Project 8: Deploying Azure Resources with Terraform and Azure DevOps

Introduction  

This blog walks through building an Azure infrastructure deployment pipeline using Terraform and Azure DevOps. It covers project setup, repository import, self-hosted agent installation, Terraform configuration, service principal creation, remote state backend setup, RBAC, pipeline execution (apply/destroy), and cleanup. Screenshots are useful for each step — placeholders are provided throughout so you can add captures of your environment.

Today, we are going to learn how to implement resources in Azure using Azure DevOps and Terraform.

Prerequisites – Collecting Required Details

1. Login to Azure Portal

• URL: portal.azure.com
• Get your Subscription details:
• Subscription ID: e11XXXXXXXXXXXXXXXXf8

2. Create App Registration

• Name: Terraform_App
• Supported account types: Accounts in this organizational directory only (Default Directory – Single tenant)
• From the Overview page, copy these details:
• Application (client) ID: bda97fce-800b-46c5-8994-67cd7e7cf928
• Directory (tenant) ID: 60e4e629-0e85-4cc5-b3d9-26d28597d8ec

Why App Registration is required?
Terraform needs a way to authenticate and interact with Azure on your behalf. App Registration in Azure AD creates a Service Principal (an identity) that Terraform will use to log in programmatically. Instead of using your personal account, this ensures secure automation with proper RBAC permissions.

3. Create Client Secret

• Navigate to Terraform_App → Certificates & Secrets
• Add new client secret:
• Description: Terraform-secret
• Expires: 90 Days

• Copy the values:
• Value: Rok8Q~D8ml.Rr7QVKWh-HGhtNdhEXTtRoW4INaKN
• Secret ID: 0673b83c-f765-4e39-a65b-ba4023a4820e

Why Certificates & Secrets are required?
The Client Secret acts like a password for the App Registration (Service Principal). When Terraform connects to Azure, it uses:
Tenant ID
Client ID
Client Secret
Subscription ID
Together, these allow Terraform to securely authenticate without needing user credentials.

Resource Group & Storage Setup

4. Create Resource Group (to manage resources easily)

• Subscription: Production Subscription
• Resource group name: RG-Terraform
• Region: East US

5. Create Storage Account

• Name: terraformstorage12
• Type: Azure Blob Storage or Data Lake Gen2

6. Create Container inside Storage

• Storage Account: terraformstorage12
• New container: tfcontainer

Why Storage Account & Container are required?
Terraform needs to keep track of the infrastructure state (terraform.tfstate file).
The Storage Account provides a reliable backend to store this file.
The Container inside it acts as a folder where Terraform keeps environment-specific state.
This ensures multiple team members can share and lock the same state, avoiding conflicts and inconsistencies.

✅ Note: Resource creation completed.

7. Assign Contributor Role to App

• Grant Terraform_App Contributor rights on the Storage Account.

8. Assign Contributor Role at Subscription Level

• Grant Terraform_App Contributor rights at Subscription scope.

Connecting Azure DevOps

9. Create PAT (Personal Access Token)

• Login to Azure DevOps Portal.
• Create token:
• Name: PAT-Terraform
• Organization: vallabhdarole
• Expiration: 30 Days
• Scopes: Full Access

• Copy the token now!
• Example: 5ZbP2AMhF9os7RHsn6vbc0PJaqGWORvNwjieV8YkDaAIMetiNsy1JQQJ99BIACAAAAAAAAAAAAASAZDO2rTv

10. Create Project in Azure DevOps

• URL: https://dev.azure.com/vallabhdarole/
• Project Name: Azure-Terraform-Infrastructure
• Description: Automates Azure infrastructure provisioning using Terraform with CI/CD in Azure DevOps. Supports environment workspaces (uat/prod), remote tfstate in Azure Storage, secure service principal authentication via pipeline secret variables, and production approvals. Includes reusable modules, runbook for agent setup, and RBAC role assignments to enforce least privilege.

Setting up VM for Agent

11. Create VM in Azure Portal

• VM Name: az-linux
• Size: Standard DS1 v2 (1 vCPU, 3.5 GiB memory)

12 Configure Agent on VM

$ mkdir myagent && cd myagent
$ wget https://download.agent.dev.azure.com/agent/4.261.0/vsts-agent-linux-x64-4.261.0.tar.gz
$ tar -zxvf vsts-agent-linux-x64-4.261.0.tar.gz
$ ./config.sh

Accept Agreement (Y)
Enter server URL: https://dev.azure.com/vallabhdarole/
Auth type: PAT
Enter PAT token
Configure agent details (pool, name, work folder).

13. Start Agent Service

$ sudo ./svc.sh install 
$ sudo ./svc.sh start 
$ sudo ./svc.sh status

Why is an Agent required?

In Azure DevOps, agents are the backbone of executing pipelines. They run the jobs defined in your pipeline (build, test, deploy, Terraform, etc.). There are two main types of agents:

• Microsoft-Hosted Agents
• Provided and maintained by Microsoft.
• Pre-installed with common tools (e.g., .NET, Node.js, Python, Terraform, Docker, Azure CLI).
• Automatically updated.
• Billed per minute of usage.
• Great for: Quick builds, short-lived jobs, proof-of-concepts, or if you don’t want to manage infrastructure.
• Limitation: Limited customization and performance. Each run starts on a new VM, so no caching beyond a single run.
• Self-Hosted Agents
• You set up and manage the machine (VM or on-prem server).
• Full control over the environment (you decide tools, versions, security hardening).
• Persistent state (can cache builds/artifacts).
• Cost-effective if you run pipelines frequently (no per-minute billing).
• Great for: Terraform deployments, long-running builds, customized environments, restricted networks, or when you need specific dependencies not included in Microsoft-hosted images.
• Limitation: You are responsible for patching, updates, and availability.
• Deployment Groups / Deployment Agents
• Special kind of self-hosted agent that is targeted for release pipelines.
• Used for multi-machine deployments (e.g., deploy app to a fleet of VMs or servers).
• Each target machine runs a deployment agent that listens for jobs from Azure DevOps.
• Great for: Rolling updates, blue-green deployments, or hybrid scenarios where resources are not directly exposed.
• Containerized Agents (Advanced use case)
• Run agents inside Docker containers.
• Useful for testing multiple versions of tools or dependencies without polluting the host system.
• Easily scalable in Kubernetes clusters.
• Great for: Dynamic workloads, testing with different toolchains, or ephemeral CI/CD runners.

Why did we use a Self-Hosted Agent here?

Since Terraform requires storing state, handling secrets, and running long-running deployments, a self-hosted Linux VM agent is more reliable. It gives us:

• Persistent caching of Terraform modules and plugins.
• Control over installed versions (Terraform, Azure CLI).
• Ability to connect securely to Azure resources within a VNET.
• It pulls jobs from Azure DevOps.
• Runs Terraform commands inside your VM.
• Reports results (success/failure) back to Azure DevOps.
• This way, DevOps pipelines can automate deployments reliably.

Install Terraform

14. Install Terraform

$ sudo yum install -y yum-utils 

$ sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo 

$ sudo yum install -y terraform 

$ terraform --version

Example Output: Terraform v1.13.2

Configure Azure DevOps Pipelines

14. Verify Agent

• Go to Azure DevOps → Project Settings → Pipelines → Agent Pools
• Confirm az-linux agent is active.

15. Import Repository

• Go to Repos → Files → Import
• Repository URL: https://github.com/vdarole/Terraform_project

• Verify backend.tf contains correct values:
• resource_group_name
• storage_account_name
• container_name

16. Configure Pipeline Variables

• Go to Pipelines → Library → Add Variable Group
• Name: terraform-secrets
• Add values:
• ARM_CLIENT_ID = bda97fce-800b-46c5-8994-67cd7e7cf928
• ARM_CLIENT_SECRET = Rok8Q~D8ml.Rr7QVKWh-HGhtNdhEXTtRoW4INaKN
• ARM_SUBSCRIPTION_ID = e113ed96-0aeb-4c7d-aa00-8c8966c3e7f8
• ARM_TENANT_ID = 60e4e629-0e85-4cc5-b3d9-26d28597d8ec

16. Create Environments

• UAT

• Prod (with approval gates enabled)

Running the Pipeline

18. Run Pipeline

• First run will create UAT Resource Group via Terraform.
• Approvals may be required.

19. Promote to Production

• After UAT success, trigger Production deployment.
• Approvals required again before provisioning.

20. Destroy Resources

• Once tested, destroy UAT and Production resource groups using Terraform destroy.

Conclusion

You have successfully implemented Azure Infrastructure Automation using Azure DevOps + Terraform.

This setup ensures:

• Secure authentication with Service Principal (App Registration + Secret)
• Remote state storage in Azure Blob (Storage Account + Container)
• CI/CD-driven provisioning with approvals
• Agent-based automation for reliable deployments
• Easy resource cleanup via Terraform destroy

Closing notes

Ensure that you delete Resource Group, VM , PAT . Project or else Azure will keep you charging for resources utilization