Managed DevOps Pools

Written by Ahmed Babić | Feb 3, 2025 11:27:47 AM

Overview

In Azure DevOps, agents are the workers that run your build and deployment jobs, and there are several types you can use, each with its own set of features and benefits.

Microsoft-hosted agents are managed by Microsoft and provide a fresh VM for each job. They are easy to use, require no maintenance, and are always up to date. However, they offer limited customization, may face performance issues for complex builds, lack storage, and do not maintain a persistent state between jobs.

Self-hosted agents are managed by the user and can be hosted on their own infrastructure. They offer full control, maintain a persistent state between jobs, and can be tailored to specific needs. The trade-off is higher maintenance, setup complexity, and potential security risks. They require regular updates and patches, which can become an operational overhead.

Azure VM Scale Set (VMSS) agents are self-hosted agents that can be auto-scaled using Azure VM Scale Sets. They offer flexibility in machine size and image, autoscaling capabilities, and cost efficiency. However, they require more setup and management compared to Microsoft-hosted agents and can be complex to configure, potentially becoming a bottleneck in your automated flow due to maintenance needs.

All of these are available for a long time in Azure as different ways to run your build and deployment jobs. However, all of them have limitations and could be hard to use for advanced project setups, large organizations, and enterprises. If you want to run a secure CI CD, making sure that agents can reach the company’s network, with enough scaling options, persisting state possibilities, easy to maintain but still cost-efficient, you would not be able to get it before mid-2024. However, Microsoft has finally published a product that would solve the typical issues we mentioned above, called Managed DevOps Pools (MDP).

Let’s take a look to its main strengths:

Powerful agents – you can choose which size to use
Custom image – you can use default ms hosted images, but also bring your own
Region – deploy it close to your resources
Scalability – fine-tuning available to scale up and down
Persisting state – you can persist the state of the agent for the next run, instead of a fresh new state
Long jobs – run jobs for two days
Network – you can reach your corporate network

And many others. But all this info is available on official Microsoft documentation, so let’s not go too much into the details. Let’s now focus on how to deploy this in Azure using Azure Bicep.

Deploy MDP using Bicep and Azure DevOps

There are some prerequisites for deploying MDP in Azure and connecting it to the Azure DevOps organization of your choice.

Pre-requisites

Azure Subscription
Azure Resource Group
Azure DevOps Organization

Implementation

The first thing to start with is to define which resources we need to create in Azure

Dev Center
Dev Center Project
Virtual Network
Subnet
Managed DevOps Pool

After that, we need some Azure DevOps resources

Azure Repo – to store the code
Azure Pipeline (YAML) – to build and deploy
Azure DevOps Service Connection – to connect Azure DevOps and Azure
1. Note: Use Workload Identity Federation
Environment – to control the environment and deployments

Here are the Dev Center and Project definitions:

1. param devCenterName string
2. param devCenterProjectName string
3. param location string = resourceGroup().location
4.
5. resource devCenter 'Microsoft.DevCenter/devcenters@2024-02-01' = {
6. name: devCenterName
7. location: location
8. }
9.
10. resource devCenterProject 'Microsoft.DevCenter/projects@2024-02-01' = {
11. name: devCenterProjectName
12. location: location
13. properties: {
14. description: 'AzDO MDP'
15. devCenterId: devCenter.id
16. }
17. }

The next is to define the Virtual Network and Subnet:

1. param virtualNetworkName string
2. param subnetName string
3. param location string = resourceGroup().location
4.
5. resource virtualNetwork 'Microsoft.Network/virtualNetworks@2023-09-01' = {
6. name: virtualNetworkName
7. location: location
8. properties: {
9. addressSpace: {
10. addressPrefixes: [
11. '10.0.0.0/16'
12. ]
13. }
14. subnets: [
15. {
16. name: subnetName
17. properties: {
18. addressPrefix: '10.0.1.0/24'
19. }
20. }
21. ]
22. }
23. }
24.

Now, our main jewel, Manged DevOps Pool:

1. param location string = resourceGroup().location
2. param managedDevOpsPoolName string
3. param devCenterName string
4. param azureDevOpsOrganizationName string
5. param virtualNetworkName string
6. param subnetName string
7.
8. resource devCenterProject 'Microsoft.DevCenter/projects@2024-02-01' existing = {
9. name: devCenterName
10. }
11.
12. resource vnet 'Microsoft.Network/virtualNetworks@2024-01-01' existing = {
13. name: virtualNetworkName
14. }
15.
16. resource subnet 'Microsoft.Network/virtualNetworks/subnets@2024-01-01' existing = {
17. name: subnetName
18. parent: vnet
19. }
20.
21. resource managedDevOpsPool 'Microsoft.DevOpsInfrastructure/pools@2024-04-04-preview' = {
22. name: managedDevOpsPoolName
23. location: location
24. properties: {
25. organizationProfile: {
26. organizations: [
27. {
28. url: 'https://dev.azure.com/${azureDevOpsOrganizationName}'
29. parallelism: 1
30. }
31. ]
32. permissionProfile: {
33. kind: 'CreatorOnly'
34. }
35. kind: 'AzureDevOps'
36. }
37. devCenterProjectResourceId: devCenterProject.id
38. maximumConcurrency: 1
39. agentProfile: {
40. kind: 'Stateless'
41. resourcePredictionsProfile: {
42. kind: 'Automatic'
43. predictionPreference: 'Balanced'
44. }
45. }
46. fabricProfile: {
47. sku: {
48. name: 'Standard_D2ads_v5'
49. }
50. images: [
51. {
52. aliases: [
53. 'ubuntu-22.04'
54. 'ubuntu-22.04/latest'
55. ]
56. wellKnownImageName: 'ubuntu-22.04'
57. }
58. ]
59. osProfile: {
60. logonType: 'Service'
61. }
62. storageProfile: {
63. osDiskStorageAccountType: 'StandardSSD'
64. }
65. networkProfile: {
66. subnetId: subnet.id
67. }
68. kind: 'Vmss'
69. }
70. }
71. }
72.

Since it is a good practice to use modules in Azure Bicep, we will define our main.bicep file where we will call the modules – resource definitions:

main.bicep

1. param devCenterName string
2. param devCenterProjectName string
3. param managedDevOpsPoolName string
4. param azureDevOpsOrganizationName string
5. param virtualNetworkName string
6. param subnetName string
7.
8. module vnet './vnet.bicep' = {
9. name: virtualNetworkName
10. params: {
11. virtualNetworkName: virtualNetworkName
12. subnetName: subnetName
13. }
14. }
15.
16. module devCenter './devCenter.bicep' = {
17. name: devCenterName
18. params: {
19. devCenterName: devCenterName
20. devCenterProjectName: devCenterProjectName
21. }
22. }
23.
24. module managedDevOpsPool './managedDevOpsPool.bicep' = {
25. name: managedDevOpsPoolName
26. params: {
27. managedDevOpsPoolName: managedDevOpsPoolName
28. devCenterName: devCenter.name
29. azureDevOpsOrganizationName: azureDevOpsOrganizationName
30. subnetName: subnetName
31. virtualNetworkName: vnet.name
32. }
33. }

In this case, use main.dev.bicepparam file to pass the parameters to the main.bicep during deployment. This is a good practice since it allows you multi-environment control.

Now we move to our Azure DevOps part. Creation of Repository, Service Connection, and Environment will not be covered here, but take a look into official documentation on these topics if you have issues. However, the azure pipeline definition in YAML will be covered here:

1. trigger:
2. - master
3.
4. pool:
5. vmImage: 'ubuntu-latest'
6.
7. variables:
8. - name: resourceGroupName
9. value: 'VALUE'
10. - name: serviceConnectionName
11. value: 'VALUE
12. - name: bicepFile
13. value: 'main.bicep'
14. - name: parametersFile
15. value: 'main.dev.bicepparam'
16. - name: environmentName
17. value: ' VALUE '
18.
19. stages:
20. - stage: build
21. displayName: 'Validate Bicep Code'
22. dependsOn: []
23. jobs:
24. - job: validation
25. displayName: 'Validation'
26. steps:
27. - script: |
28. az bicep build --file $
29. name: LintBicepCode
30. displayName: Run Bicep linter
31.
32. - stage: whatIf
33. displayName: 'Run What-If'
34. dependsOn: build
35. condition: succeeded()
36. jobs:
37. - job: PreviewAzureChanges
38. displayName: Preview Azure changes
39. steps:
40. - task: AzureCLI@2
41. name: RunWhatIf
42. displayName: Run what-if
43. inputs:
44. azureSubscription: $
45. scriptType: 'bash'
46. scriptLocation: 'inlineScript'
47. inlineScript: |
48. az deployment group what-if \
49. --resource-group $ \
50. --template-file $ \
51. --parameters $
52. - stage: deploy
53. displayName: 'Deploy $'
54. dependsOn: whatIf
55. condition: succeeded()
56. jobs:
57. - deployment: Deploy
58. displayName: Deploy
59. environment: $
60. strategy:
61. runOnce:
62. deploy:
63. steps:
64. - checkout: self
65. - task: AzureCLI@2
66. name: Deploy
67. displayName: Deploy
68. inputs:
69. azureSubscription: $
70. scriptType: 'bash'
71. scriptLocation: 'inlineScript'
72. inlineScript: |
73. az deployment group create \
74. --resource-group $ \
75. --template-file $ \
76. --parameters $

Potential obstacles

When trying to deploy, I needed to take care of a few more things:

Register Microsoft.DevCenter provider in the subscription
Register Microsoft.DevOpsInfrastructure provider in the subscription
Assign Reader and Network Contributor access to the ‘DevOpsInfrastructure’ service principal on my virtual network
If using a service principal to deploy, add that service principal as a Stakeholder to Azure DevOps and assign agent pool administrator permissions. This is required since the Agent pool will be auto-created using that account.

Result

Azure Pipeline:

Azure Resources:

Agent Pool:

Now you can use this agent pool in your Azure Pipelines to build and deploy. Another concept here mentioned is What-if - good practice to use when working with Azure Bicep where you can preview which changes will be applied to your Azure Resources, such as creating resources, modifying, deleting, or ignoring. After that, approval is required before deploying to the environment. This allows a pre-check of changes that are going to be applied to your Azure Resources. On top of that, prior to the deployment, you can run template analyzer or similar tools to scan the bicep/arm files.

Next Steps

In case you have some issues accessing your resources due to policy restrictions, you should configure NSG rules so that the MDP agent from this VNet can reach VNet where your resources are. This can be configured with inbound and outbound Network Security Group rules on both VNets. Another concept is fine-tuning your MDP resource, where you can scale a number of agents manually or automatically, you can define different images, SKU sizes or other properties.

Hopefully, this will be useful for you who are currently struggling with agents in Azure DevOps.

View full post