Leveraging Bicep modules empowers teams to encapsulate and reuse resource configurations effectively, promoting consistency, modularity, and collaboration. As organizations strive to adhere to best practices for security and compliance, the use of private container registries for managing and distributing container images has gained significant traction. This introduces a layer of complexity to the deployment pipeline, as the integration of Bicep modules with private registries demands careful orchestration and configuration.
In this article, we delve into the fusion of these two crucial concepts: Azure Bicep modules and private registries. We will explore the benefits of utilizing Bicep modules in Azure deployments and discuss the challenges and solutions for incorporating private registries into the equation. Complete setup will be described in detail on how to develop Azure Bicep modules using Azure DevOps, push them to Azure Container Registry while using proper versioning, and consume the modules
Pre-requisites
- Visual Studio Code or similar
- Az Bicep
- Azure DevOps instance
- Valid Azure Subscription
Code
The public code repository can be found here:
Structure
Our Azure Repo will have the following structure
Core
• main.bicep
Modules
• serverfarms
Pipelines
• pipelineScripts
• azure-pipeline.yml
• build-validation.yml
• genrateReadme.ps1
Tests
• serverfarms
• bicepconfig.json
Core
The core folder is container main.bicep which is used for deploying Azure Container Registry in two instances. This is a regular bicep deployment where you can use built-in tasks in Azure DevOps for deploying Azure Resources. This part is very well described in Microsoft documentation and will be omitted in this article. More about it can be found here:
Modules
The folder ‘modules’ contains a subfolder for each of the modules that we are developing. So, based on the number of resources that you want to create and consume as modules, how many subfolders you will have within this folder? In our case, we have only one module called ‘serverfarms’ and main.bicep file where we define our resources. Each time you want to update the module, you need to upgrade the version as well. By using this approach, we make sure that the current usage of the module is not disturbed and multiple versions are available at the same time.
The module development depends on your organization's needs so you can use the parameters option to limit specific types based on your organizational needs. In the following example, resource appServicePlan accepts only Windows and Linux values for serverOS. Using default values is recommended if you have a basic template on the most common usage for that parameter.
Besides main.bicep file, each module contains a metadata.json file that describes the basic details of our module. This file will be crucial for our versioning process which will be described later.
- Version, major and minor
- Item Display Name
- Description
- Summary – points to official documentation
In order to automatically generate a README file for each module, we will use a .ps1 script created by Tao Yang which can be found here. It is simple and fast, so you can avoid creating documentation for each module manually. It combines your metadata.json and main.bicep into a well-defined README.md file. Open your folder in the repo where your script is found and run:
./generateBicepReadme.ps1 -templatePath ../modules/
For each module, just change the path or module folder name. In this case, we have added the script .ps1 to the pipelines folder. The final outcome looks like this:
Tests
The test folder needs to have the same structure as your modules folder, so each module is a separate subfolder. This is a convenient way for scalability and easier development activities.
We are not using a separate config file for deploying as the testing is performed in only one environment. The container registry is showing still ‘red’ as we have not deployed the 0.1 version yet to our Azure Container Registry.
Pipelines
Our setup will have two pipelines: Build Validation and CI-CD.
Variable groups:
- Prod
- Test
These variable groups contain corresponding variables such as serviceConnectionName, resourceGroupName, acrServer, acrServerName, environmentName, etc. None of the variables contain sensitive information, so you can push them to the YAML file as well, or keep them in the variable group for reusability across multiple pipelines.
Service connections:
- azdoteam-bicep-prod
- azdoteam-bicep-test
- azdoteam-bicep-dev
Each of the variable groups uses a service principal for authentication and has access to only one group. More about service connections and service principal authentication can be found here.
Environments
- SyVe-Prod
- SyVe-Test
Build Validation
- Pull request build validation pipeline
- Executed when merging from any to dev, or dev to the main branch
- Check version
- Build modules (Lint)
build-validation.yml contains Azure Pipeline definition with 3 tasks:
- Checkout
- Get changed file and version
- Calling checkModuleVersion.ps1 file
- Build changed files
- Calling buildModules.ps1
Depending on whether is it the main branch or not, it uses the corresponding variable group.
In essence, the checkModuleVersion.ps1 acts as a guardian of module changes, applying systematic checks, version comparisons, and issue detection to uphold the quality and reliability of the repository's contents. Its role extends beyond mere file comparisons; it contributes to the overall stability of the development process by ensuring that modifications are accurate, versioning is coherent, and potential errors are promptly addressed.
The buildModules.ps1 script automates the building of Azure Bicep modules within a "modules" folder. It checks for changes using Git and verifies module existence in an Azure Container Registry (ACR). If changes exist and the module isn't in ACR, it builds the module using Azure Bicep CLI.
CI CD
- Executed when changes are pushed to the dev or main branch
- Build modules
- Calling buildModules.ps1
- Deploy to test (if not main branch)
- Executed after build
- Calling deployModules.ps1
- Execute tests (if not main branch)
- Executed after deploy to test
- Calling testModules.ps1 and cleanResources.ps1
- Deploy to prod (if main branch)
- Executed after build
- Calling deployModules.ps1
Azure-pipeline.yml is long and you can find it in the repository. But, in simple words, each stage has a job with AzureCLI@2 task that calls 1 or 2 (TestExecution stage) PowerShell script.
The script deployModules.ps1 automates the handling and deployment of Azure Bicep modules. It starts by checking the Azure Bicep version and exploring a designated "modules" folder where these modules are stored. For each module, the script processes its information, checks if it's in an Azure Container Registry (ACR), and identifies Git-based changes. If changes are found and the module isn't in the ACR, the script compiles the module's Bicep file and publishes it to the ACR. Throughout the process, it provides informative console updates. This script streamlines the management of Bicep modules, ensuring efficient deployment while keeping users informed.
The script testModules.ps1 automates the testing of Azure Bicep modules found in the "modules" and "tests" folders. It loops through each module, checking for changes using Git. If changes are detected, it builds and deploys the module's Bicep file to an Azure resource group. The script keeps track of processed modules and provides clear console messages about its actions. This ensures efficient testing while keeping users informed.
The script cleanResources.ps1 removes resources created within a specific Azure resource group. It starts by fetching a list of resource IDs using the Azure CLI (az resource list). The query excludes the Azure Container Registry (ACR) to avoid deletion. Then, for each resource ID, the script uses the Azure CLI to delete the resource (az resource delete). Informative messages are displayed using Write-Host indicating the deleted resource. This script effectively cleans up resources from the designated resource group. Avoiding ACR is added as a safety if for some reason you are using the same RG where your ACR is found which is not recommended.
Module development and deployment
Pull Request is created with a new module added. The build validation pipeline is triggered and completed successfully.
After all validations (peer review, work item connected, and build validation) are passed during Pull Request, the changes are merged to ‘dev’ and our regular CI CD pipeline is triggered. First, the build is performed where we are now building/linting ‘dev’ branch. After that, we publish new bicep modules to ‘syvemodulestest.azurecr.io’ registry.
After our module is published, our ‘tests’ can be executed. In this case, Attempt 1 has failed since the service principal which is being used in this stage did not have proper access to the ACR syvemodulestest.azurecr.io so it could not pull the modules. The access is given and then Attempt 2 is completed successfully. Clean resources are being performed every time, no matter what the outcome of ‘Test changed files’ since we do not want our resources to be running in Azure as it generates cost depending on resource type and configuration.
Using the same approach, you can create a PR from ‘dev’ to ‘main’ and push the modules to your production environment of ACR.
Conclusion
In conclusion, this article has explored the integration of Azure Bicep modules and private container registries within the context of Azure DevOps. By leveraging Bicep modules, development, and operations teams can enhance their resource configuration practices, fostering consistency, modularity, and collaboration. The fusion of Azure Bicep modules and private container registries empowers teams to achieve efficient, secure, and scalable deployments within their organization. By following the outlined structure, utilizing automation scripts, and adhering to the defined pipeline processes, organizations can elevate their deployment practices to new heights of efficiency and reliability, ultimately driving successful Azure-based projects. The final result is that you have a versioned Azure Bicep module in your ACR that can be consumed by all other members of your organization.
