1.4.5.1. Continuous Integration Pipelines

Currently, Continuous Integration pipelines have become a standard configuration for many software projects. They serve as the foundation for advancing what we refer to as large-scale Terraform module governance. Without continuous integration pipelines, we cannot enforce that all modules adhere to the various standards and specifications we have established.

There are many excellent Continuous Integration (CI) solutions available on the market. Considering that the Terraform Registry syncs Terraform modules from GitHub repositories by default, and that GitHub Actions itself is a very mature and widely used pipeline solution, this book will focus primarily on GitHub Actions.

Intuitively, we might utilize various mature Actions to stitch together a simple pipeline, for example:

name: Terraform CI

on:
  pull_request:

jobs:
  terraform:
    name: Terraform Validation and Planning
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Terraform
        uses: hashicorp/setup-terraform@v3

      - name: Terraform Validate
        run: terraform validate

      - name: Terraform Plan for each example
        run: |
          has_error=false

          if [ ! -d "examples" ]; then
            echo "No 'examples' folder found."
            exit 0
          fi

          cd examples

          for d in */; do
            if [ -d "$d" ]; then
              if ls "$d"/*.tf > /dev/null 2>&1; then
                echo "==> Checking $d"
                cd "$d" || { echo "Failed to enter directory $d"; has_error=true; cd ..; continue; }

                terraform init -input=false || has_error=true
                terraform plan -input=false || has_error=true

                cd ..
              else
                echo "No Terraform configuration files found in $d, skipping."
              fi
            fi
          done

          cd ..

          if [ "$has_error" = true ]; then
            echo "At least one 'examples' folder failed."
            exit 1
          fi

          echo "All 'examples' folders passed."
          exit 0

The workflow above might essentially function when maintaining a single Terraform module code repository. However, when we need to maintain hundreds of module repositories simultaneously, this method exposes the following issues:

  • Duplicate Maintenance: Every time the workflow logic needs updating, it must be manually modified in all repositories. The workload is enormous and error-prone.
  • Consistency Issues: Manual updates may lead to inconsistent workflow versions across different repositories, increasing maintenance difficulty.
  • Poor Scalability: It is difficult to quickly introduce new features or best practices because they need to be implemented one by one in all repositories.

Imagine if we decide to add a check before terraform init to confirm that all Terraform code has been correctly formatted; we would need to submit the same change to hundreds of code repositories, which would be a terrible experience.

1.4.5.1.1. Reusable Workflows and Composite Actions

GitHub Actions provides two mechanisms to achieve workflow reuse and modularity:

1.4.5.1.1.1. Reusable Workflows

Reusable workflows allow us to define an entire workflow as an independent YAML file and reference it in other workflows using the uses keyword. This enables multiple repositories to share the same workflow definition, achieving centralized management.

Advantages:

  • Centralized Management: All workflow logic is centralized in one repository, making it easy to maintain and update.
  • Reduce Duplication: Avoid copying the same workflow code across multiple repositories.
  • Version Control: Workflow versions can be controlled via Git tags or branches, ensuring different projects use compatible versions.

Usage:

  1. Create a reusable workflow file in a repository (e.g., .github/workflows/terraform.yml) and add the workflow_call trigger to the on field.
  2. In the workflows of other repositories, reference this reusable workflow using the uses keyword, for example:
jobs:
  terraform:
    uses: org-name/repo-name/.github/workflows/terraform.yml@v1
    secrets: inherit

In this way, all repositories referencing this workflow will automatically use the latest definition, simplifying maintenance work.

1.4.5.1.1.2. Composite Actions

Composite actions allow us to combine multiple steps into a single custom action that can be called within the steps of a workflow. This is very useful for encapsulating common step combinations, such as Terraform initialization, formatting checks, and validation.

Advantages:

  • Modularity: Combine common steps into one action for easy reuse.
  • Simplify Workflows: Calling composite actions makes workflow files more concise.
  • Easy to Maintain: Updating the logic of a composite action automatically affects all workflows referencing it.

Usage:

  1. Create a composite action directory in the repository (e.g., .github/actions/terraform-setup), and create an action.yml file inside it to define the inputs and steps of the action.
  2. In a workflow, call this composite action using the uses keyword, for example:
steps:
  - uses: ./.github/actions/terraform-setup

Through this approach, we can encapsulate common operations into modules for reuse by multiple workflows, improving workflow maintainability and consistency.

1.4.5.1.2. Centralized Actions Repository

If we simply save the definition YAML files for reusable workflows and composite actions in every Terraform module's code repository, we haven't actually improved anything; every update still requires upgrading hundreds of code repositories simultaneously. We can adopt a centralized management strategy by storing reusable workflows and composite actions in a dedicated repository. Thus, all module repositories can reference these centrally managed workflows, achieving a unified CI/CD process.

1.4.5.1.3. Case Studies: Centralized Management of GitHub Actions Workflows for Terraform Modules

When maintaining Terraform modules at scale, centrally managing GitHub Actions workflows can significantly improve efficiency and consistency. The following are two organizational cases adopting a centralized management strategy:

1.4.5.1.3.1. Cloud Posse's Centralized Workflow Management

Cloud Posse centrally manages the reusable GitHub Actions workflows within its organization through the cloudposse/github-actions-workflows repository. These workflows cover various stages from build and test to deployment, available for reference by multiple repositories, ensuring the consistency and maintainability of the CI/CD process.

Additionally, Cloud Posse provides the cloudposse/github-actions-workflows-terraform-module repository, specifically designed for managing workflows for Terraform modules. By referencing these centrally managed workflows in module repositories, Cloud Posse achieves workflow standardization and centralized maintenance.

1.4.5.1.3.2. Azure Verified Modules' Scaffold Project

Azure Verified Modules provides the Azure/avm-terraform-governance project as a template for Terraform modules, containing standard GitHub Actions workflows, Makefiles, and scripts. Developers(Microsoft FTE only) can quickly create new module repositories based on this tutorial, ensuring consistency in project structure and workflows.

results matching ""

    No results matching ""