Open in app

Sign up

Sign in

Write

Sign up

Sign in

CI/CD Pipeline for Terraform Workflow Using Amazon CodeCatalyst

Luthfi Anandra
AWS in Plain English
Published in
11 min readNov 21, 2023

--

Terraform workflow can be ran using several methods. One of them is running Terraform workflow inside CI/CD pipeline. Running Terraform workflow inside CI/CD pipeline can have several benefits, such as: automate creation or provision resources, simplify collaboration between engineers/developers, etc.

Scenario

In this blog, I will explain how to run Terraform workflow inside CI/CD pipeline which in this blog is Amazon CodeCatalyst. GitHub is used as source code repository and has been connected with CodeCatalyst. Next in CodeCatalyst pipeline/workflow, We will provision resources via Terraform. In this blog, Terraform is using S3 backend. Here is the topology used in this blog:

This blog is using reference from tutorial “Bootstrapping your Terraform automation with Amazon CodeCatalyst” written by Cobus Bernard with some adjustment in configuration.

If You want to know more details about CodeCatalyst, please check CodeCatalyst documentation.

Prerequisites

  • Before starting configuration, if you have not got AWS Builder ID account, please sign up first
  • After sign up, go to Amazon CodeCatalyst web console to start project and configuration. Apply initial configuration that needed by CodeCatalyst such as:
    - Creating space
    - Connect CodeCatalyst space with AWS account
    - Create and connect IAM role that will be used for running CodeCatalyst workflow/pipeline
    - Connect GitHub source code repository
    - Define environment that will be used by workflow/pipeline
    - Please check my blog “Build and Release Container Image to Amazon Elastic Container Registry (ECR) via Amazon CodeCatalyst” to find reference for configuration above. Please check section Amazon CodeCatalyst Pipeline/Workflow Configuration
  • Prepare S3 bucket and DynamoDB lock table (optional) that will be used by Terraform backend

Configuration Steps

Directory Structure

Below is directory structure used in this blog:

.
├── .codecatalyst/
│   └── workflows/
│       └── tf-sbx2-vpc-apse1.yml
└── sandbox2/
    ├── vpc/
    │   └── ap-southeast-1/
    │       ├── main.tf
    │       ├── resources.tf
    │       └── variables.tf
    └── tf-backend/
        ├── main.tf
        ├── resources.tf
        └── variables.tf

Terraform Backend Configurations

Configuration start with provisioning resources that will be needed by Terraform backend such as S3 bucket and DynamoDB table. Beside that, We also need to prepare IAM role that will be needed to provision resources from CodeCatalyst workflow/pipeline.

  1. At first, We haven’t had any resource for storing Terraform state in S3, so that we need to run and store terraform state on our local, for example: laptop. Later, We will move that state from local to S3. We start by defining configuration in sandbox2/tf-backend/main.tf
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.11.0"
    }
  }
 
  required_version = ">= 1.3.0"
}
 
provider "aws" {
  region = var.aws_region
}

2. Define variable region in file sandbox2/tf-backend/variables.tf

variable "aws_region" {
  type        = string
  description = "AWS Region"
  default     = "ap-southeast-1"
}

3. Define resources S3, DynamoDB and IAM that needed by CodeCatalyst in file sandbox2/tf-backend/resources.tf. In this blog, IAM role CodeCatalyst has administrator access just for demo purposes. Please bear in mind to always use least privilege method on your production to prevent any security breaches

######
# S3 #
######
 
resource "aws_s3_bucket" "my_sandbox2_tfbucket" {
  bucket = "my-sandbox2-tfbucket"
 
  lifecycle {
    prevent_destroy = true
  }
}
 
resource "aws_s3_bucket_versioning" "my_sandbox2_tfbucket" {
  bucket = aws_s3_bucket.my_sandbox2_tfbucket.id
 
  versioning_configuration {
    status = "Enabled"
  }
}
 
resource "aws_s3_bucket_server_side_encryption_configuration" "my_sandbox2_tfbucket" {
  bucket = aws_s3_bucket.my_sandbox2_tfbucket.id
 
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "AES256"
    }
  }
}
 
resource "aws_s3_bucket_public_access_block" "my_sandbox2_tfbucket" {
  bucket                  = aws_s3_bucket.my_sandbox2_tfbucket.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}
 
############
# DynamoDB #
############
 
resource "aws_dynamodb_table" "my_sandbox2_tflocks" {
  name         = "my-sandbox2-tflocks"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"
 
  attribute {
    name = "LockID"
    type = "S"
  }
}
 
#######
# IAM #
#######
 
data "aws_iam_policy_document" "codecatalyst_assume_role_policy" {
  statement {
    actions = ["sts:AssumeRole"]
    principals {
      type = "Service"
      identifiers = [
        "codecatalyst.amazonaws.com",
        "codecatalyst-runner.amazonaws.com"
      ]
    }
  }
}
 
resource "aws_iam_role" "codecatalyst_admin" {
  name               = "codecatalyst-admin"
  assume_role_policy = data.aws_iam_policy_document.codecatalyst_assume_role_policy.json
}
 
resource "aws_iam_role_policy_attachment" "codecatalyst_admin" {
  role       = aws_iam_role.codecatalyst_admin.name
  policy_arn = "arn:aws:iam::aws:policy/AdministratorAccess"
}

4. Refer AWS keys that needed for running terraform workflow commands, for example by exporting environment variables as explained in this document

5. Run terraform cli commands that needed for provisioning resources

terraform init
terraform plan
terraform apply

6. Verify terraform apply is successful and resources successfully provisioned

7. Back to file sandbox2/tf-backend/main.tf, add additional configurations related to S3 backend

###########################
# Terraform Configuration #
###########################
 
terraform {
  backend "s3" {
    bucket         = "my-sandbox2-tfbucket"
    key            = "path/to/terraform.tfstate"
    region         = "ap-southeast-1"
    dynamodb_table = "my-sandbox2-tflocks"
    encrypt        = true
  }
}

8. Run terraform init command again. Terraform will detect changes related to S3 backend and will move terraform state to S3 bucket that has been defined. Choose yes if asked for input

terraform init

Initializing the backend...
Do you want to copy existing state to the new backend?
  Pre-existing state was found while migrating the previous "local" backend to the
  newly configured "s3" backend. No existing state was found in the newly
  configured "s3" backend. Do you want to copy this state to the new "s3"
  backend? Enter "yes" to copy and "no" to start with an empty state.

  Enter a value: yes

Releasing state lock. This may take a few moments...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

9. Go to Amazon Web Services (AWS) web console, go to S3 service menu. Verify terraform.tfstate file has been copied to defined S3 bucket/path

CodeCatalyst Workflow File Configurations

After configurations for terraform backend has been finished as described above, next we continue configure CodeCatalyst workflow/pipeline.

Before starting workflow configuration, please make sure to configure all configurations that needed by CodeCatalyst as mentioned in Prerequisites.

As mentioned in document Build, test, and deploy with workflows in CodeCatalyst, CodeCatalyst use workflow definition file as reference workflow/pipeline configuration and this file is saved in directory ~/.codecatalyst/workflows/.

In this blog, I will create basic resources that needed by VPC on AWS. As mentioned on Directory Structure above, pipeline will be ran when there is changes inside directory sandbox2/vpc/ap-southeast-1.

Below is full content of workflow file codecatalyst/workflows/tf-sbx2-vpc-apse1.yml that used as example in this blog.

Name: tf-sbx2-vpc-apse1
SchemaVersion: "1.0"
 
Triggers:
  - Type: PUSH
    Branches:
      - master
    FilesChanged:
      - sandbox2\/vpc\/ap-southeast-1\/.*
 
Actions:
  terraform-plan:
    Identifier: aws/build@v1
    Inputs:
      Sources:
        - WorkflowSource
    Environment:
      Name: sandbox
      Connections:
        - Name: lanandra-sandbox
          Role: codecatalyst-admin
    Configuration:
      Container:
        Registry: DockerHub
        Image: hashicorp/terraform:1.5.7
      Steps:
      - Run: cd sandbox2/vpc/ap-southeast-1
      - Run: terraform fmt -check -no-color
      - Run: terraform init -no-color
      - Run: terraform validate -no-color
      - Run: terraform plan -no-color -input=false
    Compute:
      Type: EC2
  wait-period:
    DependsOn:
      - terraform-plan
    Identifier: aws/build@v1
    Inputs:
      Sources:
        - WorkflowSource
    Environment:
      Name: sandbox
      Connections:
        - Name: lanandra-sandbox
          Role: codecatalyst-admin
    Configuration:
      Steps:
      - Run: echo "Please wait for a while before terraform apply"
      - Run: echo "If you wish to cancel terraform apply, please stop this run"
      - Run: sleep 60
    Compute:
      Type: EC2
  terraform-apply:
    DependsOn:
      - wait-period
    Identifier: aws/build@v1
    Inputs:
      Sources:
        - WorkflowSource
    Environment:
      Name: sandbox
      Connections:
        - Name: lanandra-sandbox
          Role: codecatalyst-admin
    Configuration:
      Container:
        Registry: DockerHub
        Image: hashicorp/terraform:1.5.7
      Steps:
      - Run: cd sandbox2/vpc/ap-southeast-1
      - Run: terraform init -no-color
      - Run: terraform apply -auto-approve -no-color -input=false
    Compute:
      Type: EC2

Next, I will explain more detail each sections that defined on workflow file above.

  1. Define workflow name. Then define how workflow will be ran. In this blog, workflow will be ran when there is a push to master branch but only ran when changes is under directory sandbox2/vpc/ap-southeast-1/ (regular expression or regex is used to detect changes)
Triggers:
  - Type: PUSH
    Branches:
      - master
    FilesChanged:
      - sandbox2\/vpc\/ap-southeast-1\/.*

2. Define action. Action will be divided into 3 sections. The first one is used to run workflow terraform plan. In this section, action uses identifier aws/build@v1. Then action will be ran on sandbox environment that has been connected with AWS account and also has IAM role associated with that account. In Configuration section, has been defined that action will be ran on top of container using specific version of terraform public image that pulled from DockerHub. Also has defined working directory and several terraform commands which include terraform plan. Lastly, define compute type which is EC2

Actions:
  terraform-plan:
    Identifier: aws/build@v1
    Inputs:
      Sources:
        - WorkflowSource
    Environment:
      Name: sandbox
      Connections:
        - Name: lanandra-sandbox
          Role: codecatalyst-admin
    Configuration:
      Container:
        Registry: DockerHub
        Image: hashicorp/terraform:1.5.7
      Steps:
      - Run: cd sandbox2/vpc/ap-southeast-1
      - Run: terraform fmt -check -no-color
      - Run: terraform init -no-color
      - Run: terraform validate -no-color
      - Run: terraform plan -no-color -input=false
    Compute:
      Type: EC2

3. Next action that defined is called wait-period. This action is defined so that there is an interlude between terraform plan and terraform apply. In case there is something that unwanted in terraform plan, so that we have a time to not continue to next action or process which is terraform apply. This action is workaround because by the time this blog is released, there is no out of the box solution provided by CodeCatalyst, for example manual approval. So in case there is something unwanted, we can abort the workflow. In this action, I created a logic to pause the workflow for 60 seconds and this action is depends on previous action which is terraform plan

wait-period:
    DependsOn:
      - terraform-plan
    Identifier: aws/build@v1
    Inputs:
      Sources:
        - WorkflowSource
    Environment:
      Name: sandbox
      Connections:
        - Name: lanandra-sandbox
          Role: codecatalyst-admin
    Configuration:
      Steps:
      - Run: echo "Please wait for a while before terraform apply"
      - Run: echo "If you wish to cancel terraform apply, please stop this run"
      - Run: sleep 60
    Compute:
      Type: EC2

4. Next action that defined is action to run terraform apply. This action more or less similar with action terraform-plan, the difference is just terraform command that declared on configuration. In this section, action will run commands that needed for terraform apply. This action is depends on previous action which is wait-period

terraform-apply:
    DependsOn:
      - wait-period
    Identifier: aws/build@v1
    Inputs:
      Sources:
        - WorkflowSource
    Environment:
      Name: sandbox
      Connections:
        - Name: lanandra-sandbox
          Role: codecatalyst-admin
    Configuration:
      Container:
        Registry: DockerHub
        Image: hashicorp/terraform:1.5.7
      Steps:
      - Run: cd sandbox2/vpc/ap-southeast-1
      - Run: terraform init -no-color
      - Run: terraform apply -auto-approve -no-color -input=false
    Compute:
      Type: EC2

Example of Terraform Workflow Process in CodeCatalyst CI/CD

After configure CodeCatalyst workflow file, next I will create example of Terraform codes. In this blog, I will create VPC resources using public module terraform-aws-modules/vpc/aws as reference. Here is the initial configuration/codes that defined:

sandbox2/vpc/ap-souhteast-1/main.tf

###########################
# Terraform Configuration #
###########################
 
terraform {
  backend "s3" {
    bucket         = "my-sandbox-tfbucket"
    key            = "path/to/terraform.tfstate"
    region         = "ap-southeast-1"
    dynamodb_table = "my-sandbox-tflocks"
    encrypt        = true
  }
}
 
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.11.0"
    }
  }
 
  required_version = ">= 1.3.0"
}
 
provider "aws" {
  region = var.aws_region
}

sandbox2/vpc/ap-souhteast-1/variables.tf

variable "aws_region" {
  type        = string
  description = "AWS Region"
  default     = "ap-southeast-1"
}
 
variable "env_name" {
  type    = string
  default = "sbx2"
}

sandbox2/vpc/ap-souhteast-1/resources.tf

#######
# VPC #
#######
 
 
data "aws_availability_zones" "available" {}
 
locals {
  azs = slice(data.aws_availability_zones.available.names, 0, 3)
 
  tags = {
    "myTag:environment" = "sbx2"
    "myTag:managedBy"   = "terraform"
  }
}
 
module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "5.1.2"
 
  name = var.env_name
  cidr = "10.255.8.0/21"
 
  azs             = local.azs
  private_subnets = ["10.255.8.0/24", "10.255.9.0/24", "10.255.10.0/24"]
  public_subnets  = ["10.255.15.0/24", "10.255.14.0/24", "10.255.13.0/24"]
 
  enable_nat_gateway = false
  single_nat_gateway = true
 
  tags = local.tags
}
  1. To conduct testing of Terraform workflow in CI/CD, I will change 1 line of code inside file sandbox2/vpc/ap-souhteast-1/resources.tf. I will activate NAT gateway with changing this line of code that current value is false
enable_nat_gateway = true

2. As defined in workflow file, all changes inside directory sandbox/vpc/ap-southeast-1 will trigger workflow run. To verify, go to CodeCatalyst web console, go to space and project where codes resided. Then go to CI/CD menu, choose Workflows. Go to tab Runs, verify there is new run that currently running.

3. Go to run details by clicking name/id of run. Then We will be redirected to overview page of run. Click action name terraform-plan, on the right will be showed tray box that display details configuration/steps of action. For example We want to see details of terraform plan, go to tab log, then click step terraform plan -no-color -input-false. Verify action succeeded

4. Verify terraform plan show expected result. Terraform will create resources releated to NAT gateway as mentioned in point number 1

5. As defined in workflow file, after terraform-plan action ran successfully, next workflow will proceed next action which is wait-period to give some interlude whether terraform plan will be executed to next action which is terraform-apply or apply will be aborted. If plan has met expectation and We want to proceed to terraform apply, no action needed, just wait 60 seconds and then verify action succeeded

6. But, if want to abort workflow and don’t want continue to terraform apply, click stop button when action still in progress

7. Let’s say We want to continue to terraform apply process, then terraform-apply action will be ran. We can see the details of the action inside tray box similar with previous actions. To verify resources has been provisioned via terraform apply, click step terraform apply -auto-approve -no-color -input=false

8. Verify terraform apply has been expected and action is succeeded

9. Go to VPC service page in AWS web console to verify NAT gateway has been provisioned successfully

Summary

We have reached the last section of this blog. Here are some key takeaways that can be summarized:

  • Amazon CodeCatalyst can act as alternative for CI/CD engine/tools that can be used to run Terraform workflow
  • Amazon CodeCatalyst use IAM role to interact with AWS services. By using this method, Terraform doesn’t need to inject static credentials such as AWS Access Key and AWS Secret Key into the pipeline. This can’t help prevent security breaches
  • Amazon CodeCatalyst can give seamless experience if We want to deploy application to AWS environments
  • But some workflow or logic haven’t been provided by CodeCatalyst by the time this blog is released. Maybe there will be some improvement in the future

Please check out my GitHub repository to see source code example that used in this blog:

Please comment if you have any suggestions, critiques, or thoughts.

Hope this article will benefit you. Thank you!

PlainEnglish.io 🚀

Thank you for being a part of the In Plain English community! Before you go:

AWS
Cicd
Codecatalyst
Terraform
Cloud Computing

--

--

Luthfi Anandra
AWS in Plain English
Follow

Written by Luthfi Anandra

168 Followers

Site Reliability Engineer | AWS Community Builder | HashiCorp Ambassador

Follow

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams