Also in this series:

Terragrunt Example

Let’s consider the situation where we want to maintain the infrastructure for a system with two major components: an API and a database solution. You must also deploy dev, test, stage, and production environments for this system. Dev and Test environments are deployed to one region, while stage and production environments are deployed to two regions.

We’ve created a preconfigured sample repository to demonstrate how we might handle something like this with Terragrunt. Now, although the requirements and scenario described above may not pertain to you, the preconfigured sample repository should give you a good idea of what you can accomplish with Terragrunt and the benefits it provides in the context of keeping your Terraform code organized. Also, keep in mind that Terragrunt is unopinionated and allows you to configure it in several ways to accomplish similar results; we will only cover a few of the benefits Terragrunt provides, but be sure to check out their documentation site for more information.

To get the most out of the code sample, you should have the following:

Run through the setup steps if you need to. This will involve running a mini terraform project to provision a few resource groups in addition to a storage account to store your terraform state files.
The sample repo contains several top-level directories:

  • /_base_modules
  • /bootstrap
  • /dev
  • /test
  • /stage
  • /prod
  • _base_modules folder – contains the top-level terraform modules that your application will use. There are subfolders for each application type, the API, and the storage solution (/api and /sql). For example, there is a subfolder for the API, which contains the terraform code for your API application, and one for SQL, which will include the terraform code for your storage/database solution; take note of the main.tf, variables.tf, and outputs.tf files in each subfolder. Each application type folder will also contain a .hcl file that contains global configuration values for all environments that consume that respective application type
  • [dev/test/stage/prod] – environment folders that contain subfolders for each application type. Each subfolder for each application type will contain Terragrunt configuration files that contain variables and inputs specific to that environment
  • Bootstrap – a small isolated terraform project that will spin up placeholder resource groups in addition to a storage account that can be used to maintain remote terraform state files

As mentioned above, there are several .hcl files in a few different places within this folder structure. These are Terragrunt configuration files. You will see one within each sub folder inside the _base_modules directory and one in every subfolder within each environment folder. These files are how Terragrunt knows what terraform commands to use, where to store each application’s remote state, and what variable files and input values to use for your terraform modules defined in the _base_modules directory. Read more about how this file is structured on Gruntwork’s website. With this sample repository, global configurations are maintained in the /_base_modules folder and consumed by configurations in the environment folders.

Let’s go over some of the basic features that Terragrunt offers.

Keeping your Remote State Configuration DRY

I immediately noticed when writing my first bits of Terraform code that I couldn’t use variables, expressions, or functions within the terraform configuration block. You can override specific parts of this configuration through the command line, but there was no way to do this from code.

Terragrunt allows you to keep your backend and remote state configuration DRY by allowing you to share the code for backend configuration across multiple environments. Look at the /_base_modules/global.hcl file in conjunction with the /dev/Terragrunt.hcl file.

/_base_modules/global.hcl:

remote_state {
  backend = "azurerm"
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite"
  }
  config = {
    resource_group_name  = "shared"
    storage_account_name = "4a16aa0287e60d48tf"
    container_name       = "example"
    key            = "example/${path_relative_to_include()}.tfstate"
  }
}

This file defines the remote state that will be used for all environments that utilize the api module. Take special note of the ${path_relative_to_include} expression – more on this later.
A remote state Terragrunt block that looks like this:

remote_state {
    backend = "s3"
    config = {
      bucket = "mybucket"
      key    = "path/for/my/key"
      region = "us-east-1"
    }
  }

Is equivalent to terraform block that looks like this:

  terraform {
    backend "s3" {
      bucket = "mybucket"
      key    = "path/to/my/key"
      region = "us-east-1"
    }
  }

To inherit this configuration into a child sub folder or environment folder you can do this:

/dev/api/terragrunt.hcl

include "global" {
  path = "${get_terragrunt_dir()}/../../_base_modules/global.hcl"
  expose = true
  merge_strategy = "deep"
}

The included statement above tells Terragrunt to merge the configuration file found at _base_modules/global.hcl with its local configuration. The ${path_relative_to_include} in the global.hcl file is a predefined variable that will return the relative path of the calling .hcl file, in this case,/dev/api/terragrunt.hcl. Therefore, the resulting state file for this module would be in the example container at dev/api.tfstate. For the sql application in the dev environment, the resulting state file would be dev/sql.tfstate; look at the _base_modules/sql/sql.hcl file. For the api application in the test environment, the resulting state file would be, test/api.tfstate. Be sure to check out all of the built-in functions Terragrunt offers out of the box.

Using the feature just mentioned, we only define the details of the remote state once, allowing us to cut down on code repetition. Read more about the remote_state and include blocks and how you can configure them by visiting the Terragrunt documentation. Pay special attention to merge strategy options, how you can override includes in child modules, and the specific limitations of configuration inheritance in Terragrunt.

Keeping your Terraform Configuration DRY

Merging of configuration files do not only apply to remote state configurations – you can also apply them to the sources and inputs of your modules.
In Terragrunt, you can define the source of your module (main.tf or top level terraform module) within the terraform block. Let’s consider the api application:

/_base_modules/api/api.hcl

terraform {
  source = "${get_terragrunt_dir()}/../../_base_modules/api"

  extra_arguments "common_vars" {
    commands = get_terraform_commands_that_need_vars()

    required_var_files = [
      
    ]
  }
}

You’ll notice this is referencing a local path; alternatively, you can also set this to use a module from a remote git repo or terraform registry.
The api.hcl configuration is then imported as a configuration into each environment folder for the api application type:

Ex. /dev/api/terragrunt.hcl

include "env" {
  path = "${get_terragrunt_dir()}/../../_base_modules/api/api.hcl"
  expose = true
  merge_strategy = "deep"
}

Include statements with specific merge strategies can also be overwritten by configurations in child modules, allowing you to configure each environment separately if needed.
Merging inputs before they are applied to your terraform module is also extremely helpful if you need to share variables across environments. For example, all the names of your resources in your project might be prefixed with a specific character set. You can define any global inputs in the inputs section of the _base_modules/global.hcl file. Because Terragrunt configuration files are written in the HCL language, you can also utilize all the expressions and functions you use in Terraform to modify or restructure input values before they are applied. Look at how we are defining the identifier input variable found in both sql and api modules:

Here is the terraform variable:

/_base_modules/api/variables.tf and /_base_modules/sql/variables.tf

variable "identifier" {
  type = object({
      primary = string
      secondary = string
      type = string
  })
}

Here is the primary property being assigned from the global env:

/_base_modules/global.hcl

... 
inputs = {
    identifier = {
        primary = "EXAMPLE"
    }
}
...

Here is the secondary property being assigned from the dev/dev.hcl file:

/dev/dev.hcl

inputs = {
 identifier = {
     secondary = "DEV"
 }
}

And here is the type property being applied in the module folders

/_base_modules/sql/sql.env.tf

...
inputs = {
    identifier = {
        type = "SQL"
    }
}

/_base_modules/api/api.hcl

...
inputs = {
    identifier = {
        type = "API"
    }
}

All configurations are included in the environment configuration files with:

include "global" {
  path = "${get_terragrunt_dir()}/../../_base_modules/global.hcl"
  expose = true
  merge_strategy = "deep"
}

include "api" {
  path = "${get_terragrunt_dir()}/../../_base_modules/api/api.hcl"
  expose = true
  merge_strategy = "deep"
}

include "dev" {
  path = "../dev.hcl"
  expose = true
  merge_strategy = "deep"
}

would result in something like:

inputs = {
    identifier = {
      primary = "EXAMPLE"
secondary = "DEV"  
type = "API"
    }
}

We utilize this pattern to share variables across all environments and applications within a specific environment without having to declare them multiple times.
It is also important to note that because Terragrunt configuration files are written in the HCL language, you can access all of Terraform’s functions and expressions. As a result, because you can inherit Terragrunt configuration files into a specific environment, you can restructure, merge, or alter input variables before they are sent to terraform to be processed.

Running Multiple Modules at once

You can also run multiple terraform modules with one command using Terragrunt. For example, if you wanted to provision dev, test, stage, and prod with one command, you could run the following command in the root directory:

terragrunt run-all [init|plan|apply]

If you wanted to provision the infrastructure for a specific tier, you could run the same command inside an environment folder (dev, test, stage etc.).
This allows you to neatly organize your environments instead of maintaining everything in one state file or trying to remember what variable, backend, and provider configurations to pass in your CLI commands when you want to target a specific environment.

It is important to note that you can maintain dependencies between application types within an environment (between the sql and api application) and pass outputs from one application to another. Look at the dev/api environment configuration file.

/dev/api/terragrunt.hcl

dependency "sql" {
  config_path = "../sql"
    mock_outputs = {
    database_id = "temporary-dummy-id"
  }
}

locals {
}
inputs = {
    database_id = dependency.sql.outputs.database_id
    ...
}

Notice that it references the dev/sql environment as a dependency. The dev/sql environment uses the _base_modules/sql application so look at that module, specifically the outputs.tf file.

/_base_modules/sql/outputs.tf

output "database_id" {
  value = azurerm_mssql_database.test.id
}

Notice that this output is being referenced in the /dev/api/terragrunt.hcl file as a dependency.

The client requirements described earlier in this post proved to be especially difficult to maintain without the benefit of being able to configure separate modules that depend on one another. With the ability to isolate different components of each environment and share their code and dependencies across environments, we could maintain multiple environments effectively and efficiently with different configurations.

Conclusion

Terraform as an infrastructure as code tool has helped us reliably develop, maintain, and scale our infrastructure demands. However, because our client work involved maintaining multiple environments and projects simultaneously, we needed specific declarative design patterns to organize our infrastructure development. Terragrunt offered us a simple way to develop multiple environments and components of a given application in a way that was repeatable and distributable to other project pipelines.

There are several features of Terragrunt we did not discuss in this post:
Before, After, and Error Hooks
Maintaining CLI flags

We would like to see some of the functionality Terragrunt offers baked into Terraform by default. However, we do not feel like Terragrunt is a final solution; Terraform is rather unopinionated and less concerned with how you set up your project structure, while Terragrunt is only slightly more opinionated in your setup. Terragrunt claims to be DRY, but there is still a lot of code duplication involved when creating multiple environments or trying to duplicate infrastructure across regions. For example, creating the folder structure for an environment is cumbersome, especially when you want to add another tier.

Part one of this series focused on several methods and patterns to make the most out of your terraform code, explicitly focusing on keeping your terraform code and configurations organized and easy to maintain.

Utilizing registries to modularize your infrastructure is only a small part of the improvements you can make to your Terraform code.

One of the significant concepts of Terraform is how it tracks the state of your infrastructure with a state file. In Terraform, you need to define a “remote state” for each grouping of infrastructure you are trying to deploy/provision. This remote state could be stored in an S3 bucket, Azure Storage account, Terraform Cloud, or another applicable service. This is how Terraform knows where to track the state of your infrastructure to determine if any changes need to be applied or if the configuration has drifted away from the baseline defined in your source code. The organization of your state files is essential, especially if you are managing it yourself and not using Terraform Cloud to perform your infrastructure runs.

Keep in mind that as your development teams and applications grow, you will frequently need to manage multiple developments, testing, quality assurance, and production environments. Several projects/components will need to be managed simultaneously, as well as the number of configurations, variable files, CLI arguments, and provider information will become untenable over time.

How do you maintain all this infrastructure reliably? Do you track all the applications for one tier in one state file? Or do you break them up and track them separately?

How do you efficiently share variables across environments and applications without defining them multiple times? How do you successfully apply these terraform projects in a continuous deployment pipeline that is consistent and repeatable for different types of Terraform projects?

At one of our current clients, we were involved with onboarding Terraform as an Infrastructure as Code (IaC) tool. However, we ran into many challenges when trying to deploy multiple tiers (dev, test, stage, production, etc.) across several workstreams, specifically in a continuous manner within a deployment pipeline.

BLOG POST
Learn how to automate a file transfer from FTP server to AWS S3 bucket using Terraform to provide you flexible means when downloading files from an FTP server.

The client I work for has the following requirements for the web UI portion of the services they offer (consider Azure Cloud provider for context):

  • Each tier has *six applications* for different areas of the United States
  • *Each application* has a web server, a redis cache, app service plan, a storage account, and a key vault access policy to access a central key store
  • Development and test tiers are deployed to a single region
  • Applications in development and test tier both share a redis cache
  • Applications in Staging and production environments have individual redis caches
  • Stage and production environments are deployed to two regions, east and central

Application Development

Stage and Production tiers have up to 48 resources, respectively; the diagram above only represents three applications and excludes some services. Our client also had several other IT services that needed similar architectural setups; most projects involved deploying six application instances (for each service area of the United States), each configured differently through application settings.

Initially, our team decided to use the Terraform CLI and track our state files using an Azure Storage Account. Within the application repository, we would store several backend.tf files alongside our terraform code for each tier and pass them dynamically to terraform init –backend-config=<PATH> when we want to initialize a specific environment. We also passed variable files to terraform [plan|apply|destroy] –var-file=<PATH> to combine common and tier-specific application setting template files. We adopted this process in our continuous deployment pipeline by ensuring the necessary authentication principals and terraform CLI packages were available on our deployment agents and then running the appropriate constructed terraform command in the appropriate directory.

This is great but presented a few problems when scaling our process. The process we used initially allowed developers to create their own terraform modules specific to their application, utilizing either local or shared modules in our private registry. One of the significant problems came when trying to apply these modules in a continuous integration pipeline. Each project had its unique terraform project and its configurations that our constant deployment pipeline needed to adhere to.

Naming Conventions

Let’s also consider something relatively simple, like the naming convention of all the resources in a particular project. Usually, you would want the same named prefix on your resources (or apply it as a resource tag) to visually identify what projects the resources belong to. Since we had to maintain multiple tiers of environments for these projects (dev, test, stage, prod), we wanted to share this variable across environments, only needing to declare it once. We also wanted to declare other variables and optionally override them in specific environments. With the Terraform CLI, there is no way to merge inputs and share variables across environments. In addition, you cannot use expressions, functions, or variables in the terraform remote state configuration blocks, forcing you to either hardcode your configuration or apply it dynamically through the CLI; see this issue.

We began to wonder if there was a better way to organize ourselves. This is where Terragrunt comes into play. Instead of keeping track of the various terraform commands, remote state configurations, variable files, and input parameters we needed to consolidate to provision our terraform projects, what if we had a declarative way of defining the configuration of our Terraform?

Terragrunt is a minimal wrapper around Terraform that allows you to dynamically assign and inherit remote state configurations, provider information, local variables, and module inputs for your terraform projects through a hierarchal folder structure with declarative configuration files. It also gives you a flexible and unopinionated way of consolidating everything Terraform does before it runs a command. Every option you pass to terraform can be specifically configured through a configuration file that inherits, merges, or overrides components from other higher-level configuration files.

Terragrunt allowed us to do these essential things:

  • Define a configuration file to tell what remote state file to save based on application/tier (using folder structure)
  • It allowed us to run multiple terraform projects at once with a single command
  • Pass outputs from one terraform project to another using a dependency chain.
  • Define a configuration file that tells terraform what application setting template file to apply to an application; we used .tpl files to
  • apply application settings to our Azure compute resources.
  • Define a configuration file that tells terraform what variable files to include in your terraform commands
  • It allowed us to merge-common input variables with tier-specific input variables with the desired precedence.
  • It allowed us to consistently name and create state files.
Defining your cloud computing infrastructure as code (IAC) is becoming an industry standard for enterprise IT teams to scale effectively as their development teams and applications grow.

Terraform, by Hashicorp, is an infrastructure as code tool that lets you define both cloud and on-prem resources using human-readable configuration files that you can version, reuse, and share across your various projects and development teams.

This post will focus on several methods and patterns to make the most out of your terraform code, explicitly focusing on keeping your terraform code and configurations organized and easy to maintain; we will strive to implement DRY principles wherever possible. As a result, this post assumes you’ve worked with Terraform before and have a general understanding of how to use it. If you want to know more about Terraform and what it can offer, look at Hashicorp’s website. Also, check out this video by Hashicorp’s cofounder for a short summary of what Terraform can offer.

Modularization & Terraform Registry

Terraform has a few different offerings that provide various features. The most basic of which is open source, the Terraform CLI. Although Terraform CLI is an excellent tool, it is significantly more helpful when best practices are implemented.

One of the first things to do is organize your terraform code. Break it apart into child modules or components that encapsulate smaller pieces of your infrastructure. Instead of having one module file that provisions all of your resources, break up your architecture into several components (i.e., AppService plan, AppService, storage account, Redis cache) and reference them as dependencies in your encapsulating module (ex. main.tf).

Hashicorp also gives you access to their public Terraform registry for various providers like AWS and Azure. The public registry already has re-usable modules for the simplest cloud resource blocks for you to extend and utilize.

Although these pre-defined modules exist to standardize resource naming conventions, cloud computing sizing/scaling, and other restrictions, you may want to impose on your development teams to create modules that utilize modules in the public Terraform Registry. Create them with distinctly defined validation parameters for input variables.

If you want to read more about creating Terraform modules, check out this blog post by the creators of Terragrunt, as well as Hashicorp’s documentation.

Now the question arises: where do I keep the code for my child modules to be easily distributable, re-usable, and maintainable? Creating these child modules and storing the code within the application code repository creates some disadvantages. What if you want to re-use these sub-modules for other repositories or different projects? You’d have to duplicate the child module code and track it in several places. That’s not ideal.

To address this, you should utilize a shared registry or storage location for your child modules to reference them from multiple repositories and even distribute different versions to different projects. This would involve moving each submodule to its individual repo to be maintained independently and then uploading it to your registry or central storage location. Acceptable methods that Terraform can work with are:

  • GitHub
  • Bitbucket
  • Generic Git, Mercurial repositories
  • HTTP URLs
  • S3 buckets
  • GCS buckets

See the Terraform documentation on module sources for more information.

Utilizing these methods allows you to maintain your sub modules independently and distribute different versions that larger applications can choose to inherit. You might go from Terraform projects like this:

Maintain Sub Modules

to this:

Consolidating duplicate code

Terragrunt

Preface

Utilizing registries to modularize your infrastructure is only a small part of the improvements you can make to your Terraform code.One of the significant concepts of Terraform is how it tracks the state of your infrastructure with a state file. In Terraform, you need to define a “remote state” for each grouping of infrastructure you are trying to deploy/provision. This remote state could be stored in an S3 bucket, Azure Storage account, Terraform Cloud, or another applicable service. This is how Terraform knows where to track the state of your infrastructure to determine if any changes need to be applied or if the configuration has drifted away from the baseline that is defined in your source code. The organization of your state files is essential, especially if you are managing it yourself and not using Terraform Cloud to perform your infrastructure runs.

In addition, you must keep in mind that when your development teams and applications grow. You will frequently need to manage multiple developments, testing, quality assurance, and production environments for several projects/components simultaneously; the number of configurations, variable files, CLI arguments, and provider information will become untenable over time.

How do you maintain all this infrastructure reliably? Do you track all the applications for one tier in one state file? Or do you break them up and track them separately?

How do you efficiently share variables across environments and applications without defining them multiple times? How do you successfully apply these terraform projects in a continuous deployment pipeline that is consistent and repeatable for different types of Terraform projects?

At one of our current clients, we were involved with onboarding Terraform as an Infrastructure as Code (IaC) tool. However, we ran into many challenges when trying to deploy multiple tiers (dev, test, stage, production, etc.) across several workstreams, specifically in a continuous manner within a deployment pipeline.

The client I work for has the following requirements for the web UI portion of the services they offer (consider Azure Cloud provider for context):

  • each tier has *six applications* for different areas of the United States
  • *Each application* has a web server, a Redis cache, app service plan, a storage account, and a key vault access policy to access a central key store
  • Development and test tiers are deployed to a single region.
  • Applications in the development and test tier both share a Redis cache
  • Applications in Staging and production environments have individual Redis caches
  • Stage and production environments are deployed to two regions, east and central.

Stage and Production Environments

Stage and Production tiers have up to 48 resources, respectively; the diagram above only represents three applications and excludes some services. Our client also had several other IT services that needed similar architectural setups; most projects involved deploying six application instances (for each service area of the United States), each configured differently through application settings.
Initially, our team decided to use the Terraform CLI and track our state files using an Azure Storage Account. Within the application repository, we would store several backend.tf files alongside our terraform code for each tier and pass them dynamically to terraform init –backend-config= when we want to initialize a specific environment. We also passed variable files dynamically to terraform [plan|apply|destroy] –var-file= to combine common and tier-specific application setting template files. We adopted this process in our continuous deployment pipeline by ensuring the necessary authentication principals and terraform CLI packages were available on our deployment agents and then running the appropriate constructed terraform command in the appropriate directory on that agent.

This is great but presented a few problems when scaling our process. The process we used initially allowed developers to create their own terraform modules specific to their application, utilizing either local or shared modules in our private registry. One of the significant problems came when trying to apply these modules in a continuous integration pipeline. Each project had its own unique terraform project and its own configurations that our constant deployment pipeline needed to adhere to.

Let’s also consider something relatively simple, like the naming convention of all the resources in a particular project. Usually, you would want the same-named prefix on your resources (or apply it as a resource tag) to visually identify what projects the resources belong to. Since we had to maintain multiple tiers of environments for these projects (dev, test, stage, prod), we wanted to share this variable across environments, only needing to declare it once. We also wanted to declare other variables and optionally override them in specific environments. With the Terraform CLI, there is no way to merge inputs and share variables across environments. In addition, you cannot use expressions, functions, or variables in the terraform remote state configuration blocks, forcing you to either hardcode your configuration or apply it dynamically through the CLI; see this issue.

We began to wonder if there was a better way to organize ourselves. This is where Terragrunt comes into play. Instead of keeping track of the various terraform commands, remote state configurations, variable files, and input parameters we needed to consolidate to provision our terraform projects. What if we had a declarative way of defining how our terraform project was configured? Terragrunt is a minimal wrapper around Terraform that allows you to dynamically assign and inherit remote state configurations, provider information, local variables, and module inputs for your terraform projects through a hierarchal folder structure with declarative configuration files. It also gives you a flexible and unopinionated way of consolidating everything Terraform does before it runs a command. Every option you pass to Terraform can be specifically configured through a configuration file that inherits, merges, or overrides components from other higher-level configuration files.

Terragrunt allowed us to do these important things:

  • Define a configuration file to tell what remote state file to save based on application/tier (using folder structure)
  • It allowed us to run multiple terraform projects at once with a single command
  • Pass outputs from one terraform project to another using a dependency chain.
  • Define a configuration file that tells terraform what application setting template file to apply to an application; we used .tpl files to apply application settings to our Azure compute resources.
  • Define a configuration file that tells terraform what variable files to include in your terraform commands
  • Allowed us to merge-common input variables with tier-specific input variables with the desired precedence
  • It allowed us to consistently name and create state files

Example

Let’s consider the situation where we want to maintain the infrastructure for a system with two major components: an API and a database solution. You must also deploy dev, test, stage, and production environments for this system. Dev and Test environments are deployed to one region, while stage and production environments are deployed to two regions.

We’ve created a preconfigured sample repository to demonstrate how we might handle something like this with Terragrunt. Now, although the requirements and scenario described above may not pertain to you, the preconfigured sample repository should give you a good idea of what you can accomplish with Terragrunt and the benefits it provides in the context of keeping your Terraform code organized. Also, remember that Terragrunt is unopinionated and allows you to configure it in several ways to accomplish similar results; we will only cover a few of the benefits Terragrunt provides, but be sure to check out their documentation site for more information.

To get the most out of the code sample, you should have the following:
– Terraform CLI
– Terragrunt CLI
– An Azure Subscription
– AZ CLI

Run through the setup steps if you need to. This will involve running a mini terraform project to provision a few resource groups in addition to a storage account to store your Terraform state files.
The sample repo contains several top-level directories:

  • /_base_modules
  • /bootstrap
  • /dev
  • /test
  • /stage
  • /prod
    • _base_modules – folder contains the top-level Terraform modules that your application will use. There are subfolders for each application type, the API, and storage solution (/api and /sql). For example, there is a subfolder for the API, which contains the terraform code for your API application, and one for SQL, which will contain the terraform code for your storage/database solution; take note of the main.tf, variables.tf, and outputs.tf files in each sub folder. Each application type folder will also contain a .hcl file that contains global configuration values for all environments that consume that respective application type
    • [dev/test/stage/prod] – environment folders that contain sub folders for each application type. Each sub folder for each application type will contain Terragrunt configuration files that contain variables and inputs specific to that environment
    • bootstrap – a small isolated terraform project that will spin up placeholder resource groups in addition to a storage account used to maintain remote Terraform state files

As mentioned above, there are several .hcl files in a few different places within this folder structure. These are Terragrunt configuration files. You will see one within each sub folder inside the _base_modules directory and one in every sub folder within each environment folder. These files are how Terragrunt knows what terraform commands to use, where to store each application’s remote state, and what variable files and input values to use for your terraform modules defined in the _base_modules directory. Read more about how this file is structured on Gruntwork’s website. With this sample repository, global configurations are maintained in the /_base_modules folder and consumed by configurations in the environment folders.

Let’s go over some of the basic features that Terragrunt offers.

Keeping your Remote State Configuration DRY

I immediately noticed when writing my first bits of Terraform code that I couldn’t use variables, expressions, or functions within the Terraform configuration block. You can override specific parts of this configuration through the command line, but there was no way to do this from code.

Terragrunt allows you to keep your backend and remote state configuration DRY by allowing you to share the code for backend configuration across multiple environments. Look at the /_base_modules/global.hcl file in conjunction with the /dev/Terragrunt.hcl file.

/_base_modules/global.hcl:

remote_state {
  backend = "azurerm"
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite"
  }
  config = {
    resource_group_name  = "shared"
    storage_account_name = "4a16aa0287e60d48tf"
    container_name       = "example"
    key            = "example/${path_relative_to_include()}.tfstate"
  }
}

This file defines the remote state that will be used for all environments that utilize the API module. Take special note of the ${path_relative_to_include} expression – more on this later.
A remote state Terragrunt block that like this:

remote_state {
    backend = "s3"
    config = {
      bucket = "mybucket"
      key    = "path/for/my/key"
      region = "us-east-1"
    }
  }

Is equivalent to a Terraform block that looks like:

terraform {
    backend "s3" {
      bucket = "mybucket"
      key    = "path/to/my/key"
      region = "us-east-1"
    }
    }

To inherit this configuration into a child subfolder or environment folder you can do this:

/dev/api/terragrunt.hcl


include "global" {
  path = "${get_terragrunt_dir()}/../../_base_modules/global.hcl"
  expose = true
  merge_strategy = "deep"
}

The include statement above tells Terragrunt to merge the configuration file found at _base_modules/global.hcl with its local configuration. The ${path_relative_to_include} in the global.hcl file is a predefined variable that will return the relative path of the calling .hcl file, in this case,/dev/api/terragrunt.hcl. Therefore, the resulting state file for this module would be in the example container at dev/api.tfstate. For the SQL application in the dev environment, the resulting state file would be dev/sql.tfstate; look at the _base_modules/sql/sql.hcl file. For the API application in the test environment, the resulting state file would be test/api.tfstate. Be sure to check out all of the built-in functions Terragrunt offers out of the box.

Using the feature just mentioned, we only define the details of the remote state once, allowing us to cut down on code repetition. Read more about the remote_state and include blocks and how you can configure them by visiting the Terragrunt documentation. Pay special attention to merge strategy options, how you can override includes in child modules, and the specific limitations of configuration inheritance in Terragrunt.

Keeping your Terraform Configuration DRY

Merging configuration files does not only apply to remote state configurations – but you can also apply them to the sources and inputs of your modules.

In Terragrunt, you can define the source of your module (main.tf or top-level terraform module) within the Terraform block. Let’s consider the API application:

/_base_modules/api/api.hcl

terraform {
  source = "${get_terragrunt_dir()}/../../_base_modules/api"

  extra_arguments "common_vars" {
    commands = get_terraform_commands_that_need_vars()

    required_var_files = [
      
    ]
  }
}

You’ll notice this is referencing a local path; alternatively, you can also set this to use a module from a remote git repo or terraform registry.

The api.hcl configuration is then imported as a configuration into each environment folder for the API application type:

Ex. /dev/api/terragrunt.hcl

include "env" {
  path = "${get_terragrunt_dir()}/../../_base_modules/api/api.hcl"
  expose = true
  merge_strategy = "deep"
}

Include statements with specific merge strategies can also be overwritten by configurations in child modules, allowing you to configure each environment separately if needed.

Merging inputs before they are applied to your terraform module is also extremely helpful if you need to share variables across environments. For example, all the names of your resources in your project might be prefixed with a particular character set. You can define any global inputs in the inputs section of the _base_modules/global.hcl file. Because Terragrunt configuration files are written in the HCL language, you can also utilize all the expressions and functions you use in Terraform to modify or restructure input values before they are applied. Look at how we are defining the identifier input variable found in both SQL and API modules:

Here is the terraform variable:

/_base_modules/api/variables.tf and /_base_modules/sql/variables.tf

variable "identifier" {
  type = object({
      primary = string
      secondary = string
      type = string
  })
}

Here is the primary property being assigned from the global env:

/_base_modules/global.hcl

... 
inputs = {
    identifier = {
        primary = "EXAMPLE"
    }
}
...

Here is the secondary property being assigned from the dev/dev.hcl file:

/dev/dev.hcl

inputs = {
 identifier = {
     secondary = "DEV"
 }
}

And here is the type property being applied in the module folders:

/_base_modules/sql/sql.env.tf

...
inputs = {
    identifier = {
        type = "SQL"
    }
}

/_base_modules/api/api.hcl

...
inputs = {
    identifier = {
        type = "API"
    }
}

All configurations are included in the environment configuration files with:

include "global" {
  path = "${get_terragrunt_dir()}/../../_base_modules/global.hcl"
  expose = true
  merge_strategy = "deep"
}

include "api" {
  path = "${get_terragrunt_dir()}/../../_base_modules/api/api.hcl"
  expose = true
  merge_strategy = "deep"
}

include "dev" {
  path = "../dev.hcl"
  expose = true
  merge_strategy = "deep"
}

would result in something like:

inputs = {
    identifier = {
      primary = "EXAMPLE"
secondary = "DEV"  
type = "API"
    }
}

We utilize this pattern to share variables across all environments and applications within a specific environment without having to declare them multiple times.

It is also important to note that because Terragrunt configuration files are written in the HCL language, you have access to all Terraform’s functions and expressions. As a result, because you can inherit Terragrunt configuration files into a specific environment, you can restructure, merge, or alter input variables before they are sent to terraform to be processed.

Running Multiple Modules at once

You can also run multiple terraform modules with one command using Terragrunt. For example, if you wanted to provision dev, test, stage, and prod with one command, you could run the following command in the root directory:

terragrunt run-all [init|plan|apply]

If you wanted to provision the infrastructure for a specific tier, you could run the same command inside an environment folder (dev, test, stage, etc.). This allows you to neatly organize your environments instead of maintaining everything in one state file or trying to remember what variable, backend, and provider configurations to pass in your CLI commands when you want to target a specific environment.

It is important to note that you can maintain dependencies between application types within an environment (between the SQL and API applications) and pass outputs from one application to another. Look at the dev/api environment configuration file:

/dev/api/terragrunt.hcl

dependency "sql" {
  config_path = "../sql"
    mock_outputs = {
    database_id = "temporary-dummy-id"
  }
}

locals {
}
inputs = {
    database_id = dependency.sql.outputs.database_id
    ...
}

Notice that it references the dev/sql environment as a dependency. The dev/sql environment uses the _base_modules/sql application so look at that module, specifically the outputs.tf file.

/_base_modules/sql/outputs.tf

output "database_id" {
  value = azurerm_mssql_database.test.id
}

Notice that this output is being referenced in the /dev/api/terragrunt.hcl file as dependency.

The client requirements described earlier in this post proved to be especially difficult to maintain without the benefit of being able to configure separate modules that depend on one another. With the ability to isolate different components of each environment and share their code and dependencies across environments, we could maintain multiple environments effectively and efficiently with different configurations.

Conclusion

Terraform as an IaC tool has helped us reliably develop, maintain, and scale our infrastructure demands. However, because our client work involved maintaining multiple environments and projects simultaneously, we needed specific declarative design patterns to organize our infrastructure development. Terragrunt offered us a simple way to develop numerous environments and components of a given application in a way that was repeatable and distributable to other project pipelines.

There are several features of Terragrunt we did not discuss in this post:

Before, After, and Error Hooks
Maintaining CLI flags

We would like to see some of the functionality Terragrunt offers baked into Terraform by default. However, we do not feel like Terragrunt is a final solution; Terraform is rather unopinionated and less concerned with how you set up your project structure, while Terragrunt is slightly more opinionated in your setup. Terragrunt claims to be DRY, but there is still a lot of code duplication involved when creating multiple environments or trying to duplicate infrastructure across regions. For example, creating the folder structure for an environment is cumbersome, especially when you want to add another tier.

I am currently working as a development operation engineer at a client that maintains a large and complex infrastructure with an even more complicated development workflow. Cloud infrastructure had already been provisioned, and several development cycles had been completed when I joined the team a couple of years ago. As a result, I came into an environment that had already established a development workflow, with barely any infrastructure as code (IaC) practices in place.

What is unique about this client is that it maintains six separate web applications, each designated for a different area in the United States; the backends for each application vary from application to application, but the front ends are mostly similar. To avoid a maintenance nightmare, we use code from one repo to deploy the main UI of all six applications; the same code is deployed to each application but configured to run differently through a master configuration file. Each web application is compromised of several sub-applications, each with its codebase.

One of my responsibilities is to spin up and maintain application stacks for new development teams that are working on a specific feature for this client. That feature may or may not be utilized by all six web applications. For this team to develop rapidly, they need a stable testing environment, equipped with continuous delivery, to do their work.

This “application stack” is comprised of:

  • 1 Azure App Service Plan
  • 1 App Service (up to 6)
  • 1 Redis Cache
  • 1 Storage Account

These are Microsoft Azure constructs and are not essential to understand in the context of this post. Keep in mind that 6 separate applications utilize a set of shared resources.

Since the DevOps team at this particular client is small, we were only able to create a script to deploy an application stack for a specific development effort, but we did not get the chance to implement any strategies around maintaining these stacks once they were deployed. As a result, we are applying configurations that need to be applied across the 25-30 application stacks that we maintain has turned out to be a logistical nightmare.

Model 1 Main App RepoThe diagram above represents a high-level overview of a single application stack that we need to maintain; the diagram is scaled down from 6 applications to 3 applications for clarity.

The current script we use to provision application stacks does the following:

  1. Creates all Azure resources
  2. Performs KeyVault configuration
    1. Enabled managed identity on all app services
    2. Adds access policy to the key vault for all app services
    3. Adds key vault references for secret app settings on all app services
  3. Creates 1 Azure DevOps release pipeline to deploy code to all app services; this is done by cloning a base template for a release pipeline that exists before the script is run.

The Problem

Although the deployment script that is currently in use saves time, it falls short on a few things:

If we want to make configuration changes across all the application stacks, we would have to make edits to the deployment script and rerun it on each application stack. This process is entirely too manual and can be improved.

  1. If we want to change the mechanics of how the underlying deployment pipeline functions across all the application stacks, we have to make edits on each deployment pipeline that is tied to a given application stack. This is too tedious and leaves lots of room for error; we initially mitigated some of this by utilizing Azure DevOps task groups.
  2. Configuration drift is widely prevalent; because we do not have an easy way to maintain all of the application environments across the board, minor configuration changes during development are difficult to track and often fail to propagate to our main application stacks.
  3. The Solution: TEMPLATE YOUR LIFE

Azure YAML Templates

This client is relatively young in terms of their cloud journey, especially with Azure and Azure DevOps. They currently rely heavily on Release Pipeline User Interface within Azure DevOps to deploy code to their applications. In recent years, Azure has been subtly urging teams to utilize multi-stage YAML templates instead of the Release Pipeline User Interface to adopt an “infrastructure as code” mindset to the continuous delivery process. With this mindset, there is no difference between “Build” pipelines and “Release” pipelines; the only construct is a pipeline where you can perform any task (build, test, deploy, etc.).

I encourage you to read more about Azure DevOps YAML templates. I’ve included some relevant links below:

Azure DevOps YAML Templates
Azure DevOps Multistage YAML UI Experience
YAML Schema

Given the problem statement above, there is a large need to develop a framework and process around maintaining the cloud infrastructure and its respective continuous delivery process that is easy to manage and propagate changes through.

This is where Azure DevOps YAML templates become useful.

All you need to create a pipeline in Azure DevOps is a YAML file. This YAML file can exist in any code repository. Once this file is imported into a code repository, you can use the Azure DevOps UI to create a pipeline from this file. While you can run the pipeline from the Azure DevOps UI, the pipeline’s changes will be maintained and version within the YAML file itself, just like you would with any application code.

This YAML file can inherit and reference another YAML file that exists in other code repositories.

Template Repository

I am currently developing a code repository that will contain:

  1. Azure ARM templates to version app service configurations (including virtual/sub applications)
  2. Deployment script to deploy ARM templates mentioned above
  3. Azure DevOps YAML files that will function as “release pipelines”:
    1. Create / Refresh shared resources
    2. Create / Refresh app services
    3. Deploy code to app services

When a new workstream or development effort begins, all they need to do is create a simple YAML file that extends from the main (release.yaml) template file in the repo mentioned above (see azurepipeline.yaml below). Once that is done, the first time the development team pushes code to a specified branch, they will be equipped with a new environment with their code changes.

Hierarchy of the Templates

The diagram above represents the hierarchy of the templates in the code repository mentioned above. You can see there are only 7 YAML template files to maintain. The azurepipeline.yaml file inherits these template files. This helps address the challenge mentioned above related to the daunting task of maintaining 25+ release pipelines; changes made to any of the template files will propagate to any azurepipeline.yaml files that inherit from the release.yaml file.

The idea is that we can import the azurepipeline.yaml file that can be into any repo or branch. This file is relatively simple:

azurepipeline.yaml

# run on schedule instead of utilizing ci triggers
name: $(Date:yyyyMMdd)$(Rev:.r)
trigger:
- none
  # pipeline artifact list (previously source artifacts in release pipeline)
extends:
  template: release.yaml@templates

The release.yaml file that the azurepipeline.yaml file extends from looks similar to the one below:

release.yaml

resources:
  # script and template reference
  repositories:
    - repository: templates
      type: git
      name: template-repo
  pipelines:
    - pipeline: main
      project: ExampleProject
      source: YAML\main
      branch: stage
    - pipeline: subapp1
      project: ExampleProject
      source: YAML\subapp1
      branch: stage
    - pipeline: subapp2
      project: ExampleProject
      source: YAML\subapp2
      branch: stage
stages:
- stage: SHARED
  dependsOn: []
  displayName: refresh shared resources
  jobs: 
  - job: refresh_shared_resources
    pool:
      vmImage: 'ubuntu-latest'
    steps:
      - template: templates/update-infrastructure.yaml@templates
        parameters:
          sharedOnly: true
- stage: APP1
  dependsOn: ['SHARED']
  variables:
  - template: templates/appvars.yaml@templates
    parameters:
      appName: 'app1'
  displayName: app1 
  jobs: 
  - job: REFRESH_APP1
    pool:
      vmImage: 'ubuntu-latest'
    steps:
      - template: templates/update-infrastructure.yaml@templates
        parameters:
          sharedOnly: true
          appName: app1
  - template: templates/app1.yaml@templates
- stage: APP1
  dependsOn: ['SHARED']
  variables:
  - template: templates/appvars.yaml@templates
    parameters:
      appName: 'app1'
  displayName: app1 
  jobs: 
  - job: REFRESH_APP1
    pool:
      vmImage: 'ubuntu-latest'
    steps:
      - template: templates/update-infrastructure.yaml@templates
        parameters:
          sharedOnly: true
          appName: app1
  - template: templates/app1.yaml@templates

App stages are created for each app (not shown).

The app template files referenced by the release.yaml file looks similar to the file below:

appl.yaml

jobs:
- deployment: app1
  dependsOn: REFRESH_APP1
  displayName: "DEPLOY APP1"
  environment: stage
  pool:
    vmImage: windows-latest    
  strategy:
    runOnce:
      deploy:
        steps:
        - checkout: self
        - template: configure-app.yaml@templates
        - template: deploy-app.yaml@templates
          parameters:
            isVirtualApp: false
            appName: 'ui'
        - template: deploy-app.yaml@templates
          parameters:
            isVirtualApp: true
            appName: 'subapp1'
        - template: deploy-app.yaml@templates
          parameters:
            isVirtualApp: true
            appName: 'subapp2'

Take note of the different steps that are used to deploy an application. It comprises a configuration step and a deployment step for each sub-application that utilizes the same azure YAML template file with different parameters.

The release.yaml file results in a multi-stage YAML pipeline that looks like the one below:

Release.yaml file results

Resources

The resource section of the template defines what resources your pipeline will use; this is analogous to the artifact section in the release pipeline UI view. You can only have the resources section declared once in your series of templates for a given pipeline. In the example above, it is defined in the release.yaml file, the template that other templates will extend from.

In this example, the resource section references the template repository itself and other build pipelines that produce code artifacts that we will need to deploy to our applications.

Defining the resources section in the base template (release.yaml) allows us to be abstract the choice of artifact sources away from the end-user. This is advantageous if we want to add more sub-applications to our release pipeline; we would only need to change the release resources section.yaml file (changes will propagate to all inherited/extended pipelines).

At the client I work with; this is problematic. In the solution above, all pipelines that extend from release.yaml (that contains the resources section) is limited to only use the specific artifacts AND the branches they are set to pull from, as defined in the base template (release.yaml). We consistently have project teams that need sub-application artifacts from specific branches that their development work exists on. To solve this, we moved the resources section into the extended template (azurepipeline.yaml). However, this isn’t optimal because we would still need to update every comprehensive template if we wanted to add to the resources section across all application stacks.

As far as I know, there is no way to use pipeline variables or template expressions to determine what resource artifacts are needed dynamically. Keep the resources section in the base template and override them within the extended templates.

Dynamic Insertion
Currently, we must maintain a manifest that describes the relationship between each application and its respective sub-applications. For example, app1 could have sub-applications subapp1 and subapp2, and app2 could have sub-applications subapp1 and subapp3 (or any combination of sub-applications). We utilize a JSON file that defines the mappings between application and sub-application. This JSON file is parsed in the deployment script to ensure the correct sub-applications exist before code deployment; in Azure, the sub-application must live before you push code to it. As a result, we also need to maintain sub-applications in each of the different YAML step templates for each application. At this point, I am unaware of an elegant way to iterate over an Azure DevOps pipeline variable to create steps dynamically.

Variable Templates
Template expressions can be applied to variables. This was extremely useful in the context of the solution I was trying to solve. For example, each stage in the pipeline described above is for a particular application with a distinct name. This name is used to determine several properties for its respective pipeline stage. For example, the app1 pipeline stage uses the site name/url app1.azurewebsites.net. The app2 stage uses the site name/url app2.azurewebsites.net. You can see both site names follow the same naming convention. This is a great use case for a variable template.

Here is the variable template I used:

appvars.yaml

parameters:
  appName: ""
  deployToSlot: false
variables:
  siteName: "$(tier)-$(region)-$(app)-${{parameters.appName}}"
  ${{ if eq(parameters.deployToSlot, true) }}:
    corsOrigin:"https://${{parameters.appName}}-$(slotName).azurewebsites.net"
  ${{ if eq(parameters.deployToSlot, false) }}:  
    corsOrigin: “https://${{parameters.appName}}.azurewebsites.net"
  appName: ${{parameters.appName}}
  template: "$(System.DefaultWorkingDirectory)/arm_templates/azuredeploy.json"
  virtualApps: "$(System.DefaultWorkingDirectory)/manifest.json"

You can see that I’ve included a parameters section that takes in appName as a parameter. You can see it in the release.yaml file, this parameter is being applied with:

variables:
  - template: templates/appvars.yaml@templates
    parameters:
      appName: 'app1'

This allows us to cut down on repeated code by using an extendable template for the variables needed for each deployment stage. It is essential to understand how variables and parameters work in Azure pipelines and the different ways they can be expressed in your YAML.

Check out these resources:
YAML Variables Documentation
YAML Parameters Documentation
Developer Community Forum Post

Conclusion

Development operation decisions should be considered in architecture design. In the scenario described above, the architecture I inherited did not foster the most efficient development and deployment workflow. For example, the decision to have sub-applications exist on an individual app service instead of assigning them to their app services. This limited us to deploy the individual applications in series, as you are limited to run one deployment at a time to give app service.

Infrastructure as code implemented in the continuous integration and delivery process can be equally as important as implementing it for your existing infrastructure.

Suppose you are working on a project that requires a complicated development lifecycle with multiple teams needing distinct environments. In this case, it becomes increasingly important to formulate a strategy around maintaining the various environments. Environment creation and code deployment should be as automated as possible. Applying configurations across all environments should be an easy task.

AIS is pleased to announce the availability of the NIST SP 800-53 Rev 4 Azure Blueprint Architecture.

(NIST SP 800-53 security controls could be an entire series of blog posts in itself…so if you want to learn more than I cover here, check out NIST’s website.)

The NIST SP 800-53 Rev 4 Azure Blueprint Architecture applies a NIST SP 800-53 Rev 4 compliant architecture with the click of a button to the Azure Government Cloud subscription of your choice. Okay, maybe there are a few clicks with some scripts to prep the environment, but I swear that’s it!  This allows organizations to quickly apply a secure baseline architecture build to their DevOps pipeline. They can also add it to their source control themselves to accompany their application source code. (If you want assistance in implementing this within your greater cloud and/or DevOps journey, AIS can help with our Compliance and Security Services offering.) Read More…

Please read Part One and Part Two

Part 3: Azure Automation, Azure RunBooks, and Octopus Deploy

With just PowerShell and an Azure ARM template, we can kick off a deployment in just a few minutes. But there are still some manual steps involved – you still need to login to your Azure subscription, enter a command to create a new resource group, and enter another command to kick off a deployment. With the help of an Azure automation account and a platform called Octopus Deploy, we can automate this process even further to a point where it takes as little as three clicks to deploy your whole infrastructure! Read More…

(Part One of this series can be found here.)

Deploying Azure ARM Templates with PowerShell

After you’ve created your template, you can use PowerShell to kick off the deployment process. PowerShell is a great tool with a ton of features to help automate Azure processes. In order to deploy Azure ARM Templates with PowerShell, you will need to install the Azure PowerShell cmdlets. You can do this by simply running the command Install-Module AzureRM inside a PowerShell session.

Check out this link for more information on installing Azure PowerShell cmdlets. PowerShell works best on a Windows platform, although there is a version now out for Mac that you can check out here. You can also use Azure CLI to do the same thing. PowerShell and Azure CLI are quick and easy ways to create resources without using the Portal. I still stick with PowerShell, even though I primarily use a Mac computer for development work. (I’ll talk more about this in the next section.) Read More…

(Don’t miss Part Two and Part Three of this series!)

Early in my web development career, I always tried to avoid deployment work. It made me uneasy to watch other developers constantly bang their heads against their desks, frustrated with getting our app deployed to whatever cloud service we were using at the time. Deployment work became the “short straw” assignment because it was always a long, unpredictable and thankless task. It wasn’t until I advanced in my tech career that I realized why I felt this way.

My experience with deployment activities, up to this point, always involved a manual process. I thought that the time it took to set up an automated deployment mechanism was a lot of unnecessary overhead – I’d much rather spend my time developing the actual application and spend just a few hours every so often on a manual deployment process when I was ready. However, as I got to work with more and more experienced developers, I began to understand that a manual deployment process is slow, unreliable, unrepeatable, and rarely ever consistent across environments. A manual deployment process also requires detailed documentation that can be hard to follow and in constant need of updating.

As a result, the deployment process becomes this mysterious beast that only a few experts on your development team can tame. This will ultimately isolate the members of your development team, who could be spending more time working on features or fixing bugs related to your application or software. Although there is some initial overhead involved when creating a fully automated deployment pipeline, subsequent deployments of the same infrastructure can be done in a matter of seconds. And since validation is also baked into the automated process, your developers will only have to devote time to application deployment if something fails or goes wrong.

This three-part blog series will serve to provide a general set of instructions on how to build an automated deployment pipeline using Azure cloud services and Octopus Deploy, a user-friendly automation tool that integrates well with Azure. It might not detail out every step you need, but it will point you in the right direction, and show you the value of utilizing automated deployment mechanisms. Let’s get started. Read More…