OPG Terraform Practices
General Principles
Simple is better than Complex - Code for readability and inheritability
Layers of loops, dynamic blocks, and ternaries obscure what it is you’re actually doing, making it difficult to work with for junior engineers, you in 12 months time or people trying to get an understanding of Terraform.
Code should be correct, clear and concise, in that order.
Repo Organisation
As a general practice across OPG we have a separation of accounts for development, preproduction, and production; with a goal of multiple environments in development, with preproduction and production being identical and isolated to one environment per account.
The high level folder structure should contain an account and environment folder, to separate out the resources. Each folder may have a sub folder for modules relevant to that layer. VPC resources should be accessed as data sources, and should be stored in vpc.tf within the environment directory.
/account
/account/modules
/environment
/environment/modules
/environment/vpc.tf
Use Terraform Workspaces
We use Terraform workspaces to stop the duplication and customisation of code for each account or enviroment which increases maintainability and ease of promotion. Workspaces allow for environments to be consistent by default, but allow features and resources to be toggled to enable specificfunctionality.
Usage in Pipelines
Whenever running Terraform in a CI/CD pipeline always run a terraform plan stage, even
if applying with an auto-approve. This allows you to understand exactly what Terraform
changed and why should there ever be an issue, and allows traceability of when specific
changes were applied to an environment.
Use .editorconfig in all repos for consistent whitespace
Every mainstream IDE supports plugins for the .editorconfig standard to make it easier to enforce whitespace consistency.
We recommend adopting the whitespace convention of a particular language or project.
This is the standard .editorconfig that we use.
# Override for Makefile
[{Makefile, makefile, GNUmakefile, Makefile.*}]
indent_style = tab
indent_size = 4
[*.yaml]
intent_style = space
indent_size = 2
[*.sh]
indent_style = tab
indent_size = 4
[*.{tf,tfvars,tpl}]
indent_size = 2
indent_style = space
Naming Conventions
Resource Names
Resource names should be descriptive to allow the usage of the resource to be easily understood.
Resources defined in Terraform should be all lowercase with underscores as
separators for consistency with Terraform resource object names, also known as
snake_case. Filenames within the Terraform project should also follow this
convention.
Example:
resource "aws_security_group" "api_ecs_task" { … }
Resources that exist in AWS should be all lowercase separated by hyphens, they should all include the environment or account that they belong to in order to determine uniqueness and to be easily identifiable in the AWS Console. Using hyphens rather than underscores allows a resource to be easily identified as either a Terraform definition of a resource, or a “real” AWS resource.
Example:
resource "aws_security_group" "api_ecs" {
name = "api-ecs-${terraform.workspace}"
}
Language
Version pin all providers
Terraform’s providers are constantly in flux. To avoid unexpected instability we should ensure that all providers are pinned.
Use Terraform Linting
Linting helps to ensure a consistent code formatting, improves code quality and catches common errors with syntax.
Run terraform fmt before committing all code. Use a pre-commit hook to do this
automatically.
Do not use HEREDOC for JSON, YAML or IAM Policies
There are better ways to achieve the same outcome using Terraform interpolations or resources
For JSON, use a combination of a local and the jsonencode function.
For YAML, use a combination of a local and the yamlencode function.
For IAM Policy Documents, use the native iam_policy_document resource.
Use proper datatype
Using proper datatypes in Terraform makes it easier to validate inputs and document usage.
- Use null instead of empty strings (“”)
- Use bool instead of strings or integers for binary true/false
- Use string for freeform text
- Use object sparingly as it makes it harder to document and validate
Use locals to baptize opaque resource IDs
Using locals makes code more descriptive and maintainable.
Rather than using complex expressions as parameters to some Terraform resource,
instead move that expression to a local and reference the local in the resource.
Variables
Use upstream module or provider variable names where applicable
When writing a module that accepts variable inputs, make sure to use the same names as the upstream to avoid confusion and ambiguity.
Use all lowercase with underscores as separators
Avoid introducing any other syntaxes commonly found in other languages such as CamelCase or pascalCase. For consistency we want all variables to look uniform. This is also inline with the HashiCorp naming conventions.
Use positive variable names to avoid double negatives
All variable inputs that enable/disable a setting should be formatted
...._enabled (e.g. encryption_enabled).
It is acceptable for default values to be either false or true.
Use feature flags to enable/disable functionality
All modules should incorporate feature flags to enable or disable functionality.
All feature flags should end in _enabled and should be of type bool.
Use description field for all inputs
All variable inputs need a description field. When the field is provided by an
upstream provider (e.g. terraform-aws-provider), use same wording as the upstream docs.
Use sane defaults where applicable
Modules should be as turnkey as possible. The default value should ensure
the most secure configuration (E.g. with encryption enabled).
Use variables for all secrets with no default value
All variable inputs for secrets must never define a default value.
This ensures that terraform is able to validate user input. The exception to
this is if the secret is optional and will be generated for the user
automatically when left null or "" (empty).
Outputs
Use description field for all outputs
All outputs must have a description set. The description should be based on (or adapted from) the upstream Terraform provider where applicable. Avoid simply repeating the variable name as the output description.
Use well-formatted snake case output names
Avoid introducing any other syntaxes commonly found in other languages such as CamelCase or pascalCase. For consistency we want all variables to look uniform. It also makes code more consistent when using outputs together with Terraform remote_state to access those settings from across modules.
Never output secrets
Secrets should never be outputs of modules. Rather, they should be written to secure storage such as AWS Secrets Manager, AWS SSM Parameter Store with KMS encryption, or S3 with KMS encryption at rest.
Our preferred mechanism on AWS is using Secrets Manager.
Use symmetrical names
We prefer to keep Terraform outputs symmetrical as much as possible with the
upstream resource or module, with exception of prefixes. This reduces the amount
of entropy in the code or possible ambiguity, while increasing consistency.
Below is an example of what *not to do. The expected output name is user_secret_access_key.
This is because the other IAM user outputs in the upstream module are prefixed with user_,
and then we should borrow the upstream’s output name of secret_access_key to become
user_secret_access_key for consistency.
State
Use remote state
Use Terraform to create state bucket
This requires a two-phased approach, whereby you first provision the bucket
without the remote state enabled. Then enable remote state (e.g. s3 {}) and import
remote state by simply rerunning terraform init. We recommend this strategy
because it promotes using the best tool for the job and makes it easier to
define requirements and use consistent tooling.
Use backend with support for state locking
We recommend using the S3 backend with DynamoDB for state locking.
Use encrypted S3 bucket with versioning, encryption and strict IAM policies
We recommend not commingling state in the same bucket. This could cause the state to get overridden or compromised. Note, the state contains cached values of all outputs.
Use .gitignore to exclude Terraform state files, state directory backups and core dumps
.terraform
.terraform.tfstate.lock.info
*.tfstate
*.tfstate.backup
Module Design
Small Opinionated Modules
We believe that modules should do one thing very well. But in order to do that, it requires being opinionated on the design. Simply wrapping Terraform resources for the purposes of modularizing code is not that helpful. Implementing a specific use-case of those resource is more helpful.