Terraform is a popular tool for managing infrastructure as code. Terraform supports a variety of cloud provider plugins for seamless infrastructure management. Before we get into the problem statement, let’s look at how Terraform tracks infrastructure.

Terraform State

Terraform state is a straightforward concept that stores the fundamental metadata required to refer to real-world infrastructure with the configuration. This terraform state also helps in performance optimization. Terraform state is stored in local by default as plain text in the file terraform.tfstate. This state file can also be stored remotely in an S3 bucket, which is appropriate for project team structure.

Problem Statement

As previously stated, Terraform saves all configurations in the Terraform state file as plain text. The issue arises when we want to keep sensitive data using Terraform. As an example, consider the creation of a new RDS cluster.

Let’s take a quick look at how to use Terraform to create an RDS cluster —

# rds.tf
resource "aws_db_parameter_group" "default_13" {
  name   = "rds-pg-13"
  family = "aurora-postgresql13"
  parameter {
    name  = "log_min_duration_statement"
    value = "5000"
  }
}

resource "aws_rds_cluster_parameter_group" "default_13" {
  name        = "rds-cluster-pg-13"
  family      = "aurora-postgresql13"
  description = "RDS default cluster parameter group"
}

module "aurora-postgresql-rds" {
  source = "git@github.com:terraform-aws-modules/terraform-aws-rds-aurora?ref=v7.6.0"

  name          = "test-cluster"
  database_name = "test"

  engine                      = "aurora-postgresql"
  engine_version              = "13.4"

  vpc_id  = "vpc-12345678"
  subnets = ["subnet-12345678", "subnet-87654321"]

  iam_role_name            = "rds-enhanced-monitoring-"
  iam_role_use_name_prefix = true

  instance_class = "db.r5.large"

  storage_encrypted            = true
  apply_immediately            = true
  monitoring_interval          = 10
  performance_insights_enabled = true

  db_parameter_group_name                     = aws_db_parameter_group.default_13.name
  db_cluster_parameter_group_name             = aws_rds_cluster_parameter_group.default_13.name
  db_cluster_db_instance_parameter_group_name = aws_rds_cluster_parameter_group.default_13.name

  enabled_cloudwatch_logs_exports = ["postgresql"]

  create_random_password = true
  master_username        = "test_user"
}

We’re putting together the aws_db_parameter_group and aws_rds_cluster_parameter_group required to set up an RDS cluster. The RDS module from terraform-aws-modules/terraform-aws-rds-aurora is then configured with some basic fields such as name, database_name, engine, master_username, and so on.

If we use terraform apply to apply the changes, it will create the test-cluster on AWS RDS with the username test-user and a random password. Everything is fine so far; now let’s look at the Terraform state file, which looks like this (with much more information) —

The password is stored in plain text if you see the highlighted lines above. Anyone who gains access to the state file will be able to access the database. This is a security issue. So, how do we address this concern?

Solution

To address the above concern, Terraform recommends two options:

1. Utilize the Terraform Cloud.
2. Encrypt the state file before storing it in a remote location.

Option 1 is a paid solution, and option 2 seems to be a viable alternative, but the problem persists when users download the state file to their local machine. So, how can we totally eradicate the security concerns?

We need to do something other than terraforming on a conceptual level. The simple solution is to change the password after the cluster is created. We can easily perform the manual update from the web console, but we have to repeat the process each time we deploy a new RDS cluster. So, how can we automate it?

We’ll use null_resource with the aws cli command to automate it. The null_resource implements the standard lifecycle but does nothing else.

Let’s include a script to rotate the RDS master password —

First, we generate a random password of the specified length (default 64 chars). Once the random password is generated, we save the value to secrets manager so that we can securely access the secret later. The final step is to set the newly created password as the master password for the specified RDS cluster id.

Note: It is entirely optional to generate a random password and store it in Secrets Manager. We can also pass specific passwords from outside.

Now that our script is complete, let’s use null_resource to integrate it with Terraform —

resource "null_resource" "change_rds_password" {
  triggers = {
    updated_count = 1
  }

  provisioner "local-exec" {
    working_dir = path.module
    command     = "./rotate_rds_master_password.sh $rds_cluster_id $secret_id"
    environment = {
      secret_id      = "database_master_password"
      rds_cluster_id = module.aurora-postgresql-rds.cluster_id
    }
  }
}

In this case, we used local-exec provisioner to execute the shell script that will rotate the password. The triggers attribute allows us to easily replace the resource when necessary, such as re-rotating the password.

The executable permission for rotate_rds_master_password.sh is required. We used the script’s default password length (64) here, but a custom value can be provided. For easier access, we created a secret on the secret manager called database_master_password, which will contain the newly generated secret.

Once we’ve completed the above steps, Terraform will automatically rotate passwords based on triggers.

With the above changes, Terraform State will still contain the old password, but it is no longer a security concern because the password no longer works.

Final Thoughts

With a few minor changes to our existing Terraform code, we were able to securely create an RDS cluster using Terraform. There are now three options for managing sensitive content, such as an RDS password —

1. Utilize the Terraform Cloud.
2. When storing the state file in Remote, encrypt it.
3. Use null_resource and the aws cli command to rotate the RDS password.

You must decide which solution is best suited to your use case and team. However, it is recommended that sensitive data should be handled properly.

We focused on RDS here, but similar concepts can be applied to other databases.

Note: To understand how to rotate the password for MongoDB Atlas, see this gist.

Originally posted on medium.com.