Terraform Remote State
Hello World, today we will go through a few steps on how to use Terraform remote state. Teams use terraform as a key part of infrastructure change management and deployment pipeline. Network team works on networking aspects of infrastructure whereas the application team works on managing instances in given network space. In the same way, other teams work on provisioning other aspects of infrastructure which are managed by the same terraform configuration. As multiple members of a team work on provisioning infrastructure, they all need to have the same state file. Use of local file makes terraform usage hard among team members because every user needs to have the latest state file data before running terraform configuration. Everyone among the team needs to know the current state of infrastructure so they can create what is not present and modify what is present to achieve the desired result. Team members need to make sure that no two members run terraform configuration at the same time to prevent corruption, data loss, inconsistent state.
Benefits of remote state:
- Safer storage: Storing state on the remote server helps prevent sensitive information. State file remains same but remote storage like S3 provides a layer to security like making S3 bucket private and giving limited access.
- Auditing: Invalid access can be identified by enabling logging.
- Share data: Remote storage helps share state files with other members of the team.
This allows us to break our infrastructure in different components which are managed by different teams, as everyone shares a common view of infrastructure.
Let’s us use S3 as our remote storage for terraform configuration to create EC2 instance. To use S3 as a backend, we first need to create an S3 bucket, let’s call it “terraformseau001”.
Terraform configuration to create S3:
We need to define provider with access and secret key for all separate terraform configurations:
Once we have our S3 bucket ready, let’s setup S3 as our remote backend by adding below code in our existing terraform.tf code.
With this code, we have informed terraform to use backend as S3 with bucket name to be “terraformseau001” to store its state. The path to the state file inside the bucket can be defined using the key. In our case, state file name is “terraform” and located in region “us-east-1”. There are more parameters to tune our backend which can be found here. With S3 backend, we need to define an IAM user with ListBucket permissions and permission to GetObject and PutObject in our “terraformseau001” s3 bucket. Once we are done with our configuration, we need to run “terraform init” to make sure S3 backend comes into play.
Terraform provides locking to prevent concurrent runs against the same state. Locking helps make sure that only one team member runs terraform configuration. Locking helps us prevent conflicts, data loss and state file corruption due to multiple runs on the same state file.
DynamoDB can be used as a locking mechanism to remote storage backend S3 to store state files. The DynamoDB table is keyed on “LockID” which is set as a bucketName/path, so as long as we have a unique combination of this we don’t have any problem in acquiring locks and running everything in a safe way.
To use DynamoDB as a locking mechanism, we first need to create a dynamoDB table, let’s call it “terraform-lock”.
This terraform code is going to create a dynamo DB table with the name “terraform-lock” with a key type string named “LockID” which is also a hash key.
Once we have our dynamo DB table “terraform-lock” ready, let’s setup dynamo DB as our locking mechanism with s3 remote backend by adding below code in our existing terraform.tf code:
Once this is added, we need to run “terraform init” to make sure our backend is initialised properly for usage by terraform.
With this, we can use S3 as our storage and dynamoDB as our locking mechanism. When we run “terraform apply”, it first acquires lock on dynamo db using key as “terraformbackend/terraform” with some unique value, once lock is acquired, then it start applying changes on infrastructure and then store state file on S3.
Complete code can be found in this git repository: https://github.com/MiteshSharma/TerraformWithS3Backend
Have fun learning. Cheers!
