Terraform for Data Engineers: How to Automate Your Database Setup

 

Stop Manual Setup: Deploy a PostgreSQL Database with Terraform

If you are still manually creating databases in the AWS or Azure console, you are creating a "Snowflake Server"—a unique setup that no one can replicate if it breaks.

In 2026, professional data teams use Terraform. It allows you to write your infrastructure as code, version control it on GitHub, and deploy it perfectly every single time.

1. What is Terraform?

Terraform is a tool that lets you define your infrastructure (Databases, S3 Buckets, Kubernetes clusters) using a simple language called HCL (HashiCorp Configuration Language).

2. The Setup: provider.tf

First, we tell Terraform which cloud we are using. For this guide, we’ll use AWS, but the logic works for any cloud.

provider "aws" {

region = "us-east-1"

}

3. The Code: main.tf

Instead of clicking "Create Database," we write this block. This defines a small, cost-effective PostgreSQL instance.

resource "aws_db_instance" "datatipss_db" {

allocated_storage = 20

engine = "postgres"

engine_version = "15.4"

instance_class = "db.t3.micro" # Free tier eligible!

db_name = "analytics_db"

username = "admin_user"

password = var.db_password # Use a variable for security!

skip_final_snapshot = true

publicly_accessible = true

}

4. The Magic Commands

Once your code is written, you only need three commands to rule your infrastructure:

1. terraform init: Downloads the AWS plugins.

2. terraform plan: Shows you exactly what will happen (The "Preview" mode).

3. terraform apply: Build the database!

Reduce AWS Bills by 60%: Automate EC2 Stop/Start with Python & Lambda

In 2026, cloud bills have become a top expense for most tech companies. One of the biggest "money-wasters" is leaving Development and Staging servers running over the weekend or at night when no one is using them.

If you have a t3.large instance running 24/7, you're paying for 168 hours a week. By shutting it down outside of 9-to-5 working hours, you can save over 65% on your monthly bill.

The Solution: "The DevSecOps Auto-Stop"

We will use a small Python script running on AWS Lambda that looks for a specific tag (like Schedule: OfficeHours) and shuts down any instance that shouldn't be running.

Step 1: Tag Your Instances

First, go to your AWS Console and add a tag to the instances you want to automate:

• Key: AutoStop

• Value: True

Step 2: The Python Script (Boto3)

Create a new AWS Lambda function and paste this code. It uses the boto3 library to talk to your EC2 instances.

import boto3

ec2 = boto3.client('ec2', region_name='us-east-1') # Change to your region

def lambda_handler(event, context):

# Search for all running instances with the 'AutoStop' tag

filters = [

{'Name': 'tag:AutoStop', 'Values': ['True']},

{'Name': 'instance-state-name', 'Values': ['running']}

]

instances = ec2.describe_instances(Filters=filters)

instance_ids = []

for reservation in instances['Reservations']:

for instance in reservation['Instances']:

instance_ids.append(instance['InstanceId'])

if instance_ids:

ec2.stop_instances(InstanceIds=instance_ids)

print(f"Successfully stopped instances: {instance_ids}")

else:

print("No running instances found with AutoStop=True tag.")

Step 3: Set the Schedule (EventBridge)

You don't want to run this manually.

1. Go to Amazon EventBridge.

2. Create a "Schedule."

3. Use a Cron expression to trigger the Lambda at 7:00 PM every evening:

   • cron(0 19 ? * MON-FRI *)

How to Fix Terraform Error: "Error acquiring the state lock"

 You try to run terraform plan or apply, and instead of seeing your infrastructure changes, you get hit with this wall of text:

Error: Error locking state: Error acquiring the state lock Lock Info:

ID: a1b2c3d4-e5f6-g7h8-i9j0 Operation: OperationTypePlan

Who: user@workstation Created: 2026-01-17 10:00:00 UTC

Why does this happen?

Terraform locks your State File to prevent two people (or two CI/CD jobs) from making changes at the exact same time. This prevents infrastructure corruption. However, if your terminal crashes or your internet drops during an apply, Terraform might not have the chance to "unlock" the file.

Step 1: The Safe Way (Wait)

Before you do anything, check the Who and Created section in the error. If it says your colleague is currently running a plan, don't touch it. Wait for them to finish.

Step 2: The Manual Fix (Force Unlock)

If you are 100% sure that no one else is running Terraform (e.g., your own previous process crashed), you can manually break the lock using the Lock ID provided in the error message.

Run this command:

terraform force-unlock <LOCK_ID>

Example: terraform force-unlock a1b2c3d4-e5f6-g7h8-i9p0

Step 3: Handling Remote State (S3 + DynamoDB)

If you are using AWS S3 as a backend, Terraform uses a DynamoDB table to manage locks. If force-unlock fails, you can:

1. Go to the AWS Console.

2. Open the DynamoDB table used for your state locking.

3. Find the item with the Lock ID and manually delete the item from the table.

Pro-Tip: Preventing Future Locks

If this happens frequently in your CI/CD (like GitHub Actions or Jenkins), ensure you have a "Timeout" set. Also, always use a Remote Backend rather than local state files to ensure that if your local machine dies, the lock is manageable by the team.

terraform {

backend "s3" {

bucket = "my-terraform-state"

key = "network/terraform.tfstate"

region = "us-east-1"

dynamodb_table = "terraform-lock-table" # Always use this!

}

}