Migrating NAT instance from NAT Gateway

This is the sequel to my previous blog post Set Elastic IP to a Lambda function with NAT instance, using CDK. I’m going to write about how we can migrate an existing NAT gateway to a NAT instance. Yes, we can replace them, but there’s a catch!

What you should be aware of

As of April 2024, we can not update the VPC setup through CDK due to an issue reported in https://github.com/aws/aws-cdk/issues/6683. This issue applies when we want to remove NatGateway from CDK. Therefore, you need to understand that this migration requires a manual step, even if you are using CDK. If you deploy your CDK with CodeBuild, you must consider a workaround.

Migration steps overview

There are three steps:

1. Deploy a Nat Instance in the same VPC in a public Subnet.
2. Manually remove NAT Gateway.
3. Switch the network routing to NAT instance.

Let’s walk through this step with an example.

Actual example

For this example, we create a VPC with NAT Gateway, and Lambda function and Aurora Postgres DB is in the same network. If you want to follow this example on your on, here’s a CDK stack:

elastic-ip-lambda/bin/simple-nat.ts at main · tomoima525/elastic-ip-lambda

pnpm i
pnpm cdk deploy --app 'npx ts-node bin/simple-nat.ts'

Internally, this stack will call https://github.com/tomoima525/elastic-ip-lambda/blob/main/lib/vpc-rds-stack.ts , which creates the setup below.

NAT Gateway

The Lambda function checks IP address of the located network by accessing https://api.ipify.org?format=json. In this scenario, the lambda function accesses this external service through NATGateway which routes to one of PublicSubnets’ public IP.

IP is 52.26.200.190

52.26.200.190 is assigned to PublicSubnet

Now let’s start the migration!

Step1. Create NAT Instance

You can create NAT instance by using this Stack:

elastic-ip-lambda/bin/nat-migration.ts at main · tomoima525/elastic-ip-lambda

pnpm cdk deploy --app 'npx ts-node bin/nat-migration.ts'

In the CDK, I used Amazon Linux 2023. The AMI is deployed under the following settings:

• Source/Dest. check is disabled
• Exists in the public subnet of VPC created
• Set Keypair for SSH access
• Set Security Group for SSH access (Accept port 22)

    new ec2.Instance(this, `nat-instance`, {
      vpc,
      securityGroup: publicSg,
      vpcSubnets: { subnetType: ec2.SubnetType.PUBLIC },
      instanceType: ec2.InstanceType.of(
        ec2.InstanceClass.T3,
        ec2.InstanceSize.MICRO,
      ),
      machineImage: ec2.MachineImage.latestAmazonLinux2023(),
      keyName: "testkeypair2", // replace with your own keypair
      sourceDestCheck: false, // should be false for NAT instances
      associatePublicIpAddress: true, // assigns public IPs to this instance
    });

We also run a script to set up iptable routing for NAT after the instance is deployed

const initScriptPath = path.join(`${__dirname}/`, "init-script.sh");
const userData = fs.readFileSync(initScriptPath, "utf8");
customNat.addUserData(userData);

//init-script.sh
# install iptable
sudo yum install iptables-services -y
sudo systemctl enable iptables
sudo systemctl start iptables

# Turning on IP Forwarding
echo "net.ipv4.ip_forward = 1" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

# Making a catchall rule for routing and masking the private IP
# Amazon Linux 2023 primay network interface is ens5
sudo iptables -t nat -A POSTROUTING -o ens5 -s 0.0.0.0/0 -j MASQUERADE
sudo /sbin/iptables -F FORWARD
sudo service iptables save

Step 2: Remove NAT Gateway manually from AWS console

Go to NAT gateway in VPC console to delete the NAT Gateway.

Step 3: Route all traffic from private subnets to NAT instance

0.0.0.0/0 routing to NAT Gateway in Private Subnet will cause conflicts when we route all traffic from Private Subnets to NAT instance route. You can do so from AWS console.

Head to VPC and select Resource map tab.

Select Private Subnet from Route tables in Resource map. You will see that Status is Blackhole for nat since it no longer exists.

Select NAT Gateway routes and remove.

Then add NAT instance to 0.0.0.0/0 routing.

You should do this for both of your private Subnets.

Updated network

Now, your network should look like this:

NAT instance

Test the Lambda function again and see if it returns the IP assigned to the NAT instance.

35.166.128.31 is assigned

Wrap up

We walked through the migration step using the actual example. It’s quite tedious! However, once you switch to a NAT instance, the cost will be lower than that of a NAT Gateway.

One final note: I’d recommend thoroughly testing the migration before applying it to your environment. In the example, we had Aurora DB and Lambda, but the AWS resource that you have in your VPC might be different and they could cause unforeseen issues.

In Plain English 🚀

Thank you for being a part of the In Plain English community! Before you go:

• Be sure to clap and follow the writer ️👏️️
• Follow us: X | LinkedIn | YouTube | Discord | Newsletter
• Visit our other platforms: Stackademic | CoFeed | Venture | Cubed
• More content at PlainEnglish.io