Efficient Data Migration Between Existing AWS DynamoDB Tables

Photo by Mehmet Ali Peker on Unsplash

When it comes to migrating data within AWS DynamoDB, the challenge often lies not in the complexity of the data itself, but in the limitations posed by the available tools and methods. Traditional approaches, such as backup and restore operations via S3 buckets, are well-suited to scenarios where the target table is created as part of the migration process. However, they fall short when the task at hand involves moving data to a pre-existing table, especially when the migration needs to be both simple and quick.

The Need for a Custom Solution

The primary motivation behind this custom solution was the absence of a straightforward method for migrating data directly between existing DynamoDB tables. Traditional methods, while effective for certain use cases, did not offer the simplicity or directness needed for quick, one-time migrations. This prompted the development of a solution that could meet these specific requirements without the need for intermediary storage or complex setup procedures. We use DynamoDB streams and Lambda to implement this solution.

Lambda Function for Data Migration

The core of our migration logic resides within a Lambda function that listens to DynamoDB Stream events. This function is designed to handle both the creation of new records and the modification of existing ones. For records flagged with the migrate boolean, the function removes this field before inserting the record into the target table. Additionally, it handles the deletion of records from the target table if they are removed from the source table.

import {
  DynamoDBClient,
  DeleteItemCommand,
  PutItemCommand,
  AttributeValue,
} from '@aws-sdk/client-dynamodb';
import { DynamoDBStreamEvent } from 'aws-lambda';

const client = new DynamoDBClient();
const tableName = process.env.TARGET_TABLE;

export const handler = async (event: DynamoDBStreamEvent) => {
  console.log('Processing DynamoDBStreamEvent', JSON.stringify(event));
  for (const record of event.Records) {
    try {
      if (record.eventName === 'REMOVE') {
        const input = {
          TableName: tableName,
          Key: record.dynamodb?.Keys as Record<string, AttributeValue>,
        };
        await client.send(new DeleteItemCommand(input));
      } else {
        let item = record.dynamodb?.NewImage;
        if (item && item.migrate) {
          delete item.migrate;
        }
        if (item.Envelope && item.Envelope.B) {
          item.Envelope.B = Uint8Array.from(Buffer.from(item.Envelope.B, 'base64'));
        }
        let input = {
          TableName: tableName,
          Item: item
        };
        await client.send(new PutItemCommand(input));
      }
    } catch (error) {
      console.error('Error processing record', JSON.stringify(record), error);
    }
  }
};

Marking Records for Migration

The final step in this custom migration process involves marking the records in the source table that need to be migrated. This is accomplished using a simple script that scans the source table and adds a boolean field named migrate to each record. This field serves as a flag to identify records ready for migration, ensuring that only the necessary data is moved.

import { DynamoDBClient, ScanCommand, PutItemCommand } from "@aws-sdk/client-dynamodb";

const client = new DynamoDBClient({ region: "us-west-2" });
async function readAndWriteItems(tableName: string) {
  const scanCommand = new ScanCommand({ TableName: tableName });
  const scanResponse = await client.send(scanCommand);
  for (const item of scanResponse.Items || []) {
    item.migrate = { BOOL: true };
    const putCommand = new PutItemCommand({ TableName: tableName, Item: item });
    await client.send(putCommand);
  }
}

This script is then executed for each source table that contains records to be migrated, effectively migrating our data to the new table.

Conclusion

This custom solution for migrating data between existing AWS DynamoDB tables fills a crucial gap in the available data migration methods. By focusing on simplicity and directness, it provides a viable option for scenarios where traditional methods are either too complex or not applicable. While this approach is tailored to specific needs and may not be universally applicable, it offers a streamlined alternative for simple, one-time data migrations within the DynamoDB ecosystem.

It’s important to note that while this method worked well for our specific use case, it is by no means the only solution available. Depending on the complexity and scale of your data migration needs, other approaches might be more suitable. However, for those facing similar challenges, this custom approach offers a practical and efficient solution for migrating data between pre-existing DynamoDB tables.

In Plain English 🚀

Thank you for being a part of the In Plain English community! Before you go:

• Be sure to clap and follow the writer ️👏️️
• Follow us: X | LinkedIn | YouTube | Discord | Newsletter
• Visit our other platforms: Stackademic | CoFeed | Venture | Cubed
• More content at PlainEnglish.io