Streaming Amazon DynamoDB Data Into a Centralised Data Lake

For organizations moving towards a serverless microservice approach, Amazon DynamoDB has become a preferred backend database due to its fully managed, multi-region and multi-active durability.

It also boasts built-in security controls, backup and restore functions and in-memory caching for internet-scale applications. This in turn allows you to derive near-real-time business insights. The data lake provides capabilities to business teams to plug in BI tools for analysis and to data science teams to train models.

This post from the AWS Big Data team demonstrates some common use cases for streaming an Amazon DynamoDB table into an Amazon Simple Storage Service (Amazon S3). This stream uses a multitude of services, including Amazon Kinesis Data Streams, AWS Lambda and Amazon Kinesis Data Firehose.

The architecture diagram above illustrates the solution in use.

What You’ll Find in the AWS Post

If you visit the article, you’ll be able to see further information on:

1. Security being the primary focus of both use cases.
2. Using Amazon DynamoDB and Amazon S3 in the same AWS account.
3. Detailed instructions on deploying and testing the solutions.
4. Using Amazon DynamoDB and Amazon S3 in different AWS accounts.
5. The potential of these solutions when combining them with analysis/reporting tools.

How can Firemind assist with Data & Analytical Projects?

Here at Firemind, we’ve worked with countless clients to help decipher and best use large volumes of data within their businesses.

If you’d like to speak with us about a Data or Analytical project, please Get in Touch today and a member of our Sales team would be happy to discuss your requirements.