Event-Driven Workflow With AWS Glue

Recently published by the AWS Big Data team is the exciting tutorial on how to configure AWS Glue workflows to run based on real-time events.

This means that you no longer have to build complex solutions to trigger jobs based on events or set annoying schedules – AWS Glue event-driven workflows take care of that heavy lifting, allowing Data Scientists and Developers to spend their time elsewhere.

Why is this important? There is a constant need to find, train and use simplified tools with rich data processing features. Tools that build pipelines that enrich data, allowing it to move in and out of their data lake and data warehouses freely, whilst maximising analytical features. AWS Glue is that such tool.

It makes it easier to discover, prepare and combine multiple streams of data for analytical and machine learned solutions.

Typical Architecture Overview of the Solution

The AWS Big Data post starts with the core features of AWS Glue and delves into setting up the ‘hydration’ of both data lakes and data warehouses.

We’ve found the breakdown incredibly useful as it lists, step by step, the configuration of a rule in Amazon EventBridge to forward specific events into AWS Glue.

Be sure to visit the AWS Blog to view the setup in detail.

Would you like to know more about AWS CloudFormation, AWS Glue and Amazon EventBridge?

You may be wondering how these services can apply to your own use of data? Reach out to us below and we’ll be in touch.