Process Data at Volume
With a Custom BI Solution

Visualising large volumes of data into a purpose built Business Intelligence platform. Enabling live content analysis in an intuitive user interface.

Client: DFL
Industry: Sports & Media
Services: Data Analysis & BI Platforms

Opportunity

DFL specialise in marketing and organising football matches, then broadcasting them to millions of users around the world. DFL contacted Firemind to create a new system that would assist them in processing large volumes of data.

DFL had existing data processing platforms, such as open sourced Matomo setup on Elastic Beanstalk, using Google Analytics and Google Tag Manager to view data and forward data actions. However, they wanted to use Amazon Kinesis to visualise the data sent through, converting data into readable and actionable files. This would then be used by their applications team to display data to users in their iOS and Android apps after being outputted to S3.

If DFL’s challenge wasn’t addressed, they wouldn’t be able to get data to display to users in any of their mobile applications.

Custom Analytics Visualisations

Firemind crafted a one of a kind Business Intelligence solution that could display data from all users and deliver actionable information.

Unlimited Scaling

We knew that with a wealth of data and users, data storage and scaling had to be flexible and unlimited in capacity.

GDPR Compliant

AWS Lambda used to modify the records sent through the Firehose stream, complying with GDPR regulations on data privacy and control.

Firemind crafted a one of a kind Business Intelligence solution that could display data from all users and deliver actionable information.

We knew that with a wealth of data and users, data storage and scaling had to be flexible and unlimited in capacity.

AWS Lambda used to modify the records sent through the Firehose stream, complying with GDPR regulations on data privacy and control.

Solution

Firemind created a fully managed solution with the use of Amazon Kinesis Data Streams, Firehose, Amazon S3, CloudFront and AWS Lambda.

This system was built from the ground up purely on AWS services. Firemind ensured that the methods used could scale to demand and, given that there was very frequent live traffic, auto scaling would have to be in place to ensure that data sent through the Kinesis Data Streams could be processed without being discarded.

Included in this solution was Kinesis Data Streams, Firehose, Amazon S3 and AWS Lambda. Data Streams were used to receive logs from CloudFront Real-time logging, which were then forwarded to Kinesis Firehose, an AWS Managed service.

Firemind then created a Lambda to modify the records sent through the Firehose stream, to comply with GDPR. Specifically, Firemind removed the last two octets of the IP addresses from CloudFront logs on modification (12.34.0.0), and removed any null values returned by CloudFront logs.

Once records had been processed through Amazon Kinesis using its conversion Lambda modification option, Firemind then utilised Kinesis Firehose to output this data into an S3 bucket in a different account. This data was outputted in Apache Parquet format and provided DFL members access to the business intelligence platform and reports.

Firemind also setup an AWS Glue crawler, to automatically go through the Parquet files to scrape new data and observe if the data model had changed – this ran every 24 hours.

The Architecture

Outcome

Once the solution was completed by Firemind, it was tested thoroughly and enabled for use. Instantly, using Amazon Kinesis and CloudFront, DFL saw exceptional results with their data being outputted to S3 in a short span of 5 minutes from the start of flow to display. This new solution was extremely fast and met the needs of DFL, allowing their internal teams to visualise their data in file form.

Given that requests are sent off by Google Tag Manager to the CloudFront Distribution at a rapid rate, results were seen quickly with Amazon Kinesis Firehose and Amazon S3, outputting Parquet files to the relevant users. This was then consumed by DFL’s Matomo configuration which then gave the user who logged in an instant understanding of what data has been processed.

Overall, due to CloudFront being in place, S3 costs were cut down for DFL by 35%.

CloudFront was used for content acceleration and origin security. This also gave Firemind the option to enable AWS WAF for the distribution to enhance security even further.

AWS

Being AWS Partners, we leverage AWS services for everything from hosting your cloud environment to storing, computing and transforming your data.

Here are some of the AWS services that allowed us to deliver on this project.

Amazon Kinesis FireHose
Amazon CloudFront
AWS Lambda

Start your journey with us today