CustomerSCANOSS
IndustryInformation Technology
ServiceCloud Storage
SegmentSMB

Improved storage optimisation and retrieval time for SCANOSS

ServiceCloud Storage
IndustryDigital Solutions

Improved storage optimisation and retrieval time for an Open Source Inventory engine

Meet SCANOSS

SCANOSS have revolutionised the SCA space with their 100% Open Source platform – made by Developers, for Developers. The SCANOSS team have empowered Developers worldwide to confidently produce compliant code from the very moment they begin typing.

SCANOSS transforms software bill of materials (SBOM) creation from ‘write now, audit later’ to an always-on analysis of live code. By freeing developers to focus on writing great and compliant code, applications are finished earlier, quality is consistently higher and development costs are dramatically lowered for businesses.

65%

Reduction in codebase scanning times.

Allowing codebase edits and updates to be performed faster and more efficiently.

Business challenges

The challenges that SCANOSS faced were quite unique, in comparison to other data retrieval businesses. As one of the first companies to provide a completely open source SCA tooling system, they have an extremely large dataset of 11 Terabytes! This means they’ve been able to index the majority of public open source code (from sources such as GitHub, Bitbucket, GitLab, RhodeCode).

In order for SCANOSS to scan the open source material and produce results in a timely manner, they needed incredibly high IOPS (Input/Output Operations per second). Such a high level of reads/writes puts incredible strain, as well as cost, on the processes involved.


Why Firemind?

SCANOSS needed the support of an AWS specialist that could provide immediate assistance. The retrieval timings and associated costs were something that had been a priority change for the team, so they knew they needed to partner with a consultancy who could have the technical understanding and innovative capabilities, as well as ask ‘fast and thorough’.

Firemind were ready to assist, with a dedicated Solutions Architect and Developer team who could not only theorise ways to reduce this IOPS burden, but also action architecture changes and costing analysis.


The focus on this project was to test storage optimised compute options in Amazon Web Services (AWS), with the ultimate goal to fully migrate to AWS. In order to achieve this, we had to prove the viability of a new architecture, one that could outperform their current production and workflows.

Due to the level of incredibly high IOPS, we knew that current AWS data services (such as Amazon RDS, Amazon Aurora and Amazon DynamoDB) would not be able to match the current performance they were experiencing, whilst providing speed and cost benefits. They were using Microsoft Azure with a dedicated disk and flat files. To combat this, we instead took the route of using Amazon S3 combined with Amazon EC2 I3 Instances.

Amazon EC2 I3 instances are the next generation of Storage Optimised instances for high transaction, low latency workloads. I3 instances offer the best price per I/O performance for workloads such as NoSQL databases, in-memory databases, data warehousing, Elasticsearch and analytics workloads.

From the utilisation of EC2 I3, we could take our sample data (40GB from the 11TB) and begin to check time and cost savings against the data updates and high IOPS. We found that we could cut down the scan and produce results from 79 minutes to around 29-31 minutes. This led to a 65% reduction in overall time against further tests, with cost savings equally reflecting the speed.

Discussion with Lead Solutions Architect

We caught up with the Firemind lead Solutions Architect on the SCANOSS project, Jason Golding, to find out more about the requirements, challenges and overall successes.

Hit play for a quick summary on the customer and how we shaped a new cloud architecture with transformative effect, using the best in AWS services.

Added value

Taking SCANOSS on a fast and effective journey that would enable their future data scaling enabled a swift and ‘jam packed’ timeline. From finding the right tools that could enable faster IOPS, to figuring out the migration from Microsoft Azure to AWS.

8 days

From ideation to migration to Amazon Web Services using Amazon EC2 I3.

Time to value

As with many of our data migration projects, we were able to quickly understand the customer’s goals and work backwards to ensure we crafted an architecture fit for purpose. Within 8 days, split across 7 weeks, we worked closely with SCANOSS and were able to deliver the high IOPS scan times and costs savings they wanted.


Integrated training

Both Firemind and SCANOSS worked very closely to ensure that we could quickly identify challenges in the project, as well as provide ongoing training on the benefits of EC2 I3 instances. This would ensure the SCANOSS Lead Developer would feel confident moving forwards as their Microsoft Azure environment moved to AWS.


High transaction workloads

11TB of open source data is no joke! And having a business that needs to scan, update and address edits and changes to such a vast amount takes some high transaction workloads. This project showcased the next generation usage of optimised storage solutions, with results that will have a direct effect on the future of SCANOSS and the services they provide to their customer base.


Customer satisfaction

“AWAITING CUSTOMER QUOTE – 22.08.22”

TBC

TBC

How can we help you?

Speak with us to help with your high transaction data workloads. Providing an accelerated journey to AWS high transaction and low latency services.

Get in touch