VGW: Cloud data migration

The Client

Virtual Gaming Worlds is an innovative and highly profitable game design workshop. VGW’s turnover in 2018 was nearly $380m with tens of thousands of players across multiple product lines including Online Social Casinos and Poker platforms.

The VGW platform is a highly scalable, highly available distributed system built on AWS's global cloud platform. With customers all over the world, they leverage the power of AWS managed services to deliver a seamless customer experience and a rapid development lifecycle.

The volume of traffic and the demands of analytics, compliance and reporting on a gaming platform means that the systems generate millions of records every hour. Sitting on top of this is a sophisticated suite of financial and operational analytics.

But VGW's rapid growth outstripped the capabilities of their data platform and they required a new strategy.

The Problem

Due to the growth of the platform VGW needed to move away from point solutions to a unified data platform and pipelines.

Part of this involved the live migration of 15TB of data from the US to Europe to consolidate the platform. VGW turned to Mechanical Rock as trusted partners to deliver a seamless, scalable and repeatable solution for the data migration and ongoing transfer.

Using a combination of AWS managed data services, continuous delivery and infrastructure-as-code, we have delivered an automated CI/ CD pipeline for the data platform.

Using these techniques delivers a platform that is:

• Serverless, with minimal operational overhead and TCO.
• Secured using managed and encrypted credential handling and least privilege security principles.
• Consistent using immutable data infrastructure with the entire pipeline configured from source control and deployable with a single click.

The Solution

Various options for pipelines were investigated and discarded as not meeting VGW’s requirements and operating environment.

In the end a highly configurable custom pipeline was deployed, consisting of the following elements:

• A CI/CD pipeline for platform infrastructure, built on AWS CodePipeline and using Behaviour Driven Infrastructure to verify that deployed components are compliant and behave as expected.
• Automated provisioning of Redshift clusters using Cloudformation, as part of data platform pipeline.
• An automated data pipeline consisting of automatically provisioned and configured Apache Airflow DAGs to control flow of data, across systems and across regions.
• Improved security and compliance through implementing consistent code driven security standards across the entire platform (verified using BDI).

AWS products & services utilised for VGW's data migration.

The Benefits

Beyond the immediate goal of migrating data across AWS regions, the solution Mechanical Rock delivered offered a number of key benefits:

• Reduced latency of data availability from hours to minutes by splitting the load across multiple instances.
• Enhanced security by enforcing encryption of data in transit and at rest.
• A serverless pipeline for infrastructure-as-code with offers low management overheads and TCO, secure credential handling and consistent deployment.
• Visible compliance using Behaviour Driven Infrastructure (BDI) to assure security and configuration.
• Consistent staging environments which allow for non-production testing of infrastructure changes, pipeline changes or database schema migrations.
• Repeatable patterns and tools for cross account and region migrations which can be easily re-used for future migrations.

“Mechanical Rock helped us put our first automated deployment pipelines together and kicked off our DevOps practices in the company. They’re a talented team who have helped us with a lot of projects over the last couple of years.”

Matt James
Chief Technology Officer
VGW Holdings Ltd