Pioneer Credit: Snowflake data platform

The Challenge

Pioneer Credit, an Australian-based business with a mission to put an end to debt stress, has embarked on a bold technology modernisation strategy that has them making significant investments in cloud-based systems. A critical step in this journey is the replacement of their on premises data and analytics platforms with a robust, secure and scalable Snowflake data warehouse that they are able to grow and maintain.

The Solution

Mechanical Rock, in close collaboration with Pioneer’s Data & Analytics team, designed and implemented a tailored data platform solution that prioritised ease of adoption, scalability, and ongoing self-sufficiency.

Snowflake infrastructure set up

Set up and configuration of the Snowflake platform in readiness for data ingestion, including:

Role-based access control (RBAC) and security architecture
Databases and schemas
Virtual warehouses, resource monitors, masking policies, alerting and tags
Resources management and testing framework
Git integration and automated deployment to multiple environments

Tooling set up and training

Adoption of fit-for-purpose ingestion and transformation tooling to enable a diverse range of use cases (and data sources) in a cost effective way, including Fivetran, Azure Data Factory (ADF), and dbt. Leveraged infrastructure-as-code (IaC) for the deployment of connectors (Terraform) and resources (Azure Resource Manager) enabling significant speed and ease of ongoing deployments. The Pioneer team paired extensively with Mechanical Rock engineers throughout this project to build their knowledge and confidence around these tools. The rationale for using the selected technologies, and their associated benefits and challenges, was well documented.

Integration between the platform and key data sources

Establishment of secure ingestion connections between a wide variety of data sources both internal and external to the organisation, including:

Key operational systems
Legacy on premises data stores
Third-party SFTP servers
Flat file data

The Benefits

The implementation of the Snowflake data platform, and the accompanying capability uplift, brought several significant benefits to Pioneer Credit:

Enablement of use cases: Mechanical Rock established reusable patterns to enable the flow of data into and out of the platform in support of Pioneer’s broad range of use cases. Pairing closely with Pioneer’s data engineers, key business requirements were implemented to pilot each of these patterns, creating both the knowledge and framework for broader application and reuse.

Adoption of modern engineering practices: Mechanical Rock introduced the Pioneer team to best practices in version control, including deployment pipeline design and peer reviews. By the end of the engagement there are git integrations across all core development activities allowing traceability of what changes were made and by who, as well as a broader understanding of good coding practices. Further, all resources were managed as code using IaC - Terraform and schema change tools.

Improved data governance and security posture: A framework for data classification and security was implemented that included Snowflake tag based masking policies and row based access controls. Further, dbt documentation was exposed as an initial data catalog that can be accessed by non-Engineers to understand data lineage and metadata.

Enhanced scalability: With Snowflake running on Azure, Pioneer can easily scale their resources to accommodate varying workloads over time, without compromising performance.

Significant upskilling and capability build: A key focus throughout the engagement was to ensure that the Pioneer team were able to confidently own and operate the new Snowflake data platform. This involved working closely with Pioneer Systems Administrators and Data Engineers at every stage, and conducting all implementation activities as Mechanical Rock / Pioneer Credit pairs. The new roles and responsibilities arising from the introduction of the new data platform were carefully documented, along with a framework detailing the skill sets required to cover different parts of the data lifecycle.