Date
Nov 2025
Client
Mineral Resources
The Challenge
Mineral Resources, a leader in the Australian mining and mining services sector, is renowned for its focus on innovation and operational excellence. As part of its ongoing technology modernisation journey, Mineral Resources is transforming its data and analytics environment to unify data from key operational systems and improve scalability, performance, and accessibility across the organisation.
The existing data warehousing solution had become a bottleneck for growth - limiting flexibility, slowing delivery cycles, and making it difficult to integrate new data sources and analytical tools. To overcome these challenges, Mineral Resources set out to establish an enterprise-grade data platform on Databricks, designed to gradually replace the legacy data warehouse and provide a single, scalable environment that brings together all critical data sources. The new platform enables governed, consistent access to data across business units, laying a strong foundation for advanced analytics, AI workloads, and data-driven decision-making.
The Solution
To accelerate the transition and ensure alignment with best practices, the Data and Insights (DIA) team engaged Mechanical Rock to assist with the initial platform development, infrastructure setup, and migration of key ingestion pipelines. Working closely with Mineral Resources, Mechanical Rock delivered foundational platform components and the first production-ready use cases, providing a blueprint for future data initiatives.
Databricks Infrastructure and Workspace Setup using Terraform
The engagement began with the design and deployment of the Databricks infrastructure using Terraform. An existing framework was significantly enhanced to meet Mineral Resources’ operational and governance requirements, ensuring repeatable and reliable provisioning of both account-level and workspace-level resources.
New Terraform modules were developed to streamline the creation of key components - including service principals, SQL warehouses, Unity Catalog objects, and access permissions - as well as data access to AWS resources such as S3 and Kinesis. This enabled Mineral Resources to deploy new workspaces with minimal manual effort, providing a consistent and scalable foundation for future growth.
Config-Driven Declarative Pipelines for Data Ingestion
A configuration-driven ingestion framework was developed to simplify and standardise the creation of new data pipelines. Leveraging Databricks’ native declarative capabilities, this framework allows new ingestion pipelines to be defined through configuration files rather than custom code.
The framework supports a range of data sources - including S3, on-premises Kafka servers, and AWS Kinesis - and was extended to support historical backfill from S3 for Kafka topics with limited data retention. This modular design improved maintainability, accelerated onboarding for new data sources, and ensured consistent ingestion patterns across environments.
Introduction of DevOps Practices and CI/CD Pipelines
To bring consistency and automation to platform delivery, Mechanical Rock introduced a comprehensive DevOps approach using Azure DevOps CI/CD pipelines.These pipelines automate deployment workflows for Databricks assets, dbt resources, and ingestion pipelines (via Fivetran), ensuring that all changes are reproducible, traceable, and governed.
The new CI/CD process reduced manual intervention, improved deployment reliability, and established strong environment parity across Development, Staging, and Production.
Alignment with Mineral Resources Release and Governance Framework
Recognising Mineral Resources’ existing governance and change management practices, the deployment and release process for Databricks was integrated with the organisation’s established framework.
All changes now flow through a structured promotion path - from local development to staging and production - with peer review, approval workflows, and formal change requests ensuring compliance and traceability.
Data Governance Foundations: Tagging, Access Control, and Metadata Integration
As part of a broader enterprise data governance initiative, Mechanical Rock supported the integration of Databricks with Atlan, Mineral Resources’ new enterprise data catalog. The engagement introduced:
- Automated tagging of Databricks assets using dbt tags and metadata propagation.
- An investigation into reverse tagging from Atlan to Databricks to inform future roadmap opportunities.
- Config-driven user group management, aligning access control with governance policies.
This established a foundation for enhanced visibility, security, and governance across the data platform.
Commenced the Migration of operational data sources from Redshift to Databricks
Mechanical Rock assisted Mineral Resources with the migration of four critical operational systems from legacy AWS infrastructure (Lambda, Step Functions, EventBridge, Redshift) to the new unified Databricks-centric platform. The migrated systems included:
- Three Custom API Connectors - Fivetran SDK connectors were developed using modern, maintainable approaches for these systems. The process required reverse-engineering existing Lambda code to understand historical data ingestion patterns, API behaviours, and configuration requirements.
- One Native DynamoDB Connector - this system was integrated via Fivetran's native DynamoDB connector with cross-account IAM role configuration, eliminating complex AWS Step Functions orchestration.
This process marked the commencement of the planned migration journey from Redshift to Databricks for Mineral Resources, which will see them maximising the value of their Databricks investment.
The Benefits
The implementation of the Databricks platform represents a major step in Mineral Resources’ data modernisation journey, delivering both technical and organisational benefits:
- A Scalable, Modern Data Platform - By consolidating data from multiple operational systems into a single Databricks environment, Mineral Resources now has a unified, cloud-native platform that supports both batch and streaming workloads. The new platform provides the scalability, flexibility, and governance needed to support enterprise-wide analytics and prepare for AI and Machine Learning initiatives.
- Increased Agility and Delivery Speed - Automated CI/CD pipelines, declarative configuration patterns, and Terraform-based infrastructure deployment have dramatically shortened delivery cycles. Teams can now deploy new pipelines and environments with minimal manual effort, enabling rapid experimentation and faster time-to-value for new use cases.
- Upskilling and Capability Building - Throughout the engagement, Mineral Resources engineers were deeply involved in the design and implementation process, gaining practical experience in Databricks, dbt, Terraform, and modern DevOps practices. This hands-on collaboration fostered a stronger internal capability within the DIA team, enabling them to confidently manage, extend, and operate the platform independently.
- Strong Governance and Compliance - The integration of Databricks with Atlan and alignment with Mineral Resources’ existing change control processes ensures that all platform changes are traceable, controlled, and compliant with internal governance requirements. This structure balances agility with control - allowing innovation while maintaining oversight and data security.
- Reusable Framework for Future Growth - The infrastructure modules, CI/CD pipelines, and ingestion frameworks developed during this engagement serve as reusable building blocks that can be extended to additional domains and use cases. This not only accelerates future platform adoption but also ensures a consistent and standardised approach across all data engineering efforts.
- Foundations for Advanced Analytics and AI - By establishing a modern, governed, and scalable platform, Mineral Resources is now positioned to introduce AI and ML workloads with confidence. The platform’s unified data model, robust governance, and modern orchestration capabilities provide a trusted foundation for future predictive analytics, optimisation models, and automation initiatives.
Through close collaboration with Mechanical Rock, Mineral Resources successfully laid the foundations for a modern, scalable, and unified data platform on Databricks. This transformation not only enables the gradual migration away from their existing data warehousing solution but also positions the organisation to unlock new value from its data - improving agility, strengthening governance, and preparing the business for a future driven by advanced analytics and AI.