Databricks Engineer

Experience Level: Associate

Practice Area: Engineering

Engagement Type: Full time

Location: Hybrid

‍

Role Summary

We are looking for a Databricks Engineer to design, build, and optimise data pipelines, Delta Lake workloads, and ML/AI-enabled data processing on the Databricks Lakehouse Platform. This role focuses on hands-on development across ingestion, transformation, workflow orchestration, Databricks Serverless, and Unity Catalog governance. You will work closely with solution architects, data engineers, and platform teams to deliver reliable, secure, and scalable data solutions for financial services clients.

‍

The Difference You’ll Make

In this role you will help FSI clients modernise core data processing by building efficient and well-governed pipelines on the Databricks Lakehouse. Your work will improve the speed and quality of analytics, reporting, fraud insights, and customer intelligence. You will directly contribute to faster claims processing, better risk modelling, and more accurate regulatory reporting across the FSI sector.

‍

Key Responsibilities

• Develop and maintain scalable data pipelines using Databricks, Delta Lake, and Databricks Serverless
• Implement ingestion frameworks for batch, streaming, and semi-structured data using Autoloader, Delta Live Tables, and Structured Streaming
• Build transformation logic in Python, SQL, and PySpark aligned with Lakehouse standards
• Implement governance controls using Unity Catalog including schema management, access controls, lineage, and data quality checks
• Automate workflows using Jobs, Workflows, and CI/CD pipelines
• Optimise compute usage, job performance, and pipeline cost efficiency
• Develop data validation and monitoring routines to ensure accuracy and reliability
• Support ML/AI workflows including feature engineering, model scoring pipelines, and MLflow integration
• Collaborate with architects and platform engineers to implement integration, networking, and security patterns
• Conduct code reviews, testing, troubleshooting, and ongoing production support
• Document data flows, development standards, and operational procedures
• Work closely with FSI stakeholders to refine requirements and deliver high-quality engineering outputs

‍

Required Skills and Competencies

• Hands-on experience engineering data pipelines on Databricks
• Strong development skills with Python, SQL, and PySpark
• Practical experience with Databricks Serverless (jobs, workflows, notebooks, Lakebase)
• Good understanding of Delta Lake internals: ACID, schema evolution, Z-ordering, clustering, optimisation techniques
• Experience with streaming ingestion such as Autoloader, DLT, and Structured Streaming
• Familiarity with Unity Catalog, data governance, and fine-grained access controls
• Experience integrating with cloud storage (ADLS, S3, GCS) and cloud-native services
• Understanding of networking and connectivity concepts including PrivateLink, APIM, service endpoints, and controlled egress
• Exposure to MLflow, model serving, or feature engineering workflows is an advantage
• Knowledge of CI/CD, Git workflows, Infrastructure as Code, and DevOps practices
• Strong problem-solving and debugging abilities
• Ability to operate within agile, collaborative, and cross-functional teams

‍

Qualifications and Experience

• 3+ years experience in data engineering or software engineering
• 1–3 years hands-on experience with Databricks in production environments
• Familiarity with FSI use cases such as claims, customer 360, fraud detection, and regulatory reporting is preferred
• Relevant certifications (Databricks Data Engineer Associate/Professional) are desirable
• Experience working with AWS, Azure, or GCP cloud environments

‍

What We Offer

• Competitive salary and performance incentives
• Opportunities to work on high-impact Lakehouse programs across the FSI sector
• Exposure to modern data engineering, AI/ML integration, and enterprise-scale platforms
• A collaborative environment with global engineering and architecture teams
• Clear pathways for progression into senior engineering or architecture roles

‍