Position Overview
My client is looking for a Lead Data Engineer to design and build scalable data pipelines that deliver trusted, analytics‑ready datasets for BI, AI, and operational use cases across a hybrid environment.
Key Responsibilities - Build pipelines across bronze, silver & gold layers (Databricks, Spark, dbt)
- Implement data quality checks, contracts & schema validation
- Apply governance (catalog, lineage, RBAC, metadata)
- Deliver curated datasets, features & embeddings for AI/BI
- Monitor pipeline health, performance & cost to meet SLAs
Tech Stack - Databricks
- Spark
- Delta Lake
- dbt
- Azure Data Factory
- Kafka/Event Hubs
- CI/CD (Azure DevOps/GitHub)
Governance & Ops - Enforce data contracts, lineage & cataloging
- Apply masking, tokenisation & access controls (PII/PHI)
- Build observable pipelines with alerts, dashboards & runbooks <...