Position Overview
Job Description
Senior Data Infrastructure Engineer
Job Description
Manage and optimize Databricks infrastructure, including cluster lifecycle management, job scheduling, Unity Catalog administration, user permissions, and cost monitoring. Manage Airflow clusters deployed on Kubernetes via Helm charts, including upgrades, configuration management, and integration with ArgoCD and Vault. Write and maintain Terraform modules to provision and manage cloud infrastructure in a scalable, reliable, and secure manner. Develop and maintain data pipelines and workflows using Python, PySpark, and dbt to support data processing and analytics use cases. Use GitHub Copilot with custom skills and agents to improve team productivity, automate repetitive development tasks, and accelerate code reviews and documentation. Communicate with stakeholders across the organization to understand their data needs ...