Position Overview
- 12+ years of experience in data engineering using Databricks or Apache Spark-based platforms.
- Proven track record of building and optimizing ETL/ELT pipelines for batch and streaming data ingestion.
- Hands-on experience with Azure services such as Azure Data Factory, Azure Data Lake Storage, Azure Synapse Analytics, or Azure SQL Data Warehouse.
- Proficiency in programming languages such as Python, Scala, or SQL for data processing and transformation.
- Expertise in Spark (PySpark, Spark SQL, or Scala) and Databricks notebooks for large-scale data processing.
- Familiarity with Delta Lake, Delta Live Tables, and medallion architecture for data lakehouse implementations.
- Build and query deltalake storage solutions
- Process streaming data with Azure Databricks structured streaming
- Design Azure Databricks security and data protection solutions
- Flatten nested structures and explode arrays with spa...