Job Description

<p><b>Senior Data Engineer (Azure & Databricks)</b></p><p><b>Experience Level: 5–6 Years</b></p><p><b>Role Overview</b></p><p><b>We are looking for a Senior Data Engineer to architect and implement our next-generation</b></p><p><b>data platform. You will lead the transition toward a unified data experience, leveraging</b></p><p><b>Azure Databricks for high-scale processing and Microsoft Fabric for integrated data</b></p><p><b>engineering, with a heavy emphasis on automated data quality.</b></p><p><b>Key Responsibilities</b></p><p><b>● Lakehouse Implementation: Build and optimize Medallion architectures using</b></p><p><b>Databricks and Fabric Data Engineering (Lakehouses and Notebooks).</b></p><p><b>● Governance & Cataloging: Implement data discovery, lineage, and classification using</b></p><p><b>Microsoft Purview.</b></p><p><b>● Hybrid Orchestration: Design complex workflows using Azure Data Factory and</b></p><p><b>Synapse Pipelines (Good to have).</b></p><p><b>● Data Transformation: Develop highly efficient ETL processes using PySpark and Spark</b></p><p><b>SQL.</b></p><p><b>● High-Performance ETL: Architect and maintain robust ETL/ELT processes using Delta</b></p><p><b>Live Tables (DLT) and Spark SQL, ensuring data is ready for downstream ML and BI</b></p><p><b>workloads.</b></p><p><b>● Performance Engineering: Deep-dive into Spark UI and query plans to eliminate</b></p><p><b>bottlenecks, optimize shuffles, and reduce cloud compute costs.</b></p><p><b>Technical Requirements</b></p><p><b>● SQL: Expert-level complex joins, window functions, and query optimization.</b></p><p><b>● Python/PySpark: Advanced capability in writing modular, unit-tested PySpark code</b></p><p><b>and tuning Spark UI/Shuffle partitions.</b></p><p><b>● Deep experience in Databricks (Unity Catalog, Delta Live Tables, Databricks</b></p><p><b>Workflows/Lakeflow).</b></p><p><b>● Deep expertise with Azure cloud with Azure Databricks, Fabric, Synapse & ADF.</b></p><p><b>● Experience with Azure Purview for enterprise-grade data governance.</b></p><p><b>● Big Data: Proficiency with technologies like Spark, Delta Lake, and Structured</b></p><p><b>Streaming.</b></p><p><b>● Strong understanding of Data Warehouse/Lake/Lakehouse architecture and</b></p><p><b>concepts.</b></p><p><b><br /></b></p><p><b>● Programming: Strong Python/PySpark skills for data engineering tasks.</b></p><p><b><br /></b></p><p><b>Good to have :</b></p><p><b>● Certifications,</b></p><p><b>○ DP-600/DP-700 - Microsoft Certified: Fabric Analytics Engineer</b></p><p><b>Associate / Microsoft Certified: Fabric Data Engineer Associate</b></p><p><b>○ DP-203 - Data Engineering on Microsoft Azure</b></p><p><b>○ Databricks Certified Data Engineer</b></p><p><b>● Exposure of Databricks AI agent and AgentBricks</b></p>