ADF Data Engineer

Experience : 8–10 years overall, with 3+ years of Azure experience.

Required Skills

Strong hands-on experience with Azure Data Factory (ADF) — pipelines, data flows, triggers, integration runtimes, parameterization, error handling, and retries.

Strong experience with Azure Databricks — Notebooks, Jobs/Workflows, PySpark/Scala, Delta Lake, cluster configuration, and optimization.

Advanced Python or Scala with strong coding standards, modularization, and unit testing (pytest).

Expert SQL — CTEs, window functions, query tuning, and large datasets.

Experience with ADLS Gen2, Delta Lake, Parquet, partitioning, Z-Ordering, OPTIMIZE, and VACUUM.

Experience with Git, branching strategies, and CI/CD for ADF/Databricks.

Exposure to Infrastructure as Code (IaC) is a plus.

Roles and Responsibilities

Coding & Development: Design, develop, and optimize ADF/Databricks data pipelines and solutions.

UI Testing: Perform functional/UI testing, data validation, and troubleshooting.

Stakeholder Management: Collaborate with stakeholders, understand requirements, and communicate project status/issues effectively.