Lead Data Engineer
Humana, Denver, CO, United States
Overview
Lead Data Engineer
to join the ITSM Applied AI and Analytics team. This role leads the development and evolution of an enterprise-facing data store built on Azure Databricks, driving architectural decisions, mentoring team members, and ensuring scalable, high-quality data solutions across the organization.
Responsibilities
Lead the design, development, and optimization of ETL/ELT pipelines using Databricks, Apache Spark, Unity Catalog, and Delta Lake.
Architect data workflows following Medallion Architecture (Bronze, Silver, Gold layers).
Guide the integration of external APIs and systems, including ServiceNow and Kafka for real-time data streaming.
Establish and enforce best practices in CI/CD, GitHub workflows, and code quality.
Manage compute appropriately to ensure cost control.
Collaborate with stakeholders to define data strategy, governance, and platform enhancements.
Provide technical leadership and mentorship to data engineers and analysts.
Conduct code reviews and ensure adherence to engineering standards.
Communicate effectively with technical and non-technical stakeholders.
Required Qualifications
5+ years of experience in data engineering, with at least 2 years in a lead or senior role.
Deep expertise in Databricks, Apache Spark, Unity Catalog, and Delta Lake.
Strong understanding of Medallion Architecture and scalable data design.
Strong proficiency in Python for data engineering tasks (APIs, Spark data frames, OAuth, etc.).
Familiarity with Databricks Materialized views.
Experience with Kafka for real-time data ingestion and processing.
Proficiency in GitHub, CI/CD pipelines, and DevOps practices.
Experience building and integrating REST APIs.
Excellent communication, leadership, and documentation skills.
Preferred Qualifications
Databricks certifications (e.g., Data Engineer Professional).
Familiarity with ServiceNow data structures and integration.
Experience with enterprise analytics platforms like Power BI or Power Apps.
Background in enterprise environments with data governance and compliance requirements.
Background in enterprise data architecture and governance.
Additional Information
Candidates may be asked to present a GitHub-hosted project as part of the interview process.
#J-18808-Ljbffr
Lead Data Engineer
to join the ITSM Applied AI and Analytics team. This role leads the development and evolution of an enterprise-facing data store built on Azure Databricks, driving architectural decisions, mentoring team members, and ensuring scalable, high-quality data solutions across the organization.
Responsibilities
Lead the design, development, and optimization of ETL/ELT pipelines using Databricks, Apache Spark, Unity Catalog, and Delta Lake.
Architect data workflows following Medallion Architecture (Bronze, Silver, Gold layers).
Guide the integration of external APIs and systems, including ServiceNow and Kafka for real-time data streaming.
Establish and enforce best practices in CI/CD, GitHub workflows, and code quality.
Manage compute appropriately to ensure cost control.
Collaborate with stakeholders to define data strategy, governance, and platform enhancements.
Provide technical leadership and mentorship to data engineers and analysts.
Conduct code reviews and ensure adherence to engineering standards.
Communicate effectively with technical and non-technical stakeholders.
Required Qualifications
5+ years of experience in data engineering, with at least 2 years in a lead or senior role.
Deep expertise in Databricks, Apache Spark, Unity Catalog, and Delta Lake.
Strong understanding of Medallion Architecture and scalable data design.
Strong proficiency in Python for data engineering tasks (APIs, Spark data frames, OAuth, etc.).
Familiarity with Databricks Materialized views.
Experience with Kafka for real-time data ingestion and processing.
Proficiency in GitHub, CI/CD pipelines, and DevOps practices.
Experience building and integrating REST APIs.
Excellent communication, leadership, and documentation skills.
Preferred Qualifications
Databricks certifications (e.g., Data Engineer Professional).
Familiarity with ServiceNow data structures and integration.
Experience with enterprise analytics platforms like Power BI or Power Apps.
Background in enterprise environments with data governance and compliance requirements.
Background in enterprise data architecture and governance.
Additional Information
Candidates may be asked to present a GitHub-hosted project as part of the interview process.
#J-18808-Ljbffr