Lead Data Engineer (PySpark)

Wymagania

• 9+ years of experience in data engineering with AI exposure

• Expertise in Azure Fabric and end-to-end Fabric experience

• Knowledge of OneLake (Delta / OpenLake)

• Advanced skills in Python, PySpark and SparkSQL

• Proficiency in Cosmos DB (NoSQL API) and other Cosmos DB variants

• Understanding of DF Gen2 and M-code

• Capability to implement CI/CD pipelines using Azure DevOps or equivalent tools

• Experience with generic Azure services and Power BI integration, semantic models and performance considerations

• Background in Agile or Scrum environments

• Strong ownership mindset and ability to lead by example

• Excellent communication skills for technical and non-technical audiences

• Good understanding of the financial domain

Zakres obowiązków

• Lead the design, development and optimization of scalable data engineering solutions using Azure Fabric and cloud-native technologies

• Own end-to-end data pipelines including ingestion, transformation, storage and analytics

• Architect and implement solutions leveraging OneLake (Delta / OpenLake) and Fabric experiences

• Develop and optimize PySpark, SparkSQL and Python-based data processing pipelines

• Work with Cosmos DB (NoSQL API) and other Cosmos DB variants to support high-performance data access patterns

• Implement and maintain CI/CD pipelines and promote DevOps best practices

• Collaborate with data scientists, AI engineers and product stakeholders to enable AI-driven analytics and insights

• Mentor and guide junior engineers, setting coding standards and best practices

• Ensure data quality, security, governance and performance across platforms

• Contribute to technical decision-making and solution architecture discussions

Seniority

• Lead

Mile widziane

• Experience with code generation, including non-AI and AI-assisted approaches

• Azure AI Foundry experience

• Data Science fundamentals and collaboration with DS teams

• Strong background in Big Data architectures and Spark ecosystems

• Familiarity with financial instruments and financial services data

• Hands-on experience with industry-standard LLMs (including GPT, Claude or similar)

• Exposure to AI-enabled data platforms and intelligent analytics use cases

Opis

We are seeking a Lead Data Engineer with deep expertise in Microsoft Azure Fabric–based data platforms and AI-enabled data engineering. This role blends hands-on technical leadership with architectural and team mentoring responsibilities, focusing on modern data engineering, big data processing, and AI-driven workflows in complex enterprise environments. Responsibilities Lead the design, development and optimization of scalable data engineering solutions using Azure Fabric and cloud-native technologies Own end-to-end data pipelines including ingestion, transformation, storage and analytics Architect and implement solutions leveraging OneLake (Delta / OpenLake) and Fabric experiences Develop and optimize PySpark, SparkSQL and Python-based data processing pipelines Work with Cosmos DB (NoSQL API) and other Cosmos DB variants to support high-performance data access patterns Implement and maintain CI/CD pipelines and promote DevOps best practices Collaborate with data scientists, AI engineers and product stakeholders to enable AI-driven analytics and insights Mentor and guide junior engineers, setting coding standards and best practices Ensure data quality, security, governance and performance across platforms Contribute to technical decision-making and solution architecture discussions Requirements 9+ years of experience in data engineering with AI exposure Expertise in Azure Fabric and end-to-end Fabric experience Knowledge of OneLake (Delta / OpenLake) Advanced skills in Python, PySpark and SparkSQL Proficiency in Cosmos DB (NoSQL API) and other Cosmos DB variants Understanding of DF Gen2 and M-code Capability to implement CI/CD pipelines using Azure DevOps or equivalent tools Experience with generic Azure services and Power BI integration, semantic models and performance considerations Background in Agile or Scrum environments Strong ownership mindset and ability to lead by example Excellent communication skills for technical and non-technical audiences Good understanding of the financial domain Nice to have Experience with code generation, including non-AI and AI-assisted approaches Azure AI Foundry experience Data Science fundamentals and collaboration with DS teams Strong background in Big Data architectures and Spark ecosystems Familiarity with financial instruments and financial services data Hands-on experience with industry-standard LLMs (including GPT, Claude or similar) Exposure to AI-enabled data platforms and intelligent analytics use cases

Słowa kluczowe / Umiejętności