Lead Apache Airflow Engineer

Beschreibung

We are seeking a Lead Apache Airflow Engineer to spearhead the development and enhancement of our workflow orchestration platform. In this role, you will drive the evolution of Airflow components, integrate new versions with Cloud Composer and lead a talented team of engineers in building robust data pipelines. Please note that working from the office in Krakow 3 days per week is required. Responsibilities Develop and maintain Airflow Operators, Hooks and Sensors Integrate new Airflow versions with Cloud Composer Fix bugs in Airflow and implement improvements Implement tests for Airflow, including Unit Testing and Solution Testing Create internal documentation for work artifacts Lead and mentor a team of Airflow engineers Collaborate with Open Source communities to contribute improvements Requirements 5+ years of experience in Python 1+ years of relevant leadership experience Programming experience with Cloud APIs (REST/RPC) Basic knowledge of DevOps/SysAdmin Basic understanding of Linux and Kubernetes, including how to build container images Familiarity with Cloud platforms Expertise in Airflow or similar experience in building workflows (e.g. based on Luigi, Oozie, Argo, Azkaban, Apache HiFi) Experience working with Open Source communities Communication skills in English, spoken and written (B2+) Nice to have Understanding of Cloud Architecture, including scalability and IaaS principles

Anforderungen

• 5+ years of experience in Python

• 1+ years of relevant leadership experience

• Programming experience with Cloud APIs (REST/RPC)

• Basic knowledge of DevOps/SysAdmin

• Basic understanding of Linux and Kubernetes, including how to build container images

• Familiarity with Cloud platforms

• Expertise in Airflow or similar experience in building workflows (e.g. based on Luigi, Oozie, Argo, Azkaban, Apache HiFi)

• Experience working with Open Source communities

• Communication skills in English, spoken and written (B2+)

Zakres obowiązków

• Develop and maintain Airflow Operators, Hooks and Sensors

• Integrate new Airflow versions with Cloud Composer

• Fix bugs in Airflow and implement improvements

• Implement tests for Airflow, including Unit Testing and Solution Testing

• Create internal documentation for work artifacts

• Lead and mentor a team of Airflow engineers

• Collaborate with Open Source communities to contribute improvements

Seniority

• Lead

Mile widziane

• Understanding of Cloud Architecture, including scalability and IaaS principles