Batch Engineer

As a Batch Engineer, you will be responsible for designing, developing, and maintaining batch processing pipelines for our data platform. You will work closely with our data engineering team to ensure the reliability, scalability, and efficiency of our batch processing workflows. Key responsibilities include: - Designing and implementing batch processing pipelines using tools like Apache Airflow, Luigi, or Prefect - Developing and optimizing ETL (Extract, Transform, Load) processes to ingest and transform data from various sources - Automating the deployment and monitoring of batch processing jobs - Collaborating with data scientists and analysts to understand their data requirements and translate them into efficient batch processing solutions - Troubleshooting and resolving issues related to batch processing failures or performance bottlenecks - Documenting your work and sharing knowledge with the team Requirements: - 2-5 years of experience in batch data processing using tools like Apache Airflow, Luigi, or Prefect - Proficient in Python and SQL, with a good understanding of data engineering principles - Experience with cloud-based data platforms (e.g., AWS, GCP, Azure) - Strong problem-solving and analytical skills - Ability to work collaboratively in a team-oriented environment - Familiarity with containerization and orchestration tools like Docker and Kubernetes is a plus

Welcome to the Jungle is a platform that connects job seekers with the most exciting, innovative and fast-moving companies. We provide a tailored job-matching experience, allowing users to find roles that align with their preferences and skills.