Ecolab
Website:
ecolab.com
Job details:
Lead Databricks Data Engineer
Ecolab is looking for a Data Engineer to be part of a dynamic team that's at the forefront of technological innovation. We're leveraging cutting-edge AI to create novel solutions that optimize operations for our clients, particularly within the restaurant industry. Our work is transforming how restaurants operate, making them more efficient and sustainable.
As a key player in our new division, you'll have the unique opportunity to help shape its culture and direction. Your contributions will directly impact the success of our innovative projects and help define the future of our product offerings. Additionally, you will experience the best of both worlds with this team at Ecolab: the agility and creativity of a startup paired with the stability and resources of a global leader. Our collaborative environment fosters innovation while providing the support and security you need to thrive.
Responsibilities
- Design, develop, and maintain scalable and robust data pipelines on Databricks (Spark SQL, PySpark, Delta Lake).
- Collaborate with data scientists and analysts to understand data requirements and deliver solutions.
- Optimize and troubleshoot existing data pipelines for performance and reliability.
- Ensure data quality and integrity across various data sources.
- Implement data security and compliance best practices.
- Monitor data pipeline performance, implement data quality checks, and conduct necessary maintenance and updates.
- Document data pipeline processes and technical specifications.
- Implement robust pipeline orchestration using tools like Databricks Workflows, dbt, or similar.
- Generate and maintain data quality reports and dashboards.
- Implement Infrastructure as Code (IaC) principles for managing Databricks infrastructure.
Minimum Qualifications
- Bachelor’s degree and 8 years work experience; or no degree and 12 years combined education and equivalent work experience.
- 3 years of experience (work or educational) with a data engineering focus.
- Proven experience in Databricks (Delta Lake, Workflows, Asset Bundles).
- Proven experience in distributed data processing technologies (Spark SQL, PySpark).
- Strong knowledge in designing and developing ETL pipelines.
- Experience with data quality monitoring and reporting.
- Experience working in a collaborative environment with data scientists and software engineers.
Preferred Qualifications
- Master’s degree (MS) in Computer Science or related engineering field.
- Proficiency in Databricks (Delta Lake, Workflows, Asset Bundles).
- Proficiency in distributed data processing technologies (Spark SQL, PySpark).
- Experience with pipeline orchestration tools (Databricks Workflows, dbt, etc.).
- Experience with data visualization tools (e.g., Tableau, Power BI).
- Working experience with machine learning platforms and tools.
- Experience with real-time data streaming technologies (e.g., Kafka, Kinesis).
Nuestro compromiso con una cultura de inclusión y pertenencia
Ecolab está comprometido con el trato justo e igualitario de todas las personas colaboradoras y postulantes, y con la promoción de los principios de igualdad de oportunidades en el empleo. Reclutaremos, contrataremos, promoveremos, transferiremos y brindaremos oportunidades de desarrollo con base en las calificaciones individuales y el desempeño laboral, en todos los aspectos relacionados con el empleo, la compensación, los beneficios, las condiciones laborales y las oportunidades de crecimiento. Ecolab no discriminará a ninguna persona colaboradora ni postulante por motivos de raza, religión, color, credo, nacionalidad, estado de ciudadanía, sexo, orientación sexual, identidad y expresión de género, información genética, estado civil, edad o discapacidad.
Click on Apply to know more.