Data and Analytics Lead (Consulting)

About the Role:

We are looking for a skilled and innovative Data & Analytics Lead to join our dynamic Data team. In this role, you will focus on leveraging Databricks to design, develop, and optimize data pipelines. You will create and maintain data models while collaborating closely with solution architects and analysts to build scalable data solutions that support advanced analytics and machine learning initiatives.

Key Responsibilities:

• Design, develop, and maintain end-to-end data pipelines using Databricks and Apache Spark.
  
• Build and manage ETL processes for transforming and processing large datasets in real-time and batch modes.
  
• Define and govern data modelling and design standards, tools, and best practices, developing and maintaining data models and schemas to support analytics and machine learning applications.
  
• Collaborate with cross-functional teams, including data scientists, business analysts, and IT, to understand data requirements and deliver high-quality solutions.
  
• Optimize performance and ensure scalability of Databricks-based solutions, addressing data quality, processing time, and resource usage.
  
• Implement data governance best practices, ensuring data integrity, privacy, and security across all data assets.
  
• Monitor and troubleshoot pipeline performance, ensuring reliability and quick resolution of data issues.
  
• Automate data workflows and implement CI/CD pipelines for smooth deployment of data applications.
  
• Stay up-to-date with emerging technologies and best practices related to Databricks, Spark, and cloud data solutions.
  
  
  
  

Requirements

Required Qualifications:

• Bachelor’s or Master’s degree in Computer Science, Engineering, Data Science, or a related field.
  
• Minimum 8 years of experience as a Data Engineer with Databricks and Apache Spark.
  
• Strong programming skills in Python, Scala, or SQL.
  
• Hands-on experience working with cloud platforms such as AWS, Azure, or Google Cloud.
  
• Solid understanding of data warehousing, ETL, and data modelling principles.
  
• Experience with Databricks Delta Lake and building scalable data pipelines.
  
• Familiarity with Apache Kafka, Airflow, or other workflow automation tools.
  
• Knowledge of version control tools like Git and experience with continuous integration and deployment pipelines.
  
• Excellent analytical, data profiling, and problem-solving abilities
  
• Excellent oral and written communication abilities
  

Preferred Qualifications:

• Familiarity with machine learning pipelines and working knowledge of MLflow.
  
• Experience with large-scale data processing frameworks and distributed systems.
  
• Understanding of data visualization and reporting tools such as Tableau or Power BI.
  
• Excellent problem-solving and troubleshooting skills.
  
• Strong communication skills and the ability to work effectively in a team environment.
  
  
  
  

Benefits

Everyone who joins our team is treated as a trusted member of the team, not a number. Ultimately your success is our success, so we invest in our People heavily.

Here are the top reasons to join us :

- Developing you is a key focus - we help you craft your career
- Pioneers in Parental Leave benefits - we provide equality in our parental leave for all genders and parental types
- Doona Days - additional two days off for your mental health
- Fun is an everyday experience - we challenge you in a positive way so you enjoy your growth journey
- Competitive Compensation & Pay for Performance
- Opportunities to be more for yourself and others