zorba ai
Website:
zorbaconsulting.in
Job details:
We are looking for a highly skilled
Azure Databricks & Snowflake Data Engineer with strong experience in
Spark, PySpark, Python/Scala, Azure Cloud, SQL, ETL/ELT pipelines, and Snowflake development. The ideal candidate should have hands-on experience in building and migrating enterprise data pipelines and should be comfortable working across the complete Software Development Life Cycle.
Exposure to
Snowflake Cortex/CoCo, Databricks Genie, and AI-assisted coding tools such as GitHub Copilot is highly desirable. Candidates with strong core engineering skills and a willingness to learn emerging AI capabilities will also be considered.
Key Responsibilities
- Design, develop, optimize, and maintain scalable data engineering solutions using Azure Databricks, Apache Spark, PySpark, Python, and/or Scala.
- Develop and optimize complex ETL/ELT data pipelines for large-scale data processing.
- Perform migration of existing ETL/ELT pipelines and data workloads to Azure Databricks and Snowflake.
- Develop robust data transformation, ingestion, validation, and processing frameworks.
- Work extensively with Snowflake for data loading, transformation, optimization, and integration.
- Collaborate with data architects, developers, analysts, and business teams to understand requirements and deliver scalable solutions.
- Implement CI/CD pipelines and automated deployment processes using Git, GitHub, GitHub Workflows, and Azure DevOps.
- Follow software engineering best practices including version control, code reviews, unit testing, debugging, and documentation.
- Work with workflow/orchestration technologies such as Azure Data Factory (ADF), Azure Synapse, and Airflow.
- Participate in architecture discussions and contribute to improving data platform performance, reliability, and scalability.
- Utilize Terraform and Infrastructure-as-Code concepts where applicable.
- Explore and leverage AI-assisted development tools such as GitHub Copilot and Databricks Genie to improve engineering productivity.
- Gain exposure to and contribute to solutions using Snowflake Cortex/CoCo and other AI capabilities.
- Troubleshoot production issues and provide root-cause analysis and permanent fixes.
- Participate in all phases of the Software Development Life Cycle (SDLC).
Mandatory Technical Skills
- 5+ years of hands-on experience with Azure Databricks.
- 5+ years of experience with Apache Spark / PySpark.
- 5+ years of programming experience using Python and/or Scala.
- 5+ years of experience working on the Azure Cloud Platform.
- 3+ years of hands-on Snowflake development experience.
- Strong knowledge of SQL and database concepts with 5+ years of experience.
- Strong experience in ETL/ELT pipeline development and migration.
- Hands-on experience with CI/CD, Git, and GitHub Workflows.
- Strong software development and coding skills.
- 5+ years of complete SDLC experience.
Good-to-Have Skills
- Knowledge or hands-on experience with Snowflake Cortex / CoCo.
- Exposure to Databricks Genie.
- Experience using GitHub Copilot or other AI-assisted coding tools.
- Knowledge of Azure DevOps.
- Knowledge of Terraform / Infrastructure as Code.
- Experience with Azure Data Factory (ADF).
- Exposure to Azure Synapse Analytics.
- Exposure to Apache Airflow.
- Experience with Delta Lake and modern data lakehouse architecture.
- Understanding of data engineering best practices, performance tuning, and cloud-native architectures.
Core Competencies
- Strong programming and problem-solving skills.
- Excellent understanding of data structures, SQL, databases, and data processing concepts.
- Ability to develop clean, maintainable, reusable, and production-ready code.
- Strong analytical and debugging skills.
- Ability to work independently as well as collaboratively in a distributed team.
- Good communication and stakeholder-management skills.
- Strong willingness to learn emerging technologies, particularly Generative AI and AI-assisted development.
Education
Bachelor’s or Master’s degree in
Computer Science, Information Technology, Engineering, or a related discipline.
Evaluation / Skill WeightageEvaluation AreaWeightageAzure Databricks / Spark / PySpark25%Python / Scala & Coding Skills15%Snowflake Development15%Azure Cloud & Data Engineering10%ETL/ELT & Pipeline Migration10%SQL / Database Expertise10%CI/CD, Git, GitHub & DevOps5%ADF / Synapse / Airflow / Terraform5%Snowflake Cortex/CoCo, Databricks Genie & AI-assisted Coding5%
Total100%Key Search Keywords
Azure Databricks, Databricks, Apache Spark, PySpark, Python, Scala, Azure, Snowflake, Snowflake Cortex, CoCo, Databricks Genie, GitHub Copilot, ETL, ELT, Data Migration, SQL, Azure Data Factory, ADF, Azure Synapse, Airflow, Azure DevOps, Terraform, CI/CD, Git, GitHub Actions, Data Engineering, SDLC.
Ideal Candidate Profile
A candidate with
5+ years of experience in Azure Databricks, Spark/PySpark, Python/Scala, Azure Cloud, SQL, and end-to-end data engineering, along with
3+ years of Snowflake development experience. Strong ETL/ELT migration and CI/CD experience is essential. Exposure to
Snowflake Cortex/CoCo, Databricks Genie, and GitHub Copilot will be an added advantage.
Skills: snowflake,azure,pyspark,sql
Click on Apply to know more.