zorba ai
Website:
zorbaconsulting.in
Job details:
Primary Skills: Databricks, Azure Data Factory, Azure Function App, PySpark, Python, SQL, GitHub Job Summary
We are seeking an experienced Data Engineer Lead with strong expertise in Azure cloud data engineering technologies, scalable ETL pipeline development, and modern big data processing frameworks. The ideal candidate will be responsible for designing, developing, optimizing, and leading enterprise-grade data engineering solutions using Databricks, Azure Data Factory, Azure Function Apps, PySpark, Python, and SQL.
The candidate should possess hands-on experience building scalable batch and real-time data pipelines, implementing cloud-native architectures, managing large datasets, and collaborating with cross-functional teams to deliver reliable and high-performance data solutions.
Key Responsibilities Data Pipeline Development
- Design, develop, and maintain scalable ETL/ELT pipelines using:
- Azure Databricks
- Azure Data Factory (ADF)
- PySpark
- Python
- Build efficient data ingestion frameworks for structured and unstructured data sources.
Azure Cloud Data Engineering
- Develop cloud-native data solutions on Microsoft Azure.
- Work extensively with:
- Azure Databricks
- Azure Function Apps
- Azure Data Lake
- Azure Blob Storage
- Azure SQL
- Azure Synapse Analytics
- Implement secure, scalable, and high-performance data architectures.
Big Data Processing
- Develop and optimize distributed data processing workflows using PySpark.
- Handle large-scale datasets with focus on:
- Performance optimization
- Cost efficiency
- Scalability
- Reliability
Data Transformation & Integration
- Develop complex SQL transformations and data models.
- Integrate data from multiple enterprise systems, APIs, and third-party sources.
- Ensure data quality, consistency, and governance standards.
Automation & CI/CD
- Manage code repositories and deployment workflows using GitHub.
- Implement CI/CD best practices for data engineering deployments.
- Automate operational workflows and monitoring processes.
Leadership & Collaboration
- Lead and mentor junior data engineers.
- Collaborate with:
- Architects
- Product teams
- Business analysts
- DevOps teams
- Stakeholders
- Participate in design discussions and technical decision-making.
Monitoring & Optimization
- Monitor pipeline performance and troubleshoot failures.
- Optimize Spark jobs, SQL queries, and data workflows.
- Implement logging, alerting, and operational monitoring mechanisms.
Strong Hands-on Experience With
Required Technical Skills Must-Have Skills Azure Data Engineering
- Azure Databricks
- Azure Data Factory (ADF)
- Azure Function Apps
- Azure Data Lake
- Azure Blob Storage
Programming & Processing
Strong Expertise In
Data Engineering
Experience With
- ETL/ELT pipelines
- Batch processing
- Data transformation
- Data integration
- Data warehousing
Version Control & CI/CD
Hands-on Experience With
- GitHub
- Git workflows
- CI/CD pipelines
Cloud & Big Data
Experience Working With
- Distributed data processing
- Large-scale datasets
- Cloud-native architectures
Skills: azure,pyspark,python,sql
Click on Apply to know more.