TekPillar®
Website:
tekpillar.com
Job details:
Job Role: Data Engineer - Azure
Experience: 6+ Years
Location: Bangalore
We are looking for an experienced Data Engineer with strong expertise in Azure Databricks, PySpark, Azure Data Factory, Azure Data Lake, Azure DevOps, and API integrations.
The ideal candidate will be responsible for designing and building scalable data platforms, developing ETL/ELT pipelines, implementing Lakehouse architectures, and enabling secure, high-performance data solutions on Microsoft Azure.
Key Responsibilities
- Design, develop, and optimize ETL/ELT pipelines using Azure Databricks and Azure Data Factory
- Build scalable batch and streaming data processing solutions using Python and PySpark
- Implement and maintain Delta Lake and Lakehouse architectures
- Design and manage data models, including Dimensional Modeling and SCD Type 1 & Type 2 implementations
- Develop and maintain data storage solutions using Azure Data Lake Storage Gen2, Delta Tables, and Parquet formats
- Integrate external systems using REST APIs, Graph APIs, and secure authentication mechanisms
- Build and maintain CI/CD pipelines using Azure DevOps
- Implement data governance, security, and access control using Azure security services
- Optimize SQL queries, data processing performance, and workload efficiency
- Collaborate with cross-functional teams to deliver scalable and reliable data solutions
- Maintain technical documentation, coding standards, and best practices
- Support Databricks CI/CD implementations using Databricks Asset Bundles (DABs)
Required Qualifications & Skills
- 6–8 years of experience in Data Engineering
- Strong hands-on experience with Azure Databricks
- Expertise in Python and PySpark
- Experience with Azure Data Factory (ADF)
- Strong knowledge of Azure Data Lake Storage Gen2
- Hands-on experience with Azure DevOps, Git, and CI/CD pipelines
- Experience integrating REST APIs and secure authentication mechanisms
- Strong SQL skills including query optimization, CTEs, and window functions
- Knowledge of distributed systems, partitioning, and big data processing
- Experience with Delta Lake concepts including ACID transactions, Time Travel, Vacuum, and Z-Ordering
- Understanding of Data Governance and Security practices
Click on Apply to know more.