Website:
mycareernet.in
Job details:
Company:IT Services Organization
Key Skills: Data Engineer, Azure, Python, Pyspark, MySQL, Azure Databricks
Roles and Responsibilities:
- Design and implement scalable, metadata-driven frameworks for data ingestion, quality, and transformation across both batch and streaming datasets.
- Develop and optimize end-to-end data pipelines to process structured and unstructured data, enabling the creation of analytical data products.
- Build robust exception handling, logging, and monitoring mechanisms for better observability and operational support.
- Take ownership of complex modules and lead the development of critical data workflows and components.
- Provide guidance to data engineers and peers on best practices, conduct code reviews, and enforces standards.
- Collaborate with cross-functional teams--including business consultants, data architects, scientists, and application developers--to deliver impactful analytics solutions.
Skills Required:
- 5-10 years of total IT experience, including at least 3 years in big data engineering on Microsoft Azure
- Strong SQL expertise with experience in Azure SQL, Synapse, or similar cloud databases
- Proven experience in building high-performance data pipelines using Azure Databricks (PySpark, Spark SQL)
- Hands-on experience with Azure Data Factory for pipeline orchestration
- Experience with batch and streaming data processing
- Delivered at least one end-to-end Data Lakehouse solution (Medallion Architecture)
- Strong programming and debugging skills in Python, PySpark, and SQL
- Knowledge of data governance, security, and lifecycle management
Good to Have
- Exposure to LLM / Generative AI applications
- Knowledge of NoSQL databases
- Experience supporting BI and Data Science teams
- Azure / Databricks certifications
Education: Bachelor's Degree in related field
Click on Apply to know more.