Website:
amrapali.org.in
Job details:
Senior Data Engineer
Location - Chennai
Experience - 8 -12 years
Seeking an experienced Senior Data Engineer to drive our data engineering initiatives. The ideal candidate will have extensive experience with Azure cloud technologies and a proven track record of designing, implementing, and maintaining scalable data solutions. You will play a crucial role in building robust data pipelines, optimizing data workflows, and enabling data-driven decision making across the organization.
Key Responsibilities
- Design, develop, and maintain complex data pipelines using Azure Data Factory (ADF) and other Azure data services
- Build and optimize data processing workflows using Azure Databricks with Spark and PySpark
- Implement and manage Lake flow Declarative Pipelines for efficient data processing and transformation
- Develop and optimize SQL queries and stored procedures for data extraction, transformation, and loading (ETL) processes
- Create and maintain data models, schemas, and documentation to ensure data quality and consistency
- Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver solutions
- Monitor and troubleshoot data pipeline performance, implementing improvements and optimizations as needed
- Ensure data security, privacy, and compliance with industry standards and regulations
- Mentor junior team members and provide technical guidance on data engineering best practices
- Stay current with emerging technologies and industry trends in data engineering and cloud computing
- Participate in code reviews, testing, and deployment processes to maintain high-quality deliverables
- Design and implement data governance frameworks and data quality monitoring solutions
Technical Skills
- Cloud Platforms: Microsoft Azure (primary focus)
- Data Processing: Azure Databricks, Apache Spark, PySpark
- Data Orchestration: Azure Data Factory (ADF), Lake flow Declarative Pipelines
- Databases: SQL Server, Azure SQL Database, Azure Synapse Analytics, Azure Data Lake
- Programming Languages: Python, SQL, PySpark
- Development Tools: Git, Azure DevOps, Visual Studio Code
- Data Formats: JSON, Parquet, Avro, CSV, XML
- Monitoring & Logging: Azure Monitor, Application Insights
Mandatory Skills:
- Overall 8+yrs of software knowledge
- 7+ years of relevant experience in Data Engineering
- Strong hands-on experience with Azure Data Engineering services
- 4+ yrs of Databricks and Lakeflow Declarative Pipelines
- Experience in building scalable data pipelines and ETL/ELT workflows
- Good understanding of data processing, transformation, and optimization techniques
- Experience working with large-scale structured and unstructured data
- Strong problem-solving and performance optimization skills in data platforms
- Advanced SQL skills with experience in query optimization and database design
- Proficient in PySpark and Python for data processing and transformation tasks
- Experience with Azure cloud services including Azure Data Lake Storage, Azure SQL Database, and Azure Synapse Analytics
If you meet the requirements outlined in the job description and are interested in this opportunity, please email your resume to tanvi@amrapalis.co.in, mentioning your current CTC, expected CTC, and Notice Period.
Click on Apply to know more.