Website:
mycareernet.in
Job details:
Company:IT Services Organization
Key Skills: AWS, Pyspark, Databricks, SQL, Python, Data Engineer, AWS cloud
Roles and Responsibilities:
- Design, develop, and maintain scalable data pipelines using Pyspark and Databricks.
- Utilize AWS services such as Glue, Lambda, and S3 for data processing and storage.
- Write efficient SQL queries for data manipulation, aggregation, and retrieval.
- Collaborate with cross-functional teams to understand business data requirements and deliver solutions.
- Ensure data quality, consistency, and integrity throughout the data lifecycle.
- Optimize data workflows for performance, scalability, and cost-efficiency.
- Troubleshoot and resolve issues in data pipelines and ETL processes.
- Document pipeline architecture, data flows, and processes for future maintenance and audits.
Skills Required:
- Strong expertise in Pyspark and Databricks for data pipeline development
- Hands-on experience with AWS services (S3, Lambda, Glue, Redshift)
- Proficiency in Python and SQL for data manipulation and transformation
- Solid understanding of data engineering concepts and best practices
- Experience designing scalable and efficient ETL/ELT workflows
- Ability to ensure data governance, quality, and security in cloud environments
- Collaboration skills for working with cross-functional teams and stakeholders
- Analytical and problem-solving skills to troubleshoot and optimize data pipelines
- Awareness of cloud architecture patterns and cost optimization strategies
Education: Bachelor's degree in Computer Science, Information Technology, or a related field.
Click on Apply to know more.