US Technology International Private Limited
Seniority: Mid-Senior level
Job details:
Senior Data Engineer - Scala and Spark
We are looking for a Senior Level, full-time permanent Senior Data Engineer to join our team in India on a hybrid work model. In this role, you will focus on system design, best practices, and ownership for large-scale data processing platforms and performance optimization using Scala, Apache Spark, AWS, and Apache Iceberg.
Responsibilities
- Lead the design, development, and ownership of large-scale data processing platforms.
- Build and optimize data pipelines using Scala, Apache Spark, AWS EMR, S3, and Glue Data Catalog.
- Implement and maintain Apache Iceberg tables, including updates, merges, compaction, and performance tuning.
- Design and optimize data models in DynamoDB for high performance.
- Develop serverless workflows using AWS Lambda and Step Functions.
- Write advanced Spark SQL queries and perform data modeling.
- Apply system design best practices and mentor team members.
Requirements
- Strong hands-on experience in Scala and Apache Spark.
- Proven expertise in AWS EMR, S3, and Glue Data Catalog.
- Hands-on experience with Apache Iceberg and its performance tuning, updates, merges, and compaction.
- Experience in DynamoDB data modeling and performance optimization.
- Proficiency in AWS Lambda and Step Functions.
- Strong background in Spark SQL and data modeling.
- Experience in large-scale data processing platforms and performance optimization.
Click on Apply to know more.