About the Role
We’re seeking a Data Engineer with 3–4 years of experience to join our growing tech team. The
ideal candidate will have hands-on experience in building and managing scalable data systems
on AWS, with strong expertise in data pipelines, real-time streaming, and modern data
frameworks in building and managing scalable data systems on AWS using Spark and modern
data frameworks.
Key Responsibilities
● Design, build, and maintain end-to-end data pipelines for ingestion, transformation, and
delivery of high-volume data.
● Develop Spark-based ETL/ELT workflows for both batch and real-time streaming data.
● Core Skills: Spark, AWS (S3, Glue, Redshift, Kinesis), Kafka, Airflow, Python, SQL
● Integrate data from multiple internal and external systems using Kafka, Kinesis, or other
streaming frameworks.
● Build and manage data models, warehouses, and lakehouses using AWS services such
as S3, Glue, Redshift, Athena, etc.
● Implement data quality checks, validation rules, and monitoring to ensure reliability and
consistency.
● Collaborate with data analysts and scientists to provide clean, structured datasets
optimized for analytics and ML.
● Work with orchestration tools (Airflow, MWAA, Step Functions, etc.) for automated
workflow scheduling.
● Continuously optimize data pipelines for cost, scalability, and performance.
Required Skills
● Strong programming skills in Python for data manipulation and automation.
● Hands-on expertise in Apache Spark (PySpark or Spark SQL) for large-scale data
processing.
● Deep understanding of AWS data ecosystem — S3, Glue, Lambda, Redshift, Athena,
EMR, Kinesis, IAM.
● Experience with real-time streaming platforms such as Kafka, Kinesis, or Flink.
● Strong command of SQL and data modeling (star schema, dimensional modeling,
partitioning).
● Proficiency with data orchestration and workflow management tools (Airflow, Step
Functions, etc.).
● Familiarity with Git, CI/CD, and modern development best practices.
● Experience working in Linux/Unix environments and handling large datasets efficiently.
Good to Have
● Exposure to data lakehouse technologies (Delta Lake, Iceberg, Hudi).
● Understanding of data governance, cataloging, and lineage tools (Glue Data Catalog,
Amundsen, DataHub).
● Familiarity with containerization and deployment (Docker, ECS, EKS).
● Basic understanding of AI/ML
Qualifications
● Bachelor’s degree in Computer Science, IT, Engineering, or related technical field.
● 3-4 years of experience in data engineering, big data, or analytics infrastructure.