SMARTWORK IT SERVICES LLC (SWITS)
Website:
smartworkitservices.com
Job details:
Senior Data Engineer
Experience: 7 12 Years
Location: Hyderabad
Employment Type: Full-Time
About The Role
We are seeking a highly skilled
Senior Data Engineer to join our Data Science and Engineering team. The ideal candidate will have strong expertise in building scalable data platforms, developing robust ETL/ELT pipelines, and enabling machine learning and analytics workloads through reliable, production-grade data infrastructure.
This role requires hands-on experience with modern data engineering technologies including
DBT, Databricks, PySpark, Airflow, SQL, and Lakehouse architectures.
Key Responsibilities
- Design, develop, and maintain scalable DBT models for analytics, reporting, and machine learning use cases.
- Build and optimize Databricks and PySpark pipelines for processing large-scale structured and unstructured data.
- Develop and manage workflow orchestration using Airflow (Prefect experience is a plus).
- Create and maintain Delta Lake/Iceberg tables and Lakehouse architectures.
- Collaborate with Data Scientists and ML Engineers to productionize feature engineering and machine learning data pipelines.
- Improve performance, scalability, reliability, and cost efficiency of Spark and DBT workloads.
- Implement data quality checks, testing frameworks, monitoring, alerting, and documentation standards.
- Participate in code reviews and contribute to data engineering best practices and architecture discussions.
- Work with cloud-native data platforms and distributed data processing systems.
Required Skills
- Strong hands-on experience in SQL, DBT, Databricks, PySpark, and Airflow.
- Experience building and maintaining large-scale ETL/ELT pipelines.
- Strong understanding of data modeling, data warehousing, and Lakehouse architectures.
- Experience with Delta Lake and/or Apache Iceberg.
- Knowledge of distributed data processing frameworks and performance optimization techniques.
- Experience with Infrastructure as Code (IaC) tools such as Terraform, AWS CDK, or Pulumi.
- Familiarity with cloud platforms such as AWS, Azure, or GCP.
- Strong problem-solving, communication, and stakeholder management skills.
Preferred Skills
- Experience with Prefect orchestration.
- Exposure to machine learning feature stores and MLOps workflows.
- Experience with Kafka, Spark Streaming, or real-time data processing.
- Knowledge of CI/CD, Git, and DevOps practices.
Keywords for Search
DBT, Databricks, PySpark, Airflow, SQL, Delta Lake, Apache Iceberg, Terraform, Data Engineering, ETL, ELT, Lakehouse, Data Modeling, Spark, AWS, Azure, Feature Engineering, Data Warehouse
Click on Apply to know more.