Atyeti Inc
Website:
atyeti.com
Company:
https://www.linkedin.com/company/atyeti-inc
Seniority: Mid-Senior level
Industries: Information Services
Job details:
Key Responsibilities
- Lead the modernization and migration of legacy data applications, ETL pipelines, and data platforms to Databricks.
- Assess existing applications, data pipelines, databases, and workloads and define appropriate migration strategies.
- Design scalable Databricks Lakehouse architectures using Delta Lake, Unity Catalog, Databricks SQL, and Apache Spark.
- Develop and optimize PySpark / Spark SQL workloads for large-scale data processing.
- Modernize legacy ETL/data-processing workloads and convert them into Databricks-native pipelines.
- Design Bronze, Silver, and Gold / Medallion architecture for enterprise data platforms.
- Work on batch and near-real-time data ingestion and transformation pipelines.
- Implement data governance, security, access control, lineage, and data quality using Unity Catalog.
- Optimize Databricks jobs for performance, scalability, reliability, and cost.
- Support migration from traditional data warehouses, Hadoop, on-premises platforms, or legacy ETL tools to Databricks.
- Define migration frameworks, reusable patterns, coding standards, and best practices.
- Collaborate with application, cloud, DevOps, data engineering, and business teams.
- Perform technical POCs and evaluate modernization approaches.
- Troubleshoot complex Spark/Databricks performance and production issues.
- Provide technical mentoring and guidance to data engineering teams.
Mandatory Technical Skills
- 10+ years of experience in Data Engineering / Data Architecture.
- Strong hands-on experience with Databricks.
- Excellent knowledge of Apache Spark, PySpark, and Spark SQL.
- Strong experience with Delta Lake and Lakehouse architecture.
- Hands-on experience with Unity Catalog.
- Strong SQL and database concepts.
- Experience in ETL/ELT pipeline development and modernization.
- Experience migrating legacy/on-premises data workloads to cloud platforms.
- Strong understanding of data modeling and data architecture.
- Experience with Databricks Jobs / Workflows / Pipelines.
- Experience with Git, CI/CD and DevOps practices
Click on Apply to know more.
