VaporVM
Website:
vaporvm.com
Job details:
We are seeking an experienced Databricks Engineer to design, develop, and optimize scalable data engineering solutions using the Databricks Lakehouse Platform. The ideal candidate will have strong expertise in Databricks, Apache Spark, PySpark, SQL, Delta Lake, cloud platforms, and enterprise data pipelines.
Key Responsibilities
- Design and develop scalable ETL/ELT pipelines using Databricks, Spark, and PySpark.
- Develop and maintain Lakehouse architecture using Delta Lake and Bronze/Silver/Gold layers.
- Build batch and real-time data processing and streaming solutions.
- Manage Databricks Jobs, Workflows, and Delta Live Tables (DLT).
- Implement data governance and access controls using Unity Catalog.
- Optimize Spark jobs, SQL queries, pipelines, and cloud resources for performance and cost.
- Integrate Databricks with enterprise data sources and cloud platforms.
- Implement data quality, validation, monitoring, and error-handling processes.
- Support Git, CI/CD, and DevOps practices for automated deployments.
- Troubleshoot production issues and ensure reliable data platform operations.
- Collaborate with Data Architects, Developers, Analysts, and business stakeholders.
Required Skills
- 5–10 years of experience in Data Engineering with strong hands-on Databricks experience.
- Advanced knowledge of Databricks, Apache Spark, PySpark, and SQL.
- Strong expertise in Delta Lake and Lakehouse architecture.
- Hands-on experience with Unity Catalog, Workflows, and DLT.
- Experience with batch and streaming data processing.
- Strong understanding of ETL/ELT, data modeling, and data warehousing.
- Experience with Azure, AWS, or GCP.
- Good understanding of Git, CI/CD, and DevOps.
- Strong analytical, troubleshooting, and problem-solving skills.
Certifications (It is Mandatory to have any one of these certifications)
- Databricks Certified Data Engineer – Associate
- Databricks Certified Data Engineer – Professional
- Azure/AWS/GCP Data Engineering certification
Education
Bachelor’s degree in Computer Science, Information Technology, Data Engineering, or a related field.
Click on Apply to know more.
