Bridgenext
Website:
bridgenext.com
Job details:
Job ID: Sen-Eng-Pun-1277
Location: Pune,Remote
Position: Senior Databricks Engineer
Experience: 3–6 Years
Location: Pune / Remote
Employment Type: Full-time
Role Overview
We are seeking a highly skilled
Senior Databricks Engineer with 3–6 years of experience in modern data engineering, distributed data processing, and cloud-based analytics. The ideal candidate will have strong hands-on expertise with
Databricks,
PySpark, and
Delta Lake, along with experience on at least one major cloud platform (
Azure, AWS, or GCP). Knowledge of Azure data ecosystem is a strong plus.
This role involves designing scalable pipelines, optimizing Databricks workloads, and collaborating closely with cross-functional teams to deliver enterprise-grade data solutions.
Key Responsibilities
Databricks Engineering
- Develop, optimize, and maintain ETL/ELT pipelines using Databricks (PySpark, Spark SQL, Delta Lake).
- Design and deploy distributed data processing workflows for batch and streaming use cases.
- Implement best practices for performance tuning, cost optimization, and cluster configuration.
- Work with Delta Lake for data versioning, incremental pipelines, and reliability.
Cloud Data Platform Integration
- Build solutions on Azure, AWS, or GCP using cloud-native services integrated with Databricks.
- Ingest, transform, and process large datasets using cloud storage and compute services.
- Work with APIs, connectors, and cloud-native data orchestration tools.
Azure Data Services (Good to Have)
- Exposure to Azure Data Lake (ADLS), Azure Data Factory, Azure Synapse Analytics, Azure SQL, Event Hub, Azure Functions, etc.
- Support end-to-end pipelines covering ingestion, transformation, storage, governance, and monitoring.
Data Engineering & Development
- Write high-quality, production-grade Python, PySpark, and SQL code.
- Develop reusable data frameworks, utilities, and automation scripts.
- Participate in code reviews and enforce engineering best practices.
Collaboration & Delivery
- Work closely with Data Architects, Analysts, and Scientists to implement scalable data solutions.
- Contribute to solution design documents, data models, and architecture diagrams.
- Ensure solutions adhere to security, governance, and compliance standards (e.g., Unity Catalog).
Required Skills & Qualifications
- 3–6 years of experience in Data Engineering or Big Data platforms.
- Strong hands-on experience with Databricks, PySpark, Spark SQL, and Delta Lake.
- Experience with at least one cloud provider (Azure, AWS, or GCP).
- Strong SQL programming and data modeling concepts.
- Understanding of distributed computing, performance tuning, and cost-efficient design.
- Experience with Git, CI/CD, and basic DevOps practices.
- Familiarity with workflow orchestration (Databricks Workflows, Airflow, ADF, etc.).
Preferred / Good-to-Have
- Experience with Azure data services (ADLS, ADF, Synapse, Key Vault, Event Hub).
- Understanding of Unity Catalog, RBAC, and data governance practices.
- Experience with MLflow, serverless compute, or Delta Live Tables.
- Knowledge of containerization and serverless technologies (Docker, Kubernetes, Functions/Lambda).
- Relevant certifications:
- Databricks Data Engineer Associate or Professional
Click on Apply to know more.