zorba ai
Website:
zorbaconsulting.in
Job details:
We are looking for a skilled and motivated Databricks Developer with strong expertise in SQL, Python, Apache Spark, and cloud-based data engineering solutions. The ideal candidate will be responsible for building scalable data pipelines, enabling analytics and reporting solutions, and supporting AI/ML workloads using the Databricks platform.
- Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark.
- Develop ETL/ELT workflows for processing large-scale structured and unstructured datasets.
- Write optimized SQL queries and Python-based data transformation scripts.
- Build and manage data ingestion frameworks from multiple data sources.
- Implement batch and real-time data processing solutions.
- Collaborate with analytics, reporting, and business teams to deliver data-driven insights.
- Support AI/ML workloads by preparing and transforming datasets for model training and inference.
- Optimize Spark jobs for performance, scalability, and cost efficiency.
- Develop reusable notebooks, workflows, and orchestration pipelines.
- Ensure data quality, governance, security, and compliance standards are maintained.
- Troubleshoot production issues and provide performance tuning recommendations.
- Work closely with cross-functional teams including Data Engineers, Data Scientists, Analysts, and Cloud Architects.
Required Skills
- Strong experience with SQL and Python programming.
- Hands-on experience with Apache Spark and PySpark.
- Expertise in Databricks platform and notebook development.
- Experience in building data pipelines and ETL workflows.
- Good understanding of Data Lake, Delta Lake, and Lakehouse architecture.
- Experience with analytics and reporting solutions.
- Knowledge of AI/ML data preparation and workload integration.
- Familiarity with workflow orchestration tools and CI/CD practices.
- Strong understanding of data modeling and performance optimization techniques.
- Experience working with cloud platforms such as Azure, AWS, or GCP.
Preferred Skills
- Experience with Azure Data Factory (ADF), Synapse, or similar services.
- Knowledge of streaming technologies like Kafka or Spark Streaming.
- Familiarity with Power BI, Tableau, or other reporting tools.
- Exposure to MLflow, model deployment, or MLOps concepts.
- Understanding of DevOps and Infrastructure as Code (IaC).
Qualifications
- Bachelor’s degree in Computer Science, Information Technology, Engineering, or related field.
- Relevant certifications in Databricks, Spark, or Cloud technologies are a plus.
Key Competencies
- Strong analytical and problem-solving skills.
- Excellent communication and collaboration abilities.
- Ability to work in agile and fast-paced environments.
- Strong attention to detail and data accuracy.
Nice to Have
- Experience with Generative AI or AI-driven analytics solutions.
- Exposure to modern data architecture and cloud-native technologies.
- Experience in enterprise-scale data migration projects.
Skills: python,apache spark,databricks,gen ai
Click on Apply to know more.