Tredence Inc.
Website:
tredence.com
Job details:
Role description
Python + Gen AI + Databricks Engineer
Role Summary
We are seeking a highly skilled Python + Gen AI + Databricks Engineer to design, build, and deploy scalable data and AI solutions. The role involves developing data pipelines, advanced analytics, and Generative AI applications on cloud-based platforms using Databricks, Python, and modern LLM frameworks.
Key Responsibilities
- Develop and maintain data pipelines and analytics workflows using Databricks (Spark, PySpark)
- Build, deploy, and optimize Python-based data engineering and machine learning solutions
- Design and implement Generative AI use cases such as chatbots, summarization, content generation, and retrieval-augmented generation (RAG)
- Integrate LLMs (OpenAI, Azure OpenAI, Hugging Face, etc.) into enterprise applications
- Perform data preprocessing, feature engineering, and model validation
- Optimize data workflows for performance, scalability, and cost efficiency
- Collaborate with business stakeholders, data scientists, and product teams to translate requirements into technical solutions
- Ensure data quality, security, and compliance standards are met
- Document solutions and provide production support where required
Required Skills & Qualifications
Technical Skills
- Strong proficiency in Python (data structures, OOP, APIs, performance tuning)
- Hands-on experience with Databricks:
- Apache Spark / PySpark
- Delta Lake
- Databricks workflows and jobs
- Experience with Generative AI / LLMs:
- Prompt engineering
- Fine-tuning or adapter-based approaches (LoRA, adapters)
- RAG architectures
- Experience with ML/AI frameworks: LangChain, LlamaIndex, TensorFlow, PyTorch (any)
- Strong SQL skills and experience with large datasets
- Experience with cloud platforms (Azure preferred; AWS/GCP acceptable)
Data & MLOps
- Model deployment and monitoring
- CI/CD for data and ML pipelines
- Experiment tracking and version control
- Familiarity with MLflow is a plus
Preferred Qualifications
- Experience with Azure Databricks and Azure OpenAI
- Knowledge of vector databases (FAISS, Pinecone, Weaviate)
- Exposure to real-time or streaming data (Kafka, Event Hubs)
- Understanding of data governance and security concepts
- Prior experience in retail, BFSI, healthcare, or large-scale enterprise analytics
Education
- Bachelor’s or Master’s degree in Computer Science, Data Science, Engineering, or related field
Experience
- 4–8 years of experience in data engineering, analytics, or AI development
- At least 1–2 years of hands-on experience with Generative AI or LLM-based solutions
Click on Apply to know more.