Deccan AI Experts
Website:
deccan360.in
Company:
https://www.linkedin.com/company/deccan-ai-experts
Seniority: Associate
Industries: IT Services and IT Consulting
Job details:
About Us
Deccan AI Experts is a pioneering AI company founded by IIT Bombay and IIM Ahmedabad alumni, with a strong founding team from IITs, NITs, and BITS. We specialize in high-quality human-curated data, AI-first operations, and advanced AI evaluation systems. Our global network of financial crime, compliance, risk, and AI experts helps train and evaluate next-generation AI models through expert AML knowledge, transaction monitoring expertise, and regulatory compliance experience.
Role Overview
We are looking for an AI/ML Engineer to work primarily on managing and evaluating existing AI/ML systems, analyzing AI agents, identifying model failures, and improving overall system performance.
The role requires a strong understanding of LLM/AI fundamentals, model evaluation, AI agent architecture, RAG, model improvement/fine-tuning, and production ML systems.
This is a hands-on role for someone who can not only evaluate AI systems but also understand why models or agents fail and identify practical ways to improve their performance.
Key Responsibilities:
- 2+ years of experience in AI/ML, Manage and monitor existing AI/ML and GenAI systems.
- Conduct structured LLM and model evaluations using defined benchmarks and evaluation criteria.
- Analyze AI/agent outputs and identify accuracy, reasoning, retrieval, tool-use, and behavioral failures.
- Analyze AI agent architecture and workflows to identify failure points and performance bottlenecks.
- Evaluate and improve RAG pipelines, including retrieval quality and response generation.
- Identify opportunities for model improvement, prompt optimization, and fine-tuning.
- Analyze evaluation results and translate findings into actionable improvements.
- Support testing and validation of new model/agent versions.
- Work with production ML systems and help identify reliability, latency, scalability, and performance issues.
- Collaborate with engineering and AI teams to implement and validate improvements.
- Document evaluation results, failure patterns, root causes, and recommended solutions.
Must-Have Skills:
- Strong understanding of LLM and AI fundamentals.
- Experience working with production ML/AI systems.
- Hands-on experience with model and LLM evaluation.
- Understanding of AI agent architecture, agentic workflows, and failure analysis.
- Experience with RAG (Retrieval-Augmented Generation) systems.
- Understanding of model improvement, fine-tuning, and prompt optimization.
- Strong Python programming and analytical skills.
- Ability to investigate model failures and perform root-cause analysis.
- Ability to work with evaluation datasets, benchmarks, metrics, and structured testing.
Who Would Be a Good Fit?
We’re looking for someone who has worked beyond simply building AI models and has hands-on experience with evaluating, debugging, analyzing, and improving AI/LLM systems.
Candidates with experience in LLM evaluation + AI agents + RAG + model improvement + production ML systems would be particularly relevant..
Why Join Us?
- Work on real-world LLM and agentic AI systems.
- Gain hands-on exposure to AI evaluation and model improvement.
- Work on challenging problems involving AI agent failures, RAG, and model performance.
- Opportunity to contribute to production-oriented AI projects.
Click on Apply to know more.
