Quality Analyst for AI/ML
Trusys
- Location
- Hyderabad, Telangana, India
- Job type
- Full-time
Required skills
- Python
- end-to-end
- JSON
- regression
- REST APIs
About the role
Website:
trusys.ai
Job details:
Required Skills (AI/ML-1 Level)
Technical Skills
- Strong experience in Generative AI, LLMs, and Agentic AI systems
- Hands-on expertise with AI evaluation frameworks (RAGAS, DeepEval, TruLens, LangSmith, Promptfoo, etc.)
- Proficiency in Python and AI/ML development libraries
- Knowledge of Prompt Engineering, prompt testing, and optimization
- Ability to define and track evaluation metrics such as accuracy, relevance, groundedness, hallucination rate, latency, and user satisfaction
- Experience in creating automated evaluation pipelines and benchmarking frameworks
- Strong understanding of AI safety, guardrails, bias testing, and responsible AI practices
- Familiarity with REST APIs, JSON, vector databases, and knowledge retrieval systems
- Experience in A/B testing, human-in-the-loop evaluation, and red teaming
- Strong experience in Manual Testing of AI/GenAI applications, including functional, exploratory, UAT, regression, and end-to-end testing
- Expertise in validating Agent Reasoning, Tool Calling, Workflow Execution, and Response Quality
- Hands-on experience in Automation Testing using Python frameworks
Key Responsibilities
- Design, execute, and automate evaluation strategies for Agentic AI applications.
- Develop evaluation datasets, test cases, and benchmark suites.
- Measure and improve agent performance, reasoning quality, tool usage, and workflow effectiveness.
- Analyze model outputs and identify hallucinations, biases, safety risks, and failure patterns.
- Collaborate with AI Engineers, Product Teams, and Domain Experts to improve agent quality and reliability.
- Generate evaluation reports, dashboards, and actionable recommendations.
Click on Apply to know more.
This page is fully interactive when JavaScript is enabled. Please enable JavaScript to apply or browse related roles.
