Website:
Job details:
AI QA Automation Engineer - Immediate Joiners only (0-15 Days)
Bangalore / Pune / Mumbai
Exp Range - 5-8 years
Key Responsibilities
Eval Framework & Quality Delivery
- Design and build --- eval/golden-set regression framework for non-deterministic GenAI agent output, from scratch.
- Partner with each AI Engineer to scope and build a golden-set test suite for every agent they ship, landing alongside the agent rather than after.
- Define and track what "done" looks like for agent output quality, including golden-set pass rate and the rate of defects/false positives that escape to real users.
- Extend and maintain general automated test coverage across the agents module - backend (xUnit) and frontend E2E (Playwirght) - where gaps exist alongside the eval work.
- Flag where eval suite depth or scope needs to flex to keep pace with the delivery cadence, and work with the lead engineer to re-balance rather than let a quality backlog build up silently.
- Beyond the agents module, contribute to and help scale test automation practice across the wider --- platform - frontend E2E (Playwright) and backend (xUnit) - as the team's senior dedicated test engineer.
Required Experience & Skills
- 3-5+ years' experience in a software engineering or test automation role.
- Experience designing test strategy from scratch, not just executing an existing suite.
- Experience with C#/.NET testing (xUnit or similar) and/or JavaScript E2E testing frameworks (Playwright or similar).
- Comfortable reading and reasoning about LLM/agent output well enough to write meaningful test assertions against it.
- Ability to work independently across two engineers' agents in parallel, without becoming a bottleneck on their delivery cadence.
- Experience working in a sprint-based development team.
Desirable / Nice-to-Have Skills
Experience with some of the following is beneficial but not required:
- Hands-on experience evaluating GenAI/LLM applications (golden sets, LLM-as-judge, eval frameworks such as Promptfoo, DeepEval, Ragas, or similar).
- Experience with Google Vertex AI, Gemini, or a comparable LLM platform.
- Exposure to MCPs, or tool-use integrations for LLMs.
- Familiarity with ad platform APIs (Google Ads, Microsoft Advertising, Meta, TikTok, DV360).
- Experience with prompt management tooling (we use PostHog).
- API testing experience (e.g. via Playwright's API testing capabilities).
Tech Environment
The --- product is built using:
- Backend: C# (.NET 10), Microsoft Agent Framework, Google Vertex AI (Gemini)
- Frontend: ReactJS
- Data: MySQL, Firestore, Google Cloud Storage
- Infrastructure: Google Cloud Platform (Cloud Run, Cloud Functions, Cloud Scheduler, Docker)
- Prompt Management: PostHog
- Testing: xUnit (C#), Playwright (frontend E2E) - general repo conventions. The eval/golden-set framework for agent output does not exist yet and is what this role builds and owns
Click on Apply to know more.
