About the role
We're looking for an experienced QA engineer who can protect the quality of both our conventional software and our LLM-powered features, where pass/fail scripts alone are not enough.
You will own testing end to end: reviewing requirements, running manual and automated tests, tracking defects to closure, and building evaluation suites that catch hallucinations, regressions and safety issues before customers do. You will set measurable release criteria for RAG pipelines, multi-step agents and the APIs and apps around them.
The ideal candidate has strong analytical and problem-solving skills, a solid grounding in testing methodologies, and enjoys working with engineering, product and customer teams to keep improving how we test.
Key responsibilities
Core QA and test planning
Review product specifications, designs and requirements to flag risks and testability gaps before testing starts.
Write test plans, test cases and test scripts with engineers and product managers.
Run manual and automated functional, regression and exploratory tests across environments for new and existing products.
Identify, document and track defects to closure in JIRA, with clear reproduction steps.
Reproduce customer-reported and production issues in test environments and help troubleshoot them.
GenAI and LLM evaluation
Design test strategies and test plans for LLM features, RAG pipelines and agentic workflows.
Build and maintain golden datasets, evaluation rubrics and adversarial test sets for each use case.
Measure output quality with automated metrics (faithfulness, answer relevance, context precision/recall, groundedness) and LLM-as-judge evaluations, calibrated against human review.
Detect and track hallucinations, factual errors, bias, toxicity and policy violations.
Agent and RAG testing
Test multi-step agents end to end: tool and function calling, planning, memory, error recovery and human-in-the-loop handoffs.
Validate retrieval quality, chunking and embedding changes, and the effect of prompt or model changes on downstream answers.
Run red-teaming for prompt injection, jailbreaks, data leakage and PII exposure.
Automation and CI/CD
Build automated evaluation and regression suites in Python and integrate them into CI/CD so every prompt, model or code change is scored before release.
Automate API, integration, web UI and mobile app tests for the applications that wrap our GenAI services.
Maintain and improve test scripts, tools and frameworks to make each testing cycle faster.
Run performance and load tests covering scalability, latency, throughput, token usage and cost per request.
Quality ownership
Define release gates and quality KPIs, take part in release planning, and confirm every component meets the bar before release.
Report quality metrics and test results to engineering, product and management.
Check that products meet security standards and industry regulations, including data privacy and PII handling.
Monitor production quality through tracing and observability tools, and turn user-reported failures into new test cases.
Review release notes and product documentation for accuracy and completeness.
Give developers feedback on design changes that improve testability.
Support UAT and help customers validate the product during POCs and enterprise deployments.
Improve QA processes, tools and best practices, and keep up with new testing and LLM evaluation tools.
Required qualifications
Bachelor's degree in Computer Science, Information Technology, Computer Engineering or a related field, or equivalent experience.
3–6 years in software QA or SDET roles, covering both manual and automated testing, including at least 1 year testing LLM, ML or GenAI applications.
Strong Python for test automation, with pytest or a similar framework; working knowledge of Java or JavaScript is a plus.
Hands-on experience with at least one LLM evaluation framework, such as Ragas, DeepEval, promptfoo, TruLens or LangSmith evaluations.
Working knowledge of LLM application stacks: prompting, RAG, vector databases, embeddings, and agent frameworks such as LangChain, LangGraph, LlamaIndex or CrewAI.
Solid grasp of testing methodologies, with UI and API automation experience: Selenium or Playwright, JUnit or TestNG, and Postman, REST Assured or Python requests.
Experience with CI/CD pipelines (GitHub Actions, Azure DevOps or Jenkins), Git, JIRA or similar bug tracking, and Agile ways of working (Scrum or Kanban).
Understanding of non-deterministic system testing: statistical pass rates, sampling, and when to use human review.
Clear written communication for test reports, defect write-ups and quality dashboards, plus strong attention to detail and the ability to work independently or in a team.
Nice to have
Experience with cloud AI platforms and cloud-based test environments: Azure OpenAI, AWS Bedrock, Google Vertex AI or Databricks Mosaic AI.
LLM observability tools such as Langfuse, Arize Phoenix or MLflow tracing.
Security testing practices and AI safety tools such as Garak or PyRIT, and familiarity with the OWASP Top 10 for LLM Applications.
Performance and load testing with JMeter, LoadRunner, Locust or k6.
Mobile app testing on iOS and Android.
SQL and database knowledge for data verification.
Testing ML models for accuracy, drift, fairness or explainability.
ISTQB, CSTE or similar QA certification.
Experience in fast-paced startups or with enterprise customer deployments, handling several priorities at once.
This page is fully interactive when JavaScript is enabled. Please enable JavaScript to apply or browse related roles.
