Disprz
Website:
disprz.ai
Job details:
SDET – AI Innovation Team
Job Overview
As an SDET in the AI Innovation Team, you will be responsible for ensuring the reliability, accuracy, scalability, and quality of AI-powered products and intelligent workflows. You will work closely with AI engineers, backend developers, and product teams to build automated quality frameworks for next-generation AI systems.
This role goes beyond traditional UI automation and focuses on validating LLM-powered applications, AI workflows, APIs, intelligent agents, and data-driven systems. You will design evaluation strategies, automate regression detection, and establish quality guardrails for rapidly evolving AI products.
Responsibilities
AI Quality Engineering
- Design and implement testing strategies for AI/LLM-powered applications and intelligent workflows.
- Build automated evaluation frameworks for validating AI responses, workflows, prompts, and model behaviour.
- Create scalable regression suites to detect quality degradation across AI releases and prompt/model changes.
- Validate AI outputs for accuracy, consistency, hallucination risks, safety, and business relevance.
Automation & Framework Development
- Develop and maintain robust automation frameworks for API, integration, backend, and workflow testing.
- Build reusable testing utilities, evaluation pipelines, and quality tooling for AI systems.
- Integrate automated tests into CI/CD pipelines to ensure fast and reliable feedback cycles.
AI Workflow Validation
- Test multi-step AI workflows, agent-based systems, orchestration layers, and decision-making pipelines.
- Validate prompt engineering changes, retrieval pipelines, embeddings, and response orchestration logic.
- Design synthetic and real-world test datasets to improve AI evaluation coverage.
Reliability & Observability
- Monitor AI system behaviour using logs, metrics, traces, and evaluation dashboards.
- Identify flaky behaviour, non-deterministic failures, latency issues, and workflow inconsistencies.
- Collaborate with engineering teams to improve system reliability, testability, and operational excellence.
Collaboration & Innovation
- Partner with AI engineers, developers, and product stakeholders to define quality standards for AI products.
- Drive continuous improvements in AI testing methodologies, tooling, and automation practices.
- Research and adopt emerging approaches in AI testing, evaluation, and quality engineering.
Qualifications
- 3+ years of experience in SDET, QA Automation, or Software Engineering roles.
- Strong programming skills in TypeScript, Java, or Python with a solid understanding of OOP and software design principles.
- Hands-on experience building automation frameworks for APIs, backend systems, or distributed applications.
- Experience with automation tools such as Playwright, Selenium, Cypress, Postman, RestAssured, or equivalent frameworks.
- Understanding of CI/CD pipelines, test orchestration, and quality engineering practices.
- Familiarity with AI/LLM concepts such as prompts, embeddings, RAG pipelines, AI agents, or model evaluation is highly preferred.
- Strong analytical, debugging, and problem-solving skills.
- Ability to work in fast-paced innovation environments with evolving product requirements.
- Excellent communication and collaboration skills.
Good-to-Have Skills
- Experience testing AI/LLM-powered applications or conversational systems.
- Familiarity with OpenAI, Anthropic, Gemini, or open-source LLM ecosystems.
- Knowledge of observability platforms, telemetry analysis, and performance monitoring.
- Experience with cloud platforms and containerised environments.
Click on Apply to know more.