Adeptmind
Website:
adeptmind.ai
Job details:
QA Automation engineer
LLM Exposure Required | Remote | Full-Time
About the Role
We are looking for a passionate and detail-oriented QA Automation Engineer to join our growing QA & Engineering team. In this role, you will design, build, and maintain automated test frameworks for web, API, and AI-powered applications — with a special focus on testing Large Language Model (LLM)-based features. You will collaborate closely with quality analysts, product managers, and data analysts to ensure the quality, reliability, and performance of our software products.
Key Responsibilities
Automation & Testing
- Design, develop, and maintain automated test scripts for validating the output of an LLM-based product
- Integrate automated tests into CI/CD pipelines (GitHub Actions, Jenkins, etc.)
- Perform regression, smoke, sanity, and end-to-end testing across environments
- Identify, log, and track defects using tools like Jira
- Review test coverage and continuously improve test suite efficiency
LLM / AI Testing
- Write automated and manual test cases for LLM-powered features (chatbots, copilots, summarization, etc.)
- Validate LLM outputs for accuracy, hallucination, tone, format, and edge-case behavior
- Design evaluation frameworks to measure prompt quality and model response consistency
- Collaborate with AI/ML teams to define acceptance criteria for model outputs
- Use AI coding assistants ( ChatGPT, Claude code) to accelerate test script development
- Stay updated with emerging LLM testing tools, techniques, and industry benchmarks
Collaboration & Reporting
- Work in Agile/Scrum sprints and participate in daily standups and sprint reviews
- Collaborate with cross-functional teams to understand requirements and define test strategies
- Produce clear test reports, coverage metrics, and quality dashboards
- Document test plans, test cases, and automation strategies
Required Skills & Qualifications
- 2–3 years of hands-on experience in test automation
- Proficiency in at least one language: Python, JavaScript, Java, or TypeScript
- Experience with one or more frameworks: Selenium, Playwright, Cypress, or Robot Framework
- Working knowledge of REST API testing (Postman, RestAssured, or similar)
- Familiarity with Git version control and branching strategies
- Exposure to CI/CD pipelines and DevOps practices
- Basic understanding of LLMs and how they work (GPT, Claude, Gemini, etc.)
- Ability to evaluate and test AI-generated outputs across key quality dimensions, including accuracy, bias, precision, recall, F1 score, and hallucinations
- Strong analytical, problem-solving, and communication skills
Good to Have (Preferred)
- Hands-on experience testing LLM-based products or AI features
- Familiarity with LLM APIs: OpenAI, Anthropic Claude, or Google Gemini
- Experience with prompt engineering and prompt evaluation techniques
- Understanding of ML concepts: tokenization, embeddings, RAG pipelines
- Hands-on experience with Google Sheets formulae
Perks & Benefits
- Work on cutting-edge AI & LLM products
- A remote-first culture with a lean, high-performing team of AI experts.
- Collaborative, inclusive, and supportive team culture
- Health & wellness benefits
Click on Apply to know more.