AI & GenAI Test Engineer

Intellectt Inc

Website: intellectt.com
Job details:

Job Title: AI Tester / AI QA Engineer

Location: Hyderabad

Role Overview:

We are seeking an experienced AI Tester / AI QA Engineer to lead quality assurance efforts for AI-driven applications, including GenAI, LLM-based systems, chatbots, and cloud-native platforms. The ideal candidate will combine strong QA fundamentals, automation expertise, and hands-on AI/LLM testing experience to ensure reliability, fairness, scalability, and production readiness of intelligent systems. This role plays a key part in validating AI behavior, model outputs, data pipelines, APIs, and UI workflows, while driving AI-assisted test automation and governance.

Key Responsibilities:

AI & GenAI Testing:

Test LLM-based applications including chatbots, conversational agents, and AI-powered workflows.

Validate prompt-response accuracy, context retention, multi-turn conversations, and fallback handling.

Perform hallucination testing, response consistency checks, bias & fairness validation, and confidence threshold testing.

Design and execute AI model evaluation frameworks covering accuracy, precision/recall (where applicable), latency, safety, and reliability.

Validate RAG (Retrieval-Augmented Generation) pipelines, including data retrieval accuracy and grounding quality.

AI-Driven Test Automation:

Build and maintain AI-powered test generation frameworks using tools such as AWS Bedrock, OpenAI, Azure OpenAI, LangChain, and Hugging Face.

Integrate AI-assisted coding tools (GitHub Copilot, Microsoft Copilot) to accelerate test development and improve coverage.

Automate AI test execution with Python, PyTest, Selenium, Playwright, Cypress, and CI/CD pipelines.

Implement human-in-the-loop validation for AI-generated test cases.

Automation, API & UI Testing:

Develop and maintain robust test automation frameworks using Selenium, Playwright, Cypress, Appium, and WebDriverIO.

Perform API testing using Postman, Rest Assured, Swagger/OpenAPI, and schema validation.

Validate modern SPA and cloud-native UIs with role-based workflows and responsive behaviors.

Integrate automated tests into CI/CD pipelines (Jenkins, GitHub Actions, GitLab CI/CD, Azure DevOps).

Data, Cloud & Pipeline Validation:

Test event-driven data pipelines on AWS and Azure (S3, Lambda, Glue, EMR, Kinesis, DynamoDB, Athena).

Validate Big Data and streaming systems (Hadoop, Spark, Kafka) for data completeness, consistency, and accuracy.

Perform ETL and data quality testing using SQL, reconciliation checks, and profiling tools.

Validate microservices deployed on AWS EKS and serverless architectures.

Observability, Performance & Governance:

Monitor AI and application behavior using CloudWatch, Grafana, ELK/OpenSearch, Splunk, AppDynamics.

Execute performance testing for AI services, APIs, and streaming workloads using JMeter.

Build dashboards tracking AI KPIs such as response accuracy, hallucination rate, latency, and user satisfaction.

Contribute to Responsible AI governance, documentation, and QA best practices.

Leadership & Collaboration:

Lead QA strategy, planning, and execution across Agile/Scrum and SAFe teams.

Mentor QA engineers and support adoption of AI testing practices.

Collaborate with Product, Engineering, Data Science, and DevOps teams to align quality with business goals.

Required Skills & Qualifications:

Technical Skills:

8+ years of experience in Software QA / Automation, with hands-on AI/GenAI testing experience.

Strong programming skills in Python, Java, or JavaScript.

Experience testing LLMs, chatbots, conversational AI, and AI-driven applications.

Hands-on with AWS Bedrock, OpenAI, Azure OpenAI, Hugging Face, LangChain, RAG.

Expertise in test automation frameworks (Selenium, Playwright, Cypress, PyTest).

Strong API testing and SQL/data validation skills.

Experience with CI/CD pipelines, Docker, and cloud platforms (AWS/Azure).

Nice to Have:

Big Data & streaming validation (Hadoop, Spark, Kafka).

Performance engineering with JMeter.

Observability and monitoring experience.

Exposure to AI governance, fairness testing, and compliance.

Click on Apply to know more.