Senior Architect, Quality Engineering

Icertis

Website: icertis.com
Job details:

Job Description

We are looking for an experienced QA AI Architect to design and lead the architecture of AI-driven Quality Engineering systems for next-generation AI-powered products.

This role will focus on building scalable frameworks for testing and validating AI systems built on Large Language Models (LLMs), embeddings, vector databases, and modern AI pipelines. The architect will work closely with product, engineering, and AI teams to design intelligent QA solutions including AI-assisted test generation, intelligent automation, document understanding, and AI-driven validation systems.

The ideal candidate combines deep QA architecture experience with strong AI/ML system knowledge, and is passionate about redefining quality engineering for AI products.

 

Responsibilities

Key Responsibilities

AI Architecture & LLM Systems

• Architect and build AI-powered quality engineering platforms using LLMs and SLMs
• Evaluate and integrate open-source models such as Llama and Mistral
• Design prompt engineering workflows and Retrieval-Augmented Generation (RAG) pipelines
• Architect systems for LLM evaluation, guardrails, and validation frameworks
• Optimize architectures for token efficiency, cost, and performance

Embeddings & Vector Search

• Design and implement embedding pipelines for semantic retrieval
• Implement vectorization strategies for documents, test artifacts, and product knowledge bases
• Architect vector storage using technologies such as
• Pinecone
• Weaviate
• Milvus
• FAISS
• Optimize chunking and indexing strategies for large knowledge repositories

Model Optimization & Fine-Tuning

• Implement model fine-tuning pipelines using LoRA / PEFT techniques
• Evaluate model performance, reliability, and hallucination mitigation
• Manage training datasets and model evaluation metrics

Backend & System Architecture

• Design microservice-based AI system architectures
• Build scalable backend services and APIs for model inference and evaluation
• Develop data pipelines supporting AI workflows and large-scale inference systems

MLOps & AI Lifecycle

• Build MLOps pipelines for model deployment, monitoring, and retraining
• Implement CI/CD pipelines for AI systems
• Enable model observability, performance monitoring, and version control

OCR & Document Intelligence

• Design systems for OCR-based document processing
• Integrate OCR pipelines with embedding and AI workflows
• Build solutions for document understanding and QA automation

Required Skills

AI & LLM Systems

• Strong understanding of LLM and SLM architectures
• Experience working with open-source LLMs
• Prompt engineering and RAG architectures
• Model evaluation frameworks and hallucination control
• Token optimization strategies

Embeddings & Vector Databases

• Experience working with embedding models and semantic search
• Hands-on experience with vector databases such as
• FAISS
• Pinecone
• Weaviate
• Milvus
• Infuse AI with 
• Automated Test Failure analysis
• Crawl Product/Test code to figure out whether issue is probable Product/Test/Env issue

 

Programming & AI Frameworks

• Strong Python programming skills
• Experience with AI frameworks such as
• LangChain
• LlamaIndex
• Backend development using FastAPI / Flask / Node.js

Backend & Cloud Architecture

• Microservices architecture and REST APIs
• Containerization using Docker
• Experience working on AWS / Azure / GCP

MLOps

• Model deployment pipelines
• CI/CD for machine learning systems
• Monitoring and observability of AI models

OCR & Document Processing

• Experience with OCR frameworks such as
• Tesseract OCR
• PaddleOCR

System Foundations

• Strong understanding of system design, operating systems, networking, and distributed systems

Qualifications

Nice to Have

• Experience building AI agents or autonomous systems
• should add Architect testing frameworks for agentic AI workflows and multi-agent orchestration
• Knowledge of LLM evaluation frameworks
• Understanding of AI safety and guardrails
• Experience architecting AI-powered SaaS products

Click on Apply to know more.