nomiso
Website:
nomiso.io
Job details:
GenAI Expert
Role Overview
We're seeking an AI Expert with 8+ years of combined Gen AI and QA/Testing with Test automation expertise. This is a specialized IC role for someone who bridges AI systems engineering with rigorous quality assurance. You'll architect production-grade AI systems (RAG pipelines, fine-tuning, prompt optimization, inference scaling) while simultaneously designing comprehensive testing strategies, test cases, test plans, test automation, and execution frameworks that ensure AI systems behave reliably at scale. You own both the AI implementation and the quality validation end-to-end.
This role is for someone with deep testing domain knowledge who has evolved into Gen AI, or vice versa, equally strong in both. You're not just implementing off-the-shelf models; you're designing retrieval systems, optimizing inference pipelines, evaluating model trade-offs, and building systems that reliably serve millions of transactions. You understand that GenAI is as much about solving the "why" (business impact) as the "how" (technical implementation)
What We’re Looking For (Who You Are)
- AI Systems Architect: 4+ years of experience, and building and shipping LLM systems in production. You understand model selection, RAG architecture, fine-tuning strategies, inference optimization, and cost management. You make pragmatic trade-offs between accuracy, latency, and cost.
- Testing Domain Expert: Rigorous QA/testing background atleast 5+ years. You're fluent in test case design, test plan creation, test automation frameworks, test execution strategies, and test reporting. You understand different testing types (unit, integration, end-to-end, performance, regression, smoke, sanity) and when to apply each. You can architect testing strategies from scratch.
- Independent Operator: You own AI features end-to-end—from problem definition through evaluation, deployment, and production monitoring. You unblock yourself and make good decisions with incomplete information.
- Production-Focused Builder: You've shipped LLM systems at scale. Comfortable with tokenization, context windows, vector database optimization, handling hallucinations and failures.
- Team Amplifier: You mentor engineers on AI best practices, participate in code reviews, and help scale GenAI adoption.
- Continuous Experimenter: You A/B test approaches, iterate based on production data, and stay current with rapid LLM evolution.
Technical Stack & Qualifications
Category
Requirement
GenAI Core
- LLM APIs (OpenAI, Anthropic, Hugging Face), model trade-offs and selection
- RAG pipeline architecture, vector databases (Pinecone, Weaviate, FAISS)
- Prompt engineering, fine-tuning (PEFT/LoRA), inference optimization
- Cost and latency management, hallucination detection and mitigation
- Agentic patterns, multi-step reasoning, reliability under uncertainty
Testing Skills
- Test case design (positive, negative, edge cases, boundary conditions)
- Test planning and coverage analysis
- Test types: Unit, integration, end-to-end, performance, regression, smoke, sanity testing
- Test automation frameworks (pytest, Jest, Selenium, Cypress)
- Test execution, CI/CD pipeline setup, test reporting and metrics dashboards
- Defect tracking and root cause analysis
- AI-specific testing: Evaluation frameworks (accuracy, F1, precision, recall), regression testing for models, hallucination testing, bias detection, A/B testing for AI features, mocking LLM APIs in testing, handling non-deterministic outputs through deterministic test design
Education
Degree in Computer Science, Mathematics, or equivalent professional experience. Practical portfolio or published work demonstrating GenAI/ML implementation is valued highly.
Responsibilities
- Build: Design and implement end-to-end GenAI systems—RAG pipelines, fine-tuning strategies, prompt optimization. Make pragmatic choices between model options, latency/cost trade-offs, and production constraints.
- Ship: Own deployment of AI features. Ensure reliability, scalability, and proper handling of edge cases (hallucinations, rate limits, failures).
- Measure: Instrument systems with telemetry. Run experiments comparing approaches. Iterate based on production data and user feedback.
- Collaborate: Mentor engineers on AI best practices. Participate in code reviews. Help the team adopt and scale GenAI capabilities
Click on Apply to know more.
