Sycamore Informatics Inc.
Website:
sycamoreinformatics.com
Job details:
Role - Lead QA Analyst – Agentic AI
Designation - Principal Lead Engineer
Experience (in years) - 9 – 15 Years Experience
Education - e.g., Bachelor’s degree, preferably in Computer Science, Electrical Engineering, Physics, Math or any other related discipline. (Education requirements to be entered here)
Summary
The Senior Agentic AI QA Lead Principle owns the quality strategy for Agentic AI platforms. This role focuses on defining AI test strategies, evaluating systemic risks, validating autonomous behaviors, and ensuring production readiness of agentic systems.
Roles & Responsibilities
● Define end-to-end testing strategy for agentic and autonomous AI systems
● Design complex test scenarios for reasoning chains, tool orchestration, and multi-agent collaboration
● Lead testing for hallucinations, bias, ethical risks, and unsafe behaviors
● Validate human-in-the-loop controls, auditability, and traceability
● Review and approve AI test cases, defect reports, and validation evidence
● Partner with engineering, product, and compliance teams on risk mitigation
● Mentor QA teams on agentic AI testing techniques
● Provide release recommendations based on risk-based quality assessments
● Define and conduct any scale/performance testing for AI Agents built as well
● Train Sycamore resources with Agentic AI skills where applicable
Essential Experience
Technical Skills – Advanced Agentic AI Testing
● Deep understanding of agent frameworks and orchestration layers
● Expertise in testing autonomous workflows and decision policies
● Experience evaluating planning failures, looping behaviors, and dead-ends
● Advanced prompt evaluation, adversarial testing, and red-teaming
● Strong experience testing API-driven and distributed agent architectures
● Ability to design evaluation metrics for accuracy, reliability, and safety
● Experience reviewing logs, traces, and telemetry for agent behavior analysis
● Knowledge of AI governance, model risk management, and guardrails
● Exposure to automation (Python, Robot Framework, API automation
Deep-Dive Testing Experience (Senior)
● Designing AI test strategies under uncertainty and non-determinism
● Validating autonomous escalation and self-correction mechanisms
● Testing long-running agents and stateful workflows
● Defining acceptance criteria for probabilistic outputs
● Assessing business and compliance risks of agent decisions
● Establishing testing standards and review processes for AI systems
Desired Experience
● Experience defining AI validation approaches under CSA or GxP
● Automation framework ownership or strategy experience
● Experience in AI governance, audits, or regulatory inspections
● Working with Any LLMs like Claude with aws bedrock
Click on Apply to know more.
