Website:
bls360.com
Job details:
· Remote (India)
· 12 PM – 8 PM IST
· Contract: 2 Years
· Onboarding: Immediate
About BLS360
BLS360 is a fast-growing cybersecurity, identity & access management (IAM), and digital transformation company, helping enterprises secure and modernize their technology landscape. Founded in 2019, BLS360 delivers scalable, secure, and customized solutions across industries including finance, healthcare, and government.
With strong partnerships across leading cloud and identity platforms, BLS360 focuses on identity-centric security, compliance, and next-gen digital innovation, enabling organizations to navigate complex, multi-cloud environments with confidence.
Role Overview
We’re looking for a Senior AI QA / QE Engineer to lead quality engineering efforts for cutting-edge AI-powered applications. This role is ideal for someone who thrives at the intersection of QA, automation, and AI systems, and can ensure reliability in complex, non-deterministic environments.
What You’ll Do
- Lead QA strategy and execution for AI-driven applications and systems
- Build and maintain automated test frameworks (Python, Robot Framework) across APIs, UI, backend, and data pipelines
- Integrate automated tests into CI/CD pipelines
- Perform functional, performance, and exploratory testing
- Validate AI/LLM outputs using structured and non-deterministic testing approaches
- Conduct prompt testing and evaluate LLM behavior (PromptFoo experience preferred)
- Test and validate RAG-based systems, including ground truth evaluation
- Collaborate with engineering and product teams to define quality benchmarks
- Debug complex issues, including AI/LLM-specific challenges
Must-Have Skills
- Strong experience in QA automation and test strategy
- Hands-on experience testing AI/LLM-powered applications and agents
- Expertise in non-deterministic output validation and ground truth comparison
- Experience with RAG testing and validation
- Proficiency in Python-based automation frameworks
- Solid understanding of defect analysis and system debugging
- Experience working in Agile environments with tools like Jira
- Test and validate Retrieval-Augmented Generation (RAG-based systems), including ground truth evaluation
Nice to Have
- Experience with Azure OpenAI
- Familiarity with OpenSearch (vector databases)
- Exposure to monitoring tools like Datadog
What Makes You a Great Fit
- You can identify gaps in AI response validation and improve quality rigor
- You’re comfortable working with ambiguous or partially correct AI outputs
- You bring a hands-on approach to problem-solving—not just strategy
- You can help ensure release readiness under tight timelines
Why Join BLS360?
- Work on cutting-edge AI + cybersecurity solutions
- Be part of a rapidly growing global CyberTech company
- Collaborate with global teams on impactful projects
- Opportunity to build impactful products, solutions, services and beyond.
- Play a key role in shaping quality standards for AI applications
Click on Apply to know more.