Website:
Job details:
Role Overview
We are looking for a
QA Engineer to validate AI-powered analytical solutions where Agents use
Databricks Genie to interpret natural-language questions and generate data-driven insights. The role will focus on validating
data accuracy, response quality, business logic, and end-to-end reliability.
Key Responsibilities
- Test AI Agents and analytical solutions powered by Databricks Genie.
- Validate natural-language questions and Genie-generated analytical responses against source data and SQL results.
- Test analytical scenarios including KPIs, aggregations, trends, comparisons, filters, and drill-downs.
- Validate responses across different question phrasings, ambiguous queries, edge cases, and multi-turn conversations.
- Identify incorrect calculations, inconsistent responses, hallucinations, and incorrect interpretation of business terminology.
- Perform functional, regression, API, integration, and data validation testing.
- Build reusable test cases and automated regression suites for analytical queries.
- Collaborate with Data Engineers, AI Engineers, business SMEs, and Product Owners on acceptance criteria and defect resolution.
Required Skills
- 3-6 years of QA/testing experience, preferably in data, analytics, BI, or AI solutions.
- Strong SQL and data validation skills.
- Experience testing analytical or data-driven applications.
- Experience with API testing using tools such as Postman.
- Automation experience using Python/PyTest or similar frameworks.
- Good understanding of data pipelines, KPIs, aggregations, and analytical datasets.
- Exposure to Databricks, Databricks Genie, LLMs, or GenAI testing is highly preferred.
- Understanding of AI testing concepts such as accuracy, groundedness, hallucination, relevance, and response consistency.
Good to Have
- Azure Databricks / PySpark knowledge.
- Experience testing conversational BI or natural-language-to-SQL solutions.
- Experience automating comparison of Agent responses against expected SQL/data outputs.
Click on Apply to know more.