Role Summary
We’re hiring an AI Product & Evals Analyst to own evaluation, quality measurement, dashboarding, and enterprise analytics for AI voice agents.
You will work at the intersection of AI Engineering, Product, and Customer Delivery to define how voice agent quality is measured, monitored, and improved at scale. This includes building evaluation frameworks, detecting regressions, analyzing business impact, and enabling enterprise teams with clear, actionable insights.
Beyond evaluation, we’re looking for someone with an AI-first mindset who can leverage modern AI tools and automation workflows to improve analytics, reporting, and operational efficiency. The role will also contribute toward building reusable analytics frameworks, internal intelligence tooling, and self-serve analytics capabilities for enterprise clients.
Your work will directly influence launch quality, customer trust, operational transparency, and how enterprise AI systems are measured and scaled.
What will you do?
- Business & Enterprise Analytics
- Track and analyze business KPIs tied to voice agent deployments (automation, containment, conversion, CSAT, cost savings, etc.)
- Support enterprise billing, audit, and reconciliation workflows through reliable usage and conversation analytics
- Translate AI behavior into clear operational and business impact insights for enterprise clients
- AI-First Analytics & Automation
- Leverage AI tools and agentic workflows to improve speed, quality, and scalability of analytics, reporting, and RCA workflows
- Automate repetitive analytics and reporting tasks using LLMs, copilots, and workflow tooling
- Drive AI-native ways of working across SQL, Python, dashboards, and insight generation
- Analytics Platform & Internal Tooling
- Build reusable analytics frameworks, standardized dashboards, and internal tooling for enterprise deployments
- Contribute to self-serve analytics and intelligence capabilities such as text-to-data, AI-powered insights, and conversational analytics interfaces
- Help evolve Bluemachines’ analytics and observability layer for enterprise AI voice agents
- Voice Agent Evaluation
- Design and own AI eval frameworks specifically for voice agents
- Define conversation- and turn-level metrics
- Build and maintain golden voice datasets across accents, languages, and noise conditions
What We’re Looking For
Must-Have
- 2–5 years of experience in Product Analytics, Data Analytics, AI Quality, or related analytical roles
- Strong analytical thinking, metric design, and problem-solving skills
- Hands-on proficiency in SQL and Python for analysis, evaluation, and data investigation
- Ability to reason deeply about conversational, voice, and workflow-driven systems
- Experience building dashboards and enterprise-grade visualizations using tools like Tableau, Power BI, Looker, or equivalent
- Strong communication and storytelling skills with the ability to translate AI/system behavior into business insights
- Comfort working cross-functionally with AI Engineering, Product, Operations, and customer-facing teams
- AI-first mindset with strong adoption of modern AI tools, copilots, and automation workflows to improve productivity and scale analytics
Good-to-Have
- Experience with voice systems such as STT, TTS, call analytics, or conversational AI platforms
- Exposure to LLM-based products, prompt engineering, AI evals, or agentic workflows
- Experience with enterprise analytics, client-facing reporting, billing/reconciliation, or operational analytics
- Interest in building internal analytics tooling, reusable frameworks, or self-serve intelligence systems
- Experience working in fast-moving AI or startup environments
