Josh Talks
Website:
joshtalks.com
Job details:
Head of Operations, Voice of IndiaJosh Talks · Voice of India · Full-time · Gurgaon
About Josh TalksJosh Talks helps enterprises, governments and not-for-profits make better decisions about which AI services to buy and deploy, through independent, human-powered evaluation. Voice of India, its evaluation platform, sits at the centre of this work.
Josh Talks began as one of India's largest regional-language storytelling platforms, and the reach it built over a decade now underpins its work in AI.
About Voice of IndiaVoice of India is India's first human-powered multimodal evaluation platform, built by Josh Talks and co-developed with IIT Madras. It measures how well AI models work for the people of India, in their own languages and on the tasks they care about.
Our founding principle is that AI should be judged by people, not by other AI systems. Every ranking is backed by independent human evaluation, a published methodology and transparent evidence, rather than vendor claims or automated judges.
Live leaderboards today cover speech to text, text to speech, speech to speech and text to image, across 15 Indian languages. Leaderboards for language models, translation, legal tasks, document OCR, video and AI agents are in development. What sets the platform apart:
- Independent human judgment. Evaluations are carried out by people under blind conditions. Model names remain hidden until a judgment is recorded, the order of answers is randomised, and no vendor evaluates its own model.
- Built for procurement. Results are designed to inform procurement and deployment decisions. Scores are broken down by language, domain, geography and use case.
- Academically rigorous. Every benchmark follows a published protocol with sound statistics and reproducible experiments.
- Built from how India speaks. The speech to text benchmark uses 536 hours of unscripted telephone conversations from 36,691 speakers across 139 regional clusters. Its test set is closed, so models cannot be trained on it.
- Sovereign and global models side by side. Indian and global, open-source and proprietary models are ranked on the same boards.
Milestones so farSince its launch, Voice of India has established itself as an independent reference for AI evaluation in India:
- Government engagement at launch. Launched in New Delhi on 4 August 2026. Shri S. Krishnan, Secretary, Ministry of Electronics and Information Technology, attended the launch.
- Financial regulators in attendance. Representatives of the Reserve Bank Innovation Hub (RBIH) and the Securities and Exchange Board of India (SEBI) attended the launch.
- Peer-reviewed research. The speech to text benchmark is described in a paper accepted at Interspeech 2026.
- Trusted by model providers. Global and Indian model providers submit their models to Voice of India for evaluation and re-evaluation.
Press coverage: Business Standard, 4 August 2026
About the roleJosh Talks is hiring a Head of Operations to own Voice of India end to end. You will decide what Voice of India evaluates, run every evaluation from methodology to publication, make each release widely cited, and build the engine that keeps it rigorous and low-cost to run.
The pace is the defining feature of this role. Voice of India aims to publish a new evaluation almost every week, and to be the first in the world to put numbers on every major new model for India. You will set that cadence and hold it.
This role requires a high agency person who is hands on and not afraid to get their hands dirty. You will work directly with the founders of Josh Talks, our research and engineering teams, and the leads for enterprise partnerships and policy, and you will be accountable for what Voice of India publishes and its influence on AI deployment in the country.
How we workThis is a senior role by title, but it does not sit above the work. Our founder is deeply hands-on and involved in every detail of the company, from the wording of a methodology to the timing of a release, and expects the same of you.
- Know every moving part. You will understand how every evaluation works, from the data and the raters to the leaderboard, and know every word of every methodology document.
- Own every evaluation. Every evaluation Voice of India publishes is either written by you or thoroughly reviewed by you before it goes out.
- Stay close to the detail. You will work side by side with the founder on the details that decide whether a release is credible, rather than managing from a distance.
How this role is measuredWe believe the companies that matter in AI right now will be the ones that move fastest without losing rigour. Speed is therefore the primary measure of this role:
- Speed of releases. How quickly you ship new evaluations and put numbers on every major new model.
- Speed of partnerships. How quickly you turn conversations with labs, enterprises and government into working partnerships.
- Speed of impact. How quickly Voice of India's evaluations come to shape how AI is deployed in India.
If you would rather read the methodology than a summary of it, if a late release keeps you up at night, and if you want to work with a founder who cares about every detail as much as you do, you will feel at home here. If you are looking for a role where you manage from a distance, this is not it.
In this role, you willShip a new evaluation almost every week- Decide what to evaluate. Choose the models, tasks, languages and use cases that matter most, and own the evaluation calendar.
- Design the methodology. Work with our research team to define the protocol, test sets, rater instructions and statistics for each evaluation.
- Run production. Coordinate raters, vendors, data and engineering so each evaluation is delivered on time and to standard.
- Publish. Release results, leaderboards and methodology notes that stand up to scrutiny.
Be first in the world to put numbers on every major new model- Track every major release. Follow new models from global and Indian labs as they launch.
- Keep evaluations ready to run. Maintain the pipelines, test sets and rater capacity needed to evaluate a new model within days of its release.
- Publish first. Make Voice of India the first credible source of results for how each major model performs in Indian languages and conditions.
Make every release widely cited- Plan each release. Give every evaluation a clear headline finding and a release plan, working with our communications team.
- Reach the people who matter. Ensure labs, researchers and the press see, discuss and cite each release.
- Build credibility quickly. Respond to questions on methodology openly, establish collaborations across industry and government and publish corrections where needed.
Get Voice of India adopted by enterprises and government- Make scores part of AI decisions. Work with our Lead - AI Partnerships, India and our VP, Policy Programs & Partnerships so that a Voice of India score becomes part of how Indian enterprises and government choose AI.
- Shape evaluations around real decisions. Turn what enterprises and government need to know into evaluations Voice of India can deliver.
Build a low-cost, rigorous evaluation engine- Raters. Build and manage a reliable network of human raters across Indian languages, with clear quality controls.
- Vendors. Select and manage data, annotation and technology vendors, and get quality work from people you do not fully control.
- Tooling. Work with engineering to build the tools that make each evaluation faster and cheaper to run.
- Cost and quality. Track the cost and quality of every evaluation, and reduce cost without losing rigour.
What success looks likeWe expect the following milestones, which we will refine together in your first month:
- First 90 days. Own the evaluation calendar, ship 6 evaluations at a regular cadence, and map the raters, vendors and tooling behind each one, with a clear view of its cost.
- Six months. A new evaluation published almost every week, major new models evaluated within days of release, and a lower cost per evaluation with no loss of rigour.
- Twelve months. Voice of India regularly cited by labs, researchers and the press, its scores used in enterprise and government AI decisions, and a repeatable engine that runs at low cost.
Who we are looking for- A builder's track record. A past founder or early operator who shipped something every week.
- Technical rigour. Semi-technical and rigorous enough to defend a methodology to a researcher at a frontier AI lab.
- Senior credibility. Credible in conversation with a bank CTO or a ministry official.
- Ownership. You treat Voice of India as your own, and you hold a weekly cadence without cutting corners.
- Communication. Excellent writing, and the judgment to turn evaluation results into findings people cite.
- Quality through others. Experience getting quality work out of freelancers, vendors or contractors you did not fully control.
Experience in AI evaluation, data annotation, research operations or Indian-language AI is an advantage, but not required.
Why join- Own a national benchmark. Run the platform that shows how AI performs for the people of India.
- Work at the frontier. Evaluate new models from the world's leading labs as they launch.
- Shape how India chooses AI. See your work used by enterprises, government and researchers.
- Build from the ground up. Define how Voice of India operates, its team and its standards.
Working setup and assessmentThe role reports to the founder and CEO of Josh Talks and works closely with research, engineering, partnerships, policy and communications. It is a full-time position based in Gurgaon.
Assessment will cover something you shipped on a weekly cadence and a tight budget, how you managed people you did not fully control, and a short case exercise on designing and publishing an evaluation of a newly released model within one week.
How to applySend your CV and a short note to amritansh@voiceofindia.ai. Tell us about something you shipped every week on a tight budget, and what it achieved.
Click on Apply to know more.
