Objectways
Website:
objectways.com
Job details:
Role: Lead Data Annotation & AI Evaluation Specialist
Experience: 5–8+ years
Location: Chennai
Notice Period : Immediate
Role Summary
Objectways is seeking a highly technical Lead Data Annotation & AI Evaluation Specialist with at least 5 years of hands-on experience in complex data annotation, AI/ML data operations, or model evaluation.
This role will lead technically complex AI data projects involving LLMs, agentic AI, multimodal data, computer vision, NLP, model evaluation, safety evaluation, and human-in-the-loop workflows.
The ideal candidate must be able to take a detailed customer specification, read it end-to-end, understand the technical and operational intent, identify ambiguities and risks, design the annotation/evaluation workflow, create examples and edge cases, define QA criteria, and guide an annotation team through successful execution.
This is not simply a people-management or annotation-production role. We need someone who can think technically and act as the bridge between the customer specification, engineering/ML teams, and annotation operations.
Key Responsibilities
- Read and interpret complex customer annotation, data-generation, and AI-evaluation specifications.
- Convert requirements into clear SOPs, annotation guidelines, decision trees, examples, edge cases, and QA checklists.
- Independently determine how a project should be executed rather than waiting for step-by-step instructions.
- Design annotation schemas, taxonomies, labels, metadata, acceptance criteria, and review workflows.
- Understand complex multi-turn conversations, model behavior, tool calls, system prompts, agent trajectories, and contextual dependencies.
- Work with JSON/JSONL, structured data, APIs, tool-call traces, logs, model outputs, and annotation platforms.
- Create gold-standard examples and benchmark datasets before production begins.
- Conduct pilot annotations personally and identify gaps in customer guidelines before scaling to the annotation team.
- Define inter-annotator agreement, arbitration, QC sampling, error taxonomy, and acceptance thresholds.
- Analyze disagreements and distinguish annotator error, guideline ambiguity, tooling issues, and genuinely ambiguous data.
- Train annotators and reviewers on technically complex projects and certify readiness before production.
- Work closely with engineering and ML teams to understand model inputs/outputs and improve annotation tooling and workflows.
- Communicate directly with customers or internal project teams to raise technically meaningful clarification questions.
- Monitor production quality and identify systematic errors rather than simply reporting aggregate accuracy.
- Perform root-cause analysis and recommend changes to guidelines, workflows, tooling, or training.
- Own annotation quality from requirements → pilot → production → QC → delivery.
Technical Skills
The candidate should be comfortable with:
- LLMs and Generative AI
- AI agents and multi-turn conversational systems
- Prompt/response evaluation
- AI safety and model behavior evaluation
- NLP annotation and classification
- Annotation schema and taxonomy design
- JSON/JSONL and structured datasets
- Tool calls, API concepts, and structured model outputs
- Python or SQL at a working level
- Regular expressions and basic scripting/data analysis
- Annotation platforms such as CVAT, Label Studio, Encord, SageMaker Ground Truth, or equivalent
- Quality metrics including accuracy, precision/recall, confusion matrices, IAA/Cohen's Kappa/Fleiss' Kappa
- Dataset validation and error analysis
Click on Apply to know more.
