AI Evaluator / Annotator

Position Overview: iMerit seeks detail-oriented and analytically minded Multimodal GenAI Evaluation Analysts to perform highly nuanced evaluations of AI system outputs across different modalities: text, image, video, and multimodal interactions. Analysts will assess the accuracy, appropriateness, quality, clarity, and cultural alignment of model outputs against complex guidelines, ensuring that results align with project standards and real-world use cases. These evaluations will directly inform the development and fine-tuning of advanced large language models (LLMs), vision models (LVMs), and multimodal AI systems. Role Responsibilities: ● Evaluate outputs generated by LLMs across multiple modalities (text, image captions, video descriptions, and multimodal prompts). ● Assess quality against project-specific criteria such as correctness, coherence, completeness, style, cultural appropriateness, and safety. ● Identify subtle errors, hallucinations, or biases in AI responses. ● Apply domain expertise and logical reasoning to resolve ambiguous or unclear outputs. ● Provide detailed written feedback, tagging, and scoring of outputs to ensure consistency across the evaluation team. ● Escalate unclear cases and contribute to refining evaluation guidelines. ● Collaborate with Project Managers and Quality Leads to meet accuracy, reliability, and turnaround benchmarks. Skills & Competencies: ● Strong critical reading, observational, and evaluative skills across different modalities. ● Ability to articulate nuanced judgments with precision and clarity. ● Excellent English comprehension (CEFR B2 or above); additional languages a plus. ● Familiarity with LLMs, generative AI, and multimodal systems. ● Strong attention to detail and ability to apply guidelines consistently. ● Awareness of cultural and linguistic nuances, including potential bias and harm in AI outputs. ● Comfort with evolving workflows, rapid feedback cycles, and complex quality frameworks.

iMerit