GiantMind Solutions
Website:
gmindsolution.com
Job details:
Role: LLM – AI Quality Analyst (Personalization) – Korean
Experience: 1 Year
Engagement Type: Short Term Contract (2 Months)
Availability: Full Time (8 Hours) with 4 hours overlap with PST
Special Requirements
- Korean Proficiency: Strong ability to read and write in Korean.
- Personal Google Account: Must be willing to use their primary personal Google account with personal data sources enabled.
- Schedule Flexibility: Must be available full-time in local timezone as part of a 24/7 global operations team.
- Strong Analytical Thinking: Ability to evaluate complex and ambiguous AI responses.
Role Overview
As an AI Quality Analyst, you will evaluate a new Gemini personalization feature.
You will analyze how effectively the model uses information from:
- Gemini conversations
- Gmail
- Google Search
- YouTube activity
to generate relevant and personalized responses.
This role requires both creative prompt design and analytical evaluation of AI outputs.
Key Responsibilities
- Design multi-turn conversational prompts (1–5 turns) based on personal experiences.
- Evaluate model responses for personalization accuracy and relevance.
- Identify grounding issues, hallucinations, or incorrect inferences.
- Assess integration quality of personal data in responses.
- Perform Side-by-Side (SxS) evaluation of two AI responses.
- Rank responses based on helpfulness, clarity, and naturalness.
- Write clear rationales explaining the evaluation results.
- Extract and verify Debug Info to confirm correct data usage.
- Maintain data hygiene by deleting evaluation chats.
Required Skills
- Experience in AI quality evaluation / data annotation / content moderation (preferred).
- Prompt engineering and personalization testing knowledge.
- Strong attention to detail in reviewing AI responses.
- Ability to detect hallucinations and incorrect personalization.
- Excellent written communication for evaluation reports.
- Ability to work independently in a remote environment.
Education
- BS / BA degree or equivalent experience in:
- Computer Science
- Linguistics
- Journalism
- Policy / Law / Ethics
- Other analytical fields
Technical Requirements
- Desktop or Laptop
- Stable Internet Connection
Click on Apply to know more.