Website:
Job details:
Company: Lofte Studios
Workplace Type: Remote
Job Type: Contract
Location: Remote
About the Role:
We are hiring for AI Quality Analysts to evaluate personalization features on a leading LLM platform. As an AI Quality Analyst, you will evaluate a new personalisation feature for a leading LLM assistant platform. You will assess how well the model uses information from the user's past conversations, email activity, search history, and video platform activity to make responses more relevant and helpful. This role requires a unique blend of creativity and analytical rigour. You will actively design prompts from the perspective of your own personal experiences, then use your analytical skills to assess the quality of the model's personalised responses across dimensions such as Grounding, Integration, and Helpfulness.
About the Engaging Company:
The engaging company is a global AI research accelerator and trusted partner for enterprises deploying advanced AI systems. It supports clients in two ways: first, by accelerating frontier research with high-quality data, advanced training pipelines, and top AI researchers specialising in coding, reasoning, STEM, multilinguality, multimodality, and agents; and second, by applying that expertise to help enterprises move AI from proof-of-concept into proprietary intelligence that delivers measurable business impact.
What You'll Do:
- Design multi-turn conversational prompts (1–5 turns) based on your personal experiences
- Evaluate AI responses for correct and natural use of personal context
- Identify hallucinations, poor inferences, and forced personalizations
- Rank two model responses side-by-side (SxS) and write clear rationales
- Verify debug information to confirm data sources were properly used
- Maintain data hygiene by deleting evaluation conversations post-session
What We're Looking For:
- Chinese Proficiency: Strong ability to read and write in Chinese (required)
- Experience in data annotation, AI/LLM evaluation, QA, or content moderation (preferred)
- BA/BS in Linguistics, CS, Journalism, Law, Ethics, or a related field
- Ability to evaluate nuanced AI responses with analytical rigor
- Comfortable using personal accounts and enabling personal data sources for evaluation
- Self-motivated, detail-oriented, and able to work independently
Selection Process
1. Submit your application
2. Shortlisted candidates receive a Job Interest Form
3. Assessment will be shared must be completed within 24 hours
4. Pre-onboarding discussion with shortlisted candidates
Click on Apply to know more.