Website:
joinshift.com
Job details:
About shift
shift pays everyday workers to record short videos of the tasks they already do (in kitchens,
workshops, job sites, bakeries, and warehouses) to help train the next generation of AI and
robotics. We’ve paid out over $15M to 25,000+ workers across 15+ countries.
The role
We need someone to own the quality bar for shift's egocentric training data - not to work to a
spec, but to write it. You'll own the golden sets, SOPs, and calibration system that every
annotator, reviewer, and auto-labeling model is measured against, and you'll work directly with
engineering to stand up new annotation workflows in hand tracking, object identification, and
action labelling. As we scale from thousands of contributors to millions, you're the reason the
99% accuracy bar holds. Your ground truth becomes the benchmark, and the model is
measured against it.
What you'll do
- Own the quality bar end-to-end: golden/benchmark sets, acceptance criteria, frame-level tolerance thresholds, and the audit loops that hold them
- Author, version, and continuously sharpen annotation SOPs, rubrics, and edge-case rules as the client spec evolves
- Design and run inter-annotator agreement measurement (kappa/alpha, tolerance-based agreement on temporal boundaries) and publish regular quality reports
- Analyse disagreement patterns to root cause - guideline ambiguity vs. annotator error - and fix the system, not just the file
- Stand up the training and certification pipeline that brings new annotators and reviewers to the bar fast: onboarding, structured calibration sessions, targeted retraining
- Design human-in-the-loop workflows where models pre-label and humans review, correct, and escalate - so output grows faster than headcount
- Partner with engineering and product on new workflows: hand tracking (keypoints, left/right attribution, contact states), object identification (boxes/masks, ontology, naming consistency), action labelling (controlled vocabulary, segment boundaries)
- Specify label schemas, annotation-tool requirements, and QC dashboards; validate pilot batches before production scaling
- Run capacity planning across competing demands and allocate reviewer attention to the highest-impact work
- Track acceptance/rejection, rework, and quality-adjusted productivity - and drive them week over week
- Improve cost-per-accepted-hour while protecting the bar; inform the in-house vs. vendor mix with quality data
What we're looking for
- 5+ years of hands-on video annotation, including 1+ years with egocentric/first-person video for robotics or embodied AI - ideally at or for a leading robotics company or project
- 3+ years in a lead or QA capacity: calibrating annotators, adjudicating disagreements, and owning guidelines that others work to
- Track record standing up 0→1 annotation programs and validating them through pilot to production
- Deep command of quality methodology: IAA frameworks (kappa/alpha), gold-set creation, sampling strategies, tolerance thresholds - you can explain what a 99% (versus 95%) accuracy bar means in practice and design the system that holds it
- Track record authoring (not just following) annotation SOPs and rubrics; able to translate ambiguous specs into precise, actionable guideline language
- Hands-on with 2+ professional annotation tools - CVAT, Labelbox, Encord, V7, Label Studio, or equivalent - including configuring label schemas and QA workflows, not just annotating in them
- Experience integrating model-based annotation into human workflows: auditing auto-generated labels and designing human-in-the-loop review that raises throughput without sacrificing quality
- Outstanding written English - your rubrics train annotators and your labels are commands a model learns from
- Strong analytical skills with Excel/Google Sheets for quality reporting
- Data-driven, process-oriented mindset with strong ownership; startup-ready - comfortable with ambiguity, zero-to-one builds, and contributing when it matters most
- Working understanding of ML and why annotation quality drives model performance
- Bonus: Robotics, mechanical/mechatronics, or computer-vision engineering background; tier-1 institution or AI/robotics startup experience
- Bonus: Experience managing distributed/global annotator workforces or external vendors
- Bonus: Multi-modal annotation exposure: 3D/depth, joint pose, grasp outcome classification
- Bonus: Experience training or fine-tuning auto labeling models, or partnering closely with the ML teams that do
- Bonus: Built annotation tooling or partnered tightly with a tooling team
Why this role
- The quality bar is the product: shift's ground truth becomes the benchmark that physical-AI models are measured against - this is the role that sets it
- Ground-floor ownership at a fast-growing, well-funded startup featured in Forbes, BBC, and Business Insider - you'll build the quality system from scratch, not inherit someone else's
Click on Apply to know more.
_1575283587728.jpeg){kind=logo}