Talent.com
Mercor
Data Science Expert - AI EvaluationMercor • Toronto, ON, Canada
Data Science Expert - AI Evaluation

Data Science Expert - AI Evaluation

Mercor • Toronto, ON, Canada
2 days ago
Job type
  • Full-time
Job description

1. Role Overview

Mercor is partnering with a leading AI research organization to engage experienced data scientists for a project focused on evaluating how well AI systems perform real-world data science work. Rather than producing deliverables yourself, you will define what excellent work looks like: designing task-specific grading criteria and scoring completed work samples with rigorous, well-reasoned written justifications.

2. Key Responsibilities

  • Design precise, task-specific grading criteria for real-world data science deliverables (analyses, models, dashboards, experiment readouts, and written recommendations)
  • Score AI-generated and human work samples against those criteria, with detailed written justifications for every score
  • Apply consistent, evidence-based judgment so that scores are reproducible and defensible
  • Incorporate structured feedback from senior reviewers and iterate quickly on your work

3. Ideal Qualifications

  • 5+ years of professional data science experience in industry
  • Background in business operations, product, or growth data science at top-tier technology companies
  • Deep fluency in experiment design and A/B testing, metric definition, SQL/Python analysis, and communicating findings to executive stakeholders
  • Exceptionally strong written communication
  • Detail-oriented, consistent, and comfortable having your judgment reviewed and calibrated against peers
  • Prior experience with AI training, evaluation, or human-data projects is a strong plus

4. Application Process

  • Qualified applicants may be asked to complete a brief technical assessment or submit additional information
#J-18808-Ljbffr
Create a job alert for this search

Data Science Expert - AI Evaluation • Toronto, ON, Canada

Similar jobs

Data Science Leader in Automotive Innovation

Magna InternationalAurora, York region, Canada
Full-time

Become a Senior Data Scientist with Magna and drive innovation in automotive technologies through data analytics and machine learning.Empower your curiosity to make an impact.As part of the data an... Show more

 • Promoted

Expert Data Scientist For Ai Innovations

AutoTrader.caToronto, Canada
Full-time

Drive innovation in the automotive marketplace as an Expert Data Scientist focusing on AI solutions.Initiate high-impact projects using AI/ML technologies to redefine industry standards.In this rol... Show more

 • Promoted

Applied AI Research Scientist — Multimodal Evaluation

MillionLogicsToronto, ON, CA
Full-time

A global IT solutions provider is searching for skilled Applied AI Research Scientists to contribute to AI system evaluations.Candidates should hold an MS or Ph.The role involves designing rigorous... Show more

 • Promoted

Data Science Team Lead - AI & Analytics

Ample Insight Inc.Toronto, ON, CA
Full-time

Join us as a Data Science Team Lead, focusing on AI and analytics solutions.Partner with clients' teams to implement data-driven strategies that enhance business performance.In this role, you will ... Show more

 • Promoted

Senior Researcher – AI Data Systems

Huawei Canadamarkham, on, Canada
Full-time

Huawei Canada has an immediate 12‑month opening for a Researcher.Established in 2014, the Distributed Scheduling and Data Engine Lab is Huawei Cloud's technical innovation center in Canada.The lab ... Show more

 • Promoted

AI Platform Engineer: Build & Evaluate AI Systems

Armilla AItoronto, on, Canada
Full-time

A leading AI-focused startup in Toronto is seeking an experienced AI Engineer to shape the future of AI risk management.In this pivotal role, you will be responsible for building core AI tools and ... Show more

 • Promoted

Applied Scientist

ProducePayToronto, Ontario, Canada
Full-time

Overview Waabi is a leader in Physical AI, developing autonomous driving technology that powers commercial autonomous trucks and robotaxis.With offices in Toronto, San Francisco, Dallas, and Pittsb... Show more

 • Promoted

Senior Genai Data Scientist - Llm Strategy & Evaluation - C$135,000 - C$175,000 A Year

Insurtech CompanyToronto County, Canada
Full-time

Senior Data Scientist role focusing on Generative AI and LLM strategy, requiring ML/NLP expertise, Python, and SQL skills. Show more

 • Promoted

Data Science Manager

Lyft, Inc.Toronto, ON, CA
Full-time

At Lyft, our mission is to improve people’s lives with the world’s best transportation.To do this, we start with our own community by creating an open, inclusive, and diverse organization.Our trans... Show more

 • Promoted

Data Science Analyst

Omnicom MediaToronto, Canada
Full-time

Position Summary We are looking for a Data Science Analyst to join our growing data science practice.In this role, you will support the development and delivery of quantitative models and analyses... Show more

 • Promoted

Strategic AI Deployment Expert at Pigment

PigmentToronto
Full-time

Take on the role of a Strategic AI Deployment Expert at Pigment, focused on crafting and implementing cutting-edge AI planning solutions for clients.Leverage your skills to redefine organizational ... Show more

 • Promoted

Hybrid Data Scientist: Impactful Analytics & AI

VictraysToronto, ON, CA
Full-time

A leading consulting firm in Toronto is seeking an experienced Data Scientist to provide data-driven solutions in a hybrid work environment.The ideal candidate will have advanced skills in data ana... Show more

 • Promoted

Remote Ai Data Quality & Evaluation Engineer - C$85,000 - C$225,000 A Year - Remote

Cloud Technology CompanyNorth York, Canada
Remote
Full-time

AI Data Quality & Evaluation Engineer needed to validate and ensure the reliability of AI Agents using expertise in data quality, prompt engineering, and Python. Show more

 • Promoted

Data Scientist Role Specializing in AI/ML Model Innovation

Tata Consultancy ServicesToronto, Ontario, Canada
Full-time

Seize the moment as a Data Scientist specializing in cutting-edge AI/ML model development.This role embraces remote work, providing a unique opportunity to drive innovation and foster inclusivity.Y... Show more

 • Promoted

Remote Data Analytics Scientist – AI & Dashboards

Handis Consulting Inc.Toronto, ON, CA
Remote
Full-time

A consulting firm is seeking a Data Analytics Specialist for a one-year full-time contract in Edmonton, Canada.The role involves providing leadership on data initiatives, engaging with internal and... Show more

 • Promoted

Expert Data Scientist for AI Innovations

AutoTrader.caToronto, ON, CA
Full-time

Drive innovation in the automotive marketplace as an Expert Data Scientist focusing on AI solutions.Initiate high-impact projects using AI/ML technologies to redefine industry standards.In this rol... Show more

 • Promoted

Remote AI Data Quality & Evaluation Engineer

Veeva Systemstoronto, on, Canada
Remote
Full-time

A leading cloud technology company in Toronto is seeking an experienced professional to validate and ensure the reliability of Veeva AI Agents.The ideal candidate should have expertise in data qual... Show more

 • Promoted

Data Science expert

Amaris ConsultingToronto, Ontario, Canada
Full-time

We are looking for a Data Science Expert to support the development of advanced data and analytics solutions within a highly regulated manufacturing environment.The successful candidate will work c... Show more

 • Promoted

Remote CS Expert & Data Partner for AI Q&A

TELUS Digital AI Data SolutionsToronto, ON, CA
Remote
Full-time

A leading technology firm is seeking a Computer Science Expert to join as a Freelance Data Partner.This remote role involves developing complex question-and-answer pairs in your area of expertise a... Show more

 • Promoted

Remote Climate Data Scientist - AI-Driven Research

Crossing HurdlesToronto, ON, CA
Remote
Full-time

A referral partner specializing in climate data is seeking a Climate Data Specialist to develop and refine climate analysis tasks.This role involves creating evaluation rubrics for AI performance a... Show more