Talent.com
Mercor
Data Scientist - AI EvaluationMercor • Toronto, Ontario, Canada
Data Scientist - AI Evaluation

Data Scientist - AI Evaluation

Mercor • Toronto, Ontario, Canada
2 days ago
Job type
  • Part-time
  • Remote
  • Quick Apply
Job description

About the job

Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark, General Catalyst, Peter Thiel, Adam D'Angelo, Larry Summers, and Jack Dorsey.

Position: Data Scientist Talent Network
Type: Contract
Compensation: $100–$150/hour
Location: Remote

Role Responsibilities

  • Design precise, task-specific grading criteria for data science deliverables, including exploratory data analyses, statistical modeling work, machine learning pipelines, and A/B test write-ups.
  • Evaluate AI-generated or human-created work against established criteria to ensure quality and consistency.
  • Provide detailed written justifications for evaluations and scores to maintain transparency and accountability.
  • Apply consistent, evidence-based judgment to ensure assessments are reproducible and defensible.
  • Incorporate structured feedback from senior reviewers and iterate on submitted work for continuous improvement.
  • Work independently and asynchronously to meet deadlines and improve AI model performance.

Qualifications

Must-Have

  • 1+ years of professional data science experience.
  • Experience at a leading technology, research, or quantitative firm.
  • Strong command of Python, SQL, statistical modeling, machine learning, and experimentation.
  • Exceptional written communication skills.
  • Detail-oriented and consistent approach to evaluating complex work.
  • Comfort receiving feedback and calibrating judgment against established standards.

Application Process (Takes 20–30 mins to complete)

  • Upload resume
  • AI interview based on your resume
  • Submit form

Resources & Support

  • For details about the interview process and platform information, please check: https://talent.docs.mercor.com/welcome
  • For any help or support, reach out to: support@mercor.com

PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.

Create a job alert for this search

Data Scientist - AI Evaluation • Toronto, Ontario, Canada

Similar jobs

Senior Genai Data Scientist - Llm Strategy & Evaluation - C$135,000 - C$175,000 A Year

Insurtech CompanyToronto, Canada
Full-time

A leading insurtech company based in Vancouver is seeking a Senior Data Scientist for Generative AI to drive LLM strategies.You will evaluate model performance, fine-tune complex models, and collab... Show more

 • Promoted

Cohere Senior Research Scientist - AI Evaluation

CohereToronto, ON, CA
Full-time

Drive cutting-edge AI evaluation as a Senior Research Scientist at Cohere.Innovate new methods to measure large language models' performance and set the standard for future capabilities.As a Senior... Show more

 • Promoted

Applied AI Research Scientist — Multimodal Evaluation

MillionLogicsToronto, ON, CA
Full-time

A global IT solutions provider is searching for skilled Applied AI Research Scientists to contribute to AI system evaluations.Candidates should hold an MS or Ph.The role involves designing rigorous... Show more

 • Promoted

Applied Scientist

ProducePayToronto, Ontario, Canada
Full-time

Waabi is a leader in Physical AI, developing autonomous driving technology that powers commercial autonomous trucks and robotaxis.With offices in Toronto, San Francisco, Dallas, and Pittsburgh, the... Show more

 • Promoted

Data Scientist - Remote

YO IT ConsultingToronto, ON, CA
Remote
Full-time

In this role, you'll apply your expertise to help train next-generation AI systems.Your work will shape how models learn, reason, and perform through high-quality, real-world input.No prior experie... Show more

 • Promoted

Lead Data Scientist for AI Innovation

CohereToronto, Ontario, Canada
Full-time

Contribute to AI advancements as the Lead Data Scientist at Cohere.Use your analytical skills and leadership to shape future AI and analytics landscapes.In this influential position with the Analyt... Show more

 • Promoted

Data Scientist for Conversational AI Systems

DelpathToronto, ON, CA
Full-time

Shape the future of banking with your data science expertise.Focus on advanced conversational AI systems and training large language models in a hybrid work setup.In this crucial role, you will wor... Show more

 • Promoted

Data Scientist Role Specializing in AI/ML Model Innovation

Tata Consultancy ServicesToronto, ON, CA
Full-time

Seize the moment as a Data Scientist specializing in cutting-edge AI/ML model development.This role embraces remote work, providing a unique opportunity to drive innovation and foster inclusivity.Y... Show more

 • Promoted

Hands-On Data Scientist - Analytics & AI

Inizio Partners CorpToronto, ON, CA
Full-time

Step into a hands-on Data Scientist role focused on Analytics & AI within the insurance sector.Combine your data science skills and insurance know-how to deliver analytical solutions that drive bus... Show more

 • Promoted

Data Scientist - AI-Driven Insights

GorgiasToronto, ON, CA
Full-time

Become a key player at Gorgias as a Senior Data Scientist, leveraging AI to reshape customer interactions.Focus on predictive modeling and data science initiatives in a collaborative setting.This r... Show more

 • Promoted

Applied AI Scientist (Hybrid)

HomebaseToronto, ON, CA
Full-time

Leverage, fine‑tune, and deploy models—owning the lifecycle from experimentation to.Develop and track metrics that measure real customer impact, ensuring alignment with business outcomes.Partner cr... Show more

 • Promoted

Data Scientist: AI & Tax Innovations

BDO CanadaToronto, ON, CA
Full-time

Elevate your career at BDO as a Data Scientist in the Tax Transformation Office, where you will apply AI and machine learning to revolutionize tax operations.This role involves a blend of tech expe... Show more

 • Promoted

Hybrid Data Scientist: Impactful Analytics & AI

VictraysToronto, ON, CA
Full-time

A leading consulting firm in Toronto is seeking an experienced Data Scientist to provide data-driven solutions in a hybrid work environment.The ideal candidate will have advanced skills in data ana... Show more

 • Promoted

Remote AI Data Quality & Evaluation Engineer

Veeva SystemsToronto, ON, CA
Remote
Full-time

A leading cloud technology company in Toronto is seeking an experienced professional to validate and ensure the reliability of Veeva AI Agents.The ideal candidate should have expertise in data qual... Show more

 • Promoted

Remote Data Analytics Scientist – AI & Dashboards

Handis Consulting Inc.Toronto, ON, CA
Remote
Full-time

A consulting firm is seeking a Data Analytics Specialist for a one-year full-time contract in Edmonton, Canada.The role involves providing leadership on data initiatives, engaging with internal and... Show more

 • Promoted

Data Scientist with Generative AI Focus

Manulife Financial CorporationToronto, ON, CA
Full-time

Take your career to the next level as a Data Scientist at Manulife, specializing in Generative AI technologies.Drive innovation in AI/ML products that yield real-world business value.As part of the... Show more

 • Promoted

Expert Data Scientist for AI Innovations

AutoTrader.caToronto, ON, CA
Full-time

Drive innovation in the automotive marketplace as an Expert Data Scientist focusing on AI solutions.Initiate high-impact projects using AI/ML technologies to redefine industry standards.In this rol... Show more

 • Promoted

Data Scientist - AI/ML

ScotiabankToronto, ON, CA
Full-time

Select how often (in days) to receive an alert:.Select how often (in days) to receive an alert:.Please be advised that our Careers site will be.November 28 at 12am ET to November 29 12am ET for sch... Show more

 • Promoted

Remote Climate Data Scientist - AI-Driven Research

Crossing HurdlesToronto, ON, CA
Remote
Full-time

A referral partner specializing in climate data is seeking a Climate Data Specialist to develop and refine climate analysis tasks.This role involves creating evaluation rubrics for AI performance a... Show more

 • Promoted

Data Scientist | $120/hr Remote

Crossing HurdlesToronto, ON, CA
Remote
Full-time

Data Scientist | $120/hr Remote.Direct message the job poster from Crossing Hurdles.At Crossing Hurdles, we work as a referral partner.We refer candidates to our partner that collaborates with worl... Show more