Talent.com
YO AI Labs
AI Agent Evaluation Engineer - RemoteYO AI Labs • Ottawa, ON, Canada
AI Agent Evaluation Engineer - Remote

AI Agent Evaluation Engineer - Remote

YO AI Labs • Ottawa, ON, Canada
20 hours ago
Job type
  • Full-time
  • Remote
  • Quick Apply
Job description

Senior Software Engineer

Job Type: Contractor (~15 hours/week)
Location: Remote

Job Summary

We are seeking experienced Senior Software Engineers to support an AI training project by creating reinforcement learning environments that evaluate AI models on complex software engineering tasks using Model Context Protocol (MCP) tools.

You will design reproducible environments, deterministic verification, and reference solutions for tasks such as bug fixing, feature implementation, codebase refactoring, and performance optimization. No prior AI experience is required.

Key Responsibilities

  • Create reinforcement learning environments for software engineering tasks.
  • Design tasks involving bug fixing, feature development, refactoring, and performance optimization.
  • Build deterministic verification systems and golden reference solutions.
  • Evaluate AI agents' ability to reason through complex codebases and use MCP tools effectively.
  • Develop realistic, reproducible software engineering scenarios.
  • Ensure tasks accurately measure coding ability, problem-solving, and tool usage.
  • Document solutions and provide clear technical feedback.

Required Skills

  • Strong proficiency in Python 3, Java, Rust, C++, or TypeScript.
  • Strong understanding of algorithms and data structures.
  • Experience with bug fixing and debugging complex software issues.
  • Proven experience in feature implementation and codebase refactoring.
  • Strong knowledge of performance optimization and tuning.
  • Excellent written and verbal communication.
  • Strong attention to detail.

Preferred Qualifications

  • Experience working with large-scale or distributed codebases.
  • Familiarity with AI/ML systems is a plus but not required.
  • Experience with rigorous code reviews and software engineering best practices.
  • Experience working effectively in remote or cross-functional teams.

Hiring Process

  1. Submit an application and screening questions.
  2. Complete an AI interview (~30 minutes).
  3. Complete a technical assessment, if required.
  4. Hiring Manager review.

Compensation

Compensation is output-based, with payment provided per task that meets project specifications. Minimum weekly submission requirements may apply.

Availability

Selected experts should be prepared to begin their first tasks within 24–48 hours of completing onboarding.

Create a job alert for this search

AI Agent Evaluation Engineer - Remote • Ottawa, ON, Canada

Similar jobs

Remote Software Engineer – AI Trajectory Annotations

Weekday AI (YC W21)Ottawa, ON, CA
Remote
Part-time

A leading AI consultancy is seeking experienced Software Engineers to assist in creating training datasets for AI models.This part-time role, based in Canada, demands a minimum of 2 years' experien... Show more

 • Promoted

Remote Senior Solutions Engineer — Enterprise AI & Risk

GatekeeperOttawa, ON, CA
Remote
Full-time

A technology company is seeking an experienced Senior Solutions Engineer for a fully remote role in Canada.The selected candidate will be responsible for leading enterprise customer implementations... Show more

 • Promoted

Remote CS Expert & Data Partner for AI Q&A

TELUS Digital AI Data SolutionsOttawa, ON, CA
Remote
Full-time

A leading technology firm is seeking a Computer Science Expert to join as a Freelance Data Partner.This remote role involves developing complex question-and-answer pairs in your area of expertise a... Show more

 • Promoted

Remote AI Data Quality & Evaluation Engineer

Veeva SystemsOttawa, ON, CA
Remote
Full-time

A leading cloud technology company in Toronto is seeking an experienced professional to validate and ensure the reliability of Veeva AI Agents.The ideal candidate should have expertise in data qual... Show more

 • Promoted

AI Deployment Engineer for Public Sector

CohereOttawa, ON, CA
Full-time

Enable secure and efficient AI adoption in the public sector as an AI Deployment Engineer at Cohere.Work closely with clients to implement innovative solutions tailored to their operational needs.T... Show more

 • Promoted

Remote Chemistry Expert (LLM Evaluation) - 34879

TuringOttawa, ON, CA
Remote
Full-time

Remote Chemistry Expert (LLM Evaluation) - 34879.Remote Chemistry Expert (LLM Evaluation) - 34879.Be among the first 25 applicants.Get AI-powered advice on this job and more exclusive features.Turi... Show more

 • Promoted

AI Engineer

BioRenderOttawa, ON, CA
Full-time

At BioRender, we don’t settle for average, and neither should you.We’re building tools that transform how scientists communicate complex ideas, and we need someone who’s ready to turn ambitious ide... Show more

 • Promoted

Python & Kubernetes AI/ML Data Engineer — Remote/Distributed

CanonicalOttawa, ON, CA
Remote
Full-time

Canonical, a leader in open source software, seeks a Python and Kubernetes Software Engineer focused on Data, AI/ML, and Analytics Solutions.Collaborate on end-to-end data analytics solutions and c... Show more

 • Promoted

Remote Data Annotation Specialist (English & Text AI)

iMerit ScholarsOttawa, ON, CA
Remote
Full-time

A leading tech organization is seeking a meticulous Data Annotation Specialist to join their AI and ML projects team.This remote, full-time position involves generating and validating datasets acro... Show more

 • Promoted

Dynamic AI Agent Engineer Delivering Real-World Solutions

CrestaOttawa, ON, CA
Full-time

Be at the forefront of AI technology by developing innovative agents that drive business efficiency and enhance customer satisfaction.In this role, leverage your skills to implement AI solutions th... Show more

 • Promoted

UX Engineer: Firefly AI & Prompt Tech (Remote Canada)

Atimi SoftwareOttawa, ON, CA
Remote
Full-time

A software solutions company is seeking a UX Engineer to work fully remote from Canada.The ideal candidate will have extensive experience with Web Technologies and Graphics Design, and specific exp... Show more

 • Promoted

AI Solutions Engineer - Remote Opportunity

Yeah! GlobalOttawa, ON, CA
Remote
Full-time

Elevate your career as an AI Solutions Engineer with a remote role for a California-based client.Focus on designing and launching innovative AI initiatives that drive business success.We're buildin... Show more

 • Promoted

AI Engineer – Freelancer

TwineOttawa, ON, CA
Full-time

Join a project focused on advancing an AI‑powered influencer marketing platform by optimizing and scaling its data pipeline.The primary objective is to enhance the performance, cost‑efficiency, and... Show more

 • Promoted

RLHF Evaluation Specialist for AI Training

Rex.zoneOttawa, ON, CA
Full-time

A tech company specializing in AI is seeking an AI Trainer based in Canada.The role involves improving large language models through RLHF-style evaluations, prompt evaluation, and data labeling.Can... Show more

 • Promoted

Senior AI Engineer - Backend-Driven Agentic (Remote)

BayOne SolutionsOttawa, ON, CA
Remote
Full-time

A technology firm is hiring a Senior Full Stack Engineer (Backend-Leaning) with a strong focus on backend services and AI applications.The ideal candidate will have experience in building agent-bas... Show more

 • Promoted

Remote AI Software Engineer - Contractor, 10-40 hrs/wk

TuringOttawa, ON, CA
Remote
Full-time

A leading research accelerator is seeking a software engineering contractor based in Canada, Ontario, to evaluate and refine AI-generated code.This role requires over 5 years of experience, includi... Show more

 • Promoted

Remote AI Content Evaluator - Part-Time, Flexible Hours

Work VistaOttawa, ON, CA
Remote
Part-time

A global AI services provider is seeking a remote Content Evaluator to assess the quality of AI-generated content.This part-time role involves evaluating responses for fluency, coherence, and factu... Show more

 • Promoted

AI Developer

Hanalytica GmbHOttawa, ON, CA
Full-time

This range is provided by Hanalytica GmbH.Your actual pay will be based on your skills and experience — talk with your recruiter to learn more.Our client is looking for an exceptional.AI and machin... Show more

 • Promoted

Remote Chemistry Specialist — AI Content QA & Evaluation

Crossing HurdlesOttawa, ON, CA
Remote
Full-time

A recruitment firm is seeking a Chemistry Expert to review and verify AI-generated content and develop chemistry problems.This remote role requires an advanced degree and strong analytical skills.T... Show more

 • Promoted

Senior AI Engineer - Production ML for Fintech (Remote)

LumenaltaOttawa, ON, CA
Remote
Full-time

A forward-thinking technology firm is looking for experienced AI Engineers to design and deploy production-grade AI systems for community banks and insurance institutions.This role offers the chanc... Show more