Talent.com
YO AI Labs
AI Agent Evaluation Engineer - RemoteYO AI Labs • Toronto, ON, Canada
AI Agent Evaluation Engineer - Remote

AI Agent Evaluation Engineer - Remote

YO AI Labs • Toronto, ON, Canada
7 hours ago
Job type
  • Full-time
  • Remote
  • Quick Apply
Job description

Senior Software Engineer

Job Type: Contractor (~15 hours/week)
Location: Remote

Job Summary

We are seeking experienced Senior Software Engineers to support an AI training project by creating reinforcement learning environments that evaluate AI models on complex software engineering tasks using Model Context Protocol (MCP) tools.

You will design reproducible environments, deterministic verification, and reference solutions for tasks such as bug fixing, feature implementation, codebase refactoring, and performance optimization. No prior AI experience is required.

Key Responsibilities

  • Create reinforcement learning environments for software engineering tasks.
  • Design tasks involving bug fixing, feature development, refactoring, and performance optimization.
  • Build deterministic verification systems and golden reference solutions.
  • Evaluate AI agents' ability to reason through complex codebases and use MCP tools effectively.
  • Develop realistic, reproducible software engineering scenarios.
  • Ensure tasks accurately measure coding ability, problem-solving, and tool usage.
  • Document solutions and provide clear technical feedback.

Required Skills

  • Strong proficiency in Python 3, Java, Rust, C++, or TypeScript.
  • Strong understanding of algorithms and data structures.
  • Experience with bug fixing and debugging complex software issues.
  • Proven experience in feature implementation and codebase refactoring.
  • Strong knowledge of performance optimization and tuning.
  • Excellent written and verbal communication.
  • Strong attention to detail.

Preferred Qualifications

  • Experience working with large-scale or distributed codebases.
  • Familiarity with AI/ML systems is a plus but not required.
  • Experience with rigorous code reviews and software engineering best practices.
  • Experience working effectively in remote or cross-functional teams.

Hiring Process

  1. Submit an application and screening questions.
  2. Complete an AI interview (~30 minutes).
  3. Complete a technical assessment, if required.
  4. Hiring Manager review.

Compensation

Compensation is output-based, with payment provided per task that meets project specifications. Minimum weekly submission requirements may apply.

Availability

Selected experts should be prepared to begin their first tasks within 24–48 hours of completing onboarding.

Create a job alert for this search

AI Agent Evaluation Engineer - Remote • Toronto, ON, Canada

Similar jobs

Principal AI Solutions Developer

OpenTextrichmond hill, york region, Canada
Full-time

Become a Principal AI Solutions Developer at OpenText, leading the charge in creating sophisticated agentic AI systems.Use your expertise to enhance IT Operations security and observability.OpenTex... Show more

 • Promoted

Senior Agentic Ai Prototyping Engineer – Remote - $100,000 - $130,000 A Year - Remote

Software InternationalToronto County, Canada
Remote
Full-time

Senior engineer to lead AI projects, prototype concepts, and integrate AI solutions into workflows. Show more

 • Promoted

Agentic AI Engineer

NTT DATA North AmericaToronto, ON, CA
Full-time

Specialist Advisor to join our team in Toronto, Ontario (CA-ON), Canada (CA).The AI Architect will design and build agentic AI systems for multiple domains such as Healthcare, Finance, Retail, etc.... Show more

 • Promoted

AI Solutions Engineer - Remote Opportunity

Yeah! GlobalToronto, ON, CA
Remote
Full-time

Elevate your career as an AI Solutions Engineer with a remote role for a California-based client.Focus on designing and launching innovative AI initiatives that drive business success.We're buildin... Show more

 • Promoted

Remote AI Evaluation Scenario Designer

MindriftToronto, ON, CA
Remote
Full-time

A leading AI solutions company is looking for an expert to design and evaluate test scenarios for AI agents.The ideal candidate will have a Bachelor's or Master's degree in a relevant field, experi... Show more

 • Promoted

AI Agent Engineer — Remote, Equity

CrestaToronto, ON, CA
Remote
Full-time

A leading AI technology company in Toronto is seeking a Machine Learning Engineer to develop and deploy cutting-edge AI solutions.This role involves hands-on work with innovative technologies, focu... Show more

 • Promoted

Remote AI Engineer - LLM Specialist

PulsoraToronto, ON, CA
Remote
Full-time

Discover an exciting career as a Remote AI Engineer specializing in Large Language Models.Play a key role in developing AI-driven software solutions in a fully remote setting.Your expertise in AI/M... Show more

 • Promoted

AI Evaluation Engineer

ZafinToronto, ON, CA
Full-time

Zafin is an AI platform company helping regulated institutions modernize how critical work is designed, governed, and delivered.Our technology enables organizations to move faster while maintaining... Show more

 • Promoted

Intern Researcher - AI Agent Evaluation

Huawei CanadaMarkham, York Region, CA
Full-time

Huawei Canada has an immediate 4-12 month internship opening for an Intern Researcher.Established in 2014, the Distributed Scheduling and Data Engine Lab is Huawei Cloud's technical innovation cent... Show more

 • Promoted

Software Engineer Advancing Evaluation Infrastructure for AI Innovation

Waabitoronto, on, Canada
Full-time

Join a collaborative team as a Software Engineer enhancing the evaluation infrastructure for AI-driven vehicles.Be at the forefront of autonomous technology development in a flexible work environme... Show more

 • Promoted

DevOps Engineer - AI Model Evaluator

MercorToronto, Ontario, Canada
CA$85.00 hourly
Remote
Part-time
Quick Apply

Headquartered in San Francisco, our investors include.DevOps / SRE / Cloud Engineer (Coding Agent Experience).Review model-generated implementations involving.Identify bugs, edge cases, reliability... Show more

Forward Deployed AI Engineer

EQ Banktoronto, on, Canada
Full-time

We are looking for a Staff-level Forward Deployed AI Engineer to design, build, and deliver AI-powered applications that create measurable business impact.This is a hands‑on engineering role with s... Show more

 • Promoted

AI Agent Engineer: LLMs, RAG & Prompt Mastery

Linkus GroupToronto, ON, CA
Full-time

A technology company in Toronto is seeking an experienced software engineer to develop next-gen AI tools that enhance critical thinking.The ideal candidate will fine-tune language models and optimi... Show more

 • Promoted

Demo Engineer - AI [32469]

Stealth AI Startuptoronto, on, Canada
Full-time

Get AI-powered advice on this job and more exclusive features.This range is provided by Stealth AI Startup.Your actual pay will be based on your skills and experience — talk with your recruiter to ... Show more

 • Promoted

Remote Ai Data Quality & Evaluation Engineer - C$85,000 - C$225,000 A Year - Remote

Cloud Technology CompanyToronto, Canada
Remote
Full-time

A leading cloud technology company in Toronto is seeking an experienced professional to validate and ensure the reliability of Veeva AI Agents.The ideal candidate should have expertise in data qual... Show more

 • Promoted

AI Engineer – Generative AI Expertise

Synechrontoronto, on, Canada
Full-time

Join Synechron as an AI Engineer specializing in Generative AI.Leverage your advanced skills in machine learning to innovate solutions for capital markets.As a vital contributor, you will focus on ... Show more

 • Promoted

AI Engineer

BioRenderToronto, ON, CA
Full-time

At BioRender, we don’t settle for average, and neither should you.We’re building tools that transform how scientists communicate complex ideas, and we need someone who’s ready to turn ambitious ide... Show more

 • Promoted

Innovative AI Engineer – Pulsora Canada

PulsoraToronto, ON, CA
Full-time

Leverage your AI expertise at Pulsora as a Remote Engineer focused on developing impactful AI applications.This full-time role prioritizes cutting-edge technology for sustainability efforts.As part... Show more

 • Promoted

Remote AI Data Quality & Evaluation Engineer

Veeva Systemstoronto, on, Canada
Remote
Full-time

A leading cloud technology company in Toronto is seeking an experienced professional to validate and ensure the reliability of Veeva AI Agents.The ideal candidate should have expertise in data qual... Show more

 • Promoted

AI Engineer for Reliability Solutions

Mosaic.techtoronto, on, Canada
Full-time

Join Rootly as an AI Engineer specializing in developing reliable AI-driven solutions for incident management.Use your LLM experience to shape the future of business reliability.As a Senior AI Engi... Show more