Talent.com
YO AI Labs
AI Agent Evaluation Engineer - RemoteYO AI Labs • Montréal, QC, Canada
AI Agent Evaluation Engineer - Remote

AI Agent Evaluation Engineer - Remote

YO AI Labs • Montréal, QC, Canada
4 days ago
Job type
  • Full-time
  • Remote
  • Quick Apply
Job description

Senior Software Engineer

Job Type: Contractor (~15 hours/week)
Location: Remote

Job Summary

We are seeking experienced Senior Software Engineers to support an AI training project by creating reinforcement learning environments that evaluate AI models on complex software engineering tasks using Model Context Protocol (MCP) tools.

You will design reproducible environments, deterministic verification, and reference solutions for tasks such as bug fixing, feature implementation, codebase refactoring, and performance optimization. No prior AI experience is required.

Key Responsibilities

  • Create reinforcement learning environments for software engineering tasks.
  • Design tasks involving bug fixing, feature development, refactoring, and performance optimization.
  • Build deterministic verification systems and golden reference solutions.
  • Evaluate AI agents' ability to reason through complex codebases and use MCP tools effectively.
  • Develop realistic, reproducible software engineering scenarios.
  • Ensure tasks accurately measure coding ability, problem-solving, and tool usage.
  • Document solutions and provide clear technical feedback.

Required Skills

  • Strong proficiency in Python 3, Java, Rust, C++, or TypeScript.
  • Strong understanding of algorithms and data structures.
  • Experience with bug fixing and debugging complex software issues.
  • Proven experience in feature implementation and codebase refactoring.
  • Strong knowledge of performance optimization and tuning.
  • Excellent written and verbal communication.
  • Strong attention to detail.

Preferred Qualifications

  • Experience working with large-scale or distributed codebases.
  • Familiarity with AI/ML systems is a plus but not required.
  • Experience with rigorous code reviews and software engineering best practices.
  • Experience working effectively in remote or cross-functional teams.

Hiring Process

  1. Submit an application and screening questions.
  2. Complete an AI interview (~30 minutes).
  3. Complete a technical assessment, if required.
  4. Hiring Manager review.

Compensation

Compensation is output-based, with payment provided per task that meets project specifications. Minimum weekly submission requirements may apply.

Availability

Selected experts should be prepared to begin their first tasks within 24–48 hours of completing onboarding.

Create a job alert for this search

AI Agent Evaluation Engineer - Remote • Montréal, QC, Canada

Similar jobs

Remote AI Data Quality & Evaluation Engineer

Veeva SystemsMontreal (administrative region), QC, CA
Remote
Full-time

A leading cloud technology company in Toronto is seeking an experienced professional to validate and ensure the reliability of Veeva AI Agents.The ideal candidate should have expertise in data qual... Show more

 • Promoted

Generative AI Engineer

AperteraMontreal (administrative region), QC, CA
Full-time

We are looking for a Generative AI Engineer to develop our next-generation intelligent translation and translation-related service engine, using Generative AI (GenAI) and Large Language Model (LLM)... Show more

 • Promoted

AI Engineer

BioRenderMontreal (administrative region), QC, CA
Full-time

At BioRender, we don’t settle for average, and neither should you.We’re building tools that transform how scientists communicate complex ideas, and we need someone who’s ready to turn ambitious ide... Show more

 • Promoted

Remote Software Engineer — Train & Evaluate AI Chatbot Code

DataAnnotationMontreal (administrative region), QC, CA
Remote
Full-time

An innovative company is seeking a skilled Software Engineer to help train AI chatbots.This exciting remote role allows you to work on your own schedule while contributing to the development of cut... Show more

 • Promoted

Senior Applied AI Engineer

Giesecke+Devrientmontreal (administrative region), qc, Canada
Permanent

Giesecke+Devrient is a globally leading SecurityTech company.We are seeking a highly technical and execution-focused Senior Applied AI Engineer to join G+D’s new AI Hub.The ideal candidate will com... Show more

 • Promoted

AI Engineer – Freelancer

TwineMontreal (administrative region), QC, CA
Full-time

Join a project focused on advancing an AI‑powered influencer marketing platform by optimizing and scaling its data pipeline.The primary objective is to enhance the performance, cost‑efficiency, and... Show more

 • Promoted

Remote R&D Biologist: AI Model Evaluation & Design

DataAnnotationMontreal (administrative region), QC, CA
Remote
Full-time

An innovative company is seeking a Research and Development Biologist to enhance AI models through rigorous evaluation and problem-solving.This exciting remote role offers flexibility in project se... Show more

 • Promoted

Remote AI Physics Trainer — Train & Evaluate AI Models

DataAnnotationMontreal (administrative region), QC, CA
Remote
Full-time

An innovative company is seeking a physics expert to train AI models in a flexible remote role.You will engage with AI chatbots by providing complex physics problems and assessing their logic and p... Show more

 • Promoted

Applied AI Research Engineer (LLMs & Agentic Systems)

IC ResourcesMontreal, Montreal (administrative region), CA
Full-time

A growing AI company in Montreal is seeking a Research Engineer to bridge research and engineering.The role involves prototyping AI methods, evaluating optimization approaches, and writing producti... Show more

 • Promoted

Applied AI Engineer

Compunnel, Inc.Montreal (administrative region), QC, CA
Full-time

Client Fixed Income Institutional Lending Technology team is seeking two highly experienced Applied AI Engineers to design, build, and operate an enterprise-grade GenAI workflow platform.This platf... Show more

 • Promoted

Remote AI Content Evaluator - Part-Time, Flexible Hours

Work VistaMontreal (administrative region), QC, CA
Remote
Part-time

A global AI services provider is seeking a remote Content Evaluator to assess the quality of AI-generated content.This part-time role involves evaluating responses for fluency, coherence, and factu... Show more

 • Promoted

AI Solutions Engineer - Remote Opportunity

Yeah! GlobalMontreal (administrative region), QC, CA
Remote
Full-time

Elevate your career as an AI Solutions Engineer with a remote role for a California-based client.Focus on designing and launching innovative AI initiatives that drive business success.We're buildin... Show more

 • Promoted

AI Engineer - Canada (Remote)

Yeah! GlobalMontreal (administrative region), QC, CA
Remote
Full-time

About the job AI Engineer - Canada (Remote).One of our clients based in California, USA is building a new team and looking for 7 highly skilled and motivated AI Engineers globally to join their dyn... Show more

 • Promoted

RLHF Evaluation Specialist for AI Training

Rex.zoneMontreal (administrative region), QC, CA
Full-time

A tech company specializing in AI is seeking an AI Trainer based in Canada.The role involves improving large language models through RLHF-style evaluations, prompt evaluation, and data labeling.Can... Show more

 • Promoted

Content Evaluator (Remote)

Work VistaMontreal (administrative region), QC, CA
Remote
Full-time

United States, United Kingdom, Canada, Ireland, Australia & New Zealand.You must be legally authorised to work in the country where you are based).We are hiring on behalf of one of our clients, a l... Show more

 • Promoted

Dynamic AI Agent Engineer Delivering Real-World Solutions

CrestaMontreal (administrative region), QC, CA
Full-time

Be at the forefront of AI technology by developing innovative agents that drive business efficiency and enhance customer satisfaction.In this role, leverage your skills to implement AI solutions th... Show more

 • Promoted

Co-op Researcher - Embodied AI Agent (Robotic)

Huawei CanadaMontreal, Montreal (administrative region), CA
Full-time

Co-op Researcher - Embodied AI Agent (Robotic).Co-op Researcher - Embodied AI Agent (Robotic).Huawei Canada has an immediate Co-op opening for a Researcher.Founded in 2012, the Noah’s Ark lab has e... Show more

 • Promoted

Remote AI Evaluation Scenario Designer

MindriftMontreal (administrative region), QC, CA
Remote
Full-time

A leading AI solutions company is looking for an expert to design and evaluate test scenarios for AI agents.The ideal candidate will have a Bachelor's or Master's degree in a relevant field, experi... Show more

 • Promoted

AI Agent Builder for Real Impact

Valsoft Corporationmontreal (administrative region), qc, Canada
Full-time

Shape the future with Manos Software as an AI Agent Builder, where your work directly influences the success of various portfolio companies.This role is centered around practical applications of AI... Show more

 • Promoted

Staff Research Engineer: Accelerate LLM Inference (Remote)

CohereMontreal, Montreal (administrative region), CA
Remote
Full-time

A leading AI research firm in Montreal is seeking a Staff Research Engineer to enhance model efficiency and optimize inference for large language models.In this full-time role, you will develop tec... Show more