Talent.com
YO AI Labs
AI Agent Evaluation Engineer - RemoteYO AI Labs • Toronto, ON, Canada
AI Agent Evaluation Engineer - Remote

AI Agent Evaluation Engineer - Remote

YO AI Labs • Toronto, ON, Canada
Il y a 10 heures
Type de contrat
  • Temps plein
  • Télétravail
  • Quick Apply
Description de poste

Senior Software Engineer

Job Type: Contractor (~15 hours/week)
Location: Remote

Job Summary

We are seeking experienced Senior Software Engineers to support an AI training project by creating reinforcement learning environments that evaluate AI models on complex software engineering tasks using Model Context Protocol (MCP) tools.

You will design reproducible environments, deterministic verification, and reference solutions for tasks such as bug fixing, feature implementation, codebase refactoring, and performance optimization. No prior AI experience is required.

Key Responsibilities

  • Create reinforcement learning environments for software engineering tasks.
  • Design tasks involving bug fixing, feature development, refactoring, and performance optimization.
  • Build deterministic verification systems and golden reference solutions.
  • Evaluate AI agents' ability to reason through complex codebases and use MCP tools effectively.
  • Develop realistic, reproducible software engineering scenarios.
  • Ensure tasks accurately measure coding ability, problem-solving, and tool usage.
  • Document solutions and provide clear technical feedback.

Required Skills

  • Strong proficiency in Python 3, Java, Rust, C++, or TypeScript.
  • Strong understanding of algorithms and data structures.
  • Experience with bug fixing and debugging complex software issues.
  • Proven experience in feature implementation and codebase refactoring.
  • Strong knowledge of performance optimization and tuning.
  • Excellent written and verbal communication.
  • Strong attention to detail.

Preferred Qualifications

  • Experience working with large-scale or distributed codebases.
  • Familiarity with AI/ML systems is a plus but not required.
  • Experience with rigorous code reviews and software engineering best practices.
  • Experience working effectively in remote or cross-functional teams.

Hiring Process

  1. Submit an application and screening questions.
  2. Complete an AI interview (~30 minutes).
  3. Complete a technical assessment, if required.
  4. Hiring Manager review.

Compensation

Compensation is output-based, with payment provided per task that meets project specifications. Minimum weekly submission requirements may apply.

Availability

Selected experts should be prepared to begin their first tasks within 24–48 hours of completing onboarding.

Créer une alerte emploi pour cette recherche

AI Agent Evaluation Engineer - Remote • Toronto, ON, Canada

Offres similaires

Principal AI Solutions Developer

OpenTextrichmond hill, york region, Canada
Temps plein

Become a Principal AI Solutions Developer at OpenText, leading the charge in creating sophisticated agentic AI systems.Use your expertise to enhance IT Operations security and observability.OpenTex... Voir plus

 • Offre sponsorisée

Agentic AI Engineer

NTT DATA North AmericaToronto, ON, CA
Temps plein

Specialist Advisor to join our team in Toronto, Ontario (CA-ON), Canada (CA).The AI Architect will design and build agentic AI systems for multiple domains such as Healthcare, Finance, Retail, etc.... Voir plus

 • Offre sponsorisée

AI Solutions Engineer - Remote Opportunity

Yeah! GlobalToronto, ON, CA
Télétravail
Temps plein

Elevate your career as an AI Solutions Engineer with a remote role for a California-based client.Focus on designing and launching innovative AI initiatives that drive business success.We're buildin... Voir plus

 • Offre sponsorisée

Remote AI Evaluation Scenario Designer

MindriftToronto, ON, CA
Télétravail
Temps plein

A leading AI solutions company is looking for an expert to design and evaluate test scenarios for AI agents.The ideal candidate will have a Bachelor's or Master's degree in a relevant field, experi... Voir plus

 • Offre sponsorisée

Remote AI Engineer - LLM Specialist

PulsoraToronto, ON, CA
Télétravail
Temps plein

Discover an exciting career as a Remote AI Engineer specializing in Large Language Models.Play a key role in developing AI-driven software solutions in a fully remote setting.Your expertise in AI/M... Voir plus

 • Offre sponsorisée

RLHF Evaluation Specialist for AI Training

Rex.zoneToronto, ON, CA
Temps plein

A tech company specializing in AI is seeking an AI Trainer based in Canada.The role involves improving large language models through RLHF-style evaluations, prompt evaluation, and data labeling.Can... Voir plus

 • Offre sponsorisée

Intern Researcher - AI Agent Evaluation

Huawei CanadaMarkham, York Region, CA
Temps plein

Huawei Canada has an immediate 4-12 month internship opening for an Intern Researcher.Established in 2014, the Distributed Scheduling and Data Engine Lab is Huawei Cloud's technical innovation cent... Voir plus

 • Offre sponsorisée

Software Engineer Advancing Evaluation Infrastructure for AI Innovation

Waabitoronto, on, Canada
Temps plein

Join a collaborative team as a Software Engineer enhancing the evaluation infrastructure for AI-driven vehicles.Be at the forefront of autonomous technology development in a flexible work environme... Voir plus

 • Offre sponsorisée

AI Engineer – Freelancer

TwineToronto, ON, CA
Temps plein

Join a project focused on advancing an AI‑powered influencer marketing platform by optimizing and scaling its data pipeline.The primary objective is to enhance the performance, cost‑efficiency, and... Voir plus

 • Offre sponsorisée

DevOps Engineer - AI Model Evaluator

MercorToronto, Ontario, Canada
85,00 $CA par heure
Télétravail
Temps partiel
Quick Apply

Headquartered in San Francisco, our investors include.DevOps / SRE / Cloud Engineer (Coding Agent Experience).Review model-generated implementations involving.Identify bugs, edge cases, reliability... Voir plus

Senior Agentic Ai Prototyping Engineer – Remote - $100,000 - $130,000 A Year - Remote

Software InternationalToronto, Canada
Télétravail
Temps plein

A global IT services company is seeking a Senior Software Developer to lead experimental AI projects.The ideal candidate will have over 5 years of experience in software development and demonstrate... Voir plus

 • Offre sponsorisée

Forward Deployed AI Engineer

EQ Banktoronto, on, Canada
Temps plein

We are looking for a Staff-level Forward Deployed AI Engineer to design, build, and deliver AI-powered applications that create measurable business impact.This is a hands‑on engineering role with s... Voir plus

 • Offre sponsorisée

AI Agent Engineer: LLMs, RAG & Prompt Mastery

Linkus GroupToronto, ON, CA
Temps plein

A technology company in Toronto is seeking an experienced software engineer to develop next-gen AI tools that enhance critical thinking.The ideal candidate will fine-tune language models and optimi... Voir plus

 • Offre sponsorisée

Remote Ai Data Quality & Evaluation Engineer - C$85,000 - C$225,000 A Year - Remote

Cloud Technology CompanyNorth York, Canada
Télétravail
Temps plein

AI Data Quality & Evaluation Engineer needed to validate and ensure the reliability of AI Agents using expertise in data quality, prompt engineering, and Python. Voir plus

 • Offre sponsorisée

AI Engineer – Generative AI Expertise

Synechrontoronto, on, Canada
Temps plein

Join Synechron as an AI Engineer specializing in Generative AI.Leverage your advanced skills in machine learning to innovate solutions for capital markets.As a vital contributor, you will focus on ... Voir plus

 • Offre sponsorisée

AI Engineer

BioRenderToronto, ON, CA
Temps plein

At BioRender, we don’t settle for average, and neither should you.We’re building tools that transform how scientists communicate complex ideas, and we need someone who’s ready to turn ambitious ide... Voir plus

 • Offre sponsorisée

Innovative AI Engineer – Pulsora Canada

PulsoraToronto, ON, CA
Temps plein

Leverage your AI expertise at Pulsora as a Remote Engineer focused on developing impactful AI applications.This full-time role prioritizes cutting-edge technology for sustainability efforts.As part... Voir plus

 • Offre sponsorisée

Remote AI Data Quality & Evaluation Engineer

Veeva Systemstoronto, on, Canada
Télétravail
Temps plein

A leading cloud technology company in Toronto is seeking an experienced professional to validate and ensure the reliability of Veeva AI Agents.The ideal candidate should have expertise in data qual... Voir plus

 • Offre sponsorisée

AI Engineer for Reliability Solutions

Mosaic.techToronto, ON, CA
Temps plein

Join Rootly as an AI Engineer specializing in developing reliable AI-driven solutions for incident management.Use your LLM experience to shape the future of business reliability.As a Senior AI Engi... Voir plus

 • Offre sponsorisée

AI Agent Engineer — Remote, Equity

CrestaToronto, Ontario, Canada
Télétravail
Temps plein

A leading AI technology company in Toronto is seeking a Machine Learning Engineer to develop and deploy cutting-edge AI solutions.This role involves hands-on work with innovative technologies, focu... Voir plus