Talent.com
YO AI Labs
AI Agent Evaluation Engineer - RemoteYO AI Labs • Vancouver, BC, Canada
AI Agent Evaluation Engineer - Remote

AI Agent Evaluation Engineer - Remote

YO AI Labs • Vancouver, BC, Canada
2 hours ago
Job type
  • Full-time
  • Remote
  • Quick Apply
Job description
Senior Software Engineer

Job Type: Contractor (~15 hours/week)
Location: Remote

Job Summary

We are seeking experienced Senior Software Engineers to support an AI training project by creating reinforcement learning environments that evaluate AI models on complex software engineering tasks using Model Context Protocol (MCP) tools.

You will design reproducible environments, deterministic verification, and reference solutions for tasks such as bug fixing, feature implementation, codebase refactoring, and performance optimization. No prior AI experience is required.

Key Responsibilities

  • Create reinforcement learning environments for software engineering tasks.
  • Design tasks involving bug fixing, feature development, refactoring, and performance optimization.
  • Build deterministic verification systems and golden reference solutions.
  • Evaluate AI agents' ability to reason through complex codebases and use MCP tools effectively.
  • Develop realistic, reproducible software engineering scenarios.
  • Ensure tasks accurately measure coding ability, problem-solving, and tool usage.
  • Document solutions and provide clear technical feedback.

Required Skills

  • Strong proficiency in Python 3, Java, Rust, C++, or TypeScript.
  • Strong understanding of algorithms and data structures.
  • Experience with bug fixing and debugging complex software issues.
  • Proven experience in feature implementation and codebase refactoring.
  • Strong knowledge of performance optimization and tuning.
  • Excellent written and verbal communication.
  • Strong attention to detail.

Preferred Qualifications

  • Experience working with large-scale or distributed codebases.
  • Familiarity with AI/ML systems is a plus but not required.
  • Experience with rigorous code reviews and software engineering best practices.
  • Experience working effectively in remote or cross-functional teams.

Hiring Process

  1. Submit an application and screening questions.
  2. Complete an AI interview (~30 minutes).
  3. Complete a technical assessment, if required.
  4. Hiring Manager review.

Compensation

Compensation is output-based, with payment provided per task that meets project specifications. Minimum weekly submission requirements may apply.

Availability

Selected experts should be prepared to begin their first tasks within 24–48 hours of completing onboarding.

Create a job alert for this search

AI Agent Evaluation Engineer - Remote • Vancouver, BC, Canada

Similar jobs

AI Agent Engineer – Commercial AI Transformation

Diligent-14787b60Vancouver, British Columbia, Canada
Full-time

Overview Diligent's Commercial AI Transformation function builds AI agents that automate real workflows across our commercial organization, from sales and customer success to internal operations.We... Show more

 • Promoted

Applied Ai Engineer — Hybrid & Remote-Ready - $90,000 - $110,000 A Year

Building Controls ManufacturerSurrey, Canada
Remote
Full-time

A leading building controls manufacturer in Surrey is seeking an Applied AI Developer to design, develop, and improve AI tools for various business functions.This role emphasizes practical AI appli... Show more

 • Promoted

Remote Software Engineer – AI Trajectory Annotations

Weekday AI (YC W21)Vancouver, Metro Vancouver Regional District, CA
Remote
Part-time

A leading AI consultancy is seeking experienced Software Engineers to assist in creating training datasets for AI models.This part-time role, based in Canada, demands a minimum of 2 years' experien... Show more

 • Promoted

Senior AI Engineer, AI Inference Systems

BrixPacific, British Columbia, Canada
Full-time

Senior AI Engineer, AI Inference Systems.Senior AI Engineer, AI Inference Systems.Oversee and optimize the LLM inference engine to ensure performance, scalability, and cost-efficiency.Collaborate w... Show more

 • Promoted

AI Engineer

BioRenderVancouver, Metro Vancouver Regional District, CA
Full-time

At BioRender, we don’t settle for average, and neither should you.We’re building tools that transform how scientists communicate complex ideas, and we need someone who’s ready to turn ambitious ide... Show more

 • Promoted

AI Engineer: Build Innovative AI Solutions

Durable Technologies IncVancouver
Full-time

A tech company focused on AI is seeking full stack engineers to build and enhance customer-facing features.This position requires expertise in React, NextJS, and TypeScript, along with experience i... Show more

 • Promoted

Ai Enablement Advocate & Developer Evangelist (Hybrid) - C$93,000 - C$141,000 A Year

Professional Services FirmRichmond, Canada
Full-time

Developer Advocate to foster AI adoption, promote responsible AI, and build communities.Role involves training, mentoring, and ensuring compliance. Show more

 • Promoted

Ml Vision Engineer — Edge & Cloud Ai - C$84,300 - C$120,000 A Year

A leading mining technology firmRichmond, Canada
Full-time

Machine Learning Developer sought for innovative AI projects, designing algorithms for embedded, edge, and cloud environments. Show more

 • Promoted

Conversational Ai Engineer - C$95,000 - C$120,000 A Year

Nexlify SolutionsPort Moody, Canada
Full-time

Designs and builds intelligent chatbots and LLM-powered virtual assistants using NLP and machine learning to automate support, drive engagement, and deliver personalized experiences.Collaborates wi... Show more

 • Promoted

RLHF Evaluation Specialist for AI Training

Rex.zoneVancouver, Metro Vancouver Regional District, CA
Full-time

A tech company specializing in AI is seeking an AI Trainer based in Canada.The role involves improving large language models through RLHF-style evaluations, prompt evaluation, and data labeling.Can... Show more

 • Promoted

EA Research Engineer in Game AI and LLMs

Electronic Arts Inc.Vancouver, Metro Vancouver Regional District, CA
Full-time

Elevate interactive entertainment as a Research Engineer focusing on Large Language Models at Electronic Arts.This impactful role allows you to pioneer groundbreaking applications for gaming techno... Show more

 • Promoted

Remote AI Software Engineer - Contractor, 10-40 hrs/wk

TuringVancouver, Metro Vancouver Regional District, CA
Remote
Full-time

A leading research accelerator is seeking a software engineering contractor based in Canada, Ontario, to evaluate and refine AI-generated code.This role requires over 5 years of experience, includi... Show more

 • Promoted

Remote Senior Solutions Engineer — Enterprise AI & Risk

GatekeeperVancouver, Metro Vancouver Regional District, CA
Remote
Full-time

A technology company is seeking an experienced Senior Solutions Engineer for a fully remote role in Canada.The selected candidate will be responsible for leading enterprise customer implementations... Show more

 • Promoted

Senior Ai Engineer — Remote, Build Scalable Ai Features - C$122,000 - C$162,000 A Year - Remote

Procurement Solutions ProviderNorth Vancouver, Canada
Remote
Full-time

Seeking a Senior AI Engineer for a remote-first company to develop scalable AI applications, requiring strong AI/ML and backend engineering skills. Show more

 • Promoted

Ai Engineer: Build Innovative Ai Solutions - C$180,000 - C$225,000 A Year

Tech companyRichmond, Canada
Full-time

Seeking an AI Engineer to develop and improve customer-facing AI-powered features, integrating AI systems and ensuring a polished UI.Requires expertise in React, NextJS, and TypeScript. Show more

 • Promoted

Remote Js Integration Engineer - Customer Data - $80,000 - $120,000 A Year - Remote

ConstructorRichmond, Canada
Remote
Full-time

Remote Integration Engineer to enhance e-commerce with focus on customer data.Responsibilities include developing monitoring tools, supporting onboarding, and resolving integration challenges. Show more

 • Promoted

AI Engineer – Freelancer

TwineVancouver, Metro Vancouver Regional District, CA
Full-time

Join a project focused on advancing an AI‑powered influencer marketing platform by optimizing and scaling its data pipeline.The primary objective is to enhance the performance, cost‑efficiency, and... Show more

 • Promoted

Senior AI Engineer - Backend-Driven Agentic (Remote)

BayOne SolutionsVancouver, Metro Vancouver Regional District, CA
Remote
Full-time

A technology firm is hiring a Senior Full Stack Engineer (Backend-Leaning) with a strong focus on backend services and AI applications.The ideal candidate will have experience in building agent-bas... Show more

 • Promoted

Remote Chemistry Specialist — AI Content QA & Evaluation

Crossing HurdlesVancouver, Metro Vancouver Regional District, CA
Remote
Full-time

A recruitment firm is seeking a Chemistry Expert to review and verify AI-generated content and develop chemistry problems.This remote role requires an advanced degree and strong analytical skills.T... Show more

 • Promoted

Ai-Native Full Stack Engineer - C$115,000 - C$150,000 A Year - Remote

Orion Digital CorpRichmond, Canada
Remote
Full-time

Seeking a Full Stack Engineer to transform a fintech company into an AI-native system by redesigning legacy workflows, architecting LLM-backed services, and building AI agents within a regulated pr... Show more