Talent.com
YO AI Labs
AI Agent Evaluation Engineer - RemoteYO AI Labs • Vancouver, BC, Canada
AI Agent Evaluation Engineer - Remote

AI Agent Evaluation Engineer - Remote

YO AI Labs • Vancouver, BC, Canada
Il y a 3 jours
Type de contrat
  • Temps plein
  • Télétravail
  • Quick Apply
Description de poste

Senior Software Engineer

Job Type: Contractor (~15 hours/week)
Location: Remote

Job Summary

We are seeking experienced Senior Software Engineers to support an AI training project by creating reinforcement learning environments that evaluate AI models on complex software engineering tasks using Model Context Protocol (MCP) tools.

You will design reproducible environments, deterministic verification, and reference solutions for tasks such as bug fixing, feature implementation, codebase refactoring, and performance optimization. No prior AI experience is required.

Key Responsibilities

  • Create reinforcement learning environments for software engineering tasks.
  • Design tasks involving bug fixing, feature development, refactoring, and performance optimization.
  • Build deterministic verification systems and golden reference solutions.
  • Evaluate AI agents' ability to reason through complex codebases and use MCP tools effectively.
  • Develop realistic, reproducible software engineering scenarios.
  • Ensure tasks accurately measure coding ability, problem-solving, and tool usage.
  • Document solutions and provide clear technical feedback.

Required Skills

  • Strong proficiency in Python 3, Java, Rust, C++, or TypeScript.
  • Strong understanding of algorithms and data structures.
  • Experience with bug fixing and debugging complex software issues.
  • Proven experience in feature implementation and codebase refactoring.
  • Strong knowledge of performance optimization and tuning.
  • Excellent written and verbal communication.
  • Strong attention to detail.

Preferred Qualifications

  • Experience working with large-scale or distributed codebases.
  • Familiarity with AI/ML systems is a plus but not required.
  • Experience with rigorous code reviews and software engineering best practices.
  • Experience working effectively in remote or cross-functional teams.

Hiring Process

  1. Submit an application and screening questions.
  2. Complete an AI interview (~30 minutes).
  3. Complete a technical assessment, if required.
  4. Hiring Manager review.

Compensation

Compensation is output-based, with payment provided per task that meets project specifications. Minimum weekly submission requirements may apply.

Availability

Selected experts should be prepared to begin their first tasks within 24–48 hours of completing onboarding.

Créer une alerte emploi pour cette recherche

AI Agent Evaluation Engineer - Remote • Vancouver, BC, Canada

Offres similaires

Remote Chemistry Expert (LLM Evaluation) - 34879

TuringVancouver, Metro Vancouver Regional District, CA
Télétravail
Temps plein

Remote Chemistry Expert (LLM Evaluation) - 34879.Remote Chemistry Expert (LLM Evaluation) - 34879.Be among the first 25 applicants.Get AI-powered advice on this job and more exclusive features.Turi... Voir plus

 • Offre sponsorisée

Dynamic AI Agent Engineer Delivering Real-World Solutions

CrestaVancouver, Metro Vancouver Regional District, CA
Temps plein

Be at the forefront of AI technology by developing innovative agents that drive business efficiency and enhance customer satisfaction.In this role, leverage your skills to implement AI solutions th... Voir plus

 • Offre sponsorisée

Remote Software Engineer – AI Trajectory Annotations

Weekday AI (YC W21)Vancouver, Metro Vancouver Regional District, CA
Télétravail
Temps partiel

A leading AI consultancy is seeking experienced Software Engineers to assist in creating training datasets for AI models.This part-time role, based in Canada, demands a minimum of 2 years' experien... Voir plus

 • Offre sponsorisée

Remote AI Evaluation Scenario Designer

MindriftVancouver, Metro Vancouver Regional District, CA
Télétravail
Temps plein

A leading AI solutions company is looking for an expert to design and evaluate test scenarios for AI agents.The ideal candidate will have a Bachelor's or Master's degree in a relevant field, experi... Voir plus

 • Offre sponsorisée

AI Engineer

BioRenderVancouver, Metro Vancouver Regional District, CA
Temps plein

At BioRender, we don’t settle for average, and neither should you.We’re building tools that transform how scientists communicate complex ideas, and we need someone who’s ready to turn ambitious ide... Voir plus

 • Offre sponsorisée

Remote AI Content Evaluator - Part-Time, Flexible Hours

Work VistaVancouver, Metro Vancouver Regional District, CA
Télétravail
Temps partiel

A global AI services provider is seeking a remote Content Evaluator to assess the quality of AI-generated content.This part-time role involves evaluating responses for fluency, coherence, and factu... Voir plus

 • Offre sponsorisée

AI Developer

Hanalytica GmbHVancouver, Metro Vancouver Regional District, CA
Temps plein

This range is provided by Hanalytica GmbH.Your actual pay will be based on your skills and experience — talk with your recruiter to learn more.Our client is looking for an exceptional.AI and machin... Voir plus

 • Offre sponsorisée

RLHF Evaluation Specialist for AI Training

Rex.zoneVancouver, Metro Vancouver Regional District, CA
Temps plein

A tech company specializing in AI is seeking an AI Trainer based in Canada.The role involves improving large language models through RLHF-style evaluations, prompt evaluation, and data labeling.Can... Voir plus

 • Offre sponsorisée

EA Research Engineer in Game AI and LLMs

Electronic Arts Inc.Vancouver, Metro Vancouver Regional District, CA
Temps plein

Elevate interactive entertainment as a Research Engineer focusing on Large Language Models at Electronic Arts.This impactful role allows you to pioneer groundbreaking applications for gaming techno... Voir plus

 • Offre sponsorisée

UX Engineer: Firefly AI & Prompt Tech (Remote Canada)

Atimi SoftwareVancouver, Metro Vancouver Regional District, CA
Télétravail
Temps plein

A software solutions company is seeking a UX Engineer to work fully remote from Canada.The ideal candidate will have extensive experience with Web Technologies and Graphics Design, and specific exp... Voir plus

 • Offre sponsorisée

Remote AI Software Engineer - Contractor, 10-40 hrs/wk

TuringVancouver, Metro Vancouver Regional District, CA
Télétravail
Temps plein

A leading research accelerator is seeking a software engineering contractor based in Canada, Ontario, to evaluate and refine AI-generated code.This role requires over 5 years of experience, includi... Voir plus

 • Offre sponsorisée

Remote Senior Solutions Engineer — Enterprise AI & Risk

GatekeeperVancouver, Metro Vancouver Regional District, CA
Télétravail
Temps plein

A technology company is seeking an experienced Senior Solutions Engineer for a fully remote role in Canada.The selected candidate will be responsible for leading enterprise customer implementations... Voir plus

 • Offre sponsorisée

AI Engineer – Freelancer

TwineVancouver, Metro Vancouver Regional District, CA
Temps plein

Join a project focused on advancing an AI‑powered influencer marketing platform by optimizing and scaling its data pipeline.The primary objective is to enhance the performance, cost‑efficiency, and... Voir plus

 • Offre sponsorisée

Senior AI Engineer - Backend-Driven Agentic (Remote)

BayOne SolutionsVancouver, Metro Vancouver Regional District, CA
Télétravail
Temps plein

A technology firm is hiring a Senior Full Stack Engineer (Backend-Leaning) with a strong focus on backend services and AI applications.The ideal candidate will have experience in building agent-bas... Voir plus

 • Offre sponsorisée

Staff ML Engineer: Edge AI for Global Ops (Remote)

SamsaraVancouver, Metro Vancouver Regional District, CA
Télétravail
Temps plein

A technology company is looking for a Staff Machine Learning Engineer to work on transformative AI solutions.Based in Canada, this remote role involves leading AI product initiatives and collaborat... Voir plus

 • Offre sponsorisée

Remote AI Data Quality & Evaluation Engineer

Veeva SystemsVancouver, Metro Vancouver Regional District, CA
Télétravail
Temps plein

A leading cloud technology company in Toronto is seeking an experienced professional to validate and ensure the reliability of Veeva AI Agents.The ideal candidate should have expertise in data qual... Voir plus

 • Offre sponsorisée

AI Solutions Engineer - Remote Opportunity

Yeah! GlobalVancouver, Metro Vancouver Regional District, CA
Télétravail
Temps plein

Elevate your career as an AI Solutions Engineer with a remote role for a California-based client.Focus on designing and launching innovative AI initiatives that drive business success.We're buildin... Voir plus

 • Offre sponsorisée

AI Engineer (Remote)

LumenaltaVancouver, Metro Vancouver Regional District, CA
Télétravail
Temps plein

At Lumenalta, we partner with forward‑thinking organizations to build technology solutions that scale, delight users, and accelerate business growth.Our global teams bring curiosity, commitment, an... Voir plus

 • Offre sponsorisée

AI Engineer: Build Innovative AI Solutions

Durable Technologies IncVancouver, Metro Vancouver Regional District, CA
Temps plein

A tech company focused on AI is seeking full stack engineers to build and enhance customer-facing features.This position requires expertise in React, NextJS, and TypeScript, along with experience i... Voir plus

 • Offre sponsorisée

Remote Chemistry Specialist — AI Content QA & Evaluation

Crossing HurdlesVancouver, Metro Vancouver Regional District, CA
Télétravail
Temps plein

A recruitment firm is seeking a Chemistry Expert to review and verify AI-generated content and develop chemistry problems.This remote role requires an advanced degree and strong analytical skills.T... Voir plus