Talent.com
YO AI Labs
AI Agent Evaluation Engineer - RemoteYO AI Labs • Edmonton, AB, Canada
AI Agent Evaluation Engineer - Remote

AI Agent Evaluation Engineer - Remote

YO AI Labs • Edmonton, AB, Canada
1 day ago
Job type
  • Full-time
  • Remote
  • Quick Apply
Job description

Senior Software Engineer

Job Type: Contractor (~15 hours/week)
Location: Remote

Job Summary

We are seeking experienced Senior Software Engineers to support an AI training project by creating reinforcement learning environments that evaluate AI models on complex software engineering tasks using Model Context Protocol (MCP) tools.

You will design reproducible environments, deterministic verification, and reference solutions for tasks such as bug fixing, feature implementation, codebase refactoring, and performance optimization. No prior AI experience is required.

Key Responsibilities

  • Create reinforcement learning environments for software engineering tasks.
  • Design tasks involving bug fixing, feature development, refactoring, and performance optimization.
  • Build deterministic verification systems and golden reference solutions.
  • Evaluate AI agents' ability to reason through complex codebases and use MCP tools effectively.
  • Develop realistic, reproducible software engineering scenarios.
  • Ensure tasks accurately measure coding ability, problem-solving, and tool usage.
  • Document solutions and provide clear technical feedback.

Required Skills

  • Strong proficiency in Python 3, Java, Rust, C++, or TypeScript.
  • Strong understanding of algorithms and data structures.
  • Experience with bug fixing and debugging complex software issues.
  • Proven experience in feature implementation and codebase refactoring.
  • Strong knowledge of performance optimization and tuning.
  • Excellent written and verbal communication.
  • Strong attention to detail.

Preferred Qualifications

  • Experience working with large-scale or distributed codebases.
  • Familiarity with AI/ML systems is a plus but not required.
  • Experience with rigorous code reviews and software engineering best practices.
  • Experience working effectively in remote or cross-functional teams.

Hiring Process

  1. Submit an application and screening questions.
  2. Complete an AI interview (~30 minutes).
  3. Complete a technical assessment, if required.
  4. Hiring Manager review.

Compensation

Compensation is output-based, with payment provided per task that meets project specifications. Minimum weekly submission requirements may apply.

Availability

Selected experts should be prepared to begin their first tasks within 24–48 hours of completing onboarding.

Create a job alert for this search

AI Agent Evaluation Engineer - Remote • Edmonton, AB, Canada

Similar jobs

Remote Senior Solutions Engineer — Enterprise AI & Risk

GatekeeperEdmonton, Division No. 11, CA
Remote
Full-time

A technology company is seeking an experienced Senior Solutions Engineer for a fully remote role in Canada.The selected candidate will be responsible for leading enterprise customer implementations... Show more

 • Promoted

UX Engineer: Firefly AI & Prompt Tech (Remote Canada)

Atimi SoftwareEdmonton, Division No. 11, CA
Remote
Full-time

A software solutions company is seeking a UX Engineer to work fully remote from Canada.The ideal candidate will have extensive experience with Web Technologies and Graphics Design, and specific exp... Show more

 • Promoted

Remote AI Software Engineer - Contractor, 10-40 hrs/wk

TuringEdmonton, Division No. 11, CA
Remote
Full-time

A leading research accelerator is seeking a software engineering contractor based in Canada, Ontario, to evaluate and refine AI-generated code.This role requires over 5 years of experience, includi... Show more

 • Promoted

Remote Chemistry Expert (LLM Evaluation) - 34879

TuringEdmonton, Division No. 11, CA
Remote
Full-time

Remote Chemistry Expert (LLM Evaluation) - 34879.Remote Chemistry Expert (LLM Evaluation) - 34879.Be among the first 25 applicants.Get AI-powered advice on this job and more exclusive features.Turi... Show more

 • Promoted

Remote Data Annotation Specialist (English & Text AI)

iMerit ScholarsEdmonton, Division No. 11, CA
Remote
Full-time

A leading tech organization is seeking a meticulous Data Annotation Specialist to join their AI and ML projects team.This remote, full-time position involves generating and validating datasets acro... Show more

 • Promoted

AI Solutions Engineer - Remote Opportunity

Yeah! GlobalEdmonton, Division No. 11, CA
Remote
Full-time

Elevate your career as an AI Solutions Engineer with a remote role for a California-based client.Focus on designing and launching innovative AI initiatives that drive business success.We're buildin... Show more

 • Promoted

Public Sector AI Engineer — Agentic Systems & Production

CohereEdmonton, Division No. 11, CA
Full-time

A leading AI research company in Canada seeks a Member of Technical Staff - Public Sector to design and implement agentic AI systems.Responsibilities include building systems for critical use cases... Show more

 • Promoted

Senior AI Engineer - Backend-Driven Agentic (Remote)

BayOne SolutionsEdmonton, Division No. 11, CA
Remote
Full-time

A technology firm is hiring a Senior Full Stack Engineer (Backend-Leaning) with a strong focus on backend services and AI applications.The ideal candidate will have experience in building agent-bas... Show more

 • Promoted

Dynamic AI Agent Engineer Delivering Real-World Solutions

CrestaEdmonton, Division No. 11, CA
Full-time

Be at the forefront of AI technology by developing innovative agents that drive business efficiency and enhance customer satisfaction.In this role, leverage your skills to implement AI solutions th... Show more

 • Promoted

AI Developer

Hanalytica GmbHEdmonton, Division No. 11, CA
Full-time

This range is provided by Hanalytica GmbH.Your actual pay will be based on your skills and experience — talk with your recruiter to learn more.Our client is looking for an exceptional.AI and machin... Show more

 • Promoted

Remote AI Data Quality & Evaluation Engineer

Veeva SystemsEdmonton, Division No. 11, CA
Remote
Full-time

A leading cloud technology company in Toronto is seeking an experienced professional to validate and ensure the reliability of Veeva AI Agents.The ideal candidate should have expertise in data qual... Show more

 • Promoted

RLHF Evaluation Specialist for AI Training

Rex.zoneEdmonton, Division No. 11, CA
Full-time

A tech company specializing in AI is seeking an AI Trainer based in Canada.The role involves improving large language models through RLHF-style evaluations, prompt evaluation, and data labeling.Can... Show more

 • Promoted

Remote Software Engineer — Train & Evaluate AI Chatbot Code

DataAnnotationEdmonton, Division No. 11, CA
Remote
Full-time

An innovative company is seeking a skilled Software Engineer to help train AI chatbots.This exciting remote role allows you to work on your own schedule while contributing to the development of cut... Show more

 • Promoted

AI Engineer

BioRenderEdmonton, Division No. 11, CA
Full-time

At BioRender, we don’t settle for average, and neither should you.We’re building tools that transform how scientists communicate complex ideas, and we need someone who’s ready to turn ambitious ide... Show more

 • Promoted

Remote AI Content Evaluator - Part-Time, Flexible Hours

Work VistaEdmonton, Division No. 11, CA
Remote
Part-time

A global AI services provider is seeking a remote Content Evaluator to assess the quality of AI-generated content.This part-time role involves evaluating responses for fluency, coherence, and factu... Show more

 • Promoted

Remote Software Engineer – AI Trajectory Annotations

Weekday AI (YC W21)Edmonton, Division No. 11, CA
Remote
Part-time

A leading AI consultancy is seeking experienced Software Engineers to assist in creating training datasets for AI models.This part-time role, based in Canada, demands a minimum of 2 years' experien... Show more

 • Promoted

Remote Chemistry Specialist — AI Content QA & Evaluation

Crossing HurdlesEdmonton, Division No. 11, CA
Remote
Full-time

A recruitment firm is seeking a Chemistry Expert to review and verify AI-generated content and develop chemistry problems.This remote role requires an advanced degree and strong analytical skills.T... Show more

 • Promoted

Remote CS Expert & Data Partner for AI Q&A

TELUS Digital AI Data SolutionsEdmonton, Division No. 11, CA
Remote
Full-time

A leading technology firm is seeking a Computer Science Expert to join as a Freelance Data Partner.This remote role involves developing complex question-and-answer pairs in your area of expertise a... Show more

 • Promoted

AI Engineer – Freelancer

TwineEdmonton, Division No. 11, CA
Full-time

Join a project focused on advancing an AI‑powered influencer marketing platform by optimizing and scaling its data pipeline.The primary objective is to enhance the performance, cost‑efficiency, and... Show more

 • Promoted

Senior AI Engineer - Production ML for Fintech (Remote)

LumenaltaEdmonton, Division No. 11, CA
Remote
Full-time

A forward-thinking technology firm is looking for experienced AI Engineers to design and deploy production-grade AI systems for community banks and insurance institutions.This role offers the chanc... Show more