Talent.com
YO AI Labs
AI Agent Evaluation Engineer - RemoteYO AI Labs • Calgary, AB, Canada
AI Agent Evaluation Engineer - Remote

AI Agent Evaluation Engineer - Remote

YO AI Labs • Calgary, AB, Canada
1 day ago
Job type
  • Full-time
  • Remote
  • Quick Apply
Job description

Senior Software Engineer

Job Type: Contractor (~15 hours/week)
Location: Remote

Job Summary

We are seeking experienced Senior Software Engineers to support an AI training project by creating reinforcement learning environments that evaluate AI models on complex software engineering tasks using Model Context Protocol (MCP) tools.

You will design reproducible environments, deterministic verification, and reference solutions for tasks such as bug fixing, feature implementation, codebase refactoring, and performance optimization. No prior AI experience is required.

Key Responsibilities

  • Create reinforcement learning environments for software engineering tasks.
  • Design tasks involving bug fixing, feature development, refactoring, and performance optimization.
  • Build deterministic verification systems and golden reference solutions.
  • Evaluate AI agents' ability to reason through complex codebases and use MCP tools effectively.
  • Develop realistic, reproducible software engineering scenarios.
  • Ensure tasks accurately measure coding ability, problem-solving, and tool usage.
  • Document solutions and provide clear technical feedback.

Required Skills

  • Strong proficiency in Python 3, Java, Rust, C++, or TypeScript.
  • Strong understanding of algorithms and data structures.
  • Experience with bug fixing and debugging complex software issues.
  • Proven experience in feature implementation and codebase refactoring.
  • Strong knowledge of performance optimization and tuning.
  • Excellent written and verbal communication.
  • Strong attention to detail.

Preferred Qualifications

  • Experience working with large-scale or distributed codebases.
  • Familiarity with AI/ML systems is a plus but not required.
  • Experience with rigorous code reviews and software engineering best practices.
  • Experience working effectively in remote or cross-functional teams.

Hiring Process

  1. Submit an application and screening questions.
  2. Complete an AI interview (~30 minutes).
  3. Complete a technical assessment, if required.
  4. Hiring Manager review.

Compensation

Compensation is output-based, with payment provided per task that meets project specifications. Minimum weekly submission requirements may apply.

Availability

Selected experts should be prepared to begin their first tasks within 24–48 hours of completing onboarding.

Create a job alert for this search

AI Agent Evaluation Engineer - Remote • Calgary, AB, Canada

Similar jobs

Senior AI Engineer - Backend-Driven Agentic (Remote)

BayOne SolutionsCalgary, AB, CA
Remote
Full-time

A technology firm is hiring a Senior Full Stack Engineer (Backend-Leaning) with a strong focus on backend services and AI applications.The ideal candidate will have experience in building agent-bas... Show more

 • Promoted

Remote AI Software Engineer - Contractor, 10-40 hrs/wk

TuringCalgary, AB, CA
Remote
Full-time

A leading research accelerator is seeking a software engineering contractor based in Canada, Ontario, to evaluate and refine AI-generated code.This role requires over 5 years of experience, includi... Show more

 • Promoted

Remote Chemistry Specialist — AI Content QA & Evaluation

Crossing HurdlesCalgary, AB, CA
Remote
Full-time

A recruitment firm is seeking a Chemistry Expert to review and verify AI-generated content and develop chemistry problems.This remote role requires an advanced degree and strong analytical skills.T... Show more

 • Promoted

Backend Engineer: GIS APIs & AI-Driven Platform (Remote)

AmuseCalgary, AB, CA
Remote
Full-time

A culture and arts platform is seeking a Mid-Senior Back End Developer in Ottawa to enhance its API for the Amuse platform.Candidates should have experience with Go or equivalent languages, and a k... Show more

 • Promoted

AI Engineer - Generative & Agentic AI

BDO CanadaCalgary, AB, CA
Full-time

Putting people first, every day.BDO is a firm built on a foundation of positive relationships with our people and our clients.Each day, our professionals provide exceptional service, helping client... Show more

 • Promoted

Python & Kubernetes AI/ML Data Engineer — Remote/Distributed

CanonicalCalgary, AB, CA
Remote
Full-time

Canonical, a leader in open source software, seeks a Python and Kubernetes Software Engineer focused on Data, AI/ML, and Analytics Solutions.Collaborate on end-to-end data analytics solutions and c... Show more

 • Promoted

Applied AI Research Scientist — Multimodal Evaluation

MillionLogicsCalgary, AB, CA
Full-time

A global IT solutions provider is searching for skilled Applied AI Research Scientists to contribute to AI system evaluations.Candidates should hold an MS or Ph.The role involves designing rigorous... Show more

 • Promoted

Principal Data Engineer, Llm/Ai Platforms (Remote)

CrowdStrikeCalgary, Canada
Remote
Full-time

CrowdStrike is a global leader in cybersecurity dedicated to stopping breaches with an advanced AI-native platform.About the RolePrincipal Data Engineer with deep expertise in Large Language Models... Show more

 • Promoted

Senior ML Engineer — Remote, Real‑Time Ads Pipelines

StackAdaptCalgary, AB, CA
Remote
Full-time

A leading technology company is seeking Senior and Staff Machine Learning Engineers to join their Data Science team.You will be responsible for designing scalable data pipelines and implementing ML... Show more

 • Promoted

Remote Software Engineer – AI Trajectory Annotations

Weekday AI (YC W21)Calgary, AB, CA
Remote
Part-time

A leading AI consultancy is seeking experienced Software Engineers to assist in creating training datasets for AI models.This part-time role, based in Canada, demands a minimum of 2 years' experien... Show more

 • Promoted

Remote Data Annotation Specialist (English & Text AI)

iMerit ScholarsCalgary, AB, CA
Remote
Full-time

A leading tech organization is seeking a meticulous Data Annotation Specialist to join their AI and ML projects team.This remote, full-time position involves generating and validating datasets acro... Show more

 • Promoted

Remote Forward-Deployed Engineer: AI-Driven Modernization

Banyan SoftwareCalgary, AB, CA
Remote
Full-time

A leading software company is seeking a Forward Deployed Engineer to drive the modernization of applications with a focus on AI-assisted software development practices.This hybrid role requires ove... Show more

 • Promoted

RLHF Evaluation Specialist for AI Training

Rex.zoneCalgary, AB, CA
Full-time

A tech company specializing in AI is seeking an AI Trainer based in Canada.The role involves improving large language models through RLHF-style evaluations, prompt evaluation, and data labeling.Can... Show more

 • Promoted

AI Engineer – Freelancer

TwineCalgary, AB, CA
Full-time

Join a project focused on advancing an AI‑powered influencer marketing platform by optimizing and scaling its data pipeline.The primary objective is to enhance the performance, cost‑efficiency, and... Show more

 • Promoted

Spatial Ai Engineer

BigGeoCalgary, Canada
Full-time

We help companies manage and access the world’s spatial data.Any size, any slice, any insight.The Spatial Cloud is the infrastructure layer that makes spatial data accessible to any company, at any... Show more

 • Promoted

AI Engineer

BioRenderCalgary, AB, CA
Full-time

At BioRender, we don’t settle for average, and neither should you.We’re building tools that transform how scientists communicate complex ideas, and we need someone who’s ready to turn ambitious ide... Show more

 • Promoted

Remote AI/ML Engineer - NLP & LLM Deployment (3+ yrs)

Veracity Software IncCalgary, AB, CA
Remote
Full-time

A technology consulting company is seeking an AI / ML Engineer or Data Scientist with over 3 years of experience in AI/ML development.The successful candidate will have a solid educational backgrou... Show more

 • Promoted

UX Engineer: Firefly AI & Prompt Tech (Remote Canada)

Atimi SoftwareCalgary, Alberta, Canada
Remote
Full-time

A software solutions company is seeking a UX Engineer to work fully remote from Canada.The ideal candidate will have extensive experience with Web Technologies and Graphics Design, and specific exp... Show more

 • Promoted

Artificial Intelligence Engineer

Precision AICalgary, Alberta, Canada
Full-time

Precision AI is on a mission to transform agriculture with cutting‑edge drone technology.Our aerial spraying systems help farmers target weeds with surgical precision, reducing chemical use and inc... Show more

 • Promoted

Remote Access Engineer

ALLTECH CONSULTING SVC INCCalgary, AB, CA
Remote
Full-time

We are a global financial services firm leveraging technology to its highest potential.We’re looking for talented technologists to join our team.Grow your career through a wide range of professiona... Show more