Talent.com
Mindrift
Freelance Agent Evaluation EngineerMindrift • ON, CA
Freelance Agent Evaluation Engineer

Freelance Agent Evaluation Engineer

Mindrift • ON, CA
30+ days ago
Salary
CA$50.00 hourly
Job type
  • Part-time
  • Permanent
  • Remote
  • Quick Apply
Job description

Please submit your CV in English and indicate your level of English proficiency.

Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based, not permanent employment.

What this opportunity involves

We're building a dataset to evaluate AI coding agents - how well a model handles real-world developer tasks.

You'll create challenging tasks and evaluation criteria within realistic simulated environments:

  • Build realistic developer environments - a virtual company with codebase, infrastructure, and context (tickets, docs, conversations) that forms a believable development history
  • Design tasks from intermediate states of these environments - craft the prompt, define what "solved" means, and ensure the task is solvable by an AI agent
  • Write tests that verify agent solutions - accept all valid approaches and reject incorrect ones, neither too strict nor too lenient
  • Iterate on tasks and tests based on QA feedback - review agent solutions, analyze failures, and refine until the evaluation is fair and robust

What this is NOT

  • Not data labeling
  • Not prompt engineering
  • Not writing code from scratch - the agent writes most of the code; you guide and evaluate

What we look for

  • 5+ years in software development
  • Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis
  • Experience writing tests (functional, integration)
  • English proficiency - B2+

Why this is hard

Frontier models are already good at coding. Creating a task that genuinely challenges the best models is non-trivial. You need to deeply understand where models fail and what scenarios reveal the difference between a good and a bad solution. Tasks have many valid solutions - writing tests that accept all correct solutions and reject incorrect ones is harder than it sounds.

How it works

Apply → Pass qualification(s) → Join a project → Complete tasks → Get paid

Effort estimate

Tasks for this project are estimated to take 20 hours to complete, depending on complexity. This is an estimate and not a schedule requirement; you choose when and how to work. Tasks must be submitted by the deadline and meet the listed acceptance criteria to be accepted.

Compensation

Up to $50/hr equivalent, depending on level and pace. Tasks are estimated at ~20 hours each; you set your own schedule.

Create a job alert for this search

Freelance Agent Evaluation Engineer • ON, CA

Similar jobs

Implementation Engineer

SwiftConnectON, CA
Full-time

SwiftConnect is seeking an Implementation Engineer to guide our customers through their journey in deploying our best-of-breed SaaS integration solutions.The Implementation Engineer will be respons... Show more

 • Promoted

Lead Data Modelling Engineer

OpenTextsouthwestern ontario, on, Canada
Full-time

Lead Data Modelling Engineer – OpenText.Join OpenText as a Lead Data Modelling Engineer, where AI powers innovation and you will lead data architecture and software development.A Development role a... Show more

 • Promoted

Senior AI Full‐Stack Engineer

Manulife Financialsouthwestern ontario, on, Canada
Full-time

Step into a role where AI isn’t an add‑‑on it’s the foundation.We’re looking for a Senior Full‑Stack AI Engineer who wants to shape how modern AI‑driven software is built, deployed, and scaled.You’... Show more

 • Promoted

Senior AI Platform Engineer

Alpacasouthwestern ontario, on, Canada
Full-time

Alpaca is building an AI Enablement function to move the company from scattered AI experimentation to durable, company-wide productivity gains.As Senior AI Platform Engineer, you will own the techn... Show more

 • Promoted

Embedded Vision Developer

Planitar Inc.southwestern ontario, on, Canada
Full-time

At Planitar, we build technology that helps people see and understand the world’s spaces more clearly.We are the creators of iGUIDE, a powerful property technology solution that combines immersive ... Show more

 • Promoted

Developer Relations Engineer

CanonicalSouthwestern Ontario, ON, CA
Full-time

Be among the first 25 applicants.Get AI-powered advice on this job and more exclusive features.As the publisher of Ubuntu we serve millions of developers, building for the cloud, IoT and data scien... Show more

 • Promoted

Customer Service Agent - 50k-60k/Year - Remote

Spade RecruitingWhitewater, Ontario
Remote
Full-time
Quick Apply

We’re looking for enthusiastic, self-driven, individuals to assist existing and prospective clients within our organization.This position will work with multiple clients throughout the day pr... Show more

 • Promoted

Staff AI Engineer - Agentic Systems Lead

MasterClasssouthwestern ontario, on, Canada
Full-time

A leading online learning platform is seeking a Staff ML Engineer to join their AI engineering team.In this role, you will design and build innovative AI systems that enhance consumer learning prod... Show more

 • Promoted

Snowflake Tech Sales Engineer - Remote Canada

isgSearch, on, canada, on, Canada
Remote
Full-time

A leading recruitment firm is seeking a Snowflake Technical Sales Engineer in Canada.This full-time position involves driving pre-sales and post-sales technical efforts, providing product demonstra... Show more

 • Promoted

Senior Site Reliability Engineer for SaaS Platform

Magnet Forensicssouthwestern ontario, on, Canada
Full-time

Join Magnet Forensics as a Senior Site Reliability Engineer to strengthen cloud infrastructure for our global SaaS offerings.Utilize your AWS and Kubernetes knowledge for operational excellence.You... Show more

 • Promoted

Senior Operational Test and Evaluation Engineer

Spectrum Comm IncEastern Ontario
Full-time +1

Senior Operational Test and Evaluation Engineer.Spectrum is seeking highly qualified Engineers and/or Analysts to support Operational Test & Evaluation (OT&E) programs located in Edwards AFB, CA.Ke... Show more

 • Promoted

Lead Firmware Engineer for AI Integration

onsemisouthwestern ontario, on, Canada
Full-time

Join onsemi as a Lead Firmware Engineer, focusing on AI hardware integration.Your expertise will power next-gen accelerators through innovative firmware solutions.In this Principal Firmware Develop... Show more

 • Promoted

Principal Software Engineer - Billing

Priority Technology Holdings, Inc.southwestern ontario, on, Canada
Full-time

If you are unable to complete this application due to a disability, contact this employer to ask for an accommodation or an alternative application process.Principal Software Engineer - Billing.Job... Show more

 • Promoted

Claims Evaluator in Accident Benefits

Definity Financial CorporationSouthwestern Ontario, ON, CA
Full-time

Step into a rewarding role with Definity as a Claims Evaluator specializing in Accident Benefits.This hybrid position allows you to work both from the office and remotely while delivering excellent... Show more

 • Promoted

Machine Learning/Operations Research Platform Engineer

Kinaxissouthwestern ontario, on, Canada
Full-time

Are you looking to join an innovative, market‑leading company where you can truly elevate your career? At Kinaxis we are serious about culture, technology, customers, and not taking ourselves too s... Show more

 • Promoted

Machine Learning Research Engineer

Omazesouthwestern ontario, on, Canada
Full-time

The Product Operations machine learning team is seeking a machine learning research engineer to conduct research in anomaly detection and automated machine learning to address domain-specific chall... Show more

 • Promoted

Senior Site Reliability Engineer

Magnet Forensicssouthwestern ontario, on, Canada
Full-time

Who We Are; What We Do; Where We’re Going.Magnet Forensics is a global leader in the development of digital investigative software that acquires, analyzes, and shares evidence from computers, smart... Show more

 • Promoted

Lead Data Modeling Engineer — AI & SaaS

OpenTextsouthwestern ontario, on, Canada
Full-time

A leading technology company in Canada seeks a Lead Data Modelling Engineer to lead the design of SaaS products and implement scalable data systems.The role requires 8+ years of software developmen... Show more

 • Promoted

Senior Full Stack Engineer

Musashi AI North Americasouthwestern ontario, on, Canada
Full-time

Based in Waterloo, Ontario, Musashi AI North America employs a talented and dynamic team of R&D and Applications Engineers who provide unique engineering development activities in design, prototypi... Show more

 • Promoted

Machine Learning Engineer, Edge AI

onsemisouthwestern ontario, on, Canada
Full-time

We are looking for a Principal Firmware Engineer to lead the integration and control of our next‑generation AI accelerators.As our products evolve to include dedicated neural network hardware, the ... Show more