Talent.com
Mindrift
Freelance Agent Evaluation EngineerMindrift • QC, CA
Freelance Agent Evaluation Engineer

Freelance Agent Evaluation Engineer

Mindrift • QC, CA
Il y a 2 jours
Salaire
50,00 $CA par heure
Type de contrat
  • Temps partiel
  • Permanent
  • Télétravail
  • Quick Apply
Description de poste

Please submit your CV in English and indicate your level of English proficiency.


Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based, not permanent employment.


We're building a dataset to evaluate AI coding agents - how well a model handles real-world developer tasks.

You'll create challenging tasks and evaluation criteria within realistic simulated environments:

  • Build realistic developer environments - a virtual company with codebase, infrastructure, and context (tickets, docs, conversations) that forms a believable development history
  • Design tasks from intermediate states of these environments - craft the prompt, define what "solved" means, and ensure the task is solvable by an AI agent
  • Write tests that verify agent solutions - accept all valid approaches and reject incorrect ones, neither too strict nor too lenient
  • Iterate on tasks and tests based on QA feedback - review agent solutions, analyze failures, and refine until the evaluation is fair and robust

What this is NOT:

  • Not data labeling
  • Not prompt engineering
  • Not writing code from scratch - the agent writes most of the code; you guide and evaluate

What we look for:

  • 5+ years in software development
  • Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis
  • Experience writing tests (functional, integration)
  • English proficiency - B2+

Why this is hard:
Frontier models are already good at coding. Creating a task that genuinely challenges the best models is non-trivial. You need to deeply understand where models fail and what scenarios reveal the difference between a good and a bad solution. Tasks have many valid solutions - writing tests that accept all correct solutions and reject incorrect ones is harder than it sounds.

Requirements and benefits

Educational qualifications

  • A Master’s Degree in Computer Science, Software Engineering, Data Science / Data Analytics, Artificial Intelligence / Machine Learning, Computational Linguistics / Natural Language Processing (NLP), Information Systems or other related fields.
  • Bachelor’s degree is accepted if only candidate has 5 years of experience in the field.

Academic and/or Professional Experience

Candidates should have a minimum of 3 years of professional experience in related roles or domain - specifically for QA-automation/testing or cybersecurity roles

How it works
Apply → Pass qualification(s) → Join a project → Complete tasks → Get paid

Compensation:
Paid per accepted task. Your rate depends on the qualification tier you reach and how efficiently you complete tasks — up to the equivalent of $50/hr. Because payment is per task, a faster pace raises your effective hourly rate.

Why this freelance opportunity might be a great fit for you?

  • Take part in a part-time, remote, freelance project that fits around your primary professional or academic commitments.
  • Work on advanced AI projects and gain valuable experience that enhances your portfolio. - Influence how future AI models understand and communicate in your field of expertise.
Créer une alerte emploi pour cette recherche

Freelance Agent Evaluation Engineer • QC, CA

Offres similaires

Cloud Cost Management (FinOps) Business Analyst

ALLTECH CONSULTING SVC INCQC, CA
Temps plein

Drive adoption of Cloud FinOps capabilities and best practices within the organization.Develop cost management processes, mechanisms and controls to provide ongoing cost transparency and governance... Voir plus

 • Offre sponsorisée

Project Manager, Sustainable Land Use Planning - C$60,074 - C$102,839 A Year

Kativik Regional GovernmentNord-Du-Québec, Canada
Temps plein +1

The Kativik Regional Government (KRG), a supra-municipal organization with jurisdiction over the territory located north of the 55th parallel, is currently seeking a motivated and dynamic individua... Voir plus

 • Offre sponsorisée • Nouvelle offre

Senior Front-End Architect (Vue, TS) – Scalable Web Apps

ALLTECH CONSULTING SVC INCQC, CA
Temps plein

A leading tech consulting firm in Quebec is seeking an experienced Senior Front-End Developer to enhance their vulnerability management platforms.The ideal candidate should have over 5 years of pro... Voir plus

 • Offre sponsorisée

Livreur en voiture Instacart - Horaire flexible

Instacart Shoppersglen, qc, Canada
Temps partiel +1

Start earning money fast and be your own boss.Instacart is a leading retail enablement platform that works with grocers and retailers to transform how people shop.Instacart’s platform helps people ... Voir plus

 • Offre sponsorisée

Tech Lead, LiveOps Monetization Platform

XsollaQC, CA
Temps plein

A technology company in gaming is seeking a Tech Lead for their LiveOps Monetization platform.This hands-on role emphasizes leadership, overseeing scalable service design, and mentoring engineers.T... Voir plus

 • Offre sponsorisée

Senior Sales Development Partner (Remote)

Canadian Consultancy GroupQC, CA
Télétravail
Temps plein

With the objective of consolidating our international strategy through expansion into the African market, we are looking to bring on board seasoned talents for strategic partnership projects.We are... Voir plus

 • Offre sponsorisée

Ingénieur(e) Intégration du poste de pilotage et facteurs humains

SOGECLAIRQC, CA
Temps plein

Participate in the elicitation of Human Machine Interface (HMI) requirements for multiple embedded systems, and validation of these requirements.Coordinate design activities with system integrators... Voir plus

 • Offre sponsorisée

Remote Market Research Assistant – Earn Up to $50/Task

Occupons QuebecPort Terry, QC, Canada
Télétravail
Temps partiel

Compensation: Up to $50 per task.We are seeking individuals to complete small, remote market research tasks from home.Your insights help shape future products, services, and public trends.All tasks... Voir plus

 • Offre sponsorisée

Unreal Content Tech Director - Lead AAA Game Team

Highdive, qc, canada, qc, Canada
Temps plein

Une entreprise de jeux vidéo à Montréal cherche un Directeur Technique Généraliste de Contenu.Vous dirigerez une équipe technique tout en collaborant avec des partenaires internationaux.Les respons... Voir plus

 • Offre sponsorisée

Tech Lead — LiveOps Team

XsollaQC, CA
Temps plein

Xsolla is a global commerce company with robust tools and services to help developers solve the inherent challenges of the video game industry.From indie to AAA, companies partner with Xsolla to he... Voir plus

 • Offre sponsorisée

Senior Cloud FinOps Analyst: Cost Optimization

ALLTECH CONSULTING SVC INCQC, CA
Temps plein

A leading consulting firm is seeking a senior professional to drive Cloud FinOps capabilities and optimize public cloud spending.The ideal candidate will have extensive experience in technology fin... Voir plus

 • Offre sponsorisée

Tech Lead, Subscriptions Platform — Scale & Deliver

XsollaQC, CA
Temps plein

A leading global commerce company is seeking a Tech Lead for their Subscriptions platform in Canada, Quebec.This role combines management and hands-on engineering, focusing on leading a backend tea... Voir plus

 • Offre sponsorisée

Qualified Teachers - French Sector For The 2026-2027 School Year

Kativik IlisarniliriniqNord-Du-Québec, Canada
Temps plein

We are looking for passionate and creative teachers for the upcoming school year at the primary and secondary levels in the French sectors.SUBJECTS TAUGHT : General education Second languages a... Voir plus

 • Offre sponsorisée

Lead Engineer for LiveOps Services

XsollaQC, CA
Temps plein

Join Xsolla as a Lead Engineer, directing the LiveOps Monetization platform’s technical vision.Enjoy a hands-on role leading a skilled team to deliver exceptional gaming solutions.As a Tech Lead at... Voir plus

 • Offre sponsorisée

Senior Front-end Developer

ALLTECH CONSULTING SVC INCQC, CA
Temps plein

The vulnerability management platforms team is looking for an experienced and proactive senior front-end developer with solid background in enterprise web app development.The candidate should be ab... Voir plus

 • Offre sponsorisée

Ingénieur Intégration HMI & Facteurs Humains

SOGECLAIRQC, CA
Temps plein

A leading aerospace firm based in Canada seeks candidates for an engineering position focused on developing human-machine interface requirements for embedded systems.Candidates should have a Bachel... Voir plus