Talent.com
Mindrift
AI Evaluation Engineer (Python, QA or Security)Mindrift • QC, CA
AI Evaluation Engineer (Python, QA or Security)

AI Evaluation Engineer (Python, QA or Security)

Mindrift • QC, CA
5 days ago
Salary
CA$50.00 hourly
Job type
  • Part-time
  • Permanent
  • Remote
  • Quick Apply
Job description

Please submit your CV in English and indicate your level of English proficiency.

Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based, not permanent employment.

What this opportunity involves We're building a dataset to evaluate AI coding agents - how well a model handles real-world developer tasks.

You'll create challenging tasks and evaluation criteria within realistic simulated environments:

  • Build realistic developer environments - a virtual company with codebase, infrastructure, and context (tickets, docs, conversations) that forms a believable development history
  • Design tasks from intermediate states of these environments - craft the prompt, define what "solved" means, and ensure the task is solvable by an AI agent
  • Write tests that verify agent solutions - accept all valid approaches and reject incorrect ones, neither too strict nor too lenient
  • Iterate on tasks and tests based on QA feedback - review agent solutions, analyze failures, and refine until the evaluation is fair and robust

What this is NOT

  • Not data labeling
  • Not prompt engineering
  • Not writing code from scratch - the agent writes most of the code; you guide and evaluate

What we look for

  • 5+ years in software development
  • Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis
  • Experience writing tests (functional, integration)
  • English proficiency - B2+

Why this is hard

Frontier models are already good at coding. Creating a task that genuinely challenges the best models is non-trivial. You need to deeply understand where models fail and what scenarios reveal the difference between a good and a bad solution. Tasks have many valid solutions - writing tests that accept all correct solutions and reject incorrect ones is harder than it sounds.

How it works

Apply → Pass qualification(s) → Join a project → Complete tasks → Get paid

Compensation

Up to $50/hr equivalent, depending on level and pace. Tasks are estimated at ~20 hours each; you set your own schedule.

Create a job alert for this search

AI Evaluation Engineer (Python, QA or Security) • QC, CA

Similar jobs

Lead Engineer for LiveOps Services

Xsolla, qc, canada, qc, Canada
Full-time

Join Xsolla as a Lead Engineer, directing the LiveOps Monetization platform’s technical vision.Enjoy a hands-on role leading a skilled team to deliver exceptional gaming solutions.As a Tech Lead at... Show more

 • Promoted

Elodie looking for a babysitter or nanny in Québec

SitlyQuébec, CA
Part-time

Je suis nouvelle rsidente dans la ville de Qubec , mre monoparentale avec 2 enfants , je souhaite laisser les enfants occasionnellement afin de commencer mes dmarches administratives. Show more

 • Promoted

Ingénieur ML Senior – Perception Hors Ligne & Annotation

Torc RoboticsQC, CA
Full-time

Une entreprise de technologie autonome à Québec est à la recherche d'un Ingénieur(e) Senior(e) en apprentissage automatique pour concevoir des systèmes innovants de détection et de suivi.Le candida... Show more

 • Promoted

Senior Front-End Architect (Vue, TS) – Scalable Web Apps

ALLTECH CONSULTING SVC INCQC, CA
Full-time

A leading tech consulting firm in Quebec is seeking an experienced Senior Front-End Developer to enhance their vulnerability management platforms.The ideal candidate should have over 5 years of pro... Show more

 • Promoted

PeopleSoft Functional Consultants - Exciting Roles

ROSSQC, CA
Permanent

Our client is a global professional services organization.With lots of career growth opportunity, great training and interesting projects, use your PeopleSoft 8.Implementation Consultant, Senior Co... Show more

 • Promoted

Emilie looking for a babysitter or nanny in Québec

SitlyQuébec, CA
Part-time

Jai 2 enfants (1 garon et 1 fille, 5 et 8 ans).Je cherche quelquun de fiable occasionnellement un aprs-midi aux 2 semaines pour avoir un rpit. Show more

 • Promoted

Ingénieur(e) Intégration du poste de pilotage et facteurs humains

SOGECLAIRQC, CA
Full-time

Participate in the elicitation of Human Machine Interface (HMI) requirements for multiple embedded systems, and validation of these requirements.Coordinate design activities with system integrators... Show more

 • Promoted

Larissa looking for a babysitter or nanny in Québec

SitlyQuébec, CA
Part-time

J'ai deux garons,un de 5 ans 6 mois ,et un bb de 4 mois, il sont adorables,et aiment les jeux, trs gentil,nous sommes des personnes attentionn. Show more

 • Promoted

Zeynaa looking for a babysitter or nanny in Québec

SitlyQuébec, CA
Part-time

Jai un seul enfant de 11 mois nergique qui ne va pas encore en garderie mais qui aime jouer.Il ya mon conjoint qui va au travail 5/7 et fini la plupart du temps 17:30.Il ya aussi mon petit frre qu... Show more

 • Promoted

PeopleSoft Functional Consultant — Travel‑Ready

ROSSQC, CA
Full-time

A global professional services organization in Canada is seeking an experienced Implementation Consultant with a proven track record in PeopleSoft 8.The role includes providing client advice, ensur... Show more

 • Promoted

Earn Up To $400 Per Day By Playing Games And Taking Surveys

AlmediaNord-Du-Québec, Canada
Full-time

Com is an innovative new platform for side hustle seekers who want to earn money online by playing games, taking surveys, and completing various assigned tasks.Since its launch 5 years ago, over 50... Show more

 • Promoted

Jennifer looking for a babysitter or nanny in Québec

SitlyQuébec, CA
Part-time

We have 2childs Girl, 3years old and 17 months We are looking for a babysitter, nanny, or home care provider.Feel free to contact me so we can make an appointment. Show more

 • Promoted

Israelly looking for a babysitter or nanny in Québec

SitlyQuébec, CA
Part-time

Nous avons 1 enfant (1 garon, 0 ans).N'hsitez pas me contacter pour prendre rendez-vous. Show more

 • Promoted

Kara looking for a babysitter or nanny in Québec

SitlyQuébec, CA
Part-time

Hello, we are in town from the US and are looking for someone to help watch our 7 month old daughter while we are some work outings.She is very sweet and snuggly and loves to play games with you an... Show more

 • Promoted

Julie looking for a babysitter or nanny in Québec

SitlyQuébec, CA
Part-time

Jai deux garons, ils sont gentils et poli.Ils ont aussi beaucoup dnergie.Mais ils sont capable de rester tranquille quand il le faut.Ils aiment beaucoup parler et poser des questions.Je vais avoir ... Show more

 • Promoted

Geneviève looking for a babysitter or nanny in Québec

SitlyQuébec, CA
Part-time

Bonjour, Nous sommes la recherche d'une ou d'un babysitter pour garder occasionnellement nos deux enfants : Clara, 4 ans, et Raphal, 2 ans et demi. Show more

 • Promoted

Philibert looking for a babysitter or nanny in Québec

SitlyQuébec, CA
Part-time

Salut, j ai un garon de 5 ans qui a besoin beaucoup de bouger.Et deux sont autonomes, juste leur faire manger. Show more

 • Promoted

Elyana looking for a babysitter or nanny in Québec

SitlyQuébec, CA
Part-time

Nous avons 1 enfant (1 fille, gede 4 mois).N'hsitez pas me contacter pour prendre rendez-vous. Show more

 • Promoted

Senior Front-end Developer

ALLTECH CONSULTING SVC INCQC, CA
Full-time

The vulnerability management platforms team is looking for an experienced and proactive senior front-end developer with solid background in enterprise web app development.The candidate should be ab... Show more

 • Promoted

Remote Market Research Assistant – Earn Up to $50/Task

Occupons QuebecNorth Katelynville, QC, Canada
Remote
Part-time

Compensation: Up to $50 per task.We are seeking individuals to complete small, remote market research tasks from home.Your insights help shape future products, services, and public trends.All tasks... Show more