Talent.com
Mindrift
AI Evaluation Engineer (Python, QA or Security)Mindrift • MB, CA
AI Evaluation Engineer (Python, QA or Security)

AI Evaluation Engineer (Python, QA or Security)

Mindrift • MB, CA
6 days ago
Salary
CA$50.00 hourly
Job type
  • Part-time
  • Permanent
  • Remote
  • Quick Apply
Job description

Please submit your CV in English and indicate your level of English proficiency.

Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based, not permanent employment.

What this opportunity involves We're building a dataset to evaluate AI coding agents - how well a model handles real-world developer tasks.

You'll create challenging tasks and evaluation criteria within realistic simulated environments:

  • Build realistic developer environments - a virtual company with codebase, infrastructure, and context (tickets, docs, conversations) that forms a believable development history
  • Design tasks from intermediate states of these environments - craft the prompt, define what "solved" means, and ensure the task is solvable by an AI agent
  • Write tests that verify agent solutions - accept all valid approaches and reject incorrect ones, neither too strict nor too lenient
  • Iterate on tasks and tests based on QA feedback - review agent solutions, analyze failures, and refine until the evaluation is fair and robust

What this is NOT

  • Not data labeling
  • Not prompt engineering
  • Not writing code from scratch - the agent writes most of the code; you guide and evaluate

What we look for

  • 5+ years in software development
  • Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis
  • Experience writing tests (functional, integration)
  • English proficiency - B2+

Why this is hard

Frontier models are already good at coding. Creating a task that genuinely challenges the best models is non-trivial. You need to deeply understand where models fail and what scenarios reveal the difference between a good and a bad solution. Tasks have many valid solutions - writing tests that accept all correct solutions and reject incorrect ones is harder than it sounds.

How it works

Apply → Pass qualification(s) → Join a project → Complete tasks → Get paid

Compensation

Up to $50/hr equivalent, depending on level and pace. Tasks are estimated at ~20 hours each; you set your own schedule.

Create a job alert for this search

AI Evaluation Engineer (Python, QA or Security) • MB, CA

Similar jobs

Math Private Tutoring Jobs Manitoba

SuperprofManitoba, Canada
CA$20.00 hourly
Full-time +1

Superprof is Canada's #1 tutoring platform, and we're actively recruiting passionate tutors! Whether you're a student, a professional, or simply someone who loves teaching, join the largest communi... Show more

 • Promoted

Chemistry Private Tutoring Jobs Manitoba

SuperprofManitoba, Canada
CA$20.00 hourly
Full-time +1

Superprof is Canada's #1 tutoring platform, and we're actively recruiting passionate tutors! Whether you're a student, a professional, or simply someone who loves teaching, join the largest communi... Show more

 • Promoted

French Private Tutoring Jobs Manitoba

SuperprofManitoba, Canada
CA$20.00 hourly
Full-time +1

Superprof is Canada's #1 tutoring platform, and we're actively recruiting passionate tutors! Whether you're a student, a professional, or simply someone who loves teaching, join the largest communi... Show more

 • Promoted

Technical Interviewer Lead, Data and AI

LumenaltaMB, CA
Full-time

At Lumenalta, we create impactful software solutions that drive innovation and transform businesses.Since 2000, we’ve partnered with visionary leaders to build cutting‑edge tech, solve complex chal... Show more

 • Promoted

Remote CS Expert & Data Partner for AI Q&A

TELUS Digital AI Data SolutionsMB, CA
Remote
Full-time

A leading technology firm is seeking a Computer Science Expert to join as a Freelance Data Partner.This remote role involves developing complex question-and-answer pairs in your area of expertise a... Show more

 • Promoted

Remote Data Analyst (Latvian) — Map & Research

TELUS Digital AI Data SolutionsMB, CA
Remote
Part-time

A leading digital AI firm is seeking an Online Data Analyst to work remotely, specifically targeting Latvian speakers.In this entry-level part-time role, you will enhance digital maps used by milli... Show more

 • Promoted

AI Trainer - Chemistry

DataAnnotationMB, CA
Full-time +1

We are seeking a chemistry professional to join our team to train AI models.Your responsibilities will include measuring the progress of AI chatbots, evaluating their logic, and solving problems to... Show more

 • Promoted

Trigonometry Private Tutoring Jobs Manitoba

SuperprofManitoba, Canada
CA$20.00 hourly
Full-time +1

Superprof is Canada's #1 tutoring platform, and we're actively recruiting passionate tutors! Whether you're a student, a professional, or simply someone who loves teaching, join the largest communi... Show more

 • Promoted

Remote Technical Interview Lead — Data & AI

LumenaltaMB, CA
Remote
Full-time

A tech-driven company specializing in innovative software solutions is looking for a Mid-Senior Level Engineer.This fully remote position involves conducting candidate interviews, coordinating with... Show more

 • Promoted

Mechanical Project Manager for Advanced AI Data Centre Projects

NscaleMB, CA
Full-time

Lead the mechanical delivery of cutting-edge data centre projects as a Mechanical Project Manager.Focus on HVAC and plant infrastructure in a site-based/hybrid position.In this essential role, you'... Show more

 • Promoted

Geometry Private Tutoring Jobs Manitoba

SuperprofManitoba, Canada
CA$20.00 hourly
Full-time +1

Superprof is Canada's #1 tutoring platform, and we're actively recruiting passionate tutors! Whether you're a student, a professional, or simply someone who loves teaching, join the largest communi... Show more

 • Promoted

English Private Tutoring Jobs Manitoba

SuperprofManitoba, Canada
CA$20.00 hourly
Full-time +1

Superprof is Canada's #1 tutoring platform, and we're actively recruiting passionate tutors! Whether you're a student, a professional, or simply someone who loves teaching, join the largest communi... Show more

 • Promoted

Media Search Analyst - Work from Home (Canada)

TELUS Digital AI Data SolutionsMB, CA
Remote
Part-time

Media Search Analyst - Work from Home (Canada).Get AI-powered advice on this job and more exclusive features.Direct message the job poster from TELUS Digital AI Data Solutions.Sourcing Analyst @ TE... Show more

 • Promoted

Biology Private Tutoring Jobs Manitoba

SuperprofManitoba, Canada
CA$20.00 hourly
Full-time +1

Superprof is Canada's #1 tutoring platform, and we're actively recruiting passionate tutors! Whether you're a student, a professional, or simply someone who loves teaching, join the largest communi... Show more

 • Promoted

Arithmetic Private Tutoring Jobs Manitoba

SuperprofManitoba, Canada
CA$20.00 hourly
Full-time +1

Superprof is Canada's #1 tutoring platform, and we're actively recruiting passionate tutors! Whether you're a student, a professional, or simply someone who loves teaching, join the largest communi... Show more

 • Promoted

Algebra Private Tutoring Jobs Manitoba

SuperprofManitoba, Canada
CA$20.00 hourly
Full-time +1

Superprof is Canada's #1 tutoring platform, and we're actively recruiting passionate tutors! Whether you're a student, a professional, or simply someone who loves teaching, join the largest communi... Show more

 • Promoted

Senior Software Developer - Fintech AI Planning

Conquest PlanningMB, CA
Full-time

A dynamic fintech company is seeking a Senior Software Developer to join their team in Winnipeg, Manitoba.The role demands strong coding abilities, excellent communication skills, and a minimum of ... Show more

 • Promoted

Computer Science Expert - Data Partner (Remote)

TELUS Digital AI Data SolutionsMB, CA
Remote
Full-time

Computer Science Expert - Data Partner (Remote).Join us as a Freelance Data Partner and work remotely with flexible hours.Are you ready to use your domain knowledge to advance AI? We are seeking hi... Show more

 • Promoted

ESL Private Tutoring Jobs Manitoba

SuperprofManitoba, Canada
CA$20.00 hourly
Full-time +1

Superprof is Canada's #1 tutoring platform, and we're actively recruiting passionate tutors! Whether you're a student, a professional, or simply someone who loves teaching, join the largest communi... Show more

 • Promoted

Physics Private Tutoring Jobs Manitoba

SuperprofManitoba, Canada
CA$20.00 hourly
Full-time +1

Superprof is Canada's #1 tutoring platform, and we're actively recruiting passionate tutors! Whether you're a student, a professional, or simply someone who loves teaching, join the largest communi... Show more