Talent.com
Mindrift
Senior Python Engineer - AI Coding Agent Evaluation (Freelance)Mindrift • AB, CA
No longer accepting applications
Senior Python Engineer - AI Coding Agent Evaluation (Freelance)

Senior Python Engineer - AI Coding Agent Evaluation (Freelance)

Mindrift • AB, CA
23 days ago
Salary
CA$200.00 daily
Job type
  • Part-time
  • Permanent
  • Remote
  • Quick Apply
Job description

Please submit your CV in English and indicate your level of English proficiency.


Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based, not permanent employment.What this opportunity involves:
We're building a dataset to evaluate AI coding agents - how well a model handles real-world developer tasks.


You'll create challenging tasks and evaluation criteria within realistic simulated environments:

  • Build realistic developer environments - a virtual company with codebase, infrastructure, and context (tickets, docs, conversations) that forms a believable development history
  • Design tasks from intermediate states of these environments - craft the prompt, define what "solved" means, and ensure the task is solvable by an AI agent
  • Write tests that verify agent solutions - accept all valid approaches and reject incorrect ones, neither too strict nor too lenient
  • Iterate on tasks and tests based on QA feedback - review agent solutions, analyze failures, and refine until the evaluation is fair and robust

What this is NOT:

  • Not data labeling
  • Not prompt engineering
  • Not writing code from scratch - the agent writes most of the code; you guide and evaluate

What we look for:

  • 8+ years in software development
  • Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis
  • Experience writing tests (functional, integration)
  • English proficiency - B2+

Why this is hard:
Frontier models are already good at coding. Creating a task that genuinely challenges the best models is non-trivial. You need to deeply understand where models fail and what scenarios reveal the difference between a good and a bad solution. Tasks have many valid solutions - writing tests that accept all correct solutions and reject incorrect ones is harder than it sounds.

How it works
Apply → Pass qualification(s) → Join a project → Complete tasks → Get paidEffort estimate


Tasks for this project are estimated to take 30 hours to complete, depending on complexity. This is an estimate and not a schedule requirement; you choose when and how to work. Tasks must be submitted by the deadline and meet the listed acceptance criteria to be accepted.

Compensation:
Up to $200/hr equivalent, depending on level and pace. Tasks are estimated at ~30 hours each; you set your own schedule.

Create a job alert for this search

Senior Python Engineer - AI Coding Agent Evaluation (Freelance) • AB, CA

Similar jobs

Data Partner - Biology Expert (Remote)

TELUS Digital AI Data SolutionsAB, CA
Remote
Part-time

Data Partner - Biology Expert (Remote).TELUS Digital AI Data Solutions.Get AI-powered advice on this job and more exclusive features.Direct message the job poster from TELUS Digital AI Data Solutio... Show more

 • Promoted

Senior React Developer

Altis Technology, ab, canada, ab, Canada
Full-time

We are assisting a reputable client in the geoscience industry with the search for a Senior Front End Developer.Calgary, AB – Hybrid (2 days in the office).You will work on a cloud-based geoscience... Show more

 • Promoted

Senior Software Engineer Manager

Randstad DigitalAB, CA
Full-time

This range is provided by Randstad Digital.Your actual pay will be based on your skills and experience — talk with your recruiter to learn more.Role: People-First Recruiter | 10+ Years in Executive... Show more

 • Promoted

Full Stack Developer – Senior

GuuZ Consulting Services IncAB, CA
Full-time

Our government client is looking to contract a Full Stack Developer to join their Digital Design and Delivery Division (3D), formerly the Digital Innovation Office (DIO).The client requires a numbe... Show more

 • Promoted

Algebra Private Tutoring Jobs North Central Alberta

SuperprofNorth Central Alberta, Canada
CA$20.00 hourly
Full-time +1

Superprof is Canada's #1 tutoring platform, and we're actively recruiting passionate tutors! Whether you're a student, a professional, or simply someone who loves teaching, join the largest communi... Show more

 • Promoted

Geometry Private Tutoring Jobs Central Alberta

SuperprofCentral Alberta, Canada
CA$20.00 hourly
Full-time +1

Superprof is Canada's #1 tutoring platform, and we're actively recruiting passionate tutors! Whether you're a student, a professional, or simply someone who loves teaching, join the largest communi... Show more

 • Promoted

Senior Embedded Systems Integration Engineer – Defense/Aerospace

Wolf Advanced TechnologyAB, CA
Full-time

A leading technology company in defense and aerospace is seeking a full-time Senior Systems Integration Engineer to join their customer support team.The role is critical in resolving technical issu... Show more

 • Promoted

Online Data Analyst | WFH Odia Speakers (Canada)

TELUS Digital AI Data SolutionsAB, CA
Remote
Part-time

Online Data Analyst | WFH Odia Speakers (Canada).Be among the first 25 applicants.Get AI-powered advice on this job and more exclusive features.Direct message the job poster from TELUS Digital AI D... Show more

 • Promoted

Algebra Private Tutoring Jobs Alberta

SuperprofAlberta, Canada
CA$20.00 hourly
Full-time +1

Superprof is Canada's #1 tutoring platform, and we're actively recruiting passionate tutors! Whether you're a student, a professional, or simply someone who loves teaching, join the largest communi... Show more

 • Promoted

Computer programming Private Tutoring Jobs Central Alberta

SuperprofCentral Alberta, Canada
CA$20.00 hourly
Full-time +1

Superprof is Canada's #1 tutoring platform, and we're actively recruiting passionate tutors! Whether you're a student, a professional, or simply someone who loves teaching, join the largest communi... Show more

 • Promoted

Computer programming Private Tutoring Jobs Alberta

SuperprofAlberta, Canada
CA$20.00 hourly
Full-time +1

Superprof is Canada's #1 tutoring platform, and we're actively recruiting passionate tutors! Whether you're a student, a professional, or simply someone who loves teaching, join the largest communi... Show more

 • Promoted

English vocabulary Private Tutoring Jobs Central Alberta

SuperprofCentral Alberta, Canada
CA$20.00 hourly
Full-time +1

Superprof is Canada's #1 tutoring platform, and we're actively recruiting passionate tutors! Whether you're a student, a professional, or simply someone who loves teaching, join the largest communi... Show more

 • Promoted

Remote Odia Data Analyst - Part-Time Freelance

TELUS Digital AI Data SolutionsAB, CA
Remote
Part-time

A leading digital solutions company is hiring an Online Data Analyst for a remote role that enhances digital map quality.Ideal candidates are proficient in Odia and English, familiar with Canadian ... Show more

 • Promoted

Online Data Analyst - Remote Odia Speakers

TELUS Digital AI Data SolutionsAB, CA
Remote
Part-time

Online Data Analyst – Remote Odia Speakers.Online Data Analyst – Remote Odia Speakers.TELUS Digital AI Data Solutions.Posted 2 days ago – be among the first 25 applicants.Are you a detail-oriented ... Show more

 • Promoted

Senior DevOps Engineer

Targeted TalentAB, CA
Full-time

The Senior DevOps Engineer will provide hands-on experience in design and engineering efforts implementing DevOps Platforms, common set of products and services used across development and infrastr... Show more

 • Promoted

Algebra Private Tutoring Jobs Central Alberta

SuperprofCentral Alberta, Canada
CA$20.00 hourly
Full-time +1

Superprof is Canada's #1 tutoring platform, and we're actively recruiting passionate tutors! Whether you're a student, a professional, or simply someone who loves teaching, join the largest communi... Show more

 • Promoted

Customer Service Agent - 50k-60k/Year - Remote

Spade RecruitingFox Creek, Alberta
Remote
Full-time
Quick Apply

We’re looking for enthusiastic, self-driven, individuals to assist existing and prospective clients within our organization.This position will work with multiple clients throughout the day pr... Show more

 • Promoted

AI Accelerator Software Architect: Compiler & Runtime Lead

Targeted TalentAB, CA
Full-time

A leading technology firm in Alberta is seeking a Software Architect to lead software efforts for their AI inference chip.The role involves designing software interacting with the chip, optimizing ... Show more

 • Promoted

Remote Data Analyst for Latvian Speakers

TELUS Digital AI Data SolutionsAB, CA
Remote
Part-time

A global technology firm based in Canada is seeking a Remote Online Data Analyst fluent in Latvian and English.This part-time role involves enhancing digital maps through online research and data e... Show more

 • Promoted

Senior Software Engineer (Architecture)

Targeted TalentAB, CA
Full-time

Our client is a company building the world's highest-performance pure digital AI inference chip.They are seeking a Software Architect to lead their software efforts and advance the software stack t... Show more