Talent.com
Mindrift
Senior Python Engineer - AI Coding Agent Evaluation (Freelance)Mindrift • BC, CA
Senior Python Engineer - AI Coding Agent Evaluation (Freelance)

Senior Python Engineer - AI Coding Agent Evaluation (Freelance)

Mindrift • BC, CA
23 days ago
Salary
CA$200.00 daily
Job type
  • Part-time
  • Permanent
  • Remote
  • Quick Apply
Job description

Please submit your CV in English and indicate your level of English proficiency.


Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based, not permanent employment.What this opportunity involves:
We're building a dataset to evaluate AI coding agents - how well a model handles real-world developer tasks.


You'll create challenging tasks and evaluation criteria within realistic simulated environments:

  • Build realistic developer environments - a virtual company with codebase, infrastructure, and context (tickets, docs, conversations) that forms a believable development history
  • Design tasks from intermediate states of these environments - craft the prompt, define what "solved" means, and ensure the task is solvable by an AI agent
  • Write tests that verify agent solutions - accept all valid approaches and reject incorrect ones, neither too strict nor too lenient
  • Iterate on tasks and tests based on QA feedback - review agent solutions, analyze failures, and refine until the evaluation is fair and robust

What this is NOT:

  • Not data labeling
  • Not prompt engineering
  • Not writing code from scratch - the agent writes most of the code; you guide and evaluate

What we look for:

  • 8+ years in software development
  • Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis
  • Experience writing tests (functional, integration)
  • English proficiency - B2+

Why this is hard:
Frontier models are already good at coding. Creating a task that genuinely challenges the best models is non-trivial. You need to deeply understand where models fail and what scenarios reveal the difference between a good and a bad solution. Tasks have many valid solutions - writing tests that accept all correct solutions and reject incorrect ones is harder than it sounds.

How it works
Apply → Pass qualification(s) → Join a project → Complete tasks → Get paidEffort estimate


Tasks for this project are estimated to take 30 hours to complete, depending on complexity. This is an estimate and not a schedule requirement; you choose when and how to work. Tasks must be submitted by the deadline and meet the listed acceptance criteria to be accepted.

Compensation:
Up to $200/hr equivalent, depending on level and pace. Tasks are estimated at ~30 hours each; you set your own schedule.

Create a job alert for this search

Senior Python Engineer - AI Coding Agent Evaluation (Freelance) • BC, CA

Similar jobs

Senior Software Engineer, Product Data Platform — Hybrid

Brex, bc, canada, bc, Canada
Full-time

A financial technology company in Vancouver is seeking a Senior Software Engineer for their Product Data Platform team.This role involves analyzing and optimizing query performance and designing ro... Show more

 • Promoted

Senior Sales Engineer - Data & AI Security

Veeam Software, bc, canada, bc, Canada
Full-time

Veeam, following its acquisition of Securiti AI - the leader in AI-powered data security posture management (DSPM) - is seeking a Senior Sales Engineer to drive technical leadership in our sales te... Show more

 • Promoted

Lead Software Engineer for Developer Experience and Automation

OmnissaBC, CA
Full-time

Elevate developer efficiency as a Senior Software Engineer specializing in automation tools.Drive impactful solutions across a hybrid model, enhancing product quality through robust processes.In th... Show more

 • Promoted

Senior Software Engineer, Product Data Platform

Brex, bc, canada, bc, Canada
Full-time

Brex is the AI-powered spend platform.We help companies spend with confidence with integrated corporate cards, banking, and global payments, plus intuitive software for travel and expenses.Tens of ... Show more

 • Promoted

Senior DevOps Engineer — Cloud, CI/CD & Observability

Remarcable, bc, canada, bc, Canada
Full-time

A growing SaaS company in British Columbia is seeking a Senior DevOps Engineer to streamline software deployment through CI/CD pipelines and optimize cloud infrastructure.The ideal candidate will h... Show more

 • Promoted

Onboarding Engineering Lead — AI-Powered, Scalable

Brex, bc, canada, bc, Canada
Full-time

A leading financial technology company is seeking an Engineering Leader in Vancouver to spearhead the development of systems that enhance customer onboarding.This role involves strategic thinking, ... Show more

 • Promoted

Senior Manager, Solutions Architecture - Data & AI

TELUS Digital, bc, canada, bc, Canada
Full-time

Welcome to TELUS Digital — where innovation drives impact at a global scale.As an award-winning digital product consultancy and the digital division of TELUS, one of Canada’s largest telecommunicat... Show more

 • Promoted

Senior Software Engineer – Transforming Transit

Swiftly, Inc., bc, canada, bc, Canada
Full-time

Shape the future of public transit at Swiftly as a Senior Software Engineer focused on IoT and edge devices.Collaborate to develop innovative solutions that enhance service delivery.At Swiftly, we ... Show more

 • Promoted

Senior Software Engineer

King River Capital Group, bc, canada, bc, Canada
Full-time

As a Senior Software Engineer on our Ancillaries squad, you will build integrations that expand what we can offer travelers—connecting external providers to our platform so partners like Booking.Sk... Show more

 • Promoted

Engineering Leader — Growth, Salesforce & AI

Brex, bc, canada, bc, Canada
Full-time

A technology-focused finance company in Canada is seeking an experienced engineering leader to drive growth through technical guidance and team management.You will oversee project execution for the... Show more

 • Promoted

Remote Senior Engineer – AI & SaaS at Sycle

SycleBC, CA
Remote
Full-time

Transform audiology practice management at Sycle as a Remote Senior Software Engineer, focusing on AI-driven development using Node.This role enhances patient care through innovative technology.Syc... Show more

 • Promoted

Senior Software Engineer, Full Stack

Brex, bc, canada, bc, Canada
Full-time

Brex is the AI-powered spend platform.We help companies spend with confidence with integrated corporate cards, banking, and global payments, plus intuitive software for travel and expenses.Tens of ... Show more

 • Promoted

System Engineer - Golang & Kubernetes CRDs

PortBlueSkyBC, CA
Full-time

System Engineer – Golang & Kubernetes CRDs.Are you an expert Go engineer who understands Kubernetes beyond ordinary controller development? Do CRDs, watches, informers, admission webhooks, resource... Show more

 • Promoted

Senior Full-Stack Engineer - AI-Driven Task Platform Hybrid

Brex, bc, canada, bc, Canada
Full-time

A leading tech financial services company in Canada is seeking a Senior Full-Stack Software Engineer to develop and scale their foundational infrastructure.This role involves tackling complex chall... Show more

 • Promoted

Innovative Senior Full-Stack Engineer Driving AI-Powered Solutions

Brex, bc, canada, bc, Canada
Full-time

Lead the development of scalable task systems as a Senior Full-Stack Software Engineer.Engage in a hybrid work model to innovate core infrastructures across multiple platforms.You will be integral ... Show more

 • Promoted

Senior Software Engineer - Healthcare / SaaS / Agentic

SycleBC, CA
Full-time

At Sycle, we’re transforming how audiology practices operate.Our cloud-based practice management platform powers over 20,000 users worldwide - from local clinics to enterprise retailers - helping t... Show more

 • Promoted

Senior System Engineer - Golang & Kubernetes CRDs

PortBlueSkyBC, CA
Full-time

Elevate your career at PortBlueSky as a Senior System Engineer, focusing on Golang and Kubernetes CRDs.This role demands expertise in systems engineering for a Kubernetes-native enterprise automati... Show more

 • Promoted

Senior Software Engineer

Cut+DryBritish Columbia, British Columbia, CA
Full-time

Cut+Dry is a fast-growing FoodTech startup searching for flexible go-getters who welcome the challenge of meeting the needs of a rapidly expanding business.Our company is revolutionizing the $300B ... Show more

AI-Driven Software Engineer for Integrations

King River Capital Group, bc, canada, bc, Canada
Full-time

Become an AI-Driven Software Engineer focusing on integrating innovative services for travelers.Utilize advanced AI tools to enhance integration workflows across multiple platforms.This high-owners... Show more

 • Promoted

Senior DevOps Engineer

Remarcable, bc, canada, bc, Canada
Full-time

Location: Greater Vancouver, British Columbia, Canada.Design, implement, and own CI/CD pipelines across multiple services to streamline software development and deployment.Maintain, optimize, and a... Show more