Talent.com
Mindrift
Freelance Agent Evaluation EngineerMindrift • BC, CA
Freelance Agent Evaluation Engineer

Freelance Agent Evaluation Engineer

Mindrift • BC, CA
Il y a plus de 30 jours
Salaire
50,00 $CA par heure
Type de contrat
  • Temps partiel
  • Permanent
  • Télétravail
  • Quick Apply
Description de poste

Please submit your CV in English and indicate your level of English proficiency.


Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based, not permanent employment.


We're building a dataset to evaluate AI coding agents - how well a model handles real-world developer tasks.

You'll create challenging tasks and evaluation criteria within realistic simulated environments:

  • Build realistic developer environments - a virtual company with codebase, infrastructure, and context (tickets, docs, conversations) that forms a believable development history
  • Design tasks from intermediate states of these environments - craft the prompt, define what "solved" means, and ensure the task is solvable by an AI agent
  • Write tests that verify agent solutions - accept all valid approaches and reject incorrect ones, neither too strict nor too lenient
  • Iterate on tasks and tests based on QA feedback - review agent solutions, analyze failures, and refine until the evaluation is fair and robust

What this is NOT:

  • Not data labeling
  • Not prompt engineering
  • Not writing code from scratch - the agent writes most of the code; you guide and evaluate

What we look for:

  • 5+ years in software development
  • Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis
  • Experience writing tests (functional, integration)
  • English proficiency - B2+

Why this is hard:
Frontier models are already good at coding. Creating a task that genuinely challenges the best models is non-trivial. You need to deeply understand where models fail and what scenarios reveal the difference between a good and a bad solution. Tasks have many valid solutions - writing tests that accept all correct solutions and reject incorrect ones is harder than it sounds.

Requirements and benefits

Educational qualifications

  • A Master’s Degree in Computer Science, Software Engineering, Data Science / Data Analytics, Artificial Intelligence / Machine Learning, Computational Linguistics / Natural Language Processing (NLP), Information Systems or other related fields.
  • Bachelor’s degree is accepted if only candidate has 5 years of experience in the field.

Academic and/or Professional Experience

Candidates should have a minimum of 3 years of professional experience in related roles or domain - specifically for QA-automation/testing or cybersecurity roles

How it works
Apply → Pass qualification(s) → Join a project → Complete tasks → Get paid

Compensation:
Paid per accepted task. Your rate depends on the qualification tier you reach and how efficiently you complete tasks — up to the equivalent of $50/hr. Because payment is per task, a faster pace raises your effective hourly rate.

Why this freelance opportunity might be a great fit for you?

  • Take part in a part-time, remote, freelance project that fits around your primary professional or academic commitments.
  • Work on advanced AI projects and gain valuable experience that enhances your portfolio. - Influence how future AI models understand and communicate in your field of expertise.
Créer une alerte emploi pour cette recherche

Freelance Agent Evaluation Engineer • BC, CA

Offres similaires

Senior Product Security Engineer – Remote

GithubBC, CA
Télétravail
Temps plein

Become a pivotal force in securing software as a Senior Product Security Engineer III at GitHub, where you'll focus on building and advancing security tooling and automation remotely.This hands-on ... Voir plus

 • Offre sponsorisée

Advanced PIC Design Engineer Role

Photonic Inc.BC, CA
Temps plein

Join Photonic as an Advanced PIC Design Engineer and contribute to the evolution of quantum computing in British Columbia.Utilizing your skills, design cutting-edge photonic circuits for groundbrea... Voir plus

 • Offre sponsorisée

Senior Software Engineer, Product Data Platform

Brex, bc, canada, bc, Canada
Temps plein

Brex is the AI-powered spend platform.We help companies spend with confidence with integrated corporate cards, banking, and global payments, plus intuitive software for travel and expenses.Tens of ... Voir plus

 • Offre sponsorisée

Algebra Private Tutoring Jobs Diamond

SuperprofDiamond, British Columbia, CA
Temps plein

Superprof is Canada's #1 tutoring platform, and we're actively recruiting passionate tutors! Whether you're a student, a professional, or simply someone who loves teaching, join the largest communi... Voir plus

 • Offre sponsorisée

Senior Site Reliability Engineer

Orion InnovationBC, CA
Temps plein

Senior Site Reliability Engineer (SRE) with Kubernetes and Rancher.Full-time role focused on building and maintaining highly resilient, secure systems, including in air-gapped environments.System A... Voir plus

 • Offre sponsorisée

Firmware Engineer

YochanaBC, CA
Temps plein

Strong Firmware design and debug experience using assembly and C/C++ Programming.Knowledge of NAND Flash, SSD Flash translation layer, Wear leveling, Garbage collection.Knowledge of mass storage in... Voir plus

 • Offre sponsorisée

Algebra Private Tutoring Jobs Bonnington

SuperprofBonnington, British Columbia, CA
Temps plein

Superprof is Canada's #1 tutoring platform, and we're actively recruiting passionate tutors! Whether you're a student, a professional, or simply someone who loves teaching, join the largest communi... Voir plus

 • Offre sponsorisée

Enterprise Implementation Consultant

Brex, bc, canada, bc, Canada
Temps plein

Brex is the AI-powered spend platform.We help companies spend with confidence with integrated corporate cards, banking, and global payments, plus intuitive software for travel and expenses.Tens of ... Voir plus

 • Offre sponsorisée

Design Quality Engineer II — Medical Device Compliance

VerathonBC, CA
Temps plein

A healthcare technology innovator based in British Columbia is seeking a Design Quality Engineer II to support quality aspects of new product development.The ideal candidate will have a solid backg... Voir plus

 • Offre sponsorisée

Software Engineer, Distribution

Warner Music GroupBC, CA
Temps plein

We are seeking a full‑stack Software Engineer to join our Distribution team at a pivotal moment.We are reimagining how Warner Music delivers and monetizes music globally.You’ll work on cutting‑edge... Voir plus

 • Offre sponsorisée

English Private Tutoring Jobs Bonnington

SuperprofBonnington, British Columbia, CA
Temps plein

Superprof is Canada's #1 tutoring platform, and we're actively recruiting passionate tutors! Whether you're a student, a professional, or simply someone who loves teaching, join the largest communi... Voir plus

 • Offre sponsorisée

Innovative Senior Full-Stack Engineer Driving AI-Powered Solutions

Brex, bc, canada, bc, Canada
Temps plein

Lead the development of scalable task systems as a Senior Full-Stack Software Engineer.Engage in a hybrid work model to innovate core infrastructures across multiple platforms.You will be integral ... Voir plus

 • Offre sponsorisée

Senior Full-Stack Engineer - AI-Driven Task Platform Hybrid

Brex, bc, canada, bc, Canada
Temps plein

A leading tech financial services company in Canada is seeking a Senior Full-Stack Software Engineer to develop and scale their foundational infrastructure.This role involves tackling complex chall... Voir plus

 • Offre sponsorisée

Senior Software Engineer - Healthcare / SaaS / Agentic

SycleBC, CA
Temps plein

At Sycle, we’re transforming how audiology practices operate.Our cloud-based practice management platform powers over 20,000 users worldwide - from local clinics to enterprise retailers - helping t... Voir plus

 • Offre sponsorisée

Software Development Engineer in Test (SDET)

NearSourceBritish Columbia, Canada
45,00 $CA par heure
Temps plein
Quick Apply

Job Title: Software Development Engineer in Test (SDET).Job Location: Hybrid (3 days), Vancouver, British Columbia, Canada.NearSource Technologies is seeking a.Software Development Engineer in Test... Voir plus

Engineer, PCIe Validation

TenstorrentBC, CA
Permanent

Tenstorrent is leading the industry on cutting-edge AI technology, revolutionizing performance expectations, ease of use, and cost efficiency.With AI redefining the computing paradigm, solutions mu... Voir plus

 • Offre sponsorisée

Design Quality Engineer

VerathonBC, CA
Temps plein

Verathon is a leading provider of innovative laryngoscopy and endoscopy products.We are committed to enhancing patient care through advanced medical technologies.Join our dynamic team in Burnaby, B... Voir plus

 • Offre sponsorisée

AI-Driven Software Engineer for Integrations

King River Capital Group, bc, canada, bc, Canada
Temps plein

Become an AI-Driven Software Engineer focusing on integrating innovative services for travelers.Utilize advanced AI tools to enhance integration workflows across multiple platforms.This high-owners... Voir plus

 • Offre sponsorisée

Senior PCIe Validation Engineer – FW & Hardware Bring-Up

TenstorrentBC, CA
Temps plein

A technology company in British Columbia seeks a Senior PCIe Validation Engineer to validate next-generation PCIe interfaces while collaborating closely with firmware teams.This role includes devel... Voir plus

 • Offre sponsorisée

English Private Tutoring Jobs Diamond

SuperprofDiamond, British Columbia, CA
Temps plein

Superprof is Canada's #1 tutoring platform, and we're actively recruiting passionate tutors! Whether you're a student, a professional, or simply someone who loves teaching, join the largest communi... Voir plus