Talent.com
Outlier AI
AI Evaluation Engineer (Remote)Outlier AI • Kitchener, Ontario, Canada
No longer accepting applications
AI Evaluation Engineer (Remote)

AI Evaluation Engineer (Remote)

Outlier AI • Kitchener, Ontario, Canada
3 days ago
Job type
  • Full-time
  • Remote
Job description

About the Project

Outlier helps the world’s most innovative companies improve their AI agents by providing human feedback. Do you want to shape the future of autonomous agents like OpenClaw?

We collaborate with leading AI organizations to train Large Language Models (LLMs) to function as proactive, multi-step agents. Our projects focus on teaching these systems how to design, coordinate, and optimize complex, real-world architectural workflows.

Whether you are a passionate orchestration guru or experienced software developer — we want you to help us train the world's most advanced generative systems.

Ideal Qualifications

  • 2+ years of experience in backend engineering, AI automation, or complex systems integration.
  • Proven ability to build and maintain production-grade software with modular separation (e.g., distinct services for data parsing, logic processing, and reporting).
  • Strong command of at least two major languages (e.g., Python, JavaScript, Go, or Java) and experience working with SQL databases.
  • Practical experience building for live, non-mocked environments and handling multi-turn system interactions.
  • Outstanding attention to detail and the ability to provide clear, high-density technical feedback on complex system behaviors.

Nice to have

  • Expertise building multi-stage coordination tasks where data acquisition leads to reasoned output.
  • Hands-on experience integrating agents with live tools such as Supabase, Gmail, and various APIs to solve real-world problems.
  • High level of comfort implementing persistent state and session discovery using MEMORY.md to track agent progress.
  • Experience identifying subtle failures like privacy leaks, authority escalation, or indirect prompt injections.

Create a job alert for this search

AI Evaluation Engineer (Remote) • Kitchener, Ontario, Canada

Similar jobs

Generative AI Developer – Retail/CPG Domain

Comprehensive Resources Incguelph, on, ca
Full-time

Generative AI Developer (Retail/CPG).Seeking a GenAI Developer with expertise in LLMs, RAG, and prompt engineering.Build scalable AI solutions using Python, LangChain, HuggingFace, and vector DBs l... Show more

 • Promoted

Generative AI Developer – Retail/CPG Domain - Comprehensive Resources Inc

Comprehensive Resources Incguelph, on, ca
Full-time

Generative AI Developer (Retail/CPG).Seeking a GenAI Developer with expertise in LLMs, RAG, and prompt engineering.Build scalable AI solutions using Python, LangChain, HuggingFace, and vector DBs l... Show more

 • Promoted

Backend Engineer, AI Agents - Remote

AffirmKitchener
Remote
Full-time

A financial technology company is looking for a Software Engineer II to join their remote team in Canada.This role involves developing APIs, shipping web applications, and collaborating with produc... Show more

 • Promoted

Machine Learning Engineer for Real Estate AI

CambioKitchener, Region of Waterloo, CA
Full-time

Drive innovation in real estate sustainability as a Machine Learning Engineer at Cambio, specializing in AI solutions.This hybrid role integrates machine learning with an emphasis on climate action... Show more

 • Promoted

Senior ASIC Design & Verification Engineer for AI Inference

Targeted TalentKitchener, Region of Waterloo, CA
Full-time

A leading technology recruitment agency is seeking an experienced hardware design engineer located in Kitchener, Canada.The ideal candidate will have more than 5 years in hardware systems design, p... Show more

 • Promoted

DevOps Engineer (AI)

Affinityguelph, on, ca
Full-time

Client: Enterprise class EMR provider.On behalf of our client, Affinity is seeking a DevOps Engineer to build and evolve the infrastructure behind its flagship EMR platform.You’ll own Kubernetes-ba... Show more

 • Promoted

Director of AI - guelph

People In AIguelph, on, ca
Full-time

Director, AI / ML (Applied AI & Agentic Systems).A scaled, product-led technology company operating at the intersection of data, AI, and vertical SaaS—focused on transforming how complex, real-worl... Show more

 • Promoted

Applied AI Engineer

Jetson home incKitchener
Full-time

Jetson is on a mission to accelerate the transition of 100 million homes across North America away from fossil fuels toward sustainable energy use.We believe in a future that is 100% electric and 1... Show more

 • Promoted

Expert AI Engineer Driving AI Initiatives with Advanced Technologies

GEI ConsultantsKitchener, Region of Waterloo, CA
Full-time

Elevate your career as an AI Engineer responsible for pioneering AI solutions that transform business workflows.Focus on deploying scalable AI initiatives integrating modern technologies to meet st... Show more

 • Promoted

AI Model Evaluation Analyst

Workerbeekitchener, on, ca
Permanent

No C2C or third-party submissions.Workerbee connects workers with employers through trusted introductions.By joining Workerbee you can be matched for project-based, contract, or permanent opportuni... Show more

 • Promoted

AI Engineer - Generative & Agentic AI

BDO CanadaKitchener, Region of Waterloo, CA
Full-time

Putting people first, every day.BDO is a firm built on a foundation of positive relationships with our people and our clients.Each day, our professionals provide exceptional service, helping client... Show more

 • Promoted

Senior Software Engineer - Remote/Hybrid, AI-Enabled Impact

SkillsWaveKitchener, Region of Waterloo, CA
Remote
Full-time

A leading software development company is seeking an experienced senior software developer to join their innovative team in Kitchener.The role offers remote and hybrid work options, focusing on bui... Show more

 • Promoted

Remote Biology Tutor: Train & Evaluate AI Models

DataAnnotationGuelph, ON, CA
Remote
Full-time

An innovative firm is seeking a Biology Tutor to enhance AI models by providing complex biology problems and evaluating chatbot outputs.This role allows for flexibility in project selection and sch... Show more

 • Promoted

Senior Data Engineer — Remote/Hybrid, AI‑Driven Insights

WIN Home InspectionDelhi, ON, CA
Remote
Full-time

A leading PropTech company based in Ontario is seeking a Senior Data Engineer to join their team.The candidate will analyze complex data sets and collaborate with cross-functional teams to harness ... Show more

 • Promoted

Lead Generative AI Engineer - guelph

Inizio Partnersguelph, on, ca
Full-time

Role - Lead Generative AI Engineer.Location - Can be based anywhere in Canada.Travel - Should be open to frequent travel to client sites.We are looking for a Staff Generative AI Engineer to design,... Show more

 • Promoted

AI Model Evaluation Analyst - kitchener

Workerbeekitchener, on, ca
Permanent

No C2C or third-party submissions.Workerbee connects workers with employers through trusted introductions.By joining Workerbee you can be matched for project-based, contract, or permanent opportuni... Show more

 • Promoted

Director of AI

People In AIguelph, on, ca
Full-time

Director, AI / ML (Applied AI & Agentic Systems).A scaled, product-led technology company operating at the intersection of data, AI, and vertical SaaS—focused on transforming how complex, real-worl... Show more

 • Promoted

AI Engineer

GEI Consultants, Inc.Guelph
Full-time

The AI Engineer is responsible for the development of AI solutions, typically leveraging pretrained models and copilots, to support GEI’s priority digital and AI initiatives.This role focuses on bu... Show more

 • Promoted

Lead Generative AI Engineer - Inizio Partners

Inizio Partnersguelph, on, ca
Full-time

Role - Lead Generative AI Engineer.Location - Can be based anywhere in Canada.Travel - Should be open to frequent travel to client sites.We are looking for a Staff Generative AI Engineer to design,... Show more

 • Promoted

DevOps Engineer (AI) - Affinity

Affinityguelph, on, ca
Full-time

Client: Enterprise class EMR provider.On behalf of our client, Affinity is seeking a DevOps Engineer to build and evolve the infrastructure behind its flagship EMR platform.You’ll own Kubernetes-ba... Show more