Talent.com
Outlier AI
AI Evaluation Engineer (Remote)Outlier AI • Calgary, Alberta, Canada
AI Evaluation Engineer (Remote)

AI Evaluation Engineer (Remote)

Outlier AI • Calgary, Alberta, Canada
5 hours ago
Job type
  • Full-time
  • Remote
Job description

About the Project

Outlier helps the world’s most innovative companies improve their AI agents by providing human feedback. Do you want to shape the future of autonomous agents like OpenClaw?

We collaborate with leading AI organizations to train Large Language Models (LLMs) to function as proactive, multi-step agents. Our projects focus on teaching these systems how to design, coordinate, and optimize complex, real-world architectural workflows.

Whether you are a passionate orchestration guru or experienced software developer — we want you to help us train the world's most advanced generative systems.

Ideal Qualifications

  • 2+ years of experience in backend engineering, AI automation, or complex systems integration.
  • Proven ability to build and maintain production-grade software with modular separation (e.g., distinct services for data parsing, logic processing, and reporting).
  • Strong command of at least two major languages (e.g., Python, JavaScript, Go, or Java) and experience working with SQL databases.
  • Practical experience building for live, non-mocked environments and handling multi-turn system interactions.
  • Outstanding attention to detail and the ability to provide clear, high-density technical feedback on complex system behaviors.

Nice to have

  • Expertise building multi-stage coordination tasks where data acquisition leads to reasoned output.
  • Hands-on experience integrating agents with live tools such as Supabase, Gmail, and various APIs to solve real-world problems.
  • High level of comfort implementing persistent state and session discovery using MEMORY.md to track agent progress.
  • Experience identifying subtle failures like privacy leaks, authority escalation, or indirect prompt injections.

Create a job alert for this search

AI Evaluation Engineer (Remote) • Calgary, Alberta, Canada

Similar jobs

Lead Generative AI Engineer

Inizio Partnerscalgary, ab, ca
Full-time

Role - Lead Generative AI Engineer.Location - Can be based anywhere in Canada.Travel - Should be open to frequent travel to client sites.We are looking for a Staff Generative AI Engineer to design,... Show more

 • Promoted • New!

Generative AI Developer – Retail/CPG Domain

Comprehensive Resources Inccalgary, ab, ca
Full-time

Generative AI Developer (Retail/CPG).Seeking a GenAI Developer with expertise in LLMs, RAG, and prompt engineering.Build scalable AI solutions using Python, LangChain, HuggingFace, and vector DBs l... Show more

 • Promoted

Remote AI Data Quality & Evaluation Engineer

Veeva SystemsCalgary, AB, CA
Remote
Full-time

A leading cloud technology company in Toronto is seeking an experienced professional to validate and ensure the reliability of Veeva AI Agents.The ideal candidate should have expertise in data qual... Show more

 • Promoted

Remote AI Evaluation Scenario Designer

MindriftCalgary, AB, CA
Remote
Full-time

A leading AI solutions company is looking for an expert to design and evaluate test scenarios for AI agents.The ideal candidate will have a Bachelor's or Master's degree in a relevant field, experi... Show more

 • Promoted

Remote AI UI/UX Design Evaluator

DataAnnotationCalgary, Alberta, Canada
Remote
Full-time

A leading AI design firm in Canada is seeking a Product UI Designer to enhance AI capabilities through UI/UX assessment.This independent contract role allows designers to critique AI-generated visu... Show more

 • Promoted

Remote Research Engineer - Decentralized AI Systems

Yotta LabsCalgary, AB, CA
Remote
Full-time

A leading tech company is seeking a Research Engineer specializing in decentralized AI systems.The role involves designing efficient workload orchestration for AI applications across a global netwo... Show more

 • Promoted

Climate AI Research Engineer — Remote/Hybrid

ErodeCalgary, AB, CA
Remote
Full-time

A climate-tech startup is seeking a Machine Learning Research Engineer to develop AI models for weather forecasting and energy production.The role allows for remote or hybrid work in Calgary or Cha... Show more

 • Promoted

Web3 Research & Intelligence Associate (Remote - Canada)

C–CorpInvest Bankcalgary, ab, ca
Remote
Full-time

C–CorpInvest Bank is a boutique investment firm dedicated to empowering small and medium-sized enterprises (SMEs) with tailored solutions to achieve growth and success.Our team of experienced profe... Show more

 • Promoted

Remote AI Content Evaluation Specialist

Work VistaCalgary, AB, CA
Remote
Part-time

A global AI company is looking for a Content Evaluation Specialist to assess AI-generated content.This part-time, remote role requires conducting thorough evaluations and comparisons of AI response... Show more

 • Promoted

Applied AI Engineer - Levanta Labs

Levanta Labscalgary, ab, ca
Full-time

Levanta Labs is the data engineering, software development, and AI partner for high-growth B2B companies & SaaS startups.We design, build, and scale custom software platforms, internal systems, and... Show more

 • Promoted • New!

Remote Engineer for AI-Driven Solutions

KnakCalgary, AB, CA
Remote
Full-time

Advance AI tools as a Remote Engineer focused on internal solutions.Leverage full-stack development and cloud infrastructure expertise to enhance team efficiencies and technology workflows.In this ... Show more

 • Promoted

AI Engineer (Remote)

LumenaltaCalgary, AB, CA
Remote
Full-time

At Lumenalta, we partner with forward‑thinking organizations to build technology solutions that scale, delight users, and accelerate business growth.Our global teams bring curiosity, commitment, an... Show more

 • Promoted

AI Model Evaluation Analyst

Workerbeecalgary, ab, ca
Permanent

No C2C or third-party submissions.Workerbee connects workers with employers through trusted introductions.By joining Workerbee you can be matched for project-based, contract, or permanent opportuni... Show more

 • Promoted

Remote Bilingual AI Trainer & Evaluator (EN/ZH)

DataannotationCalgary, AB, CA
Remote
Full-time

A tech company specializing in AI models is searching for a Bilingual Community Manager.This role requires fluency in both English and Chinese as you will train chatbots by evaluating their outputs... Show more

 • Promoted

Innovative Senior AI Quality Engineer for Cutting-Edge Evaluations

RoofrCalgary, AB, CA
Full-time

Be at the forefront of AI quality as a Senior AI Quality Engineer.Your mission will be to implement comprehensive testing standards while enjoying the flexibility of a remote-first workplace.This r... Show more

 • Promoted

Generative AI Engineer

Galentcalgary, ab, ca
Full-time

Design, develop, and maintain scalable Python-based applications and automation workflows.Build and implement AI/LLM-powered agents and orchestration frameworks.Implement MCP (Model Context Protoco... Show more

 • Promoted • New!

Senior AI Engineer - Observability AI/ML (Remote)

Grafana LabsCalgary, AB, CA
Remote
Full-time

A leading open-source technology company is seeking a Senior AI Engineer to develop AI features for observability tools.You will collaborate with cross-functional teams to enhance user experience a... Show more

 • Promoted

AI Development Engineer - Remote

NTT DATA, Inc.Calgary, AB, CA
Remote
Full-time

AI Development Engineer - Remote.We are seeking a highly motivated and skilled AI Engineer to join our team.This role focuses on designing, developing, and deploying AI-driven solutions that addres... Show more

 • Promoted

Remote Open Source AI Prototyping Engineer

MozillaCalgary, AB, CA
Remote
Full-time

A leading organization in open-source technology is seeking a contractor to build prototypes for open source AI infrastructure.The role involves developing working demonstrations that promote engag... Show more

 • Promoted

AI Engineer

NTT DATA North AmericaCalgary, AB, CA
Full-time

NTT DATA strives to hire exceptional, innovative, and passionate individuals who want to grow with us.We are an inclusive, adaptable, and forward‑thinking organization.We are seeking a highly motiv... Show more