Talent.com
Outlier AI
AI Evaluation Engineer (Remote)Outlier AI • Calgary, Alberta, Canada
AI Evaluation Engineer (Remote)

AI Evaluation Engineer (Remote)

Outlier AI • Calgary, Alberta, Canada
1 day ago
Job type
  • Full-time
  • Remote
Job description

About the Project

Outlier helps the world’s most innovative companies improve their AI agents by providing human feedback. Do you want to shape the future of autonomous agents like OpenClaw?

We collaborate with leading AI organizations to train Large Language Models (LLMs) to function as proactive, multi-step agents. Our projects focus on teaching these systems how to design, coordinate, and optimize complex, real-world architectural workflows.

Whether you are a passionate orchestration guru or experienced software developer — we want you to help us train the world's most advanced generative systems.

Ideal Qualifications

  • 2+ years of experience in backend engineering, AI automation, or complex systems integration.
  • Proven ability to build and maintain production-grade software with modular separation (e.g., distinct services for data parsing, logic processing, and reporting).
  • Strong command of at least two major languages (e.g., Python, JavaScript, Go, or Java) and experience working with SQL databases.
  • Practical experience building for live, non-mocked environments and handling multi-turn system interactions.
  • Outstanding attention to detail and the ability to provide clear, high-density technical feedback on complex system behaviors.

Nice to have

  • Expertise building multi-stage coordination tasks where data acquisition leads to reasoned output.
  • Hands-on experience integrating agents with live tools such as Supabase, Gmail, and various APIs to solve real-world problems.
  • High level of comfort implementing persistent state and session discovery using MEMORY.md to track agent progress.
  • Experience identifying subtle failures like privacy leaks, authority escalation, or indirect prompt injections.

Create a job alert for this search

AI Evaluation Engineer (Remote) • Calgary, Alberta, Canada

Similar jobs

Remote AI Data Quality & Evaluation Engineer

Veeva SystemsCalgary, AB, CA
Remote
Full-time

A leading cloud technology company in Toronto is seeking an experienced professional to validate and ensure the reliability of Veeva AI Agents.The ideal candidate should have expertise in data qual... Show more

 • Promoted

Remote AI Evaluation Scenario Designer

MindriftCalgary, AB, CA
Remote
Full-time

A leading AI solutions company is looking for an expert to design and evaluate test scenarios for AI agents.The ideal candidate will have a Bachelor's or Master's degree in a relevant field, experi... Show more

 • Promoted

Remote Research Engineer - Decentralized AI Systems

Yotta LabsCalgary, AB, CA
Remote
Full-time

A leading tech company is seeking a Research Engineer specializing in decentralized AI systems.The role involves designing efficient workload orchestration for AI applications across a global netwo... Show more

 • Promoted

Generative AI Developer – Retail/CPG Domain

Comprehensive Resources Inccalgary, ab, ca
Full-time

Generative AI Developer (Retail/CPG).Seeking a GenAI Developer with expertise in LLMs, RAG, and prompt engineering.Build scalable AI solutions using Python, LangChain, HuggingFace, and vector DBs l... Show more

 • Promoted

Web3 Research & Intelligence Associate (Remote - Canada)

C–CorpInvest Bankcalgary, ab, ca
Remote
Full-time

C–CorpInvest Bank is a boutique investment firm dedicated to empowering small and medium-sized enterprises (SMEs) with tailored solutions to achieve growth and success.Our team of experienced profe... Show more

 • Promoted

Remote AI Content Evaluation Specialist

Work VistaCalgary, AB, CA
Remote
Part-time

A global AI company is looking for a Content Evaluation Specialist to assess AI-generated content.This part-time, remote role requires conducting thorough evaluations and comparisons of AI response... Show more

 • Promoted

Remote Engineer for AI-Driven Solutions

KnakCalgary, AB, CA
Remote
Full-time

Advance AI tools as a Remote Engineer focused on internal solutions.Leverage full-stack development and cloud infrastructure expertise to enhance team efficiencies and technology workflows.In this ... Show more

 • Promoted

AI Engineer (Remote)

LumenaltaCalgary, AB, CA
Remote
Full-time

At Lumenalta, we partner with forward‑thinking organizations to build technology solutions that scale, delight users, and accelerate business growth.Our global teams bring curiosity, commitment, an... Show more

 • Promoted

Remote AI Development Engineer Crafting Innovative Solutions

NTT Data Americas, Inc.Calgary, AB, CA
Remote
Full-time

Elevate your career as a Remote AI Development Engineer focused on cutting-edge AI solutions.Utilize your expertise in machine learning and data engineering to transform complex business challenges... Show more

 • Promoted

RLHF Evaluation Specialist for AI Training

Rex.zoneCalgary, AB, CA
Full-time

A tech company specializing in AI is seeking an AI Trainer based in Canada.The role involves improving large language models through RLHF-style evaluations, prompt evaluation, and data labeling.Can... Show more

 • Promoted

AI Enablement Advocate & Developer Evangelist (Hybrid)

BDO CanadaCalgary, AB, CA
Full-time

A leading professional services firm in Calgary is seeking a Developer Advocate to bridge collaboration between AI Studio and Service Line Labs.This hybrid role focuses on promoting AI practices, e... Show more

 • Promoted

AI Model Evaluation Analyst

Workerbeecalgary, ab, ca
Permanent

No C2C or third-party submissions.Workerbee connects workers with employers through trusted introductions.By joining Workerbee you can be matched for project-based, contract, or permanent opportuni... Show more

 • Promoted

Remote Bilingual AI Trainer & Evaluator (EN/ZH)

DataannotationCalgary, AB, CA
Remote
Full-time

A tech company specializing in AI models is searching for a Bilingual Community Manager.This role requires fluency in both English and Chinese as you will train chatbots by evaluating their outputs... Show more

 • Promoted

Innovative Senior AI Quality Engineer for Cutting-Edge Evaluations

RoofrCalgary, AB, CA
Full-time

Be at the forefront of AI quality as a Senior AI Quality Engineer.Your mission will be to implement comprehensive testing standards while enjoying the flexibility of a remote-first workplace.This r... Show more

 • Promoted

AI Engineer

BioRenderCalgary, AB, CA
Full-time

At BioRender, we don’t settle for average, and neither should you.We’re building tools that transform how scientists communicate complex ideas, and we need someone who’s ready to turn ambitious ide... Show more

 • Promoted

Senior AI Engineer - Observability AI/ML (Remote)

Grafana LabsCalgary, AB, CA
Remote
Full-time

A leading open-source technology company is seeking a Senior AI Engineer to develop AI features for observability tools.You will collaborate with cross-functional teams to enhance user experience a... Show more

 • Promoted

Remote AI UI/UX Design Evaluator

DataAnnotationCalgary, AB, CA
Remote
Full-time

A leading AI design firm in Canada is seeking a Product UI Designer to enhance AI capabilities through UI/UX assessment.This independent contract role allows designers to critique AI-generated visu... Show more

 • Promoted

AI Development Engineer - Remote

NTT DATA, Inc.Calgary, AB, CA
Remote
Full-time

AI Development Engineer - Remote.We are seeking a highly motivated and skilled AI Engineer to join our team.This role focuses on designing, developing, and deploying AI-driven solutions that addres... Show more

 • Promoted

Remote Open Source AI Prototyping Engineer

MozillaCalgary, AB, CA
Remote
Full-time

A leading organization in open-source technology is seeking a contractor to build prototypes for open source AI infrastructure.The role involves developing working demonstrations that promote engag... Show more

 • Promoted

AI Engineer

NTT DATA North AmericaCalgary, AB, CA
Full-time

NTT DATA strives to hire exceptional, innovative, and passionate individuals who want to grow with us.We are an inclusive, adaptable, and forward‑thinking organization.We are seeking a highly motiv... Show more