Talent.com
Jay Analytix
QA Lead — AI Systems & Models TestingJay Analytix • Montreal, Quebec, Canada
QA Lead — AI Systems & Models Testing

QA Lead — AI Systems & Models Testing

Jay Analytix • Montreal, Quebec, Canada
23 days ago
Job type
  • Full-time
Job description

QA Lead — AI Systems & Models Testing

Quality Assurance • Artificial Intelligence • Contract Position

Contract

Montreal, QC

AI / ML Testing

LLM / RAG / LangChain

ABOUT THE ROLE

We are seeking an experienced QA Lead with deep expertise in AI systems testing to join our team on a contract basis in Montreal, Québec. This role sits at the intersection of quality engineering and artificial intelligence, requiring hands-on proficiency in LLM behavior analysis, RAG pipeline validation, and modern AI orchestration frameworks. You will own the end-to-end test strategy for complex AI products and help define quality standards in a rapidly evolving space.

MUST-HAVE SKILLS

  • Proven QA leadership experience designing and executing test strategies for AI/ML systems or LLM-powered applications.
  • Strong understanding of LLM internals: tokenization, embeddings, attention mechanisms, and inference behavior to anticipate and diagnose failure modes.
  • Hands-on experience with prompt engineering — constructing effective prompts, detecting hallucinations, and evaluating outputs across accuracy, tone, coherence, and bias dimensions.
  • Experience testing RAG pipelines and knowledge base integrations, including validation of data quality and retrieval accuracy as they impact model outputs.
  • Familiarity with vector database mechanics: similarity search thresholds, embedding drift, near-duplicate documents, and sparse vs. dense embeddings.
  • Practical experience with LangChain and/or LangGraph — able to read chain/graph construction code, identify failure points, and write test harnesses.
  • Ability to validate MCP (Model Context Protocol) integration points, including tool availability and error-handling scenarios.
  • Proficiency applying generative AI evaluation metrics and establishing quality thresholds appropriate for production AI systems.
  • Excellent written and verbal communication in English; bilingualism (English/French) is a plus for the Montreal market.

NICE-TO-HAVE SKILLS

  • Experience with bias detection and safety testing frameworks for AI systems.
  • Exposure to performance and scalability testing of vector databases under high load.
  • Familiarity with CI/CD pipelines for ML model deployment and automated regression testing.
  • Knowledge of responsible AI principles and AI governance frameworks.
  • Contributions to or experience with open-source AI testing or evaluation tooling (e.g., DeepEval, Ragas, PromptFlow).
  • Background in data engineering or data quality practices relevant to AI pipeline inputs.
  • Cloud platform experience (AWS, Azure, or GCP) in the context of deploying or testing AI workloads.

KEY RESPONSIBILITIES

  • Lead design and execution of comprehensive test strategies across AI systems, including prompt evaluation, output quality assessment, and bias/safety analysis.
  • Develop and maintain test harnesses for LangChain and LangGraph-based applications; review chain and graph construction code to proactively surface integration risks.
  • Validate RAG pipeline integrity — data ingestion, chunking, retrieval accuracy, and embedding consistency — and define edge-case coverage for vector database interactions.
  • Establish and track generative AI quality metrics and thresholds; report on model output quality across multiple evaluation dimensions.
  • Collaborate with ML engineers, data scientists, and product teams to embed quality practices throughout the AI development lifecycle.
  • Document test findings clearly for both technical and non-technical stakeholders.

Contract position based in Montreal, Québec, Canada • On-site / Hybrid

Create a job alert for this search

QA Lead — AI Systems & Models Testing • Montreal, Quebec, Canada

Similar jobs

QA Lead — AI Systems & Models Testing

Jay AnalytixMontreal (administrative region), QC, CA
Full-time

QA Lead — AI Systems & Models Testing.Contract • Montreal, QC • AI / ML Testing • LLM / RAG / LangChain.We are seeking an experienced QA Lead with deep expertise in AI systems testing to join our t... Show more

 • Promoted

Senior Applied AI/ML Developer

Autodesk, Inc.montreal (administrative region), qc, Canada
Full-time

As a Senior ML Developer you will lead and contribute to the design, analysis, and delivery of data‑driven solutions that address significant business challenges across our RAG programs.Your work w... Show more

 • Promoted

AI Systems Architect

DarkRoast DesignMontreal (administrative region), QC, CA
Full-time

Get AI-powered advice on this job and more exclusive features.We’re building an AI intelligence infrastructure embedded inside a B2B platform used by consumer brands to manage creative work, assets... Show more

 • Promoted

Senior QA Lead – Geomodelling, Cloud & Automation

FPT AmericasMontreal (administrative region), QC, CA
Full-time

A leading software provider in Canada seeks a Technical Recruiter to join the team, focusing on leading testing efforts for a cloud-based computation engine.The successful candidate will leverage e... Show more

 • Promoted

AI Lead

Super.comMontreal (administrative region), QC, CA
Full-time

This range is provided by Super.Your actual pay will be based on your skills and experience — talk with your recruiter to learn more.For our employees, our promise is that Super.We are more than a ... Show more

 • Promoted

Montreal QA Lead - AI Systems Testing

Jay AnalytixMontreal (administrative region), QC, CA
Full-time

Shape the future of AI quality assurance as a QA Lead in Montreal, focusing on testing AI systems and models.This contract role incorporates prompt evaluation and bias analysis.You will employ your... Show more

 • Promoted

AI Model Engineer at Baseten

Basetenmontreal (administrative region), qc, Canada
Full-time

Empower cutting-edge AI solutions at Baseten as an AI Model Engineer.Focus on the development and optimization of APIs essential for high-performance model inference.Join Baseten’s Model Performanc... Show more

 • Promoted

Baseten Solution Architect - AI Deployment

Basetenmontreal (administrative region), qc, Canada
Full-time

Elevate AI solutions as a Solution Architect with Baseten, focusing on customer engagement and technical integrations.Work directly with Sales and clients to foster successful deployments.In this r... Show more

 • Promoted

AI Solutions Consultant

Quandrimontreal (administrative region), qc, Canada
Full-time

Our mission at Quandri is to transform insurance into a trusted and delightful experience using AI.We’re doing this by delivering an AI operating system for North America's leading insurance agenci... Show more

 • Promoted

Senior AI Machine Learning Engineer

SAP SEmontreal (administrative region), qc, Canada
Full-time

We are hiring a Senior Machine Learning Engineer Scientist to lead the development of scalable graph‑based and transformer‑based modeling systems, along with production‑grade ML pipelines.This role... Show more

 • Promoted

Senior AI Developer: Agentic Systems & Platform Lead

Constellation Dealer GroupMontreal (administrative region), QC, CA
Full-time

A North American software provider is seeking a Senior Developer to design and build AI-native systems that impact business outcomes.The role emphasizes collaborative system design and engineering ... Show more

 • Promoted

Applied AI Technical Lead — Enterprise Solutions

The Resume Databasemontreal (administrative region), qc, Canada
Full-time

A leading AI company in Montreal is looking for a Technical Lead, Applied AI to drive technical strategy and deliver complex AI solutions for enterprise customers.This role requires a strong backgr... Show more

 • Promoted

Machine Learning Architect Driving Next-Gen AI Capabilities

Kinaxismontreal (administrative region), qc, Canada
Full-time

Become a Machine Learning Architect and lead AI innovation efforts in supply chain management.Work in a flexible hybrid role focused on practical solutions for real customer challenges.This role re... Show more

 • Promoted

Architecte de solutions IA | AI Solutions Architect

Valsoft Corporationmontreal (administrative region), qc, Canada
Full-time

Manos Software Group fait partie de Valsoft Corporation, un investisseur à long terme spécialisé dans les entreprises de logiciels verticaux.Valsoft acquiert et exploite des sociétés de logiciels a... Show more

 • Promoted

AI/ML Lead (Senior Machine Learning Engineer – Full-time Leadership Role)

Aurelian Venture AIMontreal (administrative region), QC, CA
Full-time

AI/ML Lead (Senior Machine Learning Engineer – Full-time Leadership Role).Full-Time, Contract, Hands‑on Technical Leadership.CAD $180,000 – $250,000 base + significant equity + performance bonuses ... Show more

 • Promoted

Global Robotics R&D Product Lead – Agile & AI

Planiform ConveyorsTerrebonne, Lanaudière, CA
Full-time

A leader in automated conveyor systems is seeking a Bilingual Project Manager in Terrebonne.This role involves managing R&D and product development in robotics, requiring strong technical skills an... Show more

 • Promoted

Applied AI, Technical Lead, Forward Deployed AI Engineer

Mistralmontreal (administrative region), qc, Canada
Full-time

Mistral provides full-stack AI solutions: from frontier models to developer tools, applications, and compute.We partner with enterprises tackling the hardest problems—across high-stakes industries ... Show more

 • Promoted

Montreal AI Change Enablement Lead

WSPmontreal (administrative region), qc, Canada
Full-time

Become WSP's AI Change Enablement and Communications Lead in Montreal, QC, and spearhead the adoption of cutting-edge AI practices.Your role will focus on crafting persuasive communications that il... Show more

 • Promoted

Giesecke+Devrient Senior AI Systems Engineer

Giesecke+Devrientmontreal (administrative region), qc, Canada
Full-time

Drive Generative AI innovation at Giesecke+Devrient as a Senior Applied AI Engineer in the newly established AI Hub.Combine your strong Python skills with advanced AI engineering knowledge.This key... Show more

 • Promoted

Senior AI Systems Architect - Intelligence Layer & RAG

DarkRoast DesignMontreal (administrative region), QC, CA
Full-time

A leading technology firm in Canada is seeking a Senior AI Systems Architect to design the foundational intelligence layer for innovative AI-driven features.The role includes defining robust retrie... Show more