Talent.com
Jay Analytix
QA Lead — AI Systems & Models TestingJay Analytix • Montreal, Quebec, Canada
QA Lead — AI Systems & Models Testing

QA Lead — AI Systems & Models Testing

Jay Analytix • Montreal, Quebec, Canada
10 days ago
Job type
  • Full-time
Job description

QA Lead — AI Systems & Models Testing

Quality Assurance • Artificial Intelligence • Contract Position

Contract

Montreal, QC

AI / ML Testing

LLM / RAG / LangChain

ABOUT THE ROLE

We are seeking an experienced QA Lead with deep expertise in AI systems testing to join our team on a contract basis in Montreal, Québec. This role sits at the intersection of quality engineering and artificial intelligence, requiring hands-on proficiency in LLM behavior analysis, RAG pipeline validation, and modern AI orchestration frameworks. You will own the end-to-end test strategy for complex AI products and help define quality standards in a rapidly evolving space.

MUST-HAVE SKILLS

  • Proven QA leadership experience designing and executing test strategies for AI/ML systems or LLM-powered applications.
  • Strong understanding of LLM internals: tokenization, embeddings, attention mechanisms, and inference behavior to anticipate and diagnose failure modes.
  • Hands-on experience with prompt engineering — constructing effective prompts, detecting hallucinations, and evaluating outputs across accuracy, tone, coherence, and bias dimensions.
  • Experience testing RAG pipelines and knowledge base integrations, including validation of data quality and retrieval accuracy as they impact model outputs.
  • Familiarity with vector database mechanics: similarity search thresholds, embedding drift, near-duplicate documents, and sparse vs. dense embeddings.
  • Practical experience with LangChain and/or LangGraph — able to read chain/graph construction code, identify failure points, and write test harnesses.
  • Ability to validate MCP (Model Context Protocol) integration points, including tool availability and error-handling scenarios.
  • Proficiency applying generative AI evaluation metrics and establishing quality thresholds appropriate for production AI systems.
  • Excellent written and verbal communication in English; bilingualism (English/French) is a plus for the Montreal market.

NICE-TO-HAVE SKILLS

  • Experience with bias detection and safety testing frameworks for AI systems.
  • Exposure to performance and scalability testing of vector databases under high load.
  • Familiarity with CI/CD pipelines for ML model deployment and automated regression testing.
  • Knowledge of responsible AI principles and AI governance frameworks.
  • Contributions to or experience with open-source AI testing or evaluation tooling (e.g., DeepEval, Ragas, PromptFlow).
  • Background in data engineering or data quality practices relevant to AI pipeline inputs.
  • Cloud platform experience (AWS, Azure, or GCP) in the context of deploying or testing AI workloads.

KEY RESPONSIBILITIES

  • Lead design and execution of comprehensive test strategies across AI systems, including prompt evaluation, output quality assessment, and bias/safety analysis.
  • Develop and maintain test harnesses for LangChain and LangGraph-based applications; review chain and graph construction code to proactively surface integration risks.
  • Validate RAG pipeline integrity — data ingestion, chunking, retrieval accuracy, and embedding consistency — and define edge-case coverage for vector database interactions.
  • Establish and track generative AI quality metrics and thresholds; report on model output quality across multiple evaluation dimensions.
  • Collaborate with ML engineers, data scientists, and product teams to embed quality practices throughout the AI development lifecycle.
  • Document test findings clearly for both technical and non-technical stakeholders.

Contract position based in Montreal, Québec, Canada • On-site / Hybrid

Create a job alert for this search

QA Lead — AI Systems & Models Testing • Montreal, Quebec, Canada

Similar jobs

AI Software Development Lead

Aspen Technology, Inc.Montreal (administrative region), QC, CA
Full-time

AspenTech seeks a passionate AI Software Development Lead to enhance its R&D capabilities.Focus on mentorship, delivering top-notch code, and fostering team collaboration.In this role, you will uti... Show more

 • Promoted

Applied AI Technical Lead — Enterprise Solutions

The Resume DatabaseMontreal (administrative region), QC, CA
Full-time

A leading AI company in Montreal is looking for a Technical Lead, Applied AI to drive technical strategy and deliver complex AI solutions for enterprise customers.This role requires a strong backgr... Show more

 • Promoted

AI Engineer

BioRenderMontreal (administrative region), QC, CA
Full-time

At BioRender, we don’t settle for average, and neither should you.We’re building tools that transform how scientists communicate complex ideas, and we need someone who’s ready to turn ambitious ide... Show more

 • Promoted

Lead Architect for Autonomous Vehicle Control Systems

CS GROUP - CanadaMontreal (administrative region), QC, CA
Full-time

Advance autonomous vehicle safety with CS Group Canada as a Lead Architect.Build essential software for safe steering and fault management in critical scenarios.CS Group Canada invites you to join ... Show more

 • Promoted

AI Systems Architect

DarkRoast DesignMontreal (administrative region), QC, CA
Full-time

Get AI-powered advice on this job and more exclusive features.We’re building an AI intelligence infrastructure embedded inside a B2B platform used by consumer brands to manage creative work, assets... Show more

 • Promoted

Senior QA Lead – Geomodelling, Cloud & Automation

FPT AmericasMontreal (administrative region), QC, CA
Full-time

A leading software provider in Canada seeks a Technical Recruiter to join the team, focusing on leading testing efforts for a cloud-based computation engine.The successful candidate will leverage e... Show more

 • Promoted

AI Lead

Super.comMontreal (administrative region), QC, CA
Full-time

This range is provided by Super.Your actual pay will be based on your skills and experience — talk with your recruiter to learn more.For our employees, our promise is that Super.We are more than a ... Show more

 • Promoted

Automation Test Analyst (Selenium/C#) - Remote & AI-Driven QA

ResonaiteMontreal (administrative region), QC, CA
Remote
Full-time

A technology consulting firm is seeking a contract Automation Tester to design and maintain automated testing solutions.Candidates should have over 4 years of QA experience, focusing on Selenium-ba... Show more

 • Promoted

Senior Applied AI/ML Developer

Autodesk, Inc.Montreal (administrative region), QC, CA
Full-time

As a Senior ML Developer you will lead and contribute to the design, analysis, and delivery of data‑driven solutions that address significant business challenges across our RAG programs.Your work w... Show more

 • Promoted

Senior AI Developer: Agentic Systems & Platform Lead

Constellation Dealer GroupMontreal (administrative region), QC, CA
Full-time

A North American software provider is seeking a Senior Developer to design and build AI-native systems that impact business outcomes.The role emphasizes collaborative system design and engineering ... Show more

 • Promoted

Leader en Tests Logiciels auprès de CAE

CAEMontreal (administrative region), QC, CA
Full-time

Intégrez CAE en tant que Leader en Tests Logiciels au sein d'une équipe dynamique pour garantir la qualité des logiciels dans l’industrie.Apportez votre expertise en validation logicielle.Le Leader... Show more

 • Promoted

AI/ML Lead (Senior Machine Learning Engineer – Full-time Leadership Role)

Aurelian Venture AIMontreal (administrative region), QC, CA
Full-time

AI/ML Lead (Senior Machine Learning Engineer – Full-time Leadership Role).Full-Time, Contract, Hands‑on Technical Leadership.CAD $180,000 – $250,000 base + significant equity + performance bonuses ... Show more

 • Promoted

Global Robotics R&D Product Lead – Agile & AI

Planiform ConveyorsTerrebonne, Lanaudière, CA
Full-time

A leader in automated conveyor systems is seeking a Bilingual Project Manager in Terrebonne.This role involves managing R&D and product development in robotics, requiring strong technical skills an... Show more

 • Promoted

Shape the Future of AI/ML with High-Impact Research Queries

Great Value HiringMontreal (administrative region), QC, CA
Full-time

Engage as a pivotal AI/ML Research Expert by identifying game-changing questions in your field.Your work could lead to groundbreaking breakthroughs we need to tackle today.This role is crafted for ... Show more

 • Promoted

Giesecke+Devrient Senior AI Systems Engineer

Giesecke+DevrientMontreal
Full-time

Drive Generative AI innovation at Giesecke+Devrient as a Senior Applied AI Engineer in the newly established AI Hub.Combine your strong Python skills with advanced AI engineering knowledge.This key... Show more

 • Promoted

Remote AI & ML Developer — LLMs, RAG & APIs

OrganimiMontreal (administrative region), QC, CA
Remote
Full-time

A leading B2B SaaS company is seeking an experienced AI / Machine Learning Developer to work on innovative AI models and systems.The candidate will train and deploy large language models and relate... Show more

 • Promoted

AI Team Lead

Maya HTTMontreal
Full-time +1

Maya HTT is a world leading developer of digital industries software solutions.The world’s top tier engineering and manufacturing organizations rely on Maya HTT’s experience and expertise to achiev... Show more

 • Promoted

Remote AI Engineer - Build Next-Gen ML Solutions

Yeah! GlobalMontreal (administrative region), QC, CA
Remote
Full-time

A global technology firm is seeking highly skilled AI Engineers to join their team.This remote position requires proficiency in programming languages like Python and experience in developing AI mod... Show more

 • Promoted

AI / ML Engineer

US MobileToronto, Montreal (Administrative Region)
Full-time

US Mobile is a venture‑backed company on a mission to revolutionize connectivity with a software platform for 5G and IoT.We are expanding rapidly and seek an AI/ML Engineer to develop and scale mac... Show more

 • Promoted

Lead Software Engineer in AI Development

AlphaSenseMontreal (administrative region), QC, CA
Full-time

Take the lead as a Software Engineer focusing on AI at a premier Market Intelligence firm.Drive innovation and architecture while fostering a collaborative and inclusive workplace.This role seeks c... Show more