GenAI Testing: LLM, RAG, Agentic AI, API and Test Automation
What are the top 3 skills required for this role?
1. GenAI, LLM, RAG and agentic AI testing
2. Python-based API and test automation, including PyTest or Robot Framework
3. LLM evaluation, prompt-injection and hallucination testing, CI/CD and observability
Additional Information:
Team size, direct reports, key deliverables, unique selling points, additional qualifications, team culture etc. Key Responsibilities
Develop and execute comprehensive test strategies for GenAI applications, including LLM-powered solutions, RAG pipelines, chatbots, copilots and agentic AI workflows.
Validate functional and non-functional requirements across prompts, model responses, APIs, user interfaces, integrations and end-to-end business workflows.
Create reusable evaluation datasets, test cases and scoring rubrics for relevance, groundedness, factual accuracy, completeness, coherence, instruction adherence and response consistency.
Test RAG solutions by evaluating retrieval quality, context precision and recall, source attribution, answer faithfulness and resistance to unsupported or hallucinated responses.
Validate agentic AI behaviour, including tool selection, function calling, reasoning flows, multi-step task completion, memory handling, fallback paths and failure recovery.
Perform adversarial, security and safety testing for prompt injection, jailbreaks, sensitive-data exposure, toxic or prohibited content, authorization bypass and insecure tool use.
Conduct performance, scalability and reliability testing for response latency, throughput, token usage, cost, concurrency, rate limits and degradation under load.
Build automated regression suites in Python and integrate GenAI evaluations, API tests and quality gates into CI/CD pipelines.
Monitor production quality through traces, evaluation metrics, user feedback and drift indicators; identify defects, analyse root causes and support remediation.
Collaborate with product, development, data science, security and business teams to define acceptance criteria, test coverage and release-readiness standards.
Required Qualifications and Skills:
Strong experience in software quality engineering, test strategy, test design, defect management and automation for enterprise applications.
Hands-on knowledge of GenAI concepts, LLM behaviour, prompt engineering, embeddings, vector databases, RAG architectures and agentic AI systems.
Proficiency in Python and experience with automation frameworks such as PyTest or Robot Framework, together with API testing tools and libraries.
Experience evaluating LLM outputs using deterministic checks, semantic similarity, model-based evaluation and human-review workflows.
Practical experience testing hallucinations, bias, robustness, non-deterministic behaviour, prompt injection, guardrails and content-safety controls.
Working knowledge of GenAI evaluation and observability tools such as Ragas, DeepEval, Promptfoo, LangSmith, Arize Phoenix or equivalent platforms.
Strong SQL skills and experience validating source data, document ingestion, chunking, metadata, embeddings and retrieval results.
Experience with GitHub, GitHub Actions or comparable CI/CD tools; familiarity with Docker, Kubernetes, AWS or Azure is desirable.
Ability to define measurable acceptance thresholds, analyse evaluation results and communicate quality risks clearly to technical and business stakeholders.
Strong analytical, investigative and problem-solving skills, with attention to responsible AI, privacy, security and regulatory expectations.
Preferred Qualifications:
Bachelor's or Master's degree in Computer Science, Software Engineering, Data Science or a related field, with at least five years of experience in quality engineering, test automation, data testing, AI/ML testing or a related discipline.
Experience testing applications in banking or another regulated environment, including model governance, audit evidence and responsible-AI controls, is an asset.
GenAI Test Engineer • Halifax, NS, Canada
Unleash the Power of Multi-Mission Excellence: Safeguarding the Seas for the World's Navies.Step into a world where cutting-edge technology meets unparalleled expertise.Ultra Maritime pioneers deli... Voir plus
Location: Halifax, Nova Scotia, CA, B3B 1E6.Onsite or Hybrid: Senior Test Specialist, Mechanical or Electrical (Permanent) – Halifax, NS.Expected Salary: $80,458 to $98,337.To determine final salar... Voir plus
Compensation: Up to $50 per task.We are seeking individuals to complete small, remote market research tasks from home.Your insights help shape future products, services, and public trends.All tasks... Voir plus
Assistant Engineer – Seaward Services – Alakai.Assistant Engineer – Seaward Services – Alakai.Hornblower is seeking an Assistant Engineer for our Seaward Services operation aboard.The vessel is a h... Voir plus
We are seeking a Senior Civil Engineer to join our Municipal Engineering team, based in either our Newfoundland and Labrador or Nova Scotia offices.This role offers the opportunity to work on meani... Voir plus
A marine service company is seeking an Assistant Engineer for their vessel operation.Key responsibilities include the upkeep and maintenance of engineering systems, supervising engineering staff, a... Voir plus
Senior Test Specialist, Mechanical or Electrical (Permanent) – Halifax, NS.Salary: expected $80,458 to $98,337.Final base salary is determined by the successful candidate’s skills and experience an... Voir plus
Location: Halifax (4 days a week on site).Develop and maintain automation test scripts using Python and Robot Framework.Execute automated and manual test cases as required.Identify logs and track d... Voir plus
Senior Technical Project Manager – Capital Markets.Hybrid in Halifax, Nova scotia, Canada.What is the opportunity Join the Shared Services Technology team in Capital Markets as we continuously deli... Voir plus
An established industry player is seeking a talented Manufacturing Engineer to join their dynamic team in Halifax, Nova Scotia.This role is pivotal in collaborating with various departments to enha... Voir plus
An innovative company is seeking a physics expert for a remote Content Developer role.This position involves training AI models by providing complex physics problems and evaluating their logic.Cand... Voir plus
Step into a vital role as an AI Optimization Engineer for HPC Systems.Focus on maximizing research efficiency by refining AI workflows within a high-performance computing environment.In this positi... Voir plus
As a Senior AI Engineer, you will deliver advanced AI solutions leveraging state-of-the‑art machine learning, generative and agentic AI technologies.You will drive the adoption of modern AI framewo... Voir plus
Advance as a Senior Production Support Engineer, ensuring system performance and reliability.You will troubleshoot critical issues and provide exceptional support for production systems.This role i... Voir plus
Senior Systems Integration and Test Engineer -WOME.Senior Systems Integration and Test Engineer -WOME.This position is part of the Rotary and Mission Systems business area, where employees across C... Voir plus
Senior Test Specialist, Mechanical or Electrical (Permanent).Expected Salary: $80,458 to $98,337.To determine final salary, Babcock considers a variety of factors including the successful Candidate... Voir plus
Senior Test Specialist, Mechanical or Electrical (Permanent).Expected Salary: $80,458 to $98,337.To determine final salary, Babcock considers a variety of factors including the successful Candidate... Voir plus
Enfield, NS (249 Aerotech Dr, Halifax).The candidate will work in the Halifax Avionics Systems Engineering (AVSE) section for the military CP‑140 Aurora and other military platforms.The candidate w... Voir plus
Lead meaningful municipal projects as a Senior Civil Engineer in our Nova Scotia or Newfoundland and Labrador offices.This position requires a highly skilled engineer ready to make a difference.As ... Voir plus
Your responsibilities will include measuring the progress of AI chatbots, evaluating their logic, and solving problems to enhance each model's quality.Requirements include a strong understanding of... Voir plus
