GenAI Testing: LLM, RAG, Agentic AI, API and Test Automation
What are the top 3 skills required for this role?
1. GenAI, LLM, RAG and agentic AI testing
2. Python-based API and test automation, including PyTest or Robot Framework
3. LLM evaluation, prompt-injection and hallucination testing, CI/CD and observability
Additional Information:
Team size, direct reports, key deliverables, unique selling points, additional qualifications, team culture etc. Key Responsibilities
Develop and execute comprehensive test strategies for GenAI applications, including LLM-powered solutions, RAG pipelines, chatbots, copilots and agentic AI workflows.
Validate functional and non-functional requirements across prompts, model responses, APIs, user interfaces, integrations and end-to-end business workflows.
Create reusable evaluation datasets, test cases and scoring rubrics for relevance, groundedness, factual accuracy, completeness, coherence, instruction adherence and response consistency.
Test RAG solutions by evaluating retrieval quality, context precision and recall, source attribution, answer faithfulness and resistance to unsupported or hallucinated responses.
Validate agentic AI behaviour, including tool selection, function calling, reasoning flows, multi-step task completion, memory handling, fallback paths and failure recovery.
Perform adversarial, security and safety testing for prompt injection, jailbreaks, sensitive-data exposure, toxic or prohibited content, authorization bypass and insecure tool use.
Conduct performance, scalability and reliability testing for response latency, throughput, token usage, cost, concurrency, rate limits and degradation under load.
Build automated regression suites in Python and integrate GenAI evaluations, API tests and quality gates into CI/CD pipelines.
Monitor production quality through traces, evaluation metrics, user feedback and drift indicators; identify defects, analyse root causes and support remediation.
Collaborate with product, development, data science, security and business teams to define acceptance criteria, test coverage and release-readiness standards.
Required Qualifications and Skills:
Strong experience in software quality engineering, test strategy, test design, defect management and automation for enterprise applications.
Hands-on knowledge of GenAI concepts, LLM behaviour, prompt engineering, embeddings, vector databases, RAG architectures and agentic AI systems.
Proficiency in Python and experience with automation frameworks such as PyTest or Robot Framework, together with API testing tools and libraries.
Experience evaluating LLM outputs using deterministic checks, semantic similarity, model-based evaluation and human-review workflows.
Practical experience testing hallucinations, bias, robustness, non-deterministic behaviour, prompt injection, guardrails and content-safety controls.
Working knowledge of GenAI evaluation and observability tools such as Ragas, DeepEval, Promptfoo, LangSmith, Arize Phoenix or equivalent platforms.
Strong SQL skills and experience validating source data, document ingestion, chunking, metadata, embeddings and retrieval results.
Experience with GitHub, GitHub Actions or comparable CI/CD tools; familiarity with Docker, Kubernetes, AWS or Azure is desirable.
Ability to define measurable acceptance thresholds, analyse evaluation results and communicate quality risks clearly to technical and business stakeholders.
Strong analytical, investigative and problem-solving skills, with attention to responsible AI, privacy, security and regulatory expectations.
Preferred Qualifications:
Bachelor's or Master's degree in Computer Science, Software Engineering, Data Science or a related field, with at least five years of experience in quality engineering, test automation, data testing, AI/ML testing or a related discipline.
Experience testing applications in banking or another regulated environment, including model governance, audit evidence and responsible-AI controls, is an asset.
GenAI Test Engineer • Halifax, NS, Canada
Location: Halifax, Nova Scotia, CA, B3B 1E6.Onsite or Hybrid: Senior Test Specialist, Mechanical or Electrical (Permanent) – Halifax, NS.Expected Salary: $80,458 to $98,337.To determine final salar... Show more
Assistant Engineer – Seaward Services – Alakai.Assistant Engineer – Seaward Services – Alakai.Hornblower is seeking an Assistant Engineer for our Seaward Services operation aboard.The vessel is a h... Show more
Join 49North as a Systems Engineer focused on ship integration for advanced UAS systems.Use your experience to drive innovative solutions in the defense sector.In this role, you will be an integral... Show more
At Synechron, we believe in the power of digital to transform businesses for the better.Our global consulting firm combines creativity and innovative technology to deliver industry‑leading digital ... Show more
Compensation: Up to $50 per task.We are seeking individuals to complete small, remote market research tasks from home.Your insights help shape future products, services, and public trends.All tasks... Show more
A marine service company is seeking an Assistant Engineer for their vessel operation.Key responsibilities include the upkeep and maintenance of engineering systems, supervising engineering staff, a... Show more
Remote Work‑at‑Home | Canada – Bilingual Call Center Representative (English & German).MCI is a leading Business Process Outsourcing (BPO) company that delivers tailored solutions to clients.Handle... Show more
Location: Halifax (4 days a week on site).Develop and maintain automation test scripts using Python and Robot Framework.Execute automated and manual test cases as required.Identify logs and track d... Show more
A technology solutions provider is seeking a skilled developer with over 6 years of experience in multi-tier applications using the Microsoft.NET technology stack, including strong expertise in C#,... Show more
An established industry player is seeking a talented Manufacturing Engineer to join their dynamic team in Halifax, Nova Scotia.This role is pivotal in collaborating with various departments to enha... Show more
Join Synechron as a Quantitative Development Specialist in Toronto and enhance investment solutions through advanced technology.Your skills in MATLAB and Python will be central to this role.As part... Show more
Online Data Analyst - Odia Speakers (Remote).Remote opportunity with TELUS Digital AI Data Solutions.Join us in enhancing digital maps used worldwide.Conduct research and evaluation tasks in a web-... Show more
An innovative company is seeking a physics expert for a remote Content Developer role.This position involves training AI models by providing complex physics problems and evaluating their logic.Cand... Show more
Ultra Maritime seeks a dedicated Night Shift Test Engineering Electronics Technician to advance automated testing in defense systems.Leverage your electronics expertise and strong analytical skills... Show more
Step into a vital role as an AI Optimization Engineer for HPC Systems.Focus on maximizing research efficiency by refining AI workflows within a high-performance computing environment.In this positi... Show more
Advance as a Senior Production Support Engineer, ensuring system performance and reliability.You will troubleshoot critical issues and provide exceptional support for production systems.This role i... Show more
Enfield, NS (249 Aerotech Dr, Halifax).The candidate will work in the Halifax Avionics Systems Engineering (AVSE) section for the military CP‑140 Aurora and other military platforms.The candidate w... Show more
Lead meaningful municipal projects as a Senior Civil Engineer in our Nova Scotia or Newfoundland and Labrador offices.This position requires a highly skilled engineer ready to make a difference.As ... Show more
Your responsibilities will include measuring the progress of AI chatbots, evaluating their logic, and solving problems to enhance each model's quality.Requirements include a strong understanding of... Show more
Get AI-powered advice on this job and more exclusive features.If you are a talented and experienced Manufacturing Engineer, Aplin has the right opportunity for you! Our client, a leader in their fi... Show more
