Senior GenAI QA Engineer - Toronto
- Full-time
Salary: $104,000 - 124,800 per year
Requirements:
- 4+ years of experience in software QA engineering, SDET, or AI/ML engineering, with a strong foundation in API testing and automation.
- 2+ years of hands-on experience evaluating production GenAI/LLM applications, including RAG-based or Agentic AI systems.
- Strong Python skills for test automation and CI/CD integration.
- In-depth understanding of AI quality, hallucination risk, guardrail evaluation, and adversarial or red-team testing.
- Ability to turn conceptual AI evaluation into practical, measurable, repeatable metrics.
- Experience with or familiarity with Python, AWS Bedrock, Azure OpenAI, Anthropic Claude, LangGraph, LangChain, Semantic Kernel, Promptfoo, Langfuse, and DeepEval.
Responsibilities:
- Lead quality assurance and evaluation strategy for our Agentic AI platform, with a focus on safety, hallucination risk, and PII/PHI protection.
- Develop automated evaluation tools in Python and integrate quality checks into CI/CD pipelines.
- Create practical, repeatable evaluation frameworks and test datasets for non-deterministic AI systems.
- Test the effectiveness of AI guardrails using functional, adversarial, and red-team approaches.
- Own the core software QA foundation for APIs, backend services, and user-facing workflows.
Technologies:
- Agentic AI
- AI
- AI Agents
- API
- AWS
- Azure
- Backend
- CI/CD
- Support
- LLM
- Langfuse
- Python
- RAG
- Cloud
More:
We are seeking a Senior GenAI QA Engineer for a live, production-grade Agentic AI platform in the healthcare domain. This hands-on, high-impact role involves close collaboration with senior AI engineers to support the safety and reliability of our evolving AI agents. The position is a three-month contract with a strong possibility of extension or conversion to hire. It is remote in Canada, with occasional on-site work in downtown Toronto; the listed work arrangement is hybrid remote in Toronto, Ontario. Compensation is $50–$60 per hour, with expected hours capped at 40 per week.
last updated 47 week of 2026