AI QA Analyst

Job Title: AI QA Analyst
Location: 100% Remote in Canada
Duration: 6 months (with potential to extend)
Working Hours: Eastern or Central Time Zone
Pay Rate: 40-50/h INC

About the Role
Our client is integrating generative AI across its platform to enhance meeting outcomes, automate summaries, and support decision-making. As part of this transformation, we are seeking a hands-on AI QA Analyst to help define and validate the quality of AI-generated content.
This role will focus on building scalable QA strategies, evaluating LLM outputs, and ensuring the reliability and trustworthiness of GenAI features. You’ll collaborate closely with QA, engineering, and product teams to shape the future of AI quality.

Key Responsibilities

• Design and implement QA strategies for validating LLM-generated outputs (e.g., meeting summaries, task extraction, document search).
• Evaluate GenAI features across different formatting styles (e.g., Robert’s Rules, anonymous vs. named notes).
• Develop prompt-scoring and confidence models to detect regressions and behavior shifts.
• Align GenAI validation with real-world use cases and customer expectations.
• Work within a Microsoft-based tech stack: .NET Core (C#), Azure, microservices, Cypress.
• Utilize tools like Azure OpenAI, Azure AI Search, Recall.AI, Windsurf IDE, and RAG pipelines.
• Leverage AI-based tools for regression testing, test generation, and PR reviews.
• Document QA methodologies and build reusable playbooks.
• Contribute to redefining quality metrics for GenAI (e.g., usability, trust, value).
• Support broader QA initiatives including test coverage, smoke testing, and reliability improvements.

Minimum Qualifications

• 2+ years in QA, test engineering, or quality automation in a software product environment.
• Experience validating generative AI/LLM outputs (e.g., OpenAI, Claude, Cohere).
• Strong understanding of prompt engineering, hallucination risks, and LLM behavior.
• Familiarity with prompt scoring, fuzzy matching, domain validation, and consistency testing.
• Experience with both manual and automated testing in dynamic, prompt-based systems.
• Ability to work independently in remote, fast-paced environments.
• Excellent communication and documentation skills.

Preferred Qualifications

• Experience testing GenAI in B2B SaaS, enterprise, or regulated industries (e.g., education, healthcare, finance).
• Familiarity with:Azure OpenAI, Azure AI Search, Recall.AI, Windsurf IDECypress, .NET Core/C#, Vue.js (basic)Vector databases, RAG pipelines, AI-enhanced search
• Experience using GenAI to accelerate QA (e.g., test case generation, regression automation).
• Knowledge of Agile/DevOps practices, CI/CD pipelines, and shift-left QA.
• Experience collaborating with offshore/nearshore QA teams.

If this is a role that interests you and you’d like to learn more, click apply now and a recruiter will be in touch with you to discuss this great opportunity. We look forward to speaking with you!