AI QA Analyst

Job Title : AI QA Analyst

Location : 100% Remote in Canada

Duration : 6 months (with potential to extend)

Working Hours : Eastern or Central Time Zone

Pay Rate : 40-50 / h INC

About the Role

Our client is integrating generative AI across its platform to enhance meeting outcomes, automate summaries, and support decision-making. As part of this transformation, we are seeking a hands-on AI QA Analyst to help define and validate the quality of AI-generated content.

This role will focus on building scalable QA strategies, evaluating LLM outputs, and ensuring the reliability and trustworthiness of GenAI features. You’ll collaborate closely with QA, engineering, and product teams to shape the future of AI quality.

Key Responsibilities

• Design and implement QA strategies for validating LLM-generated outputs (e.g., meeting summaries, task extraction, document search).
• Evaluate GenAI features across different formatting styles (e.g., Robert’s Rules, anonymous vs. named notes).
• Develop prompt-scoring and confidence models to detect regressions and behavior shifts.
• Align GenAI validation with real-world use cases and customer expectations.
• Work within a Microsoft-based tech stack : .NET Core (C#), Azure, microservices, Cypress.
• Utilize tools like Azure OpenAI, Azure AI Search, Recall.AI, Windsurf IDE, and RAG pipelines.
• Leverage AI-based tools for regression testing, test generation, and PR reviews.
• Document QA methodologies and build reusable playbooks.
• Contribute to redefining quality metrics for GenAI (e.g., usability, trust, value).
• Support broader QA initiatives including test coverage, smoke testing, and reliability improvements.

Minimum Qualifications

Preferred Qualifications

If this is a role that interests you and you’d like to learn more, click apply now and a recruiter will be in touch with you to discuss this great opportunity. We look forward to speaking with you!