About the Role
This role sits at the intersection of AI safety research, LLM evaluation, and applied data science. You will design and run adversarial evaluations against production AI agents, develop quantitative frameworks to measure evaluation quality, and translate findings into actionable insights for application teams and business stakeholders. You will also mentor junior data scientists and contribute to the growth of our AI safety research practice.
Responsibilities
Design and execute adversarial evaluations of generative AI and agentic systems, probing for safety failures, policy violations, and unexpected behaviors under realistic conditions
Apply rigorous experimental design and statistical methodology to AI safety research questions, including evaluation of model robustness and the reliability of automated evaluation systems
Develop and validate quantitative frameworks for assessing LLM outputs and measuring evaluation quality at scale
Build and maintain data pipelines to process large-scale evaluation outputs, aggregate metrics, and surface trends for research and stakeholder consumption
Engage directly with internal stakeholders including application teams, security analysts, and business leaders to explain findings and translate complex analytical results into clear, actionable recommendations
Mentor and develop junior data scientists and analysts on the team
Stay current with emerging literature in adversarial ML, LLM safety, and AI evaluation; contribute to the team's evolving research agenda
Participate in special projects and perform other duties as assigned
Qualifications
Minimum 5 years of experience in data science, applied ML research, or AI evaluation roles
Hands-on experience with large language models and agentic AI systems, with working knowledge of LLM behavior, failure modes, and safety evaluation techniques
Familiarity with adversarial AI concepts including jailbreaks, prompt injection, and model robustness, whether through direct research or applied work
Strong programming skills in Python, including experience with data pipelines, large-scale experimentation, and ML libraries such as PyTorch, Hugging Face, or Scikit-learn
Solid foundation in statistical reasoning and experimental design: hypothesis formulation, significance testing, and the ability to identify confounds and methodological flaws in existing analyses
Experience in financial services, AI safety, trust and safety, or a regulated industry is a strong plus
Clear, confident communication skills with the ability to explain technical findings to both research peers and non-technical stakeholders
Bachelor's degree in Computer Science, Statistics, Applied Mathematics, or a related quantitative field; Master's degree or equivalent research experience preferred
How We Work
Vanguard has implemented a hybrid working model for the majority of our crew members, designed to capture the benefits of enhanced flexibility while enabling in-person learning, collaboration, and connection. We believe our mission-driven and highly collaborative culture is a critical enabler to support long-term client outcomes and enrich the employee experience.
Senior AI Scientist • Toronto, ON, CA
The Senior AI Engineer will work on the development and deployment of cutting‑edge machine learning models, drive innovation, and collaborate with cross‑functional teams to support business growth ... Show more
A fintech AI platform is seeking a Senior Enterprise Sales Executive in Toronto.The role involves driving enterprise sales cycles, managing complex deals, and serving as a trusted advisor to financ... Show more
Join MSCI as a Senior AI Researcher, enhancing investment research methodologies through innovative AI solutions.Collaborate with a leading team focusing on key emerging economic themes in a flexib... Show more
Empower your career as a Senior Applied Scientist driving cutting-edge research.Utilize AI and machine learning to address industry needs in a flexible, hybrid environment.This role entails leading... Show more
Make a difference in AI technology with NTT DATA as a Remote Senior AI Engineer from Toronto, Ontario.Focus on designing AI models and enhancing existing architectures for impactful solutions.As a ... Show more
The Emerging Risks & Opportunities R&D team is building MSCI’s next-generation capability for measuring and classifying portfolio exposure to economy-wide structural themes.Our team develops AI-nat... Show more
Elevate collaboration at Fitch Group as a Senior Engineer focused on AI innovations in Toronto.Spearhead projects with Microsoft 365 to drive impactful business outcomes.Become part of Fitch’s Mode... Show more
The Emerging Risks & Opportunities R&D team is building MSCI’s next-generation capability for measuring and classifying portfolio exposure to economy-wide structural themes.Our team develops AI-nat... Show more
Boston Consulting Group partners with leaders in business and society to tackle their most important challenges and capture their greatest opportunities.BCG was the pioneer in business strategy whe... Show more
Elevate your career as a Senior AI Engineer, where you’ll create innovative features powered by AI technologies in TypeScript, Go, and AWS.In this pivotal role, you will design and build workflows ... Show more
Elevate user experiences at Autodesk as a Senior Scientist specializing in AI personalization systems.This hands-on role focuses on delivering cutting-edge agentic solutions.In this role, you will ... Show more
We believe conversations will become the #1 way to shop.At Gorgias, we’re building the platform that makes this real: a unified AI agent that sells, supports, and re‑engages customers across the en... Show more
Drive cutting-edge AI evaluation as a Senior Research Scientist at Cohere.Innovate new methods to measure large language models' performance and set the standard for future capabilities.As a Senior... Show more
Senior Machine Learning Engineer.Direct message the job poster from MIR Digital Solutions.At MIR Digital, we’re more than a company - we’re a collective of innovators, builders, and change-makers.W... Show more
SecurityScorecard is the global leader in cybersecurity ratings, with over 12 million companies continuously rated, operating in 64 countries.Founded in 2013 by security and risk experts Dr.Alex Ya... Show more
Are you interested in shaping the future of Advertising and B2B Sales? We are a growing team with an exciting AI‑first charter and need your passion, innovative thinking, and creativity to help tak... Show more
The Senior Data Scientist role provides analytical and technical support in the scope of executing targeted campaigns for our customers.The Senior Data Scientist will be the subject matter expert i... Show more
Maximum up to 5% travel across North America.North American dealership business management software provider specializing in solutions for outdoor power equipment, agricultural and powersports deal... Show more
We do this by building an industry-leading incident management platform that allows companies around the world to consistently and quickly resolve incidents.We are not simply transforming an indust... Show more
Emerging Risks & Opportunities R&D team is building MSCI's next-generation capability for measuring and classifying portfolio exposure to economy-wide structural themes.Our team develops AI-native ... Show more
