Talent.com
Autodesk
Applied AI Developer (Agent Evaluation)Autodesk • Toronto, ON, Canada
Applied AI Developer (Agent Evaluation)

Applied AI Developer (Agent Evaluation)

Autodesk • Toronto, ON, Canada
12 days ago
Job type
  • Full-time
Job description

Position Overview

As a Software Developer on the Fusion platform services team within Product Development and Manufacturing Solutions (PDMS), you'll be part of a team of technologists dedicated to creating cutting-edge AI and generative AI solutions that enhance developer productivity and experience. You'll work closely with AI engineers, software architects, and product engineering teams to build and rigorously evaluate intelligent agentic systems — including benchmarking AI agents against commercial solvers — and develop MCP (Model Context Protocol)-based tooling that integrates seamlessly with IDEs such as VS Code and Cursor.

Responsibilities

  • Develop and orchestrate multi-agent AI systems for automated test generation, test execution, and end-to-end development workflow optimization using frameworks like LangGraph, AutoGen, or the Anthropic Agent SDK (Claude Code)
  • Design and implement agentic workflows that coordinate multiple AI agents to autonomously drive test automation across UI, API, integration, and system levels, from test case synthesis to result evaluation, ensuring seamless integration with existing developer tools and MCP-compatible services
  • Build evaluation frameworks and custom benchmarks for agentic systems, including comparisons of AI agents against commercial solvers, using tools like AgentBench and Langfuse
  • Evaluate MCP server and tool performance across agentic pipelines, measuring latency, accuracy, context fidelity, and end-to-end task completion rates

Minimum Qualifications

  • BS/MS in Computer Science, Machine Learning, or a related applied AI field
  • Expertise in Python and ML frameworks (PyTorch, Transformers, scikit-learn)
  • Experience with Large Language Models applied to software understanding or test generation
  • Knowledge of AI evaluation methodologies and metrics for agentic task completion and test quality
  • Strong foundation in statistical analysis and experimental design
  • Experience with developer workflow and productivity measurement frameworks

Preferred Qualifications

  • Background in software engineering or QA with close collaboration with development teams
  • Familiarity with test automation frameworks (e.g., Playwright, Selenium, Pytest, Appium) and CI/CD pipelines
  • Experience designing benchmarks that compare AI agents against commercial or domain-specific solvers
  • Hands-on experience with MCP (Model Context Protocol), building, evaluating, and optimizing MCP servers and tool integrations within agentic pipelines
  • Experience with agentic AI frameworks including LangGraph, AutoGen, or the Anthropic Agent SDK / Claude Code
  • Knowledge in vision-language models or multi-modal AI for UI and system-level understanding and evaluation
  • Experience with Azure AI Foundry/ML or AWS cloud ML platforms
#J-18808-Ljbffr
Create a job alert for this search

Applied AI Developer (Agent Evaluation) • Toronto, ON, Canada

Similar jobs

AI Platform Engineer: Build & Evaluate AI Systems

Armilla AItoronto, on, Canada
Full-time

A leading AI-focused startup in Toronto is seeking an experienced AI Engineer to shape the future of AI risk management.In this pivotal role, you will be responsible for building core AI tools and ... Show more

 • Promoted

Applied Ai Developer - C$90,000 - C$110,000 A Year

Delta Intelligent Building TechnologiesEast York, Canada
Full-time

Seeking an Applied AI Developer to design, develop, and deploy AI-powered internal tools and agents.Responsibilities include integrating AI, ensuring responsible use, monitoring costs, and enabling... Show more

 • Promoted

Agentic AI Developer - Senior Level

Tech Talent InternationalToronto, ON, CA
Full-time

Shape the future of AI applications as a Senior Developer.Build robust systems with LLMs and autonomous agents while leveraging the benefits of remote work within the U.This role involves leading t... Show more

 • Promoted

Remote AI Evaluation Scenario Designer

MindriftToronto, ON, CA
Remote
Full-time

A leading AI solutions company is looking for an expert to design and evaluate test scenarios for AI agents.The ideal candidate will have a Bachelor's or Master's degree in a relevant field, experi... Show more

 • Promoted

Agentic AI Developer

Kumaran Systemstoronto, on, Canada
Full-time

AI agents that automate business processes, enhance customer experiences, and drive operational efficiency.The ideal candidate will have strong expertise in Agentic AI systems, AI agent development... Show more

 • Promoted

Senior, Applied AI Engineer

Headstart AItoronto, on, Canada
Full-time

This range is provided by Headstart AI.Your actual pay will be based on your skills and experience — talk with your recruiter to learn more.At Headstart our mission is to bring companies into the a... Show more

 • Promoted

Intern Researcher - AI Agent Evaluation

Huawei CanadaMarkham, York Region, CA
Full-time

Huawei Canada has an immediate 4-12 month internship opening for an Intern Researcher.Established in 2014, the Distributed Scheduling and Data Engine Lab is Huawei Cloud's technical innovation cent... Show more

 • Promoted

AI Evaluation Engineer

ZafinToronto, ON, CA
Full-time

Zafin is an AI platform company helping regulated institutions modernize how critical work is designed, governed, and delivered.Our technology enables organizations to move faster while maintaining... Show more

 • Promoted

Italian Fluency Needed for AI Evaluators

ProlificToronto, ON, CA
Full-time

Become an AI evaluator with Prolific, focusing on your Italian fluency.Earn $30 per hour while engaging in remote tasks that improve AI systems with flexible hours.We are looking for individuals wh... Show more

 • Promoted

Applied AI Implementation Specialist Toronto

Pear VCToronto, ON, CA
Full-time

Join Paxos Health in Toronto as an Applied AI Implementation Specialist, focusing on transforming healthcare workflows.You'll work hands-on to structure AI outputs, enhancing patient access to medi... Show more

 • Promoted

Generative AI Implementation Expert

SiaToronto
Full-time

Drive the future of AI solutions as a Generative AI Implementation Expert.Utilize your skills to design and optimize generative AI systems for widespread impact across various industries.You will p... Show more

 • Promoted

Applied AI Engineer

Nexxa.AIToronto, ON, CA
Full-time

Our mission is to translate deep technical breakthroughs into operational reality, solving some of the hardest systems-level problems in industry.We’re looking for a Applied AI Engineer to work as ... Show more

 • Promoted

AI Agent Evaluation Intern - Research & Reasoning

Huawei CanadaMarkham, York Region, CA
Full-time

A leading tech company in York Region, Canada, seeks an Intern Researcher for a 4-12 month position focusing on cutting-edge AI research.This role requires candidates enrolled in a Master's or PhD ... Show more

 • Promoted

Generative AI Developer

Stantec Consulting International Ltd.Toronto, ON, CA
Full-time

We are seeking a GenAI Developer to drive the design, maturation, and enterprise deployment of generative AI and agentic AI systems within Stantec’s Digital Center of Excellence (DCoE).This role wi... Show more

 • Promoted

AI Developer

Hanalytica GmbHToronto, ON, CA
Full-time

This range is provided by Hanalytica GmbH.Your actual pay will be based on your skills and experience — talk with your recruiter to learn more.Our client is looking for an exceptional.AI and machin... Show more

 • Promoted

Ai Developer – Agentic System - C$140,000 - C$190,000 A Year

XanaduNorth York, Canada
Full-time

AI Developer role focused on integrating generative AI into R&D, developing agentic systems, and building automated research processes.Requires SWE experience in AI/ML tools and production syst... Show more

 • Promoted

Applied AI Engineer

Nexxa.aiToronto, Ontario, Canada
Full-time

Our mission is to translate deep technical breakthroughs into operational reality, solving some of the hardest systems‑level problems in industry.Role Overview We’re looking for a Senior Forward De... Show more

 • Promoted

Remote AI Data Quality & Evaluation Engineer

Veeva Systemstoronto, on, Canada
Remote
Full-time

A leading cloud technology company in Toronto is seeking an experienced professional to validate and ensure the reliability of Veeva AI Agents.The ideal candidate should have expertise in data qual... Show more

 • Promoted

Expert Generative AI Developer Role

LanternToronto, ON, CA
Full-time

Leverage your skills as a Generative AI Developer with Lantern, focusing on Microsoft Azure AI technologies.Create exceptional applications that solve complex problems in the digital landscape.Lant... Show more

 • Promoted

Applied Ai Developer - C$90,000 - C$110,000 A Year

Delta ControlsEast York, Canada
Full-time

Seeking an Applied AI Developer to design, develop, and deploy internal AI tools and agents, focusing on practical, secure, and scalable AI adoption across various business functions. Show more