Validation engineer Jobs in North York, ON

About PureFacts Financial Solutions

PureFacts is the leader in the Revenue Performance Management category for wealth and asset management firms. The PureRevenue™ Platform helps organizations maximize revenue potential by connecting pricing, billing, compensation, advisor behavior, and AI-powered intelligence within a single Revenue Book of Record. By transforming fragmented revenue processes into a coordinated growth system, firms gain greater visibility, stronger pricing discipline, improved revenue capture, and more effective advisor alignment. The result is faster organic growth, improved profitability, and increased enterprise value. For more than 25 years, PureFacts has helped leading financial institutions turn revenue from an operational process into a strategic advantage.

At PureFacts, we are building an AI-native platform and company. We embed AI, intelligent automation, and agentic workflows across our products and operations to detect anomalies, surface insights, streamline repetitive work, and support faster, better decision-making. In a highly regulated industry, we believe AI must be practical, governed, and auditable—amplifying human expertise while helping our teams and clients focus on higher-value, strategic work.

About the role

The AI Engineer (LLM/Agent) will own the conversational layer that describes Purefacts’ ML model outputs to end users, develop a “Revenue Assistant” Agent from R&D through to prototype, and design context architecture grounded in client-specific pricing data. Builds evaluation and safety frameworks. This role sits at the intersection of machine learning, software engineering, and product, focusing on building intelligent systems that can reason, automate workflows, and augment human decision-making.

You will play a key role in advancing PureFacts’ AI-first strategy, developing AI-powered copilots, agents, and automation tools that reduce manual work, improve productivity, and deliver meaningful client value.

What you'll do

LLM & Agent Development

• Design and build LLM-powered applications and AI agents for both internal and client-facing use cases
• Develop solutions such as:
• • AI copilots for internal teams and clients
  • Intelligent workflow automation agents
  • Natural language interfaces for data and reporting
• Implement prompt engineering, tool usage, and agent orchestration frameworks

AI-First Automation & Use Cases

• Identify opportunities to replace manual processes with AI-driven automation
• Build systems that enable users to interact with complex data through natural language
• Develop AI solutions that enhance:
• • Revenue insights and analytics
  • Client reporting and communication
  • Operational efficiency across workflows

System Design & Integration

• Integrate LLMs into PureFacts’ SaaS platform and data systems
• Build APIs and services to support AI-powered features
• Work with data and engineering teams to ensure secure, scalable, and reliable integrations

Retrieval-Augmented Generation (RAG) & Data Integration

• Design and implement RAG pipelines using structured and unstructured data sources
• Work with:
• • Vector databases (e.g., Pinecone, Weaviate)
  • Embedding models and semantic search
• Ensure accurate, relevant, and context-aware outputs from AI systems

Evaluation, Testing & Optimization

• Develop frameworks to evaluate LLM outputs for quality, accuracy, and reliability
• Continuously optimize prompts, models, and workflows
• Monitor system performance and implement improvements

AI Infrastructure & Tooling

• Leverage and integrate tools such as:
• • OpenAI, Azure OpenAI, or similar LLM providers
  • LangChain, LlamaIndex, or agent frameworks
  • APIs, microservices, and cloud infrastructure
• Collaborate with MLOps to ensure scalable and maintainable deployments

Responsible AI & Governance

• Ensure AI solutions are secure, compliant, and aligned with responsible AI principles
• Address:
• • Data privacy and security
  • Model hallucination and reliability
  • Explainability and transparency

Cross-Functional Collaboration

• Partner with Product, Engineering, and Client teams to translate AI capabilities into business value
• Help stakeholders identify opportunities to increase efficiency and reduce manual effort
• Communicate technical concepts in a clear, practical way

Qualifications

• 2+ years building production, customer-facing LLM/GenAI applications, including vector databases, RAG pipelines , agent orchestration
• 6-8+ years of total back-end software engineering experience
• Deep, hands-on experience building with agent frameworks (e.g., Microsoft Agent Framework, Google ADK, LangGraph, etc.), including designing custom orchestration patterns beyond out-of-the-box templates
• Demonstrated ownership of evaluation frameworks and pipelines and design of deterministic guardrails/safety controls in regulated or compliance-sensitive contexts
• Experience in SaaS, fintech, or other data-driven, regulated environments strongly preferred

Technical Skills

• Expert-level Python (required)
• Deep experience with:
• • LLM APIs (OpenAI, Azure OpenAI, Anthropic, etc.), including cost/latency tradeoffs at scale
  • Agent development frameworks (LangChain, LlamaIndex, LangGraph, or similar)
  • API and microservices architecture, including integrating LLM and ML model components into larger systems
  • Data processing (SQL, Python data libraries) and pipeline design for both retrieval-quality data and model training data
• Solid working knowledge of:
• • Vector databases and embedding strategies (selection, tuning, hybrid search)
  • Cloud platforms (AWS, Azure), including deployment and scaling of AI/ML workloads

AI, Agent & ML Model Expertise

• Proven experience architecting and shipping:
• • Retrieval-Augmented Generation (RAG) systems at production scale
  • Multi-step agent workflows with error handling and recovery
  • Tool-using agents and end-to-end automation systems
• Strong, applied understanding of:
• • LLM limitations, failure modes, and optimization techniques (prompt design, fine-tuning vs. RAG tradeoffs, latency/cost optimization)
  • Evaluation methods for generative AI (offline eval sets, human-in-the-loop review, regression testing for prompt/model changes, model performance monitoring)
  • Safety and guardrail design appropriate to regulated environments (PII handling, hallucination mitigation, model bias/fairness checks, audit trails)

Operations & Reliability

• Experience operating AI/ML systems in production, including uptime, latency, and cost monitoring for both LLM and model-serving infrastructure
• Familiarity with incident response and root-cause analysis for model or agent failures (degraded outputs, drift, hallucination spikes, pipeline breakages)
• Ability to define and track model/agent health metrics (accuracy, drift, confidence calibration, usage patterns) and act on them proactively

Automation & Product Mindset

• Genuine passion for using AI/ML to automate workflows and eliminate low-value work
• Ability to independently translate ambiguous AI/ML capabilities into practical, high-impact products with minimal guidance
• Strong bias toward user experience, reliability, and real-world adoption over technical novelty

Communication & Collaboration

• Ability to work fluidly across technical and non-technical teams, including practice/business stakeholders
• Strong systems thinking — able to reason about tradeoffs across the full stack, from retrieval/model quality to UX to compliance to operational cost
• Comfortable communicating complex AI/ML concepts and tradeoffs to non-technical leadership, and mentoring less-experienced engineers

Key Success Metrics

• Successful deployment of AI-powered copilots, agents, and ML-driven products into production, owned end-to-end
• Measurable reduction in manual effort through AI/ML-driven automation
• Adoption and sustained usage of AI/ML features by internal teams and clients
• Quality, reliability, and accuracy of AI-generated and model-generated outputs, validated through rigorous evaluation and ongoing monitoring
• Model/system uptime, drift management, and operational stability in production
• Speed and quality of development/iteration cycles, and contribution to team technical standards

Education

• Degree in Computer Science, Engineering, Data Science, or related field
• Advanced degree is a plus but not required

Use of AI in our Hiring Process:

We are committed to a fair and transparent hiring process. As part of our recruitment practices, we may use artificial intelligence (AI)–based tools. Our tool is designed to assess role-related qualifications in a consistent way and is always used together with human review and decision-making. If you have any questions or concerns about the use of AI in our hiring process, or if you would prefer an alternative assessment method, please let us know and we will be happy to accommodate.

The pay range for this role is:
100,000 - 120,000 CAD per year(Toronto, Canada)

PId88f7a4323cf-30511-41623865