Staff Software Engineer

Job Description

What's the opportunity?We're seeking a seasoned Staff Software Engineer to join the RBC Borealis AI Platform team and own the end-to-end lifecycle of machine learning systems—from experimentation and validation through to high-throughput production serving at scale. You'll be the technical anchor for operationalizing vision language models and document processing systems that handle thousands of documents per minute, setting the bar for reliability, observability, and engineering excellence across our AI platform.You'll lead the design and evolution of our scalable document processing platform—a production system that combines event-driven architecture, vision language models, and cloud-native infrastructure to extract intelligence from financial documents at enterprise scale:

• ML System Operationalization: Own the production lifecycle of LLM and computer vision models, from integration and validation to serving, monitoring, and continuous improvement at 1000+ documents/minute throughput

• Platform Architecture: Design resilient microservices using FastAPI and event-driven patterns with Apache Kafka, ensuring 99.5%+ reliability for mission-critical financial document processing

• Scalable Infrastructure: Build and optimize Kubernetes-native workloads with KEDA-based autoscaling (3-50 replicas dynamically), PostgreSQL/MongoDB data layers, and S3 object storage with lifecycle management

• Observability & Reliability: Establish comprehensive monitoring, alerting, and SRE practices that provide deep visibility into model performance, system health, and business metrics across distributed services

This is a rare opportunity to shape the foundation on which Canada's largest financial institution runs its most critical AI workloads, working directly with leading researchers in machine learning while having access to rich, massive datasets and the computational resources to support groundbreaking innovationYour responsibilities include:Technical Leadership & ML Engineering

• Architect production ML pipelines that seamlessly integrate vision language models, OCR engines, and document extraction services into scalable, fault-tolerant systems

• Drive technical decisions on complex distributed systems challenges involving data consistency, exactly-once processing semantics, and sub-500ms API response times

• Collaborate closely with ML researchers to translate cutting-edge models in computer vision, NLP, and reinforcement learning into production-ready services

• Set engineering standards for model serving, A/B testing, feature flags, and gradual rollouts that enable safe, data-driven experimentation at scale

Platform Development & Innovation

• Build sophisticated retry mechanisms with exponential backoff, circuit breakers, dead-letter queues, and fallback strategies that ensure system resilience

• Implement advanced event-driven patterns across Kafka topics (ingestion, processing, callbacks, DLQ) with precise consumer group management and lag-based autoscaling

• Develop reusable frameworks and libraries for async processing, template-based document parsing, and callback orchestration that accelerate team productivity

• Lead the evaluation and adoption of emerging AI technologies, ensuring alignment with enterprise security, compliance, and data governance requirements

Cross-Functional Collaboration

• Partner with data scientists and ML researchers to understand model requirements, performance characteristics, and integration patterns for production deployment

• Work with process engineers and business stakeholders to translate financial document processing needs into robust, scalable technical solutions

• Foster strong relationships across platform, infrastructure, and security teams to deliver end-to-end capabilities that span multiple domains

• Mentor engineers on distributed systems design, event-driven architecture, ML ops best practices, and cloud-native development patterns

Strategic Problem Solving

• Navigate ambiguity in complex technical challenges, from Kafka partition strategies to LLM provider selection to autoscaling configurations

• Identify and mitigate architectural risks before they impact production, using techniques like chaos engineering, load testing, and failure mode analysis

• Provide clear, data-driven recommendations to engineering leadership on infrastructure investments, technology choices, and platform roadmap priorities

• Drive continuous improvement in system performance, cost efficiency, and developer experience through metrics-driven iteration

You're our ideal candidate if you have:

• 5-8+ years of software engineering experience with 3+ years focused on ML systems, data platforms, or high-scale distributed systems

• Deep expertise in Python and production-grade API frameworks (FastAPI, Flask, or similar) with strong software design principles

• Proven track record operationalizing ML models in production—you've integrated LLMs, vision models, or similar AI services into scalable systems

• Strong hands-on experience with event-driven architectures using Apache Kafka, RabbitMQ, or cloud-native messaging platforms

• Production experience with both SQL (PostgreSQL) and NoSQL (MongoDB, DynamoDB) databases, understanding tradeoffs and optimization strategies

• Expert-level knowledge of containerization (Docker) and Kubernetes/OpenShift orchestration, including custom resources, operators, and autoscaling

What's in it for you?

• Become part of a team that thinks progressively and works collaboratively. We care about seeing each other reach full potential;

• A comprehensive Total Rewards Program including bonuses and flexible benefits, competitive compensation, commissions, and stock options where applicable;

• Leaders who support your development through coaching and managing opportunities;

• Ability to make a difference and lasting impact from a local-to-global scale.

About RBC BorealisRBC Borealis is the driving force behind Royal Bank of Canada’s AI and data innovation. As part of Canada’s largest financial institution, we bring together a team of architects, engineers, scientists, and product experts on a mission to revolutionize finance through world-class research, solutions, and a resilient data platform. With locations across Toronto, Waterloo, Montreal, Calgary, and Vancouver, we’re at the forefront of AI research and platform development. With a focus on cutting-edge research in areas like time series forecasting, causal machine learning, and responsible AI, we are seamlessly integrating AI research and data engineering, to solve critical challenges in the financial industry. We are building intelligent, and scalable, data-driven solutions that will help communities thrive and drive innovation for our customers across the bank.