Staff AI Engineer

Job Description

WHAT IS THE OPPORTUNITY?

We're building the engine that judges how good our agents actually are. Claims

have to be data-driven: you can't build on what you can't see, so how can you

honestly say one version is 10% better than the last? Evaluation runs both before

we ship and after; this role owns the runtime side — judging agents live in

production, from the traces they generate serving real traffic.

The hard part is the data. Agent behaviour generates verbose traces with high

cardinality, and we need a system that can analyze them real-time, providing

actionable insights in low latency. Join us to build it: the engineering looks a

lot like site reliability engineering meeting user analytics, combining

high-throughput low latency data with evaluating user behaviour and outcomes.

WHAT WILL YOU DO?

• Build the ingestion path that takes agent traces at production volume and keeps up with it.

• Score agent behaviour live — judge quality straight from the trace as it happens, not in a batch job hours later.

• Enforce quality and safety guardrails in the request path stopping it before it reaches the user, within a fixed latency budget and at predictable cost.

• Correlate spans across services so one request reads as one trace.

• Own the experience of turning production traces back into datasets and test cases the next version is measured against.

• Set the technical direction for this burgeoning field, and push it into the open through open source contributions and conference talks.

• In this role, you will communicate and interact frequently with RBC partners and/or employees located across Canada and/or worldwide

WHAT DO YOU NEED TO SUCCEED?

Must have

• 8+ years in software or platform engineering, with 5+ in SRE, real-time data infrastructure, observability, or large-scale stream processing.

• A track record running high-volume telemetry in production with hands-on work on ingestion, storage, and query at scale.

• Distributed tracing and Open Telemetry: semantic conventions, collector configuration, span correlation across services.

• Familiarity with routing traffic on live signal, whether that's weighted load balancing, canary rollouts, or multi-armed-bandit routing.

• Turning telemetry into decisions in real time — scoring, anomaly detection, or

• rule/threshold evaluation on streaming data.

• An LLM observability platform (`Langfuse`, `MLFlow`, or equivalent) and the trace-to-evaluation feedback loop.

Nice-to-have

• A feel for the latency and backpressure trade-offs of doing work in the live request path — collectors, proxies, sidecars.

• Experience in a regulated industry (financial services, healthcare) and its constraints on AI infrastructure.

• AI security controls in the request path: prompt-injection mitigation, output filtering, PII detection.

• AI governance, model audit logging, and runtime drift detection.

• Open-source contributions or published work in observability, tracing, or LLM evaluation.

WHAT'S IN IT FOR YOU?

We thrive on the challenge to be our best, progressive thinking to keep growing, and working together to deliver trusted advice to help our clients thrive and communities prosper. We care about each other, reaching our potential, making a difference to our communities, and achieving success that is mutual.

• A comprehensive Total Rewards Program including bonuses and flexible benefits, competitive compensation, commissions, and stock where applicable

• Leaders who support your development through coaching and managing opportunities

• Ability to make a difference and lasting impact

• Work in a dynamic, collaborative, progressive, and high-performing team

• A world-class training program in financial services

• Opportunities to do challenging work

#LI-Post

#TECHPJ

Job Skills

Big Data Management, Data Mining, Data Science, Deep Learning, Machine Learning (ML), Predictive Analytics, Programming Languages

Additional Job Details

Address:

RBC WATERPARK PLACE, 88 QUEENS QUAY W:TORONTO

City:

Toronto

Country:

Canada

Work hours/week:

37.5

Employment Type:

Full time

Platform:

TECHNOLOGY AND OPERATIONS

Job Type:

Regular

Pay Type:

Salaried

Posted Date:

2026-06-22

Application Deadline:

2026-09-30

Note: Applications will be accepted until 11:59 PM on the day prior to the application deadline date above

Our Employment Opportunities

At RBC, we are guided by living shared values of Client First, Integrity, Collaboration, Respect and Excellence and winning together as One RBC. We believe an inclusive workplace that has diverse perspectives is core to our continued growth as one of the largest and most successful banks in the world. Maintaining a workplace where our employees feel supported to perform at their best, effectively collaborate, drive innovation, and grow professionally helps to bring our Purpose to life and create value for our clients and communities. RBC strives to deliver this through policies and programs intended to foster a workplace based on respect, belonging and opportunity for all.

Join our Talent Community

Stay in-the-know about great career opportunities at RBC. Sign up and get customized info on our latest jobs, career tips and Recruitment events that matter to you.

Expand your limits and create a new future together at RBC. Find out how we use our passion and drive to enhance the well-being of our clients and communities at jobs.rbc.com.

RBC is presently inviting candidates to apply for this existing vacancy. Applying to this posting allows you to express your interest in this current career opportunity at RBC. Qualified applicants may be contacted to review their resume in more detail.