AI/ML Engineer – LLM Inference & Evaluation - Intermediate

Assignment: RQ11828 - Software Developer - Intermediate

Job Title: AI/ML Engineer LLM Inference & Evaluation - Intermediate

Requisition (SS): RQ11828

Start Date: 2026-11-05

Client: Government Services Integration Cluster

End Date: 2027-03-31

Office Location: 222 Jarvis St, Toronto

Organization: Government Services Integration Cluster

Ministry: Ministry of Public and Business Service Delivery and Procurement

# Business Days: 66.00

5 days onsite

Must Have:

• Background in Computer Science, Data Science, or AI/ML (student or recent grad is good).
• Familiarity with Python and libraries like pandas, scikit-learn, or PyTorch.
• Exposure to LLM frameworks (DSPy, LangChain, Hugging Face, or equivalent).
• Comfortable working with Databricks / MLflow or other cloud ML environments.
• Detail-oriented with strong organizational skills for tracking experiments.

Description

Role Overview:

• The Government of Ontario is embarking on a large Data & AI transformation initiative. Core to the transformation is the enablement of the Ontario Data Platform (ODP), which is being developed as the enterprise technology ecosystem for everything Data & AI. ODP is a modern ecosystem of data solution consisting of data ingestion technologies, storage, processing and visualization as well as solutions related to data governance, quality, marketplace, etc. ODP will initially serve all internal government clusters, ministries and program areas, but also external agencies in the future. We are looking for a data scientist to support various initiatives related to the platform as well as the execution of use cases to support the needs of various businesses within the government.

Experience and Skill Set Requirements

Key Responsibilities

• Run and Monitor Inferences
  • Execute batch inference jobs on Databricks and other environments.
  • Troubleshoot inference issues, log results, and escalate model anomalies.
• Pipeline & Evaluation Support
  • Assist with running DSPy pipelines and loading custom models/configs.
  • Support evaluation runs using internal metrics
  • Track and summarize performance metrics for inference experiments.
• Data Handling & Experimentation
  • Help prepare and clean legal text datasets for inference and evaluation.
  • Annotate outputs to support error analysis and benchmarking.
  • Maintain reproducible scripts/notebooks for inference runs.

Qualifications

• Background in Computer Science, Data Science, or AI/ML (student or recent grad is good).
• Familiarity with Python and libraries like pandas, scikit-learn, or PyTorch.
• Exposure to LLM frameworks (DSPy, LangChain, Hugging Face, or equivalent).
• Comfortable working with Databricks / MLflow or other cloud ML environments.
• Detail-oriented with strong organizational skills for tracking experiments.

Evaluation:

• Technical Skills - 50%
• LLM frameworks - 25%
• Databricks/MLflow - 25%