AI Engineer (AI System Calibration & Optimization)Robots & Pencils • Calgary, AB, CA
Les candidatures ne sont plus acceptées
AI Engineer (AI System Calibration & Optimization)
Robots & Pencils • Calgary, AB, CA
Il y a plus de 30 jours
Type de contrat
- Temps plein
Description de posteOwn end-to-end delivery of calibration systems from initial baseline to production-ready optimization workflows. Establish measurable KPIs and demonstrate accuracy improvements, latency reduction, and hallucination mitigation. Provide strategic guidance on RAG architecture improvements and retrieval parameter optimization. Accelerate client time-to-value through hands-on development and comprehensive knowledge transfer. Deliver operational playbooks and documentation enabling the client team to maintain calibration systems independently. Lead complex, multi-stakeholder calibration initiatives on-site and remotely; drive clarity, remove blockers, and keep execution on track. Set coding standards and architectural patterns for calibration components; write clear docs, runbooks, and technical specifications. Mentor client engineers through code reviews, pairing sessions, and technical workshops on DSPy, GEPA, and evaluation best practices. Make sound trade-offs under real-world constraints – Azure cost optimization, data quality, performance requirements, and security. Align delivery with Robots & Pencils' responsible AI practices and client governance requirements. Work closely with client's AI SMEs and product engineering teams to understand product catalog structure and validation workflows. Collaborate with internal R&P product, engineering, and delivery teams on calibration methodology and best practices. Share insights from client engagement to improve R&P's prompt optimization frameworks and tooling. Contribute reusable patterns, evaluation frameworks, and documentation back to R&P's core platform. Collaborate across time zones with distributed teams. Bachelor's degree in computer science, Engineering, or equivalent experience. 7+ years of professional software development with significant ownership of architecture and delivery. 3+ years of Python in ML / AI systems with a strong focus on data processing and evaluation pipelines. 2+ years building with Generative AI including hands‑on prompt engineering and optimization work. Experience with prompt optimization frameworks – DSPy strongly preferred, or similar systematic approaches to prompt improvement. Deep understanding of RAG architectures – retrieval quality, latency / cost tuning, hallucination mitigation, and evaluation methods. Hands‑on experience designing evaluation metrics and building assessment frameworks for LLM systems. Knowledge of systematic experimentation methods – A / B testing, parameter tuning, performance benchmarking. Experience with data curation, labeling workflows, and dataset quality management for AI systems. Strong Azure cloud experience with focus on AI / ML services – Azure Machine Learning, Azure AI Search, Azure OpenAI Service. Experience with Azure Data Labeling, Azure Blob Storage, and Azure infrastructure fundamentals. Understanding vector search platforms and retrieval optimization (Azure AI Search, Weaviate, Qdrant, Pinecone). Strong IaC background (Terraform or ARM templates) plus containerization and distributed systems knowledge. Solid SDLC practices – testing strategies, CI / CD, code reviews, observability, and operational excellence. Upper‑intermediate English for client communication. Experience leading complex technical projects with multiple stakeholders. Strong communication skills for technical and executive audiences. Ability to context‑switch and adapt to client environments. Willingness to travel to client sites. Direct hands‑on experience with DSPy framework and GEPA optimizer. Understanding systematic optimization principles – evolutionary algorithms, Bayesian optimization, multi‑objective optimization, and Pareto efficiency concepts. Familiarity with prompt optimization frameworks and methods – experience with any of : MIPROv2, TextGrad, EvoPrompt, AutoPrompt, or reinforcement learning approaches (GRPO, PPO). Experience with LLM‑as‑judge patterns and automated evaluation pipelines. Knowledge of advanced RAG patterns – Adaptive RAG, Self‑RAG, Corrective RAG – and retrieval evaluation methods (MRR, NDCG, precision@k). Understanding of agentic AI patterns – ReAct, Chain‑of‑Thought, Tool Use – and their application in RAG systems. Experience building evaluation dashboards with Azure Monitor, Application Insights, or similar observability tools. Familiarity with MLOps practices – model versioning, experiment tracking, metric logging for evaluation systems. Experience with AWS or GCP AI / ML platforms (Bedrock, SageMaker, Vertex AI) and cross‑cloud architecture patterns. Experience with product catalog systems, cross‑reference matching, or e‑commerce search optimization. Background in manufacturing, industrial equipment, or technical specification systems. Prior consulting or professional services experience with enterprise clients. Accountability – Owns full client engagement cycle with quality, reliability, and attention to detail. Adaptability – Thrives in dynamic, fast‑paced client environments. Collaboration – Builds strong partnerships across teams and time zones. Execution‑Focused – Delivers maintainable, scalable solutions without overengineering. Innovation‑Minded – Brings curiosity and experimentation to technology decisions. Craftsmanship – Cares deeply about documentation and code quality, architecture, and user experience.
Join to apply for the AI Engineer (AI System Calibration & Optimization) role at Robots & Pencils
1 day ago Be among the first 25 applicants
Join to apply for the AI Engineer (AI System Calibration & Optimization) role at Robots & Pencils
Get AI-powered advice on this job and more exclusive features.
Robots & Pencils is seeking an outcome-oriented AI Engineer to partner with a strategic client on a high-impact AI system calibration and optimization engagement. You'll embed directly with the client's AI and product engineering teams to improve the accuracy, reliability, and transparency of their Azure-hosted, fine-tuned GPT model through systematic prompt optimization and RAG calibration.
As an AI Engineer, you'll serve as a technical thought partner, actively coding and leveraging your software engineering experience to build calibration pipelines, optimize prompts using prompt optimization frameworks, and establish repeatable improvement workflows. You'll work on-site with the client, driving measurable outcomes that maximize their AI system performance.
Key Responsibilities
- Embed with strategic client as their technical partner for AI system calibration and prompt optimization.
- Build production-grade calibration systems using Python within the client's Azure environment.
- Implement DSPy framework and GEPA optimizer to systematically improve prompt quality and retrieval performance.
- Design and develop Golden Dataset curation workflows using Azure Data Labeling, establishing gold / silver data tier schemas.
- Create evaluation frameworks to measure model accuracy, precision / recall, latency, and hallucination rates.
- Architect prompt optimization pipelines for retrieval, context synthesis, and answer generation tailored to client needs.
- Own the path to production – evaluation pipelines, Azure ML workflows, KPI dashboards, and optimization automation.
- Iterate rapidly based on client feedback and KPI results, translating business goals into technical calibration improvements.
Outcome Ownership & Business Impact
Engineering Leadership & In-Field Delivery Excellence
Cross-Functional Collaboration
Required Skills & Qualifications
Nice to Have
Personal Competencies
Why Join Robots & Pencils?
We don't just ship features, we build digital‑first products that matter. As an AI Engineer, you'll join a team that values deep craft, cross‑functional collaboration, and relentless focus on quality. You'll work on impactful AI applications using modern technologies, while influencing engineering culture and best practices across the organization.
Seniority level
Mid‑Senior level
Employment type
Full‑time
Job function
Engineering and Information Technology
Industries
IT Services and IT Consulting
We’re unlocking community knowledge in a new way. Experts add insights directly into each article, started with the help of AI.
#J-18808-Ljbffr
Créer une alerte emploi pour cette recherche
Ai Engineer • Calgary, AB, CA
Offres similaires
Parallelz enables developers to instantly port their existing mobile apps / games to the web, without any SDKs, code changes, or engineering efforts.
Developers can improve user acquisition, organic v...Voir plus
Qualitest looking for a Senior Automation Engineer with strong technical expertise and leadership qualities.The ideal candidate should be a self-starter who can quickly contribute, lead automation ...Voir plus
We’re looking for enthusiastic, self-driven, individuals to assist existing and prospective clients within our organization.
This position will work with multiple clients throughout the day pr...Voir plus
The mission of Speechify is to make sure that reading is never a barrier to learning.Over 50 million people use Speechify's text-to-speech products to turn whatever they're reading – ...Voir plus
We are seeking a Senior Engineer, in this role you will perform a variety of senior level engineering tasks which are broad in nature and to interact with Clients to accurately interpret Client, or...Voir plus
The position is responsible for planning and directing large engineering projects, setting priorities, allocating resources, and making necessary decisions on day-to-day operating matters within th...Voir plus
Hifyre provides market intelligence for the cannabis industry, analyzing retail data to help.Our models power product recommendations, sales forecasting, and market analysis for both internal opera...Voir plus
Actalent's client is seeking a SENIOR EI&C LEAD to take a technical leadership and engineering responsibility role in the delivery of control system migration and modernization projects.Thi...Voir plus
Department : Software Engineering.Location : Calgary, Alberta, Canada.Reports to : Director, Software Engineering.Senior Control Systems Engineer with extensive experience designing and tuning control...Voir plus
You have 2-5 years’ experience in high-volume production bioinformatics workflows for WGS and WTS analysis.You've worked in a regulated clinical lab environment and have built tools that help scien...Voir plus
Acero”), a growing oil & gas facilities engineering company providing engineering, procurement and construction management expertise is currently seeking a motivated and enthusiastic indi...Voir plus
Hello me and my wife are expecting another child.Our son Gus has a lot of energy and normally we would be fine, but I am always at work and sometimes for a week at a time.We need someone to watch h...Voir plus
We're seeking top-notch engineers to join our team.As part of our group, you'll collaborate with hardware and software engineers to design, develop, and optimize software for our chip, maki...Voir plus
We believe that by embedding expert forensic reasoning into scalable AI, we can transform the way risk is understood and adjudicated in property insurance losses.
We build enterprise-grade generativ...Voir plus
Responsible for providing quality workmanship, consistent service, and technical knowledge to the customers.Performs all tasks related to the testing, diagnosing, repair, maintenance, and reconditi...Voir plus
The R&D Engineer / Technologist supports product design, research, and development.Director of Engineering, this role works with a multidisciplinary team to develop new equipment, improve exist...Voir plus
As a Senior C Engineer, you will work remotely on an hourly paid basis to review AI-generated C code, low-level systems designs, and technical explanations, as well as generate high-quality referen...Voir plus
Data Theorem is an exciting company focused on creating a more secure world for data.Rooted in a strong engineer first culture, every employee has an impact on product and direction.We are searchin...Voir plus
Senior Systems & Graphics Engineer
Parallelz • Calgary, AB, Canada
Temps plein
Dernière mise à jour : il y a plus de 30 jours • Offre sponsorisée
Senior Automation Engineer - Qualitest
Qualitest • calgary, ab, ca
Temps plein
Dernière mise à jour : il y a 5 jours • Offre sponsorisée
Work From Home - Sales Representative - 60k / Year
Spade Recruiting • High River, Alberta
Télétravail
Temps plein
Quick Apply
Dernière mise à jour : il y a plus de 30 jours • Offre sponsorisée
Senior Software Engineer, AI Model Serving - Calgary, Canada
Speechify • Calgary, AB, Canada
Temps plein
Dernière mise à jour : il y a 3 jours • Offre sponsorisée
Senior Engineer - KCA
Kestrel Power Engineering • Calgary, AB, Canada
Temps plein
Dernière mise à jour : il y a plus de 30 jours • Offre sponsorisée
Senior Controls Engineer
Propak Systems Ltd. • Airdrie, AB, Canada
Temps plein
Dernière mise à jour : il y a plus de 30 jours • Offre sponsorisée
Machine Learning Engineer - calgary
Hifyre • calgary, ab, ca
Temps plein
Dernière mise à jour : il y a 19 jours • Offre sponsorisée
Ei&c Engineer
Actalent • Calgary, AB, Canada
Temps plein
Dernière mise à jour : il y a 6 jours • Offre sponsorisée
Senior Control Systems Engineer
Eavor • Calgary, AB, Canada
Temps plein
Dernière mise à jour : il y a plus de 30 jours • Offre sponsorisée
AWS Bioinformatics Engineer (Full time remote US / Canada) - calgary
Juniper Genomics • calgary, ab, ca
Télétravail
Temps plein
Dernière mise à jour : il y a 19 jours • Offre sponsorisée
I&C Engineer
Acero Engineering Inc. • Calgary, AB, Canada
Temps plein +2
Dernière mise à jour : il y a 2 jours • Offre sponsorisée
Fredie looking for a babysitter or nanny in Okotoks
Sitly • Okotoks, CA
Temps partiel
Dernière mise à jour : il y a 1 jour • Offre sponsorisée
Senior Deep Learning Engineer
Targeted Talent • Calgary, AB, Canada
Temps plein +1
Dernière mise à jour : il y a plus de 30 jours • Offre sponsorisée
Forensic Engineer SME - calgary
Mitigateway • calgary, ab, ca
Temps plein
Dernière mise à jour : il y a plus de 30 jours • Offre sponsorisée
HET Field Technician
Wajax Limited • Rocky View County, AB, Canada
Temps plein
Dernière mise à jour : il y a plus de 30 jours • Offre sponsorisée
R&D Engineer / Technologist
Forum Canada, ULC • Calgary, AB, Canada
Temps plein
Dernière mise à jour : il y a 6 jours • Offre sponsorisée
Remote Senior C Engineer - AI Trainer
SuperAnnotate • Okotoks, Alberta, CA
Télétravail
Temps plein
Dernière mise à jour : il y a 23 jours
iOS Engineer, Observability SDK
Data Theorem • Calgary, AB, CA
Temps plein
Quick Apply
Dernière mise à jour : il y a plus de 30 jours