AI Engineer (AI System Calibration & Optimization)

Join to apply for the AI Engineer (AI System Calibration & Optimization) role at Robots & Pencils

1 day ago Be among the first 25 applicants

Join to apply for the AI Engineer (AI System Calibration & Optimization) role at Robots & Pencils

Get AI-powered advice on this job and more exclusive features.

Robots & Pencils is seeking an outcome-oriented AI Engineer to partner with a strategic client on a high-impact AI system calibration and optimization engagement. You'll embed directly with the client's AI and product engineering teams to improve the accuracy, reliability, and transparency of their Azure-hosted, fine-tuned GPT model through systematic prompt optimization and RAG calibration.

As an AI Engineer, you'll serve as a technical thought partner, actively coding and leveraging your software engineering experience to build calibration pipelines, optimize prompts using prompt optimization frameworks, and establish repeatable improvement workflows. You'll work on-site with the client, driving measurable outcomes that maximize their AI system performance.

Key Responsibilities

• Embed with strategic client as their technical partner for AI system calibration and prompt optimization.
• Build production-grade calibration systems using Python within the client's Azure environment.
• Implement DSPy framework and GEPA optimizer to systematically improve prompt quality and retrieval performance.
• Design and develop Golden Dataset curation workflows using Azure Data Labeling, establishing gold / silver data tier schemas.
• Create evaluation frameworks to measure model accuracy, precision / recall, latency, and hallucination rates.
• Architect prompt optimization pipelines for retrieval, context synthesis, and answer generation tailored to client needs.
• Own the path to production – evaluation pipelines, Azure ML workflows, KPI dashboards, and optimization automation.
• Iterate rapidly based on client feedback and KPI results, translating business goals into technical calibration improvements.

Outcome Ownership & Business Impact

Engineering Leadership & In-Field Delivery Excellence

Cross-Functional Collaboration

Required Skills & Qualifications

Nice to Have

Personal Competencies

Why Join Robots & Pencils?

We don't just ship features, we build digital‑first products that matter. As an AI Engineer, you'll join a team that values deep craft, cross‑functional collaboration, and relentless focus on quality. You'll work on impactful AI applications using modern technologies, while influencing engineering culture and best practices across the organization.

Seniority level

Mid‑Senior level

Employment type

Full‑time

Job function

Engineering and Information Technology

Industries

IT Services and IT Consulting

We’re unlocking community knowledge in a new way. Experts add insights directly into each article, started with the help of AI.

#J-18808-Ljbffr