Program evaluation Jobs in Longueuil, QC

Job Description

As a Senior Prompt Designer, you will play a defining role in shaping the quality, reliability, and evolution of MEDFAR's generative AI features — most notably CoeurWay, our AI-powered clinical scribe, and emerging AI capabilities within MYLE EMR. Reporting to the Director of UX, you will sit at the intersection of product, clinical quality, and AI craft, working closely with product designers, developers, QA, and client-facing teams to ensure that every AI-generated output meets the high standards required in a healthcare setting.

This is a senior individual contributor role with a clear growth path toward mentoring and leading a prompt engineering practice as MEDFAR's AI product surface expands.

Main Responsibilities

• Audit and re-architect CoeurWay's existing monolithic prompts into a modular chain of focused, maintainable prompts — improving robustness, debuggability, and adaptability.

• Lead the selection, evaluation, and pairing of language models to specific prompt tasks, fostering a model-agnostic approach that reduces vendor dependency and optimizes for quality and cost.

• Own the quality of AI-generated clinical outputs: define what "good" looks like, identify failure modes, and drive continuous improvement through structured validation processes.

• Evolve and re-align our existing evaluation framework, currently covering 7 clinical note quality rubrics, improving its coverage, automation potential, and alignment with clinical ground truth.

• Establish and maintain a systematic approach to output validation, including edge case identification, regression testing for prompt changes, and documentation of known limitations.

• Design and implement new prompts and evaluation suites to support new features in CoeurWay and MYLE EMR.

• Collaborate with the GenAIOps to ensure prompts are production-ready — observable, versioned, and deployable through established CI/CD processes.

• Develop a deep understanding of clinical workflows and the needs of healthcare providers, maintaining close relationships with client-facing teams and end users.

• Stay current on the rapidly evolving LLM landscape — new models, prompting techniques, evaluation methodologies — and bring relevant findings back to the team.

• Contribute to internal documentation, standards, and best practices for prompt engineering at MEDFAR.

• Mentor future prompt designers as the team grows, and participate in hiring decisions as the practice scales.

Working conditions:

• Contract: Permanent, full time (40h/week)

• Working mode: Hybrid or remote

  • Occasional in-office presence may be required during the year (for events or team meetings, for example).

  • Candidates must reside in the province of Quebec.

Qualifications

Contribute with your strengths:

• 5+ years of professional experience in AI/ML, product development, computational linguistics, or a closely related field.

• 2+ years of hands-on prompt engineering experience in production LLM systems, ideally in a B2B SaaS or regulated industry context.

• Demonstrable experience decomposing complex prompts into modular, chained architectures (e.g., using patterns such as prompt chaining, routing, reflection, or tool use).

• Experience designing or significantly improving LLM evaluation frameworks — rubric design, inter-rater alignment, and failure mode analysis.

• Strong familiarity with frontier commercial LLM APIs (OpenAI, Anthropic, Google, etc.) and the practical tradeoffs between them.

• Experience working within or alongside healthcare, clinical, or life sciences domains is a strong asset.

• Proficiency in Python or another scripting language for building eval pipelines or prompt tooling is an asset.

• Experience with self-hosted or open-weight models is an asset.

• Advanced proficiency in English, both written and spoken;

• French is also required due to frequent client interactions

.

Who you are:

• You understand that great prompt engineering is as much about thinking clearly as it is about technical execution. You break complex problems into focused, well-scoped sub-problems.

• You are model-agnostic in mindset: you care about output quality and understand that the best tool for a task may not be the most popular or most expensive one.

• You take clinical accuracy seriously. You appreciate that in a healthcare context, the quality bar is not just about user experience — it has direct implications for patient safety.

• You are comfortable operating in ambiguity and iterating quickly, while maintaining rigorous documentation of what you've tried and why.

• You are a strong collaborator who can communicate nuanced AI quality concepts to non-technical stakeholders, clinical users, and developers alike.

• You have the seniority and judgment to work with significant autonomy, while knowing when to align and escalate.

Additional Information

• Remote work and flexibility (supporting work-life balance)

• RRSP contribution

• Healthcare insurance from day one

• Paid time off: 3 weeks + 1 additional week between Christmas and New Year

• Annual training allowance to support your professional development

• An onboarding program to help you get familiar with our environment and the digital healthcare field

• All IT equipment is provided, with additional gear if needed

• Internal growth opportunities (promotions, internal mobility)

• Support from a wellness and social committee, with initiatives to foster team cohesion, mental health, and employee well-being

• A company culture focused on transparency, collaboration, and innovation

• Join a dynamic and innovative environment where your work has a real and wide-reaching impact, helping to modernize healthcare in Canada and internationally

With offices around the world, fluency in both French and English is a must at MEDFAR. Because of the need to communicate with colleagues and/or customers in other provinces or countries, bilingualism enables us to communicate in both languages while promoting the use of French.

At MEDFAR, we value diversity, equity and inclusion within our team. We are committed to providing a work environment where every individual feels respected and supported, regardless of their background, identity or abilities.As part of our commitment to a fair and inclusive recruitment process, we offer accommodation to candidates who request it. If you need accommodation during your interview, please let us know so that we can provide you with an adapted experience.

MEDFAR has voluntarily subscribed to an Equal Employment Opportunity Program (EEOP). We encourage applications from women, visible minorities, ethnic minorities, aboriginal peoples and people with disabilities. When applying, we invite you to complete this section, which enables us to implement our Equal Employment Opportunity Program (EEOP). Self-identification is not compulsory, but may enable you to benefit from hiring or promotion measures if you have the skills required for the job.

To better understand the self-identification process, please consult this guide.