Java Software Engineer

About The Job

Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .

Position : AI Model Evaluator

Type : Full‑time or Part‑time Contract Work

Compensation : $60–$100 / hour

Location : Remote

Role Responsibilities

• Evaluate LLM-generated responses to coding and software engineering queries for accuracy, reasoning, clarity, and completeness.
• Conduct fact‑checking using trusted public sources and authoritative references.
• Execute code and validate outputs using appropriate tools to conduct accuracy testing.
• Annotate model responses by identifying strengths, areas of improvement, and factual or conceptual inaccuracies.
• Assess code quality, readability, algorithmic soundness, and explanation quality.
• Ensure model responses align with expected conversational behavior and system guidelines.
• Apply consistent evaluation standards by following clear taxonomies, benchmarks, and detailed evaluation guidelines.

Qualifications

Must‑Have

Preferred

Application Process (Takes 20–30 mins to complete)

Resources & Support

PS : Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.

#J-18808-Ljbffr