Java Software Engineer

About The Job

Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark, General Catalyst, Peter Thiel, Adam D'Angelo, Larry Summers, and Jack Dorsey.

Position

Conversational AI Evaluator

Type

Full-time or Part-time Contract Work

Compensation

$45–$80 / hour

Location

Remote

Commitment

20+ hours / week

Role Responsibilities

• Evaluate LLM-generated responses to coding and software engineering queries for accuracy, reasoning, clarity, and completeness.
• Conduct fact‑checking using trusted public sources and authoritative references.
• Execute code and validate outputs using appropriate tools to ensure accuracy.
• Annotate model responses by identifying strengths, areas of improvement, and factual or conceptual inaccuracies.
• Assess code quality, readability, algorithmic soundness, and explanation quality.
• Apply consistent evaluation standards by following clear taxonomies, benchmarks, and detailed evaluation guidelines.

Qualifications

Must‑Have

Preferred

Application Process (Takes 20–30 mins to complete)

Resources & Support

PS : Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.

#J-18808-Ljbffr