LLM Evaluator (Model Response Analyst)

About the role

We are looking for a detail‑oriented LLM Evaluator to assess and improve the performance of large language models. Working remotely with a global team, you will analyze AI‑generated content for factual accuracy, coherence, safety, bias, and alignment with our guidelines.

Key responsibilities

• Evaluate and rank model‑generated text using complex rubrics covering factuality, coherence, safety, instruction‑following, and creativity.
• Compare multiple model responses to the same prompt, select the preferred output, and justify your choice.
• Provide concise feedback to modeling and training teams about recurring failure patterns.
• Design adversarial prompts to expose biased, harmful, or insecure model behavior.
• Collaborate with QA to refine evaluation guidelines and calibrate scoring standards across the team.
• Investigate underperforming outputs, hypothesize root causes, and flag novel model behaviors for research.

Required profile

• Minimum 2 years of professional experience in computational linguistics, data analysis, technical writing, NLP quality assurance, or cognitive science.
• Bachelor’s degree in computer science or a related field.
• Proven ability to craft prompts that test model limits and to explain why outputs are good or bad.
• Experience with Reinforcement Learning from Human Feedback (RLHF) data collection and evaluation consistency.
• Familiarity with IAA scoring, calibration sessions, dataset annotation, and A/B testing for AI models.

Required skills

• Prompt engineering
• Data annotation for LLMs
• RLHF data handling
• A/B testing concepts applied to AI