Snorkel AI: Frontier Model Evals
Software Engineer, AI Evaluation (Contract). Build and review coding evaluations with repeatable grading and anti-cheat checks.
50+ evaluations built200+ evaluations reviewedAnti-cheat validated
- 50+ evals built
- 200+ evals reviewed
- Reward-hacking resistant
Or skim the summary
Impact
- Built more than 50 coding evaluations with reference solutions and automated grading.
- Reviewed more than 200 evaluations for correct tests, repeatable results, and anti-cheat controls.
- Used code review and failure analysis to improve evaluation quality.
Problem
Coding evaluations must give the same result on every run. Unclear tasks, unstable tests, and weak graders produce false scores.
Approach
- Wrote reference solutions and pytest graders for each task.
- Put each evaluation in a container with fixed dependencies.
- Added checks for reward hacking and shortcut solutions.
- Reviewed task instructions, tests, and grader behavior.
- Used code review and failure analysis to improve the tasks.
PythonpytestDockerReproducible containersDeterministic oraclesCI/CD
Results
- Delivered more than 50 evaluations with repeatable grading.
- Reviewed more than 200 evaluations.
- Made the tasks clearer and harder to exploit.
