Snorkel AI: Evaluations for Frontier AI Coding Models

Snorkel AI: Evaluations for Frontier AI Coding Models

Impact

• Designed and delivered 50+ software-engineering evaluations for frontier AI coding models, each with a deterministic reference solution and automated grading.
• Reviewed 200+ evaluations for correctness, test coverage, reproducibility, and resistance to reward hacking.
• Strengthened model-assessment signal by applying AI-assisted code review and systematic failure analysis.

Problem Evaluating whether a frontier model can actually ship working code requires tasks that are unambiguous, reproducible, and impossible to game. Weak specs, flaky tests, or exploitable graders produce misleading scores that overstate model capability.

Constraints

• Determinism: grading must produce the same verdict on every run.
• Reproducibility: every task pinned to a self-contained container.
• Integrity: verifiers must resist reward hacking and shortcut solutions.
• Signal: instructions and tests must align so failures are diagnostic.

Role Software Engineer, AI Evaluation (Contract). Contributor and reviewer of software-engineering evaluations.

Approach

1. Authored deterministic reference solutions and pytest grading harnesses for each task.
2. Packaged every evaluation in a reproducible container so results are environment-independent.
3. Designed anti-cheat validation to catch reward hacking and shortcut solutions.
4. Reviewed peers' evaluations for verifier strength, instruction/test alignment, and reproducibility.
5. Used AI-assisted code review and failure analysis to raise task quality and code quality.

Results

• 50+ evaluations delivered with deterministic grading and anti-cheat validation.
• 200+ evaluations reviewed, improving correctness, coverage, and reproducibility across the set.
• Clearer, harder-to-game tasks that produce more trustworthy model-capability signal.

Stack Python, pytest, Docker, reproducible containers, deterministic oracles, CI/CD