Expand description
Perplexity and zero-shot multiple-choice evaluation (eval.py parity).
Structs§
- Multiple
Choice Example - One multiple-choice zero-shot evaluation example (
boolq,piqa,hellaswag,arc, etc.). - PplResult
- Perplexity measurement result for one dataset split.
- Zero
Shot Task Result - Aggregated accuracy metrics for a zero-shot multiple-choice benchmark.
Functions§
- evaluate_
multiple_ choice - Evaluate conditional log-likelihood accuracy across multiple-choice examples,
matching EleutherAI
lm-evalscoring (accand byte-normalizedacc_norm). - evaluate_
perplexity - Evaluate autoregressive perplexity on a flat token stream matching
eval.py. - load_
multiple_ choice_ jsonl - Load multiple-choice examples from a JSONL file.