Skip to main content

Module eval

Module eval 

Source
Expand description

Perplexity and zero-shot multiple-choice evaluation (eval.py parity).

Structs§

MultipleChoiceExample
One multiple-choice zero-shot evaluation example (boolq, piqa, hellaswag, arc, etc.).
PplResult
Perplexity measurement result for one dataset split.
ZeroShotTaskResult
Aggregated accuracy metrics for a zero-shot multiple-choice benchmark.

Functions§

evaluate_multiple_choice
Evaluate conditional log-likelihood accuracy across multiple-choice examples, matching EleutherAI lm-eval scoring (acc and byte-normalized acc_norm).
evaluate_perplexity
Evaluate autoregressive perplexity on a flat token stream matching eval.py.
load_multiple_choice_jsonl
Load multiple-choice examples from a JSONL file.