Skip to main content

evaluate_multiple_choice

Function evaluate_multiple_choice 

Source
pub fn evaluate_multiple_choice(
    model: &CausalLm,
    tokenizer: &Tokenizer,
    task_name: &str,
    examples: &[MultipleChoiceExample],
    limit: Option<usize>,
    device: &Device,
) -> Result<ZeroShotTaskResult>
Expand description

Evaluate conditional log-likelihood accuracy across multiple-choice examples, matching EleutherAI lm-eval scoring (acc and byte-normalized acc_norm).

ยงErrors

Returns an error if examples is empty, tokenization fails, or model forward fails.