LLM Evaluation Metrics
Overview Evaluating LLMs is challenging because quality is subjective. This guide covers automated metrics, benchmarks, and human evaluation approaches. Automated Metrics Perplexity Measures how well a model predicts text. Lower is better. import torch from transformers import GPT2LMHeadModel, GPT2Tokenizer model = GPT2LMHeadModel.from_pretrained("gpt2") tokenizer = GPT2Tokenizer.from_pretrained("gpt2") def calculate_perplexity(text): inputs = tokenizer(text, return_tensors="pt") with torch.no_grad(): outputs = model(**inputs, labels=inputs["input_ids"]) return torch.exp(outputs.loss).item() perplexity = calculate_perplexity("The quick brown fox jumps over the lazy dog.") BLEU Score Measures n-gram overlap with reference text. Used for translation. ...