NLP · Transformers
Mitigating Answer-Length Bias in Fine-Tuned ELECTRA
Evaluates ELECTRA-small on SNLI and SQuAD, revealing lower exact-match scores for answers of six or more tokens. Oversampling improves long-answer F1 from 78.4% to 80.3%, with a trade-off in overall performance.
- 90% SNLI Accuracy
- 85.9% SQuAD F1
- 78.1% SQuAD EM
Stack PyTorch / Hugging Face / ELECTRA
Question
Benchmark scores can obscure whether a model is reasoning or pattern-matching. Two models with similar accuracy can behave very differently across slices of the data. This project examines what a strong aggregate score conceals.
Approach
Using a shared fine-tuning pipeline, google/electra-small-discriminator is trained on two benchmarks: SNLI for natural language inference and SQuAD for extractive question answering. A separate analysis script breaks results down by slice, including confusion matrices, recall, and performance as a function of answer length.
Findings
- Baselines: SNLI 90% accuracy; SQuAD 78.1% EM / 85.9% F1
- On NLI, the model systematically confused neutral with entailment
- On QA, performance fell off for long answers, with exact match dropping to 60.4% (from 81.4%) once gold answers reached six or more tokens
- Oversampling long-answer training examples lifted long-answer F1 from 78.4% to 80.3%, at a cost to overall performance
The conclusion: answer-length bias is real and partially correctable, but the correction shifts behavior elsewhere. Slice-level evaluation reveals more than a single headline number.