NLP · Transformers
Mitigating Answer-Length Bias in Fine-Tuned ELECTRA
A single ELECTRA-small model is fine-tuned on SNLI and SQuAD, then evaluated past its headline scores. The model degrades on long answer spans, with exact match dropping from 81% to 60% past six tokens. Targeted oversampling recovers some long-answer performance at a measurable cost to overall accuracy.
- 90% SNLI Accuracy
- 85.9% SQuAD F1
- 78.1% SQuAD EM
Stack PyTorch / Hugging Face / ELECTRA
Question
Benchmark scores can obscure whether a model is reasoning or pattern-matching. Two models with similar accuracy can behave very differently across slices of the data. This project examines what a strong aggregate score conceals.
Approach
Using a shared fine-tuning pipeline, google/electra-small-discriminator is trained on two benchmarks: SNLI for natural language inference and SQuAD for extractive question answering. A separate analysis script breaks results down by slice, including confusion matrices, recall, and performance as a function of answer length.
Findings
- Baselines: SNLI 90% accuracy; SQuAD 78.1% EM / 85.9% F1
- On NLI, the model systematically confused neutral with entailment
- On QA, performance fell off for long answers, with exact match dropping to 60.4% (from 81.4%) once gold answers reached six or more tokens
- Oversampling long-answer training examples lifted long-answer F1 from 78.4% to 80.3%, at a cost to overall performance
The conclusion: answer-length bias is real and partially correctable, but the correction shifts behavior elsewhere. Slice-level evaluation reveals more than a single headline number.