A fine-tuning job finishing without an error tells you nothing about whether the resulting model is actually good — only real evaluation does.
1Overfitting Is a Real, Not Theoretical, Risk
With too few examples and too many training passes, a model can start memorizing the specific wording of your training tickets rather than learning the general priority pattern behind them. A memorizing model looks perfect on the exact examples it trained on and then performs poorly on any new, differently-worded ticket — exactly the opposite of what fine-tuning was supposed to achieve.
2Holdout Accuracy Is the Honest Number
Because a holdout set was deliberately excluded from training, a model's performance on it reflects genuine generalization rather than memorization. Reporting accuracy on the training set instead is a common shortcut that produces an inflated, unreliable number — the whole reason a portion of labeled examples has to be set aside and never trained on.
3Step-by-Step Breakdown
What Actually Happens During Training. Submitting your JSONL dataset kicks off a real training job: the provider adjusts the base model's weights over several passes (epochs) through your examples, trying to reduce the gap between its predictions and your labels. Too many epochs on too small a dataset and the model memorizes your exact examples instead of learning the general pattern — a real failure mode called overfitting.
Evaluate on a Real Holdout Set. true_labels holds the real correct priorities for tickets the model never trained on; predicted holds what the fine-tuned model actually said. Finish accuracy() so it correctly counts how many of those predictions really matched.
Why evaluate the fine-tuned model on a holdout set it never saw during training, instead of just re-checking its accuracy on the training examples themselves?
- →Accuracy on training examples can look artificially high due to memorization (overfitting), while holdout accuracy measures how well the model actually generalizes to real tickets it hasn't seen before.
- →Holdout data is only used because it makes the accuracy calculation run faster.
A Real, Evaluated Classifier. You now have a real methodology for judging whether a fine-tuned model is trustworthy — accuracy measured on data it never trained on. Module 3 is complete. Next: taking TriageAgent out of your browser entirely and deploying it behind a real AWS Lambda handler.
Level Up 🚀
Advanced cheat sheets, SEO tricks, and interview prep for this topic.
Browser Support
Fully supported.
Fully supported.
Fully supported.
Fully supported.
Accessibility (A11y)
1Report Evaluation Results With the Holdout Size Alongside the Percentage
"80% accuracy" means something very different evaluated on 5 examples versus 5,000 — always report the holdout set size next to the percentage so the number can be judged for statistical reliability.
"80% accuracy (n=5)" // vs "80% accuracy (n=5000)"SEO Implications
- 1
Target 'evaluate fine-tuned model accuracy' and 'overfitting in LLM fine-tuning' separately
Developers checking a training job's result search for the evaluation methodology and the specific failure mode as distinct concerns.
Best Practices
Compare the Fine-Tuned Model's Holdout Accuracy Against the Original Prompting Approach's Accuracy
Fine-tuning is only worth shipping if it measurably outperforms the few-shot prompt it's meant to replace — evaluate both on the exact same holdout set to make that comparison fair.
Frequent Bugs
Accidentally including holdout examples in the training file.
This lets the model see the 'unseen' evaluation data during training, inflating holdout accuracy and hiding real overfitting — keep the split enforced before training ever starts.
Real-World Examples
Silent Overfitting
A model fine-tuned on 40 examples for too many epochs can show 100% accuracy on those same 40 tickets while performing barely better than random guessing on new ones — a gap only a real holdout evaluation would reveal.
train_accuracy: 100% vs holdout_accuracy: 54%