The model that matched the guess
A neural network built in R to flag MBA students at risk of not being placed. Its accuracy turned out to match always guessing "placed".
The chart
01 / Symptom
A school wanted to know, at admission, which of its MBA students might struggle to get placed and need support.
02 / Tests
Selected features with t-tests and chi-square tests, trained a neural network in R on 391 students, and evaluated it with a confusion matrix.
03 / Diagnosis
85.38% accuracy looked strong, but always guessing "placed" scores the same. The model found only 1 of the 19 students who weren't placed.
04 / Prescription
Judge the model on the students who matter: report a baseline first, and use class weights or a different threshold for the rare group.
05 / Outcome
A documented warning not to trust the headline accuracy, and a clear plan for a model that could actually find the students who need help.
How the tests ran
- Dropped fields that would leak the outcome, such as MBA marks, specialization and salary.
- Selected features with t-tests and chi-square tests (p < 0.05), applied min-max scaling and split the data 75/25.
- Trained a single-hidden-node neural network in R, with a logistic regression in Python as a comparison.
- Evaluated with a confusion matrix: accuracy, precision, recall, specificity, F1 and Youden's J.
Tools
- R / RStudio
- neuralnet
- Python
- scikit-learn
Findings
85.38%
test accuracy, exactly what always predicting "placed" scores
1 of 19
non-placed students identified
Accuracy hid the failure. When the rare group is the one that matters, judge the model on that group.
Limits
- With 391 students and one split, every metric depends on which students land in the test set.