Skip to content
Chayan Aggarwal Résumé
All case files
Case 02AcademicCourse projectAdvanced Business Analytics, team of three

The model that matched the guess

A neural network built in R to flag MBA students at risk of not being placed. Its accuracy turned out to match always guessing "placed".

Pred. placedPred. notPlacedNotplacedPlaced, predicted placed110Placed, predicted not1Not placed, predicted placed18Not placed, predicted not1
Test-set confusion matrix for 130 students, reconstructed from the reported metrics.

The chart

  1. 01 / Symptom

    A school wanted to know, at admission, which of its MBA students might struggle to get placed and need support.

  2. 02 / Tests

    Selected features with t-tests and chi-square tests, trained a neural network in R on 391 students, and evaluated it with a confusion matrix.

  3. 03 / Diagnosis

    85.38% accuracy looked strong, but always guessing "placed" scores the same. The model found only 1 of the 19 students who weren't placed.

  4. 04 / Prescription

    Judge the model on the students who matter: report a baseline first, and use class weights or a different threshold for the rare group.

  5. 05 / Outcome

    A documented warning not to trust the headline accuracy, and a clear plan for a model that could actually find the students who need help.

How the tests ran

  1. Dropped fields that would leak the outcome, such as MBA marks, specialization and salary.
  2. Selected features with t-tests and chi-square tests (p < 0.05), applied min-max scaling and split the data 75/25.
  3. Trained a single-hidden-node neural network in R, with a logistic regression in Python as a comparison.
  4. Evaluated with a confusion matrix: accuracy, precision, recall, specificity, F1 and Youden's J.

Tools

  • R / RStudio
  • neuralnet
  • Python
  • scikit-learn

Findings

  • 85.38%

    test accuracy, exactly what always predicting "placed" scores

  • 1 of 19

    non-placed students identified

  • Accuracy hid the failure. When the rare group is the one that matters, judge the model on that group.

Limits

  • With 391 students and one split, every metric depends on which students land in the test set.
Next case / 03Is this data ready for the VP's dashboard?