← The six schools

Divination · data and game AI

Bobathon

A Kaggle competition inside a machine-learning course at ESILV: predict how far Parkinson's disease has progressed in patients the model has never seen. Team of three, worked through IBM's agent manager, Bob.

The problem

About 44,600 clinical visits from roughly 5,600 patients. For each visit, predict the motor score of the disease measured without medication. The test patients never appear in training, so the model has to generalise to new people, not remember known ones.

Working with Bob

The course required every submission to come out of Bob, IBM's agent manager, with each result logged in Skore Hub by Probabl. Bob is not a chat assistant like Claude or ChatGPT. It runs a fixed workflow of skills: explore the data, write a design note, run the experiment, record the result in a journal. It was built with a very different idea of what the tool is for, and learning to work inside its rules was part of the exercise.

Measuring honestly

From the second experiment on, every score came from a five-fold cross-validation grouped by patient. Shuffling visits at random would have let the model see the same patient in training and validation, and given flattering numbers that Kaggle would then have refused.

From 16.5 to 3.4

  • A constant guess: 16.5. A linear model: 10.6. Gradient boosting that keeps missing values as information: 7.8.
  • Per-patient features, such as averages over all their visits: 4.9. The biggest single step.
  • Tuning the boosting and trying an automatic tabular pipeline went nowhere. Temporal features with LightGBM: 4.7.
  • The turning point: each patient's score follows a smooth curve with age. Fitting that curve per patient, then correcting for time since the last dose and for who rated the visit: 3.7.
  • Adding each patient's own response to levodopa: 3.4 in cross-validation, 3.20 on the public leaderboard.

What I take from it

  • Looking at the data per patient taught us more than any tuning did.
  • A clean validation split matters more than the model; it is what made each step trustworthy.
  • Keeping a journal of every experiment, including the failed ones, made the path easy to retrace.
  • Not done: predicting the curve's coefficients directly, or putting LightGBM inside the per-patient model.
Error per experiment, from a constant guess to the final model. Lower is better.
Error per experiment, from a constant guess to the final model. Lower is better.