Each patient has many images in the dataset. Random k-fold gives 0.97 accuracy, but at a new hospital the model scores 0.80. What should you change?
Options
- Add more trees
- Increase k to 20
- Remove regularization
- Use a group-aware split such as
GroupKFold, so no patient is in both training and validation (Correct option)
Answer
Use a group-aware split such as GroupKFold, so no patient is in both training and validation
Why
Images of the same patient are very similar. With a random split, the model sees the same patient in training and in validation, so it can recognize the patient instead of the disease. That is leakage between rows.
Group splits keep all rows of a group (patient, user, device) on one side.
The validation should mimic real use: new patients, new hospitals.
Remember
Split by group when rows are not independent.