Only 1% of your examples are positive. What is a sensible way to start?
Options
- Oversample the positives in the whole dataset and then split it into train and test
- Report accuracy: it is the standard metric for classifiers
- Use a stratified split and judge the model with precision, recall or PR-AUC instead of accuracy (Correct option)
- Use a bigger model: imbalance disappears with enough capacity
Answer
Use a stratified split and judge the model with precision, recall or PR-AUC instead of accuracy
Why
A stratified split keeps the 1% share in every part, so each one has positives to learn from and to be tested on. Accuracy would reward always predicting the majority class, so use metrics for the rare class.
Resampling is fine, but only on the training part: oversampling before the split puts copies of the same row in train and test (leakage).
Class weights or threshold tuning often work as well as resampling.
Remember
Stratify, pick the right metric, resample only training data.