Retraining strategy with evolving classes + imbalanced labels?
\n \t
\n
<\/BR>
\n
\n \t
\n
<\/BR><\/p>\n
Hi all \u2014 I\u2019m looking for advice on the best retraining strategy for a multi-class classifier in a setting where the label space can evolve. Right now I have about 6 labels, but I don\u2019t know how many will show up over time, and some labels appear inconsistently or disappear for long stretches. My initial labeled dataset is ~6,000 rows and it\u2019s extremely imbalanced: one class dominates and the smallest class has only a single example. New data keeps coming in, and my boss wants us to retrain using the model\u2019s inferences plus the human corrections made afterward by someone with domain knowledge. I have concerns about retraining on inferences, but that’s a different story.<\/p>\n
Given this setup, should retraining typically use all accumulated labeled data, a sliding window of recent data, or something like a recent window plus a replay buffer for rare but important classes? Would incremental\/online learning (e.g., partial_fit style updates or stream-learning libraries) help here, or is periodic full retraining generally safer with this kind of label churn and imbalance? I\u2019d really appreciate any recommendations on a robust policy that won\u2019t collapse into the dominant class, plus how you\u2019d evaluate it (e.g., fixed \u201cgolden\u201d test set vs rolling test, per-class metrics) when new labels can appear.<\/p>\n<\/p><\/div>\n
submitted by \/u\/fleeced-artichoke <\/a>
[link]<\/a><\/span> [comments]<\/a><\/span>\n<\/div>\n