What general framework can be established for understanding how the scope, cleaning criteria, and class balance of datasets used for fine-tuning LLMs impact model performance? Especially in low-resource settings, what data selection strategies should be employed? In your view, how can balancing techniques help minimize the risk of overfitting?
What strategies should be used for data selection and balancing in the fine-tuning process of LLMs?
👁️ 94 views💬 1 replies❤️ 0 likes
1 Replies
For effective fine-tuning in a limited environment, I first prioritize **quality over quantity**: I filter the dataset using a deduplication script (SHA-256 hashing) and remove entries with excessive tabs or non-printable characters. Then, I categorize the examples by **functional category** (e.g., Q&A, instructions, dialogues) and maintain a balanced number of each class, even if it means drastically reducing the total. In practice, I use stratified subsampling: I set a quota (e.g., 2,000 examples) per category and then select the cleanest examples (low unknown token rate, good length). This approach allows me to obtain a training set of around 10k examples while maintaining sufficient thematic coverage.
To limit overfitting, I combine **class weights** (or focal loss) with **early stopping** based on a small validation set that reflects the real-world distribution of the problem. A little trick that worked for me: I apply slight **data augmentation**—such as synonym replacement or back-translation—only to underrepresented classes, which increases diversity without inflating the dataset. Finally, a slow-decaying **learning-rate scheduler** (cosine annealing) and fine-tuning in LoRA or QLoRA mode help keep the model robust even with very little data. By following these steps, I’ve reduced the performance gap by over 8% between a model fine-tuned on 1% of the original corpus and the same model trained on the full corpus.