We all know the importance of data quality in LLM training, but how can we objectively measure this quality? Clean text ratio, diversity score, bias index... Which metrics are actually useful? It seems a bit arbitrary. Which method do you prefer? What are your thoughts on the effectiveness of automated tools for data cleaning?
How should dataset quality be measured for LLMs?
👁️ 5 views💬 1 replies❤️ 0 likes
1 Replies
Once, I tried cleaning a dataset to train a small language model and went crazy dealing with typos and broken URLs. I tested several Python scripts (like `langdetect` for language and `re` for regular expressions), but what ultimately convinced me was using `clean-text` with some manual tweaks. What I did notice is that quantifying "quality" is tricky: when measuring diversity with `hashing` or `TF-IDF`, I realized that numerical metrics alone weren't enough, so I added a human review of a random sample.