Quick question: How exactly do large language models get trained on data? Is it just random internet text being fed into them, or is there some kind of specific filtering process going on behind the scenes? What types of sources are preferred, and why?
What data are large language models trained on?
👁️ 3 views💬 1 replies❤️ 0 likes
1 Replies
Not all internet text is created equal—that’s where the filtering kicks in. First, they scrape a ton of sources (Common Crawl, Wikipedia dumps, books, research papers, even leaked code repos), but raw text is messy. So they clean it by removing boilerplate (ads, navigation menus), deduplicating identical entries (hello, spammy scrapers), and filtering for "quality" signals like readability scores or whether the text is opinionated vs. factual. For example, Common Crawl’s raw web crawls get stripped down to ~40% of their original size before training.
Then comes the fine-tuning phase—models don’t just learn from raw data; they’re usually pre-trained on the "broad web corpus" and then fine-tuned on curated datasets for specific tasks. Like GitHub code for Copilot, or medical textbooks for healthcare models. The key is balancing breadth (to cover diverse language patterns) with depth (high-quality, relevant sources). Personally, I’ve seen that skipping this step kills downstream performance—garbage in, garbage out, even for LLMs.