Which do you think we should focus more on when developing large language models (LLMs): ensuring the dataset is clean, diverse, and high-quality, or optimizing the model architecture (e.g., transformer variations, attention mechanisms)? Why did you choose your preference? If there's any confusion, I can clarify.
In LLM training, is data quality or model architecture more important?
👁️ 8 views💬 4 replies❤️ 0 likes
4 Replies
Man, bro, I always advocate for a "data-first" approach in LLMs. Because even the best model architecture can tank performance with bad data. For example, in the projects I’ve worked on, after putting in the effort to clean and diversify the data, tweaking the architecture led to explosive improvements. Even transformers don’t mean much if they’re not fed well-tokenized, richly diverse data.
But of course, architecture matters too—it’s not an either/or thing. Like, for smaller systems, lightening up the attention mechanism (e.g., FlashAttention) boosts both speed and memory efficiency. So if you’ve got solid data and you’re used to it, a few tweaks to the architecture can work wonders. But if your data’s trash, no matter how many transformer variations you throw at it, the results will always be disappointing.
Last year, we went through a similar crisis with my buddy on a project. We were training a new LLM in-house with our own data, and the team was obsessed with the architecture. They kept saying, "Look, we've got this 405B parameter micro-Transformer. If we optimize it with 10 tweaks, we'll get the results we need." I kept pushing back, saying, "Come on, let's not ignore the data," but eventually, I ended up manually sifting through our dataset. Turns out, after three months of training, the model flopped, and the output quality was terrible. Why? Because 20% of our dataset consisted of low-quality sources that hadn’t been cleaned or corrected by humans. If we’d spent twice as much time cleaning the data instead of tweaking the architecture, the model would’ve produced solid results on its own.
So for me, the answer’s clear: data quality > model architecture. Architecture’s important, sure, but if you’re not using clean, diverse, high-quality data, you’re wasting your time—even with the best architecture in the world. A friend of mine at Google confirmed this too; their top LLMs win competitions using the simplest transformers with the cleanest data. Bottom line: fix the data first, then ask for the budget for architecture.
Data quality is more important, I think bro. I had a class project like that too, if you don’t clean the data properly, the chatbot starts giving nonsense answers. The architecture is important too, but without data, the model doesn’t mean anything, man.
Honestly, bro, data quality is way more important. I mean, no matter how good a model's architecture is, if it's fed with bad data, it's just gonna be a "parrot that talks pretty." For my small projects, I always try to start with high-quality data, and the results show a huge difference.