¿Qué métodos son efectivos para mejorar la eficiencia de datos al entrenar LLM?

30 Tem 06:26

Smart sampling is probably the low‑hanging fruit most teams overlook. Instead of feeding the model every token you have, you can construct a weighted‑by‑information loss sampler that prefers sentences where the current model’s perplexity is high. In practice I’ve seen a 10‑15 % reduction in epochs to reach a target validation loss when using a simple “hard‑example” buffer combined with periodic shuffling. The trade‑off is that you need a reliable proxy for “hardness” early in training, which usually means a cheap checkpoint model that you keep updating. Curriculum learning, when done right, can actually smooth the data‑efficiency curve. The key is to start with a narrow, high‑quality subset—think well‑structured news articles or code snippets—and progressively broaden the distribution. I ran a two‑stage curriculum on a 6‑B parameter model: first 30 % of training steps on curated technical documentation, then the full web crawl. The final model matched baseline performance but consumed roughly 20 % fewer total tokens. The downside is the upfront effort to define a sensible curriculum and the risk of biasing the model toward the early domains. Synthetic augmentation is a double‑edged sword. Techniques like back‑translation or paraphrase generation can inflate the effective data size, but they also introduce noise that can confuse the model if the synthetic data isn’t filtered. In my recent experiment with a 2‑B parameter model, adding a modest 5 % of high‑quality paraphrases—generated by a smaller T5‑like model—gave a small boost in downstream fine‑tuning tasks, yet the same pipeline hurt performance on factual QA because the synthetic sentences contained subtle factual errors. So the rule of thumb: augment only when you have a strict quality filter and keep the proportion low. Finally, consider token‑level sparsity tricks like “mix‑up” embeddings or dropout on the input embeddings themselves. These don’t reduce the raw data volume, but they can make each token count more by forcing the model to learn more robust representations. The cost is extra compute per batch, but it’s often cheaper than pulling in more raw data. All of these methods have a sweet spot; the best practice is to start with a cheap smart sampler, layer on a well‑planned curriculum, and only then experiment with synthetic augmentation if you still need a squeeze.