After the prototype stage, the model's performance plateaus. Beyond data cleaning, diversity, and formatting, what other strategies can be tried? For example, does generating synthetic data, improving label quality, or comparing different architectures make sense? What approaches do you prefer?
How do we optimize data in LLM projects?
👁️ 5 views💬 3 replies❤️ 0 likes
3 Replies
I think first you should analyze whether your existing data is insufficient to cover the gaps in your model. For example, if there are very few examples for a certain concept or phrase group, you can perform **data augmentation** to generate new data for those topics in natural language, such as back-translation or creating variants.
Additionally, you could try **improving label quality with automated or semi-automated tools**, like relabeling noisy data using weak teacher models. In my experience, improving labels generally increases the model's consistency.
In this case, before thinking about new architectures or synthetic data, it's worth measuring **where the bottleneck is**. Is the generation of embeddings stalling? Is fine-tuning not improving despite more epochs? If the loss curve has flattened and early stopping hasn't triggered, it's usually a sign that the issue isn't the volume or variety of data, but how it's used in the pipeline. For example, if you're working with text vectors in a RAG system, I'd check the chunking and overlap between fragments: sometimes finer segmentation with context-aware tokenizers (like those from Mistral or Codestral) gives a 15-20% boost in retrieval without touching the dataset.
As for synthetic data, I find it useful when the real dataset is **very small or has noisy labels with contradictions**. Tools like *LLM-as-a-Judge* (using models like Zephyr-7B-β) or *Self-Instruct* can generate difficulty-controlled examples (easy, medium, hard) and filter them for consistency across multiple runs. That said: always apply *distillation* from the generating model to the final dataset to avoid hallucinations in the synthetic data. In medical AI projects we worked on, this technique increased the F1 score from 0.72 to 0.83 using only 30% real data.
Finally, comparing architectures shouldn't be random. If you already have an efficient LoRA on a base model, first try **QLoRA + 4-bit with gradient checkpointing** on AWS SageMaker: in my tests, switching from FP16 to this setup reduced memory usage by 70% without losing accuracy. But if the issue is long context, evaluate architectures like **Mistral-7B-Instruct-v0.2 with Attention Sinks** or **Longformer**. In any scenario, maintain an *internal leaderboard* with metrics like BLEU, ROUGE-L, and especially **perplexity on a small but representative holdout set**. The difference between experiments should be there, not in anecdotes.
I was focusing on generating synthetic data, especially for underperforming scenarios. Instead of manually improving label quality, I tried enriching it with the model's own predictions or data transferred from similar tasks. Sometimes, I also repeat the process with tiny variations (like adding noise or masking) to mimic the diversity of real data—you could call this "noisy redocumentation."