What factors should we consider when choosing the learning rate for fine-tuning large language models for a specific task? Especially with challenges like low data volume, adapting to different language structures, and balancing between different optimizer types, it can be tricky. Could you share your experiences and recommendations on common learning rate scheduling strategies and recommended ranges?
How should the learning rate be adjusted during the fine-tuning process of large language models?
👁️ 85 views💬 2 replies❤️ 0 likes
2 Replies
When fine-tuning a model like GPT-2 for a task with few examples, the first thing I tried was using a fairly low learning rate (between 1e-5 and 5e-5) with the AdamW optimizer, which helps maintain the stability of the pre-trained weights. In my case, when adapting the model to Catalan texts, I found that a "linear warm-up" scheduler of 5% of the total steps helped prevent training from saturating in the early batches; after the warm-up, I linearly reduced the LR to 0. This mitigated the overfitting that often occurs with datasets of <2k sentences.
When I had to mix multiple languages (English, Spanish, and French), I used the same LR range but split the training into two phases: first, a "head" layer with LR=3e-5, and then the entire model with LR=1e-5, also using the Adafactor optimizer to reduce memory usage. In tests, the "cosine decay" of the LR after the warm-up proved more robust than a constant decay, especially when the number of epochs was limited. In short, start with LR ≈ 1e-5–5e-5, apply warm-up (3–5% of the steps), and then use a linear or cosine scheduler; adjust the "head-only" phase if your data is very scarce or if you're changing the target language.
In my latest fine-tuning project with a ~2B parameter model, I found that the learning rate needs to scale with three key variables: dataset size, domain similarity, and the optimizer you're using. With small datasets (under 1k examples), I usually start with 1e-5 to 2e-5 and apply a cosine-annealing or linear decay scheduler; this prevents the model from overfitting too quickly to the limited examples. If the target language differs significantly from the pre-training language (e.g., switching from English to Finnish), slightly increasing the LR to 3e-5 helps the network adjust the sub-word embeddings, though I always keep a 5-10% warm-up phase to stabilize the initial training.
As for the optimizer, AdamW remains the safest choice; however, I’ve noticed that with LoRA-based adapters, AdamW with β₁=0.9 and β₂=0.999 outperforms Adafactor, which tends to be too aggressive with higher LRs. In practice, I test a range of values (1e-5, 2e-5, 3e-5) and use early-stopping based on validation loss. The combination of a low LR and a gradual scheduler usually yields the most robust results, especially when data is scarce and the language is structurally different.