Hello fellow developers, I've been diving into LLM development lately. Is fine-tuning or prompt engineering more effective? Or should I mix both approaches? Which data sources are generally preferred for training datasets? Does it make more sense to start with a small model or a large one? Please share your experiences and approaches, thanks!
What methods do you prefer when developing LLMs?
👁️ 4 views💬 3 replies❤️ 0 likes
3 Replies
I think when developing an LLM, first and foremost, you need to clearly define the purpose of the project, bro. If you're aiming for general capabilities (like a chatbot), you can start with prompt engineering and enhance it with fine-tuning. But if it's for a specific task (like medical text analysis), you might have to fine-tune a massive model, seriously.
I usually prefer starting with smaller models because fine-tuning large models is both expensive and hard to control. For example, I optimize a 7B model using methods like LoRA or QLoRA and train it with my own dataset. For training data, I use open-source datasets from GitHub (like Stack Exchange, Reddit, etc.) or domain-specific datasets available on Hugging Face.
However, starting with prompt engineering has the advantage of saving you from constantly retraining the model. You can improve results with specific prompts like "Give a shorter response" or "Present 3 different perspectives on the same topic." So, a mix of both is usually the best approach, depending on your budget and goals.
Also, from my experience, when fine-tuning, regulations (like overfitting, learning rate, etc.) are crucial; otherwise, the model's performance can tank. Optimizing a smaller model often saves you from the issues you'd face when fine-tuning a large model.
Starting with prompt engineering and then moving on to fine-tuning makes sense to me, much like designing with Canva instead of Photoshop. Playing around with prompts on a smaller model gives you quick feedback—testing a prompt on a 7B model and seeing results in 5 minutes is easy, whereas doing the same on a 70B model means waiting and risking disappointment. For this, built-in models on Hugging Face or their demo interfaces are more than enough.
When it comes to fine-tuning, I usually start with open-source datasets like OpenWebText or The Pile, just like washing vegetables from the local market before cooking. After optimizing a small model (e.g., 1.5B–3B parameters), moving to a larger model makes sense because the knowledge learned from the small model transfers more consistently to the large one compared to prompt engineering. Plus, fine-tuning a large model is expensive, so working on a small model first and then refining it further is the smarter approach.
I find prompt engineering powerful enough to outperform fine-tuning in many cases, but mixing both usually gives the most stable results. I typically start with a smaller model (like a 7B parameter one) and use open datasets from Hugging Face or domain-specific data for training.