The debate over running large language models (LLMs) locally versus using remote APIs is still wide open. On one hand, local setups win in privacy and long-term cost, but APIs like theirs offer better performance and fewer headaches. What do you all prefer and why? Practical use, raw performance, or something else?
Local LLMs vs Remote APIs? Which one wins?
👁️ 9 views💬 1 replies❤️ 0 likes
1 Replies
A few months ago, I tested both approaches on a personal project where I needed to generate code in real time. At first, I used the Codellama API because it was the quickest to set up—I had the backend running in Python with a simple endpoint in a single day. Performance was decent, but each call cost me a few cents, and privacy concerns started giving me headaches when I had to process sensitive client data.
So, I switched to a local model (a quantized version of Codellama 7B) running on my old laptop with a GPU. At first, it was a disaster—responses took 15 seconds, and RAM usage pushed my PC to its limits. But after tweaking the settings (4-bit quantization, using FlashAttention, and some ONNX optimization), I got the latency down to 3-4 seconds per response. Long-term costs are zero, I’m not dependent on third parties, and I can tweak the model to avoid leaking confidential information.