Local LLMs

1) Introduction

Everything keeps changing in the realm of Large Language Models.
As of September 2026 I decided to create a blog to keep track of my experiments, avoid forgetting details, consolidate information spread across various discussion & issue threads.

Writing old-style posts like this nowadays is against the  current trend as most people (myself included) tend to simply query an LLM for information.
But on the other hand maybe somebody will like such a centralized list of information + the LLMs will scan and assimilate it providing more reliable answers + it will allow me to better track my past experiences and experiments.

Fyi I created this blog in September 2026 and I'm backdating the first posts.

2) Why run LLMs locally?

Why run LLMs locally instead of just paying an external provider like Google/OpenAI/Anthropic/Mistral/Alibaba/DeepSeek/etc...?

  1. I'm a tinkerer at heart + I love "poking at things" -> experimenting with local LLMs was irresistible.
  2. Peace of mind regarding costs.
    • I'm thinking especially of when I used for a while Gemini in Google's "AI studio" (end of 2025 / beginning of 2026):
      I could set a max budget of "tokens" in Google's UI but it wasn't enforced (if I remember correctly I would have gotten an email when it was reached -> bad luck if I don't continuously check my inbox...).

      That requirement is in my opinion a "no-brainer": if a company isn't able to control well enough the usage of its offerings then it should at least cover the costs that go beyond the approved budget for the limited timespan during which that can happen.

      When I was clueless in regards to LLMs I initially did not know that each additional post done in the same chat/thread would make the LLM reprocess the whole history of the chat/thread -> when I happened to have a discussion that had a single very long post of logs, each additional question/post increased my tokens consumption exponentially, which made me reach almost 1M tokens usage in a few minutes -> luckily I was still in the free/prepaid usage phase so I did not have to pay anything, but I got my lesson.
       

    • Maybe your project/usecase needs a high amount of processing but not much "intelligence"?
      I have a usecase of scanning and doing OCR of old magazines before throwing them away, to free up space in the cellar -> with ~20 years worth of monthly issues * 12 issues per year * an average of 150 pages per issue that resulted in ~36000 pages to be OCR'd and interpreted (multiple columns of text had to be understood and merged to form coherent sentences).

      I didn't have a good feeling about the costs that would have been involved.
      (e.g. for a single page, ingesting a scan uses ~1000-2000 tokens + there might be a ~long "thinking"-phase of ~2000-25000 tokens depending on the complexity of the text columns that need to be merged and the quality of the user prompt + finally the output of ~0-400 tokens per page = from ~3400 to ~40000 tokens per page?).

  3. Data privacy. 
    Sometimes I do not want to forward personal information to external companies.
    (especially non-european ones which may not have to comply to data privacy guidelines like GDPR).

    Additionally, I envision a future when we'll all be running LLMs locally to perform at least e.g. small/medium-sized recurring tasks, having a permanently open audio chat with a LLM without having to worry about releasing personal information, etc... -> being able to run most of this on your private infrastructure gets rid of many disclosure release problems.

  4. Models ("brains") change, therefore their behaviours change as well.
    If I have a usecase working perfectly using version Y of model X then I just want to keep it like that; I don't want to run after any change of behaviour caused by upgrades/changes that external providers do when switching to their next newest/greatest/alternative model (this includes deprecation of older models).
  5. Understand exactly what's going on and what hackers are talking about.
    For example when people talk about "speculative decoding"/"multi tokens prediction" or about "Caveman" then you'll be able to quickly assess what's that for and what its dis/advantages might be.
     

Of course running models locally has as well disadvantages such as:

  • Having to organize and keep track of everything can become quite stressful.
    On the other hand you can leave everything untouched for a quite long time (e.g. as long as you're able to compile your specific version of "llama.cpp") and decide & plan on your own when to switch to the next newer/better/alternative model and/or software.
  • Initial provisioning costs.
    As of 2026 cost increases have become unbearable:
    e.g. an nVidia RTX 5060 with 16 GiB VRAM or an RTX 5070 with 12 GiB VRAM used to cost max CHF ~500 in 2025 respectively H1 2026, but as of H2 2026 prices have increased again to almost CHF 700, let's not even mention DDR5 RAM or storage (both flash and even normal HDDs!!!) -> if you don't already own the hardware or have won the lottery then currently you're out of luck :o(
    But soon or later prices will have to come down again, it's just a question of "when".
  • Local electricity costs.
  • Temperature! Noise!
    My home office room got hot and noisy this summer (2026) when my "big PC" (3 GPUs) was crunching & generating tokens.

3) Why run LLMs locally "now"?

In the initial years things were (at least for me) absolutely confusing, for example:

  • information was incomplete and spread all around the Internet.
  • models were not yet good enough or were very big (therefore needing a big infrastructure).
  • the "good" programs that could run the models reliably couldn't be identified in discussion posts/threads; almost every single person was shouting about the one that s/he was using, many criticized software others were using.
  • I understood nothing about the involved terminology.
    (e.g. I used to program in C/C++ so the name "llama.cpp" generated maximum confusion, the website called "huggingface" did not contribute to alleviate my doubts, the last time I dealt with "tokens" was in arcade rooms in Italy/Jesolo when I was a teenager)

Things changed for me when the LLM model "Qwen" version 2.5 created by the chinese company "Alibaba" was published in 2024/2025 and became then widely known in ~2025:

  • the model had variants that were definitely small enough to be handled using normal GPUs & CPUs.
  • the community focused and mentioned everywhere mainly 3 programs to be used to run the model, being "Ollama" & "llama.cpp" & "vLLM".
  • information about compilation & runtimes options became more accessible.

Therefore in my opinion, starting from 2026 onwards, the Internet does mention most of the stuff that would enable most people to successfully install and run a local LLM
-> nothing can stop you now, if you're ready to give it a try (at least if you already own a ~recent GPU with at least 12 GiB VRAM, mh :oP ).