20260301 | Compilation of "llama.cpp"

You can use "llama.cpp" as the program that runs the Large Language Model (LLM) of your choice (such as "Gemma" by Google, "Qwen" by Alibaba, etc...), offering a standard OpenAI I/O interface.
You can then make other applications (such as your favorite Web UI for general chat activities, your code assistant, your automation tool, ....) connect to Llama's listening port to use/interface with that LLM.

1) Preparations

  • These instructions focus on compiling "llama.cpp" on Linux and in combination with CUDA for nVidia GPUs.
  • As of H1 2026 when using CUDA you will need GCC version 14 (see below).
  • For each version of "llama.cpp" that you will handle you will need quite a bit of free disk space (0.5-1.5 GiB) to host the sourcecode and the compilation output.
    You will want to host multiple versions because of new features, new bugs, new support for different models and/or model versions, etc... and then switch back and forth as needed.

2) Get the sourcecode

The releases of "llama.cpp" are listed here: https://github.com/ggml-org/llama.cpp/releases

 

As of 2026, the project publishes new versions very frequently in the form "bXXXXX" (kind of "nightly releases" - e.g. "b10817") basically for each individual change, and from time to time aggregated ones in the form "vX.X.X" (e.g. "v0.4.0") which have a nice aggregated changelog.

Some releases are minor, while others are important because they might introduce support for new language models or new features (e.g. "Multi Token Prediction"/"Speculative Decoding" for specific models or model versions).

 

Choose a specific version (usually the latest one), expand its "Assets" menu, download the file "Source code (tar.gz)"... 

$ wget "https://github.com/ggml-org/llama.cpp/archive/refs/tags/b10867.tar.gz"

...and then unpack & extract it: 

$ gunzip b10867.tar.gz
$ tar xvf b10867.tar

The unpacked sourcecode will then be located in the subdirectory "llama.cpp-b10867".

3) Configure the build

In the sourcecode directory create a "build" subdirectory and go in there: 

$ mkdir llama.cpp-b10867/build
$ cd llama.cpp-b10867/build

From within the build directory then execute the configuration command -> here an example of the basic parameters:  

cmake .. \
-DGGML_CCACHE=OFF \
-DGGML_NATIVE=ON \
-DGGML_LTO=ON \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_C_FLAGS="-O3 -march=znver5 -pipe -fomit-frame-pointer" \
-DCMAKE_CXX_FLAGS="-O3 -march=znver5 -pipe -fomit-frame-pointer" \
&& echo ok || echo NOK
3a) CPU flags

Llama.cpp should be able to automatically identify your CPU and use the ideal options, but to avoid any surprises I usually set explicitly the C and CXX flags: 

-DCMAKE_C_FLAGS="-O3 -march=znver5 -pipe -fomit-frame-pointer" \
-DCMAKE_CXX_FLAGS="-O3 -march=znver5 -pipe -fomit-frame-pointer" \
  • "-O3":
    Uses max optimizations (level 3).
    Some extreme optimizations might backfire on some architectures so if you want to avoid risks then use only "-O2".
  • "-march=znver5":
    In my case my PC has as CPU an "AMD Ryzen 7 9700X" -> as that processor belongs to the "zen 5" architecture I tell the compiler with "-march" to implement full optimizations for the architecture "znver5".
  • "-pipe":
    Has only an impact on the compilation speed (compiles using pipes instead of temporary files).
  • "-fomit-frame-pointer":
    Compiled result might be smaller and run a little bit faster.
    If this is specified then debuggers won't work.
3b) If you plan to use CUDA with an nVidia GPU

3b1)

Ensure that you have already installed the nVidia CUDA toolkit; in Gentoo that's the package "dev-util/nvidia-cuda-toolkit".

3b2)

Additionally (after having installed the CUDA-toolkit), CUDA itself will be very picky in regards to the version of the compiler that will be used to compile code so doublecheck which version of GCC you're using: 

$ gcc --version
gcc (Gentoo 14.3.1_p20260604 p6) 14.3.1 20260604

As of 1H 2026, the CUDA-related code of "llama.cpp" will complain if you use a version of GCC newer than 14:

#error -- unsupported GNU version! gcc versions later than 14 are not supported! The nvcc flag '-allow-unsupported-compiler' can be used to override this version check; however, using an unsupported host compiler may cause compilation failure or incorrect run time execution. Use at your own risk.

This means that if you have e.g. version 15 or 16 then you'll have to install additionally GCC v14 and then either:

  • switch your main compiler to GCC version 14.
    (e.g. on Gentoo with "gcc-config -l" and then selecting v14 with "gcc-config X")

  • instruct the "cmake" configuration command to use the older version 14 instead of the current main version 15 or 16 -> this is achieved by giving "cmake" the following additional parameters: 

    -DCMAKE_C_COMPILER=x86_64-pc-linux-gnu-gcc-14 \
-DCMAKE_CXX_COMPILER=x86_64-pc-linux-gnu-g++-14 \
-DCMAKE_CUDA_HOST_COMPILER=x86_64-pc-linux-gnu-g++-14 \

3b3)

Specify your nVidia GPU architecture / "compute capabilities" with the parameter 

-DCMAKE_CUDA_ARCHITECTURES="<your_gpu_arch>"

This page shows a mapping of the "compute capabilities" of various nVidia GPUs: https://developer.nvidia.com/cuda/gpus
Here is a copy:

Compute capability GPU
12.1 NVIDIA GB10 (DGX Spark)
12.0 NVIDIA RTX PRO 6000 | Blackwell Server Edition | NVIDIA RTX PRO 4500 | Blackwell Server Edition | NVIDIA RTX PRO 6000 Blackwell Workstation Edition | NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition | NVIDIA RTX PRO 5000 Blackwell | NVIDIA RTX PRO 4500 Blackwell | NVIDIA RTX PRO 4000 Blackwell | NVIDIA RTX PRO 4000 Blackwell SFF Edition | NVIDIA RTX PRO 2000 Blackwell | GeForce RTX 5090 | GeForce RTX 5080 | GeForce RTX 5070 Ti | GeForce RTX 5070 | GeForce RTX 5060 Ti | GeForce RTX 5060 | GeForce RTX 5050
11.0 Jetson T5000 | Jetson T4000
10.3 NVIDIA GB300 | NVIDIA B300 | NVIDIA GB300 (DGX Station)
10.0 NVIDIA GB200 | NVIDIA B200
9.0 NVIDIA GH200 | NVIDIA H200 | NVIDIA H100
8.9 NVIDIA L4 | NVIDIA L40 | NVIDIA L40S | NVIDIA RTX 6000 Ada | NVIDIA RTX 5000 Ada | NVIDIA RTX 4500 Ada | NVIDIA RTX 4000 Ada | NVIDIA RTX 4000 SFF Ada | NVIDIA RTX 2000 Ada | GeForce RTX 4090 | GeForce RTX 4080 | GeForce RTX 4070 Ti | GeForce RTX 4070 | GeForce RTX 4060 Ti | GeForce RTX 4060 | GeForce RTX 4050
8.7 Jetson AGX Orin | Jetson Orin NX | Jetson Orin Nano
8.6 NVIDIA A40 | NVIDIA A10 | NVIDIA A16 | NVIDIA A2 | NVIDIA RTX A6000 | NVIDIA RTX A5000 | NVIDIA RTX A4000 | NVIDIA RTX A3000 | NVIDIA RTX A2000 | GeForce RTX 3090 Ti | GeForce RTX 3090 | GeForce RTX 3080 Ti | GeForce RTX 3080 | GeForce RTX 3070 Ti | GeForce RTX 3070 | GeForce RTX 3060 Ti | GeForce RTX 3060 | GeForce RTX 3050 Ti | GeForce RTX 3050
8.0 NVIDIA A100 | NVIDIA A30
7.5 NVIDIA T4 | QUADRO RTX 8000 | QUADRO RTX 6000 | QUADRO RTX 5000 | QUADRO RTX 4000 | QUADRO RTX 3000 | QUADRO  T2000 | NVIDIA T1200 | NVIDIA T1000 | NVIDIA T600 | NVIDIA T500 | NVIDIA T400 | GeForce GTX 1650 Ti | NVIDIA TITAN RTX | GeForce RTX 2080 Ti | GeForce RTX 2080 | GeForce RTX 2070 | GeForce RTX 2060

Therefore if you want to use e.g. an RTX 5070 then specify... 

-DCMAKE_CUDA_ARCHITECTURES="120" \

...or if you want to use e.g. an RTX 5070 and as well an RTX 3060 then specify... 

-DCMAKE_CUDA_ARCHITECTURES="86;120" \

...and so on.

3b4)

In the shell/terminal which you plan to use to run the "configuration" and then later the "build", optionally set the paths where your CUDA executables and libraries are located with e.g.  

export PATH="/opt/cuda/bin:$PATH"
export LD_LIBRARY_PATH="/opt/cuda/lib64:$LD_LIBRARY_PATH"

 

In my case the binaries are located in "/opt/cuda/bin"... 

$ ls /opt/cuda/bin
bin2c  cuda-config  cu++filt   fatbinary  __nvcc_device_query  nvdisasm  nvprune
crt    cudafe++     cuobjdump  nvcc       nvcc.profile         nvlink    ptxas

...and the libraries are located in "/opt/cuda/lib64": 

$ ls /opt/cuda/lib64
cmake libcublasLt.so libcublasLt.so.12
libcublasLt.so.12.9.1.4 libcublasLt_static.a
...
3c) Wrap-up

This would be the final configuration command... 

$ export PATH="/opt/cuda/bin:$PATH"
$ export LD_LIBRARY_PATH="/opt/cuda/lib64:$LD_LIBRARY_PATH"

$ cd llama.cpp-b10867/build

$ cmake .. \
-DCMAKE_C_COMPILER=x86_64-pc-linux-gnu-gcc-14 \
-DCMAKE_CXX_COMPILER=x86_64-pc-linux-gnu-g++-14 \
-DCMAKE_CUDA_HOST_COMPILER=x86_64-pc-linux-gnu-g++-14 \
-DGGML_CCACHE=OFF \
-DGGML_CUDA=ON \
-DCMAKE_CUDA_ARCHITECTURES="86;120" \
-DGGML_NATIVE=ON \
-DGGML_LTO=ON \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_C_FLAGS="-O3 -march=znver5 -pipe -fomit-frame-pointer" \
-DCMAKE_CXX_FLAGS="-O3 -march=znver5 -pipe -fomit-frame-pointer" \
&& echo ok || echo NOK

...which would set in my case the following:

  • I have in my Gentoo Linux distro both GCC 14 and 15 installed, version 15 is set as the main version, therefore I tell cmake to compile "llama.cpp" by using GCC version 14.
  • Compile the support for my "RTX 5070" + "RTX 5060" (compute capability "120") and my "RTX 3060" (compute capability "86").
  • Optimize the code for my zen 5 CPU.

If you make a mistake and want to rerun the configuration then you might potentially want to first delete all the contents of your "build" directory.

If everything goes well you should get at the very end a confirmation:

...
-- Configuring done (3.6s)
-- Generating done (0.2s)
-- Build files have been written to: /......./llama.cpp-b10867/build
ok

4) Compile / Build

To build/compile "llama.cpp" then finally execute in your "build" directory the following: 

cmake --build . --config Release -j14 && echo ok || echo NOK

With "-j<some_number>" you specify how many max parallel processes to use for the compilation.
E.g. my CPU has 16 logical cores so I use 14 parallel processes (a bit less than what is available).

The compilation time varies depending on the version of "llama.cpp", your compilation options, your CPU and the amount of parallel processes; e.g. on my "AMD Ryzen 7 9700X" with "boost" disabled compiling version "b10867" with max 14 parallel processes takes ~8.5 minutes.

If everything goes well then you should get the final "ok":

...
[100%] Linking CXX executable ../../bin/llama-cli
[100%] Built target llama-cli
[100%] Linking CXX executable ../bin/llama
[100%] Built target llama-app
[100%] Linking CXX executable ../bin/test-chat
[100%] Built target test-chat
[100%] Linking CXX executable ../bin/test-backend-ops
[100%] Built target test-backend-ops
ok

The executables will be located in the "bin" subdirectory (in this example that would be "llama.cpp-b10867/build/bin").

EDIT 15.Sept.2026:
if you get a build error similar to this one...

...
[  4%] Linking CXX executable ../../bin/llama-gemma3-cli
[  4%] Built target llama-common-base
[  4%] Built target llama-gemma3-cli
-- UI: archive verified and extracted
-- UI: HF download succeeded, stamp updated (ggml-org/llama-ui|b0, resolved: latest)
CMake Error: Unable to read from file:
../../../../../../ssd/ai/llama.cpp-selfcompiled/llama.cpp-b10976/build/tools/ui/dist/_app/immutable/assets/bundle.oAmIsIaD.css
because:
Can't lstat ../../../../../../ssd/ai/llama.cpp-selfcompiled/llama.cpp-b10976/build/tools/ui/dist/_app/immutable/assets/bundle.oAmIsIaD.css
CMake Error at /opt/<redacted>/ai/llama.cpp-selfcompiled/llama.cpp-b10976/scripts/ui-assets.cmake:218 (file):
file failed to compress:
/opt/<redacted>/ai/llama.cpp-selfcompiled/llama.cpp-b10976/build/tools/ui/ui-gzip/_gzip/_app/immutable/assets/bundle.oAmIsIaD.css
Call Stack (most recent call first):
/opt/<redacted>/ai/llama.cpp-selfcompiled/llama.cpp-b10976/scripts/ui-assets.cmake:592 (emit_files)


[  5%] Linking CXX static library libvendor-hash.a
gmake[2]: *** [tools/ui/CMakeFiles/llama-ui-assets.dir/build.make:71: tools/ui/CMakeFiles/llama-ui-assets] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:6102: tools/ui/CMakeFiles/llama-ui-assets.dir/all] Error 2
gmake[1]: *** Waiting for unfinished jobs....
[  5%] Built target vendor-hash
[  5%] Linking CXX shared library ../../bin/libggml-base.so
[  5%] Built target ggml-base
[  5%] Linking CXX static library libcpp-httplib.a
[  5%] Built target cpp-httplib
gmake: *** [Makefile:146: all] Error 2
NOK

...then it's probably because of how the embedded Web UI has recently been set up in Llama and the reason might be that you don't have "nodejs" installed -> you have two options:

  • if you don't need the embedded Web UI then just add the two following options to your configuration command... 

    -DLLAMA_BUILD_UI=OFF -DLLAMA_USE_PREBUILT_UI=OFF \

    ...then delete all the contents of your "build"-directory and execute again the configuration and then the build commands.

  • if you need the embedded Web UI then installing "nodejs" (package "net-libs/nodejs" in Gentoo Linux) should get rid of the error message.

5) Run "llama.cpp"

Out of context in relation to this post, anyway to run "llama.cpp" I recommend to use individual shell scripts and to mention in their name some extra information like:

  • the context / type of usage (e.g. OCR, RAG, code assistant, general chat, ...) as you will want to customize Llama's parameters as needed for the specific usecase (e.g. a small context size for OCR but a big one for the general chat, a fast more quantized model for low-risk tasks but a slow less quantized model for important tasks).
  • maybe the version of the script (you might want to test multiple similar variants and keep track of what you did in the past, and then to move old scripts to an "archive/" subdirectory to keep track of all your experiments).
  • maybe the model name and/or as well its size (not sure - here we go with my usual "clash of naming conventions").

E.g. for a long time I did not mention a context and after a while I forgot which model & model size worked well with my code assistant.

Posting here as an example my shell script "natgeo-qwen36-clever_but_slow.sh" which starts Llama loading the "Qwen" LLM version 3.6, used as an OCR engine to identify text and then assemble complete sentences spread over multiple columns of my old National Geographic magazines:
(yes, I do have as well another script called "natgeo-qwen36-dumb_but_fast.sh" - it uses only 2 GPUs and works ~95 % of the times) 

#!/bin/bash
LLAMA_DIR="/<redacted>/llama.cpp-selfcompiled/llama.cpp-b10867/build/bin"
MODEL_DIR="/<redacted>/models"

MODEL_FILE="Qwen3.6-27B-Q6_K_L-bartowski-20260524.gguf"
MMPROJ_FILE="Qwen3.6-27B-mmproj-f16-unsloth.gguf"

export CUDA_DEVICE_ORDER=PCI_BUS_ID
export CUDA_VISIBLE_DEVICES=0,1,2

$LLAMA_DIR/llama-server \
--model "$MODEL_DIR/$MODEL_FILE" \
--mmproj "$MODEL_DIR/$MMPROJ_FILE" \
--no-ui \
--flash-attn on \
--main-gpu 0 \
--threads 8 \
--ctx-size 50000 \
--split-mode layer \
--temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0 --repeat-penalty 1.1 --presence-penalty 0.0 \
--host quad \
--fit on \
--fit-target 512 \
--parallel 1 \
--no-mmap \
--mmproj-offload \
-ctk f16 -ctv f16 \
--cache-ram 0 \
--tensor-split 15,24,12 \
--ubatch-size 2048 \
--spec-type draft-mtp --spec-draft-n-max 3 \
--port 10123

Instead of executing "$LLAMA_DIR/llama-server" you can replace that with "$LLAMA_DIR/llama-cli" to get immediately a terminal interface which allows you to post your first query -> you can install later one of the many web UIs such as "Open Web UI" (that you'll configure to connect to the "llama-server" process) to have a more comfortable UI.

Discussions about Llama's parameters will hopefully be part of other posts.

In general: start with a minimum set of options, test, then if results match your expectations then add candidate options as needed and repeat; keep in mind that things keep changing with each Llama & model version and that your config will grow together with your knowledge.

DO NOT BLINDLY TRUST STUFF independently from where you find hints/recommendations - test test test on your own, on your own hardware, using your own usecase!!! Try to target the scientific method!

:o)