20260301 | Compilation of "llama.cpp"

1) Preparations

  • These instructions focus on compiling "llama.cpp" on Linux and in combination with CUDA on nVidia GPUs.
  • As of H1 2026 when using CUDA you will need GCC version 14 (see below).
  • For each version of "llama.cpp" that you will handle you will need quite a bit of free disk space (0.5-1.5 GiB) to host the sourcecode and the compilation output.
    You will want to host multiple versions because of new features, new bugs, new support for different models and/or model versions, etc... and then switch back and forth as needed.

2) Get the sourcecode

The releases of "llama.cpp" are listed here: https://github.com/ggml-org/llama.cpp/releases

 

As of 2026, the project publishes new versions very frequently in the form "bXXXXX" (kind of "nightly releases" - e.g. "b10817") basically for each individual change, and from time to time aggregated ones in the form "vX.X.X" (e.g. "v0.4.0") which have a nice aggregated changelog.

Some releases are minor, while others are important because they might introduce support for new language models or new features (e.g. "Multi Token Prediction"/"Speculative Decoding" for specific models or model versions).

 

Choose a specific version (usually the latest one), expand its "Assets" menu, download the file "Source code (tar.gz)"... 

$ wget "https://github.com/ggml-org/llama.cpp/archive/refs/tags/b10867.tar.gz"

...and then unpack & extract it: 

$ gunzip b10867.tar.gz
$ tar xvf b10867.tar

The unpacked sourcecode will then be located in the subdirectory "llama.cpp-b10867".

3) Configure the build

In the sourcecode directory create a "build" subdirectory and go in there: 

$ mkdir llama.cpp-b10867/build
$ cd llama.cpp-b10867/build

From within the build directory then execute the configuration command -> here an example of the basic parameters:  

cmake .. \
-DGGML_CCACHE=OFF \
-DGGML_NATIVE=ON \
-DGGML_LTO=ON \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_C_FLAGS="-O3 -march=znver5 -pipe -fomit-frame-pointer" \
-DCMAKE_CXX_FLAGS="-O3 -march=znver5 -pipe -fomit-frame-pointer" \
&& echo ok || echo NOK
3a) CPU flags

Llama.cpp should be able to automatically identify your CPU and use the ideal options, but to avoid any surprises I usually set explicitly the C and CXX flags: 

-DCMAKE_C_FLAGS="-O3 -march=znver5 -pipe -fomit-frame-pointer" \
-DCMAKE_CXX_FLAGS="-O3 -march=znver5 -pipe -fomit-frame-pointer" \
  • "-O3":
    Uses max optimizations (level 3).
    Some extreme optimizations might backfire on some architectures so if you want to avoid risks then use only "-O2".
  • "-march=znver5":
    In my case my PC has as CPU an "AMD Ryzen 7 9700X" -> as that processor belongs to the "zen 5" architecture I tell the compiler with "-march" to implement full optimizations for the architecture "znver5".
  • "-pipe":
    Has only an impact on the compilation speed (compiles using pipes instead of temporary files).
  • "-fomit-frame-pointer":
    Compiled result might be smaller and run a little bit faster.
    If this is specified then debuggers won't work.
3b) If you plan to use CUDA with an nVidia GPU

3b1)

Ensure that you have already installed the nVidia CUDA toolkit; in Gentoo that's the package "dev-util/nvidia-cuda-toolkit".

3b2)

Additionally, CUDA is picky in regards to the version of the compiler that will be used so doublecheck which version of GCC you're using: 

$ gcc --version
gcc (Gentoo 14.3.1_p20260604 p6) 14.3.1 20260604

As of 1H 2026, the CUDA-related code of "llama.cpp" will complain if you use a version of GCC newer than 14:

#error -- unsupported GNU version! gcc versions later than 14 are not supported! The nvcc flag '-allow-unsupported-compiler' can be used to override this version check; however, using an unsupported host compiler may cause compilation failure or incorrect run time execution. Use at your own risk.

This means that if you have e.g. version 15 or 16 then you'll have to install additionally GCC v14 and then either:

  • switch your main compiler to GCC version 14.
    (e.g. on Gentoo with "gcc-config -l" and then selecting v14 with "gcc-config X")

  • instruct the "cmake" configuration command to use the older version 14 instead of the current main version 15 or 16 -> this is achieved by providing "cmake" the following additional parameters: 

    -DCMAKE_C_COMPILER=x86_64-pc-linux-gnu-gcc-14 \
-DCMAKE_CXX_COMPILER=x86_64-pc-linux-gnu-g++-14 \
-DCMAKE_CUDA_HOST_COMPILER=x86_64-pc-linux-gnu-g++-14 \

3b3)

Specify your nVidia GPU architecture / "compute capabilities" with the parameter "-DCMAKE_CUDA_ARCHITECTURES="<your_gpu_arch>"".

This page shows a mapping of the "compute capabilities" of various nVidia GPUs: https://developer.nvidia.com/cuda/gpus
Here is a copy:

Compute capability GPU
12.1 NVIDIA GB10 (DGX Spark)
12.0 NVIDIA RTX PRO 6000 | Blackwell Server Edition | NVIDIA RTX PRO 4500 | Blackwell Server Edition | NVIDIA RTX PRO 6000 Blackwell Workstation Edition | NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition | NVIDIA RTX PRO 5000 Blackwell | NVIDIA RTX PRO 4500 Blackwell | NVIDIA RTX PRO 4000 Blackwell | NVIDIA RTX PRO 4000 Blackwell SFF Edition | NVIDIA RTX PRO 2000 Blackwell | GeForce RTX 5090 | GeForce RTX 5080 | GeForce RTX 5070 Ti | GeForce RTX 5070 | GeForce RTX 5060 Ti | GeForce RTX 5060 | GeForce RTX 5050
11.0 Jetson T5000 | Jetson T4000
10.3 NVIDIA GB300 | NVIDIA B300 | NVIDIA GB300 (DGX Station)
10.0 NVIDIA GB200 | NVIDIA B200
9.0 NVIDIA GH200 | NVIDIA H200 | NVIDIA H100
8.9 NVIDIA L4 | NVIDIA L40 | NVIDIA L40S | NVIDIA RTX 6000 Ada | NVIDIA RTX 5000 Ada | NVIDIA RTX 4500 Ada | NVIDIA RTX 4000 Ada | NVIDIA RTX 4000 SFF Ada | NVIDIA RTX 2000 Ada | GeForce RTX 4090 | GeForce RTX 4080 | GeForce RTX 4070 Ti | GeForce RTX 4070 | GeForce RTX 4060 Ti | GeForce RTX 4060 | GeForce RTX 4050
8.7 Jetson AGX Orin | Jetson Orin NX | Jetson Orin Nano
8.6 NVIDIA A40 | NVIDIA A10 | NVIDIA A16 | NVIDIA A2 | NVIDIA RTX A6000 | NVIDIA RTX A5000 | NVIDIA RTX A4000 | NVIDIA RTX A3000 | NVIDIA RTX A2000 | GeForce RTX 3090 Ti | GeForce RTX 3090 | GeForce RTX 3080 Ti | GeForce RTX 3080 | GeForce RTX 3070 Ti | GeForce RTX 3070 | GeForce RTX 3060 Ti | GeForce RTX 3060 | GeForce RTX 3050 Ti | GeForce RTX 3050
8.0 NVIDIA A100 | NVIDIA A30
7.5 NVIDIA T4 | QUADRO RTX 8000 | QUADRO RTX 6000 | QUADRO RTX 5000 | QUADRO RTX 4000 | QUADRO RTX 3000 | QUADRO  T2000 | NVIDIA T1200 | NVIDIA T1000 | NVIDIA T600 | NVIDIA T500 | NVIDIA T400 | GeForce GTX 1650 Ti | NVIDIA TITAN RTX | GeForce RTX 2080 Ti | GeForce RTX 2080 | GeForce RTX 2070 | GeForce RTX 2060

Therefore if you want to use e.g. an RTX 5070 then specify... 

-DCMAKE_CUDA_ARCHITECTURES="120" \

...or if you want to use e.g. an RTX 5070 and an RTX 3060 then specify... 

-DCMAKE_CUDA_ARCHITECTURES="86;120" \

...and so on.

3b4)

In the shell/terminal which you plan to use to run the configuration and then later the build set the paths where your CUDA executables and libraries are located with e.g.  

export PATH="/opt/cuda/bin:$PATH"
export LD_LIBRARY_PATH="/opt/cuda/lib64:$LD_LIBRARY_PATH"

 

In my case the binaries are located in "/opt/cuda/bin"... 

$ ls /opt/cuda/bin
bin2c  cuda-config  cu++filt   fatbinary  __nvcc_device_query  nvdisasm  nvprune
crt    cudafe++     cuobjdump  nvcc       nvcc.profile         nvlink    ptxas

...and the libraries are located in "/opt/cuda/lib64": 

$ ls /opt/cuda/lib64
cmake libcublasLt.so libcublasLt.so.12
libcublasLt.so.12.9.1.4 libcublasLt_static.a
...
3c) Wrap-up

This would be the final configuration command... 

$ export PATH="/opt/cuda/bin:$PATH"
$ export LD_LIBRARY_PATH="/opt/cuda/lib64:$LD_LIBRARY_PATH"

$ cd llama.cpp-b10867/build

$ cmake .. \
-DCMAKE_C_COMPILER=x86_64-pc-linux-gnu-gcc-14 \
-DCMAKE_CXX_COMPILER=x86_64-pc-linux-gnu-g++-14 \
-DCMAKE_CUDA_HOST_COMPILER=x86_64-pc-linux-gnu-g++-14 \
-DGGML_CCACHE=OFF \
-DGGML_CUDA=ON \
-DCMAKE_CUDA_ARCHITECTURES="86;120" \
-DGGML_NATIVE=ON \
-DGGML_LTO=ON \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_C_FLAGS="-O3 -march=znver5 -pipe -fomit-frame-pointer" \
-DCMAKE_CXX_FLAGS="-O3 -march=znver5 -pipe -fomit-frame-pointer" \
&& echo ok || echo NOK

...which would set in my case the following:

  • I have in my Gentoo Linux distro both GCC 14 and 15 installed and version 15 is set as the main version, therefore I tell cmake to compile "llama.cpp" by using GCC version 14.
  • Compile the support for my "RTX 5070" + "RTX 5060" (compute capability "120") and my "RTX 3060" (compute capability "86").
  • Optimize the code for my zen 5 CPU.

If you make a mistake and want to rerun the configuration then you might potentially want to first delete all the contents of your "build" directory.

If everything goes well you should get at the very end a confirmation:

...
-- Configuring done (3.6s)
-- Generating done (0.2s)
-- Build files have been written to: /......./llama.cpp-b10867/build
ok

4) Compile / Build

To build/compile "llama.cpp" then finally execute in your "build" directory the following: 

cmake --build . --config Release -j14 && echo ok || echo NOK

With "-j<some_number>" you specify how many max parallel processes to use for the compilation.
E.g. my CPU has 16 logical cores so I use 14 parallel processes (a bit less than what is available).

The compilation time varies depending on the version lf "llama.cpp", your compilation options, your CPU and the amount of parallel processes; e.g. on my "AMD Ryzen 7 9700X" with "boost" disabled compiling version "b10867" with max 14 parallel processes takes ~8.5 minutes.

If everything goes well then you should get the final "ok":

...
[100%] Linking CXX executable ../../bin/llama-cli
[100%] Built target llama-cli
[100%] Linking CXX executable ../bin/llama
[100%] Built target llama-app
[100%] Linking CXX executable ../bin/test-chat
[100%] Built target test-chat
[100%] Linking CXX executable ../bin/test-backend-ops
[100%] Built target test-backend-ops
ok

The executables will be located in the "bin" subdirectory "llama.cpp-b10867/build/bin/".

5) Run "llama.cpp"

Not part of this post, anyway to run "llama.cpp" I recommend to use individual shell scripts and to mention in their name some extra informations:

  • the context / type of usage (e.g. OCR, RAG, code assistant, general chat, ...) as you will want to customize Llama's parameters as needed for the specific usecase (e.g. a small context size for OCR but a big one for the general chat).
  • maybe the version of the script (you might want to test multiple similar variants and keep track of what you did in the past).
  • maybe the model name and/or as well its size.

E.g. for a long time I did not mention a context and after a while I forgot which model & model size worked well with my code assistant.

Here as an example my shell script "natgeo-qwen36-clever_but_slow.sh" which starts Llama to be used as an OCR engine to scan and assemble text columns of my old National Geographic magazines:
(yes, I do have as well another script called "natgeo-qwen36-dumb_but_fast.sh") 

#!/bin/bash
LLAMA_DIR="/<redacted>/llama.cpp-selfcompiled/llama.cpp-b10867/build/bin"
MODEL_DIR="/<redacted>/models"

MODEL_FILE="Qwen3.6-27B-Q6_K_L-bartowski-20260524.gguf"
MMPROJ_FILE="Qwen3.6-27B-mmproj-f16-unsloth.gguf"

export CUDA_DEVICE_ORDER=PCI_BUS_ID
export CUDA_VISIBLE_DEVICES=0,1,2

$LLAMA_DIR/llama-server \
--model "$MODEL_DIR/$MODEL_FILE" \
--mmproj "$MODEL_DIR/$MMPROJ_FILE" \
--no-ui \
--flash-attn on \
--main-gpu 0 \
--threads 8 \
--ctx-size 50000 \
--split-mode layer \
--temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0 --repeat-penalty 1.1 --presence-penalty 0.0 \
--host quad \
--fit on \
--fit-target 512 \
--parallel 1 \
--no-mmap \
--mmproj-offload \
-ctk f16 -ctv f16 \
--cache-ram 0 \
--tensor-split 15,24,12 \
--ubatch-size 2048 \
--spec-type draft-mtp --spec-draft-n-max 3 \
--port 10123

Discussions about Llama's parameters will be part of another post.
In general start with a minimum set, then add as needed; keep in mind that things keep changing with each different Llama & model version and that your config will grow together with your knowledge.