llama.cpp: LLM inference in C/C++

Once installed:

1. Example 1
# Download and run a model directly from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF
2. Example 2
# Launch OpenAI-compatible API server
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF
3. Example 3
# Run a vision model for tagging photos.
wget 'https://huggingface.co/openbmb/MiniCPM-V-4.7-1B-gguf\
/resolve/main/MiniCPM-V-4_7-1B-Q4_0.gguf?download=true
wget 'https://huggingface.co/openbmb/MiniCPM-V-4.7-1B-gguf\
/resolve/main/mmproj-model-f16-4x.gguf?download=true'
export GGML_CUDA_ENABLE_UNIFIED_MEMORY=1
llama-mtmd-cli -m MiniCPM-V-4_7-1B-Q4_0.gguf --mmproj \
mmproj-model-f16.gguf -c 4096 --jinja --image my-image.jpg\
-p "What is in the image?"

*Note*: The LLM world evolves very quickly, so even the most recent
released llama.cpp does not support the most recent models, which are
targeting git-master. The version 0.6.0 (which was the packaged
version at 2026-10-14) supports models created before mid-2026, and
after about 2024.

Description
The main goal of llama.cpp is to enable LLM (and VLM) inference with
minimal setup and state-of-the-art performance on a wide range of
hardware - locally and in the cloud.

Plain C/C++ implementation without any dependencies
AVX, AVX2, AVX512 and AMX support for x86 architectures
1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer
quantization for faster inference and reduced memory use
Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD
GPUs via HIP and Moore Threads GPUs via MUSA)
Vulkan and SYCL backend support
CPU+GPU hybrid inference to partially accelerate models larger than
the total VRAM capacity

TODOs: At the moment the build works for Nvidia CUDA with Unified
Memory. In theory, it is possible to also support the following
configurations, if somebody tells me how to compile the deps:

1. rocmtoolkit_7  -- for AMD GPU support
2. openvino       -- for Intel GPU support
3. nccl           -- support for multiple Nvidia GPUs