{"id":1168,"date":"2025-01-14T07:02:35","date_gmt":"2025-01-14T07:02:35","guid":{"rendered":"https:\/\/mailitics.com\/index.php\/2025\/01\/14\/llama-cpp-writing-a-simple-c-inference-program-for-gguf-llm-models-12bc5f58505f\/"},"modified":"2025-01-14T07:02:35","modified_gmt":"2025-01-14T07:02:35","slug":"llama-cpp-writing-a-simple-c-inference-program-for-gguf-llm-models-12bc5f58505f","status":"publish","type":"post","link":"https:\/\/mailitics.com\/index.php\/2025\/01\/14\/llama-cpp-writing-a-simple-c-inference-program-for-gguf-llm-models-12bc5f58505f\/","title":{"rendered":"llama.cpp: Writing A Simple C++ Inference Program for GGUF LLM Models"},"content":{"rendered":"

llama.cpp: Writing A Simple C++ Inference Program for GGUF LLM Models
\n \t

\n
<\/BR>
\n
\n \t

\n
<\/BR><\/p>\n

\n

Exploring llama.cpp internals and a basic chat program\u00a0flow<\/h4>\n
\"\"
Photo by Mathew Schwartz<\/a> on\u00a0Unsplash<\/a><\/figcaption><\/figure>\n

llama.cpp has revolutionized the space of LLM inference by the means of wide adoption and simplicity. It has enabled enterprises and individual developers to deploy LLMs on devices ranging from SBCs to multi-GPU clusters. Though working with llama.cpp has been made easy by its language bindings, working in C\/C++ might be a viable choice for performance sensitive or resource constrained scenarios.<\/p>\n

This tutorial aims to let readers have a detailed look on how LLM inference is performed using low-level functions coming directly from llama.cpp. We discuss the program flow, llama.cpp constructs and have a simple chat at the\u00a0end.<\/p>\n

The C++ code that we will write in this blog is also used in SmolChat, a native Android application that allows users to interact with LLMs\/SLMs in the chat interface, completely on-device. Specifically, the LLMInference class we define ahead is used with the JNI binding to execute GGUF\u00a0models.<\/p>\n

GitHub – shubham0204\/SmolChat-Android: Running any GGUF SLMs\/LLMs locally, on-device in Android<\/a><\/p>\n

The code for this tutorial can be found\u00a0here:<\/p>\n

shubham0204\/llama.cpp-simple-chat-interface<\/a><\/p>\n

The code is also derived from the official <\/a>simple-chat example<\/a> from llama.cpp.<\/p>\n

Contents<\/h3>\n
    \n
  1. About llama.cpp<\/a><\/li>\n
  2. Setup<\/a><\/li>\n
  3. Loading the\u00a0Model<\/a><\/li>\n
  4. Performing Inference<\/a><\/li>\n
  5. Good Habits: Writing a Destructor<\/a><\/li>\n
  6. Running the Application<\/a><\/li>\n<\/ol>\n

    About llama.cpp<\/h3>\n

    llama.cpp<\/a> is a C\/C++ framework to infer machine learning models defined in the GGUF format<\/a> on multiple execution backends<\/em>. It started as a pure C\/C++ implementation of the famous Llama series LLMs from Meta that can be inferred on Apple\u2019s silicon, AVX\/AVX-512, CUDA, and Arm Neon-based environments. It also includes a CLI-based tool llama-cli to run GGUF LLM models and llama-server to execute models via HTTP requests (OpenAI compatible server).<\/p>\n

    llama.cpp uses ggml<\/a>, a low-level framework that provides primitive functions required by deep learning models and abstracts backend implementation details from the user. Georgi Gerganov<\/a> is the creator of ggml and llama.cpp.<\/p>\n

    The repository\u2019s README<\/a> also lists wrappers built on top of llama.cpp in other programming languages. Popular tools like Ollama<\/a> and LM Studio<\/a> also use bindings over llama.cpp to enhance user friendliness. The project has no dependencies on other third-party libraries<\/p>\n

    How is llama.cpp different from PyTorch\/TensorFlow?<\/h4>\n

    llama.cpp has emphasis on inference of ML models<\/em><\/strong> from its inception, whereas PyTorch<\/a> and TensorFlow<\/a> are end-to-end solutions offering data processing, model training\/validation, and efficient inference in one\u00a0package.<\/p>\n

    PyTorch and TensorFlow also have their lightweight inference-only extensions namely ExecuTorch<\/a> and TensorFlow Lite<\/a>\n<\/p><\/blockquote>\n

    Considering only the inference phase of a model, llama.cpp is lightweight<\/em><\/strong> in its implementation due to the absence of third-party dependencies and an extensive set of available operators or model formats to support. Also, as the name suggests, the project started as an efficient library to infer LLMs (the Llama model<\/a> from Meta) and continues to support a wide range of open-source LLM architectures<\/em><\/strong>.<\/p>\n

    \nAnalogy<\/strong>: If PyTorch\/TensorFlow are luxurious, power-hungry cruise ships, llama.cpp is small, speedy motorboat. PyTorch\/TF and llama.cpp have their own use-cases.<\/p><\/blockquote>\n

    Setup<\/h3>\n

    We start our implementation in a Linux-based environment (native or WSL) with cmake installed and the GNU\/clang toolchain installed. We\u2019ll compile llama.cpp from source and add it as a shared library to our executable chat\u00a0program.<\/p>\n

    We create our project directory smol_chat with aexternals directory to store the cloned llama.cpp repository.<\/p>\n

    mkdir smol_chat
    cd smol_chat

    mkdir src
    mkdir externals
    touch CMakeLists.txt

    cd externals
    git clone --depth=1 https:\/\/github.com\/ggerganov\/llama.cpp<\/pre>\n
    CMakeLists.txt is where we define our build, allowing CMake to compile our C\/C++ code using the default toolchain (GNU\/clang) by including headers and shared libraries from externals\/llama.cpp\u2063.<\/p>\n
    cmake_minimum_required(VERSION 3.10)
    project(llama_inference)

    set(CMAKE_CXX_STANDARD 17)
    set(LLAMA_BUILD_COMMON On)
    add_subdirectory(\"${CMAKE_CURRENT_SOURCE_DIR}\/externals\/llama.cpp\")

    add_executable(
        chat
        src\/LLMInference.cpp src\/main.cpp
    )
    target_link_libraries(
        chat 
        PRIVATE
        common llama ggml
    )<\/pre>\n
    Loading the\u00a0Model<\/h3>\n
    We have now defined how our project should be built by CMake. Next, we create a header file LLMInference.h which declares a class containing high-level functions to interact with the LLM. llama.cpp provides a C-style API, thus embedding it within a class will help us abstract\/hide the inner working\u00a0details.<\/p>\n
    #ifndef LLMINFERENCE_H
    #define LLMINFERENCE_H

    #include \"common.h\"
    #include \"llama.h\"
    #include <string>
    #include <vector>

    class LLMInference {

        \/\/ llama.cpp-specific types
        llama_context* _ctx;
        llama_model* _model;
        llama_sampler* _sampler;
        llama_batch _batch;
        llama_token _currToken;
        
        \/\/ container to store user\/assistant messages in the chat
        std::vector<llama_chat_message> _messages;
        \/\/ stores the string generated after applying
        \/\/ the chat-template to all messages in `_messages`
        std::vector<char> _formattedMessages;
        \/\/ stores the tokens for the last query
        \/\/ appended to `_messages`
        std::vector<llama_token> _promptTokens;
        int _prevLen = 0;

        \/\/ stores the complete response for the given query
        std::string _response = \"\";

        public:

        void loadModel(const std::string& modelPath, float minP, float temperature);

        void addChatMessage(const std::string& message, const std::string& role);
        
        void startCompletion(const std::string& query);

        std::string completionLoop();

        void stopCompletion();

        ~LLMInference();
    };

    #endif<\/pre>\n
    The private members declared in the header above will be used in the implementation of the public member functions described in the further sections of the blog. Let us define each of these member functions in LLMInference.cpp\u00a0.<\/p>\n
    #include \"LLMInference.h\"
    #include <cstring>
    #include <iostream>

    void LLMInference::loadModel(const std::string& model_path, float min_p, float temperature) {
        \/\/ create an instance of llama_model
        llama_model_params model_params = llama_model_default_params();
        _model = llama_load_model_from_file(model_path.data(), model_params);

        if (!_model) {
            throw std::runtime_error(\"load_model() failed\");
        }

        \/\/ create an instance of llama_context
        llama_context_params ctx_params = llama_context_default_params();
        ctx_params.n_ctx = 0;               \/\/ take context size from the model GGUF file
        ctx_params.no_perf = true;          \/\/ disable performance metrics
        _ctx = llama_new_context_with_model(_model, ctx_params);

        if (!_ctx) {
            throw std::runtime_error(\"llama_new_context_with_model() returned null\");
        }

        \/\/ initialize sampler
        llama_sampler_chain_params sampler_params = llama_sampler_chain_default_params();
        sampler_params.no_perf = true;      \/\/ disable performance metrics
        _sampler = llama_sampler_chain_init(sampler_params);
        llama_sampler_chain_add(_sampler, llama_sampler_init_min_p(min_p, 1));
        llama_sampler_chain_add(_sampler, llama_sampler_init_temp(temperature));
        llama_sampler_chain_add(_sampler, llama_sampler_init_dist(LLAMA_DEFAULT_SEED));

        _formattedMessages = std::vector<char>(llama_n_ctx(_ctx));
        _messages.clear();
    }<\/pre>\n
    llama_load_model_from_filereads the model from the file using llama_load_model internally and populates the llama_model instance using the given llama_model_params\u00a0. The user can give the parameters, but we can get a pre-initialized default<\/em> struct for it with llama_model_default_params\u00a0.<\/p>\n
    llama_context represents the execution environment for the GGUF model loaded. The llama_new_context_with_model instantiates a new llama_context and prepares a backend for execution by either reading the llama_model_params or by automatically detecting the available backends. It also initializes the K-V cache, which is important in the decoding or inference step. A backend scheduler that manages computations across multiple backends is also initialized.<\/p>\n
    Optimizing LLM Inference: Managing the KV Cache<\/a><\/p>\n
    A llama_sampler determines how we sample\/choose tokens from the probability distribution derived from the outputs (logits) of the model (specifically the decoder of the LLM). LLMs assign a probability to each token present in the vocabulary, representing the chances of the token appearing next in the sequence. The temperature and min-p that we are setting with llama_sampler_init_temp and llama_sampler_init_min_p are two parameters controlling the token sampling\u00a0process.<\/p>\n
    Setting Top-K, Top-P and Temperature in LLMs<\/a><\/p>\n
    Performing Inference<\/h3>\n
    There are multiple steps involved in the inference process that takes a text query from the user as input and returns the LLM\u2019s response.<\/p>\n
    1. Applying the chat template to the\u00a0queries<\/strong><\/h4>\n
    For an LLM, the incoming messages are classified as belonging to three roles, user\u00a0, assistant and system\u00a0. user and assistant messages given by the user and the LLM, respectively, whereas system denotes a system-wide prompt that is followed across the entire conversation. Each message consists of a role and content where content is the actual text and role is any one of the three\u00a0roles.<\/p>\n
    <example><\/pre>\n
    The system prompt is the first message of the conversation. In our code, the messages are stored as a std::vector<llama_chat_message> named _messages where llama_chat_message is a llama.cpp struct with role and content attributes. We use the llama_chat_apply_template function from llama.cpp to apply the chat template stored in the GGUF file as metadata. We store the string or std::vector<char> obtained after applying the chat template in _formattedMessages\u00a0.<\/p>\n
    2. Tokenization<\/h4>\n
    Tokenization is the process of dividing a given text into smaller parts (tokens). We assign each part\/token a unique integer ID, thus transforming the input text to a sequence of integers that form the input to the LLM. llama.cpp provides the common_tokenize or llama_tokenize functions to perform tokenization, where common_tokenize returns the sequence of tokens as a std::vector<llama_token>\u00a0.<\/p>\n
    void LLMInference::startCompletion(const std::string& query) {
        addChatMessage(query, \"user\");

        \/\/ apply the chat-template 
        int new_len = llama_chat_apply_template(
                _model,
                nullptr,
                _messages.data(),
                _messages.size(),
                true,
                _formattedMessages.data(),
                _formattedMessages.size()
        );
        if (new_len > (int)_formattedMessages.size()) {
            \/\/ resize the output buffer `_formattedMessages`
            \/\/ and re-apply the chat template
            _formattedMessages.resize(new_len);
            new_len = llama_chat_apply_template(_model, nullptr, _messages.data(), _messages.size(), true, _formattedMessages.data(), _formattedMessages.size());
        }
        if (new_len < 0) {
            throw std::runtime_error(\"llama_chat_apply_template() in LLMInference::start_completion() failed\");
        }
        std::string prompt(_formattedMessages.begin() + _prevLen, _formattedMessages.begin() + new_len);
        
        \/\/ tokenization
        _promptTokens = common_tokenize(_model, prompt, true, true);

        \/\/ create a llama_batch containing a single sequence
        \/\/ see llama_batch_init for more details
        _batch.token = _promptTokens.data();
        _batch.n_tokens = _promptTokens.size();
    }<\/pre>\n
    In the code, we apply the chat template and perform tokenization in the LLMInference::startCompletion method and then create a llama_batch instance holding the final inputs for the\u00a0model.<\/p>\n
    3. Decoding, Sampling and the KV\u00a0Cache<\/h4>\n
    As highlighted earlier, LLMs generate a response by successively predicting the next token in the given sequence. LLMs are also trained to predict a special end-of-generation (EOG) token, indicating the end of the sequence of the predicted tokens. The completion_loop function returns the next token in the sequence and keeps getting called until the token it returns is the EOG\u00a0token.<\/p>\n