With the recent explosion of interest in large language models (LLMs), they often seem almost magical. But let\u2019s demystify them.<\/p>\n
I wanted to step back and unpack the fundamentals\u200a\u2014\u200abreaking down how LLMs are built, trained, and fine-tuned to become the AI systems we interact with today.<\/p>\n
This two-part deep dive<\/strong> is something I\u2019ve been meaning to do for a while and was also inspired by Andrej Karpathy\u2019s widely popular 3.5-hour YouTube<\/a> video, which has racked up 800,000+ views in just 10 days. Andrej is a founding member of OpenAI, his insights are gold\u2014 you get the idea.<\/p>\n If you have the time, his video is definitely worth watching<\/strong>. But let\u2019s be real\u200a\u2014\u200a3.5 hours is a long watch. So, for all the busy folks who don\u2019t want to miss out, I\u2019ve distilled the key concepts from the first 1.5 hours into this 10-minute read<\/strong>, adding my own breakdowns to help you build a solid intuition.<\/p>\n Part 1 (this article): Covers the fundamentals of LLMs, including pre-training to post-training, neural networks, Hallucinations<\/a>, and inference.<\/p>\n Part 2: Reinforcement learning with human\/AI feedback, investigating o1 models, DeepSeek R1, AlphaGo<\/p>\n Let\u2019s go! I\u2019ll start with looking at how LLMs are being built.<\/p>\n At a high level, there are 2 key phases: pre-training and post-training.<\/p>\n Before an LLM can generate text, it must first learn how language works. This happens through pre-training, a highly computationally intensive task.<\/p>\n The first step in training an LLM is gathering as much high-quality text as possible. The goal is to create a massive and diverse dataset containing a wide range of human knowledge.<\/p>\n One source is Common Crawl<\/a>, which is a free, open repository of web crawl data containing 250 billion web pages over 18 years. However, raw web data is noisy\u200a\u2014\u200acontaining spam, duplicates and low quality content\u200a\u2014\u200aso preprocessing is essential.If you\u2019re interested in preprocessed datasets, FineWeb offers a curated version of Common Crawl, and is made available on Hugging Face<\/a>.<\/p>\n Once cleaned, the text corpus is ready for tokenization.<\/p>\n Before a neural network can process text, it must be converted into numerical form. This is done through tokenization<\/strong>, where words, subwords, or characters are mapped to unique numerical tokens.<\/p>\n Think of tokens as the building blocks\u200a\u2014\u200athe fundamental building blocks of all language models. In GPT4, there are 100,277 possible tokens.A popular tokenizer, Tiktokenizer<\/a>, allows you to experiment with tokenization and see how text is broken down into tokens. Try entering a sentence, and you\u2019ll see each word or subword assigned a series of numerical IDs.<\/p>\n Once the text is tokenized, the neural network learns to predict the next token based on its context. As shown above, the model takes an input sequence of tokens (e.g., \u201cwe are cook ing\u201d<\/em>) and processes it through a giant mathematical expression\u200a\u2014\u200awhich represents the model\u2019s architecture\u200a\u2014\u200ato predict the next token.<\/p>\n A neural network consists of 2 key parts:<\/p>\n Initially, the model\u2019s predictions are random, but as training progresses, it learns to assign probabilities to possible next tokens.<\/p>\n When the correct token (e.g. \u201cfood\u201d) is identified, the model adjusts its billions of parameters (weights) through backpropagation<\/strong>\u200a\u2014\u200aan optimization process that reinforces correct predictions by increasing their probabilities while reducing the likelihood of incorrect ones.<\/p>\n This process is repeated billions of times across massive datasets.<\/p>\n At this stage, the base model has learned:<\/p>\n However, base models are not yet optimised for real-world tasks<\/strong>. You can think of them as an advanced autocomplete system\u200a\u2014\u200athey predict the next token based on probability, but with limited instruction-following ability.<\/p>\n A base model can sometimes recite training data verbatim and can be used for certain applications through in-context learning<\/strong>, where you guide its responses by providing examples in your prompt. However, to make the model truly useful and reliable, it requires further training.<\/p>\n Base models are raw and unrefined. To make them helpful, reliable, and safe, they go through post-training, where they are fine-tuned on smaller, specialised datasets.<\/p>\n Because the model is a neural network, it cannot be explicitly programmed like traditional software. Instead, we \u201cprogram\u201d it implicitly by training it on structured labeled datasets that represent examples of desired interactions.<\/strong><\/p>\n Specialised datasets are created, consisting of structured examples on how the model should respond in different situations.\u00a0<\/p>\n Some types of post training include:<\/p>\n Post training also introduces special tokens\u200a<\/strong>\u2014\u200asymbols that were not used during pre-training\u200a\u2014\u200ato help the model understand the structure of interactions. These tokens signal where a user\u2019s input starts and ends and where the AI\u2019s response begins, ensuring that the model correctly distinguishes between prompts and replies.<\/p>\n Now, we\u2019ll move on to some other key concepts.<\/p>\n Inference<\/a> can be performed at any stage, even midway through pre-training, to evaluate how well the model has learned.<\/p>\n When given an input sequence of tokens, the model assigns probabilities to all possible next tokens based on patterns it has learned during training.<\/p>\n Instead of always choosing the most likely token, it samples from this probability distribution\u200a\u2014\u200asimilar to flipping a biased coin, where higher-probability tokens are more likely to be selected.<\/strong><\/p>\n This process repeats iteratively, with each newly generated token becoming part of the input for the next prediction.\u00a0<\/p>\n Token selection is stochastic <\/strong>and <\/strong>the same input can produce different outputs. Over time, the model generates text that wasn\u2019t explicitly in its training data but follows the same statistical patterns.<\/p>\n Hallucinations happen because LLMs do not \u201cknow\u201d facts\u200a\u2014\u200athey simply predict the most statistically likely sequence of words based on their training data.<\/p>\n Early models struggled significantly with hallucinations.<\/p>\n For instance, in the example below, if the training data contains many \u201cWho is\u2026\u201d questions with definitive answers, the model learns that such queries should always have confident responses, even when it lacks the necessary knowledge.<\/p>\n When asked about an unknown person, the model does not default to \u201cI don\u2019t know\u201d because this pattern was not reinforced during training. Instead, it generates its best guess, often leading to fabricated information.<\/p>\n Improving factual accuracy requires explicitly training the model to recognise what it does not know\u200a\u2014\u200aa task that is more complex than it seems.<\/p>\n This is done via self interrogation<\/strong>, a process that helps define the model\u2019s knowledge boundaries.<\/p>\n Self interrogation can be automated using another AI model, which generates questions to probe knowledge gaps. If it produces a false answer, new training examples are added, where the correct response is: \u201cI\u2019m not sure. Could you provide more context?\u201d<\/em><\/p>\n If a model has seen a question many times in training, it will assign a high probability to the correct answer.<\/p>\n If the model has not encountered the question before, it distributes probability more evenly across multiple possible tokens, making the output more randomised. No single token stands out as the most likely choice.<\/strong><\/p>\n Fine tuning explicitly trains the model to handle low-confidence outputs with predefined responses.\u00a0<\/p>\n For example, when I asked ChatGPT-4o, \u201cWho is asdja rkjgklfj?<\/em>\u201d, it correctly responded: \u201cI\u2019m not sure who that is. Could you provide more context?\u201d<\/p>\n A more advanced method is to extend the model\u2019s knowledge beyond its training data by giving it access to external search tools.<\/p>\n At a high level, when a model detects uncertainty, it can trigger a web search. The search results are then inserted into a model\u2019s context window\u200a\u2014\u200aessentially allowing this new data to be part of it\u2019s working memory. The model references this new information while generating a response.<\/p>\n Generally speaking, LLMs have two types of knowledge access.<\/p>\n Adding relevant facts within the context window significantly improves response quality.<\/p>\n When asked questions like \u201cWho are you?\u201d<\/em> or \u201cWhat built you?\u201d<\/em>, an LLM will generate a statistical best guess based on its training data, unless explicitly programmed to respond accurately.\u00a0<\/p>\n LLMs do not have true self-awareness, their responses depend on patterns seen during training.<\/p>\n One way to provide the model with a consistent identity is by using a system prompt<\/strong>, which sets predefined instructions about how it should describe itself, its capabilities, and its limitations.<\/p>\n That\u2019s a wrap for Part 1! I hope this has helped you build intuition on how LLMs work. In Part 2, we\u2019ll dive deeper into reinforcement learning and some of the latest models.<\/p>\n Got questions or ideas for what I should cover next? Drop them in the comments\u200a\u2014\u200aI\u2019d love to hear your thoughts. See you in Part 2! The post How LLMs Work: Pre-Training to Post-Training, Neural Networks, Hallucinations, and Inference<\/a> appeared first on Towards Data Science<\/a>.<\/p>\n<\/div>\nWhat you\u2019ll get<\/strong><\/h2>\n
1. Pre-training<\/strong><\/h3>\n
Step 1: Data collection and preprocessing<\/h4>\n
![\"\"](\"https:\/\/i0.wp.com\/towardsdatascience.com\/wp-content\/uploads\/2025\/02\/Screenshot-2025-02-18-at-9.25.27%25E2%2580%25AFAM-1024x583.png?resize=1024%2C583&ssl=1\")
<\/figure>\nStep 2: Tokenization<\/h4>\n
![\"\"](\"https:\/\/i0.wp.com\/towardsdatascience.com\/wp-content\/uploads\/2025\/02\/Screenshot-2025-02-18-at-9.25.48%25E2%2580%25AFAM-1024x611.png?resize=1024%2C611&ssl=1\")
<\/figure>\nStep 3: Neural network training<\/h4>\n
\n
![\"\"](\"https:\/\/i0.wp.com\/towardsdatascience.com\/wp-content\/uploads\/2025\/02\/Screenshot-2025-02-18-at-9.26.09%25E2%2580%25AFAM-1024x527.png?resize=1024%2C527&ssl=1\")
<\/figure>\nBase model\u200a\u2014\u200athe output of pre-training<\/strong><\/h4>\n
\n
2. Post training\u200a\u2014\u200aMaking the model useful<\/h3>\n
How post training works<\/h4>\n
\n
<\/strong>Goal: To teach the model to follow instructions, be task oriented, engage in multi-turn conversations, follow safety guidelines and refuse malicious requests, etc.
Eg: InstructGPT (2022)<\/a>: OpenAI hired some 40 contractors to create these labelled datasets. These human annotators wrote prompts and provided ideal responses based on safety guidelines. Today, many datasets are generated automatically, with humans reviewing and editing them for quality.<\/li>\n
<\/strong>Goal: Adapt the model for specialised fields like medicine, law and programming.<\/li>\n<\/ol>\nInference\u200a\u2014\u200ahow the model generates new text<\/strong><\/h2>\n
Hallucinations\u200a\u2014\u200awhen LLMs generate false info<\/strong><\/h2>\n
Why do hallucinations occur?<\/strong><\/h3>\n
![\"\"](\"https:\/\/i0.wp.com\/towardsdatascience.com\/wp-content\/uploads\/2025\/02\/Screenshot-2025-02-18-at-9.26.24%25E2%2580%25AFAM-1024x583.png?resize=1024%2C583&ssl=1\")
<\/figure>\nHow do you reduce hallucinations?<\/strong><\/h3>\n
Method 1: Saying \u201cI don\u2019t know\u201d<\/h4>\n
Method 2: Doing a web search<\/h4>\n
Vague recollections vs working memory<\/h2>\n
\n
Knowledge of self\u00a0<\/h2>\n
To end off<\/h2>\n
![\"\ud83d\ude42\"](\"https:\/\/i0.wp.com\/s.w.org\/images\/core\/emoji\/15.0.3\/72x72\/1f642.png?ssl=1\")
<\/p>\n