Concerns about the environmental impacts of Large Language Models (LLMs) are growing. Although detailed information about the actual costs of LLMs can be difficult to find, let\u2019s attempt to gather some facts to understand the\u00a0scale.<\/p>\n Since comprehensive data on ChatGPT-4 is not readily available, we can consider Llama 3.1 405B as an example. This open-source model from Meta is arguably the most \u201ctransparent\u201d LLM to date. Based on various benchmarks<\/a>, Llama 3.1 405B is comparable to ChatGPT-4, providing a reasonable basis for understanding LLMs within this\u00a0range.<\/p>\n The hardware requirements to run the 32-bit version of this model range from 1,620 to 1,944 GB of GPU memory, depending on the source (substratus<\/a>, HuggingFace<\/a>). For a conservative estimate, let\u2019s use the lower figure of 1,620 GB. To put this into perspective\u200a\u2014\u200aacknowledging that this is a simplified analogy\u200a\u2014\u200a1,620 GB of GPU memory is roughly equivalent to the combined memory of 100 standard MacBook Pros (16GB each). So, when you ask one of these LLMs for a tiramisu recipe in Shakespearean style, it takes the power of 100 MacBook Pros to give you an\u00a0answer.<\/p>\n I\u2019m attempting to translate these figures into something more tangible\u2026 though this doesn\u2019t include the training costs<\/a>, which are estimated to involve around 16,000 GPUs at an approximate cost of $60 million USD (excluding hardware costs)\u200a\u2014\u200aa significant investment from Meta\u200a\u2014\u200ain a process that took around 80 days. In terms of electricity consumption, training required 11\u00a0GWh<\/a>.<\/p>\n The annual electricity consumption per person<\/a> in a country like France is approximately 2,300 kWh. Thus, 11 GWh corresponds to the yearly electricity usage of about 4,782 people. This consumption resulted in the release of approximately 5,000 tons of CO\u2082-equivalent greenhouse gases (based on the European average<\/a>),\u00a0, although this figure can easily double depending on the country where the model was\u00a0trained.<\/p>\n For comparison, burning 1 liter of diesel produces 2.54 kg of CO\u2082. Therefore, training Llama 3.1 405B\u200a\u2014\u200ain a country like France\u200a\u2014\u200ais roughly equivalent to the emissions from burning around 2 million liters of diesel. This translates to approximately 28 million kilometers of car travel. I think that provides enough perspective\u2026 and I haven\u2019t even mentioned the water required to cool the\u00a0GPUs!<\/p>\n Clearly, AI is still in its infancy, and we can anticipate more optimal and sustainable solutions to emerge over time. However, in this intense race, OpenAI\u2019s financial landscape highlights a significant disparity between its revenues and operational expenses, particularly in relation to inference costs. In 2024, the company is projected to spend approximately $4 billion on processing power provided by Microsoft for inference workloads, while its annual revenue is estimated to range between $3.5 billion and $4.5 billion. This means that inference costs alone nearly match\u200a\u2014\u200aor even exceed\u200a\u2014\u200aOpenAI\u2019s total revenue (deeplearning.ai<\/a>).<\/p>\n All of this is happening in a context where experts are announcing a performance plateau for AI models (scaling paradigm). Increasing model size and GPUs are yielding significantly diminished returns compared to previous leaps, such as the advancements GPT-4 achieved over GPT-3. \u201cThe pursuit of AGI has always been unrealistic, and the \u2018bigger is better\u2019 approach to AI was bound to hit a limit eventually\u200a\u2014\u200aand I think this is what we\u2019re seeing here\u201d said Sasha Luccioni<\/a>, researcher and AI lead at startup Hugging\u00a0Face.<\/p>\n But don\u2019t get me wrong\u200a\u2014\u200aI\u2019m not putting AI on trial, because I love it! This research phase is absolutely a normal stage in the development of AI. However, I believe we need to exercise common sense in how we use AI: we can\u2019t use a bazooka to kill a mosquito every time. AI must be made sustainable\u200a\u2014\u200anot only to protect our environment but also to address social divides. Indeed, the risk of leaving the Global South behind in the AI race due to high costs and resource demands would represent a significant failure in this new intelligence revolution..<\/p>\n So, do you really need the full power of ChatGPT to handle the simplest tasks in your RAG pipeline? Are you looking to control your operational costs? Do you want complete end-to-end control over your pipeline? Are you concerned about your private data circulating on the web? Or perhaps you\u2019re simply mindful of AI\u2019s impact and committed to its conscious use?<\/p>\n Small language models (SLMs) offer an excellent alternative worth exploring. They can run on your local infrastructure and, when combined with human intelligence, deliver substantial value. Although there is no universally agreed definition of an SLM\u200a\u2014\u200ain 2019, for instance, GPT-2 with its 1.5 billion parameters was considered an LLM, which is no longer the case\u200a\u2014\u200aI am referring to models such as Mistral 7B, Llama-3.2 3B, or Phi3.5, to name a few. These models can operate on a \u201cgood\u201d computer, resulting in a much smaller carbon footprint while ensuring the confidentiality of your data when installed on-premise. Although they are less versatile, when used wisely for specific tasks, they can still provide significant value\u200a\u2014\u200awhile being more environmentally virtuous.<\/p>\n Smaller is smarter<\/a> was originally published in Towards Data Science<\/a> on Medium, where people are continuing the conversation by highlighting and responding to this story.<\/p>\n<\/div>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2A9tyzhcJGdDxLBo_-dsoAqQ.png?ssl=1\")
Inference<\/h3>\n
Training<\/h3>\n
Sustainability<\/h3>\n
And now?<\/h3>\n
SLM can be a smarter\u00a0choice!<\/h3>\n
<\/p>\n
\n