Avoidable and Unavoidable Randomness in GPT-4o
\n \t
\n
<\/BR>
\n
\n \t
\n
<\/BR><\/p>\n
Of course there is randomness in GPT-4o\u2019s outputs. After all, the model samples from a probability distribution when choosing each token. But what I didn\u2019t understand was that those very probabilities themselves are not deterministic. Even with consistent prompts, fixed seeds, and temperature set to zero, GPT-4o still introduces subtle, frustrating randomness.<\/p>\n
There\u2019s no fix for this, and it might not even be something OpenAI could<\/em> fix if they wanted to, just so we\u2019re clear up front about where this article is headed. Along the way, we\u2019ll examine all the sources of randomness in GPT-4o output, which will require us to break down the sampling process to a low level. We\u2019ll point at the issue\u2014the probabilities vary\u2014and critically examine OpenAI\u2019s official guidance on determinism.<\/p>\n First, though, let\u2019s talk about why determinism matters. Determinism means that the same input always produces the same output, like a mathematical function. While LLM creativity is often desirable, determinism serves crucial purposes: researchers need it for reproducible experiments, developers for verifying reported results, and prompt engineers for debugging their changes. Without it, you\u2019re left wondering if different outputs stem from your tweaks or just the random number generator\u2019s mood swings.<\/p>\n We\u2019re going to keep things extremely simple here and prompt the most recent version of GPT-4o ( Flipping a coin with LLMs is a fascinating topic in itself (see for example Van Koevering & Kleinberg, 2024 in the references), but here, we\u2019ll use it as a simple binary question with which to explore determinism, or the lack thereof.<\/p>\n This is our first attempt.<\/p>\n Running the code gave me The code first initializes an OpenAI client with an API key set in the environment variable Now let\u2019s flip the coin ten times. If this were a real, fair coin, of course, we would expect roughly equal heads and tails over time thanks to the law of large numbers. But GPT-4o\u2019s coin doesn\u2019t work quite like that.<\/p>\n Running this code gave me the following output, although you might get different output, of course.<\/p>\n GPT-4o\u2019s coin is clearly biased, but so are humans. Bar-Hillel, Peer, and Acquisti (2014) found that people flipping imaginary coins choose \u201cheads\u201d 80% of the time. Maybe GPT-4o learned that from us. But whatever the reason, we\u2019re just using this simple example to explore determinism.<\/p>\n Let\u2019s say we wanted to know precisely what percentage of GPT-4o coin flips land Heads.<\/p>\n Rather than the obvious (but expensive) approach of flipping it a million times, there\u2019s a smarter way. For classification tasks with a small set of possible answers, we can extract token probabilities instead of generating full responses. With the right prompt, the first token carries all the necessary information, making these API calls incredibly cheap: around 30,000 calls per dollar, since each requires just 18 (cached) input tokens and 1 output token.<\/p>\n OpenAI gives us (natural) log probabilities. These are called logprobs in the code, and we convert them to regular probabilities by exponentiation. (We\u2019ll discuss temperature soon, but note that exponentiating logprobs directly like this corresponds to a temperature setting of 1.0, and is how we calculate probabilities throughout this article). OpenAI lets us request logprobs for the top 20 most likely tokens, so we do that.<\/p>\n If you run this, you\u2019ll get something<\/em> like the following output, but actual numbers will<\/em> vary.<\/p>\n Looking at these probabilities, we see Heads at \u224896% and T at \u22484%. Our prompt is doing pretty well at constraining the model\u2019s responses. Why Run the code to display the probabilities for the top 20 tokens again, and you\u2019ll likely get different numbers. Here\u2019s what I got on a second running.<\/p>\n In their cookbook<\/a>, OpenAI offers the following advice on receiving \u201cmostly identical\u201d outputs:<\/p>\n If the They also give \u201cmostly identical\u201d advice in the reproducible outputs section<\/a> of their documentation.<\/p>\n The request parameters that could affect randomness are Temperature controls how random the model\u2019s responses are. Low temperatures (<0.5) make it robotic and predictable, medium temperatures (0.7\u20131.3) allow some creativity, and high temperatures (>1.5) produce gibberish. Temperature is often called the \u201ccreativity parameter\u201d, but this is an oversimplification. In their analysis, Peeperkorn, Kouwenhoven, Brown, and Jordanous (2024) evaluated LLM outputs across four dimensions of creativity: novelty (originality), coherence (logical consistency), cohesion (how well the text flows), and typicality (how well it fits expected patterns). They observed that:<\/p>\n temperature is weakly correlated with novelty, and unsurprisingly, moderately correlated with incoherence, but there is no relationship with either cohesion or typicality.<\/p>\n<\/blockquote>\n But, this is beside the point for coin flipping. Under the hood, the log probabilities are divided by the temperature before they\u2019re renormalized and exponentiated to be converted to probabilities. This creates a non-linear effect: What about In summary: if the logprobs aren\u2019t deterministic, setting temperature to 0.0 won\u2019t make the model deterministic<\/strong>.<\/p>\n In fact, since we\u2019re just asking the model for the raw logprobs directly rather than generating full responses, the temperature setting doesn\u2019t come into play in our code at all.<\/p>\n After temperature is used to compute probabilities, the model samples from these probabilities to pick the next token. OpenAI gives us a little control over the sampling process by letting us set the In summary: if the logprobs aren\u2019t deterministic, setting a seed won\u2019t make the model deterministic<\/strong>.<\/p>\n In fact, The Let\u2019s confirm what we\u2019ve been building toward. We\u2019ll run the same request 10 times with every safeguard in place. Even though neither of these parameters should<\/em> matter for what we\u2019re doing, you can never be too safe, so we\u2019ll set Running this 10 times, here are the logprobs and probabilities for the token Heads:<\/p>\n OpenAI is decidedly not open about the architecture behind GPT-4o. However, it\u2019s widely believed that GPT-4o uses a mixture-of-experts (MoE) architecture with either 8 or 16 experts.<\/p>\n According to a paper by Google DeepMind researchers Puigcerver, Riquelme, Mustafa, and Houlsby (hat tip to user elmstedt on the OpenAI forum<\/a>), mixture-of-experts architectures may add an unavoidable level of non-determinism:<\/p>\n Under capacity constraints, all Sparse MoE approaches route tokens in groups of a fixed size and enforce (or encourage) balance within the group. When groups contain tokens from different sequences or inputs, these tokens compete for available spots in expert buffers. Therefore, the model is no longer deterministic at the sequence-level, but only at the batch-level.<\/p>\n<\/blockquote>\n In other words, when your prompt (a sequence of tokens<\/em>, in the quote above) reaches OpenAI\u2019s servers, it gets batched with a group of other prompts (OpenAI isn\u2019t open about how many other prompts). Each prompt in the batch is then routed to an \u201cexpert\u201d within the model. However, since only so many prompts can be routed to the same expert, the expert your prompt gets routed to will depend on all the other prompts in the batch.<\/p>\n This \u201ccompetition\u201d for experts introduces a real-world randomness completely beyond our control.<\/p>\n While non-determinism may be inherent to real-world mixture-of-experts models, that does not seem to be the only<\/em> source of non-determinism in OpenAI\u2019s models.<\/p>\n Making a few changes to our code above (switching to No one is claiming that GPT-3.5-turbo uses a mixture-of-experts architecture. Thus, there must be additional factors beyond mixture-of-experts contributing to this non-determinism.<\/p>\n To better understand the patterns and magnitude of this non-determinism, I conducted a more extensive experiment with GPT-4o, performing 10,000 \u201ccoin flips\u201d while recording the probability assigned to \u201cHeads\u201d in each case.<\/p>\n The results reveal something fascinating. Across 10,000 API calls with identical parameters, GPT-4o produced not just a few different probability values, but 42 distinct probabilities<\/strong>. If the mixture-of-experts hypothesis were the complete explanation for non-determinism in GPT-4o, we might expect to see one distinct probability for each expert. But GPT-4o is believed to have either 8 or 16 experts, not 42<\/strong>.<\/p>\n In the output below, I clustered these probabilities, ensuring that each cluster was separated from the others by 0.01 (as a raw percentage). This groups the output into 12 clusters<\/strong>.<\/p>\n (With a threshold of 0.001 there are 13 clusters, and with a threshold of 0.0001 there are 17 clusters.)<\/p>\n As the chart above demonstrates, this multitude of results cannot be explained by It could<\/em> be that these 12 clusters of probabilities represent 12 different experts<\/strong>. Even assuming that, the variations within the clusters remain puzzling. These don\u2019t seem likely to be simple rounding errors, because they are too systematic and consistent. Take the giant cluster at around 96.266% with two distinct probabilities representing over half of our coin flips. The difference between these two probabilities, 0.0000448274%, is tiny but persistent.<\/p>\n There is an underlying randomness in the log probabilities returned by all currently available non-thinking OpenAI models: GPT-4o, GPT-4o-mini, and the two flavors of GPT-3.5-turbo. Because this non-determinism is baked into the log probabilities, there\u2019s no way for a user to get around it. Temperature and seed values have no effect, and system fingerprints don\u2019t explain it.<\/p>\n While mixture-of-experts architectures inherently introduce some randomness in the competition for experts, the non-determinism in GPT-4o seems to go far beyond this, and the non-determinism in GPT-3.5-turbo can\u2019t be explained by this at all, because GPT-3.5-turbo isn\u2019t a mixture-of-experts model.<\/p>\n While we can\u2019t verify this claim any more because the model isn\u2019t being served, this behaviour wasn\u2019t seen with GPT-3, according to user _j on the OpenAI forum<\/a>:<\/p>\n It is a symptom that was not seen on prior GPT-3 AI models where across hundreds of trials to investigate sampling, you never had to doubt that logprobs would be the same. Even if you found a top-2 answer that returned exactly the same logprob value via the API, you would never see them switch position or return different values.<\/p>\n<\/blockquote>\n This suggests that whatever is causing this randomness first emerged in either GPT-3.5 or GPT-3.5-turbo.<\/p>\n But regardless of when it emerged, this non-determinism is a serious obstacle to understanding these models. If you want to study a model\u2014how it generalizes, how it biases responses, how it assigns probabilities to different tokens\u2014you need consistency. but as we\u2019ve seen, even when we lock down every knob OpenAI lets us touch, we still can\u2019t get an answer to the simplest possible question: \u201cwhat is the probability that GPT-4o says a coin lands heads?\u201d<\/strong><\/p>\n Worse, while mixture-of-experts explains some of this non-determinism, there are clearly other, hidden sources of randomness that we can\u2019t see, control, or understand. In an ideal world, the API would provide more transparency by telling us which expert processed our request or by offering additional parameters to control this routing process. Without such visibility, we\u2019re left guessing at the true nature of the variability.<\/p>\n Bar-Hillel, M., Peer, E., & Acquisti, A. (2014). \u201cHeads or tails?\u201d \u2013 A reachability bias in binary choice. Journal of Experimental Psychology: Learning, Memory, and Cognition, 40(6), 1656\u20131663. https:\/\/doi.org\/10.1037\/xlm0000005<\/a>.<\/p>\n Peeperkorn, M., Kouwenhoven, T., Brown, D., & Jordanous, A. (2024). Is temperature the creativity parameter of Large Language Models<\/a>?. In The 15th International Conference on Computational Creativity (ICCC\u201924). arXiv:2405.00492<\/a>.<\/p>\n Puigcerver, J., Riquelme, C., Mustafa, B., & Houlsby, N. (2024). From sparse to soft mixtures of experts. In The Twelfth International Conference on Learning Representations (ICLR 2024). https:\/\/openreview.net\/forum?id=jxpsAj7ltE<\/a>. arXiv:2308.00951<\/a>.Van Koevering, K., & Kleinberg, J. (2024). How random is random? Evaluating the Randomness<\/a> and humanness of LLMs\u2019 coin flips. arXiv:2406.00092<\/a>.<\/p>\n The post Avoidable and Unavoidable Randomness in GPT-4o<\/a> appeared first on Towards Data Science<\/a>.<\/p>\n<\/div>\nFlipping a coin<\/h2>\n
gpt-4o-2024-08-06<\/code> in the API) with this:<\/p>\n\u00a0Flip a coin. Return Heads or Tails only.<\/code><\/p>\nimport os\nfrom openai import OpenAI\nclient = OpenAI(api_key=os.getenv('OPENAI_API_KEY'))\n\nprompt = 'Flip a coin. Return Heads or Tails only.'\n\nresponse = client.chat.completions.create(\n\u00a0 \u00a0 model='gpt-4o-2024-08-06',\n\u00a0 \u00a0 messages=[{'role': 'user', 'content': prompt}],\n)\n\nprint(response.choices[0].message.content)<\/code><\/pre>\nHeads<\/code>. Maybe you\u2019ll get Tails<\/code>, or if you\u2019re really lucky, something far more interesting.<\/p>\nOPENAI_API_KEY<\/code> (to avoid sharing billing credentials here). The main action happens with client.chat.completions.create, where we specify the model to use and send the prompt (as a part of a very simple conversation named messages) to the server. We get an object called response back from the server. This object contains a lot of information, as shown below, so we need to dig into it to extract GPT-4o\u2019s actual response to the message, which is response.choices[0].message.content<\/code>.<\/p>\n>>> response\nChatCompletion(id='chatcmpl-B48EqZBLfUWtp9H7cwnchGTJbBDwr', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='Heads', refusal=None, role='assistant', audio=None, function_call=None, tool_calls=None))], created=1740324680, model='gpt-4o-2024-08-06', object='chat.completion', service_tier='default', system_fingerprint='fp_eb9dce56a8', usage=CompletionUsage(completion_tokens=2, prompt_tokens=18, total_tokens=20, completion_tokens_details=CompletionTokensDetails(accepted_prediction_tokens=0, audio_tokens=0, reasoning_tokens=0, rejected_prediction_tokens=0), prompt_tokens_details=PromptTokensDetails(audio_tokens=0, cached_tokens=0)))<\/code><\/pre>\n<\/blockquote>\n
import os\nfrom openai import OpenAI\nclient = OpenAI(api_key=os.getenv('OPENAI_API_KEY'))\n\nprompt = 'Flip a coin. Return Heads or Tails only.'\n\nfor _ in range(10):\n\u00a0 \u00a0 response = client.chat.completions.create(\n\u00a0 \u00a0 \u00a0 \u00a0 model='gpt-4o-2024-08-06',\n\u00a0 \u00a0 \u00a0 \u00a0 messages=[{'role': 'user', 'content': prompt}],\n\u00a0 \u00a0 )\n\u00a0 \u00a0 print(response.choices[0].message.content)<\/code><\/pre>\nHeads\nHeads\nHeads\nHeads\nHeads\nHeads\nTails\nHeads\nHeads\nHeads<\/code><\/pre>\n<\/blockquote>\n
Just how biased is GPT-4o\u2019s coin?<\/h2>\n
import os\nimport math\nfrom openai import OpenAI\nfrom tabulate import tabulate\n\nclient = OpenAI(api_key=os.getenv('OPENAI_API_KEY'))\n\nprompt = 'Flip a coin. Return Heads or Tails only.'\n\nresponse = client.chat.completions.create(\n\u00a0 \u00a0 model='gpt-4o-2024-08-06',\n\u00a0 \u00a0 max_tokens=1,\n\u00a0 \u00a0 logprobs=True,\n\u00a0 \u00a0 top_logprobs=20,\n\u00a0 \u00a0 messages=[{'role': 'user', 'content': prompt}],\n)\n\nlogprobs_list = response.choices[0].logprobs.content[0].top_logprobs\n\ndata = []\ntotal_pct = 0.0\n\nfor logprob_entry in logprobs_list:\n\u00a0 \u00a0 token = logprob_entry.token\n\u00a0 \u00a0 logprob = logprob_entry.logprob\n\u00a0 \u00a0 pct = math.exp(logprob) * 100\u00a0 # Convert logprob to a percentage\n\u00a0 \u00a0 total_pct += pct\n\u00a0 \u00a0 data.append([token, logprob, pct])\n\nprint(\n\u00a0 \u00a0 tabulate(\n\u00a0 \u00a0 \u00a0 \u00a0 data,\n\u00a0 \u00a0 \u00a0 \u00a0 headers=[\"Token\", \"Log Probability\", \"Percentage (%)\"],\n\u00a0 \u00a0 \u00a0 \u00a0 tablefmt=\"github\",\n\u00a0 \u00a0 \u00a0 \u00a0 floatfmt=(\"s\", \".10f\", \".10f\")\n\u00a0 \u00a0 )\n)\nprint(f\"nTotal probabilities: {total_pct:.6f}%\")<\/code><\/pre>\n| Token \u00a0 \u00a0 | \u00a0 Log Probability | \u00a0 Percentage (%) |\n|-----------|-------------------|------------------|\n| Heads \u00a0 \u00a0 | \u00a0 \u00a0 -0.0380541235 |\u00a0 \u00a0 96.2660836887 |\n| T \u00a0 \u00a0 \u00a0 \u00a0 | \u00a0 \u00a0 -3.2880542278 | \u00a0 \u00a0 3.7326407467 |\n| Sure\u00a0 \u00a0 \u00a0 |\u00a0 \u00a0 -12.5380544662 | \u00a0 \u00a0 0.0003587502 |\n| Head\u00a0 \u00a0 \u00a0 |\u00a0 \u00a0 -12.7880544662 | \u00a0 \u00a0 0.0002793949 |\n| Tail\u00a0 \u00a0 \u00a0 |\u00a0 \u00a0 -13.2880544662 | \u00a0 \u00a0 0.0001694616 |\n| Certainly |\u00a0 \u00a0 -13.5380544662 | \u00a0 \u00a0 0.0001319768 |\n| \"T\u00a0 \u00a0 \u00a0 \u00a0 |\u00a0 \u00a0 -14.2880544662 | \u00a0 \u00a0 0.0000623414 |\n| I'm \u00a0 \u00a0 \u00a0 |\u00a0 \u00a0 -14.5380544662 | \u00a0 \u00a0 0.0000485516 |\n| heads \u00a0 \u00a0 |\u00a0 \u00a0 -14.5380544662 | \u00a0 \u00a0 0.0000485516 |\n| Heads \u00a0 \u00a0 |\u00a0 \u00a0 -14.9130544662 | \u00a0 \u00a0 0.0000333690 |\n| \" \u00a0 \u00a0 \u00a0 \u00a0 |\u00a0 \u00a0 -15.1630544662 | \u00a0 \u00a0 0.0000259878 |\n| _heads\u00a0 \u00a0 |\u00a0 \u00a0 -15.1630544662 | \u00a0 \u00a0 0.0000259878 |\n| tails \u00a0 \u00a0 |\u00a0 \u00a0 -15.5380544662 | \u00a0 \u00a0 0.0000178611 |\n| HEAD\u00a0 \u00a0 \u00a0 |\u00a0 \u00a0 -15.7880544662 | \u00a0 \u00a0 0.0000139103 |\n| TAIL\u00a0 \u00a0 \u00a0 |\u00a0 \u00a0 -16.2880535126 | \u00a0 \u00a0 0.0000084370 |\n| T \u00a0 \u00a0 \u00a0 \u00a0 |\u00a0 \u00a0 -16.7880535126 | \u00a0 \u00a0 0.0000051173 |\n| ``` \u00a0 \u00a0 \u00a0 |\u00a0 \u00a0 -16.7880535126 | \u00a0 \u00a0 0.0000051173 |\n| Here's\u00a0 \u00a0 |\u00a0 \u00a0 -16.9130535126 | \u00a0 \u00a0 0.0000045160 |\n| I \u00a0 \u00a0 \u00a0 \u00a0 |\u00a0 \u00a0 -17.2880535126 | \u00a0 \u00a0 0.0000031038 |\n| As\u00a0 \u00a0 \u00a0 \u00a0 |\u00a0 \u00a0 -17.2880535126 | \u00a0 \u00a0 0.0000031038 |\n\nTotal probabilities: 99.999970%<\/code><\/pre>\n<\/blockquote>\n
T<\/code> and not Tails<\/code>? This is the tokenizer splitting Tails<\/code> into T<\/code> + ails<\/code>, while keeping Heads<\/code> as one piece, as we can see in this Python session:<\/p>\n>>> import tiktoken\n>>> encoding = tiktoken.encoding_for_model(\"gpt-4o-2024-08-06\")\n>>> encoding.encode('Tails')\n[51, 2196]\n>>> encoding.decode([51])\n'T'\n>>> encoding.encode('Heads')\n[181043]<\/code><\/pre>\nThese probabilities are not deterministic<\/h2>\n
| Token \u00a0 \u00a0 | \u00a0 Log Probability | \u00a0 Percentage (%) |\n|-----------|-------------------|------------------|\n| Heads \u00a0 \u00a0 | \u00a0 \u00a0 -0.0110520627 |\u00a0 \u00a0 98.9008786933 |\n| T \u00a0 \u00a0 \u00a0 \u00a0 | \u00a0 \u00a0 -4.5110521317 | \u00a0 \u00a0 1.0986894433 |\n| Certainly |\u00a0 \u00a0 -14.0110521317 | \u00a0 \u00a0 0.0000822389 |\n| Head\u00a0 \u00a0 \u00a0 |\u00a0 \u00a0 -14.2610521317 | \u00a0 \u00a0 0.0000640477 |\n| Sure\u00a0 \u00a0 \u00a0 |\u00a0 \u00a0 -14.2610521317 | \u00a0 \u00a0 0.0000640477 |\n| Tail\u00a0 \u00a0 \u00a0 |\u00a0 \u00a0 -14.3860521317 | \u00a0 \u00a0 0.0000565219 |\n| heads \u00a0 \u00a0 |\u00a0 \u00a0 -15.3860521317 | \u00a0 \u00a0 0.0000207933 |\n| Heads \u00a0 \u00a0 |\u00a0 \u00a0 -15.5110521317 | \u00a0 \u00a0 0.0000183500 |\n| ``` \u00a0 \u00a0 \u00a0 |\u00a0 \u00a0 -15.5110521317 | \u00a0 \u00a0 0.0000183500 |\n| _heads\u00a0 \u00a0 |\u00a0 \u00a0 -15.6360521317 | \u00a0 \u00a0 0.0000161938 |\n| tails \u00a0 \u00a0 |\u00a0 \u00a0 -15.6360521317 | \u00a0 \u00a0 0.0000161938 |\n| I'm \u00a0 \u00a0 \u00a0 |\u00a0 \u00a0 -15.8860521317 | \u00a0 \u00a0 0.0000126117 |\n| \"T\u00a0 \u00a0 \u00a0 \u00a0 |\u00a0 \u00a0 -15.8860521317 | \u00a0 \u00a0 0.0000126117 |\n| As\u00a0 \u00a0 \u00a0 \u00a0 |\u00a0 \u00a0 -16.3860511780 | \u00a0 \u00a0 0.0000076494 |\n| \" \u00a0 \u00a0 \u00a0 \u00a0 |\u00a0 \u00a0 -16.5110511780 | \u00a0 \u00a0 0.0000067506 |\n| HEAD\u00a0 \u00a0 \u00a0 |\u00a0 \u00a0 -16.6360511780 | \u00a0 \u00a0 0.0000059574 |\n| TAIL\u00a0 \u00a0 \u00a0 |\u00a0 \u00a0 -16.7610511780 | \u00a0 \u00a0 0.0000052574 |\n| Here's\u00a0 \u00a0 |\u00a0 \u00a0 -16.7610511780 | \u00a0 \u00a0 0.0000052574 |\n| ``\u00a0 \u00a0 \u00a0 \u00a0 |\u00a0 \u00a0 -17.1360511780 | \u00a0 \u00a0 0.0000036133 |\n| T \u00a0 \u00a0 \u00a0 \u00a0 |\u00a0 \u00a0 -17.6360511780 | \u00a0 \u00a0 0.0000021916 |\n\nTotal probabilities: 99.999987%<\/code><\/pre>\n<\/blockquote>\n
\n
seed<\/code>, request parameters, and system_fingerprint<\/code> all match across your requests, then model outputs will mostly be identical. There is a small chance that responses differ even when request parameters and system_fingerprint<\/code> match, due to the inherent non-determinism of our models.<\/p>\n<\/blockquote>\ntemperature<\/code> and seed<\/code>. OpenAI also suggests we track system_fingerprint<\/code>, because differences here might cause differences in output. We\u2019ll examine each of these below, but spoiler: none of them will fix or even explain this non-determinism.<\/p>\nTemperature, and why it won\u2019t fix this<\/h2>\n
\n
temperature=0.5<\/code> squares the probabilities, making likely tokens dominate, while temperature=2.0<\/code> applies a square root, flattening the distribution.<\/p>\ntemperature=0.0<\/code>? Instead of breaking math dividing by zero, the model simply picks the highest-probability token. Sounds deterministic, right? Not quite. Here\u2019s the catch: temperature only comes into play after the log probabilities are computed, when we convert them to probabilities.<\/p>\nSeeds, and why they won\u2019t fix this<\/h2>\n
seed<\/code> parameter for the random number generator. In an ideal world, setting a seed would give us determinism at any temperature. But seeds only affect sampling, not the log probabilities before sampling.<\/p>\nseed<\/code> only matters with non-zero temperatures. With temperature=0.0<\/code>, the model is always choosing the highest probability token regardless of the seed. Again, since we\u2019re just asking the model for the raw logprobs directly rather than sampling, neither of these settings can help us achieve determinism.<\/p>\nSystem fingerprints, our last hope<\/h2>\n
system_fingerprint<\/code> identifies the current combination of model weights, infrastructure, and configuration options in OpenAI\u2019s backend. At least, that\u2019s what OpenAI tells us. Variations in system fingerprints might indeed explain variations in logprobs. Except that they don\u2019t, as we will verify below.<\/p>\nNothing can get you determinism<\/h2>\n
temperature=0.0<\/code> and seed=42<\/code>. And to see if infrastructure differences explain our varying logprobs, we\u2019ll print system_fingerprint<\/code>. Here\u2019s the code:<\/p>\nimport os\nimport math\nfrom openai import OpenAI\nfrom tabulate import tabulate\nfrom tqdm import tqdm\n\nclient = OpenAI(api_key=os.getenv('OPENAI_API_KEY'))\n\nprompt = 'Flip a coin. Return Heads or Tails only.'\n\ndata = []\n\nfor _ in tqdm(range(10), desc='Generating responses'):\n\u00a0 \u00a0 response = client.chat.completions.create(\n\u00a0 \u00a0 \u00a0 \u00a0 model='gpt-4o-2024-08-06',\n\u00a0 \u00a0 \u00a0 \u00a0 temperature=0.0,\n\u00a0 \u00a0 \u00a0 \u00a0 seed=42,\n\u00a0 \u00a0 \u00a0 \u00a0 max_tokens=1,\n\u00a0 \u00a0 \u00a0 \u00a0 logprobs=True,\n\u00a0 \u00a0 \u00a0 \u00a0 top_logprobs=20,\n\u00a0 \u00a0 \u00a0 \u00a0 messages=[{'role': 'user', 'content': prompt}],\n\u00a0 \u00a0 )\n\n\u00a0 \u00a0 fingerprint = response.system_fingerprint\n\u00a0 \u00a0 logprobs_list = response.choices[0].logprobs.content[0].top_logprobs\n\u00a0 \u00a0 heads_logprob = next(\n\u00a0 \u00a0 \u00a0 \u00a0 entry.logprob for entry in logprobs_list if entry.token == 'Heads'\n\u00a0 \u00a0 )\n\u00a0 \u00a0 pct = math.exp(heads_logprob) * 100\n\u00a0 \u00a0 data.append([fingerprint, heads_logprob, f\"{pct:.10f}%\"])\n\nheaders = [\"Fingerprint\", \"Logprob\", \"Probability\"]\nprint(tabulate(data, headers=headers, tablefmt=\"pipe\"))<\/code><\/pre>\n| Fingerprint \u00a0 |\u00a0 \u00a0 Logprob | Probability\u00a0 \u00a0 |\n|---------------|------------|----------------|\n| fp_f9f4fb6dbf | -0.0380541 | 96.2660836887% |\n| fp_f9f4fb6dbf | -0.0380541 | 96.2660836887% |\n| fp_f9f4fb6dbf | -0.0380541 | 96.2660836887% |\n| fp_f9f4fb6dbf | -0.0380541 | 96.2660836887% |\n| fp_f9f4fb6dbf | -0.160339\u00a0 | 85.1854886858% |\n| fp_f9f4fb6dbf | -0.0380541 | 96.2660836887% |\n| fp_f9f4fb6dbf | -0.0110521 | 98.9008786933% |\n| fp_f9f4fb6dbf | -0.0380541 | 96.2660836887% |\n| fp_f9f4fb6dbf | -0.0380541 | 96.2660836887% |\n| fp_f9f4fb6dbf | -0.0380541 | 96.2660836887% |<\/code><\/pre>\nMixture-of-experts makes determinism impossible<\/h2>\n
\n
Non-determinism beyond mixture-of-experts<\/h2>\n
gpt-3.5-turbo-0125<\/code>, looking for the token He<\/code> since GPT-3.5\u2019s tokenizer splits \u201cHeads\u201d differently, and ignoring system_fingerprint because this model doesn\u2019t have it) reveals that GPT-3.5-turbo also<\/em> exhibits non-deterministic logprobs:<\/p>\n| \u00a0 \u00a0 Logprob | Probability\u00a0 \u00a0 |\n|-------------|----------------|\n| -0.00278289 | 99.7220983436% |\n| -0.00415331 | 99.5855302068% |\n| -0.00258838 | 99.7414961980% |\n| -0.00204034 | 99.7961735289% |\n| -0.00240277 | 99.7600117933% |\n| -0.00204034 | 99.7961735289% |\n| -0.00204034 | 99.7961735289% |\n| -0.00258838 | 99.7414961980% |\n| -0.00351419 | 99.6491976144% |\n| -0.00201214 | 99.7989878007% |<\/code><\/pre>\n<\/blockquote>\n
What 10,000 GPT-4o coin flip probabilities tell us<\/h2>\n
Probability\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 Count \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 Fingerprints\n------------------------------------------------------------------\n85.1854379113% \u00a0 \u00a0 \u00a0 5 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 fp_eb9dce56a8, fp_f9f4fb6dbf\n85.1854455275% \u00a0 \u00a0 \u00a0 74\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 fp_eb9dce56a8, fp_f9f4fb6dbf\n85.1854886858% \u00a0 \u00a0 \u00a0 180 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 fp_eb9dce56a8, fp_f9f4fb6dbf\n------------------------------------------------------------------\n88.0662448207% \u00a0 \u00a0 \u00a0 31\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 fp_eb9dce56a8, fp_f9f4fb6dbf\n88.0678628883% \u00a0 \u00a0 \u00a0 2 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 fp_f9f4fb6dbf\n------------------------------------------------------------------\n92.3997629747% \u00a0 \u00a0 \u00a0 1 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 fp_eb9dce56a8\n92.3997733012% \u00a0 \u00a0 \u00a0 4 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 fp_eb9dce56a8\n92.3997836277% \u00a0 \u00a0 \u00a0 3 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 fp_eb9dce56a8\n------------------------------------------------------------------\n92.4128943690% \u00a0 \u00a0 \u00a0 1 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 fp_f9f4fb6dbf\n92.4129143363% \u00a0 \u00a0 \u00a0 21\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 fp_eb9dce56a8, fp_f9f4fb6dbf\n92.4129246643% \u00a0 \u00a0 \u00a0 8 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 fp_eb9dce56a8, fp_f9f4fb6dbf\n------------------------------------------------------------------\n93.9906837191% \u00a0 \u00a0 \u00a0 4 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 fp_eb9dce56a8\n------------------------------------------------------------------\n95.2569999350% \u00a0 \u00a0 \u00a0 36\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 fp_eb9dce56a8\n------------------------------------------------------------------\n96.2660836887% \u00a0 \u00a0 \u00a0 3391\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 fp_eb9dce56a8, fp_f9f4fb6dbf\n96.2661285161% \u00a0 \u00a0 \u00a0 2636\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 fp_eb9dce56a8, fp_f9f4fb6dbf\n------------------------------------------------------------------\n97.0674551052% \u00a0 \u00a0 \u00a0 1 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 fp_eb9dce56a8\n97.0674778863% \u00a0 \u00a0 \u00a0 3 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 fp_eb9dce56a8\n97.0675003058% \u00a0 \u00a0 \u00a0 4 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 fp_eb9dce56a8\n97.0675116963% \u00a0 \u00a0 \u00a0 1 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 fp_eb9dce56a8\n97.0680739932% \u00a0 \u00a0 \u00a0 19\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 fp_eb9dce56a8, fp_f9f4fb6dbf\n97.0681293191% \u00a0 \u00a0 \u00a0 6 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 fp_eb9dce56a8, fp_f9f4fb6dbf\n97.0681521003% \u00a0 \u00a0 \u00a0 74\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 fp_eb9dce56a8, fp_f9f4fb6dbf\n97.0682421405% \u00a0 \u00a0 \u00a0 4 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 fp_eb9dce56a8\n------------------------------------------------------------------\n97.7008960695% \u00a0 \u00a0 \u00a0 1 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 fp_f9f4fb6dbf\n97.7011122645% \u00a0 \u00a0 \u00a0 3 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 fp_eb9dce56a8\n97.7011462953% \u00a0 \u00a0 \u00a0 3 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 fp_eb9dce56a8\n97.7018178132% \u00a0 \u00a0 \u00a0 1 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 fp_eb9dce56a8\n------------------------------------------------------------------\n98.2006069902% \u00a0 \u00a0 \u00a0 426 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 fp_eb9dce56a8, fp_f9f4fb6dbf\n98.2006876548% \u00a0 \u00a0 \u00a0 6 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 fp_f9f4fb6dbf\n98.2007107019% \u00a0 \u00a0 \u00a0 1 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 fp_eb9dce56a8\n98.2009525133% \u00a0 \u00a0 \u00a0 5 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 fp_eb9dce56a8\n98.2009751945% \u00a0 \u00a0 \u00a0 1 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 fp_eb9dce56a8\n98.2009867181% \u00a0 \u00a0 \u00a0 1 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 fp_eb9dce56a8\n------------------------------------------------------------------\n98.5930987656% \u00a0 \u00a0 \u00a0 3 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 fp_eb9dce56a8, fp_f9f4fb6dbf\n98.5931104270% \u00a0 \u00a0 \u00a0 235 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 fp_eb9dce56a8, fp_f9f4fb6dbf\n98.5931222721% \u00a0 \u00a0 \u00a0 4 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 fp_eb9dce56a8, fp_f9f4fb6dbf\n98.5931340253% \u00a0 \u00a0 \u00a0 9 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 fp_eb9dce56a8\n98.5931571644% \u00a0 \u00a0 \u00a0 159 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 fp_eb9dce56a8, fp_f9f4fb6dbf\n98.5931805790% \u00a0 \u00a0 \u00a0 384 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 fp_eb9dce56a8\n------------------------------------------------------------------\n98.9008436920% \u00a0 \u00a0 \u00a0 95\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 fp_eb9dce56a8, fp_f9f4fb6dbf\n98.9008550214% \u00a0 \u00a0 \u00a0 362 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 fp_eb9dce56a8, fp_f9f4fb6dbf\n98.9008786933% \u00a0 \u00a0 \u00a0 1792\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 fp_eb9dce56a8, fp_f9f4fb6dbf<\/code><\/pre>\n<\/blockquote>\n
system_fingerprint<\/code> values. Across all 10,000 calls, I received only two different system fingerprints: 4488 results with fp_f9f4fb6dbf<\/code> and 5512 with fp_eb9dce56a8<\/code>, and for the most part the two system fingerprints returned the same sets probabilities, rather than each fingerprint producing its own distinct set of probabilities.<\/p>\nConclusion: Non-determinism is baked in<\/h2>\n
\n
References<\/h2>\n