Compilers are incredibly complex things and the often have undefined behaviors that slip in because the compiler has to interact with the operating system at large, and that means the code actually doesn’t run the same on every platform, or even every version of the same operating system. So, while they are technically deterministic, the reality is that you get surprising and unexpected behaviors in practice.
Also worth nothing that the probability distributions for an LLM are fixed at training time, and that is why the weights are static. When doing inference, models are selecting from a top-k of probable tokens at each autoregressive loop for your typical LLM. When you adjust top-k to 1 or simply set the “temperature” to 0, it will always give you the same result from the same input.
all the undefined behaviors would like to disagree with you there champ
Can you elaborate?
Compilers are incredibly complex things and the often have undefined behaviors that slip in because the compiler has to interact with the operating system at large, and that means the code actually doesn’t run the same on every platform, or even every version of the same operating system. So, while they are technically deterministic, the reality is that you get surprising and unexpected behaviors in practice.
Also worth nothing that the probability distributions for an LLM are fixed at training time, and that is why the weights are static. When doing inference, models are selecting from a top-k of probable tokens at each autoregressive loop for your typical LLM. When you adjust top-k to 1 or simply set the “temperature” to 0, it will always give you the same result from the same input.
Thanks