You can predict with certainty what a compilers output would be. Linus is conflating the fact that most people don’t understand how a compiler converts higher level programming language into machine code with the reality that no one can actually predict and audit what an LLM puts out.
You cannot, that’s why reproducible builds are hard. Modern compiler are made with compilers (egg/chicken problem) not assembly nor machine code and you always loose some perfect prediction.
Yes LLMs are less predictible than compiler but their output is readable to human, while machine code is very hard (yet not impossible).
Is that a bad things that 99% of devs have forgot how to write code cutting holes in paper? Tools are changing and while a direct comparison between the move from paper to COBOL or assembly to C is not the same as python to LLM generated it’s hard to deny that LLM will change how human write code.
Thanks for that explanation. Are you saying that if I rebuild the exact same code on the same machine I could get different results for a given compiler? I think this would render solutions like docker useless. Could you elaborate?
Yes you can get a different result. Compilers are designed to be deterministic, but a build is more than the compiler. Usually the bytes differ but the behavior is identical.
Docker is far from being reproducible!
It works on my machine ≠ reproducible.
But reproducible will make it works on your machine.
Using a compiler does not guarantee you to get byte-perfect output each time, especially not on different machines.
Talk about misleading. The differences are timestamps and metadata, and some compilers have flags to let you zero out timestamps and force it to produce a bit-for-bit identical file every time.
That is, of course, the whole point of docker (making sure the results of compiling are identical). I totally get the different machines giving slightly different results. This is not really in the same realm of the dramatically different output a LLM will give though.
No the goal of Docker is not to make sure the results of compiling are identitcal. It is to make it work on your machine, there is some nuance to it.
This is more the goal of something like nixpkgs (NixOS).
This is not really in the same realm of the dramatically different output a LLM will give though.
Yes this is not same realm but my point was that eve if LLM would give you different results each time the result is transparent and editable. Also developers tools evolve and Linus have more technical knowledge to use this tool in a way that will gave him satisfying result than me or a junior dev with 1.5 years of experience. He was against LLM generated code a few years ago, not because he hated on AI but because this tool was giving him unsatisfiying results. If compilers were not working 50% of the time he would probably write everything in assembly. LLM models and their coding harness cannot be compared to what they were 2 years ago. And most of the devs are using them. Will that transform bad devs into good ones? Probably not. Does that mean that LLM generated or assisted code is the way to go? Not necessarly. But it means that it’s a tool that find its use in the majority of development teams and thus that they find it quite useful.
There is some issues with it and I’m not here to deny them. Whether they are social, environmental, capitalist, monopolist, learning. Like every new disruptive tool we use them in ways we won’t in the future and I think it’s similar with AI.
He is saying that you have to understand your contribution, so whatever the LLM spits out, you need to thoroughly review and double-check that it makes sense. So, in particular, he’s not trying to rate these tools, he’s merely saying make sure the contribution is objectively good, then he doesn’t care how you got there.
My issue is him saying don’t use a compiler if you don’t really understand the machine language it creates. This conflates LLM with a compiler as a tool when they are inherently different. This is disingenuous because it is possible to understand and audit what a compiler creates. You cannot audit what an LLM creates in even close to the same way.
But why can you not audit what the LLM creates? You can’t do it for the general case, i.e. prove that for a given prompt it always spits out a valid solution, but you can look at what it actually spit out for your concrete prompt and then you can freely decide, if those code changes are good.
I am of the opinion that a thorough review often takes similarly long to actually writing it yourself, and there’s a high chance that you miss architectural aspects, if you don’t investigate yourself.
But yeah, again, I think you’re taking the comparison too literal.
He is just saying that LLMs are a tool. You need to know how to use your tool and produce objectively good results. Then he does not care what tool you use to get there.
You don’t have to do everything with the LLM. Tracing the logical path very much still happens by looking at the code in your editor.
This is also why I’m saying it isn’t much faster. Understanding the existing code and what changes are necessary is the bulk of the work when handwriting code as well.
You *can* tell the LLM to go ham and implement the change on its own and then only look at the diff afterwards, but anyone who thinks they understand the code changes then, is lying to themselves IMHO.
Can you elaborate on the last sentence a bit from a technical perspective? I’m not using code myself to the extent that I could really judge between LLM-generated code being good or just-works. I can just imagine compiler output being strictly deterministic, while LLM-code, due to the way a LLM works in principle, should be more …unstructured?
I honestly didn’t think you could even audit an LLM. It is possible, but not in the same way you would audit code. It is my understanding, because of the probabilistic nature of how the LLM operates, you can get different results even within the same model. This makes tracing something like logic next to impossible. Why the LLM gave the result is somewhat of a mystery. It is in effect a black box. This may change in the future, but for now LLM suffers from this issue.
You could probably trace how it comes up with the probabilities of the next token. Though the meaning of all the weights are probably going to be lost on you. Once you’ve got your probabilities and you’ve limited to some top number, it’s just a dice throw at that point.
Predicting what a compiler outputs is quiet hard, especially if you enable optimization flags or get near any undefined behavior. But I think you’re right to say that a compiler is more predictable than any LLM.
I don’t get the point why you should be unable to audit the output of an LLM? If its used as a tool like described the user should have the knowlage to audit any output
Then I can run it again, and the output will be the same.
I can run a LLM prompt.
Then I can run it again, and the output should not be the same. The unpredictability is what makes the tech great - Nobody in decision making actually understands this, but the goal is not for identical outputs. Just like me and you, don’t respond to the same question in the same way every time
Compilers aren’t all that deterministic without effort. Example, both gc and clang if ran with parallelism can produce different results purely off file system lookup differences alone. This means intermediate archive files can be equivalent in content from the perspective of functions built but differ in order. This can easily filter up to the final binary being different. Making compilers bitwise deterministic is a lot more effort than running the same compiler on the same input. This is ignoring things like timestamps and other common sources of non determinism. But I can say the Debian reproducible build project has most of this figured out but compilers are not as deterministic as your comment implies. They can be, but you should really check that your final binary is bit for but the same. If you’ve used any parallelism it likely isn’t.
If we then bring in lto et al it gets worse.
Let’s take just gcc, are you setting -frandom-seed, SOURCE_DATE_EPOCH, -file-prefix-map, are you ensuring the input file list is deterministic? If you’ve used any can’t affirm yes to at least these 4, note these are just the things I can remember, you do not have a deterministic compile chain.
But I rarely want the output twice? At least in this domain of coding, the entire point is doing it once.
Secondly, I can definitely validate it in most ways that matter. Ironically, by compiling it for example, I make 100% certain the output was well defined code. Type checking, linting, tests, etc. all help ensure I got functionally valid code.
At that point I might not really care if the function name or a variable name is different, just like I don’t care that the compiler did some kind of heuristic optimization or minified the names or something.
In both cases, I have a target functionality, and I need to validate it was reached. The way I got there doesn’t matter (in most cases).
It’s a bit like Docker vs. Nix. Sure, docker isn’t really reproducible perfectly, but it fulfills it’s goal for most people most of the time for a fraction of the time and effort taken.
Compilers are incredibly complex things and the often have undefined behaviors that slip in because the compiler has to interact with the operating system at large, and that means the code actually doesn’t run the same on every platform, or even every version of the same operating system. So, while they are technically deterministic, the reality is that you get surprising and unexpected behaviors in practice.
Also worth nothing that the probability distributions for an LLM are fixed at training time, and that is why the weights are static. When doing inference, models are selecting from a top-k of probable tokens at each autoregressive loop for your typical LLM. When you adjust top-k to 1 or simply set the “temperature” to 0, it will always give you the same result from the same input.
You can predict with certainty what a compilers output would be. Linus is conflating the fact that most people don’t understand how a compiler converts higher level programming language into machine code with the reality that no one can actually predict and audit what an LLM puts out.
You cannot, that’s why reproducible builds are hard. Modern compiler are made with compilers (egg/chicken problem) not assembly nor machine code and you always loose some perfect prediction.
Yes LLMs are less predictible than compiler but their output is readable to human, while machine code is very hard (yet not impossible).
Is that a bad things that 99% of devs have forgot how to write code cutting holes in paper? Tools are changing and while a direct comparison between the move from paper to COBOL or assembly to C is not the same as python to LLM generated it’s hard to deny that LLM will change how human write code.
Honestly a lot of computing history goes ‘Okay now it should always do–’ followed by not that.
It took us a long while to tame the transistor. It took even longer to figure out why those solutions worked.
Thanks for that explanation. Are you saying that if I rebuild the exact same code on the same machine I could get different results for a given compiler? I think this would render solutions like docker useless. Could you elaborate?
Yes you can get a different result. Compilers are designed to be deterministic, but a build is more than the compiler. Usually the bytes differ but the behavior is identical.
Docker is far from being reproducible!
It works on my machine ≠ reproducible. But reproducible will make it works on your machine.
Using a compiler does not guarantee you to get byte-perfect output each time, especially not on different machines.
Talk about misleading. The differences are timestamps and metadata, and some compilers have flags to let you zero out timestamps and force it to produce a bit-for-bit identical file every time.
Yes, that is a great explanation. Thank you.
That is, of course, the whole point of docker (making sure the results of compiling are identical). I totally get the different machines giving slightly different results. This is not really in the same realm of the dramatically different output a LLM will give though.
No the goal of Docker is not to make sure the results of compiling are identitcal. It is to make it work on your machine, there is some nuance to it.
This is more the goal of something like nixpkgs (NixOS).
Yes this is not same realm but my point was that eve if LLM would give you different results each time the result is transparent and editable. Also developers tools evolve and Linus have more technical knowledge to use this tool in a way that will gave him satisfying result than me or a junior dev with 1.5 years of experience. He was against LLM generated code a few years ago, not because he hated on AI but because this tool was giving him unsatisfiying results. If compilers were not working 50% of the time he would probably write everything in assembly. LLM models and their coding harness cannot be compared to what they were 2 years ago. And most of the devs are using them. Will that transform bad devs into good ones? Probably not. Does that mean that LLM generated or assisted code is the way to go? Not necessarly. But it means that it’s a tool that find its use in the majority of development teams and thus that they find it quite useful.
There is some issues with it and I’m not here to deny them. Whether they are social, environmental, capitalist, monopolist, learning. Like every new disruptive tool we use them in ways we won’t in the future and I think it’s similar with AI.
He is saying that you have to understand your contribution, so whatever the LLM spits out, you need to thoroughly review and double-check that it makes sense. So, in particular, he’s not trying to rate these tools, he’s merely saying make sure the contribution is objectively good, then he doesn’t care how you got there.
I got that part and I agree with it.
My issue is him saying don’t use a compiler if you don’t really understand the machine language it creates. This conflates LLM with a compiler as a tool when they are inherently different. This is disingenuous because it is possible to understand and audit what a compiler creates. You cannot audit what an LLM creates in even close to the same way.
But why can you not audit what the LLM creates? You can’t do it for the general case, i.e. prove that for a given prompt it always spits out a valid solution, but you can look at what it actually spit out for your concrete prompt and then you can freely decide, if those code changes are good.
I am of the opinion that a thorough review often takes similarly long to actually writing it yourself, and there’s a high chance that you miss architectural aspects, if you don’t investigate yourself.
But yeah, again, I think you’re taking the comparison too literal.
He is just saying that LLMs are a tool. You need to know how to use your tool and produce objectively good results. Then he does not care what tool you use to get there.
Auditing typically means tracing the logical path. How do you suggest to do this with an LLM?
I am definitely taking his words literally which is why I disagree with just this one point.
You don’t have to do everything with the LLM. Tracing the logical path very much still happens by looking at the code in your editor.
This is also why I’m saying it isn’t much faster. Understanding the existing code and what changes are necessary is the bulk of the work when handwriting code as well.
You *can* tell the LLM to go ham and implement the change on its own and then only look at the diff afterwards, but anyone who thinks they understand the code changes then, is lying to themselves IMHO.
Can you elaborate on the last sentence a bit from a technical perspective? I’m not using code myself to the extent that I could really judge between LLM-generated code being good or just-works. I can just imagine compiler output being strictly deterministic, while LLM-code, due to the way a LLM works in principle, should be more …unstructured?
I honestly didn’t think you could even audit an LLM. It is possible, but not in the same way you would audit code. It is my understanding, because of the probabilistic nature of how the LLM operates, you can get different results even within the same model. This makes tracing something like logic next to impossible. Why the LLM gave the result is somewhat of a mystery. It is in effect a black box. This may change in the future, but for now LLM suffers from this issue.
You could probably trace how it comes up with the probabilities of the next token. Though the meaning of all the weights are probably going to be lost on you. Once you’ve got your probabilities and you’ve limited to some top number, it’s just a dice throw at that point.
Sleeping agent in LLM is a scary topic and the tech itself has mathematically no solution for this: https://arxiv.org/html/2401.05566v2
Backdoors through model poisoning is so evil, I love it.
Predicting what a compiler outputs is quiet hard, especially if you enable optimization flags or get near any undefined behavior. But I think you’re right to say that a compiler is more predictable than any LLM. I don’t get the point why you should be unable to audit the output of an LLM? If its used as a tool like described the user should have the knowlage to audit any output
I can run a compiler.
Then I can run it again, and the output will be the same.
I can run a LLM prompt.
Then I can run it again, and the output should not be the same. The unpredictability is what makes the tech great - Nobody in decision making actually understands this, but the goal is not for identical outputs. Just like me and you, don’t respond to the same question in the same way every time
Compilers aren’t all that deterministic without effort. Example, both gc and clang if ran with parallelism can produce different results purely off file system lookup differences alone. This means intermediate archive files can be equivalent in content from the perspective of functions built but differ in order. This can easily filter up to the final binary being different. Making compilers bitwise deterministic is a lot more effort than running the same compiler on the same input. This is ignoring things like timestamps and other common sources of non determinism. But I can say the Debian reproducible build project has most of this figured out but compilers are not as deterministic as your comment implies. They can be, but you should really check that your final binary is bit for but the same. If you’ve used any parallelism it likely isn’t.
If we then bring in lto et al it gets worse.
Let’s take just gcc, are you setting -frandom-seed, SOURCE_DATE_EPOCH, -file-prefix-map, are you ensuring the input file list is deterministic? If you’ve used any can’t affirm yes to at least these 4, note these are just the things I can remember, you do not have a deterministic compile chain.
But I rarely want the output twice? At least in this domain of coding, the entire point is doing it once.
Secondly, I can definitely validate it in most ways that matter. Ironically, by compiling it for example, I make 100% certain the output was well defined code. Type checking, linting, tests, etc. all help ensure I got functionally valid code.
At that point I might not really care if the function name or a variable name is different, just like I don’t care that the compiler did some kind of heuristic optimization or minified the names or something.
In both cases, I have a target functionality, and I need to validate it was reached. The way I got there doesn’t matter (in most cases).
It’s a bit like Docker vs. Nix. Sure, docker isn’t really reproducible perfectly, but it fulfills it’s goal for most people most of the time for a fraction of the time and effort taken.
At least clean it up so it’s maintainable, introducing tech debt is a long term impact
For local LLMs you can set the seed and generate the same output every time.
You can get the same output from the LLM if you put the appropriate guardrails and context in place. That’s what is happening in the enterprise space.
I was just pointing out Linus was conflating the two to make his point. I don’t agree with this, but everything else he said was fine.
100%, or we’d commit prompts instead of code lol
all the undefined behaviors would like to disagree with you there champ
Can you elaborate?
Compilers are incredibly complex things and the often have undefined behaviors that slip in because the compiler has to interact with the operating system at large, and that means the code actually doesn’t run the same on every platform, or even every version of the same operating system. So, while they are technically deterministic, the reality is that you get surprising and unexpected behaviors in practice.
Also worth nothing that the probability distributions for an LLM are fixed at training time, and that is why the weights are static. When doing inference, models are selecting from a top-k of probable tokens at each autoregressive loop for your typical LLM. When you adjust top-k to 1 or simply set the “temperature” to 0, it will always give you the same result from the same input.
Thanks