“An intriguing open question is whether the LLM is actually using its internal model of reality to reason about that reality as it solves the robot navigation problem,” says Rinard. “While our results are consistent with the LLM using the model in this way, our experiments are not designed to answer this next question.”
The paper, “Emergent Representations of Program Semantics in Language Models Trained on Programs” can be found here.
Abstract
We present evidence that language models (LMs) of code can learn to represent the formal semantics of programs, despite being trained only to perform next-token prediction. Specifically, we train a Transformer model on a synthetic corpus of programs written in a domain-specific language for navigating 2D grid world environments. Each program in the corpus is preceded by a (partial) specification in the form of several input-output grid world states. Despite providing no further inductive biases, we find that a probing classifier is able to extract increasingly accurate representations of the unobserved, intermediate grid world states from the LM hidden states over the course of training, suggesting the LM acquires an emergent ability to interpret programs in the formal sense. We also develop a novel interventional baseline that enables us to disambiguate what is represented by the LM as opposed to learned by the probe. We anticipate that this technique may be generally applicable to a broad range of semantic probing experiments. In summary, this paper does not propose any new techniques for training LMs of code, but develops an experimental framework for and provides insights into the acquisition and representation of formal semantics in statistical models of code.
i feel that many of us, when confronted to this, are doing like Z.B. (president of the Galaxy) in The hitchhiker guide… when he says :
… “whenever I stop and think why did I want to do something? – how did I work out how to do it? – I get a very strong desire to just to stop thinking about it” …
We don’t want to be surpassed by machines … and this explains the large amount of downvotes
I’m actually pretty sure the downvotes are because LLM’s don’t think. They don’t even process. They pick the highest number and spit out the information attached to it.
Science cannot say much about what it is to think since it doesn’t understand the brain well enough … and the day we can fully explained it, we will also be able to replicated it on computers.
Science can and does quantify what our brains do vs what an LLM does though. That’s the point. That’s why the brain knows when it’s supplying wrong information or guessing but the LLM does not.
The LLM can provide wrong information. What it can’t do is intentionally lie.
i agree with you that we are much better than LLMs in the fact we can verify our errors (and we can do much more things LLMs don’t do).
Still i am happy to have access to their vast memory and i know where they fail most of times so i can work with them in a productive way.
The day we provide them (or DNNs) with “Will” is i think when they will become (more) dangerous.