Thanks to Bruce W. Lee for valuable feedback while drafting this post!
What if natural language is not the optimal medium for LLM reasoning? It is an incredible finding that reasoning models (systems like o1, DeepSeek-R1, inter alia) have learned to produce appropriate chains-of-thought (CoT) in natural language to solve specific problems, much like the in-context learning ability of GPT-3 drove a substantial leap in capabilities. We have quickly accepted that machines can learn to iterate in human-legible terms to improve on various tasks, but the extent to which the resulting CoTs reflect LLM cognition (and relate to human reasoning processes) is yet unclear. There are several interesting research directions that focus on learning effective CoTs, with important implications on model performance and cost for increasingly harder and long-horizon tasks, as well as the need to advance our understanding of model reasoning, impacting oversight and interpretability.
The latent variable view of CoT [1,2] frames reasoning as learning a useful intermediate z that maps a context $x$ to a response $y$. This $z$ can be expressed in many forms, which change the degree to which the CoT is machine- and human-interpretable. The continuous embeddings perspective [3] suggests that $z$ may not need to be verbalized at all, while externalized (verbal) CoT produces a rationale in natural language. However, both come with tradeoffs: performance in the case of the former, and inference-time cost with the latter; verbal CoT can also be unfaithful while constraining reasoning to the space of human-legible sequences. This offers the potential to design new mechanisms along this spectrum.
We don’t believe that natural language should be done away with – it is fundamentally vital as the interface for human-AI communication and collaboration [5]. But the assumption that reasoning must happen in natural language is worth examining – and is a key driver in our exploration into other reasoning mechanisms.
Our work was motivated by the following big-picture questions: are natural language symbol embeddings, inherited from pre-training, actually necessary for effective reasoning? Can we get away with shorter, cheaper CoTs if we relax that constraint?
Our approach -- Abstract Chain-of-Thought (Abstract-CoT / ACoT) – replaces verbalized CoTs with non-pre-trained abstract symbols (e.g. <TOKEN_X>) in the form of an auxiliary vocabulary, without explicit compression. Our results show that, indeed, models can learn to use these ACoT traces with little or no performance degradation – and in some cases, outperforming verbal CoT -- while using substantially shorter sequences. This makes ACoT especially compelling in cost-sensitive regimes where inference scaling is employed, such as majority voting over many sampled outputs.
In developing Abstract-CoT, we studied how new tokens can be introduced into an existing model’s vocabulary post hoc, training the model to use them in reasoning. Given prior work on Pause Tokens [4] showed that pause-finetuning without pause-pretraining was significantly less performant, we sought to devise strategies to address this in our methodology. We find that appropriate initialization of the abstract symbol representations through a warm-up mechanism is vital to ACoT learning.
Given the community-wide interest around latent reasoning, we believe that releasing this work is a net positive from a scientific research perspective, providing a mechanism that sits at a new point in the latent CoT spectrum.
Abstract-CoT also makes the tradeoff of legibility versus cost explicit – one that has already been implicit for a while. Current verbal CoTs are not reliably faithful, so they largely offer surface-level legibility rather than true interpretability. The tradeoff this induces allows practitioners to use Abstract-CoT in cost-sensitive settings, while using verbal CoT where they specifically require human-readability. Advancing in CoT faithfulness as a community would make this tradeoff more difficult, which we believe is the right outcome. Even then, users may have certain settings where cheap latent / abstract reasoning could continue to be preferred, but the incentives for each option are much clearer for human decision-making.
We do not yet have a mechanism for monitoring Abstract-CoT in human-legible concept space: while one can analyze the token embeddings statically, it remains challenging to develop a contextual understanding of model behavior, which is necessary for proper safety assessment. The partial monitorability framing from [6] is a useful starting point, as ACoT sits in a similar landscape as other latent reasoning approaches, though the discretization into abstract tokens does offer a different intermediate interface for analysis than fully continuous methods. As we continue to explore the various “instantiations of $z$” as described earlier, our monitorability toolkits should evolve accordingly and be adaptable to new mechanisms that may emerge along this spectrum.
A note on neuralese: several people pointed out some parallels between ACoT and discussions of neuralese (e.g. AI 2027). We weren't aware of this connection while developing the work, and we appreciate the community for surfacing it, as it helps us to sharpen how we think about questions around deployment and potential impacts discussed herein. We think that marginal risk analyses for potential misuse should inform how systems like this are deployed, with careful consideration and assessment, while still recognizing ACoT’s practical value.
Again, we really appreciate the interest and discussion that Abstract-CoT has stirred, and hope this post gives some context around big-picture perspectives and directions our work can be taken so we can responsibly deploy these systems. If you are interested in any of the problems here, or something adjacent that you're looking to explore, please reach out – we're happy to discuss further!
If you found this blog helpful, please check out our paper and cite our work!
Thinking Without Words: Efficient Latent Reasoning with Abstract Chain-of-Thought