I Don’t Think LLMs Are the Final Architecture of Superintelligence

Modern LLMs are increasingly multimodal, so calling every LLM purely "text-based" is no longer entirely accurate. Images, audio, video, and other inputs can also be represented and processed by these systems. However, much of the current frontier still revolves around an autoregressive language-model core. In a number of cases, images and audio are converted into model embeddings, extended reasoning is often carried out by generating additional tokens, and actions are commonly expressed as discrete tool calls or action tokens.

I think AI models based on this architecture will continue to advance and become much more capable than they are today. I don't, however, think simply scaling this general paradigm eventually gets us to superintelligence, or at least not in the most efficient or ideal way. What "superintelligence" really means is up for debate, but I'm broadly referring to maximally capable AI. I expect increasingly capable systems to become less centered around making every kind of information and task compatible with a general-purpose language model, and to instead make greater use of modality-native representations and computation designed around the properties of the task itself.

One reason I believe this is that we are increasingly using language models to perform tasks that are not fundamentally linguistic. Computer control/use is a major example. OpenAI's current Codex computer-use system can visually inspect a computer screen, click, type, scroll, and interact with ordinary applications. The model can directly process screenshots, so it is not literally translating the screen into text first, but the overall interaction from OpenAI's computer-use systems, from my understanding, have been a largely serialized loop. Observe the current state, reason about it, generate a discrete action, execute that action, observe the new state, and repeat.

This system is becoming impressively capable, but it also feels like we are stretching an architecture built around sequential token generation into domains that are inherently visual, spatial, interactive, and often parallel. A person can look at a screen, continuously perceive changes across it, move a cursor, type, and maintain an understanding of the broader environment at the same time. An autoregressive agent instead tends to reduce this into a sequence of observations and actions that must be generated and processed one after another. The same broader pattern appears in tool use, robotics, and even reasoning itself. Because language models are currently our most capable general-purpose models, we increasingly encode non-linguistic problems into forms that they can consume and emit. I think there is an important distinction between an architecture being flexible enough to control almost anything and that architecture always being the most efficient or natural substrate for intelligence.

We have already seen a smaller-scale example of this problem with voice. Before GPT-4o, ChatGPT's Voice Mode used three separate models, with speech transcribed into text, that text processed by GPT-3.5 or GPT-4, and the resulting text converted back into speech. OpenAI specifically noted that this meant the main model could not directly perceive things like tone, multiple speakers, or background noise. Naturally, substantial latency was also a side effect, which is a very annoying problem. GPT-4o moved toward a single model trained end-to-end across text, vision, and audio, allowing audio to be processed and generated without requiring that explicit text intermediary.

I think this is an important precedent. The old system worked because speech can be approximated reasonably well as text, but speech contains information that text fundamentally does not. Converting everything into the representation most convenient for a language model meant throwing some of, or even a great deal of, that information away and adding additional processing steps. Once models could process the original/native modality directly, like audio, it was no longer necessary to force that information through text first. I believe it is likely that something similar applies much more broadly. Text and discrete tokens are extremely useful representations, but there is little reason to assume that they are the ideal representation for every kind of perception, action, thought, or task.

This is also why I think the current use of the word "multimodal" can hide an important distinction. A system can accept images, audio, video, or actions without its computation being fundamentally designed around the structure of those modalities. We have become very good at making different modalities compatible with general-purpose language-model architectures by mapping them into embeddings, tokens, discrete actions, or other representations the model can process. That is not at all the same as building modality-native systems whose representations and computation are designed around the properties of vision, audio, physical control, memory, or other forms of information themselves. I think we still have comparatively few systems that are truly modality-native in this sense.

Robotics exemplify this quite well. Google's RT-2 was built by adapting vision-language models for robotic control. To make robot actions compatible with the existing language-model architecture, the researchers represented those actions as text tokens and trained them alongside ordinary vision-language data. As expected, it worked, and allowed knowledge learned from vision and language on the web to transfer into physical robotic behavior. I still think something is inherently wrong with the design, though, and the paper's own numbers show where. Each continuous action dimension is uniformly discretized into 256 bins, introducing an additional precision limit determined by that discretization. Those action tokens are then generated autoregressively, so the control frequency becomes constrained by model inference latency rather than being chosen purely around the dynamics of the task. The 55B RT-2-PaLI-X model ran its control loop at only 1 to 3 Hz, while the 5B version reached around 5 Hz. Later work moved away from simple per-dimension binning specifically because it breaks down on higher-frequency, more precise and dexterous control. Physical Intelligence notes that dexterous systems may require 20 to 50 actions per action chunk, while its continuous-action π₀ architecture was designed for motor commands at frequencies of up to 50 Hz. A robot arm moving through three-dimensional space is not a linguistic process, and encoding it as one imposes limits. We represent the movement as tokens because token-generating models are currently an extremely powerful thing we already know how to build, scale, and advance.

The same representation question applies to reasoning itself. Reasoning models today can improve their answers by spending more test-time compute generating intermediate reasoning tokens before producing a final answer. This works, but it means much of that additional computation is implemented by extending an autoregressive token sequence. An important clarification is that LLMs already perform their internal computation in continuous, high-dimensional representations. The question is whether additional iterative reasoning needs to repeatedly be decoded into discrete language tokens at all. There is growing evidence that, on some tasks, performing more of this additional computation directly through continuous representations can improve both efficiency and reasoning performance.

Meta researchers introduced Coconut, or Chain of Continuous Thought, which feeds the model's last hidden state directly back as the representation of the next reasoning step instead of first decoding it into a word token. The results are particularly interesting on problems involving planning and search. On ProsQA, Coconut reached 97.0% accuracy compared with 77.5% for ordinary chain-of-thought, while using 14.2 continuous reasoning states on average instead of 49.4 reasoning tokens. The researchers argue that a continuous state can represent multiple possible next steps simultaneously instead of immediately committing to one discrete token and reasoning path. Importantly, this is not universally better. On GSM8K, Coconut scored 34.1% compared with 42.9% for ordinary chain-of-thought. The benefit therefore appears to depend heavily on the kind of reasoning being performed.

Other research suggests this is not unique to Coconut. CODI matched explicit chain-of-thought performance on GSM8K while compressing the reasoning process by 3.1×, while MARCOS reported outperforming token-based chain-of-thought by 4.7% on GSM8K with up to a 15.7× inference speedup. A separate recurrent-depth approach showed that reasoning performance could continue improving as additional test-time computation was applied by repeatedly processing internal representations rather than generating additional reasoning tokens. These are still early results, but they make the point substantially more interesting to me. Language-form chain-of-thought is not simply a window into reasoning. The representation and architecture through which additional reasoning occurs can affect the reasoning itself.

Continuous reasoning is not universally better, and it also makes intermediate reasoning less directly observable. Still, different kinds of reasoning may benefit from different internal representations, and natural-language tokens are unlikely to be optimal for all of them.

My expectation is therefore that LLMs will become one component of a broader architecture, and that more modality-native and optimized representations will be developed. Future systems may still use transformers, autoregression, and language extensively, while processing things like audio directly rather than through text, controlling physical motion through continuous action representations rather than language-like tokens, and carrying out more extended reasoning directly in continuous internal states rather than generated chain-of-thought. In essence, I believe language should remain how superintelligence talks to us, without being how it primarily reasons, because natural-language tokens are not always the most efficient representation for reasoning.