LSTM: The Comeback Story? [Prof. Sepp Hochreiter]
Summary
Hochreiter’s central call is that scaling language models makes them larger, not smarter. He sees an LLM as “a database technology” that stores human knowledge in text, perhaps also code, and recombines it, potentially ranging far through existing program space but failing, in his view, when progress requires a genuinely new concept. “Our path ends with scaling up.”
xLSTM’s comeback rests on three architectural changes: exponential gating, a matrix memory based on a Hopfield network, and parallelization. Exponential gates let the model revise an earlier storage decision, while normalization is needed because earlier exponential activation would otherwise break learning; the larger memory stores richer structures rather than only a scalar.
Hochreiter claims xLSTM is now faster than FlashAttention in both training and inference. Its chunkwise design alternates optimized attention with recurrence, sizing chunks for GPU caches; if xLSTM were 100 times faster at inference, an o1/Strawberry-style system could spend the same budget on “100 times more thinking.”
The more differentiated commercial opening may be embedded industrial AI, not another language product. xLSTM uses a fixed memory whether a sequence contains 100 or 100 million elements, which Hochreiter argues makes memory requirements predictable and supports fast, energy-efficient systems for robots, drones and self-driving systems. One unnamed drone company reportedly found the results “unbelievable,” and its drones are now flying autonomously, though no public benchmark was disclosed.
Hochreiter does not see a necessity to put every reasoning capability into one learned model. His preferred route is a hybrid that calls formal provers, mathematics software and knowledge tools: “For me it’s stupid to push everything in one system.” Austria’s roughly €40 million “Bilateral AI” project is intended to bring symbolic and subsymbolic AI together for robust industrial systems.
NXAI is being built around xLSTM and neural simulation rather than the crowded general-language market. The first €10 million reportedly went into compute and the first paper, and Hochreiter says a 7B xLSTM can compete with Transformer technology while offering speed and energy advantages. The company’s simulation work reportedly reduces one car-design calculation from three weeks to three minutes.
The largest industrial claim is that learned abstractions could replace prohibitively granular numerical simulation. Instead of calculating every particle or mesh point, a model can identify structures in which thousands of elements move together—the “snowball” rather than every snowflake. Hochreiter says a prototype can cost €100 million and might be avoided, but preserves the condition plainly: “if this works.”
Deep dive
1. Scaling stores more knowledge without solving intelligence
Hochreiter’s framing: an LLM is “a database technology” that captures human knowledge in text, perhaps also code, then generalizes by substitution and recombination—changing Tuesday to Wednesday, for example. That capability is powerful, but it operates on things that “already exist.”
The decisive question is whether every piece of code has effectively been written already. If so, recombination may go extremely far; if progress requires “new code, a new idea, or a new concept,” Hochreiter does not think current language models can produce it.
The host’s pushback—worth keeping: systems such as o1 can generate programs, compose primitive operations and use test-time computation to search indirectly through program space. Hochreiter concedes, “You can tweak it and you will go very far,” but retains his boundary around concepts unavailable through existing components.
His blunt conclusion is that more training data primarily makes the systems larger rather than smarter: “We’re only larger.” Models are approaching what he calls almost brain-like scale, yet humans learn from few examples, adapt, abstract and plan differently. “Something is missing.”
2. Real reasoning needs goals, rules and external tools
Hochreiter distinguishes human reasoning concepts—contradiction, induction, formal rules and structured proof—from an LLM “repeating reasoning” seen in its input. A model may substitute variables or reproduce code patterns, yet small changes can still make the procedure fail.
A model could learn one formal logic well enough to emit syntactically correct formulas and apply its rules to new things. His reservation is semantics and direction: constructing a proof requires goal-directed intermediate steps, and competence learned inside one formal system might not transfer to another. Such systems are still imperfect and often not as good as humans.
Challenged with AlphaGo’s Move 37, Hochreiter accepts that the system created new knowledge, but attributes it to a combination of understanding the game, Monte Carlo tree search and value functions. His preferred general architecture similarly delegates: call a prover, Mathematica or a lookup tool rather than forcing every operation into one network.
3. Vanishing gradients led directly to LSTM
Hochreiter remembers Jürgen as unusually persuasive and creatively restless: when three researchers offered seminar topics to roughly 50 Munich students, Jürgen arrived saying he was not prepared, yet every student selected his neural-network topic. He had also considered becoming an artist rather than a scientist.
In Hochreiter’s diploma work, a neural network needed to retain one piece of information until the end of a sequence. Watching numbers stream across the screen, he noticed “super small numbers”: the gradients had vanished, leaving the beginning of the sequence with no useful credit-assignment signal.
LSTM’s memory cell was constructed so the backward signal would not be repeatedly scaled—the gradient at the beginning could remain the same as at the end. He wrote the result into his diploma thesis; Jürgen later returned, asked whether they should publish it, and they did.
Hochreiter says LSTM still powers flood prediction in Google’s app and systems used by the US and Canadian governments, outperforming alternatives for that task. It also underpinned AlphaStar and dominated language through 2017, often alongside attention, before “Attention Is All You Need” shifted the field toward Transformers.
4. Transformers won on hardware throughput, not asymptotic efficiency
LSTM’s input gate acts like an early attention mechanism, selecting which sequence elements deserve storage; its forget gate discounts old memory. Recurrence interacts with a compressed memory at constant interaction cost per new query, so total work is linear in sequence length.
Attention instead compares each query with prior keys, making its mathematics quadratic in context length and largely pairwise. Hochreiter argues that a recurrent memory might let a new token interact with an abstraction assembled from several earlier tokens rather than only individual key-value comparisons.
Yet quadratic attention ran faster because GPUs process its comparisons in parallel, while classic LSTM updates memory sequentially. FlashAttention remains quadratic—“you cannot cheat math”—but exploits registers, fast memory access and hardware-aware implementation so effectively that its practical throughput overwhelmed LSTM.
5. xLSTM can overwrite bad memories and store richer structures
The xLSTM project asked whether LSTM could inherit the scalable backbone of Transformers while fixing its own limitations. The team parallelized recurrence, enlarged the memory, and addressed a crucial defect: classic LSTM could not revise a choice once it had stored it.
Hochreiter’s example is shopping for clothes under two criteria, price and compatibility with shoes. When a later item is better, the model should heavily upweight it and demote the previous winner; a sigmoid gate capped at one cannot counteract an earlier downweighting by applying a factor above one.
Exponential gating removes that ceiling, while normalization divides by accumulated exponential input gates—“like a rolling softmax.” The original motivation was revising decisions, though Hochreiter also observes gradient peaks associated with systems starting to learn again; why the learning dynamics improve remains explicitly speculative.
A matrix memory replaces the original scalar with a gated Hopfield network, adding outer products of key and value while discounting older contents. He describes Mamba-2 as highly convergent with this design—“xLSTM without the input gate”—but considers the missing input gate important.
6. Chunkwise recurrence targets cheaper inference and real-time control
xLSTM alternates chunks of FlashAttention with recurrent updates, choosing chunk sizes that fit GPU caches efficiently. Hochreiter says they took the idea from the FlashAttention developers, but claims the resulting system is faster than full-context FlashAttention in training as well as inference—an outcome he did not expect.
Autoregressive generation exposes attention’s weakness because each new word repeatedly consults a growing history, even with caching. Hochreiter connects xLSTM’s faster recurrent inference to o1/Strawberry-style “thinking”: under his stated 100-times condition, the system could perform proportionally more inference-time work.
The deployment advantage is fixed memory: sequences of 100 and 100 million elements use the same designed recurrent state. Hochreiter sees that predictability, plus energy efficiency and speed, as a route to embedded robotics where Transformer-driven agents may take seconds to react.
An unnamed company reportedly tested xLSTM on drone GPUs, said the results were “unbelievable,” and told Hochreiter that its drones are now flying autonomously with real-time control. He extends the opportunity to cars and possibly phones, but calls the phone case “perhaps too far-fetched” because he does not know its constraints.
7. Industrial AI requires symbolic guarantees and learned abstractions
Hochreiter says he has never seen an AI system form a proper new abstraction: language is human-made, and even ImageNet places the target object where humans chose. xLSTM might compress sun, beach and cocktail into a holiday concept, but he carefully says, “I don’t know whether it can do it.”
His strategic answer is neurosymbolic integration. Decades of symbolic methods can provide tools, robustness and guarantees around learned systems, particularly where a stopped production process is unacceptable; Austria’s roughly €40 million “Bilateral AI” project is pursuing that combination.
The present integration remains “clumsy”: symbolic researchers propose machine learning for their parameters, while subsymbolic researchers treat symbolic logic as an external shield. Hochreiter wants learning to enter formal systems and formal reasoning to become an integrated component, despite the two communities’ resistance to each other.
He also rejects a clean System 1/System 2 split. Human action ranges continuously from grabbing something without thought, through deciding between two routes, to long chess planning; AI-created abstractions may likewise differ from human concepts because AI systems live in and manipulate the same world differently.
8. NXAI pairs xLSTM with simulations that learn the right scale
Hochreiter founded NXAI after struggling to fund xLSTM at a university while keeping the technology in Europe. A local investor backed “fix the technology and then build on top of it”; the first €10 million went into compute and the first paper.
NXAI now has two industrial pillars. Hochreiter says its 7B xLSTM competes with Transformer technology while offering advantages in speed and energy, but he does not want to fight the many companies already concentrated on language, which is not the core business of much industry.
Neural simulation attacks particle and mesh problems too large for conventional numerics by identifying coherent structures. His analogies are the Moon represented by location, perhaps an impulse, and mass, or a snowball modeled as one object rather than every snowflake: the useful abstraction eliminates enormous redundant computation.
In one car-design case, he says a numerical simulation takes three weeks while the learned system takes three minutes. For steel furnaces that cannot be simulated at full scale, the ambition is to avoid a prototype that can cost €100 million and simulate the real thing instead—subject to his crucial hedge, “if this works.”