TL;DR

There is a lot of speculation about GPT-6 Astra, so let me talk about the Loop Transformer a little more seriously. By definition I think it should be split into two kinds: Loop and Recursive. The former is the widely circulated version that takes the model's Transformer blocks as one whole and runs the loop over them twice; the latter is a certain recursive structure built on fine-grained management of the Context, which is what we will develop in detail in this article.

The most widely circulated version of the Loop Transformer is Sebastian's *OpenAI Astra and Looped Transformers*^[1]. Expressed simply in pseudocode:

# Created only once at initialization.
self.block = TransformerBlock(...)

# The same set of parameters is reused in the forward pass.
for _ in range(num_loops):
    h = self.block(h)

Let us call this approach the Loop Transformer for now, but I do not think it is right, and it is not the kind I like. Of course this Weight Tying approach may have some use — for example when a foundation-model vendor is limited on compute, say it does not have enough large-VRAM Blackwell cards so its model size is capped under 1T parameters, and it still wants to Scale the model...

Also, Jakub has a tweet suggesting that this may not be the case...

So this article will expand a little on Jakub's second paragraph: how to work on the CoT. And the architecture I personally prefer, which is perhaps a Recursive one, is related to this as well, together with some composable context material, in order to achieve Recursive Self-Improvement (RSI).

1. Transformer: What Can We Actually Change?

Most work discussing LLM Memory frames it as "adding a memory module to the model". I think that framing gets the question wrong from the very start. A more worthwhile question is a different one: in a standard Transformer, how many places on its computation graph can actually be written to? Is CoT itself a kind of memory injection?

1.1 An Analysis of Memory Injection Points in a Transformer

For a standard Pre-LN Transformer Block at layer :

In the attention part, for head , position reading position :

There is one rewrite here that is absolutely crucial. Write the score as a bilinear form on the residual stream:

And write the output side as:

• is this head's "metric", which decides who it considers similar to whom, that is, the QK circuit;
• is its "readout control", which decides what to write onto the bus once something has been read, that is, the OV circuit.

The entire identity of a head is this pair of low-rank matrices , each with rank no greater than . Every modification to Attention is, in essence, a discussion of how to temporarily alter this pair of matrices.

That makes the injectable surfaces on a transformer block clear: the residual , the gain and bias of normalization, the metric , the bias , the summation range , the head set , the output gate, and the subsequent FFN. I have summarized them in the figure below:

The first thing that comes to mind is modifying the residual stream that serves as the input. There are many ways to do this, for example the much-talked-about Loop Transformer, DeepMind's Recirculation injection, and of course all kinds of HyperConnection widening. But it is worth noting that this approach changes several positions inside Attention at once, causing Q, K and V all to change.

Beyond things like the Loop Transformer, we can also modify some biases (Attention-Bias), or dynamically add and remove Heads and make other fine-grained structural adjustments. So can we find a computationally efficient way to separate the memory-injection computation for Q, K and V? Or even some Test-Time modifications? Can the Query be modified or rewritten at Test-Time? Can we make some KV-related modifications to the summation inside the Softmax?

1.2 The Attention That RSI Requires

First let us look at what CoT actually adds to the whole computation. Looking purely at the Decode stage, for the generation of the -th token in the autoregressively generated content, the Attention Score the model needs to compute is shown in the figure below.

Suppose we have finished a think block and are decoding the output token by token. At this point is only related to the current token's , so we can take the essence of CoT to be influencing the next-token prediction through K,V.

The common methods for influencing the selection of K,V are SWA (Sliding Window Attention) and some Sparse Attention algorithms: the former truncates with a fixed window, the latter truncates through TopK selection.

Just as I said in my previous article On the Loop Transformer:

So next we need to focus on the algorithmic design of Attention. Put more directly: how do we find, from the algorithmic side, the direction in which Attention can scale further? Let us first talk about some of the shortcomings of standard Attention.

When we stuff the CoT directly into the same softmax as memory columns:

then adding columns inevitably dilutes the probability mass that the context receives, and how much it is diluted depends on the norm of the memory keys, which is entirely uncontrollable. Although there are Attention Sink mechanisms to represent an empty slot that means "nothing worth taking", once we bring in the potential perturbation mechanisms of CoT, in particular some of the mechanisms at work during the RL Post-Training stage, things are not clear to us.

Another shortcoming is the problem. Although Linear Attention and Sparse Attention algorithms already alleviate it, some shortcomings remain. A while ago I did a detailed analysis of Linear Attention and Sparse Attention:

• Chinese version: https://zartbot.github.io/blog/model_arch/linear_vs_sparse/index.html
• English version: https://zartbot.github.io/blog/model_arch/linear_vs_sparse/index_en.html

Just as that analysis explains, the approach I hope for is a mechanism of some kind of BLOCK-based Composable Context. A very simple intuition is this: could the storage of these KVs be built by borrowing from the WAL and LSM mechanisms common in storage systems and databases? Or, to put it simply, for the composition of the context, can it be built composably following a block mechanism (Composable Context), and then, when it truly lands in storage, be subject to some Merge / Compaction mechanism similar to LSM? And could a compacted Memory block be recovered through LLM Decoding?

On the other hand, considering that we must not break computational parallelism and computational efficiency, for Attention I hope for some kind of multi-branch explicit Gating, for example:

In essence this constitutes a Recursive Attention structure, and Composable Context can be used within the recursive process. The next chapter will develop this in detail.

2. Recursion: Thinking About Thinking

2.1 Why Recursion Is Needed

As Chapter 1 explained, after the simple Loop Transformer's looped injection, Q, K and V all change, so what we want is some kind of QKV-separated treatment. Therefore, in the course of the Recursive treatment, we need to answer two questions: can Q be rewritten? And how do we build a Composable KV for KV?

We also note that the DeepSeek Harness framework has already laid out the view of spatiotemporal composability in the paper *A Programming Paradigm for Spatiotemporal Composability*^[2], but personally I feel these capabilities should be internalized directly inside the model, that is, the model itself has the ability to do composable context, and can perform operations such as composition / divide-and-conquer / recursion on top of the chain of thought through some recursive structure, rather than relying on an external Harness framework.

The other consideration is efficiency. For ultra-large-scale models (5T~10T parameters), we need to do our utmost to maintain high-speed inference; every pass through FFN/MoE introduces enormous memory-access and communication overhead, and the deeper the model's layer count, the greater the impact. At the same time, from some Topological Data Analysis, current Transformer models do not effectively use the model's depth, and intelligence tends to emerge at deeper positions in the model. So what I hope for is to do some recursive processing only in the Attention block, so that while keeping the model's depth relatively shallow (e.g. 60~80 layers), we pay more compute inside Attention to achieve the effect of an equivalently deeper model.

From the algorithmic side, recursion adds a few capabilities over CoT. The first is the stack structure; its essence is that it changes the shape of the computation graph, making "thinking" an object that can be thought about again. Traditional CoT, on the other hand, is essentially a serial structure: the longer the context, the bigger the trouble, and in particular wrong conclusions from past reasoning cannot be erased. If we regard the KV as a stack, then CoT in essence only has the push operation. Although we can achieve erasure through the state matrix of Linear Attention, Linear Attention has shortcomings in the composability and commutativity of the context. So in fact, can we, on top of Sparse Attention, add before the Indexer some Mask that is learnable and can be generated at inference time, as a parameter of the recursion? That would then constitute a complete stack structure.

Second, since this is a recursive structure, the return value needs to exist in token form, rather than as some simple modification, addition or removal of KV. In effect, we then have thinking about the thinking produced by , giving .

2.2 Thinking About Thinking: Why It Is Not a "Summary of Summaries"

Based on the recursive line of thought, and drawing on some of the LSM thinking from databases, we can organize the whole Context into a hierarchical structure:

So how do we split a continuous stretch of CoT into different Blocks, or rather, how does the model autonomously generate Blocks while Decoding? And L3 is where the substance of "thinking about thinking" lies. The output of first-order thinking is the answer to this problem; the output of second-order thinking is the method for this class of problems. The former evaporates along with the request, the latter can enter the rule library.

What needs attention here is that L3 must not become a "summary of summaries"; this is precisely the point where the vast majority of "hierarchical memory" designs get stuck. In essence, the summaries we generate need to satisfy the requirement that the LLM can recover the entire thinking process from them. For example, if we have a stretch of CoT for a mathematical proof, then in the L3 summary we need to rigorously record the proof's conditions, key steps and conclusion, ensuring that even if the CoT has been discarded, the original detailed proof process can still be derived from the summary.

Another question is how the L3 Blocks we have summarized should be injected into the Transformer. Chapter 1 explained why one should not loop on the residual: residual injection rewrites both score and value, addressing and content are entangled, and the ratio is not yours to decide either. So when considering other injection points, we also want to preserve the model's efficiency, which is to say such an algorithm needs to guarantee:

1. Single-version: the cache keeps only one copy, with zero increment relative to the un-looped model.
2. Train-inference consistency: the read version is paired between training and prefill, and consistent with decode.
3. Prefill parallelism: no cross-token diagonal dependency is introduced.

The potential conclusion in the end is that, in the course of the recursion, and are frozen once they have been computed in the first round, and each subsequent round only recomputes and the residual, repeatedly reading the same copy of KV. Of course the content of a new L3 Block can be Appended into the KV, but we need to construct a Query-related Mask in order to make the KV reads and the Attn Score computation differ across different iterations.

With that, the overall line of thought for building the Recursive Transformer is basically settled.

2.3 From MEMENTO to INCEPTION

In the course of writing this article, I came across a paper *MEMENTO: Teaching LLMs to Manage Their Own Context*^[3]. Its name is taken from Christopher Nolan's 2000 film, in which the protagonist suffers from anterograde amnesia and compensates for his memory deficit by maintaining external memory artifacts, which is precisely an analogy for a model that must reason by relying on compressed summaries of its own past thinking. A memento is not a summary in the usual narrative sense, but the minimal record of a reasoning Block: using as few tokens as possible to retain its conclusion, intermediate values and key directional decisions. Once a memento has been produced, the preceding Think Block is Masked inside a single uninterrupted generation call. The rough idea is as follows:

The figure below is a concrete Example:

But judging from this implementation, what it does is physical eviction: when a block completes, compact_kv_cache copies the live entries into contiguous slots and frees the trailing KV pages. The paper writes the reason for this choice very plainly: standard FlashAttention and paged-attention kernels work without modification, they simply never see the evicted tokens. So MEMENTO fits the KV Stack of LLM computation with a pop, but that pop is destructive.

As for the Recursive Transformer, it reminds me of another Nolan film, Inception. When we do Thinking on the CoT, it is as if we have entered another layer of space, while the content of the original layer does not change; it waits there. And the time scale of each layer is different, which is the fundamental reason why we need to build summaries at the L3 Block. This is also why I have previously wanted to build some algorithms from the perspective of a Nerve construction. The key problem is the same as well: how to exit from a deeper Thinking space back to the layer above.

From Memento to Inception (Recursive Transformer), what we can essentially see is that Memento mainly controls the writing of KV, while Inception mainly exercises control over the reading of KV. Therefore, we can borrow Memento's method to chunk the context and compress it into summaries, and then make recursive selections on the block structure and the block summaries it leaves behind. Through a learnable and recursive Mask placed before the Sparse Attention Indexer, we control whether a block is pushed onto the Indexer's stack, thereby forming a Composable Context. And original Think Blocks that have not been accessed for a long time can also be discarded from storage, keeping only the summary blocks; when they are genuinely needed, we can let the model Prefill once again by itself, or reason through it from the summary blocks, to obtain the complete information. But this kind of operation requires a commutativity property for KV Blocks.

3. Conclusion

This article started from the points at which memory can be injected into a Transformer, analyzed the shortcomings of the Loop Transformer and the Recirculation family, then, drawing on Memento's work and further combining it with the Nerve construction, proposed a Recursive Transformer architecture, which through a recursive construction and a stack structure, together with Block based Masking and Block Summary, reduces the active Context length. This approach is naturally suited to Sparse Attention: it only requires building a learnable and recursive Mask before the Indexer. Of course, for the composability and commutativity of Blocks, more co-optimization of the Attention computation and the related Prefill operators is still needed.

How to partition Blocks also requires fairly detailed training-data preprocessing, or SFT/RL in the Mid-training stage, so that the model learns <|Lk: block_start|>, <|Lk: block_end|>, <|Lk: summary_start|>, <|Lk: summary_end|> and so on, as well as how to enter the next layer of CoT or return to the upper layer of CoT to continue reasoning.

If we take RLHF-based ChatGPT as the first generation of LLMs, and the long-CoT OpenAI o1 / DeepSeek-R1 elicited by RL as the second generation of LLMs, then a recursive-thinking construction on top of CoT, co-designed with modifications to the Transformer architecture, can count as the third generation of LLMs. Moreover, once these capabilities have been internalized into the model, many of the complex Skills / Harness problems of the past can also be readily resolved, and perhaps we can even see a glimmer of hope for RSI.