Transfer Attention: What is the difference between Kimi Delta Network and LSTM?

Published

One of my students asked me a question last week that I couldn’t answer with a diagram alone: “How does Delta Network attention actually differ from an Long Short Term Memory (LSTM)? If the same token arrives with the same memory, what does each one do to the numbers?”

This is exactly the right question to ask, because the difference between these two models only becomes visible in arithmetic. So let’s take the sentence “your journey starts with one step” and watch both models process a single moment: the instant the token “journey” arrives right after “your”. To keep every number visible, we give the memory state just three dimensions. That is worth pausing on by itself — even if the final prediction is a single scalar like tomorrow’s air temperature, the internal memory can be as wide as we like, and each dimension gets its own gate values.

Setup. The token “journey” arrives as x₁ = [0.55, 0.87, 0.66], and yesterday’s memory (left over from “your”) is s₀ = [0.80, 0.50, 0.30]. Every gate value below is a realistic sigmoid or tanh output.

LSTM: two free dials, then a curtain

The three gates and the candidate come out as:

The memory update is element-wise multiplication, twice:

s₁ = f ∘ s₀ + i ∘ a₁ = [0.72, 0.25, 0.03] + [0.06, −0.12, 0.72] = [0.78, 0.13, 0.75]

h₁ = o ∘ tanh(s₁) = o ∘ [0.653, 0.129, 0.635] = [0.653, 0.065, 0.127]

LSTM processes one token with two independent dials and an output curtain. Dimension three, highlighted in red, keeps 0.03 of the old value and writes 0.72 of the new one, while the output gate lets only 0.127 through.

Watch the third dimension, outlined in red above. The model keeps only 0.10 × 0.30 = 0.03 of the old value and simultaneously writes 0.80 × 0.90 = 0.72 of the new one. Forget and input are independent dials, so the LSTM is free to throw the old away aggressively while writing the new aggressively — the two decisions never have to agree. Then a separate decision is made about exposure: tanh squashes 0.75 down to 0.635, and the output gate shows only 20% of it, so the outside world sees 0.127. Notice also what the LSTM cannot do: the gates act uniformly on whatever sits in each dimension, with no notion of which past token put it there.

Kimi Delta Network (KDN): memory is a dictionary, not a dial

Now replace the memory vector with a 3×3 matrix S, updated by the gated delta rule:

S₁ = S₀·(Diag(α) − β·k(α ∘ k)ᵀ) + β·v·kᵀ, with readout o = S·q

The mapping to LSTM vocabulary is direct: S is the long memory, α handles forgetting (per column), β·v·kᵀ is the write, and multiplying by a query q is the readout. The crucial addition is the key k: every write and every read is addressed to one specific slot.

The Delta Network processes two tokens as dictionary writes. “Your” writes its value into its own key column; “journey” decays that column by half, writes its own value at its own key, and the query reads back exactly what “journey” wrote.

Step 1, the token “your”: k = [1,0,0], v = [0.5, 0.5, 0], α = [0.9, 0.8, 1.0], β = 0.8, and S₀ = 0.

  • The erase-and-decay matrix Diag(α) − β·k(α ∘ k)ᵀ = [[0.18, 0, 0], [0, 0.8, 0], [0, 0, 1.0]] multiplies S₀, but S₀ is all zeros, so there is nothing to erase yet.
  • The write term β·v·kᵀ places the β-scaled value into column one, the column owned by key “your”: S₁ = [[0.4, 0, 0], [0.4, 0, 0], [0, 0, 0]].
  • Reading with q₁ = [1,0,0] returns o₁ = [0.4, 0.4, 0] — precisely the value stored under “your.”

Step 2, the token “journey”: k = [0,1,0], v = [0.0, 0.8, 0.2], α = [0.5, 0.9, 1.0], β = 1.0.

  • Decay: right-multiplying by Diag(α) scales column j of S by αⱼ, so the entire column owned by “your” is halved — both 0.4 entries become 0.2.
  • Erase: the key [0,1,0] currently holds S₁·(α ∘ k) = [0, 0, 0]. Nothing has been written at this key yet, so the correction v − old is the full v. When a key already holds a value, only the correction gets written — that is the delta rule doing its job, and it is what lets the model update one fact without disturbing the rest of memory.
  • Write: S₂ = decayed + 1.0·v·kᵀ = [[0.2, 0, 0], [0.2, 0.8, 0], [0, 0.2, 0]].
  • Read: with q₂ = [0,1,0], o₂ = [0.0, 0.8, 0.2] — exactly the value “journey” wrote, while the column belonging to “your” sits untouched at 0.2.

The same step, two behaviors

Model Kept of old 0.30 Added of new Why Readout
LSTM 0.03 (f = 0.10) 0.72 (i = 0.80) Gates are independent dials, applied to the whole state 0.127, gated and squashed
Delta Network Column “your” halved by α β·(v − old) written at key k only Writes target one key, not the whole state o = S·q, read by query

The one-line answer

  • LSTM: two free dials, keep and add, applied to the whole memory vector at once, then a curtain that chooses what to show.
  • Delta Network: decay per column, write by key, read by query — the memory is a dictionary, not a dial.

Your journey starts with one step, and now you have watched both models take it.