The Hadamard MLP for Channel Mixing for Almost No Parameters

A transformer block has two parts. Attention mixes information across tokens, and the feed-forward layer mixes the channels within a token, the 768 numbers that describe it. The feed-forward layers hold most of the parameters in a transformer, and on a small device every parameter has to be read from memory for every token. Needle replaces the feed-forward layer with a Hadamard MLP, a mixer with 25.6K parameters per layer instead of 4.7M. Below is the construction and what it costs.

The cost of a feed-forward layer

A token is a vector $x \in \mathbb{R}^{d}$ with $d = 768$. The standard feed-forward layer is two matrices with a nonlinearity between them:

$\mathrm{FFN}(x) = W_2\, \sigma(W_1 x), \qquad W_1 \in \mathbb{R}^{4d \times d},\ W_2 \in \mathbb{R}^{d \times 4d}.$

It stores $2 \cdot 768 \cdot 3072 \approx 4.7\text{M}$ numbers per layer, and it spends $4.7\text{M}$ multiply-adds on every token.

On a GPU the multiply-adds are the cost. On a phone, a watch or a microcontroller, the cost is moving the weights. Every weight travels from memory into the processor for every token, and memory is slow. A token is not done until the last weight has arrived.

Our attention-only study measured what this layer contributes. Its parameter count matters. Its exact shape matters much less. What the block needs is a mixer that connects every channel to every other, plus a nonlinearity, with as few stored numbers as possible.

A mixer with no weights

The Walsh-Hadamard matrix connects every channel to every other and stores nothing. One rule defines it at every size:

$H_1 = [1], \qquad H_{2n} = \frac{1}{\sqrt{2}} \begin{bmatrix} H_n & H_n \\ H_n & -H_n \end{bmatrix}.$

Tile the matrix four times, flip the sign of the bottom-right copy, scale by $1/\sqrt{2}$.

Three properties make it a good mixer.

1. Every entry is $+1/\sqrt{n}$ or $-1/\sqrt{n}$. Nothing is stored; the rule is the matrix.
2. It is orthogonal, $H H^{\top} = I$. It rotates a vector without stretching it, so values do not grow or vanish as they pass through.
3. Applying it is cheap. The fast transform below does it in $\log_2 n$ stages. Each stage pairs channels and replaces the pair $(a, b)$ by $(a+b, a-b)/\sqrt{2}$, which is $n \log_2 n$ additions in total.

After the last stage every output depends on every input. Needle 2 used this transform as its mixer, with a learned scale on each channel before and after it. Needle 3 keeps the idea and makes the rotation itself learnable, at the same cost. The tool for that is the Kronecker product, in the form of a Monarch matrix.

Folding a vector into a tile

Pad the 768 channels to $n = 1024$ and write them as a $32 \times 32$ tile $Z$, one row after another. Take two small matrices $A, B \in \mathbb{R}^{32 \times 32}$. Mix the rows of the tile with $A$ and the columns with $B$:

$Z \;\mapsto\; A^{\top} Z\, B.$

Flatten the tile back into a vector. The result is the same as multiplying the vector by one $1024 \times 1024$ matrix, the Kronecker product $A \otimes B$:

$(A \otimes B)\, \mathrm{vec}(Z) = \mathrm{vec}\!\left(A^{\top} Z\, B\right).$

A dense $1024 \times 1024$ matrix stores $1{,}048{,}576$ numbers and costs as many multiply-adds. The two factors store $2 \cdot 32^2 = 2{,}048$ numbers and cost $2 \cdot 32^3 = 65{,}536$ multiply-adds. That is 512 times fewer parameters and 16 times less work.

The Walsh matrix is itself a Kronecker product, $H_{1024} = H_{32} \otimes H_{32}$. So $A = B = H_{32}$ at initialisation makes the stage an exact Hadamard transform on the first step of training, and every later step is free to move $A$ and $B$.

Why one stage is not enough

One stage mixes each channel with its own row and its own column of the tile. Two channels in different rows and different columns never meet. Of the 1,024 channels, one stage lets a channel reach $2 \cdot 32 - 1 = 63$.

The fix is a fixed shuffle between stages. After the first stage, a random permutation $\Pi$ moves the 63 reached channels to new places in the tile. The second stage spreads each of them along its new row and column, and after that every channel has a path to every other.

A product of block-structured matrices with permutations between them is a Monarch matrix, the same idea as learned butterfly factorisations. Two stages give full reach. Needle uses three, because the third stage adds capacity, and each stage is still two small matrix products.

The whole layer

$M_s = A_s \otimes B_s$ are the three mixes, $\Pi_1, \Pi_2$ the fixed shuffles, $D_1 \dots D_4$ learned per-channel scales, $b$ a bias, and $\mathrm{silu}(u) = u \cdot \mathrm{sigmoid}(u)$ the nonlinearity:

$y = D_4\, M_3\, D_3\, \Pi_2\, M_2\, \mathrm{silu}\!\big(D_2\, c(x) \odot \Pi_1\, M_1\, D_1\, x + b\big).$

Reading from the inside out: scale, mix, shuffle, scale, nonlinearity, mix, shuffle, scale, mix, scale. The middle scale $D_2$ is multiplied by a gain $c(x)$ that depends on the token:

$c(x) = 1 + \mathrm{softmax}(x V)\, U, \qquad V \in \mathbb{R}^{768 \times 8},\ U \in \mathbb{R}^{8 \times 1024}.$

The token is projected to 8 numbers, softmax turns them into weights over 8 patterns, and each pattern is a vector of channel gains. $U$ starts at zero, so $c(x) = 1$ at initialisation. The gain costs 14K multiply-adds and gives each token some control over how its own channels are scaled, which a wide dense layer gets from its width. $D_4$ starts at 0.02, so a new layer writes gently into the residual stream, the same convention as the attention output.

The parameter count per layer:

Per token, the three mixes cost $3 \times 65{,}536$ multiply-adds, the scales about 5K and the gain 14K, roughly $0.21\text{M}$ in total.

The cost, compared

Per layer the Hadamard MLP is 180 times smaller than the dense layer and 22 times cheaper to run. Over the 20 blocks of Needle 3 the difference is 94M parameters, which would take the model from 121M to 215M, and 196 MFLOPs per token, which would take it from 100 to 296.

A dense feed-forward layer is also where a transformer stores facts. Needle stores them in the engram, a conditional memory of hashed n-gram tables that holds 70.8M of its 121M parameters. A table is read by gather: a few rows per token, selected by hashing the last few tokens, with no arithmetic. So the parameter budget went from a matrix that every token reads in full to a memory that is only touched where it is needed.

Results

The question is what the swap loses. We trained the same 12-layer, $d = 640$ decoder on 800M tokens of Nemotron QA text on 8 H100s, once per mixer, with the same optimiser, schedule and seed, and changed only the feed-forward block. At this width a dense $4d$ feed-forward layer is 3.28M parameters and the Hadamard MLP is 24K; at Needle 3's $d = 768$ the same two numbers are 4.7M and 25.6K. Validation cross-entropy is in nats per token. The removed columns are for the whole model, since that is what a device stores, and the change column is against the dense GELU row.

What we lose relative to what we remove: per layer the mixer goes from 3.28M parameters to 24K, 99.3% of it; the whole model gives up 63% of its parameters and 40% of its multiply-adds, because the embeddings and attention stay, and pays 18% more loss on general QA text. A low-rank feed-forward layer that removes the same 63% pays 28%. Monarch without the Walsh start pays the same as the Hadamard MLP. SwiGLU removes nothing and pays 6%. The token-dependent gain costs nothing and gains nothing measurable here; across three seeds the Hadamard MLP and the same stages without the gain differ by 0.0%, 1.0% and 1.4% of the loss, in alternating directions.

Removing one piece at a time, at a shorter 573-step screen on an 8-layer, $d = 512$ model where the dense layer scores 5.29:

Removing the diagonal scales is the one change that helps at 573 steps and hurts at the full 3,000, where it costs 2.3% of the loss, 2.756 against 2.695.

The 18% is the price on text where the feed-forward layer's storage role matters, the same query-token deficit the attention-only study measured. On tool-shaped data, where the answer is in the context, the price was not where the loss lived, and the engram carries the facts. All rows in the table are one seed; the Hadamard MLP and the gain-free stages were run at two more seeds, which is where the three paired differences come from.

On the device

Each Kronecker stage runs as the two $32 \times 32$ products of the diagram, on a tile that fits in vector registers. A layer's mixer is a few hundred fused multiply-adds per lane with no weight matrix to fetch. The three factor pairs, the scales and the gain are a little over a hundred kilobytes per layer at full precision, and much less in the shipped 2-bit archive, so the mixer stays in cache for the whole session. Decode on a Raspberry Pi 5 runs at 400 to 4,000 tokens per second across the depth ladder, and the Hadamard MLP is a large part of that.

What it gives up

• A dense layer can represent any linear map on its hidden width. Three Kronecker mixes with shuffles cannot represent all of them.
• The rank-8 gain is a small substitute for the token-dependence a wide hidden layer has.
• On the reasoning-dense, tool-shaped data Needle trains on, the block's job is to route and transform what is already in context. That expressivity was not where the loss lived, and the engram took the storage role.
• Whether the same trade holds for knowledge-heavy general text at larger scale is open, and it is the same question the attention-only study left open.
• Within Needle's regime the numbers above are the argument: a 121M model that beats models ten times its size on mobile tool calls, at a third of the compute a transformer of its shape would spend.

References

1. Vaswani et al., Attention Is All You Need, 2017.
2. Walsh, A Closed Set of Normal Orthogonal Functions, American Journal of Mathematics, 1923.
3. Fino and Algazi, Unified Matrix Treatment of the Fast Walsh-Hadamard Transform, IEEE Transactions on Computers, 1976.
4. Dao et al., Monarch: Expressive Structured Matrices for Efficient and Accurate Training, 2022.
5. Dao et al., Learning Fast Algorithms for Linear Transforms Using Butterfly Factorizations, 2019.
6. Hendrycks and Gimpel, Gaussian Error Linear Units, 2016.
7. Cheng et al., Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models, 2026.
8. Ndubuaku et al., A Controlled Study of Attention-Only Transformers, 2026.