The Hadamard MLP for Channel Mixing for Almost No Parameters

A transformer block has two parts. Attention mixes information across tokens, and the feed-forward layer mixes the channels within a token, the 768 numbers that describe it. The feed-forward layers hold most of the parameters in a transformer, and on a small device every parameter has to be read from memory for every token. Needle replaces the feed-forward layer with a Hadamard MLP, a mixer with 25.6K parameters per layer instead of 4.7M. Below is the construction and what it costs.

The cost of a feed-forward layer

A token is a vector $x \in \mathbb{R}^{d}$ with $d = 768$ . The standard feed-forward layer is two matrices with a nonlinearity between them:

$\mathrm{FFN}(x) = W_2\, \sigma(W_1 x), \qquad W_1 \in \mathbb{R}^{4d \times d},\ W_2 \in \mathbb{R}^{d \times 4d}.$

It stores $2 \cdot 768 \cdot 3072 \approx 4.7\text{M}$ numbers per layer, and it spends $4.7\text{M}$ multiply-adds on every token.

On a GPU the multiply-adds are the cost. On a phone, a watch or a microcontroller, the cost is moving the weights. Every weight travels from memory into the processor for every token, and memory is slow. A token is not done until the last weight has arrived.

The cost that matters on a small device. Every weight a layer touches has to be fetched for every token. The dense layer is 180 cells wide at this scale; the Hadamard MLP is one.

Our attention-only study measured what this layer contributes. Its parameter count matters. Its exact shape matters much less. What the block needs is a mixer that connects every channel to every other, plus a nonlinearity, with as few stored numbers as possible.

A mixer with no weights

The Walsh-Hadamard matrix connects every channel to every other and stores nothing. One rule defines it at every size:

$H_1 = [1], \qquad H_{2n} = \frac{1}{\sqrt{2}} \begin{bmatrix} H_n & H_n \\ H_n & -H_n \end{bmatrix}.$

Tile the matrix four times, flip the sign of the bottom-right copy, scale by $1/\sqrt{2}$ .

The Walsh-Hadamard matrix. Each step tiles the last matrix four times and flips the sign of one quarter. The rule is the whole definition, so the matrix costs no memory at any size.

Three properties make it a good mixer.

Every entry is $+1/\sqrt{n}$ or $-1/\sqrt{n}$ . Nothing is stored; the rule is the matrix.
It is orthogonal, $H H^{\top} = I$ . It rotates a vector without stretching it, so values do not grow or vanish as they pass through.
Applying it is cheap. The fast transform below does it in $\log_2 n$ stages. Each stage pairs channels and replaces the pair $(a, b)$ by $(a+b, a-b)/\sqrt{2}$ , which is $n \log_2 n$ additions in total.

The Walsh-Hadamard transform. Each stage pairs channels a fixed distance apart and replaces them by their sum and difference, so n channels are fully mixed in n log₂ n additions with no weights at all. Needle's MLP starts from this transform and learns away from it.

After the last stage every output depends on every input. Needle 2 used this transform as its mixer, with a learned scale on each channel before and after it. Needle 3 keeps the idea and makes the mixing learnable through small Kronecker factors. This is related to the structured matrix approach used in Monarch.

Folding a vector into a tile

Pad the 768 channels to $n = 1024$ and write them as a $32 \times 32$ tile $Z$ , one row after another. Take two small matrices $A, B \in \mathbb{R}^{32 \times 32}$ . Mix the rows of the tile with $A$ and the columns with $B$ :

$Z \;\mapsto\; A^{\top} Z\, B.$

Flatten the tile back into a vector. The result is the same as multiplying the vector by one $1024 \times 1024$ matrix, the Kronecker product $A \otimes B$ , using the row-vector convention of the implementation:

$\mathrm{rowvec}\!\left(A^{\top} Z\, B\right) = \mathrm{rowvec}(Z)\,(A \otimes B).$

One Kronecker stage. A 1,024 × 1,024 mixing matrix is replaced by the Kronecker product of two 32 × 32 matrices, which is applied by folding the token into a tile and multiplying its rows, then its columns. The engine runs each stage this way, in registers.

A dense $1024 \times 1024$ matrix stores $1{,}048{,}576$ numbers and costs as many multiply-adds. The two factors store $2 \cdot 32^2 = 2{,}048$ numbers and cost $2 \cdot 32^3 = 65{,}536$ multiply-adds. That is 512 times fewer parameters and 16 times less work.

The Walsh matrix is itself a Kronecker product, $H_{1024} = H_{32} \otimes H_{32}$ . So $A = B = H_{32}$ at initialisation makes the stage an exact Hadamard transform on the first step of training, and every later step is free to move $A$ and $B$ .

What the extra stages add

One factor mixes rows and the other mixes columns, in sequence. A value first spreads along its column under $A^{\top} Z$ , then across every column under multiplication by $B$ . One complete Kronecker stage can therefore connect all 1,024 channels.

The two factors act in sequence. One spreads a value along its column; the next spreads that column across the entire tile. A full Kronecker stage already connects all channels. Shuffles change the factor grouping between stages.

Full connectivity does not mean an arbitrary dense matrix. The weights still have the factorisation $A \otimes B$ . Needle uses three stages, fixed shuffles, learned channel scales and a nonlinearity to compose more flexible transformations. The shuffles change which channels share the same factor grouping.

The whole layer

For the remaining equations, $x$ is the padded column vector and $M_s = (A_s \otimes B_s)^{\top}$ are the three mixes, $\Pi_1, \Pi_2$ the fixed shuffles, $D_1 \dots D_4$ learned per-channel scales, $b$ a bias, and $\mathrm{silu}(u) = u \cdot \mathrm{sigmoid}(u)$ the nonlinearity:

$y = D_4\, M_3\, D_3\, \Pi_2\, M_2\, \mathrm{silu}\!\big(D_2\, c(x) \odot \Pi_1\, M_1\, D_1\, x + b\big).$

Reading from the inside out: scale, mix, shuffle, scale, nonlinearity, mix, shuffle, scale, mix, scale. The middle scale $D_2$ is multiplied by a gain $c(x)$ computed from the original $d = 768$ channels:

$c(x) = 1 + U^{\top}\,\mathrm{softmax}(V^{\top} x_{1:d}), \qquad V \in \mathbb{R}^{768 \times 8},\ U \in \mathbb{R}^{8 \times 1024}.$

The token is projected to 8 numbers, softmax turns them into weights over 8 patterns, and each pattern is a vector of channel gains. $U$ starts at zero, so $c(x) = 1$ at initialisation. The gain costs 14K multiply-adds and gives each token some control over how its own channels are scaled, which a wide dense layer gets from its width. $D_4$ starts at 0.02, so a new layer writes gently into the residual stream, the same convention as the attention output.

Three Kronecker mixes, two fixed permutations between them, four learned diagonals, one bias, one activation and a rank-8 gain that lets the input modulate its own channel scaling. Everything mixes; almost nothing is stored.

The parameter count per layer:

piece	count
three Kronecker stages, $3 \times 2 \times 32^2$	6,144
four diagonal scales and a bias, $5 \times 1024$	5,120
gain, $V$ : $768 \times 8$	6,144
gain, $U$ : $8 \times 1024$	8,192
total	25,600

Per token, the three mixes cost $3 \times 65{,}536$ multiply-adds, the scales about 5K and the gain 14K, roughly $0.21\text{M}$ in total.

The cost, compared

At d = 768 with a 4d hidden width, a dense feed-forward layer holds 4.7M parameters and spends 4.7M multiply-adds per token; the Hadamard MLP holds 25.6K and spends about 0.21M. Across 20 mixers that is about 94.37M versus 4.20M multiply-adds per token. These counts cover the channel mixers, not the whole model or measured latency.

Per layer the Hadamard MLP is about 184 times smaller than the dense layer and uses about 22 times fewer multiply-adds. Over the 20 blocks of Needle 3 the difference is 94M parameters, which would take the model from 121M to 215M, and about 90M multiply-adds per token in the channel mixers alone.

A dense feed-forward layer is also where a transformer stores facts. Needle stores them in the engram, a conditional memory of hashed n-gram tables that holds 70.8M of its 121M parameters. A table is read by gather: a few rows per token, selected by hashing the last few tokens, with no arithmetic. So the parameter budget went from a matrix that every token reads in full to a memory that is only touched where it is needed.

Results

The question is what the swap loses. We trained the same 12-layer, $d = 640$ decoder on 800M tokens of Nemotron QA text on 8 H100s, once per mixer, with the same optimiser, schedule and seed, and changed only the feed-forward block. At this width a dense $4d$ feed-forward layer is 3.28M parameters and the Hadamard MLP is 24K; at Needle 3's $d = 768$ the same two numbers are 4.7M and 25.6K. Validation cross-entropy is in nats per token. The removed columns are for the whole model, since that is what a device stores, and the change column is against the dense GELU row.

mixer, same model shape	mixer per layer	model	removed	MACs removed	validation CE	loss change
dense FFN, GELU, hidden $4d$	3.28M	61.6M			2.281
dense FFN, SwiGLU	3.32M	62.1M	0%	0%	2.422	+6.2%
Hadamard MLP	24K	22.6M	63%	40%	2.695	+18.1%
Kronecker stages without the gain	11K	22.4M	64%	40%	2.695	+18.1%
Monarch, normal init	65K	23.1M	62%	41%	2.699	+18.3%
low-rank FFN, same size	24K	22.6M	63%	42%	2.930	+28.5%

What we lose relative to what we remove: per layer the mixer goes from 3.28M parameters to 24K, 99.3% of it; the whole model gives up 63% of its parameters and 40% of its multiply-adds, because the embeddings and attention stay, and pays 18% more loss on general QA text. A low-rank feed-forward layer that removes the same 63% pays 28%. Monarch without the Walsh start pays the same as the Hadamard MLP. SwiGLU removes nothing and pays 6%. The token-dependent gain costs nothing and gains nothing measurable here; across three seeds the Hadamard MLP and the same stages without the gain differ by 0.0%, 1.0% and 1.4% of the loss, in alternating directions.

Removing one piece at a time, at a shorter 573-step screen on an 8-layer, $d = 512$ model where the dense layer scores 5.29:

ablation	validation CE	loss change
full Hadamard MLP	5.84
one Kronecker stage instead of three	5.95	+1.9%
normal initialisation instead of Walsh	5.95	+1.9%
no permutations between stages	5.90	+1.0%
rotation fixed at Walsh, only scales learn	5.86	+0.3%
no diagonal scales	5.45	−6.8%

Removing the diagonal scales is the one change that helps at 573 steps and hurts at the full 3,000, where it costs 2.3% of the loss, 2.756 against 2.695.

The 18% is the price on text where the feed-forward layer's storage role matters, the same query-token deficit the attention-only study measured. On tool-shaped data, where the answer is in the context, the price was not where the loss lived, and the engram carries the facts. All rows in the table are one seed; the Hadamard MLP and the gain-free stages were run at two more seeds, which is where the three paired differences come from.

On the device

Each Kronecker stage runs as the two $32 \times 32$ products of the diagram, on a tile that fits in vector registers. The kernel reads the small learned factors instead of a full dense mixing matrix. The three factor pairs, the scales and the gain are a little over a hundred kilobytes per layer at full precision, and much less in the shipped 2-bit archive, so the mixer stays in cache for the whole session. Decode on a Raspberry Pi 5 runs at 400 to 4,000 tokens per second across the depth ladder, and the Hadamard MLP is a large part of that.

What it gives up

A dense layer can represent any linear map on its hidden width. Three Kronecker mixes with shuffles cannot represent all of them.
The rank-8 gain is a small substitute for the token-dependence a wide hidden layer has.
On the reasoning-dense, tool-shaped data Needle trains on, the block's job is to route and transform what is already in context. That expressivity was not where the loss lived, and the engram took the storage role.
Whether the same trade holds for knowledge-heavy general text at larger scale is open, and it is the same question the attention-only study left open.
Within Needle's regime the numbers above are the argument: a 121M model that beats models ten times its size on mobile tool calls, at a third of the compute a transformer of its shape would spend.

References

Vaswani et al., Attention Is All You Need, 2017.
Walsh, A Closed Set of Normal Orthogonal Functions, American Journal of Mathematics, 1923.
Fino and Algazi, Unified Matrix Treatment of the Fast Walsh-Hadamard Transform, IEEE Transactions on Computers, 1976.
Dao et al., Monarch: Expressive Structured Matrices for Efficient and Accurate Training, 2022.
Dao et al., Learning Fast Algorithms for Linear Transforms Using Butterfly Factorizations, 2019.
Hendrycks and Gimpel, Gaussian Error Linear Units, 2016.
Cheng et al., Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models, 2026.
Ndubuaku et al., A Controlled Study of Attention-Only Transformers, 2026.

The Hadamard MLP for Channel Mixing for Almost No Parameters

Cite this post