[NEW]Needle: a 26M tool-calling model distilled from Gemini
Read moreCactus Blog
Deep dives into on-device AI, inference optimization, and the engineering behind Cactus.
Needle: We Distilled Gemini Tool Calling into a 26M Model
An open-source 26M parameter function-calling model that runs at 6000 tok/s prefill and 1200 tok/s decode on consumer devices.
Henry Ndubuaku
TurboQuant-H: Hadamard Rotation for 2-Bit Embedding Quantization
A simplified offline variant of TurboQuant using Hadamard rotation and per-group Lloyd-Max codebooks — 4× compression of per-layer embeddings in Gemma 4 E2B at +0.06 PPL.
LFM-2.5-350m on Cactus: 140 tok/sec, Single Core, 355 MB
Benchmarking Liquid's LFM-2.5-350m across seven devices with Cactus. INT8 quantization, single-core CPU decode, zero-copy loading, and why this configuration makes on-device inference practical.
Sub-150ms Transcription with Cloud-Level Accuracy: Why We Built a Hybrid Engine
How Cactus combines on-device and cloud inference for real-time speech transcription with sub-150ms latency and automatic cloud handoff for noisy audio.
Ridiculously Fast On-Device Transcription: Reviewing Parakeet CTC 1.1B with Cactus
Review of NVIDIA's Parakeet-CTC-1.1B model running locally on Mac with Cactus. Architecture breakdown, benchmarks, and transcription use cases.
The Sweet Spot for Mac Code Use: Reviewing LFM2 24B MoE A2B with Cactus
Review of LiquidAI's LFM2-24B-A2B mixture-of-experts model running locally on Mac with Cactus. Architecture breakdown, benchmarks, and coding agent use cases.