Fine-tuning Needle

Needle was designed to be customised. Its capacity is a ladder, and a subnetwork as small as two layers, fine-tuned on one product's tools, runs on devices far smaller than the full model needs. Constraining a narrow, well-defined task is what lets it reach frontier accuracy there: fine-tuning on DroidCall lifts every subnetwork by 18 to 36 points, and from four layers up the tuned subnetwork passes DeepSeek V4 Flash.

Base and tuned are both scored with forced calls; DeepSeek V4 Flash runs through its cloud API. There are two ways to fine-tune, and the cactus-needle package drives both:

The sections below are the local path. The platform section at the end shows the same package running the hosted one.

What a fine-tune is

LoRA adapters of rank 16 on the five attention projections of every layer, trained on your JSONL with the base frozen, then merged into the weights at export. Training runs at the full 20 layers through the same 4-bit quantisation-aware numerics the export uses, so the adapter matches the archive needle build writes. The engine, the tokenizer and the confidence head are untouched, and the output is a single .cact you pass as weights=.

Install

The runtime package does not carry the training stack. Add the train extra, and gpu or metal to train on an NVIDIA GPU or Apple Silicon:

pip install "cactus-needle[train]"
pip install "cactus-needle[train,gpu]"
pip install "cactus-needle[train,metal]"

Training is plain JAX. On NVIDIA the same command trains on the GPU with nothing else changed. Apple GPUs go through the jax-metal plugin, which does not work past jax 0.4.38, so the metal extra pins an older stack and Needle adapts to it (manual attention, no rematerialisation, unrolled layers). On an M5 Max a step takes 0.71 seconds against 2.90 on CPU at the same shape, with a one-time compile of about 23 seconds. Training is float32 on every backend.

Data

One JSON object per line. query and tools describe the turn, answers lists the exact calls the model should emit, and reasoning is one short line deriving each argument from its span in the query. The tools follow the same rules as at inference, one tool per action with formats in the descriptions and constraints in the schema; How to Design Tools for Needle 3 is the reference, and Structured JSON Extraction with Needle covers the case where the tool is a record to fill rather than an action:

{"query": "dim the kitchen to 10", "tools": [{"name": "set_lights", "parameters": {"type": "object", "properties": {"room": {"type": "string"}, "brightness": {"type": "integer"}}, "required": ["room"]}}], "answers": [{"name": "set_lights", "arguments": {"room": "kitchen", "brightness": 10}}], "reasoning": "'kitchen' -> room; 'to 10' -> brightness"}

reasoning is optional but include it: the model produces the derivation before the call, and examples that show where each value comes from teach grounding, not just tool selection. The rules that matter:

1. Arguments contain only values present in the query. Omit optional fields with no evidence; never fill them with placeholders or empty strings.
2. Include off-topic examples with "answers": []. The built-in generator produces about one in eight. Without them the tuned model calls a tool on everything.
3. When the catalogue has similar tools, include ambiguous queries resolved to the correct one.
4. An optional "system" field per example becomes a system turn, matching Needle(system=...) at inference.
5. Each rendered example must fit within --max-len (default 1024) tokens; longer ones are silently truncated. Padding rounds up to the longest example, so a short dataset trains fast regardless of the cap: 245-token examples train six times faster padded to 256 than to 1024 on CPU.
6. The platform's chat-format files, messages with OpenAI-form tools, train locally as well. A line with one user turn answered by one assistant turn, with or without a system turn, is read as the example above; multi-turn lines are skipped with a count.

Extraction uses the same format with the record as the tool and the passage as the query. To grow a small hand-written set, seed the generator with it (needs OPENROUTER_API_KEY; OPENROUTER_URL points it at another OpenAI-compatible gateway):

export OPENROUTER_API_KEY=sk-or-...
needle generate-data --tools my_tools.json --num-samples 500 --output data.jsonl
needle generate-data --augment data.jsonl --num-samples 1000

Train, build, run

needle finetune data.jsonl --epochs 10 --out adapter.safetensors
needle build --lora adapter.safetensors --out tuned.cact
needle build --lora adapter.safetensors --layers 8 --out tuned_8l.cact
needle build --lora adapter.safetensors --platform linux-arm64 --layers 2 --out ./device

The base checkpoint downloads from Hugging Face on the first run. needle build merges the adapter into the full base, slices the subnetwork you asked for, quantises to 4 bits and packages the tokenizer into a .cact archive; with --platform it also downloads that platform's engine and header, so --out becomes a folder ready to copy onto the device. What Devices Are Supported on Needle lists every platform folder. Defaults: batch 16, learning rate 1e-4 with warmup and cosine decay, gradient clipping at norm 1, rank 16, alpha 32, max length 1024, validation split 0.1, seed 0. --seed reproduces or deliberately varies the LoRA initialisation, the validation selection and the epoch shuffle.

agent = needle.Needle(tools=[...], weights="tuned.cact")
agent.run("...")

The engine is weights-agnostic, so a tuned archive runs on it directly, with the same API, response shape and behaviour contract as the base model; the Needle Python Docs cover them. Set NEEDLE_HF_REPO=<you>/<model> and pass --upload to publish it; needle download <you>/<model>/tuned.cact pulls it on any machine.

Reading the loss

The loss covers only the target: the reasoning line plus the JSON call. Much of the call is boilerplate the base already predicts (the name, the braces, the field names), so a run starts near 1.0 rather than near random. Judge it by the trend, not the level.

Step count is what small datasets get wrong. 200 examples at batch 16 is 13 steps per epoch, and three epochs is 39 steps, which barely moves a rank-16 adapter at the default learning rate. For a few hundred examples run 10 to 30 epochs and expect a clear downward trend. If the curve sits at its starting value after a few hundred steps, raise the epochs first, then the learning rate. A validation loss prints at each epoch end; when it rises while the training loss keeps falling, the run is overfitting: stop there, or add data.

Sizing the dataset

Tool selection moves first: a few hundred clean examples measurably improve which tool gets picked. Argument grounding moves later and needs more, on the order of thousands of examples with reasoning lines and varied phrasings and values. If evaluation shows correct tools with wrong argument values, the dataset is too small or too uniform, not mislabelled. For grounding-heavy tasks --lora-rank 32 doubles adapter capacity and the adapter stays tiny.

For a large catalogue, consider two passes at inference instead of more training: one turn against the full catalogue to pick the tool, then one turn declaring only that tool, which constrains the grammar to exactly that call.

The platform from the package

The hosted path trains the full model, every depth from 2 layers up, reinforced with Needle's own dataset so the tuned model keeps what the base knew, and it trains the confidence head with it. It needs a key from the console in NEEDLE_API_KEY and three chat-format files, training, validation and test, which you can bring or generate from your tool definitions:

export NEEDLE_API_KEY=needle_ft_...
needle platform generate --tools tools.json --examples 1000 --description "Voice control for a smart home" --out ./data
needle platform finetune data/train.jsonl data/validation.jsonl data/test.jsonl --suffix smart-home --out ./models

finetune uploads the files, submits the job, prints its dashboard page, waits, prints the validation and test accuracy of every depth, and downloads one .cact per size. A fine-tune spends one run of the plan's allowance at submission, and a generation spends its example count. Ctrl-C leaves the job running; needle platform jobs <id> --wait --out ./models resumes the wait and the download, and needle download model-<id> --depth 8 fetches one size of any model you own. The same calls are available in Python:

from needle.platform import Platform

client = Platform()
job = client.wait(client.finetune(["train.jsonl"], ["validation.jsonl"], ["test.jsonl"], suffix="smart-home"))
job["evaluations"]
paths = client.download(job["fine_tuned_model"], "models", depth=8)

A platform model runs exactly like a local one, needle.Needle(tools=[...], weights="models/smart-home-8L.cact"), and its confidence is calibrated on your tools. Once a job is submitted it can also be followed on the dashboard, and the whole loop can be handed to Claude Code or Codex with your key and cactuscompute.com/llms.txt; the API reference lists every endpoint.

What a fine-tune does not change

The confidence head. Local fine-tuning does not train it, so needle build leaves it out of the archive and Needle(weights=...) reports confidence as None, with one warning. Route locally tuned models on your own validation instead of the score, or keep the base model for the decision and the tuned one for the call; Leveraging Needle's Confidence explains what the score measures and how to route on it. Fine-tunes from the platform train the head with the model, and their confidence is calibrated on your tools.

The tokenizer. Non-English text fragments into roughly 1.7 times more tokens (measured on Spanish), which taxes both quality and the context budget.

The bits. Local fine-tuning trains and exports at 4 bits. The 2-bit post-training and quantisation behind the shipped model run on the platform, so a platform model comes back at the shipped size.

Troubleshooting

1. failed to load weights: the .cact format is tied to the engine version, so an archive exported by an older package will not load. Rebuild it with the current package.
2. Loss hovers at its starting value: the run is undertrained, not broken. See reading the loss.
3. The tuned model calls a tool on everything: the dataset has no [] examples. Add them.
4. Correct tools, wrong values: more examples with reasoning lines, more varied values, then rank 32.
5. platform error unauthorized: NEEDLE_API_KEY is missing or revoked; create a key in the console. quota_exceeded means the period's runs are spent, job_active that a job is already running; both say what to do in the message.

Where to go next

• Needle Python Docs: the API, the response shape, system facts, tool retrieval, offline devices.
• How to Design Tools for Needle 3: the schemas the model does best on, and the triggers.
• Leveraging Needle's Confidence: acting, confirming or refusing on the score.
• Structured JSON Extraction with Needle: extraction and classification with a record as the tool.
• What Devices Are Supported on Needle: every platform folder, the CLI runner, the C API, the browser.
• Intelligence Ladders: why every depth of one file is a model, and how a fine-tune is sliced.
• The .cact Format and Porting Needle 3: the file you ship and how to write your own runtime for it.