# Mirage and text-only language models

An inference experiment by Henry Ndubuaku, run on 5 November 2026. The article is [Structured Visuo-Semantic Perception](https://cactuscompute.com/blog/structured-perception#mirage-visual-perception--llm).

Mirage describes each image once without seeing the questions. Qwen receives text only. Both models retain their released weights. The final sample contains 256 questions from 128 images, disjoint from the prompt-development images.

The original experiment uses Qwen3-0.6B. The follow-up below adds three larger text-only readers while keeping the questions, observations, prompt, answer contract and scorer fixed.

## Results

| Input to Qwen | VQA consensus score / 100 |
| --- | ---: |
| Question only | 9.34 |
| Scene fields | 28.13 |
| Structured record without trace | 33.09 |
| Structured record with complete trace | 31.25 |
| Mismatched record with complete trace | 22.23 |

The matched-minus-mismatched difference is 9.02 points, with an image-cluster bootstrap 95% interval of 4.88 to 13.24. The trace-minus-no-trace difference is −1.84 points, with an interval of −4.96 to 1.17.

These scores include exact answer production. A complete sentence can receive zero even if its meaning agrees with a short reference. JSON validity alone does not ensure a short answer. No semantic judge, response repair, or answer rewriting was used. All 1,280 final responses parsed successfully; one question-only reply reached the 64-token limit and was retained.

The sample is exploratory. Training overlap was not audited, and this is neither a full-benchmark score nor a comparison to a native VLM. Scene fields are not a prose baseline containing the same information as the complete record.

## Larger text-only readers

We next tried Qwen3-1.7B and NVIDIA Nemotron 3 Nano 4B. After observing format failures with Qwen3-1.7B, we added Qwen3-4B-Instruct-2507. All three used their original BF16 weights through MLX, with thinking disabled, greedy decoding and the same 64-token limit. No model received pixels or image embeddings, and no weights were changed.

| Input | Qwen3-0.6B | Qwen3-1.7B | Nemotron 3 Nano 4B | Qwen3-4B Instruct |
| --- | ---: | ---: | ---: | ---: |
| Question only | 9.34 | 27.50 | 33.09 | 33.40 |
| Scene fields | 28.13 | 29.53 | 30.59 | 25.47 |
| Structured record without trace | 33.09 | 23.98 | 4.45 | 35.00 |
| Structured record with complete trace | 31.25 | 31.64 | 5.78 | 37.34 |
| Mismatched record with complete trace | 22.23 | 27.62 | 3.83 | 22.93 |

These remain VQA consensus scores out of 100, including strict answer-format compliance. They are not a ranking of the models' general knowledge or visual understanding.

Qwen3-4B Instruct improved on Qwen3-0.6B by 6.09 points with the trace, with a paired 95% image-bootstrap interval of 2.03 to 10.20. But the larger model already scored 33.40 without visual context. Its matched-record gain over that question-only control was 3.95 points, with an interval of −1.99 to 9.92. Its gain over the mismatched record was clearer, at 14.41 points with an interval of 10.08 to 18.95. Irrelevant context can harm a capable reader, so the mismatched comparison alone does not establish a benefit over using no visual record.

The other readers exposed a problem with the answer contract. Qwen3-1.7B produced 252 invalid-format responses out of 1,280, and Nemotron produced 713. Many were bare words, numbers or booleans instead of an object with a string-valued `answer`. All received zero credit under the frozen protocol. Both Qwen3-0.6B and Qwen3-4B Instruct produced valid JSON in all 1,280 cases. We did not wrap, rewrite or repair failed responses.

| Invalid-format responses / 256 | Qwen3-1.7B | Nemotron 3 Nano 4B |
| --- | ---: | ---: |
| Question only | 69 | 13 |
| Scene fields | 55 | 42 |
| Structured record without trace | 88 | 228 |
| Structured record with complete trace | 34 | 214 |
| Mismatched record with complete trace | 6 | 216 |

This follow-up did not establish a consistent benefit from increasing the reader's size. We therefore retain the original small-model example in the launch post and keep these additional results here, including the unsuccessful runs. A next experiment should separate answer-contract reliability from missing or incorrect visual facts, using development examples before evaluating on fresh images.

These are follow-up comparisons on the same sample, selected after inspecting earlier results. The intervals are nominal, without a multiple-comparison correction. They are not independent confirmations or evidence of unseen-training-data generalisation.

### Model selection

The requested [Qwen3.5-2B](https://huggingface.co/Qwen/Qwen3.5-2B) already has a vision encoder. [Nemotron 3.5 Bolt E4B](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB) is offered as a text-only GGUF, but descends from multimodal [Gemma 4 E4B](https://huggingface.co/google/gemma-4-E4B-it). Its BF16 configuration includes vision and audio components. Neither was used in this text-only comparison.

The selected checkpoints are

- [Qwen3-1.7B](https://huggingface.co/Qwen/Qwen3-1.7B/tree/70d244cc86ccca08cf5af4e1e306ecf908b1ad5e).
- [NVIDIA Nemotron 3 Nano 4B BF16](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16/tree/dfaf35de3e30f1867dd8dbc38a7fc9fb52d3914f).
- [Qwen3-4B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507/tree/cdbee75f17c01a7cc42f958dc650907174af0554).

Each has a text-only causal-language architecture. The runner checks the configuration and loaded parameter names for vision components. `qwen-4b-selection.json` records when and why the third checkpoint was added.

### Follow-up files and reproduction

`qwen-1.7b/`, `nemotron-4b/` and `qwen-4b/` each retain the frozen inputs, protocol, complete responses, scores, runtime metadata and generic format probes. `text-models.json` contains all four models and the paired cross-model intervals. `text-models-tracking.json` records the W&B run ID; the run stores code and machine provenance plus the complete evidence artifact.

`scale.py` uses the shared observation selection and scorer from `run.py`. For example, after installing the pinned dependencies, reproduce Qwen3-4B Instruct in a new output directory with

```bash
python scale.py --output rerun-qwen-4b \
  --model Qwen/Qwen3-4B-Instruct-2507 \
  --revision cdbee75f17c01a7cc42f958dc650907174af0554
```

The model and revision for either other reader are in its `protocol.json`. This reuses the recorded Mirage observations, so it isolates the text reader. Model weights are additional to Mirage's 22 MB. The recorded wall-clock times are incidental local stage measurements, not a controlled latency comparison between models.

`compare.py` verifies identical input hashes and complete prediction coverage before recomputing the cross-model intervals. Run it from this directory with `python compare.py`.

`reference-check.py` provides a separate Transformers CPU spot check using the same checkpoints and prompts. It requires `torch==2.15.0` in addition to the main requirements and loads full-precision weights, which requires substantial memory. Reference-runtime verification was interrupted and was not completed for this comparison. Treat the scores as recorded MLX results, without a claim of verified agreement across runtimes.

## Files

- `protocol.json` freezes the sample, conditions, prompt, model revisions and scoring method before final predictions.
- `questions.jsonl` and `references.jsonl` retain the selected question IDs and all ten human answers separately.
- `images.jsonl` records original image URLs and SHA-256 hashes.
- `observations.jsonl` retains Mirage's complete returned records and measured CPU call times. Masks and points were excluded through the documented API options.
- `predictions.jsonl` retains every Qwen raw response, parsed answer, prompt hash, token count, condition and timing.
- `scored.jsonl` and `summary.json` provide per-question scores, aggregates and paired confidence intervals.
- `runtime.json` records package versions, model and engine hashes, and hardware. Model loading and downloads are excluded from timings. Mirage used CPU; Qwen used the Metal GPU. The stages were timed separately.
- `run.py`, `ask.py` and `requirements.txt` reproduce the experiment and a single-image example.
- `format-check.json` records the three format checks on the separate release photo, before the final sample was run.

## Reproduce

Use Python 3.12 on Apple Silicon. Install the pinned dependencies in a fresh environment. The model download is roughly 1.2 GB for Qwen, plus Mirage and the engine.

```bash
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python ask.py photo.jpg "What drink is shown?"
```

To repeat the final sample without overwriting recorded evidence

```bash
python run.py prepare --output rerun
python run.py perceive --output rerun
python run.py answer --output rerun
python run.py score --output rerun
```

The image order is the SHA-256 ordering of `mirage-qwen-vqa-20261105:<image_id>`, restricted to validation images with at least two questions. The final run uses zero-based positions 128 through 255. Two questions within each image are chosen by the same hash rule applied to question IDs. Positions 0 through 127 belong to development. No image was dropped for poor predictions. The frozen question and reference hashes can be compared to verify selection.

The scorer downloads the pinned VQA reference implementation. Its only syntax conversion changes two Python 2 print statements. For the final score, its normalisation functions are applied to every prediction and reference, followed by the ten leave-one-annotator-out consensus calculation. The summary also retains the legacy evaluator's result, whose normalisation is conditional on reference disagreement. Bootstrap resampling uses images as clusters and retains their two questions.

## Development records

`coordinate-prompt/` contains the first run on the development sample. It used a longer system prompt with a coordinate definition and raw short answers. Qwen sometimes echoed the definition. Scores were 12.34, 21.68, 27.85, 28.87 and 24.26 in the condition order above.

`short-prompt/` retains a follow-up on the same development sample with a shorter system prompt. Qwen often returned sentences. Legacy exact-answer scores were 0.00, 0.78, 7.19, 9.34 and 8.32. Uniform normalisation alone did not resolve sentence-valued responses. These runs informed the final JSON answer interface. Their protocols, predictions and scores remain intact; they are not combined with the fresh comparison.

The final prompt was checked on the release photo and then run once on the disjoint final sample. This is prompt development followed by a fresh validation probe, not preregistration or proof of unseen-training-data generalisation.

## Sources

- [VQAv2 downloads and attribution](https://visualqa.org/download.html), Goyal and colleagues, *Making the V in VQA Matter*, CVPR 2017.
- [VQA evaluation](https://visualqa.org/evaluation.html) and [the pinned reference code](https://github.com/GT-Vision-Lab/VQA/tree/a013f0043c1e2cdc995922dfe257f7149aa9af06/PythonEvaluationTools).
- [COCO](https://cocodataset.org/#termsofuse). The three displayed photographs are COCO val2014 images 283, 162067 and 490624. Original image URLs and hashes are in the image manifest. Original image and annotation terms apply.
- [Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B/tree/c1899de289a04d12100db370d81485cdf75e47ca), original BF16 weights, Apache 2.0.
- [Mirage](https://huggingface.co/Cactus-Compute/mirage/tree/76c1e9e53381aca8fc3843c636216c96ff220122), released 2-bit `.cact` through `cactus-compute==4.0.1`.
- [Socratic Models](https://socraticmodels.github.io/) for prior work composing vision and language models through text.