Optimized Multimodal Support
I gave a 1-billion-parameter language model that has never seen an image or heard a sound the ability to point at objects with bounding boxes and to say when sounds happen. I did not change a single one of its weights.
Keep the brain, borrow the eyes
Vision-language models are usually built by training a language model and an image encoder together for a long time. That is expensive, and it produces a new model. I wanted the opposite: take a text-only model people already run locally, leave it exactly as it is, and plug perception into it.
The plug is a connector, a small neural network that sits between an encoder borrowed from another model and the language model. The encoder is frozen. The language model is frozen. Only the connector learns. Its job is to translate the encoder's picture of the world into vectors that the language model already knows how to read as words.
Because llama.cpp already has a file format for exactly this slot, the mmproj.gguf, the result needs no new code. You load the stock Llama GGUF, point --mmproj at the connector file, and the model can see. Swap the file and it can hear.
Why a single matrix was not enough
This project started from a cheaper trick: fit one matrix, in closed form, that maps an image's average feature to a blend of word embeddings. It works for naming the main objects, with 52% top-1 label retrieval against 0.5% chance. But averaging throws away where things are, so it can never draw a box.
Before training anything I measured what image tokens look like from the language model's side.
When does a real vision-language model "name" what it sees?
Qwen3-VL-2B, 300 COCO images, 16,915 object tokens. The share of object tokens whose object name is among the top 10 words when that layer's state is decoded. Layer 0 is the projector output, the place my connector writes into.
The name of an object only surfaces 20 layers deep, and it surfaces on the object's own tokens. A frozen text model never learned that 20-layer translation, so the connector has to do it, token by token, before the first layer.
Words don't light up the object. Pictures do.
Each square is one Qwen3.8 image token, a 32×32-pixel patch. The second panel colours every token by how similar it is to Qwen's own word for the object. The third colours it by similarity to the average token of the same object in other images. Yellow is high.
Across 298 images, tokens of the same object resemble each other (prototype AUC about 0.96) but do not resemble the object's word (AUC 0.43, below chance). Image tokens carry what and where, in a code that is not made of words.
Teaching the connector on a laptop
The image connector has exactly the shape of Qwen's own patch merger: two linear layers with a GELU between them, 30.7 million parameters. Every 32×32-pixel patch becomes one input vector for Llama, so a 448×288 image becomes 14×9 = 126 tokens and the layout survives.
Training uses ordinary next-token loss on the answer, but the gradient flows through the frozen Llama into the connector. The connector learns whatever input makes Llama say the right thing. Boxes are plain text, [x1,y1,x2,y2] on a 0-999 scale, so nothing is added to Llama's vocabulary.
List the objects in this image with their bounding boxes.→person [437,83,785,900]
skis [294,822,915,941]Where is the stop sign? Answer with bounding boxes.→[322,123,700,463]Describe this image in one sentence.→A woman skiing down a snowy hill on skis.Everything ran on an RTX 3070 laptop GPU with 8 GB of memory, while the machine stayed in normal use. Every job ran in a memory-capped sandbox, with a watchdog that pauses training when free RAM drops below 750 MB or the GPU reaches 93 °C, and a supervisor that resumes from the last checkpoint. The audio run was paused and resumed more than a dozen times while I browsed the web, and it still finished.
Image connector
run1: 4,500 COCO images. Full: 118,287 images, starting from run1.
Audio connector
Flat for 15k examples, then the connector suddenly locks on.
Pointing at things
Six held-out COCO images, answered by llama.cpp running the unmodified Llama-3.2-1B GGUF with the trained mmproj. Hover a line of output to find its box.
To score it I asked "Where is the X?" about the largest object in each of 192 held-out images and checked whether the first box overlaps the true one by at least half (Acc@0.5). Two controls matter. A guess from the question alone, using the average box for that category, gets 5%. Feeding the connector a different image gets 4%. The boxes come from the image.
"Where is the X?" Acc@0.5
192 held-out COCO images, COCO categories.
By object size
Share of the image the object covers.
More data mattered most. Going from 4,500 to 118,000 training images lifted Acc@0.5 from 48% to 74% and nearly doubled box-level recall in lists, from 0.25 to 0.47. On large objects the connector reaches 89%, six points behind Qwen3-VL-2B, a model trained end to end for grounding. The gap is on small and medium objects, and in lists, where greedy decoding sometimes loops on the same box until it runs out of tokens.
Outside the 80 COCO categories
COCO has 80 object categories, and the connector only ever practiced those. To test anything else I used LVIS annotations on the same held-out images and asked about the largest object that is not a COCO category: curtains, blankets, cabinets, street signs, mirrors. That is 246 questions over 119 categories.
Acc@0.5, COCO questions vs non-COCO questions
Same images, same prompt. Qwen3-VL-2B is a reference trained end to end on far more grounding data.
Outside its training categories the connector mostly fails: 21% instead of 74%. The hits it does get are revealing. Blankets are found where a bed is, desks where a dining table is, street signs next to stop signs. It learned "the region of the COCO object closest to this word", not the word itself. Qwen3-VL-2B drops only from 91% to 85%, so these objects are visible to a vision tower of this kind. My best guess is that the limit is the training data (80 names, 3 prompt templates) rather than the frozen encoder or the frozen Llama. That is the next experiment.
Borrow the eyes, and a few layers of the brain
The connector works on one image token at a time. It cannot see that four neighbouring patches are the same dog, or where that dog ends. A real vision-language model sorts this out inside its language model, whose attention layers let every image token read every other one. So I borrowed those layers too.
The donor is Qwen3-VL-2B. I keep its vision tower and the first k of its 28 language-model layers, freeze all of it, and throw the rest away. A new connector, half the size of the first one, learns to translate the hidden states after layer k into Llama's input, with the same three tasks and the same loss as before. Llama is still untouched.
k is a choice, so I tried three values with short runs (32k images, 3,000 steps) and trained the plain connector on the same budget for comparison.
How many layers to copy
Short runs, 3,000 steps on 32k images. "Where is the X?" Acc@0.5 on the 192 COCO questions.
Training loss, same data and schedule
Both trained from scratch on all 118,287 images, 14,800 steps. The axis starts at 1.5.
With no layers at all (k = 0), the Qwen3-VL-2B tower does slightly worse than the Qwen3.8 tower, so the gain comes from the copied layers. Four layers beat twelve. Boxes need to know where each token is. In the first layers a token is still mostly about its own patch, while deeper layers mix tokens together and drift toward predicting the next word. I only tried three values, so the best k could be 2, 6 or 8.
Then the real test: both designs trained from scratch with identical data and schedule, one full pass over COCO train2017, evaluated on the same held-out questions as everything above.
Acc@0.5 after equal training
Same 192 COCO and 246 non-COCO (LVIS) questions as the earlier sections.
Where the two disagree
The plain connector trained from scratch lands at 74%, the same as the earlier full run, so the baseline holds up. Connector + donor is better on every metric I measured while training about half as many parameters, and on COCO it gets most of the way to Qwen3-VL-2B itself. It is the first of my models that does reasonably on small objects, 6 of 9, and its object lists repeat less (13% duplicate lines against 20%).
Two things stay the same. Objects outside COCO are still hard: 28% against Qwen3-VL-2B's 85%, so the 80 training categories remain the main limit. And this version does not run in llama.cpp yet. An mmproj file holds a vision tower and a projector, and llama.cpp has no way to run a language model's decoder layers inside it. For now connector + donor runs in PyTorch, and the downloadable mmproj is the plain connector.
Hearing, with timestamps
The same recipe works for sound. I borrowed OpenAI's Whisper-large-v3-turbo encoder, froze it, and trained a 46-million-parameter connector shaped like llama.cpp's built-in Ultravox projector. Eight 20 ms Whisper frames become one Llama token, about one token per 160 ms. The audio version of a bounding box is a time interval: dog [2.3s, 6.0s].
Speech
Trained on 100 hours of LibriSpeech audiobooks. These clips are from the held-out test set, transcribed by llama.cpp. Differences from the reference are highlighted.
Scenes with several sounds
For timestamps I built tracks from ESC-50 clips, 1 to 3 sounds at random times. Press play: yellow is when each sound really happens, orange is what the model listed. The test clips use sounds never heard in training.
"When can you hear X?" Acc@0.5
300 held-out mixtures, split by how many sounds the clip contains.
Finding events vs naming them
Share of true events whose time span the model listed (tIoU ≥ 0.5).
Speech is the strong result. The frozen text model transcribes at 5.3% word error rate, about twice the error of Whisper's own purpose-built decoder (2.6%), and gets 161 of 300 test sentences exactly right. Sounds are harder to name, at 66% accuracy over 50 classes, and the mistakes are between similar sounds: coughing and sneezing, frog and dog, pig and cow.
The timestamps tell the most interesting story. The model finds when something happens 87% of the time, but attaches the right name to the right span only 49% of the time. With one sound in the clip, "when can you hear X?" is right 96% of the time. With three it falls to 40%, usually because the model returns the span of a different sound.
What the connector does to a token
So where do the tokens end up after training? Drag the slider, or let it play. Each dot is one token. Coloured dots are tokens of a labelled object or sound, grey dots are background, small light dots are words, and stars mark each class's own word.
The connector does not turn tokens into words. After training, image tokens are on average 24 times longer than Llama's word vectors and audio tokens 96 times longer. They point in directions no word uses, and their nearest words are arbitrary ones like . Llama reads them anyway. Each of its layers normalises its input before using it, so length matters less than direction, and training found directions that the frozen layers respond to even though no word lives there.
There is still a faint pull toward meaning. Before the connector, an object's tokens are no closer to the object's word than the background is. After it, they are, modestly, and the effect holds across 119 objects.
How long the vectors are
Average length, log scale. Words are what each language model normally reads.
Do object tokens point to their word?
Within each image: how often an object's token is more similar to the object's word than a background token is. 50% is chance.
Token by token: similarity to the object's word
The same image, before and after. Stronger orange means more similar to the word. Yellow outlines mark the object's tokens. Each map is scaled to its own range.
Training budget
Everything ran on one RTX 3070 Laptop GPU with 8 GB of memory and a machine with 13.8 GB of RAM that stayed in everyday use. Llama and both encoders were frozen the whole time.
| Run | Data | Steps × batch | GPU time | Wall clock | Peak VRAM | Trained / frozen |
|---|---|---|---|---|---|---|
| Vision, run1 | 4.5k COCO images | 2,500 × 8 | 24 min | 24 min | 5.3 GB | 30.7M / 1.65B |
| Vision, full | 118k COCO images | 14,800 × 8 | 3 h 15 min | 3 h 15 min | 6.1 GB | 30.7M / 1.65B |
| Audio | 100 h speech, 1.6k sound clips | 8,000 × 8 | ≈3 h 15 min | 6 h 46 min | 6.2 GB | 46.1M / 1.87B |
| Connector + donor, k = 0 / 4 / 12 | 32k COCO images | 3 × 3,000 × 8 | 40 / 42 / 50 min | n/a | 6.3-6.7 GB | 16.8M / 1.65-2.25B |
| Connector, same short budget | 32k COCO images | 3,000 × 8 | 42 min | n/a | 6.0 GB | 30.7M / 1.65B |
| Equal footing, connector + donor | 118k COCO images | 14,800 × 8 | 3 h 32 min | 3 h 55 min | 5.7 GB | 16.8M / 1.85B |
| Equal footing, connector | 118k COCO images | 14,800 × 8 | 3 h 30 min | 3 h 30 min | 4.9 GB | 30.7M / 1.65B |
The first three runs took about 7 GPU-hours, and all runs together about 17, plus a few hours of evaluation. Wall clock is longer than GPU time when the watchdog paused a run because the browser and other apps pushed free RAM under 750 MB; the equal-footing connector + donor run was paused and resumed 7 times. The trained connectors are 1.8% (vision), 2.4% (audio) and 0.9% (connector + donor) of the parameters in each pipeline.
What it cannot do yet
Grounding was trained on 80 COCO categories and sounds on 50 ESC-50 classes. Outside them, accuracy drops from 74% to 21%, or from 86% to 28% with connector + donor.
Ask the vision model "What colour is her jacket?" and it answers with a box. Three prompt templates were not enough variety.
On busy scenes the object list can repeat one line over and over. 21% of list answers run into the 160-token limit, mostly for this reason.
With several sounds in a clip, the model often returns the right kind of span for the wrong sound.
The timestamp tests use artificial mixtures without overlap. Real recordings are harder.
192 to 300 questions per test, one training run each. Treat differences of a few points as noise.
Next: train on LVIS, RefCOCO and Visual Genome for open-vocabulary grounding, mix in ordinary visual questions, get connector + donor running in llama.cpp, and repeat everything with a second language model family to test whether any text model can be given these senses.
Run it yourself
You need a current llama.cpp build, the stock Llama-3.2-1B-Instruct GGUF, and one of the two mmproj files.
# see: boxes llama-server -m Llama-3.2-1B-Instruct-Q8_0.gguf \ --mmproj mmproj-qwen38-grounding-llama32-1b-full.gguf -c 4096 --image-max-tokens 256 # hear: transcripts and timestamps llama-server -m Llama-3.2-1B-Instruct-Q8_0.gguf \ --mmproj mmproj-whisper-audio-llama32-1b.gguf -c 4096
Resize images so the longest side is 448 pixels and both sides are multiples of 32, as in training. Ask "Where is the dog? Answer with bounding boxes." or, with a WAV file, "List the sounds in this audio with their timestamps."