Teaching AI to See: A Technical Deep-Dive on Vision Language Models with Will Hardman of Veratai

Summary

• Vision-language models are becoming a practical platform layer even if multimodality’s necessity for AGI remains unresolved. Hardman points to medical assistance, insurance-claim verification, content moderation, document indexing and product catalogs, while robotics eventually demands vision alongside audio, touch and radar. The research bet is that integrating modalities might produce “another kind of transformational leap” in world understanding, though strong text-only reasoning means that conclusion is not yet proven.
• The winning architecture is surprisingly modular: encode images, connect them to a language model, and let the language model perform the reasoning. Flamingo added trainable cross-attention and a 64-token Perceiver Resampler; LLaVA showed that a comparatively simple projection can instead inject continuous visual embeddings directly into the decoder stream. Labenz frames this as “Frankensteining” across modalities: frozen systems can be connected through learned bridges without breaking the language model.
• Training data, especially its sequencing and quality, increasingly looks more defensible than any single connector design. Flamingo established the importance of interleaved web documents; LLaVA used GPT-4 and COCO metadata to synthesize roughly 150,000 instruction examples; Qwen-VL moved from broad, lower-quality alignment data to higher-resolution multitask pre-training and then supervised tuning. The recurring recipe is “going up the quality scale, down the size, and unfreezing more parameters as we go.”
• Chinese open models are highly competitive with proprietary Western systems on vision-language benchmarks. Shanghai’s InternVL 2.5, Alibaba’s Qwen2-VL and ByteDance’s LLaVA-OneVision sit beside or above several proprietary systems on cited benchmarks. InternVL’s progressive scaling reached a similar level of loss with roughly 120 billion tokens versus about 1.4 trillion for the cited Qwen2-VL comparison.
• Decoder-only fusion generally maximizes reasoning and OCR, cross-attention preserves efficiency and can protect language capabilities, and NVIDIA’s hybrid suggests the end state may use both. NVLM fed a global thumbnail into self-attention while reserving high-resolution tiles for cross-attention, beating both pure variants by roughly one point on the MMMU validation split. Hardman’s verdict remains appropriately hedged: the result is interesting, but it is too early to know whether the hybrid is generally superior.
• Benchmark gains in reasoning conceal large gaps in basic perception. MMMU rose from GPT-4V’s roughly 55% in November 2023 to a reported o1 score near 78%, yet BLINK found GPT-4 Turbo at only 32.7% on visual IQ problems against roughly 80% for humans; counting and illumination-adjusted surface color also remain weak. “Reasoning can compensate for some of these perceptual deficiencies,” but applications still need task-specific evaluation and carefully structured prompts.
• Transfusion offers a promising path from image understanding to genuinely multimodal generation and editing. Meta’s research combines continuous image representations, diffusion loss and text next-token prediction inside one Transformer, reporting comparable image quality at about one-third the compute of earlier quantized-token approaches and text losses at half the FLOPs. Its precise edits—remove an object, change a color, preserve the scene—show why unified multimodal systems could improve on pipelines that merely hand prompts to separate diffusion models.
• There is no universal best model; selection should follow the workload and benchmark. At the episode’s December 20, 2024 checkpoint, o1 was reported at roughly 78% on MMMU, Qwen2-VL reached 96.5% on DocVQA, InternVL 2.5 slightly led the cited BLINK results at 63.8%, Gemini 2.0 Flash scored around 70.7% on MMMU in the smaller-model class, and 4-billion-parameter Phi-3.5 Vision reached 58.3% on BLINK. The durable implication is that model choice depends heavily on the task, data and evaluation used.

Deep dive

1. Multimodality is useful before anyone proves it is necessary for AGI

• Hardman’s practical case begins with workloads that inherently combine evidence: medical images plus patient history, social-media imagery plus text, product photographs plus catalog descriptions, and vehicle-damage photographs checked against an insurance report. These are not merely captioning tasks; the model must decide whether visual evidence supports a contextual claim.

• The longer-term engineering argument is composability. Learning to integrate images and language is a first step toward systems that also combine audio, touch and radar—modalities a robot would need to cook a meal, handle objects and respond to a changing physical environment.

• Hardman preserves the philosophical uncertainty: frontier language models already display abstraction, world models and sophisticated reasoning without obvious evidence of a ceiling. Multimodality therefore might not be required for AGI, but it could produce “another kind of transformational leap” in a system’s ability to understand and reason about the world.

• Labenz expects the question to remain counterfactual: multimodal systems are such an obvious “path of least resistance” that researchers may never learn whether a purely linguistic AGI could have worked. He expects multimodality to become the norm “sooner rather than later.”

2. Vision Transformers turned an image into a sequence a Transformer could consume

• The 2020 paper “An Image Is Worth 16x16 Words” replaced the prevailing stack of convolutional filters with a simple recipe: split an image into non-overlapping patches, linearize each patch, map it to a continuous embedding, and send the resulting sequence through a Transformer encoder with full attention.

• Unlike causal language attention, every image patch can attend to every other patch because the image represents one simultaneous scene. The original model prepended a classification token and trained it to identify the image category, in a setup Hardman compares to BERT-style classification.

• A 224x224 image divided into 16x16 patches produces 196 visual tokens. Downstream VLMs usually retain the entire sequence of hidden states rather than only the classification vector, making the representation richer but also creating a substantial token and attention cost.

• Model notation carries useful design information: ViT-H/16 means a “huge” Vision Transformer—roughly 600 million parameters—with 16x16 patches; ViT-G denotes a “giant” model larger than huge. Resolution and patch size jointly determine how much visual detail survives.

3. Visual “tokens” are continuous representations, not a fixed vocabulary

• Labenz stresses the contrast with text tokenization. A text tokenizer selects from a fixed vocabulary—perhaps around 100,000 entries—and each code can be mapped back to a fragment of text; visual patches instead pass through a learned matrix and emerge as vectors on a continuum.

• Nothing in Transformer attention requires those inputs to be discrete. The model can operate directly on continuous vectors, even though researchers loosely call them visual tokens. In the classification setup described, no image codebook is needed.

• This distinction makes the model’s latent input space much larger than the set reached by ordinary text embeddings. Later connector architectures exploit precisely that freedom: an image projection can place vectors in regions of the language model’s embedding space that no vocabulary token itself can occupy.

• Early systems nevertheless imposed severe preprocessing. Images were resized—sometimes distorted—to a fixed 224x224 square, yet produced surprisingly capable results. Labenz notes that modern APIs obscure this burden from developers, while leading models now preserve much more high-resolution and aspect-ratio information.

4. CLIP made language-aligned vision encoders reusable infrastructure

• OpenAI’s 2021 CLIP paired a vision encoder with a text encoder and jointly trained them on web-scraped image-caption pairs. For each batch of N correct pairs, the contrastive objective rewards similarity for those N matches and penalizes similarity across the N²−N mismatches.

• The mechanism pulls an image and its caption together in embedding space while pushing unrelated combinations apart. Once trained, CLIP supports open-ended image retrieval and classification, but its more consequential legacy is a Vision Transformer already partly aligned with language.

• Researchers can detach a CLIP-trained Vision Transformer and connect it to a generative language model, beginning with visual representations that already encode many linguistically relevant concepts.

• That changed the problem from recognizing one of ImageNet’s fixed classes to matching arbitrary natural-language descriptions. For application developers, the basic procedure became almost comically simple: encode the query and candidate images, calculate cosine similarity, and sort.

5. CLIP’s noisy web supervision exposed why data quality became central

• Labenz recalls using CLIP to choose imagery for small-business videos. It could retrieve semantically relevant pictures, but a textual occurrence of “pizza” inside an image might dominate a query for pizza, revealing artifacts inherited from noisy captions, jokes, poems and weakly related descriptions.

• Aesthetics were an even larger blind spot. CLIP could identify content but carried little signal about whether a photograph was beautiful, ugly, professional or embarrassing—an economically critical distinction when a business’s Facebook photos range from polished marketing to low-quality user uploads.

• Hardman sees the field’s later “increasing obsession with filtering data for quality” as a direct response. Web-scale volume generated genuine signal, but noisy correspondence between image and text capped performance, especially for models expected to reason over rather than merely retrieve visual content.

• The lesson carries through every subsequent training recipe: data must preserve a reliable relationship between what the pixels contain and what the language says. Synthetic captions, deduplication, safety filtering and task-specific augmentation are attempts to manufacture that correspondence at scale.

6. Flamingo established the modern VLM pattern through cross-attention

• DeepMind’s Flamingo—described as a “GPT-3 moment” for VLMs—paired a visual encoder with a Chinchilla language-model backbone. Rather than retraining the whole language model, it inserted new cross-attention layers roughly between every fourth existing Transformer block.

• In cross-attention, language-derived queries retrieve keys and values from encoded images. The language model remains responsible for next-token reasoning, while the new layers learn where visual evidence should alter its output; most of the pretrained backbone can stay frozen.

• Special image markers in an interleaved prompt are not ordinary tokens to predict. They tell the language model when to retrieve visual context, giving it enough information to generate the next textual token after an image.

• Flamingo demonstrated competitive visual question answering, OCR and captioning, including few-shot behavior, without separate task-specific models. Its efficiency and task breadth made it foundational, although DeepMind did not make the model broadly available.

7. The Perceiver Resampler made arbitrary image inputs computationally manageable

• Cross-attention creates a dimensionality problem when prompts contain different numbers or sizes of images. Flamingo’s Perceiver Resampler solved it with 64 learned latent queries that attend across however many visual tokens arrive, always returning 64 visual vectors.

• Standard attention forms a sequence-length-squared matrix because every query meets every key. Replacing sequence-length queries with 64 learned queries changes that expensive dimension to sequence length by 64; Hardman’s illustrative output is 64 vectors of perhaps 768 dimensions.

• Labenz calls the result evidence of how “malleable all of these latent spaces are.” An arbitrary visual sequence can be reshaped into a fixed-size interface, then consumed by a frozen language model that was never trained on images and does not otherwise “know” their origin.

• Hardman remains amazed that it works: the operation feels like “tipping an image into a blender,” yet training preserves useful features. Compression can also affect fine spatial structure, an issue later associated with weaker OCR.

8. Interleaved documents supplied the context that caption pairs could not

• Flamingo’s other major contribution was showing that isolated image-alt-text pairs were insufficient. Web documents preserve approximate order among paragraphs and images in the document object model, letting a model learn from richer sequences where visual and linguistic context alternate.

• During next-token training, image markers trigger retrieval of the corresponding visual representation and text continues around them. This better resembles multimodal documents than one image plus one short caption.

• Hardman says interleaved data was extremely important, particularly for few-shot learning. Later ablations by Hugging Face found a dramatic performance reduction when interleaved image-text documents were removed from otherwise comparable training.

• The finding also explains why emerging corpora containing HTML, PDFs and academic documents matter. They preserve relationships across modalities rather than merely one shallow labeling relationship.

9. LLaVA proved direct visual-token injection could rival a bespoke fusion stack

• LLaVA chose an autoregressive, decoder-only architecture rather than Flamingo’s repeated cross-attention. A learned projection maps the Vision Transformer’s continuous outputs into the language model’s embedding space, then prepends those visual vectors directly to the text sequence.

• From that point onward, the backbone treats image and text representations as one causal stream. Labenz’s memorable framing is that the system has converted images into “text embedding space,” even though the resulting vectors can occupy positions that no actual text token reaches.

• This is architecturally simpler than cross-attention but computationally costly: every visual token must be unrolled through the decoder’s attention. A projection also introduces fewer trainable parameters, so after exhausting its alignment capacity, improving performance may require unfreezing the backbone.

• Unfreezing creates the risk of catastrophic forgetting. Hugging Face’s comparison, as cited by Hardman, estimated that cross-attention could introduce about 25% trainable parameters in a 7-billion-parameter-class model, versus roughly 10% for the simpler projection route.

10. LLaVA’s real breakthrough was synthetic visual instruction tuning

• LLaVA began with Microsoft COCO, roughly 200,000 images containing descriptions and bounding boxes. The team gave that structured textual information—not the image itself—to GPT-4 and asked it to invent a conversation as though it could see the scene.

• Bounding boxes let GPT-4 formulate spatial questions such as “What is to the left of the car?” Captions support factual questions, while the combined scene description enables higher-order prompts such as what difficulty people loading luggage into a vehicle might face.

• Because GPT-4 generated both question and answer, the pipeline produced supervision without manually authoring every dialogue. The resulting data covered multiround conversation, region-specific grounding and visual reasoning rather than only generic captioning.

• Training used approximately 600,000 image-text pairs for initial alignment, then around 150,000 richer synthetic instruction examples. The resulting model outperformed contemporaries on complex reasoning and slightly improved conversational tasks, validating task diversity as a major capability driver.

11. Synthetic data changed annotation economics for multimodal applications

• Labenz connects LLaVA’s recipe to his own work extracting on-screen text and duration from video. Rather than building an annotation operation, he manually solved roughly 10 examples, fine-tuned a model, let it attempt the next 100, corrected failures and iterated.

• The task was awkward for humans and infrastructure-heavy to outsource, yet this process could bootstrap a new narrow capability in days. That experience convinced him that frontier models would help developers route around natural data walls across many unfamiliar tasks.

• Hardman gives parallel multimodal examples: generate corrupted synthetic OCR images with known ground truth, or generate LaTeX documents from known text to create paired visual and linguistic examples. The crucial constraint is not merely data volume but trustworthy cross-modal correspondence.

• The highest-quality instruction data remains expensive and often proprietary. Without a frontier model such as GPT-4 to generate it, producing the required volume and quality would be difficult.

12. MMMU separates perception, knowledge and reasoning across expert domains

• Massive Multi-discipline Multimodal Understanding, or MMMU, contains roughly 11,000 questions across about 30 fields, including medicine, history, electronics, market research and music. Students from different disciplines sourced problems from textbooks, lectures and online materials that require domain expertise.

• Questions may ask which musical score has an incorrectly constructed harmonic interval or require mathematical reasoning. Each offers four choices, making random performance 25%, while successful answers require seeing the diagram, knowing the subject and reasoning through it.

• At MMMU’s November 2023 release, GPT-4V scored about 55%; the strongest cited open model, a LLaVA variant, scored around 34%. By the episode’s December 20, 2024 checkpoint, o1 was reported at roughly 78%, around eight points above the next model in Hardman’s comparison.

• A revealing baseline gave text-only GPT-4 OCR output or LLaVA-generated captions instead of raw images; it still reached about 34%. That result shows reasoning contributes heavily, while the remaining gap depends on extracting and using visual information accurately.

13. Qwen-VL turned multimodal pre-training into a staged curriculum

• Alibaba’s Qwen-VL family uses a Qwen language backbone, a Vision Transformer and a connector with learned queries that compresses visual tokens before placing them in an autoregressive decoder. Its more lasting contribution is a three-stage training curriculum.

• Stage one uses image-caption and interleaved corpora while freezing the language model. Images are resized to 224x224, and only the vision encoder and connector learn the initial alignment, insulating the already capable text model from noisy visual supervision.

• Stage two raises resolution to 448x448—effectively four tiles—unfreezes the whole model and introduces multitask material: synthetic OCR, bounding-box grounding, visual and document question answering, and text-only examples that preserve language capabilities.

• A smaller supervised fine-tuning stage then uses the most carefully constructed prompts and image augmentations. Labenz’s summary captures the pattern: first get the components “on the same page,” then let them co-adapt end to end, while continuing to mix ordinary text so the general-purpose model does not overfit vision.

14. Open multimodal data scaled from captions into trillion-token documents

• LAION supplied roughly 5.8 billion quality-filtered image-caption pairs. Recent interleaved corpora changed the scale available to open researchers. MINT-1T, associated with a multi-contributor team including Salesforce, contains about one trillion tokens across HTML, PDFs and research papers.

• Shanghai’s OpenGVLab released OmniCorpus with approximately 2.2 billion documents, 8 billion images and 1.6 trillion text tokens. These corpora capture relationships among images and passages, closer to the documents that production systems must interpret.

• Hardman contrasts InternVL 2.5’s cited 120-billion-token run with Qwen2-VL’s roughly 1.5-trillion-token training and notes that OpenGVLab had released data much larger than its reported training run. His conclusion—“scaling is a hell of a drug”—is both technical forecast and competitive warning.

• This scale helps explain rapid improvement among open models. Better public recipes and corpora diffuse quickly, while teams incorporate lessons about quality filtering, tiling and instruction diversity.

15. InternVL scaled the vision encoder instead of treating it as an accessory

• OpenGVLab’s first InternVL questioned the convention of attaching a comparatively small, separately trained CLIP encoder to a much larger decoder. It built a fresh Vision Transformer with roughly 6 billion parameters, far above the approximately 600-million-parameter ViT-H and the roughly 1.8-billion-parameter largest standard Vision Transformers cited by Hardman.

• The team contrastively trained that encoder with a frozen 7-billion-parameter Llama model. Text and images were embedded, matching pairs pulled together, and mismatches pushed apart, while the language side resembled the generative decoder that the vision encoder would eventually serve.

• After training, the original Llama model could be discarded and the vision encoder connected to another language model. The result suggests that large, carefully language-aligned visual encoders can be connected to different backbones.

• The broader implication is that VLM performance does not depend exclusively on enlarging the language model. Still, later controlled experiments found that, under a fixed total parameter budget, scaling the language backbone generally produced more gain than allocating the same parameters to vision.

16. Dynamic high resolution preserved both local detail and global context

• InternVL 1.5 introduced “dynamic high resolution”: choose a tiling arrangement that approximates the image’s native aspect ratio, split it into 448x448 regions, encode every region separately, and append a thumbnail of the complete image.

• Local tiles preserve text and fine objects; the thumbnail restores the global composition that independent cropping could destroy.

• Because tiling creates many tokens, InternVL applies pixel shuffle—reshaping information from spatial width and height into the depth dimension. Hardman describes this as storing more information in depth while producing fewer spatial tokens.

• The constraint has shifted from forcing every image into one tiny square to deciding how much compute to spend preserving detail. This is particularly relevant to high-resolution tasks such as OCR and document understanding.

17. Progressive scaling cut InternVL’s data and compute requirements sharply

• InternVL 2.5 aligned its vision encoder first with a smaller language model, trained until the process reached a plateau, then swapped in progressively larger backbones and continued. The final family ranged from roughly 7 billion parameters to a 78-billion-parameter flagship.

• Hardman’s intuition is optimization rather than magic. A smaller model has fewer free parameters and finds a useful basin faster; once the vision encoder has learned the basic correspondence, the larger model starts from a good region and spends capacity on more complex understanding.

• OpenGVLab reported reaching a similar level of loss with about 120 billion training tokens, versus approximately 1.4 trillion for the cited Qwen2-VL comparison. The savings compound because the training stages use smaller backbones before the largest one is introduced.

• InternVL 2.5’s 78-billion-parameter model then ranked just below o1 on MMMU and ahead of GPT-4o, Claude 3.5 Sonnet and Gemini 1.5 Pro on that benchmark.

18. Llama 3.2 Vision showed cross-attention is still competitive

• Meta’s Llama 3.2 Vision departed from the decoder-only trend. It used a ViT-H/14, modified the Vision Transformer in ways not fully detailed in the technical report, and added new cross-attention layers to the language model while freezing the rest of Llama during multimodal pre-training and supervised fine-tuning.

• Meta could therefore train substantial new multimodal capacity without perturbing the language model’s existing self-attention or sacrificing its text-only behavior.

• The data pipeline included OCR generation, tables, LaTeX documents, captions, safety filtering, deduplication and quality filtering. Meta also applied DPO, one of the relatively few explicit examples in the survey of preference optimization after visual instruction tuning.

• The 90-billion-parameter version ranked second among the cited open models on MMMU, behind InternVL 2.5. That result shows cross-attention is not obsolete; it may be preferable when preserving a mature language backbone matters more than maximizing fine-grained OCR.

19. Controlled comparisons expose the real cross-attention trade-off

• Hugging Face’s Idefics studies kept the decoder and Vision Transformer fixed while changing the connection strategy. With the backbone frozen, cross-attention performed better because it contributed many more trainable parameters than a simple decoder-side projection.

• When researchers updated the language backbone, the autoregressive architecture could perform much better. Full attention updates were unstable, but low-rank adapters made training workable and let the language model adapt.

• Scaling either component helped, although increasing the language model delivered more value under a fixed parameter budget. A Perceiver-style resampler accelerated training in either architecture by reducing tokens, but did not necessarily improve final quality.

• A recurring weakness was detail: compressive resampling may shuffle visual tokens, hurting OCR. Decoder-only fusion instead places image and language tokens in one stream, which some authors speculate supports better multimodal reasoning.

20. NVIDIA’s NVLM hybrid captured much of both architectures’ upside

• NVIDIA paired Qwen’s language model with the InternVL Vision Transformer and trained a decoder-only version, a cross-attention version and a hybrid. The setup made the architectural comparison informative.

• Decoder-only NVLM performed best on multimodal understanding, reasoning over images and OCR, but cross-attention was more efficient to train because it did not have to unroll the entire sequence of image tokens in the decoder.

• The hybrid placed the thumbnail directly in the decoder stream while exposing high-resolution tiles through cross-attention. The language model could reason globally over the image and use cross-attention to retrieve high-resolution detail.

• It still trailed the decoder-only version on some OCR tasks, but beat both pure variants by roughly one percentage point on the MMMU validation set. Hardman calls the result promising rather than decisive: it is unclear whether the hybrid is generally superior.

21. Visual training can improve rather than merely preserve language reasoning

• NVIDIA tested text-only MMLU, math, HumanEval and other benchmarks after multimodal training. LLaVA and InternVL models showed some degradation relative to their original language backbones, while frozen-backbone Llama 3 Vision avoided that loss.

• NVLM went further: with a high-quality text-only supervised mixture alongside multimodal data, it improved over its original backbone on text benchmarks. The gain was strongest in mathematics, where the visual corpus contained many diagrams and geometry questions.

• The authors believe that solving mathematical problems presented as images strengthened general mathematical reasoning, which then transferred back to text. Hardman presents this as evidence that multimodal data can benefit capabilities outside the visual modality.

• This does not prove grounded multimodality is necessary for AGI, but it does show that adding another modality can improve capabilities outside that modality.

22. Early and late fusion are a continuum, not settled categories

• Most systems covered are late-fusion designs: vision and language are encoded separately, then joined through projection, self-attention or cross-attention. Even CLIP begins with distinct encoders before aligning their final representation spaces.

• Very early fusion would use one representation process and token space from the beginning. Hardman warns that the terminology lacks a universally settled boundary; the Apple approach aligns modalities earlier while retaining a distinct Vision Transformer.

• NVLM already demonstrates multiple integration points: one visual representation joins text in the decoder, while another is retrieved through cross-attention. Hardman expects future architectures to combine early and late mechanisms rather than commit to a single fusion moment.

• Labenz sees modular fusion as especially plausible for biological sequences, weather or other specialist modalities. Separately competent models could communicate through learned bridges before anyone bears the cost of training one model on every modality at once.

23. VQA, DocVQA and The Cauldron map benchmarks to real workloads

• The original VQA dataset contains roughly 50,000 COCO-derived images and about one million open-ended questions produced with Mechanical Turk. Its questions require visual recognition, language and common sense—for example identifying what a banana “mustache” is made from rather than merely noticing yellow pixels.

• DocVQA contains around 50,000 questions over roughly 12,000 images extracted from industry documents: scans, invoices, tables, charts, business infographics and handwritten notes. Answers are often precise text spans, making it directly relevant to document-extraction products.

• Hugging Face bundled around 50 visual fine-tuning datasets into “The Cauldron.” Hardman recommends finding the subset closest to a production task, then studying its image augmentations and prompt structure before creating a custom dataset or settling on an inference-time prompt.

• The practical lesson is benchmark specificity. MMMU asks whether a model can reason with expert knowledge; VQA probes common objects and relationships; DocVQA measures document extraction. A high aggregate VLM reputation does not substitute for matching evaluation to the user’s actual task.

24. BLINK reveals perception gaps that reasoning-heavy benchmarks obscure

• BLINK contains just under 4,000 multiple-choice questions across 14 perceptual tasks that humans can solve “in a blink.” Human performance is generally in the mid-90s, while chance is a little above one-third overall because of the number of answer options.

• Its authors argue that many MMMU questions can be reduced to dense captioning plus language reasoning. BLINK instead tries to isolate visual abilities for which replacing the image with an excellent textual description would give away the answer.

• Art-style matching was among the best-solved categories: GPT-4o reached about 83%, against roughly 95% for humans. Visual similarity also performed reasonably well, with GPT-4 Turbo around 80% versus approximately 97% for people.

• The benchmark therefore exposes a distinction between reasoning over a rich description and directly extracting perceptual structure from an image.

25. Visual IQ tests expose missing perceptual priors

• BLINK’s IQ category presents sequences of shapes and asks which option completes the transformation. Humans score roughly four-fifths, random selection gives 25%, and the cited GPT-4 Turbo result was only 32.7%—barely above chance.

• Hardman compares the task to ARC: humans quickly constrain the possible rule using perceptual priors, whereas a model may need an expensive search over candidate transformations. There is something about the perceptual features humans naturally extract that differs from the model’s representation.

• Labenz found earlier VLMs weak even at describing ARC images: they could miscount squares or misstate grid dimensions before attempting the transformation.

• Hardman expects additional inference-time reasoning to improve IQ-task scores by brute force, but that misses BLINK’s point. A model that searches laboriously after imperfect perception is functionally different from a person whose visual system makes the correct structure immediately salient.

26. Counting and illumination remain deceptively hard visual primitives

• Relative reflectance asks which of two marked surfaces is inherently darker despite shadows or lighting. Humans infer that a cream-colored headboard remains darker than a white wall even when the illuminated pixels suggest otherwise; the best VLM cited, a LLaVA model, scored just under 40%, while humans scored around 95%.

• Counting is similarly unstable. The cited results had GPT-4V around 60% and GPT-4o around 49%, an unusual regression between models.

• A DeepMind analysis found numerical statements scarce in common image-caption corpora and potentially too sparse to bind numbers reliably to visual features. Hardman therefore suspects counting could improve through programmatically generated examples with explicit instance counts.

• Prompted decomposition already helps: ask a model to inspect regions, enumerate what it sees and then total the objects. This supports Hardman’s distinction that “reasoning can compensate for some of these perceptual deficiencies,” while leaving the underlying perceptual weakness intact.

27. A generative objective may strengthen vision encoders

• The Apple team asked whether CLIP-style contrastive training itself limits visual representations. They jointly trained a fresh Vision Transformer and decoder on image-caption pairs, placing visual tokens before text and randomly masking some portion of the image sequence.

• The decoder reconstructed missing continuous visual tokens with mean-squared-error loss, then generated caption tokens with ordinary cross-entropy. Training used prefix attention, and the model was trained from scratch on the paired data.

• After pre-training, the team discarded the decoder, attached the Vision Transformer to Llama 3 through a simple multilayer perceptron, and fine-tuned on the LLaVA supervised mixture. Against a similarly sized contrastively trained encoder, the generatively pretrained model improved every reported VLM benchmark, especially captioning and visual question answering.

• The result suggests that the contrastive objective may be limiting the Vision Transformer and that a generative objective may produce a stronger visual encoder.

28. Transfusion unifies text prediction and image diffusion inside one Transformer

• Meta’s Transfusion pre-trains a single Transformer on an even mix of text and images while preserving modality-appropriate objectives. Text remains discrete and uses next-token cross-entropy; images stay continuous and use diffusion loss instead of being forced into a shared codebook.

• Image inputs pass through a variational autoencoder and an MLP or U-Net downsampling path before entering the Transformer. Generated image representations pass through the corresponding U-Net upsampling path and VAE decoder, making the architecture resemble “a latent diffusion model split in half” with a Transformer inserted centrally.

• Attention masks also differ by modality. Text is causal; patches within one image use bidirectional attention, but they can attend only to earlier text or earlier images outside that image block. The model must therefore route outputs through the appropriate objective and decoding path.

• Earlier models such as Chameleon quantized a 512x512 image into 1,024 discrete tokens from an 8,192-entry codebook—roughly 16x16 patches. Transfusion removes that image-token quantization in favor of continuous representations.

29. Transfusion’s efficiency and editing quality point toward true multimodality

• Compared with earlier quantized-token approaches, Transfusion reportedly reached similar image quality with about one-third as many FLOPs and matched text training losses at roughly half the FLOPs. Its 7-billion-parameter model outperformed DALL-E 2 and Stable Diffusion XL on the cited image-generation evaluations while reaching roughly LLaMA 1 performance on text-only tasks.

• The especially compelling outputs are localized edits: remove an item, replace one object, change a color, or convert graffiti on a truck into calligraphy without destroying the scene. Labenz sees the control missing from pipelines that perturb an input image and regenerate a loosely related composition.

• This matters for commercial imagery because businesses want realistic representations. Pure generation can create attractive scenes that bear little resemblance to the premises customers visit; precise editing can improve a real photograph while preserving its identity.

• GPT-4o’s original announcement also appeared to promise native image output, although at recording time OpenAI had not released that capability, reportedly because of safety and infrastructure concerns. Gemini and GPT-4o initially used systems that generated a textual prompt and handed it to Imagen or DALL-E.

30. Benchmark leadership is fragmented across models and model sizes

• On MMMU, Hardman’s December 2024 compilation placed o1 at roughly 78%. Claude 3.5 Sonnet New and InternVL 2.5 were around 70%, Gemini 1.5 Pro several points lower, and Grok 2 Beta roughly 66%; newer or missing results could change the order.

• On DocVQA, most cited frontier models exceeded 90%, and Qwen2-VL led available results at 96.5%, ahead of Grok 2, Gemini 1.5 and GPT-4o.

• On BLINK, the cited scores were InternVL 2.5 at 63.8%, GPT-4o at 63.2%, Gemini around 61%, and Claude 3.5 Sonnet New around 56.5%. The relatively low absolute scores reinforce that strong expert reasoning and strong basic perception are different capabilities.

• The open-model result complicates a simple geographic narrative. InternVL comes from Shanghai’s OpenGVLab, Qwen2-VL from Alibaba and LLaVA-OneVision from ByteDance; the cited Chinese open models were highly competitive with proprietary systems.

31. Small models already offer benchmark-specific production advantages

• Gemini 2.0 Flash led the cited smaller-model class on MMMU at around 70.7%; Grok 2 Mini was around 63.2%. For economical image reasoning, Flash therefore looked especially strong at that checkpoint.

• InternVL 2.5’s 8-billion-parameter version reached 95.1% on DocVQA—the same cited result as its larger version—and 54.8% on BLINK, ahead of GPT-4o Mini’s 51.9%. Scaling the backbone is not equally valuable for every visual task.

• The surprise in the smaller-model class was Microsoft’s 4-billion-parameter Phi-3.5 Vision at 58.3% on BLINK, despite weaker MMMU performance. It used roughly half a trillion pre-training tokens, then an unusually large 33-billion-token supervised dataset with a substantial in-house Microsoft component, followed by DPO.

• Hardman cannot attribute the result to one factor, but the combination suggests that post-training scale and perceptual task coverage can matter alongside raw parameter count.

32. The next gains will come from better fusion, post-training and task-specific evidence

• Hardman expects larger open VLMs, accelerated by progressive backbone scaling and trillion-token interleaved corpora. He also expects more experiments with generatively trained or redesigned vision encoders, because the standard contrastive Vision Transformer now appears less inevitable than it did.

• Fine-tuning data will continue to accumulate programmatic or human image augmentations for neglected tasks such as counting, OCR and spatial grounding. DPO appears only sparsely in the surveyed literature, leaving alignment and preference post-training comparatively underexplored.

• For builders, task-specific evaluation remains the governing reality. A sensible workflow is to choose the benchmark nearest the business task, prototype cheaply, inspect failures, study analogous Cauldron data, and only then invest in custom prompts, synthetic supervision or fine-tuning.

• Hardman describes effective AI consulting as arriving with “a prepared mind”: maintaining a searchable archive of papers and techniques so a client’s unfamiliar problem can be pattern-matched quickly. The apparent speed of a good proof of concept rests on accumulated technical depth, not a universal model recipe.