Vision-Language Models

Defining and Describing Vision-Language Models

  • Vision-language models connect visual perception and language generation so machines can interpret images and respond in words.
  • A vision-language model (VLM) combines an image or video encoder with a language model, allowing the system to process visual inputs alongside text and produce language-based answers or actions. [1] [2] Typical capabilities include image captioning, visual question answering, instruction following, document analysis, and conversational image understanding. [1] [2]
  • Many modern VLMs use a modular architecture: a pretrained vision encoder produces visual tokens, an alignment or projection layer maps those tokens into the language model’s representation space, and an autoregressive language model generates the response. [1] [3]
  • VLMs apply when a task requires both perception and linguistic reasoning—for example, answering questions about a photograph, extracting information from a chart, interpreting a document, or discussing multiple images. [1] [2]
flowchart LR I["Image or video"] --> V["Vision encoder"] V --> P["Alignment projector"] P --> L["Language model"] T["Text prompt"] --> L L --> O["Text response"]

Uses in Context

  • Visual question answering: A user asks a question about an image, and the model identifies relevant objects, text, relationships, or events before answering. [1] [2]
  • Image conversation: VLMs support multi-turn dialogue in which users discuss an image through successive questions and instructions. [2]
  • Document and chart analysis: Models can interpret visual documents and return structured explanations or extracted information. [1]
  • Instruction following: Visual instruction tuning adapts models from simple captioning or question answering to more complex, conversational requests. [1] [3]
  • GUI and agent interfaces: The same visual-instruction capabilities can support interaction with interfaces and other environments requiring image-based reasoning. [1]

History of Use

Origins

  • The modern VLM lineage grew from academic work on joint vision-language pretraining, including models such as BLIP and BLIP-2, which sought more efficient alignment between image representations and language models. [4]
  • DeepMind’s Flamingo demonstrated few-shot visual-language learning by combining visual inputs with a language model, helping establish the pattern of adapting large language models to multimodal prompts. [5] [4]
  • The open-source LLaVA project, introduced by Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee in 2023, connected a CLIP vision encoder to a Vicuna/LLaMA language model through a learned projection and trained it on synthetic visual-instruction examples. [1] [3]
  • LLaVA’s authors described their work as “Visual Instruction Tuning,” using GPT-4-generated multimodal instruction data to teach a language model to follow conversational visual requests. [1] [5]

Evolution

  • 2022 — Few-shot multimodal prompting: Flamingo extended language-model-style few-shot learning to visual inputs, demonstrating that a model could generalize across image-and-text examples. [5] [4]
  • 2023 — Efficient modular alignment: BLIP-2 and related systems emphasized connecting frozen or pretrained vision and language components rather than training a complete multimodal model from scratch. [3] [4]
  • 2023 onward — Open visual assistants: LLaVA popularized a lightweight, reproducible recipe based on a CLIP encoder, a projection layer, a language model, and synthetic instruction data; later open-source projects expanded this direction to broader multimodal benchmarks and model families. [1] [6] [7] [3]

Best Real-World Examples

  • LLaVA — Open-source visual assistant combining CLIP-style visual encoding with a large language model for image conversation and instruction following. [1] [2] [3]
  • BLIP-2 — A vision-language system designed to bridge pretrained image encoders and large language models efficiently. [3] [4]
  • Flamingo — A research model demonstrating few-shot learning from interleaved visual and textual inputs. [5] [4]
  • LLaVA-Gemma — An open multimodal model family combining the LLaVA framework with Gemma language models. [6]
  • LLaVA-OneVision-2 — An open-source successor in the LLaVA-OneVision line targeting stronger performance across multimodal benchmarks. [8] [7]
  • LAVIS — An open framework for training and evaluating vision-language models, including BLIP-family systems. [4]
  • LLaVA-VLA — A research direction that extends a VLM backbone toward vision-language-action control for robotic systems. [9]

Case Studies

LLaVa and the open visual-assistant recipe. In 2023, Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee introduced LLaVA as an attempt to make language models follow visual instructions rather than merely caption images. [1] [5] The system used a frozen CLIP ViT-L/14 vision encoder, a trainable linear projection, and a Vicuna/LLaMA language model. [1] [3] Its training pipeline used GPT-4 to generate approximately 158,000 visual-instruction examples from image captions and region information. [1] [3] The case shows how a relatively simple connector and carefully designed instruction data could turn existing unimodal components into a useful conversational VLM, helping open-source researchers reproduce and extend multimodal assistants. [1] [3]
Flamingo and few-shot visual learning. DeepMind ’s Flamingo research demonstrated a different path: rather than focusing primarily on open visual chat, it investigated how a language model could learn visual tasks from a small number of multimodal examples. [5] [4] The work helped establish interleaved image-and-text prompting as an important VLM pattern and influenced later systems that adapt language-model interfaces to visual inputs. [5] [4] Its significance lies in showing that multimodal models could inherit the flexible, prompt-based behavior associated with large language models while incorporating images. [5] [4]
From VLMs to embodied action. More recent work extends VLMs beyond answering questions toward controlling agents and robots. LLaVA-VLA describes a vision-language-action system built around a compact VLM backbone, adding multi-view perception, proprioceptive inputs, and action chunking for navigation and manipulation. [9] The transition illustrates a broader redefinition of the concept: a VLM is no longer limited to producing text about an image but can serve as the perceptual and reasoning component of an embodied system that selects actions. [9]

Sources

[3] 2026, Sep 17. awesome.papernotes.org . Visual Instruction Tuning . Updated: 2026, Oct 13. Generated: 2026, Nov 14. Perplexity API response.

[4] 2026, Jan 23. github.com . Awesome-Multimodal-Models/README.md at master · arrdel/Awesome-Multimodal-Models . Updated: 2026, Nov 03. Generated: 2026, Nov 14. Perplexity API response.

[5] 2026, Nov 08. app.cochat.ai . Visual Instruction Tuning – Citation check . Updated: 2026, Nov 12. Generated: 2026, Nov 14. Perplexity API response.

[6] 2026, Nov 13. alphaxiv.org . Matthew L Olson on alphaXiv . Updated: 2026, Nov 14. Generated: 2026, Nov 14. Perplexity API response.

[7] 2026, Aug 10. ar5iv.labs.arxiv.org . LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligence . Updated: 2026, Nov 04. Generated: 2026, Nov 14. Perplexity API response.

[8] 2026, May 25. arxiv.org . LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligence . Updated: 2026, Jun 14. Generated: 2026, Nov 14. Perplexity API response.