Welcome back to my Friday series! Every week, we take a complex technical term or concept and strip away the jargon to see how it actually works—debunking a few myths along the way.
Last week, we looked at what kind of “text file” an LLM actually is. This week, we’re taking it a step further to explore a concept that became the industry standard over the last year: Multi-Modal Models.
The “Old Days” (All of two years ago…)
In the early days of the generative AI revolution, everything lived in its own silo. You had one app for generating images, another for text, and a different one for video.
This was because:
- Text models were trained on a “mind map” of words.
- Image models were trained on fractions of pixels.
- Video models were trained on sequences of frames.
The logic for an image model was: “Which pixel would look best next to the previous one?” That is how it builds a picture of a jaguar, much like a text model predicts the next character to write an essay.
Enter Multi-Modality
Multi-modal models change the game by generating text, images, and even audio simultaneously from the same base model. Instead of separate maps, the “mind map” these models use is built from “bits of words” and “bits of images” at the exact same time. They predict the next “bit” in the sequence regardless of whether that bit represents a syllable or a splash of color.
The Mind-Boggling Part: For an engineer, these are all just abstract numbers. In a multi-modal model, there are literal connections between digital nodes representing “canary yellow” and “submarine.” To the AI, these aren’t different categories of media; they are just “related concepts we might think about.”
(Computer Science folks: I know I’m over-simplifying here! Feel free to roast my definitions or provide the “proper” version in the comments.)
The “1+1=3” Effect
The most surprising thing about multi-modal models is that they work better than individual ones. A multi-modal model is often better at generating text than a model trained only on text. By “seeing” the world through images and “hearing” it through audio, the model develops a deeper understanding of the concepts it’s writing about.
The Catch: This efficiency is driving massive centralization. Because these “all-in-one” models are so huge and expensive to maintain, only a handful of companies can afford to keep building them.
What do you think?
Does this “universal mind map” explanation make sense to you, or have you heard a better analogy? Drop your thoughts in the comments!
Also, let me know if there are other terms you’d like me to demystify. Unless someone has a better idea, next Friday we’re diving into the “black box” of Neural Networks. 🕸️
Leave a Reply