“Tokens” might sound like the latest digital currency, and in a way, they are. Everything in the world of AI revolves around them—most importantly, the pricing. But not all tokens are created equal. Today, we’re breaking down the difference between input and output tokens.
The AI’s “Secret Code”
When you send a message to a chatbot, your prompt isn’t read as plain text. First, it’s translated into a format the “model” can actually digest. Think of this as the model’s internal dialect; every model has its own specific version.
Why not just use regular words? Efficiency. Instead of trying to memorize every possible word in every language, models “compress” the vocabulary into small chunks of characters—usually 2 to 4 letters long. They’re a bit like syllables.
For example, the word “learning” might be broken down into “learn” and “ing”.
- In this case, one word becomes two tokens.
- A 10-word sentence might actually be 15 tokens once the AI is done slicing it up.
Math in the “Synthetic Brain”
A token is essentially the basic unit of work for a Large Language Model (LLM). All that fancy math happening inside the AI occurs on these tokens. Inside its synthetic brain, tokens are compared, summed, and multiplied in a very specific sequence to produce—drumroll, please—output tokens.
Once the LLM finishes its “thinking” process, it spits out a string of these output tokens. These are then reverse-translated back into human language using the same dictionary that was used at the start.
The Economy of Compute
At its core, an LLM is a machine that does math with tokens. Because this process is highly measurable, AI companies use tokens as their primary metric for cost. If you use an API (like Claude or GPT), you’ll see a price tag for every 1,000 input tokens and a different price for output tokens.
This is because companies can measure exactly how much electricity and computing power it takes to process a single unit. It’s why prices fluctuate when a more efficient chip is invented or a new data center opens.
In an economy where AI is becoming a massive part of production, tokens are starting to behave like a currency. (Though I haven’t seen anyone call them “digital gold” quite yet!)
The Rule of Thumb: For now, just remember that tokens are pieces of text. If you count the syllables in a sentence, you’re actually pretty close to the token count.
Beyond the Chatbot
As we established in our last post, a “model” is essentially just a massive file full of numbers. While we usually interact with these through a chatbot interface, that’s not their only trick. We can also use them to measure contextual similarity—essentially figuring out how “close” two different documents are in meaning—with incredible ease.
Leave a Reply