Lexicon · II · Access
Inference
The moment the model works: it receives a context and produces its answer, token by token. Every request is one inference, and each one has a cost.
The analogy
A doctor's consultation. Ten years of study shaped their judgment: that is training, done once. Each consultation mobilizes all of that knowledge on your particular case without modifying it: the doctor does not become a different doctor between two patients. Inference is the consultation: the use of the knowledge, not its making.
A concrete example
Every chat answer, every automatic summary in your CRM, every turn of an agent's loop is an inference. At the scale of an organization, that becomes millions of inferences a day. And it is inference, not training, that fills the datacenters, strains the power grids and makes up your monthly bill.
What it is NOT
It is not learning: the model comes out of every request unchanged. It is neither instant nor free: latency and cost per token determine what a use case can be in production. And it is not a database lookup: the model does not retrieve a stored answer, it produces one. That is its strength, and the source of its confident mistakes.
Often confused with
The distinction that changes the conversation.
- Training
- Inference uses frozen weights; training makes them. In the moment, an inference therefore learns nothing: your data passes through the model without settling into it. But the provider may keep your conversations for a future training run: consumer products often do so by default, enterprise offerings do not. That is a contract clause, not a mechanism: check it.