What it means
Inference is the stage when a trained AI model is used to produce an output, such as an answer, a classification or a prediction, from new input. Every time someone uses an AI tool, the model is running inference.
Real-world example
A tax office's document classification model was trained once over several weeks on historical letters. Each morning it now runs inference on around 3,000 newly scanned letters, sorting them into categories in seconds per document.
Why it matters for AI adoption
For organisations using existing models, the ongoing cost, speed and energy use of AI come mainly from inference, not training. Inference volume drives cloud bills and hardware needs, and where inference runs determines where data is processed.
Often confused with
Fine-tuning (and training in general)
Training is how a model learns patterns from data, usually done once or occasionally and requiring large amounts of computing power. Inference is using the finished model to produce outputs, which happens every time someone uses it. Most organisations only run inference.
Questions to ask
- Where does inference happen, and where is our data processed when it does?
- What are the expected response times and costs at our real usage volume?