Inference

Inference is the processing of an input by an already trained model. The result may be text, a classification or a numerical prediction.

The model parameters are generally not changed during this process. This distinguishes inference from training or fine-tuning, where parameters are adjusted.

The cost of inference

Inference cost arises with every use and may account for a substantial part of total operating cost.

Important factors include the model and hardware, input and output length, request volume and concurrency, and requirements for response time and availability.

A larger model is not automatically the most economical choice. What matters is which model achieves the required quality under the relevant operating conditions.

Cloud API versus self-hosted inference

Inference may run through a managed API, a dedicated cloud environment or privately operated hardware.

APIs provide variable cost and limited operational work. Private infrastructure provides fixed capacity and greater control, but requires investment, utilisation planning, maintenance and responsible staff.

The economical choice can be determined only from the specific load profile and full operating cost.

Latency and context length as additional levers

Response time depends on factors including the model, hardware, utilisation, network, input context length and generated output length.

Interactive or time-sensitive applications should therefore provide only the necessary context. Realistic load tests show whether the chosen architecture meets its targets.