You send a prompt to an API, and a few hundred milliseconds later, you get a token back. That round-trip hides a lot of moving parts—each with its own cost, latency, and failure modes. If you want to tune inference or predict what will break first at scale, you need to know what actually happens between prompt and token.
AI inference is not a single operation. Each token is the result of a pipeline that includes network, pre-processing, model execution, post-processing, and sometimes more. Where the time and money go depends on your model, your serving setup, and how you call it.
The steps between prompt and token
The typical flow for text generation looks like this:
client ──▶ API server ──▶ pre-processing ──▶ model runner ──▶ post-processing ──▶ API server ──▶ client
- Network: Your prompt travels over the network to the inference server. If you are using a cloud API, this is a real contributor to latency. If you are running locally, this step is minimal.
- Pre-processing: The prompt is tokenised—split into the units the model understands. This can introduce overhead, especially for long or complex prompts.
- Model execution: The core of inference. The model loads (if not already resident), the context is assembled, and the model generates the next token. This step is compute-heavy and often the main driver of cost.
- Post-processing: The generated token is mapped back to text. Some servers also apply filters or safety checks here.
- Network (again): The token is sent back to the client. In streaming APIs, this happens once per token; in batch APIs, after the whole response.
Where the time and money go
- If the model is not already loaded in GPU memory, loading it can dominate latency. Keeping models warm is expensive but avoids this penalty.




