LLM output optimization

Scaling large language systems can increase maintenance costs if memory and maintenance design are ignored.

Stacking transformer layers improves accuracy but also increases memory usage and the cost per call. Decoder-only models process tokens in two stages during model inference: a pre-fill stage that relies on GPU computation and a decoding stage that is memory-bound The decoding stage often introduces end-to-end latency.

Batching improves throughput, but increases KV cache and memory overhead. Therefore, parallelism, cache management, and runtime controls must be used to keep GPUs busy.

This guide describes optimization techniques such as paged attention, I/O-aware attention, quantization, and KV reduction, as well as infrastructure patterns for scaling safely while measuring performance gains.

Quick Take

• Use metrics to prove throughput and latency gains.
• Focus on memory-bound decoding work to reduce real latency.
• Balance batching with the KV cache size to avoid GPU OOMs.
• Use KV quantization and reduction to reduce model memory.
• Use pipeline and tensor parallelism to scale across GPUs.

Prefill vs. Decoding

Prefill is the first phase of inference, during which the model processes the entire input prompt. At this stage, the user text is tokenized and passed through all layers of the transformer. The model computes internal representations of the tokens and generates attention states, which are stored in the key-value (KV) cache. Prefill is computationally intensive because the entire sequence of input tokens is processed at once. However, it is only performed once for each request.

Decoding is the second phase, during which the model generates a response token by token. Using the context and KV cache formed during prefill, the model predicts the next token, adds it to the sequence, and repeats this process until a response is generated or a length limit is reached. Unlike prefill, decoding is performed multiple times. Although each step requires less computation, the overall response latency depends on the decoding speed.

KV cache: memory size and formulas

KV-cache (key-value cache) is a mechanism that stores intermediate representations of keys and values ​​from the attention mechanism. Thanks to this, the model does not enumerate the entire token sequence at each generation step. It uses already stored keys and values, which speeds up the decoding phase.

However, using KV-cache consumes a lot of memory, especially for large models and long contexts. Therefore, understanding how to estimate its size and control memory usage is essential for stable LLM deployment.

Calculate the total size of KV

The size of the KV-cache depends on several model parameters: the number of transformer layers, the number of attention heads, the dimensionality of each head, the context length, and the numeric representation type. For each token, the model stores keys (K) and values ​​(V) for each layer and each attention head. The formula can estimate the total memory size:

KV cache memory ≈ 2 × L × H × D × T × bytes_per_value

where

1. L is the number of model layers,
2. H is the number of attention heads,
3. D is the dimension of one head,
4. T is the number of tokens in the context,
5. bytes_per_value is the number of bytes per numeric value.

The multiplier of 2 appears because both keys and values ​​are stored. From this formula, it is clear that the size of the KV-cache grows linearly with the length of the context and the number of model layers. In large LLMs with tens of thousands of tokens, the KV-cache can occupy several gigabytes of GPU memory, making it one of the main factors limiting inference scaling.

Controls to avoid OOM

To avoid out-of-memory (OOM) errors and maintain high throughput, use caching strategies such as KV-cache optimization, system-level caching, and precomputation.

1. Limiting the maximum length of the context, since it directly affects the cache size.
2. Using less precise numeric formats, such as FP16 or BF16, which reduce memory usage compared to FP32.
3. Paged or segmented KV-cache allows you to manage memory and distribute it between multiple requests.
4. Batch control and request scheduling are used to avoid running many long generations at once.
5. Optimized engines that support KV-cache reuse, offloading to CPU memory, or cache compression, which helps to work with longer contexts without overloading the GPU.

Fuse and reordering for speed: FlashAttention in practice

FlashAttention is an optimized algorithm for computing self-attention in transformer models, reducing memory usage and speeding up computation. In the standard implementation of attention, large intermediate matrices are created during the calculation. In long-running contexts, this can strain GPU memory and cause slowness or even out-of-memory (OOM) errors.

FlashAttention solves this problem; instead of storing large intermediate matrices, the algorithm processes data in blocks (tiles) in fast GPU memory (SRAM). This reduces the number of global memory accesses and improves GPU bandwidth efficiency.

The peculiarity of FlashAttention is that it performs exact attention calculations. The algorithm combines several stages:

• calculation of the product of queries and keys;
• application of softmax;
• multiplication by values ​​in a single optimized pass.

This avoids storing the full attention matrix in memory.

FlashAttention is useful in scenarios where long contexts or large batch sizes are used. It is used during LLM inference and training to increase GPU throughput and reduce processing latency.

Quantization for lower weights and faster inference

Quantization is a method for optimizing large language models that reduces the size of weights and speeds up inference by using lower-precision numbers. By default, model weights are stored in FP16 or FP32 format, which requires significant memory and computational resources. During quantization, these values are converted to more compact representations, such as 8- or 4-bit integers, while preserving most of the model parameter information.

The main advantage of quantization is that the model can be deployed on resource-constrained hardware without significant loss of quality. This is important for running LLM locally, inferring on edge devices, or processing a large number of requests in cloud services.

8-bit vs. 4-bit

Model service strategies that maximize throughput

In today's query processing systems, maximizing throughput is a critical performance factor. Bandwidth determines how many requests a system can process per unit of time, and without the right servicing strategies in place, it can quickly become a bottleneck.

Service strategies

Implementing these strategies allows systems to use resources efficiently, reduce processing latency, and ensure stable operation under high load. This is important for deploying LLM in a production environment or services with a large number of concurrent users.

FAQ

What is the difference between prefill and decode phases?

Prefilling forms a context by processing all incoming tokens at once, and decoding generates a response one token at a time using that context.

How to calculate the KV cache size for a model?

KV-cache size is calculated as 2 × number of layers × number of heads × head dimension × number of tokens × bytes per value (for keys and values).

What techniques help handle long contexts without OOM?

Using FlashAttention, quantization, dynamic memory offloading, and context length limits helps handle long contexts without OOM.

What is page attention and why use it?

Paged attention is a method of calculating attention in blocks to reduce memory usage for long sequences without losing accuracy.

How does the implementation of attention affect decoding throughput?

Implementing optimized attention, such as FlashAttention, increases decoding throughput, reducing memory usage and speeding up computation per token.

When should you use FlashAttention?

FlashAttention should be used when long contexts or large batches make standard attention slow or memory expensive.

How to choose between 8-bit and 4-bit quantization?

The choice between 8-bit and 4-bit quantization depends on the trade-off between model accuracy, memory savings, and inference speed.