LLM Inference Optimization: Cutting Costs with Quantization, KV Cache, and Batching

An AI model's lifecycle has two phases: training and inference. Training, where billions of parameters are learned from data, is very expensive but happens once. Inference is when the trained model answers each user request, and it repeats every day, for every request. For a company using open models, the training bill has already been paid by the model's creator; what you actually pay for is inference.

That is why inference efficiency directly determines whether an on-prem AI project is viable. Serving the same workload with one GPU instead of two, or cutting response time from 10 seconds to 2, can decide whether users adopt the system and how quickly the investment pays back.

In this article we describe the optimization techniques MeSoft applies in client deployments. The goal is to give technical teams a checklist and decision makers a clear answer to 'why this hardware and this software?'.

An LLM processes a request in two phases. In prefill, the entire input (question, system instruction, and attached documents) is processed in parallel and the KV cache is built; this phase is compute-bound. In decode, the model generates the answer token by token, and every new token requires reading all model weights from GPU memory, so this phase is memory-bandwidth-bound.

This distinction matters for tracking the right metrics. TTFT (Time To First Token) is how long a user waits to see the first word; it depends mostly on prefill and grows when long documents are attached. TPOT (Time Per Output Token), or tokens per second, shows how smoothly the answer streams and depends on decode. Throughput is the total number of tokens per second the system produces across all users, and it is the basis of any cost calculation.

A practical takeaway: because decode is limited by memory bandwidth, a GPU serving a single user leaves most of its compute idle. Using that idle capacity by processing many requests at once (batching) is the largest single source of gains in inference optimization.

Model weights are usually distributed as 16-bit numbers (FP16/BF16); an 8-billion-parameter model needs about 16 GB in that format and a 70-billion-parameter model about 140 GB. Quantization represents the weights with 8, 4, or even fewer bits, reducing memory by 2 to 4 times. With 4-bit quantization an 8B model drops to about 5 GB and a 70B model to about 40 GB.

Different methods suit different environments: GPTQ and AWQ are widely used 4-bit methods on GPU servers; the GGUF format (used by llama.cpp and Ollama) offers flexible options on CPUs and consumer GPUs; and FP8 delivers a near-lossless speedup on recent data-center GPUs. Beyond the weights, the KV cache can also be stored in 8 bits to fit more concurrent users in the same memory.

Quantization has a price: especially below 4 bits, quality loss can become noticeable in mathematical reasoning and in languages that are less represented in training data, such as Turkish. The right quantization level should therefore be chosen with a test set built from the company's own tasks, not from generic benchmarks. Quite often, a large model quantized to 4 bits outperforms a smaller model at 16 bits.

Traditional servers pre-allocate KV cache memory for the longest possible answer of every request, and most of it goes unused. PagedAttention, introduced by vLLM, allocates the KV cache in small blocks on demand, similar to virtual memory paging in operating systems. This nearly eliminates memory waste and lets far more concurrent requests fit on the same GPU. With prefix caching, requests that share a common system prompt can also reuse the prefill computation.

Continuous batching, instead of waiting to form fixed batches, adds new requests and removes finished ones at every generation step. A short question no longer waits for another request that is generating a long report. Together, these two techniques often increase throughput several times over a naive implementation on the same hardware.

Speculative decoding uses a small, fast 'draft' model to guess several tokens ahead, which the large model then verifies in a single step. Correctly guessed tokens come for free and output quality is unchanged, because the large model always has the final say. It noticeably reduces latency for predictable outputs such as templated report text and code.

The right inference server depends on the use case. vLLM and SGLang provide high throughput and an OpenAI-compatible API for multi-user GPU servers; TensorRT-LLM targets peak performance on NVIDIA hardware; llama.cpp and Ollama stand out for ease of setup on single servers, workstations, or CPU-heavy environments. Using an OpenAI-compatible API lets you swap models or servers without changing application code.

On the hardware side, the rule is simple: define the workload first, then choose hardware. For a 20-person office summarizing a few hundred documents a day, a single GPU with 24 to 48 GB of memory and a quantized 8B to 14B model is often enough. Hundreds of concurrent users or 70B-class models call for multiple GPUs and tensor parallelism. Non-interactive jobs such as batch reporting can run overnight, keeping the same hardware free for interactive use during the day.

At MeSoft we treat inference optimization as a measurement-driven process rather than a one-time setup: we load-test with real request samples, measure TTFT, throughput, and quality, and then tune quantization, batching, and caching based on that data. This approach lets SMEs in Mersin deploy AI at the lowest possible hardware cost. To understand why these optimizations work, see the attention mechanism article in this series.