The Attention Mechanism: Understanding the Heart of LLM Architectures

The Transformer architecture, introduced in the 2017 paper 'Attention Is All You Need', is the foundation of nearly every large language model in use today, and attention is at its core. Earlier RNN and LSTM models processed text one word at a time and tended to 'forget' information from the start of long sentences. Attention lets every word in a sequence look directly at every other word.

A simple example: in 'The container reached the port early because it had departed ahead of schedule', understanding what 'it' refers to requires the model to attend more to 'container' than to 'port'. Attention learns these 'how much should I look at each word' weights from data. As a model trains, the weights come to capture grammar, meaning, coreference, and even logical relationships.

This is not just academic curiosity. Knowing how attention works is the key to understanding how much GPU memory an in-house LLM needs, why context length multiplies cost, and which model architecture suits your workload. At MeSoft we factor these architectural details directly into the models we recommend to clients.

For every token (a word or word piece), attention produces three vectors: a Query (Q, 'what am I looking for?'), a Key (K, 'what do I contain?'), and a Value (V, 'the information I carry'). How much one token attends to another is measured by the similarity (dot product) between its Query and the other token's Key. These scores are turned into a probability distribution with softmax, and the Value vectors are summed using those weights.

As a formula: Attention(Q, K, V) = softmax(Q·Kᵀ / √d_k) · V. Dividing by √d_k keeps dot products from growing too large as vector dimensions increase, which would saturate the softmax and destabilize training. Q, K, and V are obtained by multiplying the token's embedding by three separate learned matrices, so what the model 'pays attention to' is determined by matrices learned during training.

Generative (decoder-only) models also apply a causal mask: each token can only attend to tokens before it and cannot see the future. This is what allows the model to generate text left to right by predicting the next token. Popular model families such as GPT, Llama, Qwen, Mistral, and Gemma all use this structure.

A single attention computation can capture only one kind of relationship between tokens. Multi-head attention therefore runs the computation in parallel across many 'heads': one head may focus on syntax, another on what pronouns refer to, another on numeric values. The heads' outputs are concatenated and passed to the next layer. Large models have dozens of layers, each with dozens of heads.

In classic multi-head attention, every Query head has its own Key and Value head. During generation, however, keeping those K and V values in memory (the KV cache) is expensive. Multi-Query Attention (MQA) shares a single K/V head across all Query heads, while Grouped-Query Attention (GQA) splits Query heads into groups that share a K/V head. A model with 32 Query heads and 8 K/V heads, for instance, cuts KV cache memory by 4x compared with the classic design at very little quality cost. Most of today's open models use GQA.

Attention by itself has no notion of word order: 'the port moved the container' and 'the container moved the port' contain the same words. Models therefore inject positional information. Most modern models use RoPE (Rotary Position Embedding), which rotates Query and Key vectors according to token position so that the relative distance between two tokens is naturally reflected in their attention score. RoPE scaling techniques such as YaRN are also used to extend models to longer contexts than they were trained on.

Attention's strength is also its cost: in a sequence of n tokens, every token looks at every other token, so compute (and, in a naive implementation, memory) grows with n². Going from 4,000 to 32,000 tokens multiplies the attention computation by roughly 64. Using a model advertised with a '128K context window' on long documents therefore has a direct impact on your hardware budget.

During generation, the K and V vectors of already-processed tokens are kept in memory so they don't have to be recomputed at every step; this is the KV cache. Its size per token is roughly 2 × number of layers × number of K/V heads × head dimension × bytes per value. For an 8-billion-parameter model with 32 layers, 8 K/V heads, and 128-dimensional heads, that comes to about 128 KB per token in FP16. A single 8,000-token conversation needs about 1 GB, and 20 concurrent users need about 20 GB of extra GPU memory, on top of the model weights themselves.

This calculation shows why choosing a GPU for an on-prem deployment is not just a question of 'does the model fit?'. The number of concurrent users, document length, and target response time determine required memory as much as model size does. The inference optimization article in this series covers techniques such as quantization and PagedAttention that shrink the KV cache.

FlashAttention is an algorithm that computes attention far more efficiently on GPUs without changing the math. The core idea is to never write the large n×n score matrix to the GPU's slower main memory (HBM), but to compute it in small tiles in fast on-chip memory (SRAM). The result is exactly the same (not an approximation), memory use grows with n instead of n², and long contexts get a substantial speedup. Modern frameworks such as vLLM, TGI, and PyTorch use it by default.

There are architectural efficiency techniques as well: sliding window attention limits each token to the last few thousand tokens; some models alternate local and global attention layers; and Mixture-of-Experts (MoE) architectures reduce compute by activating only a fraction of the feed-forward parameters for each token. When choosing a model, these architectural decisions can create large performance differences between two models with similar parameter counts.

In short, attention is the mechanism that lets LLMs understand context, and also the one that drives most of their hardware cost. When MeSoft builds AI infrastructure for companies in Mersin, we combine this architectural knowledge with measurements of the real workload to recommend hardware and models sized to what you need, not larger. To see attention's practical implications, the RAG article in this series explains why smart retrieval beats simply stuffing a long context.