How do large language models actually know where words sit in a sequence
By AI Update World · 2026-11-15

Understanding Positional Embeddings in Modern LLMs
Before transformers, language models relied on recurrence. They read words one at a time, left to right, and kept hidden state that accumulated meaning as they went. Then came 2017. Researchers introduced the transformer architecture, which threw out the sequential constraint entirely. It could look at all words at once. But this created a problem: if all words get processed simultaneously rather than in order, how does the model know which word comes first, which comes last, and what the sequence actually means? That problem led to positional embeddings, a mathematical solution that has become foundational to how modern AI reads text.
The Recurrence Problem and Its Solution
The oldest neural networks for language used loops. A recurrent neural network reads input token by token, updating an internal memory state after each step. The recurrence itself creates order. By the time the model sees word ten, it has already processed words one through nine and carries their meaning forward. Transformers broke that pattern. They use attention mechanisms that let every position in a sequence look at every other position at the same time. This parallelism made training faster and allowed the model to capture long range relationships. But without the sequential processing step, the model had no built in sense of where anything was. Two sentences with identical words in different orders would look identical to the raw transformer. So researchers needed a way to tell the model: this token is first, that token is second, that other token is eighth. Enter positional encoding.
Early Approaches: Fixed Sinusoidal Patterns
The original transformer paper proposed positional encodings based on sine and cosine functions. Each position in the sequence gets a unique numerical pattern made from waves of different frequencies. Position zero might get one sine wave pattern, position one gets a slightly different pattern, position two gets another, and so on. These patterns are deterministic and fixed. You do not train them or change them. The model learns that each unique pattern corresponds to a location, and it learns to use that information when attending to other words. This approach works well because sine and cosine waves have nice mathematical properties. The model can learn to recognize the patterns and generalize to longer sequences than it ever saw during training. The approach is elegant and requires no additional parameters to tune.
Why Sinusoids Were Not the Final Answer
Sinusoidal encodings work but they have weaknesses. They are rigid. The model cannot adapt them or learn which aspects of position matter most for its task. Also, the patterns repeat at certain intervals, which can create mathematical artifacts. For very long sequences, the repeating nature of waves becomes less useful. As language models grew larger and encountered longer texts, researchers began experimenting with alternatives. Some m