Architecture

Attention Mechanism

A mechanism in neural networks that computes weighted relevance scores between elements of a sequence, allowing the model to focus on the most pertinent information for each output.

Explained at five levels

Level 1

The way AI decides which words in a sentence are most important — like when you highlight the key words in a book.

Level 2

A technique that lets AI focus on the most relevant parts of the input when generating each word, instead of treating everything equally.

Level 3

A mechanism in neural networks that computes weighted relevance scores between elements of a sequence, allowing the model to focus on the most pertinent information for each output.

Level 4

The core operation in transformers that computes pairwise relevance via scaled dot-product of query, key, and value projections, enabling dynamic context-dependent weighting of input representations.

Level 5

Scaled dot-product attention: Attention(Q,K,V) = softmax(QKᵀ/√dₖ)V — extended via multi-head projections, causal masking, and positional encodings, with O(n²) complexity driving research into linear, sparse, and sub-quadratic alternatives.

Definitions are educational summaries. Terminology can vary by source and context.

Sources