From text to LLM
CONNECTIONS IN CONTEXT

Attention

The model iterates through all context tokens and evaluates their influence on each other

Transformer builds a table of connections: which previous tokens are important for each position now [2].

A bright cell means more weight in one attention operation.

[2] arXiv
N TOKENSN × N connections between positions
where do we get information from? →
↓ / →Mashapostcupontable,she
Masha
post
cup
on
table
,
she
← for whom we collect context
Educational attention operation: brightness shows the weight of the connection, but not the entire causality of the decision.
08/ 31