Read deeper.
Think further.

Attention is a way of letting a machine look at everything at once and decide what matters most. Think of a crowded room: you ask a question, and each person offers an answer. The machine weighs those answers, giving more weight to the ones that fit your question, then blends them together§3.2.

In this paper the authors used a version they call scaled dot-product attention. They compare each question with every key by multiplying them, shrink the results a bit, and turn them into weights that add up to one; then they take a weighted average of the values§3.2.1. That shrinking keeps things steady when the numbers grow large.

They also do this several times over, side by side, so the model can notice different kinds of relationships at once§3.2.2.

Attention takes a question, called a query, and some clue pairs, called keys and values. It is like checking clues to see which ones match your question§3.2. Each clue gets a weight. Then attention adds the values together using those weights.

The paper's special kind is Scaled Dot-Product Attention§3.2.1. It multiplies the query with all keys. Then it divides each answer by the square root of dk. A softmax turns those into weights. Then it makes a weighted sum of the values. They do many queries at once, packed as matrices Q, K, and V.

Why divide? For small dk, plain dot-product attention is fine§3.2.1. For large dk, the dot products get big. That pushes softmax into a tiny-gradient spot, which is bad. Scaling helps fix that. Multi-head attention does this many times in parallel. It is like many helpers looking at different views, then joining their answers§3.2.2.

Attention begins with a query and a set of key-value pairs§3.2. For every value, it computes a weight from how compatible the query is with that value's paired key; the output is the weighted sum of the values.

In scaled dot-product attention, queries and keys have dimension dk, while values have dimension dv§3.2.1. You take the dot product of the query with each key, divide each result by the square root of dk, then apply softmax to turn the numbers into weights. Multiply each value by its weight and add them. With many queries, keys, and values packed into matrices Q, K, and V, this is Attention(Q,K,V)=softmax(QK^T/√dk)V. The scaling by 1/√dk prevents large dot products, which occur when dk is large, from pushing softmax into regions with extremely small gradients.

Multi-head attention repeats this function in parallel on learned linear projections of queries, keys, and values, then concatenates and projects the results§3.2.2.

Attention maps a query and key-value pairs to an output; the output is a weighted sum of the values, and each weight comes from a compatibility function of the query with the corresponding key§3.2. In scaled dot-product attention, queries and keys have dimension dk and values dv. You compute the dot products of the query with all keys, divide each by sqrt(dk), and apply softmax to get weights on the values; in matrix form, Attention(Q,K,V)=softmax(QK^T/sqrt(dk))V§3.2.1. Scaling matters because for large dk the dot products grow large, pushing softmax into regions with extremely small gradients.

Multi-head attention projects queries, keys, and values h times with learned linear projections to dk, dk, and dv dimensions, runs attention in parallel on each projected version, concatenates the dv-dimensional outputs, and projects again; this lets the model jointly attend to information from different representation subspaces at different positions§3.2.2. The excerpts do not explain masked or encoder-decoder attention.

An attention function maps a query and a set of key-value pairs to an output, where all of these are vectors; the output is a weighted sum of the values, and the weight on each value comes from a compatibility function of the query with the corresponding key§3.2. In practice, queries are packed together into a matrix Q, and keys and values into K and V§3.2.1.

The paper's scaled dot-product attention takes the dot products of the query with all keys, divides each by the square root of dk, applies a softmax to obtain the weights on the values, and computes softmax(QK^T/√dk)V§3.2.1. The scaling exists because for large dk the dot products grow large in magnitude, pushing the softmax into regions with extremely small gradients.

Multi-head attention instead projects queries, keys and values h times with different learned linear projections, performs attention in parallel on each projected version, then concatenates and projects the results, letting the model attend to information from different representation subspaces at different positions§3.2.2.