~/vdanepalli

ERA V5session 01

From neural networks to the transformer.

The School of AI · ERA V5
neuron
→
layers
→
training
→
embeddings
→
attention
→
assistant

01Neural network

Learns to find patterns. It predicts, compares the prediction with the actual answer, and adjusts its state. Repeat n times until it reaches the goal state.

fig. 1 · the learning loopanimated

This figure needs JavaScript.

02Biological neurons

  1. Decides how much to pass on. Incoming signals add up in the cell body; it fires only when the total crosses a threshold.
  2. Talks to many others. Each neuron connects to thousands of others (not to every other one).
  3. Spiking. Neurons fire in spikes and can stay silent, a bit like sleeping. Spiking neural networks borrow this.
fig. 2 · a neuron charging and firinganimated

This figure needs JavaScript.

03Neuron

Multiply, add, squish. A weighted sum of the inputs, plus a bias, passed through an activation function.

z = x₁·w₁ + x₂·w₂ + x₃·w₃ + b
a = f(z)
fig. 3 · multiply → add → squishanimated

This figure needs JavaScript.

Bias lost importance. With y = mx + b it matters. But today x spans x₁ … xbillion, so the chance of every input being zero is tiny, and normalisation layers re-centre values anyway. LLaMA and PaLM drop biases from their linear layers.

04Activation functions

  • ReLU: negative → 0, else x; max(0, x). Dying ReLU: if a neuron gets stuck in the negative zone, its output and its slope are both 0, so it sleeps forever.
  • Sigmoid: squishes x between 0 and 1.
  • Step: below zero → 0, above zero → 1. Its slope is 0 everywhere, so there's nothing to learn from.
  • GELU: a smooth ReLU that lets small negatives through.
  • Swish: x · sigmoid(x), also smooth and slightly negative near 0.

Without them, stacked layers collapse into a single linear map (S1-2 proves it).

fig. 4 · the shapes, and their slopesdrag across the plot

This figure needs JavaScript.

05Feed-forward network

Neurons in layers; each layer's output is the next layer's input, flowing one way from input to output.

We can have a different number of neurons in each layer, but we prefer the same number for computation reasons: GPUs run big, regular matrix multiplies fastest.

fig. 5 · a forward passanimated

This figure needs JavaScript.

06Loss and gradient descent

Loss: how wrong the network's answer is. Gradient descent: change the weights in the direction that reduces the loss, w ← w − η · ∂L/∂w. The learning rate η sets the step size.

fig. 6 · rolling down the losspick a learning rate

This figure needs JavaScript.

07Language

Two challenges: positional information (order changes meaning) and contextual awareness (a word's meaning depends on its neighbours).

fig. 7 · order, and contextanimated

This figure needs JavaScript.

08Embeddings

Each token becomes a vector: a row of a learned table. Tokens used in similar ways end up close together. S1-3 shows this emerge from next-token training alone.

fig. 8 · meaning as geometry (2-D cartoon)hover a word

This figure needs JavaScript.

09RNN

Reads a sentence one word at a time and carries a running summary of what it has read so far: the hidden state.

hₜ = tanh(W_h · hₜ₋₁ + W_x · xₜ + b)

Because it's sequential, it can't be parallelised, and the running summary tends to forget the past when the context is long.

fig. 9 · watching an RNN forgeta real tiny RNN

This figure needs JavaScript.

10Attention

Achieved through Query, Key and Value.

  • Query: the information the word is looking for.
  • Key: the information the word can offer to other words.
  • Q·K: the strength of that connection (scaled by √dk, then softmaxed into weights).
  • Value: what actually gets passed along, weighted by that strength.
fig. 10 · what "it" attends toclick a word

This figure needs JavaScript.

11Residual connections and normalisation layers

  • Residual connection: add a layer's input to its output, x + f(x). The layer only learns a change, and gradients get a shortcut through deep stacks.
  • Normalisation layer: rescale each vector to mean 0, spread 1, so values don't blow up or fade out layer after layer.
fig. 11 · skip path, and rescalinganimated

This figure needs JavaScript.

12Assignments