~/vdanepalli

ERA V5session 1 · assignments

Four claims, tested.

Each section states a claim, shows why it should hold, then proves it with an experiment that trains in your browser. Nothing is precomputed.

01assignment 1 of 4 · S1-1 · activations

Activations exist for a reason.

Without a non-linearity, a model can only draw a straight line. Two rings, one inside the other, can't be split by a line. Add one ReLU layer and the boundary wraps around the inner ring.

Starting point: a neuron is a weighted sum plus an activation. What does the activation actually buy us?

concept · why: a line can only cut the plane in two; ReLU units add bendsanimated

the experiment

claimNo non-linearity means a straight boundary, so it can't separate concentric rings. One ReLU hidden layer can.
build300 noisy 2-D points in two rings. Train (a) one linear layer + sigmoid and (b) one ReLU hidden layer, on the same data with the same optimiser.
proofPlot both decision boundaries. The linear one is a straight line at about coin-flip accuracy; the ReLU one wraps the ring at about 99%.
(a) linear + sigmoid2 → 1 · params
–accuracy
(b) one ReLU hidden layer2 → → 1 · params
–accuracy
(a) linear · loss(b) ReLU · lossstep 0 / 400

verdictPress train. Both models see the same 300 points, the same optimiser and the same number of steps.

What to notice

  • Model (a) can only tilt and shift one straight line. Every orientation cuts each ring roughly in half, so it hovers around 50%.
  • Model (b) has a few ReLU units, each contributing one bend; together they form a closed polygon around the inner ring.
  • Try 2 or 3 hidden units: too few bends to enclose the ring. Around 4 or more, it closes.

How it's built

data 300 points, inner ring r = 1, outer ring r = 2, Gaussian radial noise
(a) z = w·x + b → sigmoid
(b) h = relu(W₁x + b₁) → z = w₂·h + b₂ → sigmoid
loss binary cross-entropy · optimiser Adam, lr 0.03, full batch, 400 steps
code plain JavaScript, hand-written backprop, runs in your browser: ml.js · session-1.js

end of S1-1next: S1-2 · depth ↓
02assignment 2 of 4 · S1-2 · depth

Depth without non-linearity is a lie.

Stack five linear layers and you still have one linear layer, just written out longer. It fails the rings exactly like a single layer. Put a ReLU between the same five layers and it works.

From S1-1: one ReLU layer solved the rings. But was it the activation, or just having more weights? Here: more layers, no activations.

concept · why: linear layers multiply into one matrixanimated

the experiment

claimFive linear layers collapse into one linear map, so they're no stronger than one layer. ReLUs between them break the tie.
buildThe same rings as S1-1. Train 1 linear layer, 5 linear layers (no activations), and 5 layers with ReLU. Same optimiser, same steps.
proofThe first two get the same accuracy and the same straight boundary; the third wraps the ring. Bonus: multiply the five weight matrices and get one 1×2 matrix.
1 linear2 → 1
–accuracy
5 linear2 → 8×4 → 1
–accuracy
5 + ReLUsame shapes
–accuracy
step 0 / 400

the collapse, computed live

verdictPress train. All three models see the same 300 points, the same optimiser and the same number of steps.

What to notice

  • The middle model has 249 parameters to the left one’s 3, and draws the same kind of line.
  • The product W₅W₄W₃W₂W₁ is a single 1×2 matrix. Feed it a point and you get exactly the 5-layer network's output.
  • The right model has the same shapes and the same parameter count as the middle one. The only change is ReLU between layers.

How it's built

data 300 points, two noisy rings (same generator as S1-1)
1 layer σ(w·x + b)
5 linear σ(W₅(W₄(W₃(W₂(W₁x + b₁) + b₂) + b₃) + b₄) + b₅)
5 + ReLU the same, with relu(·) around each hidden layer
loss binary cross-entropy · optimiser Adam, lr 0.03, full batch, 400 steps
code ml.js (see collapse()) · session-1.js

end of S1-2next: S1-3 · embeddings ↓
03assignment 3 of 4 · S1-3 · embeddings

Similarity, learned from next tokens.

A tiny model only ever learns to guess the next word. Nobody tells it that cat and dog are both animals. Words that are followed by the same things end up with similar embeddings anyway.

From S1-2: layers with non-linearities can carve almost any shape. The same machinery, trained only to guess the next word, ends up learning meaning.

concept · why: words with the same "what comes next" get pushed the same wayanimated

the experiment

claimTrained only on next-token prediction, the embedding table groups related words, though similarity was never part of the data.
buildSentences <animal> <verb> <object> . where eat takes a fruit, chase an animal, see either. Train one-hot → embedding → softmax on every (word, next token) pair.
proofProject the embeddings to 2-D and look up each word's nearest neighbour by cosine. Same-category words land together and neighbour each other.
embedding spacePCA of the -d embeddings · keeps –
animalsfruitsverbs
colours added after the fact;
the model never sees them
nearest neighbourscosine, over the 8 words
wordnearestcossame
–same-category neighbour rate
lossbest possible –now –step 0 / 500
the language8 words + "." · params

Every sentence is <animal> <verb> <object> . with animals cat dog cow, fruits apple mango, verbs eat chase see. eat takes a fruit, chase an animal, see either. Words are picked uniformly.

2,000 sentences give 6,000 training pairs. A few of them:


        

      

      

        
why they clustermodel's p(next | word)

        

        
Rows that predict the same next words need the same embedding.
Animal rows match, fruit rows match; verbs overlap through see.

      

    


    

      
      
verdictPress train. The model starts from random weights, so at first the neighbours are a matter of luck.

    


    

      

        
What to notice

        
    
          
  • At step 0 the points are scattered and most neighbours are wrong. That is the random init: nothing about categories is built in.
    • 
          
  • Animals and fruits pull together first. Each animal is followed by the same mix of verbs and "."; each fruit is always followed by ".". Identical targets, so the gradients push their embeddings the same way.
    • 
          
  • Verbs form a looser group. eat predicts fruits and chase predicts animals, so they never fully agree; see sits between them and each verb's nearest neighbour is still a verb.
    • 
          
  • Try embedding size 2. It usually still works, but now and then a word lands on the wrong side: two numbers per word is a tight fit for nine outputs.
    • 
        

      

      

        
How it's built

        
data 2,000 sentences (seed 3) → 6,000 (word, next) pairs; "." is only ever a target

          model one-hot(word) → h = W₁x + b₁ (d units, no activation) → softmax(W₂h + b₂) over 9 tokens

          embedding word i ↦ column i of W₁ (bias shared by all words, left out of cosine)

          loss cross-entropy · optimiser Adam, lr 0.005, batch 128, 500 steps

          best possible loss = entropy of the next token given the word, from the corpus counts

          2-D view PCA on the 8 embeddings, signs kept steady between frames

          code plain JavaScript, hand-written backprop, runs in your browser: ml.js · session-1.js

      

    

    
end of S1-3next: S1-4 · generalisation ↓

    


    

      

        
04assignment 4 of 4 · S1-4 · generalisation

        
More data closes the gap.

        
A big network can memorise a handful of points, noise and all, and still be wrong about everything else. Give the same network more data and memorising stops paying; it has to learn the actual pattern.

      

      
From S1-3: a model learns whatever its data supports. Give a big model too little data and it learns the data itself, noise and all.

      

        
concept · why: a big model can memorise a few points, noise and allanimated

        

      

      
the experiment



    

      
claimA high-capacity model on tiny data drives train loss to about zero while held-out loss stays high. Growing the dataset closes the gap.

      
buildNoisy 2-D points split by a sine curve, 10% of labels flipped. Train the same 2 → 64 → 64 → 1 ReLU net on 20, 200 and 2000 of them. Test all three on 1000 separate points.

      
proofAt 20 points, train loss hits zero and test loss is large. The gap (test loss − train loss) shrinks at 200 and nearly vanishes at 2000.

    


    

      

        
n = 20train points

        
        
–train acc
–test acc

      

      

        
n = 200train points

        
        
–train acc
–test acc

      

      

        
n = 2000train points

        
        
–train acc
–test acc

      

    


    

      
      
      seed 7 · 2 → 64 → 64 → 1 ·  params · dashed curve: the true boundary · ringed: misclassified
    


    

      

        
n = 20n = 200n = 2000test losstrain lossstep 0 / 2000 · log scale

        
      

      

        
generalisation gap = test loss − train lossvs training-set size, log scale

        
      

    


    

      
      
verdictPress train. The same net trains three times, on 20, 200 and 2000 points, with the same optimiser and 2000 steps each. All three are scored on the same 1000 held-out points.

    


    

      

        
What to notice

        
    
          
  • At n = 20 the net has about 220 parameters per point. It bends the boundary around single points, flipped labels included, and the test loss climbs as it grows more confident.
    • 
          
  • At n = 2000 it can't afford a bend per point, so the boundary settles onto the dashed sine. Train and test loss meet.
    • 
          
  • Neither loss reaches zero at 2000: with 10% of labels flipped, the best possible loss is about 0.33 and the best accuracy about 90%. That floor is the noise, not the model.
    • 
        

      

      

        
How it's built

        
data points uniform in [−2, 2]², class = above y = 0.9 sin(1.7x), 10% of labels flipped

          train one 2000-point draw; n = 20 and 200 are its first points (nested) · test 1000 separate points

          model 2 → 64 → 64 → 1, ReLU, sigmoid output, same init for all three · no regularisation

          loss binary cross-entropy · optimiser Adam, lr 0.01, minibatch 32 (full batch at n = 20), 2000 steps

          code plain JavaScript, hand-written backprop, runs in your browser: ml.js · session-1.js

      

    

    
end of S1-4back to top ↑