Part 3 of 4 · 2 min read
Exercises
Easy
Easy 1: Run it (30 minutes)
Run python -m pytest -q and python run_all.py in code/mini-inference.
Done when: you can explain each table in the results.
Easy 2: Cache size by hand (30 minutes)
Compute the KV cache for a model of your choice at batch 8 and sequence 4,096, in 16-bit and 8-bit.
Done when: your numbers match kv_cache_bytes.
Easy 3: Expected tokens (30 minutes)
Compute (1 - a^(K+1)) / (1 - a) for a = 0.6, 0.8 and K = 2, 4, 8.
Done when: your numbers match expected_tokens.
Medium
Medium 1: Max batch for your GPU (3 hours)
Change the memory constants to a GPU and model you can access and recompute the largest batch for three sequence lengths.
Done when: you report the batch and which term (weights or cache) dominates.
Medium 2: Heavier-tailed requests (3 hours)
Raise the length spread (sigma) in the batching simulation and compare static and continuous p95.
Done when: you plot the gap against sigma.
Medium 3: Break-even draft (3 hours)
For a draft cost c, find the smallest acceptance rate at which speculative decoding gives a 1.5x speed-up.
Done when: you give a table for c = 0.05, 0.1, 0.2, 0.4.
Hard
Hard 1: Measure a real server (8 hours)
Serve an open model, measure TTFT, TPOT and throughput versus batch and compare with the formula. (I did not run this, so your numbers are the ground truth.)
Done when: you report where the memory-bound model breaks.
Hard 2: Quantisation on a real model (8 hours)
Quantise weights and cache of a small open model separately and measure task accuracy and perplexity. (I did not run this.)
Done when: you report which tolerated fewer bits and compare it with the toy.
Hard 3: Speculative decoding in PyTorch (8 hours)
Implement draft-and-verify with two real models and measure acceptance rate and speed-up on your prompts. (I did not run this.)
Done when: your measured speed-up matches the formula within a stated tolerance, or you explain the gap.