Part 3 of 4 · 2 min read
Exercises
Easy
Easy 1: Run it (30 minutes)
Run python -m pytest -q and python run_all.py in code/mini-lora.
Done when: you can explain each row of the rank-sweep table.
Easy 2: Count parameters (30 minutes)
Compute LoRA parameters for a model with 32 layers, d=4096, rank 16 on four matrices and on all attention plus MLP matrices.
Done when: you give both counts and the fraction of a 7B model.
Hint
per matrix the count is 2 x d x r for a square d x d matrix.
Easy 3: Why B = 0 (30 minutes)
Set both A and B to zero in lora_init and train for 50 steps.
Done when: you can explain why the error does not move.
Hint
with A and B both zero, the gradient of each depends on the other.
Medium
Medium 1: More data (3 hours)
Raise n_train to 1,024 and 4,096 and rerun the rank-4 sweep.
Done when: you report whether full tuning overtakes LoRA and at what size.
Hint
keep the validation and test sets fixed while changing training size.
Medium 2: Seeds (3 hours)
Repeat the rank sweep over 10 seeds and report mean and spread.
Done when: you can say which differences are larger than seed noise.
Hint
change the seeds of data generation and of A's initialisation.
Medium 3: Real NF4 levels (3 hours)
Replace my normal-quantile table with the NF4 table from the QLoRA paper and compare errors.
Done when: you report the difference against my approximation.
Hard
Hard 1: PyTorch LoRA on a small model (8 hours)
Reproduce the rank sweep on a small open language model using PEFT on one agent step. (I did not run this, so your numbers are the ground truth.)
Done when: you plot validation quality against rank and trainable parameters.
Hint
tune on later-period data to detect overfitting.
Hard 2: QLoRA versus 16-bit (8 hours)
Compare 16-bit LoRA and 4-bit QLoRA on quality, memory and speed. (I did not run this.)
Done when: you report all three with your hardware.
Hard 3: Multi-LoRA serving (8 hours)
Serve three adapters on one base and measure latency against merged models. (I did not run this.)
Done when: you report latency and storage per adapter.