Definitions

Definitions

The structure

Results

1. Rank must cover the change. Below the needed rank the adapter underfits; above it adds nothing here.
2. LoRA trades capacity for data efficiency; it can trail full tuning when the needed change is high-rank.
3. Scale and learning rate interact; tune them together.
4. The base dominates memory, so quantising the base matters more than shrinking the adapter.

Design rules

• Start with rank 8 to 16 on all attention matrices (add MLP matrices if quality is short) and sweep rank on a validation set.
• Initialise B to zero and A small random, so training starts at the base model.
• Tune alpha and the learning rate together; keep alpha : r within a sensible band and watch for divergence.
• Merge for one model; keep adapters separate for multi-tenant or multi-task serving.
• Version adapters with the base model, since an adapter is only valid for the exact base weights it was trained on.
• Use 4-bit bases (QLoRA) when memory is the limit, and compare against a 16-bit run on your own eval.
• Smaller quantisation blocks cost extra bits; measure error against memory.
• Hold out data from a later period to check the adapter does not overfit.
• Run a regression eval on tasks the adapter should not change.
• Recompute any parameter count you copy (tables sometimes label the matrices wrongly).

Interview questions, with the follow-ups

Common mistakes

• Raising rank hoping for better quality without a validation curve.
• Applying LoRA to a single matrix without testing more.
• Random B initialisation and a worse starting point.
• Changing alpha without re-tuning the learning rate.
• Mixing adapters and base versions.
• Assuming 4-bit is free; it costs accuracy that you must measure.
• Training on 200 examples with full finetuning and calling LoRA "better" (small data favours small adapters).
• Copying parameter counts without recomputing (Table 7-6 caption).
• Ignoring regression tests on tasks the base model did well.
• Treating a toy result as evidence about LLMs.

What the example does not do

• No real model: a 64-unit network, random base, and a target built by a low-rank change.
• The low-rank assumption is built in, so the rank-4 result is flattering by design; the full-rank case is the fairer stress test.
• Small data: 256 examples make full tuning overfit; with more data the gap may flip.
• One seed per configuration, with learning rates chosen from a small grid, so small differences (for example between rank 8 and 16) are not conclusive.
• My quantiser is an approximation (symmetric normal-quantile levels, no double quantisation, no paged optimiser).
• No attention, no real training dynamics, no mixed precision.
• Optional real experiment (I did not run this): with a small open model, train LoRA at ranks 4, 8, 16, 32 on one agent step with PyTorch and PEFT, plot validation quality against rank, then repeat with a 4-bit base and compare quality and memory.

For the curious Rank, alpha and learning rate interact

In my toy the needed change was rank 4 by construction, so r=4 won (0.0003) and r=1 underfit (0.0273). Raising r to 64 did not help. When the needed change was full rank, LoRA needed a high rank to match full tuning. At scale alpha/r = 4, lr 0.1 diverged while 0.01 worked, so alpha and learning rate are not independent. All of this is a small NumPy network, not a language model.

What you must be able to say (one page)

• LoRA in one line. For a frozen weight matrix W, learn two small matrices B (n x r) and A (r x m) and use W' = W + (alpha / r) B A. Only A and B train; r is the rank. In my code B starts at zero, so training starts exactly from the base model.
• Why it is cheap. A 64 x 64 matrix has 4,096 weights; at rank 4 the adapter has 512 (2 x 64 x 4). A 9 x 9 matrix (81 weights) factors into 9 x 1 and 1 x 9 (18 weights). For GPT-3 the LoRA paper reports about 4.7M trainable parameters, which I recomputed as 0.0027 percent of 175B.
• Why it can work. The assumption is that the change a task needs is low-rank. In my toy I made the needed change exactly rank 4 by construction, so this is an assumption I built in, not evidence about real models.
• Rank sweep, my toy (3-layer net, 64 hidden units, 256 training examples, normalised test MSE where 1 means "predict the mean", learning rate chosen on validation data). Needed change is rank 4 (untuned error 0.238): full tuning of that matrix 0.0061 (4,096 parameters), LoRA r=1: 0.0273, r=2: 0.0121, r=4: 0.0003, r=8: 0.0009, r=16: 0.0016, r=32: 0.0027, r=64: 0.0032. A rank below the true one underfits; the matching rank wins; more rank did not help and slightly hurt here, so raising r does not necessarily improve results and may overfit. LoRA beat full tuning because 256 examples cannot pin down 4,096 free weights (full tuning reached 0.0001 train error but 0.0061 test error). That is a statement about my small data, not a general law.
• When the needed change is full rank (untuned 0.158): full tuning 0.0048, LoRA r=1: 0.0767, 2: 0.0368, 4: 0.0135, 8: 0.0112, 16: 0.0069, 32: 0.0059, 64: 0.0057. LoRA needs a high rank to match full tuning. This is the usual catch: LoRA can trail full finetuning when the needed change is large.
• Which matrices and what rank in practice. In the LoRA paper's GPT-3 experiment with an 18M-parameter budget, spreading rank 2 over all four attention matrices did best (WikiSQL 70.4, 70.0, 73.0 and 73.2 percent for single matrices at rank 8; MultiNLI 91.0, 90.8, 91.0 and 91.3 percent). I am quoting these from memory and did not re-check the paper. A small r between 4 and 64 is often enough; one practitioner (Raschka) reported r=256 best on his tasks. Adding feed-forward matrices often helps.
• alpha. The update is scaled by alpha / r; common practitioner advice is to keep alpha : r between 1:8 and 8:1. In my toy at rank 8, the best learning rate moves with the scale: scale 0.25 was best at lr 0.03 to 0.1 (validation error 0.0005), scale 1 at lr 0.03 (0.0008), scale 4 at lr 0.01 (0.0014) and diverged at lr 0.1 (0.633). So alpha and learning rate are not independent; tune one with the other fixed.
• Why B starts at zero. With B = 0 the adapter adds nothing at step 0 (my test checks the matrix is unchanged). With a random B my toy started at error 0.344 instead of 0.238, so training had to undo a random perturbation first.
• Merging and serving. Merge B A into W for a single model (no extra latency), or keep adapters separate to serve many fine-tunes from one base. In my test the merged and unmerged outputs agree to 1.2e-15. Example: 100 customers, 4096 x 4096 matrix, r=8: merged copies need 1.68B parameters, one base plus 100 adapters 23.3M. I recomputed 1,677,721,600 and 23,330,816.
• QLoRA: store the frozen base in 4-bit NF4 (normal-distribution-shaped levels), de-quantise to 16-bit for compute, train LoRA on top, with paged optimisers. The QLoRA authors report finetuning a 65B model on a single 48 GB GPU, and their Guanaco models came close to ChatGPT on a GPT-4-judged benchmark (from memory; not re-checked).
• My 4-bit experiment (approximate: my level table is a symmetric normal-quantile table, not the real NF4). Round-trip error relative to the weight spread, normal weights: block 16: 0.080 uniform vs 0.081 normal-quantile; block 64: 0.101 vs 0.092; block 256: 0.118 vs 0.098; one block for the whole matrix: 0.182 vs 0.126, at 5.00, 4.25, 4.06 and 4.00 bits per weight (including a 16-bit scale per block). Smaller blocks cut error but cost bits; the normal-quantile table helps most when blocks are large, and at block 16 it did not help.
• LoRA on a 4-bit base in my toy (all frozen layers quantised at block 64, LoRA r=8): untuned error rose from 0.238 to 0.264 (uniform) and 0.254 (normal-quantile); after LoRA 0.0088 and 0.0069 against 0.0009 for 16-bit LoRA. So LoRA recovers almost all of the task but not all of the quantisation damage in my toy; the QLoRA authors report parity with 16-bit on their benchmarks, which I did not test.
• Memory is mostly the base, not the adapter. Llama 2 13B weights take 26 GB (13B x 2 bytes) against a 6.55 MB adapter (3.28M parameters x 2 bytes). Cutting adapter parameters barely changes total memory; quantising the base does (16 to about 4.25 bits per weight, about 3.8 times less in my arithmetic).
• Recount parameters yourself. Rank 2 on all four attention matrices gives 3,276,800 for Llama 2 13B (d=5120, 40 layers) and 18,874,368 for GPT-3 (d=12288, 96 layers); on only two matrices it is 1,638,400 and 9,437,184. Published tables sometimes mix up which matrices they count, so check any figure against the formula before you copy it.
• What to say in an interview: "LoRA freezes W and learns a low-rank update B A scaled by alpha over r, with B initialised to zero. It trains a tiny fraction of the parameters, can be merged for no latency or kept separate to serve many adapters on one base, and QLoRA adds a 4-bit frozen base to cut memory. I pick rank by the validation curve, usually 8 to 64, apply it to all attention (and often MLP) matrices, and expect it to trail full tuning on hard shifts."
• Honest limits: a tiny NumPy network with a built-in low-rank target; the quantiser is my approximation; no real language model was tuned.