Definitions

Definitions

The pipeline

Results

1. A scope sentence with a number and a baseline is the cheapest quality gate.
2. Estimates are medians; totals and tails are longer than the sum.
3. Shared causes of delay matter more than task-by-task error.
4. The cut line belongs to the plan, not to the last week.
5. Each extra point of quality costs more time than the last.
6. A hundred labelled items is a floor for claiming a number.

Design rules

• Write the scope sentence (user, job, result, baseline) before opening an editor.
• Choose a user you can reach and show them early.
• Build the baseline and the evaluation set first.
• Fix the usefulness threshold for quality, latency and cost, and name what you will not do.
• Estimate in hours with a range, add a shared buffer and cut to the 80th percentile.
• Order features by priority, with the evaluation harness always core.
• Plan the human's role (suggest, route, or automate) and a staged rollout.
• Re-check scope at week 3 and move the cut line only by swapping work.
• Record what you cut so you can discuss it.

Interview questions, with the follow-ups

Common mistakes

• No baseline, so there is nothing to beat.
• Several users or jobs in one project.
• A target without a number or an evaluation set too small to show it.
• Estimating in medians and planning to the median.
• Building features before the evaluation harness.
• No cut line until the last week.
• Adding an agent where a rule would do.
• Never showing a real user.

What the example does not do

• No real project: the capstone, hours and spreads are invented; your own tasks will have different shapes.
• Independent task durations plus one shared factor; real tasks have dependencies, blocking and rework.
• The last-mile curve uses two points from one reported case and assumes exponential decay; it is a way to think, not a forecast.
• Item counts use a normal approximation and an assumed disagreement rate.
• The checker's text rules are crude: a vague user can slip past them and a good one can trip them; it is a prompt for questions.
• No budget model beyond a per-task cap, no user research method, no market analysis.
• Optional real experiment (I did not run this): write your own scope, estimate each task with a low, likely and high hour figure, track actuals for four weeks and compare them with the simulated 50th and 80th percentiles.

For the curious Plans slip, so cut by confidence and not hope

In my fictional plan (11 features, 116 hours of estimates against 80 available), the full plan had a 5 percent chance of finishing in 80 hours; core plus important 54 percent; core only 88 percent. A shared project-wide delay factor widens the tail. Keeping features until the 80th percentile fits left 4. The spreads, hours and the one-anecdote last-mile curve are assumptions; nothing here was built.

What you must be able to say (one page)

• A capstone is one sentence. "For this user, my agent does this job, and I can show this measured result against this baseline." If the sentence has two users, two jobs or no number, the project is not scoped yet. The sentence format is my own.
• Decide whether and why before you build: it is easy to build a demo and hard to build a profitable product. Classify the feature: critical or complementary (the more critical the AI, the higher the accuracy bar), reactive or proactive (proactive features need higher quality because users did not ask for them), dynamic or static; and decide the human's role: suggestions for a person, simple cases automated with the rest routed to humans, or full automation. One staged approach is Microsoft's Crawl-Walk-Run (human involvement mandatory, then internal users, then external users) and an example gate: when humans accept about 95 percent of suggested replies verbatim, let the AI answer those directly.
• Set the usefulness threshold first: business impact (for a support bot: share of messages automated, time saved, satisfaction), then quality, latency, cost and other metrics. Start from what existing models already do; the stronger the off-the-shelf model, the less work remains. Your goals may also change after evaluation, including dropping the project.
• Expect the last mile: a demo can take a weekend and a product months. LinkedIn's engineering team reported (from memory; I did not re-check) that it reached 80 percent of the experience in one month and needed four more months to pass 95 percent. Using only that anchor and assuming the error shrinks by a constant factor per unit time (my extrapolation), the error halves every 2 months, giving 3 months to 90 percent, 5 to 95, 7.6 to 98 and 9.6 to 99 percent. Every extra point costs more than the last, so the threshold you choose is a schedule decision.
• Start small and cheap: an MVP that "works and delivers value, even if it's basic", then feedback and iteration; weigh cost against value, and prefer a simpler or rule-based solution when it does the job. A prototype may be enough for non-critical uses, but many uses need accuracy and reliability guarantees it cannot give. So every capstone needs a non-agent baseline (rules, a single prompt, or the human process) to beat.
• Plans slip; model the slip. I wrote a fictional capstone (a support-ticket reply drafter) with 11 features and a median total of 116 hours against 80 available (four weeks at 20 hours). Drawing each feature from a lognormal around its estimate (spread 0.5, assumed) plus a shared project-wide factor (0.25, assumed): the plan with everything has a median of 130 h and a 5 percent chance of finishing in 80 h; core plus important (70 h of estimates) has a median of 78 h and 54 percent; core only (48 h) has a median of 52 h, an 80th percentile of 71 h and 88 percent. With the shared factor removed those become 0.1, 56 and 94 percent: shared causes of delay widen the tail.
• Set a cut line by confidence, not hope. Keeping features in priority order until the 80th percentile fits in 80 h leaves 4 features (the four core ones, including the evaluation harness); at 50 percent confidence 7; at 90 percent 3. Decide the line before starting, and move work above it only by removing work from it.
• Size the evaluation set to the claim. For a paired comparison with 80 percent power, a gain from 45 to 70 percent needs about 45 items, 60 to 75 percent 77, 50 to 60 percent 155. But a score from 100 items has a 95 percent interval about 18 points wide (25 at 50 items, 15 at 150, 10 at 300), so I set a floor of 100 items in the checker. These are my numbers.
• Check the scope with a checklist. My check function flags a vague user, a multi-part job, a missing baseline or target, an evaluation set that is too small, no simple baseline, no cost or latency limit, too many core features and plans that exceed the hours. On three fictional scopes it raised 5, 4 and 0 flags.
• A four-week shape (my own suggestion): week 1: pick the user, write the scope sentence, build the baseline and the evaluation set; week 2: core path end to end; week 3: improve against the evaluation set and decide the cut line; week 4: write-up, demo, public release. Reaching a real user for feedback matters more than another feature.
• What to say in an interview: "I scoped it to one user and one job, wrote the target and the baseline before building, sized the evaluation set to the gain I wanted to show, estimated hours with a spread and cut to fit the 80th percentile, and compared against a simple non-agent baseline."
• Honest limits: the hours, spreads and example capstone are invented; the last-mile curve extrapolates one reported case; the checker uses crude text rules; no real project was scoped or built here.