Definitions

Definitions

The method

1. List the criteria that matter to the user and the business, with a unit each.
2. Apply hard requirements and drop what fails.
3. Drop dominated options.
4. Compare what is left with a break-even or a weighted score using a scale fixed in advance.
5. Test the sensitivity: weights, scores, critical weights, margin.
6. State the evidence and its source, and label assumptions.
7. Write the decision and the reversal trigger.

Results

1. The multi-agent system is not "better" or "worse"; it wins above a traffic mix and loses below it.
2. Constraints did most of the work: weighting only separated two near-equals.
3. A tie in the matrix is information: it says the decision is driven by something you have not measured.
4. Uncertainty in the weights changes the confidence, not the answer, but only because both survivors were close.
5. Scaling is a modelling choice that can change winners; fix it in advance.

Design rules

• State the criteria and the hard requirements before scoring.
• Remove failing and dominated options first.
• Use a break-even when the choice depends on one quantity such as traffic mix.
• Fix the scale in advance; do not rescale by the options present.
• Report the margin and the critical weight, not only the winner.
• Label every number: measured, quoted from a source, simulated or assumed.
• Write the reversal trigger as a measurable condition.
• Revisit decisions when the evidence or the models change.
• Prefer the simpler option unless a measured benefit justifies the complexity.

Interview questions, with the follow-ups

Common mistakes

• "It depends" with no number behind it.
• Weighting before filtering by hard requirements.
• Hiding the argument in the weights and presenting the score as objective.
• Rescaling by the options present and getting rank reversals.
• Presenting an assumed number as measured.
• No reversal trigger, so the decision can never be wrong.
• Counting only the benefits of your choice.
• Choosing the complex option without a measured benefit.

What the example does not do

• No stakeholders: one weight vector, no negotiation.
• Few criteria: three in case B; real decisions include risk, team skill and maintenance.
• Additive scoring: it assumes criteria trade off linearly, which fails for hard limits (that is why they are filtered first).
• Assumed inputs: quality values and prices are mine; the case A numbers are illustrative and not from any measurement I made.
• No uncertainty on latency and cost beyond the optional score noise.
• Optional experiment (I did not run this): measure your own tool-heavy share of traffic from a month of logs and place yourself on the break-even chart.

For the curious A weighted matrix hides the argument in the weights

The winner of my two surviving options won 55 to 77 percent of the time as I became surer of my weights, because the two scored 0.520 against 0.515. The decision flipped when the p95 weight fell below 0.37; asking how wrong the weights would have to be beats asking for the right weights. With min-max scaling, adding a dominated option flipped the winner; scaling against fixed targets did not. Case A uses illustrative numbers and case B uses simulated latency and assumed quality.

What you must be able to say (one page)

• Use one sentence shape for every decision: "I chose X over Y because [the criterion that mattered most]; it costs [Z]; the evidence is [number, with its source]; I would switch if [trigger]." If you cannot fill all four slots, you have an opinion, not a decision. The sentence shape is my own.
• Constraints first, trade-offs second. A trade-off exists only among options that meet your hard requirements. In my example (module 3's design variants) requirements of p95 at most 8 s and quality at least 0.72 removed 3 of 5 options before any weighting.
• Turn a vague preference into a break-even. Take illustrative numbers (invented for this exercise, resembling a small comparison I cannot cite or verify): a direct model scores 9.7 against a multi-agent system's 9.3 on simple reasoning, but 6.8 against 9.2 on tool-heavy tasks, at 355 against 15,343 tokens per task (43 times). From those numbers, the multi-agent system has the higher mean score once more than 14.3% of traffic is tool-heavy (0.4 / 2.8), and at an assumed $1 per million tokens the extra tokens cost about $0.015 per task, so at 50% tool-heavy traffic a score point must be worth more than about $0.015 to justify it. The scores and the price are assumptions; the score is a judge's 0 to 10, not a probability.
• A weighted matrix hides the argument in the weights. Among my two surviving options, the winner won 55%, 60%, 65% and 77% of the time as I became more confident in my weights (concentration 5, 20, 50, 200). Even sure weights left a 23% chance for the other option, because the two scored 0.520 against 0.515.
• Find the critical weight. The decision flipped when the weight on p95 latency fell below 0.37 or the weight on quality rose above 0.43; cost weight never flipped it. "How wrong would my weights have to be?" is a better question than "what are the right weights?"
• When the margin is thin, measure. The winner's assumed quality was 0.73 and it would lose at 0.728: a margin of 0.002 on a number I invented. The right next step is to measure quality, not to debate weights.
• Pick the scale before you see the results. With min-max scaling (rescaling by the options present), adding a dominated option flipped my example's winner from A to B at weights 0.6 / 0.4. Scaling against fixed targets did not. Min-max with only two options also ties at equal weights, so the weights alone decide.
• Write it down as a decision record. My seven-field template (status, context, decision, alternatives, evidence, consequences, revisit-when) found 7 flags in a weak record, 4 in a partial one and 0 in a complete one. The most useful field is "revisit when": a measurable reversal trigger.
• General rules that help in a design review. Choose a pattern by asking "what does my task need?", not "which is best?"; control and autonomy move inversely. Trade-offs among performance, scalability, reliability and cost must be revisited as needs change. Adding a judge trades risk against latency; parallel plans trade latency against cost; detailed plans are harder to make and easier to execute. Keep it simple unless there is a compelling reason.
• Honest limits: the matrix is a thinking aid; two of my inputs are assumed (the quality values and the token price); the case A numbers are illustrative, not measured; real decisions have more than three criteria and more than two stakeholders.