Dozens of models, new ones every month. A four-step process that finds a good choice without trying them all.
The problem: too many models, one decision
You need a model for a feature. There are dozens, new ones appear every month, and every provider says theirs is best. Trying them all by hand is slow, and picking the most famous one is a gamble. The way out is a process where cheap, fast steps remove most options and the expensive step (testing) is done only on a few.
Filter, shortlist, test, monitor. The last step never finishes.
Filter out models that break a hard requirement.
Shortlist three to five with public information.
Test them on your own cases, and decide.
Monitor after launch, because models change.
Hard attribute
A requirement you cannot or will not change: where data may be sent, the licence you can accept, the minimum context window, the countries where you must operate. A model that fails one is out, however good it is.
Soft attribute
A property you can improve with effort: accuracy through prompting or fine-tuning, tone, cost through caching. Do not discard a model on a soft attribute before you have tried to improve it.
The same property can be either. Latency is a soft attribute if you host the model and can tune it, and a hard one if someone else runs it.
Step 1: filter by hard attributes
Write the hard attributes down first, with a reason for each. Typical entries:
Hard attribute
Example question
Data handling
May customer data leave our network? Is there an agreement that the provider does not train on our data?
Licence
May we use it commercially? Can we modify it?
Context window
Is 32,000 tokens enough for our longest realistic input?
Region
Is there an endpoint in a permitted region?
Modalities and tools
Do we need images, tool calls or structured output?
In the example code, filter_hard(cards, need_open_weights=True) removes everything except open-weight models. With the four simulated models, only open-14b remains. (An open-weight model is one whose trained numbers you can download and run yourself.)
Check yourself
Give one hard and one soft attribute for an email-drafting assistant.
Why do you filter before you test?
Step 2: shortlist with public information
Public benchmarks (standard test sets) and leaderboards (rankings on them) give you a first cut. Use them to choose three to five candidates, not to choose the winner.
Why public scores mislead
Benchmarks measure someone else's task, which may not resemble yours. They can also be contaminated. And a ranking says nothing about your price limit, speed, or constraints.
Data contamination
A model was trained, knowingly or not, on data that includes the test questions or answers, so its score reflects memory rather than ability. Also called leakage.
The example has a crude detector. contamination_rate flags a test item when it shares a run of eight words with the training text. Real return values of the function on a made-up training text and two test items:
leaked sentence (copied from the training text): flagged -> rate 1.0
fresh question (not in the training text): not flagged -> rate 0.0
both together: rate 0.5
Real detection is harder, since paraphrases slip past a word-run check, but the principle is the same: keep your own test set out of anything a model may see, including prompts you paste into public tools.
Check yourself
Why use a benchmark to shortlist rather than to decide?