Ask again, and strict or forgiving

The pretend model, and the retry loop

To build and test all this without paying for real calls, the example code includes a pretend model called FlakyModel. "Flaky" means unreliable, which is exactly its job. It is not a real model. It is a small function that, on its first call, replies with one chosen mistake from the list above, and when asked again, replies with a good answer. That lets the tests make every kind of failure happen on demand.

Now the loop. When a reply fails, send the problem back and ask again:

outcome = get_structured(call, "Customer was charged twice", Ticket, max_attempts=3)
outcome.value      # a Ticket, or None if every attempt failed
outcome.attempts   # how many calls it took
outcome.errors     # the kinds of failure seen on the way

Here call is whatever function talks to the model; in the tests it is a FlakyModel. On a failure the loop adds two messages to the conversation: the model's bad reply, and a message from the user quoting the error and saying "Reply again with only the corrected JSON object." Running the example with the wrong-enum mistake:

mistake         first parse    attempts  note
wrong_enum      schema                2  JSON did not match the schema: field 'priority': Input should be 'low'...
truncated       truncated             2  JSON object is not closed (truncated?)
no_json         no_json               2  no JSON object found

The first parse failed, the second attempt succeeded, so attempts is 2. A test checks that the error text really reaches the model. Quoting a specific error works much better than repeating the same request and hoping.

Four rules for the loop:

1. Cap the attempts, usually at two or three. Every retry costs a call and some waiting (module 6).
2. Return a clear failure when the attempts run out, and let the caller decide: pass it to a person, use a default, or stop. Never return half-checked data.
3. Log the kinds of failure. If truncated is common, raise max_tokens. If one field keeps failing, its name or its allowed values are probably unclear in your prompt.
4. Lower the temperature (Level 1, module 4) for extraction tasks, because you want the most likely answer, not a creative one.

Strict or forgiving

By default Pydantic is forgiving: it converts values that look compatible. The text "yes" or the number 1 both become True for a bool field. That is convenient, and sometimes dangerous, because it can hide a model's mistake. The example shows both behaviours:

class LaxFlag(BaseModel):    ok: bool          # forgiving: "yes" and 1 are converted to True
class StrictFlag(BaseModel): ok: StrictBool    # strict: only a real true or false is accepted

LaxFlag.model_validate({"ok": "yes"})     ->  ok=True
StrictFlag.model_validate({"ok": "yes"})  ->  Input should be a valid boolean

Choose strict types for fields where a wrong guess has consequences, such as flags that trigger actions, IDs and amounts of money. Use forgiving types where being tolerant is helpful.