Timeouts, groups and cancellation

Timeouts

Never wait forever for something external. Python 3.11 added asyncio.timeout:

async def with_deadline(call, prompt, seconds):
    async with asyncio.timeout(seconds):
        return await call(prompt)         # raises TimeoutError if it takes too long

>>> await with_deadline(fake_model, "slow", 0.01)
TimeoutError

Put the timeout around the whole operation you care about, and still use the per-request HTTP timeouts from module 3.

Task groups: all or nothing

async def all_or_nothing(prompts, call):
    async with asyncio.TaskGroup() as group:
        tasks = [group.create_task(call(p)) for p in prompts]
    return [t.result() for t in tasks]

If any task raises, the group cancels the others and raises an ExceptionGroup that holds the failures. Python 3.11 also added the matching except* syntax to catch them:

>>> try:
...     await all_or_nothing(["ok 1", "bad one", "ok 2"], fake_model)
... except* RuntimeError as eg:
...     print([str(e) for e in eg.exceptions])
["model failed on 'bad one'"]

Use a task group when the results only make sense together, for example three sub-queries that you must combine. Use gather(..., return_exceptions=True) when each result stands alone.

Cancellation

A running task can be cancelled: by a timeout, by a task group, or by task.cancel(). The cancellation arrives inside the task as an asyncio.CancelledError, raised at the await where the task was paused. Two rules:

1. Release resources in a finally block: close files and connections, mark the job as stopped.
2. Do not swallow CancelledError. Let it continue upward, or the program cannot shut down.

async def cancellable_worker(log):
    try:
        log.append("started")
        await asyncio.sleep(10)
    finally:
        log.append("cleaned up")      # runs even when the task is cancelled

Starting the worker, cancelling it after 0.05 seconds and then awaiting it raises CancelledError, and the log shows the cleanup still ran:

cancelled; log = ['started', 'cleaned up']

For agents this matters whenever a user presses "stop": every in-flight model call and tool call must be cancelled cleanly, and not left running and billing you.