Async and concurrency

In this module you will

Explain why waiting is the main cost in an agent, and what async and await do about it.
Run many model or tool calls at once, with a cap, so that you stay inside rate limits.
Handle one failure, a timeout, or a "stop" request without leaving work half-done.
Spot blocking code hidden inside async code, and fix it.

Why this matters for agents

An agent spends most of its time waiting: for a model to answer, for a search to return, for a tool to finish. If you wait for each call before you start the next, a task with twenty calls takes twenty times as long as one call. Agents that fan out (several searches at once, several sub-agents in parallel) need a way to wait for many things at the same time.

The interview question for this module: what goes wrong when blocking code runs inside an async function? By the end you will be able to answer it with a measured example.

The problem: waiting in line

Imagine you are cooking dinner. You put rice on the stove and it needs 20 minutes. A sensible cook does not stand and stare at the pot. They chop vegetables while the rice cooks, and come back when it is ready. Nothing runs faster. The cook simply uses the waiting time.

A program that calls a model is like the cook staring at the pot. It sends a question, then does nothing for a second or two until the answer comes back. With three questions, plain code looks like this:

def slow_question(q):          # an ordinary function: waits, then answers
    time.sleep(1)
    return f"answer to {q}"

print([slow_question(q) for q in "abc"])

['answer to a', 'answer to b', 'answer to c']
plain code: 3.0s

Three questions of one second each take three seconds, because each one waits for the one before. But the three questions are independent. They could all be "on the stove" at once. That is what concurrency means, and it is what async gives you. First, some words.

Concurrency

Making progress on several things during the same period by switching between them, like the cook switching between rice and vegetables. It is not the same as parallelism, which means several things running at the exact same instant on several CPU cores, like having two cooks.

I/O-bound and CPU-bound

Work is I/O-bound (input/output) if it mostly waits: for the network, a disk, a model's answer. Work is CPU-bound if it mostly computes: parsing a huge file, or embedding text with a local model. async helps with I/O-bound work. It does nothing for CPU-bound work, because a CPU that is busy computing cannot also be switching.

Calls to models and tools are I/O-bound, so async fits agents well.

Check yourself

Why does the plain program above take three seconds and not one?
Is calling a model I/O-bound or CPU-bound? What about resizing 10,000 images?
In the cooking story, what is concurrency and what would parallelism be?

Overview

Why this matters for agents

The problem: waiting in line