Fan-out with a limit

Running 100 calls at once will hit the provider's rate limit (Level 0, module 3). You want many calls at once, but not too many. A semaphore is a counter that lets only a fixed number of jobs through at the same time.

async def fan_out(prompts, call, *, limit=5):
    semaphore = asyncio.Semaphore(limit)

    async def one(prompt):
        async with semaphore:                 # waits here if `limit` calls are running
            return await call(prompt)

    return await asyncio.gather(*(one(p) for p in prompts), return_exceptions=True)

"Fan-out" means one task spreading into many parallel calls, like a hand fanning open. The function takes a list of prompts and a call function (here fake_model) and returns all the answers. This is the real timing from the example project, for 20 prompts that each wait 0.1 seconds:

sequential: 2.01s
fan_out limit=5: 0.40s
fan_out limit=20: 0.10s

With a limit of 5, the 20 calls run in 4 rounds of 5 (4 × 0.1 s). With a limit of 20 they all run together. The limit is how you trade speed against the provider's rate limit.

Two details matter.

return_exceptions=True. Normally, when one job fails, the exception comes out of gather and you lose track of the others. With this setting, failures come back as values in the result list, next to the successes. Here is a run where the middle prompt fails (the word "bad" makes fake_model raise an error):

>>> await fan_out(["ok 1", "bad one", "ok 2"], fake_model, limit=2)
['answer to ok 1', RuntimeError("model failed on 'bad one'"), 'answer to ok 2']

One failure did not cancel the others. Check each result with isinstance(result, Exception) to tell them apart.

Order is preserved. results[i] always belongs to prompts[i], even though the calls finish in a different order.