The Step That Had to Wait: asyncio Locks on a Warm Lambda

One of my workflows reads a receipt or invoice image with two steps that start at the same time. One runs OCR on the image. The other asks a vision model to pull out the fields. A third step merges what the two return. At the end of September, on our development environment, the vision step started failing in a strange way. On each worker, the first call succeeded and took 27 seconds. Every later call on the same, now warm, worker failed in about 30 milliseconds, before any request reached the tool server. In the same tasks, the OCR step worked fine.

The cause was a semaphore created once per process. An asyncio lock or semaphore binds to an event loop the first time a coroutine has to wait on it. Our Lambda handlers start a new event loop for every invocation, but module globals stay alive between invocations. So after the first invocation the semaphore stayed tied to a loop that no longer existed, and only a step that had to wait for a slot noticed. Here that was always the vision step.

Three things here apply to anyone running agent steps concurrently. A global asyncio primitive fails only for a coroutine that has to wait on it, so the bug looks like one broken step rather than a broken runtime. You can't recognise an event loop by its id(). And if your step runner gathers with return_exceptions=True, log the traceback before the exception turns into a value. Ours didn't, and I fixed a different bug first.

What we had to go on

The failures showed up when we started scoring our workflows on public datasets, using one direct model call on the same file as the control. On the receipt track, only 2 of 45 runs produced any output. Here are the worker logs for one burst of calls:

Size wasn't it: the failing images were smaller than the passing one. Beyond that, the log had only the type, RuntimeError, and the task runner's failed: None, with no message. Steps that can run in parallel form a layer, and a layer runs its steps under asyncio.gather(..., return_exceptions=True). That turns an exception into a return value, and the event we record for a failed step kept only its type. We had to rebuild the timeline from each step's start and end events.

The first fix was a real bug

The first guess was an async client reused after its event loop had closed, so I looked at the client our steps use to call tool servers. It was cached in a module global and rebuilt whenever the event loop changed. To decide whether the loop had changed, it compared id(loop), and that check is broken. Once an object is freed, CPython is free to give its id to the next object, so a new loop can get the dead loop's id. The cache then returns a client whose connection pool belongs to a closed loop. CPython makes no promise either way, but on my machine five loops, created and closed one after another, all got the same id.

We changed the cache to hold the loop through a weak reference and to compare loops by identity. The new test forces every id() call in the module to return one value, and it fails on the old code. We deployed the fix to development. The vision step still failed in exactly the same way, and the log still had no traceback.

The step that had to wait

So I read the step's code path from the start. Before a step calls its tool, it takes a slot from a concurrency limiter. It does this twice: once in the layer runner, and again in the step executor it calls. Both got the limiter from get_concurrency_limiter(), which returned one instance per process. Inside that instance are an asyncio.Lock and one Semaphore per key, and this workflow got the default limit of three.

Two steps that each take two slots need four. The OCR step gets both of its slots, the vision step gets its first, and its second acquire has to wait for OCR to finish. That happens on every run, not only under load.

Since Python 3.10, these primitives don't take a loop argument. Each one binds to the running loop the first time it needs one. A semaphore only needs a loop when a coroutine has to wait, because taking a free slot is a fast path that never touches the loop. On the first invocation in a fresh container, the waiting vision step binds the semaphore to that invocation's loop and succeeds. On every later invocation the vision step waits again, and the semaphore's loop has already ended. The probe below reproduces it with the same slot counts. Each asyncio.run plays one warm invocation:

Python 3.13.3
invocation 1: ['ok', 'ok']
invocation 2: ['ok', 'RuntimeError: <asyncio.locks.Semaphore object at 0x000002497355ABA0 [locked]> is bound to a different event loop']
invocation 3: ['ok', 'RuntimeError: <asyncio.locks.Semaphore object at 0x000002497355ABA0 [locked]> is bound to a different event loop']

That message names the semaphore. If it had been in our log, we would have gone straight to the limiter.

We fixed it the same way as the client: one limiter per running loop, with the loop held through a weak reference and compared by identity. The limiter was documented as protection within one process, and nothing relied on its state carrying over between invocations, so no other code was affected. The step executor now also logs the exception with its traceback before re-raising it.

Then we re-ran the same receipt track on development, on a build carrying both fixes. This time 42 of 45 runs produced output, up from 2. Field-level F1 went from 5.6% to 83.0%, against 70.7% for the single direct call. The three runs that still failed all used one 10.8 MB image, too large for the way we passed images at the time, which a later post covers.

Taking two slots per step turned out to be a problem of its own, and we fixed it later. The step executor awaits its idempotency check between the two acquires, so a layer of three parallel steps on a limit of three could deadlock: each step holds one slot and waits for a second. Now only the layer runner takes a slot. A test with layers of three and five real steps hangs on the old code and passes now.

Other module-level locks

We searched for other module-level asyncio primitives and found three more asyncio.Locks. One guarded a health-check cache, and the other two guarded caches of embedding vectors. None had failed yet. Each one guards a double-checked cache with an await inside the lock, so we couldn't simply remove the locks. They now use a small LoopLocalLock, which hands out the lock for the current loop.

The first version of LoopLocalLock stored one lock per loop in a WeakKeyDictionary, so that finished loops would be garbage-collected. Code review ran it: the loops weren't collected. A lock that has been waited on keeps a strong reference to its loop. Each lock is a value in the dictionary, so it keeps its own key alive, and every loop that ever waited on a lock stayed in the warm container. The probe shows this with plain asyncio: three runs whose locks were never waited on leave no loop behind, and three runs whose locks were waited on leave all three.

The final version keeps a single slot: the current loop, held through a weak reference, and that loop's lock. When a new loop asks, it replaces both. This assumes only one event loop is running at a time, which holds in a standard Lambda container because it serves one invocation at a time. A second loop running on another thread at the same time would quietly get its own lock.

To stop new ones appearing, a test parses every module under app/ and fails on any top-level assignment of asyncio.Lock, Semaphore, BoundedSemaphore, Event or Condition. A second test runs the same scan on a small sample that contains two of them and checks that it reports both lines, so the scan can't pass by matching nothing.

import asyncio, gc, platform, weakref

print(f"Python {platform.python_version()}")

# One semaphore per process, as the old get_concurrency_limiter() returned.
# A module global survives across asyncio.run() calls the way it survives
# across warm Lambda invocations.
LIMIT = asyncio.Semaphore(3)

async def step(name):
    async with LIMIT:          # the layer runner takes a slot...
        async with LIMIT:      # ...and the step executor takes another
            await asyncio.sleep(0.01)
    return name

async def invocation():
    results = await asyncio.gather(step("ocr"), step("vision"), return_exceptions=True)
    return ["ok" if isinstance(r, str) else f"{type(r).__name__}: {r}" for r in results]

for i in (1, 2, 3):
    print(f"invocation {i}:", asyncio.run(invocation()))

# id() of a dead loop is free to be reused by the next one.
ids = set()
for _ in range(5):
    loop = asyncio.new_event_loop(); ids.add(id(loop)); loop.close(); del loop
print("distinct id() over 5 loops created one after another:", len(ids))

# A lock that has been waited on keeps its loop alive; one that never was doesn't.
def alive_after(contend):
    locks, refs = weakref.WeakKeyDictionary(), []
    async def run():
        lock = locks.setdefault(asyncio.get_running_loop(), asyncio.Lock())
        async def hold():
            async with lock:
                await asyncio.sleep(0.01)
        await (asyncio.gather(hold(), hold()) if contend else hold())
        refs.append(weakref.ref(asyncio.get_running_loop()))
    for _ in range(3):
        asyncio.run(run())
    gc.collect()
    return sum(r() is not None for r in refs)
print("closed loops still alive, locks never waited on:", alive_after(False))
print("closed loops still alive, locks waited on:      ", alive_after(True))

distinct id() over 5 loops created one after another: 1
closed loops still alive, locks never waited on: 0
closed loops still alive, locks waited on:       3

What the scan doesn't see

The scan only finds the five names, called as asyncio.X(), at the top level of a module. asyncio.Queue and Barrier bind to a loop too and aren't on the list, and neither is a bare Lock() imported with from asyncio import Lock. It also can't see the limiter's own shape: a module global holding an object whose constructor creates the primitives. The limiter is covered by its per-loop factory and that factory's test, not by the scan.

A server that keeps one event loop for its whole life never runs into this. You hit it when something creates a new loop while module globals stay alive: our Lambda handlers calling asyncio.run per invocation, a test runner that makes a loop per test, or a thread that starts its own loop.