Fire and forget (or never) with Python’s asyncio

TL;DR; Python’s asyncio.create_task() can silently garbage collect your fire-and-forget tasks starting in Python 3.12 - they may never run. The fix: store task references in a set and register a done_callback to clean them up.

Do you use Python’s async/await in programming? Often you have some async task that needs to run, but you don’t care to monitor it, know when it’s done, or even if it errors.

Let’s imagine you have an async function that logs to a remote service. You want its execution out of the main-line execution. Maybe it looks like this:

async def log_account_created(username: str): ...

Someone new to Python’s odd version of async would think they could write code like this (hint, they cannot):

async def register_user():
    data = get_form_data()
    user = user_service.create_user(data)
    
    # Log in the background (actually no, but we try)
    log_account_created(user.name) # BUG!
    
    return redirect('/account/welcome')

You cannot just run an async function, you fool! Why? I don’t know. It’s a massively needless complication of modern Python. You either have to await it (which would block the main execution foiling our fire and forget intention) or you have to start it as a task separately. Here’s the working version (at least in Python 3.11 it works, Python 3.12+? Sometimes):

async def register_user():
    data = get_form_data()
    user = user_service.create_user(data)
    
    # Log in the background?
    # Runs on the asyncio loop, fixed, maybe
    asyncio.create_task(log_account_created(user.name)) 
    
    return redirect('/account/welcome')

Why asyncio.create_task loses tasks in Python 3.12+

Actually that fixed version has a tremendously subtle race condition that was introduced in Python 3.12 (seriously). In Python 3.11 or before, the async loop holds the new task and will just run it at some point soon.

Here is the first line in the docs for create_task:

Wrap the coro coroutine into a Task and schedule its execution. Return the Task object.

It schedules its execution. But in Python 3.12, it might forget about it!

I call functions like log_account_created fire and forget async functions. You don’t care to wait for it or even check its outcome. What are you going to do if logging fails anyway? Log it more? But check this out, straight from Python 3.14’s documentation:

Important: Save a reference to the result of [ asyncio.create_task ], to avoid a task disappearing mid-execution. The event loop only keeps weak references to tasks. A task that isn’t referenced elsewhere may get garbage collected at any time, even before it’s done. For reliable “fire-and-forget” background tasks, gather them in a collection:

background_tasks = set()

for i in range(10):
    task = asyncio.create_task(some_coro(param=i))

    # Add task to the set. This creates a strong reference.
    background_tasks.add(task)

    # To prevent keeping references to finished tasks forever,
    # make each task remove its own reference from the set after
    # completion:
    task.add_done_callback(background_tasks.discard)

Wait, what? If I start a Python async task it might be GC’ed before it even starts? Wow, just wow.

The fix? Well, you just create a set to track them (keep a strong reference in GC-parlance). In our example, it looks like this:

background_tasks = set()

async def register_user():
    data = get_form_data()
    user = user_service.create_user(data)
    
    # Log in the background
    task = asyncio.create_task(log_account_created(user.name)) 
    background_tasks.add(task)                       # prevent GC
    task.add_done_callback(background_tasks.discard) # cleanup when done
    
    return redirect('/account/welcome')

One obvious way to do it

This set hack is entirely non-obvious that this is required. After all, the Zen of Python states:

There should be one – and preferably only one – obvious way to do it.

But Zen doesn’t always apply, does it? I’m sure there is a reason for this change, though I’m not sure it was worth it.

If it were up to me, Python would come with one omnipresent event loop running on a background thread. Just calling an async function would schedule and run it there. The await keyword would be a control flow only construct, not the thing that actually does the execution.

But it doesn’t work that way and so we get oddities here and there I guess.