If you read my Fire and Forget (or Never) about Python and asynchronous programming, you could think it’s a super odd edge case. But a reader/listener, Richard, pointed me at Will McGugan’s article The Heisenbug lurking in your async code. This is basically the same article, but in Will-style.
Will does say “This behavior is well documented, as you can see from this excerpt.” True, but the documentation got this emphasis and warning in Python 3.12 whereas the feature create_task was added in Python 3.6/3.5 timeframe. So it’s not just a matter of did we read the docs carefully. It’s a matter of did we reread the docs carefully, years later?
Luckily Will added some nice concrete numbers I didn’t have:
https://github.com/search?q=%22asyncio.create_task%28%22&type=code
This appears in over 0.5M separate code files on GitHub. To be clear, not every search result for create_task uses the fire-and-forget pattern, but just on the first page of results there are 5 instances.
If the design pattern to fix this is to:
- Create a global set
- When a task is added to the event loop, add it to the set
- Remove it from the set when it’s done
Wouldn’t it have been better for the Python team to add this to the event loop internally once and solve this problem for everyone globally across the entire Python ecosystem?
It doesn’t look like that’s going to happen. So make sure you double check your code for create_task. And don’t let the Heisenbugs bite.
And yes, I know about task groups. Several people told me that we could use task groups to hang on to the task. Yes, that’s true. But task groups are incongruent with the fire-and-forget design pattern. Why? Because you create the group in a context manager and then you wait for all the tasks in the group to be finished. That doesn’t allow you to fire off a task and then continue working. So task groups may or may not have fixed Will’s problem, but they don’t solve the one I was originally talking about.