Company logo

Quantlane tech blog

How to make sure your asyncio tests always run

We make heavy use of asyncio and pytest, and thus are big fans of the pytest-asyncio plugin. It allows one to write async def test_... functions as well as asynchronous fixtures just as easily as normal synchronous code. Well, almost...

The one extra thing you need to do is to always mark your test functions with the @pytest.mark.asyncio decorator. See the following example (with code and its test in the same file for simplicity):

 1 import asyncio
 2 import pytest
 3 
 4 
 5 async def my_coroutine() -> float:
 6     await asyncio.sleep(0.1)
 7     return 2.71828182845
 8 
 9 
10 @pytest.mark.asyncio
11 async def test_my_coroutine() -> None:
12     assert 2 < await my_coroutine() < 3

Note the decorator on line 10. This works as expected:

$ pytest -v example1.py
============== test session starts ===============
platform darwin -- Python 3.8.2, pytest-5.4.3, py-1.9.0, pluggy-0.13.1 -- ...venv/bin/python
cachedir: .pytest_cache
rootdir: ...
plugins: asyncio-0.14.0
collected 1 item

example1.py::test_my_coroutine PASSED      [100%]

=============== 1 passed in 0.12s ================

How not to forget

With test files often spanning dozens of test functions, some of them synchronous, it is easy to forget the decorator. What happens then?

 1 import asyncio
 2 
 3 
 4 async def my_coroutine() -> float:
 5     await asyncio.sleep(0.1)
 6     return 2.71828182845
 7 
 8 
 9 async def test_my_coroutine() -> None:
10     assert 2 < await my_coroutine() < 3

Luckily, pytest developers have thought of that, so you get a helpful warning:

$ pytest -v example2.py
============== test session starts ===============
platform darwin -- Python 3.8.2, pytest-5.4.3, py-1.9.0, pluggy-0.13.1 -- ...venv/bin/python
cachedir: .pytest_cache
rootdir: ...
plugins: asyncio-0.14.0
collected 1 item

example2.py::test_my_coroutine SKIPPED     [100%]

================ warnings summary ================
example2.py::test_my_coroutine
...venv/lib/python3.8/site-packages/_pytest/python.py:171: PytestUnhandledCoroutineWarning: async def functions are not natively supported and have been skipped.
You need to install a suitable plugin for your async framework, for example:
   - pytest-asyncio
   - pytest-trio
   - pytest-tornasync
   - pytest-twisted
   warnings.warn(PytestUnhandledCoroutineWarning(msg.format(nodeid)))

-- Docs: https://docs.pytest.org/en/latest/warnings.html
========= 1 skipped, 1 warning in 0.02s ==========

This is pretty great. It tells you to install a plugin even though you already have it (plugins: asyncio-0.14.0) but it definitely points you in the right direction as it tells you that test_my_coroutine is being skipped.

Earlier versions of pytest did not even have this warning. The test function would not have been run at all, but was still marked as passing! Only an asyncio message RuntimeWarning: coroutine 'test_my_coroutine' was never awaited mixed in the output somewhere would be hinting at this omission. Over time, we discovered quite a few tests in our codebase that we thought were being run – but pytest was in fact quietly ignoring them. In that regard, this warning is a huge improvement.

How to check for @pytest.mark.asyncio automatically

Since linting and static analysis is an integral part of our development workflow we decided early on to add an automatic check that all async def test_... functions are decorated with this mark.

This proved to be trivial with the amazing bellybutton linting engine. Bellybutton allows you to create custom linting rules operating on the Abstract Syntax Tree. We created the following rule (lines 2–4) that finds asynchronous functions starting with test_ and checks whether they have the requisite decorator:

 1 rules:
 2   MissingPytestMarkAsyncio:
 3     description: "Asynchronous test function requires the `@pytest.mark.asyncio` decorator"
 4     expr: //AsyncFunctionDef[starts-with(@name, 'test_')]/decorator_list[not(Attribute[@attr = 'asyncio']/value/Attribute[@attr = 'mark']/value/Name[@id = 'pytest'])]
 5 
 6 default_settings: !settings
 7   included: ["*"]
 8   excluded: []
 9   allow_ignore: yes

Running Bellybutton with this configuration we get

$ bellybutton lint
example2.py:10       MissingPytestMarkAsyncio: Asynchronous test function requires the `@pytest.mark.asyncio` decorator
Linting failed (1 rule, 2 files, 1 violation).

We made this a part of our CI run (through our cq linter runner) and voilà, any commit that forgets the decorator will fail the build even before the tests are run.

Alternative: applying the mark on the module level

Finally, there is a way to avoid adding @pytest.mark.asyncio to every asynchronous test function in your module. Pytest allows you to mark the entire module by setting the global variable pytestmark:

 1 import asyncio
 2 import pytest
 3 
 4 
 5 pytestmark = pytest.mark.asyncio
 6 
 7 
 8 async def my_coroutine() -> float:
 9     await asyncio.sleep(0.1)
10     return 2.71828182845
11 
12 
13 async def test_my_coroutine() -> None:
14     assert 2 < await my_coroutine() < 3

If there were multiple asynchronous test functions this would apply the mark to all of them. In our code we prefer to be more explicit and add marks directly to tests, but this is a viable alternative if you like pytest's magic 😉

P.S. Even more magic: aiohttp

If you use aiohttp and its pytest plugin you will find that asynchronous test functions just work without you having to do anything at all. No decorators, no global marks. That's because the plugin, apart from providing useful fixtures, also hooks into test execution and ensures that coroutine test functions run in an event loop. Personally I prefer the explicit behaviour of pytest-asyncio but it's good to know about this auto-magical alternative too!

Vita Smid

Hi, my name is Vita. I co-founded Quantlane together with a few stock traders in 2014. I developed the first version of our Python trading platform from the ground up. After our engineering team started to grow I became Quantlane's CTO. This was quite new for me, as prior to this I had been working as a freelance developer and had a degree in financial mathematics.

I am a big fan of asyncio, static typing, wine, and motorbikes. Having lived in Australia, Hong Kong and now Europe I consider myself a global citizen, and am excited to be building towards Quantlane's vision to become a truly global trading & technology powerhouse.

You can reach me at vita@quantlane.com.