potiuk commented on code in PR #35160: URL: https://github.com/apache/airflow/pull/35160#discussion_r1375550277
########## tests/operators/test_python.py: ########## @@ -844,17 +848,30 @@ def f(exit_code): assert ti.state == expected_state +venv_cache_path = tempfile.mkdtemp(prefix="venv_cache_path") Review Comment: > Good idea to enable venv caching in tests - then we test more functionality and find nice race conditions as well :-D - can we enable x-dist also here and run them all in parallel? :-D Unfotunately not. Those are "classic" DB tests - they create Dag Run in DB and run task using Airflow's internal code which reads and updates the DB (and cleans it before /after) - so by the natur of it, it falls into "DB tests" rather than "Non-DB" tests where at most what we can do (and this is what I do already for some time) we split the test into "groups" (TEST_TYPE) and run the TEST_TYPES in parallell - each TEST_TYPE with it's own Database (via docker compose) and all the tests within each TEST_TYPE are executed sequentially because of the shared resource - DB they use. But generally speaking this whole PR is about splitting the tests of ours into DB and Non-DB - and run the Non-DB tests with `pytest-xdist` precisely. Simply the Venv and External tests we are talking about are the "DB" ones. BTW. Part of this change is documentation explaining all that and I just submitted a change explaining it - with a few TODOs for examples and more explanation so I'd really appreciate good review of those as well (and of course open to get some feedback/improvement ideas on the whole thing): The draft doc is in TESTING.rst in this PR - see https://github.com/apache/airflow/pull/35160/files#diff-35a395614be3ab2f31c88bff2e106b942c11e0416ad3a2fbbcd8223cd1387478R76 for the version I pushed just hour ago or so. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org