potiuk commented on code in PR #35160:
URL: https://github.com/apache/airflow/pull/35160#discussion_r1375550277
##########
tests/operators/test_python.py:
##########
@@ -844,17 +848,30 @@ def f(exit_code):
assert ti.state == expected_state
+venv_cache_path = tempfile.mkdtemp(prefix="venv_cache_path")
Review Comment:
> Good idea to enable venv caching in tests - then we test more
functionality and find nice race conditions as well :-D - can we enable x-dist
also here and run them all in parallel? :-D
Unfotunately not.
Those are "classic" DB tests - they create Dag Run in DB and run task using
Airflow's internal code which reads and updates the DB (and cleans it before
/after) - so by the nature of it, it falls into "DB tests" rather than "Non-DB"
tests where at most what we can do (and this is what I do already for some
time) we split the test into "groups" (TEST_TYPE) and run the TEST_TYPES in
parallell - each TEST_TYPE with it's own Database (via docker compose) and all
the tests within each TEST_TYPE are executed sequentially because of the shared
resource - DB they use.
But generally speaking this whole PR is about splitting the tests of ours
into DB and Non-DB - and run the Non-DB tests with `pytest-xdist` precisely.
Simply the Venv and External tests we are talking about are the "DB" ones.
BTW. Part of this change is documentation explaining all that and I just
submitted a change explaining it - with a few TODOs for examples and more
explanation so I'd really appreciate good review of those as well (and of
course open to get some feedback/improvement ideas on the whole thing):
The draft doc is in TESTING.rst in this PR - see
https://github.com/apache/airflow/pull/35160/files#diff-35a395614be3ab2f31c88bff2e106b942c11e0416ad3a2fbbcd8223cd1387478R76
for the version I pushed just hour ago or so.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
