Hi Jarek,
Thanks for proposing the new e2e test structure, the task-sdk e2e setup
looks great to me!
I have one question about the k8s tests in the future: should they live
under airflow-core, under task-sdk, or in a separate dedicated location?
Best regards,
Jason
On Mon, Nov 17, 2025 at 11:56 PM Jarek Potiuk <[email protected]> wrote:
Hello here,
*TL;DR; We want to have better organized, and much improved new `e2e`
test
type, a common thing in Airflow - one that others will be able to add
tests
to and add their own similar tests following the blueprint.*
Over the last few weeks quite a few of us were working on something we
have
not fully realise we will end up and it was mostly bottom -up grass root
ideas that simply "clicked" together and we would like to propose adding
(or more formalising) a new type of tests in airflow - e2e tests. The
people who worked on it were mostly Amogh, Zhe You lIu, Bugra, myself.
Wa already have a few attempts to do it (in various stages - as top-level
folders: *docker-tests, k8s-tests, airflow-e2e-tests, airflow-ctl-tests,
task-sdk-integration-tests* - yes the lack of consistency in names is
very
confusing) but with recent improvements in task-sdk-integration-tests -
we
think we have a good proposal on how we can implement those tests and
standardise it across airflow distributions.
Current (best) approach is
https://github.com/apache/airflow/tree/main/task-sdk-integration-tests
And it has the following properties:
1) uses uv pytest (standard pytest tests), docker-compose and
python-on-whales together, also has nice breeze wrapper for CI
2) automatically sets-up (and keeps running during iteration)
docker-compose based airflow deployment - with components (any -
api-server, dag processor, scheduler, triggerer) necessary during the
tests
and allows to interact with it, including triggering local dags and the
like
3) pytest tests are run in a local venv controlled by uv sync
4) it allows for extremely fast (almost like pytest-native in venv)
iteration speed with tests - it uses Zhe's hot-reload functionality
added a
week ago in order to reload components inside the docker-containers, and
local sources are mounted to those - which means that just saving a file
in
your IDE or local env will make automated (sub-second, really) reload of
the components you interact with. Basically at the time you press
Shift-F10
(or whatever shortcut you have) to re-run your tests, your just
auto-saved
modified Airflow code already runs in those docker-compose components (!)
- *this is the most important feature of it.*
5) All those tests are automatically run in CI - in relevant PRs (driven
by
selective checks) - but also they are very easily runnable locally in
local
venv (no breeze CI image needed). Basically:
cd task-sdk-integration-tests
uv run pytest
6) Last minute addition - we've been literally asked by Kacper an hour
ago
about having "something like that" for open lineage - every current
distribution is able to have their own set of tests like that if the
stewards of that part want it
Enough bragging....
*Our proposal:*
* we consistently rename those tests as `*e2e*` tests
* we adapt all of them (including *k8s* tests eventually - where we have
to
hack kind a bit) - to follow the same patterns and principles
* we extract common code for that to `*devel-common*` and reuse across
those tests
* we get rid of all the top-level `*SMTH-test*` folders we have now and
move those different kinds of tests to `*tests/e2e*` (next to `unit`,
`system`, `integration` we already have) of distributions that the tests
are relevant to
For example - instead of:
task-sdk
src
tests
task_sdk
some_unit_test.py
task-sdk-integration-tests
dags <- here e2e test dags are stored
logs <- here logs from all components are kept (.gitignored)
tests
task_sdk_tests
some_end_2_end_test.py
We propose:
task-sdk
src
tests
unit
task_sdk
some_unit_test.py
e2e
dags
logs
task_sdk
some_end_2_end_test.py
Of course - this is a proposal - and if there are other ideas on how to
restructure the test folders, that might be a good idea to do it now.
*Only
serious and complete offers are going to be considered :)*, so I propose
constructive, complete proposals rather than criticising the current one.
*Let us know what you think,*
Jarek, Zhe, Bugra, Amogh
