potiuk commented on PR #51908: URL: https://github.com/apache/airflow/pull/51908#issuecomment-2985909860
cc: @kaxil @ashb @amoghrajesh The `airflow-ctl` is the first of the distributions that we can run separately and simplify all the testing enviroment for it. previously it made no sense to do it for airlfow/providers etc. and we needed to use CI image for that - because that was the only reproducible environment that we could run tests in. And this only shows how important is to split airflow into smaller pieces- including splitting airflow core. Currently we simply cannot separate the tests of webserver, dag processor, triggerer etc. because all that is so heavily entangled, we still have airflow core depending on some provider tests etc. But if we do think seriously of our dependencies, and split them to smaller sets, this will be entirely possible to do very similar change as this one with `airflow-ctl` where we can saperate tests for each component to inndependent, workspace-managed packages and do something very similar (and only use CI image for provider tests and eventually maybe even get rid of it for most of the providers that will have "non-problematic" dependencies. This is one of the reasons why I am so keen on splitting airflow and sharding the dependencies and using separate distributions and workspace to bring it all together. This is something @ashb always complained about that we have far too complex environment - and yes, we do, because we have everything bundled together. When we split it - we will finally be able to achieve the "dream" that many of our tests will be runnable in such a super simple way as "airflow-ctl" will be after this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
