potiuk opened a new issue, #33909: URL: https://github.com/apache/airflow/issues/33909
### Body As part of making it possible to move out of providers from Airflow core repository (we have not decided yet on it, we just want to make it possible) we should turn Airflow Providers into "real" packages. Currently those packages are build "dynamically" -> https://github.com/apache/airflow/blob/main/dev/provider_packages/prepare_provider_packages.py is used as part of `breeze release-management prepare-provider-packages` to extract parts of the "airflow/providers/<provider>" sources dynamically, generate setup.* and pyproject.toml files dynamically and build the providers from those dynamically generated temporary folders. This has some disadvantages - for example it does not make reproducible builds possible, and it requires complex `breeze` command, CI image and the python script in the image to build the packages (CI image is used to make sure all dependencies are installed and to provide isolation and cleanup between builds, also it allows to isolate (security) the host from container building the packages whenbuilding the providers in case of builds from contributor forks). But this got us through last 3 years of releasing airflow and providers separately :). With recent changes (#32604 upcoming #32048, the upcoming #33907 and a number of other changes already implemented in the past - we are quite close to make it possible to split out providers to "standalone" packages - where each provider is a separate "compliant with standard" package and has a complete independent directory where you can build the package without moving the sources around, using standard python tooling. This would require everything that relates to the providers to move to those directories (docs, tests are notably shared between airflow and providers in "docs" and "tests" and they should be moved around). The early draft POC attempts to add scripts to automate such migration and result of it can be seen here: * https://github.com/apache/airflow/pull/28291 * https://github.com/apache/airflow/pull/28292 However rhe challenge to solve for this one is to make it easy for contributors to contribute to airflow and providers together. We want to make it possible - simlarly as today to have an easy environment where you can edit both airflow and provider code and run tests, run airflow, run integration tests without extra hassle of installing and reinstalling the packages in editabkle mode. Ideal workflow of the developer is where they can: * run `breeze` and be able to edit any sources locally in the host - and airflow in breeze should automatically pick the changes in both airlfow and providers when interpreter is started * install a local venv (in editable mode possibly) for airlfow and selected (all?) providers and the locally installed airflow should also pick changes for both airflow and providers when interpreter is started The way how to do it and choice of (ideally) standard python tooling to make such move is not yet determined and is open for discussion, POCs and proposals. ### Committer - [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
