I made a comparison of package files before <> after and added a few corrections. Also I've added an extra security layer for CI building of airflow packages - it runs inside a fully isolated Docker container.
Would be great to get another quick look /review before I merge it :) J, On Wed, Jan 10, 2024 at 1:36 AM Jarek Potiuk <[email protected]> wrote: > Hey Everyone, > > I got the PR green: https://github.com/apache/airflow/pull/36537 - I got > a really comprehensive review and a number of iterations with Jens (and > approval! yay!!) and a number of comments from TP. > > I would love to have some feedback from others before merging, I still > want to (I will do it tomorrow) go through the packages prepared with hatch > and make sure we have not lost (or added) too much from the packages and > add appropriate inclusions/exclusions - but other than that, I think it > could be merged even today. > > I'd love some more comments - especially from those who struggled with > local venv/editable installation and dependency management/adding provider > dependencies recently - as the way it is done now should be WAY simpler and > better. > > Just to repeat what we get with that one: > > 1. cutting-edge support for packaging Python standards (see previous mail > in the thread) - with complete configuration for project in single > pyproject.toml file. Allows to use any modern build frontend for > development (hatch, pip. poetry, pipenv etc.) > 2. nicer integration with IDEs (Pycharm/VScode etc.) with installing > dependency management > 3. nicely and logically organized dependencies - including devel > dependencies + extras per provider, nicely managed from provider.yaml > 4. seamlessly working `pip install --editable .` (it was hacked before, > and not working in recent `pip` versions - now it will `**just work**) > 5. a way to easily install provider devel dependencies for testing in > local venv (`pip install -e ".[amazon,google]"`) > 6. hatch as recommended (but not mandatory) frontend that supports > out-of-the-box: > a) installing python interpreters (`hatch python install all`) > b) creating local venvs (`hatch env create`, `hatch env shell`, `hatch > -e airflow-311 create` and so on) > c) building packages for release (`hatch build -c custom -c wheel -c > sdist`) > d) later we will use more things that hatch gives us (reproducible > builds, publishing to PyPI, possibly local testing and code formatting, > better monorepo organization in the future). > 7. Updated documentation for all the above. > > Note: It does not replace Breeze for reproducing and optimizing our CI > build (Breeze has way more optimizations and customisations needed for > Airflow). However it makes the LOCAL_VIRTUALENV option of running tests and > developing airflow much easier to manage and get it under control. > > Just as a teaser - here is the output of `hash env show`: > > > ┏━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ > ┃ Name ┃ Type ┃ Features ┃ Description > ┃ > > ┡━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ > │ default │ virtual │ devel │ Default environment with Python 3.8 > for maximum compatibility │ > > ├─────────────┼─────────┼──────────┼───────────────────────────────────────────────────────────────┤ > │ airflow-38 │ virtual │ │ Environment with Python 3.8. No devel > installed. │ > > ├─────────────┼─────────┼──────────┼───────────────────────────────────────────────────────────────┤ > │ airflow-39 │ virtual │ │ Environment with Python 3.9. No devel > installed. │ > > ├─────────────┼─────────┼──────────┼───────────────────────────────────────────────────────────────┤ > │ airflow-310 │ virtual │ │ Environment with Python 3.10. No > devel installed. │ > > ├─────────────┼─────────┼──────────┼───────────────────────────────────────────────────────────────┤ > │ airflow-311 │ virtual │ │ Environment with Python 3.11. No > devel installed │ > > └─────────────┴─────────┴──────────┴───────────────────────────────────────────────────────────────┘ > > J. > > > > On Sun, Jan 7, 2024 at 11:55 PM Jarek Potiuk <[email protected]> wrote: > >> Ah .. .And comparing to the original proposal I simplified it a LOT. >> generally speaking for both contributor and user the way how you >> install Airflow for installation and contribution is "standard" and >> basically just "fixes" what has been broken - i.e. you just install it >> as expected: >> >> * `pip install apache-airflow[google]` or `pip install .[google]` >> will install airflow + google provider (user story) >> * `pip install -e .[google]` will install airflow + all google >> provider dependencies in editable mode - ready to run tests >> >> Plus Airflow follows all the PEP-standards so that it is compatible >> with all the modern tooling for Python packaging. Here is the list of >> PEP's that it makes airflow generally compatible with: >> >> * `PEP-440 Version Identification and Dependency Specification >> <https://www.python.org/dev/peps/pep-0440/>`__ >> * `PEP-517 A build-system independent format for source trees >> <https://www.python.org/dev/peps/pep-0517/>`__ >> * `PEP-518 Specifying Minimum Build System Requirements for Python >> Projects <https://www.python.org/dev/peps/pep-0518/>`__ >> * `PEP-561 Distributing and Packaging Type Information >> <https://www.python.org/dev/peps/pep-0561/>`__ >> * `PEP-621 Storing project metadata in pyproject.toml >> <https://www.python.org/dev/peps/pep-0621/>`__ >> * `PEP-685 Comparison of extra names for optional distribution >> dependencies <https://www.python.org/dev/peps/pep-0622/>`__ >> >> J. >> >> >> On Sun, Jan 7, 2024 at 11:27 PM Jarek Potiuk <[email protected]> wrote: >> > >> > Hello everyone, >> > >> > I iterated quite a bit on the PR and I think it's ready for an even >> > more serious review: https://github.com/apache/airflow/pull/36537 . I >> > solved all of the TODOs and teething problems and while it likely >> > still has some tests to fix, all the build and packaging pieces, local >> > installation and even developer/contributor documentation should be >> > already in the state that is ready for serious scrutiny. Thanks to >> > Jens and TP for the reviews so far - I addressed all of the comments >> > already - and there are just 2 conversations left remaining. >> > >> > See the comment for status summary: >> > https://github.com/apache/airflow/pull/36537#issuecomment-1880193452 >> > >> > BTW. I found it really useful to follow the "unresolved conversation" >> > routine - it's really nice to see such things as a summary (see >> > attachment) and be able to see that there are still 2 conversations to >> > resolve. >> > That's the in-progress experiment with conversations which I >> > personally like a lot so far. It already saved me from merging a PR >> > that still had things to resolve. >> > >> > J. >> > >> > On Thu, Jan 4, 2024 at 8:04 PM Jarek Potiuk <[email protected]> wrote: >> > > >> > > I slept over it a few nights and got away of it and I have an idea to >> > > simplify it quite a bit - i.e. cut the number of extras by half and >> > > virtually make 0 impact on current editable installation so you might >> > > wnnt to hold on a bit with that (unless you want to see it changing :) >> > > ) .. The whole concept won't change, I just realized that I do not >> > > need to add new `editable_` extras to achieve the same effect. >> > > >> > > I will also attempt to split it a bit to make it easier to review. >> > > >> > > Hold tight :) - but also feel free to look and comment even now :) >> > > >> > > And yes. Exciting. It kept me awake a night or two where I could not >> > > get to sleep until I finally got it working :D >> > > >> > > J >> > > >> > > On Thu, Jan 4, 2024 at 6:52 PM Pierre Jeambrun <[email protected]> >> wrote: >> > > > >> > > > I personally think that this is a great idea. I have been following >> the >> > > > hatch project for a while and I am convinced it has a lot to offer >> for >> > > > airflow. The two big pros for me are its ease of use (backend and >> front >> > > > end) as well as the security covered aspects (reproducible builds >> to name >> > > > one). >> > > > >> > > > I will take a look at the PR later this week, but it definitely >> sounds >> > > > exciting. >> > > > >> > > > >> > > > >> > > > On Tue 2 Jan 2024 at 20:26, Jarek Potiuk <[email protected]> wrote: >> > > > >> > > > > Hello everyone. >> > > > > >> > > > > Tl;DR; I have a proposal to adopt Hatchling as a build backend >> (and >> > > > > recommend, but not require Hatch as frontend) for Airflow as our >> way >> > > > > of switching to PEP-standard compliant pyproject.toml way of >> > > > > installing Airflow (including local venvs) and building the >> Airflow >> > > > > package. >> > > > > >> > > > > I have a working implementation that needs polishing and taking a >> few >> > > > > less important decisions and rather simple TODOS). Here is draft >> PR: >> > > > > https://github.com/apache/airflow/pull/36537 >> > > > > >> > > > > I've spent a better part of the Xmas/New Years break on >> implementing >> > > > > it - something that we've been discussing for - literally - years >> - >> > > > > and several people (including myself) made several attempts in the >> > > > > past - unsuccessfully- with standardising python packaging/ build >> > > > > process for Airflow to use modern standard-driven tooling. >> > > > > >> > > > > I think I succeeded. finally. >> > > > > >> > > > > In short, what it means: >> > > > > >> > > > > When this change is merged, Airflow will have a nice and slick and >> > > > > modern, standard compliant contributor's experience - with >> editable >> > > > > installation that will **just work**, that will work with multiple >> > > > > build front-ends and it will make it very easy to install and >> manage >> > > > > local virtualenv(s) to contribute to Airflow. The extras >> structure and >> > > > > airflow configuration will be in one place (pyproject.toml) and it >> > > > > will be much easier to reason about our extras and dependencies. >> As a >> > > > > bonus point - with tools like Hatch, contributors will get the >> > > > > canonical way of managing local virtualenvs for Airflow >> development >> > > > > and a very easy recommended way to manage both Python and Venvs >> (but >> > > > > without forcing a single frontend). >> > > > > >> > > > > From the user perspective Airflow packages will be more >> standardised, >> > > > > with just user extras defined. From maintainers and PMC members, >> we >> > > > > will get reproducible builds (similarly as we have now for >> Providers) >> > > > > - which means that it will be easier and more robust to verify >> > > > > provenance of the packages (security!) >> > > > > >> > > > > Why can we do it now and we could not do it before ? >> > > > > >> > > > > This is mostly thanks to Herculean efforts of Python Packaging >> team >> > > > > (hats off to TP being part of the team and leading a lot of >> > > > > standardisation efforts there) - after a few years of relentless >> > > > > introduction and implementation of many PEPs and releasing new >> tooling >> > > > > (particularly Hatch, but also Flit that we already use for >> providers) >> > > > > it seems finally Airflow can move away from a very complex, >> completely >> > > > > custom setup.py and setup tools being abused by us in ways that >> > > > > authors and Packaging team did not originally anticipate. >> > > > > >> > > > > What problems does the change solve? >> > > > > >> > > > > My PR solves all the difficult requirements of our custom >> solution, >> > > > > but also (mostly thanks to standardisation efforts by the >> packaging >> > > > > team), it improves on a lot of problems we could not solve. >> > > > > >> > > > > Happy to have a detailed discussion here, and more detailed in >> the PR >> > > > > (I added a lot more context and documentation- showing how this >> will >> > > > > work when we merge it). but here is the list of things such a move >> > > > > provides: >> > > > > >> > > > > * We are using hatchling build backend, that follows appropriate >> PEP >> > > > > standards and makes it work with any "frontend" you choose to >> install >> > > > > and manage your local installation (You can use modern Hatch >> which is >> > > > > counterpart to hatchling - highly recommended, but also it will >> work >> > > > > with just pip, poetry, flit, and any other standard-compliant >> tool in >> > > > > the future. No habits of the contributors need to be changed, it >> will >> > > > > **just** work >> > > > > >> > > > > * our editable installation has been broken for some time (mostly >> > > > > because we were abusing setuptools and setup.py A LOT). See >> > > > > https://github.com/apache/airflow/issues/30764 . This change >> puts the >> > > > > shine back on being able to make editable install of airflow work >> as >> > > > > expected and getting a first-class experience for contributors >> with >> > > > > local virtualenvs >> > > > > >> > > > > * all Airflow package configuration is now merged into a single >> > > > > appropriate PEP-compliant pyproject.toml - no more setup.py, >> > > > > setup.cfg, MANIFEST.in. >> > > > > >> > > > > * the extras are refactored and organized into logical groups and >> > > > > start to make sense. I introduced new "editable" extras to allow >> you >> > > > > to easily install provider dependencies locally and reorganized >> devel >> > > > > extras to make it easy to understand what you should install in >> your >> > > > > editable environment to run tests. More importantly those "devel" >> > > > > extras - while present in pyproject.toml are stripped off (thanks >> to >> > > > > custom hooks) from the final package - so final package has just >> > > > > things that are important to our users >> > > > > >> > > > > * we use pre-commit to automatically use provider.yaml >> dependencies >> > > > > and merge them into pyproject.toml - thanks to that provider.yaml >> will >> > > > > remain the single source of truth for providers. This provides a >> > > > > single source of truth for provider configuration, while it also >> > > > > allows one local installation to develop them all together" - and >> in a >> > > > > very seamless way. >> > > > > >> > > > > * no more INSTALL_PROVIDERS_FROM_SOURCES hack when you install >> airflow >> > > > > for local development. I figured a nice way to avoid installing >> > > > > pre-installed providers, and to make it super-easy to install >> > > > > dependencies of providers in editable installation (hint: `pip >> install >> > > > > -e .[editable_google]` . This thanks to custom build hooks the PEP >> > > > > standardized. >> > > > > >> > > > > * I also recommend Hatch as a Python/Venv management tool and >> used it >> > > > > for testing - it's a great tool for managing both - Python >> > > > > installations and Virtualenv management. For many people - >> providing >> > > > > such a canonical way (while following the standards and not >> forcing >> > > > > Hatch) will be really great to simplify their local environment >> > > > > installation. >> > > > > >> > > > > * Hatchling supports reproducible builds out-of-the-box, which is >> > > > > great for security - and it will make our package generation much >> > > > > safer and easier to verify (as we do with our providers now). >> > > > > >> > > > > There are many more details and thoughts (and also some future >> > > > > possible developments) that I am aware of, but this mail is >> already >> > > > > too long. and we can discuss it in the thread/PR or future >> threads. >> > > > > >> > > > > Happy to take any questions, critique, proposals and feedback - I >> got >> > > > > quite deep into how modern package building works so I likely made >> > > > > some mistakes / bad assumptions or things can be improved or >> maybe we >> > > > > can take other directions. It will take some time to merge and >> > > > > discuss details, and if this one gets approved it's likely going >> to be >> > > > > targeted for Airflow 2.9. >> > > > > >> > > > > J. >> > > > > >> > > > > >> --------------------------------------------------------------------- >> > > > > To unsubscribe, e-mail: [email protected] >> > > > > For additional commands, e-mail: [email protected] >> > > > > >> > > > > >> >
