potiuk commented on PR #49799: URL: https://github.com/apache/airflow/pull/49799#issuecomment-2831599725
BTW. @kaxil @eladkal, @gopidesupavan @ashb and others potentially interested - some more explaation why previos rules were so complex and brittle, and why after removing ".dev0" prefix from our sources suddenly it became so much easier. -> this is what .dev0 removal allowed for as simplification (it's not the last cleanup - there is one more cleanup in flooring the suffix and provider preparation) What we can do here is we can rely on `>=x,y.z` dependendencies between Airflow and providers, because both airlfow and providers do not have the suffixes any more when we prepare then on CI. So if somoene says `apache-airflow==3.1.0` (in development) depends on (say) comon-compat>=1.6.1 and `common.compat==1.6.1` depending on `apache-airflow>=3.1.0` the CI will continue to work nicely, This is becuase in CI we will build apache airflow==3.1.0 and common-compat==1.6.1 and we will use those packages for building PROD image and generarting `PyPI` constraints. And both packages installed in CI will satisfy each-others requirements. Previously we had to do some wicked magic because we produced `apache-airflow==3.1.0.dev0` and `common-compat==1.6.1dev0` and we had to do the magic because: * apache-airflow >= 3.1.0 was NOT satisfied by apache-airflow==3.1.0.dev0 * comon-compat >= 1.6.1 was NOT satisifed by comon-compat==1.6.1.dev0 This also applied to `workspace` dependencies in sources. When we had "apache-ariflow==3.1.0.dev0" - we could never work on a provider package that had "apache-airflow>=3.1.0" - because "3.1.0.dev0" does not satisfy ">=3.1.0". If not that limitation, we could simply strip-down '.dev0" always in CI and keep it in the sources, however with workspace the same dependencies that we have in requirements are used to determine in workspace can be installed together, we would not be able to work at the same time on provider / airflow that depend on each other's current main version if we kept the .dev0 suffix. All this is an unfortunate consequence of https://packaging.python.org/en/latest/specifications/version-specifiers/#summary-of-permitted-suffixes-and-relative-ordering which says that "3.1.0.dev0" < " 3.1.0.rc0" < "3.1.0.rc1" < "3.1.0". If we had other rules and "3.1.0,dev0" was considered as satisfying the ">=3.1.0" - that would be a different story. But it's not, unfortunately. For some reason (probably not realising the consequences) made the decision that strict ordering is better and that ',dev0` always goes before `final version` - which is logical, but also does not allow to have such pre-release cross-dependencies. There are a number of discussion about this choice (in this context) in pip: for example https://github.com/pypa/pip/issues/12049 where maintainers confirmed that `pip` behaves according to the specification (hard to argue with it because it does) but still being open to control that behaviour - but that would be a difficult one to define nicely. Also `uv` foll ows the spec in workspaces and it would be hard to change the spec and adjust thing in all the tooling if we would like to change it. So what we had to do previously (this part is still to be removed in a separate PR) we had to artifficially modify those >= requirements - when we prepared `apache-airflow==3.1.0.dev0 we had to make sure that all provider packages it referred to also had >= x.y.z.dev0 (and the other way round). And it got even weirder when we had to build stuff for rc packages for PROD images. If we stick to non-dev0 non-rc suffix anywhere in CI, things are suddenly **way** simpler: * workspaces work out of the box. We can have in main "apache-airflow==3.1.0" depending on "common.compat>=1.6.1" that depends on "apache-airlfow>=3.1.0" and uv will nicely install such workspace (this is the main reason why removal of .dev0 suffix hard-coded in our sources was needed - cc: @kaxil ) * if a provider package is not released yet (either not RC-d or RC in PyPI) we will always build such provider package in `main` build with "final" version - wich means we will not have to modify airflow's dependency to have >= x.y.z.rc0 or in any other way - we can **just** build both packages in CI without any suffixes and they will install together nicely (we do it for constraint generation and PROD image building). * simlarly, even if Airflow is not released yet but we have (say) 3.1.0 in main - we will always build the "final" version in our CI and even if we release a new provider that depends on >=3.1.0 - such provider will nicely build and install in Ci - without having to modify the requirements at all. * when we release Dockerhub RC PROD airflow image - we will first have to at least release an RC image. And here we will still have modify the requirements in Airlfow and providers to have `>=X.Y.Zrc1` - and there is really no other way, but this is already way simpler, knowing that we have no other suffixes to strip. And in this case we will not use locally built providers at all - when we release Dockerhub RC PROD image we ALWAYS will have to install things from PyPI. We will use "--pre" flag so we will install pre-release providers that are already released in PyPi - but we will never build the providers locally in CI for those image (cc: @kaxil - that will avoid the issue we had with the broken standard provided in RC5 of 3.0.0) - this happened because we installed provider from locally modified sources in main that were not released even in RC yet. * when we release FINAL PROD airflow image - we will not use the `--pre` flag, and airflow will not have`rc` added to >= - which means that FINAL PROD airflow image will always require "final" providers released in PyPI. I **think** that handles all the edge cases and combinations of released, RC, non-released providers and dependencies between those we had encountered so far. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
