Accidentally It turned out - with today's investigation of why RC images are not building that it is VERY needed. It seems that `pip` resolution works currently with our meta-package in a very bad way. If in our pyproject.toml we have just "amazon' = "apache-airflow-providers-amazon", "google" = "apache-airflow-providers-google" ....
then apache-airflow[google,amazon] - will lead pip to download and try to resolve ALL VERSIONS or apparently ALL providers that have ever been released. That means we have 100 amazon, 100 google - and if you have more extras - 10s or 100s of other providers to consider. And essentially what pip tries to do is - try all combinations of those to find out which ones are good. Or at least very, very big subset of those providers. Which actually explains the "Resolution Too Deep" problem we tried to diagnose and investigate over the last few weeks. So summarising: * resolution too deep was actually "real" issue * it was caused by no lower-bounds in providers in meta package * we will have to set those lower-binds to something reasonable (and I might start later today with simply latest released versions if i have internet on my flight back from the US) * and we will have to keep those min versions updated and bumped periodically The last case is interesting - because it will finally force us to introduce some policy on how old providers we support when we release airflow. With Airflow 3 I think we are good with basically "latest" versions (and maybe get a few versions back for crucial providers). But we can discuss it later. J. On Thu, Apr 10, 2025 at 2:49 PM Vincent Beck <vincb...@apache.org> wrote: > I assume this is not necessary when the constraint is already set on the > provider side? For example, FAB provider has a dependency on Airflow 3 > (`apache-airflow>=3.0.0` in `providers/fab/pyproject.toml`) > > On 2025/04/10 17:22:13 Jarek Potiuk wrote: > > Hello, > > > > TL;DR; Do people who work on some providers think that Airflow 3 should > > have minimum version of those? > > > > While discussing a question about chicken-egg providers with Kaxil, I > > realized that we can finally have min versions of providers easily added > to > > "apache-airflow" meta-package. > > > > Simply speaking - we can say that for example > > > > apache-airflow[openlineage] -> > apache-airflow-providers-openlineage>=2.1.3 > > > > Previously it was not that easy or straightforward, but now we can do it > > easily - just changing: > > > > "openlineage" = [ > > "apache-airflow-providers-openlineage" > > ] > > into > > > > "openlineage" = [ > > "apache-airflow-providers-openlineage>=2.1.3" > > ] > > > > We will likely have to change the tooling a bit to adapt to RC / Dev > > packaging versions for chicken-egg-providers - but this should be rather > > easy. > > > > And I think this is mostly about openlineage, standard, common and all > the > > other kinds of providers that are "special" in some ways. > > > > Are we aware of some minimum versions we **SHOULD** put on those ? > > > > J. > > > > > > > > So the question is. > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org > >
