For OpenLineage we'd definitely like something like this. IMO there is no downside to restricting the providers to the latest released version. We could go with the chicken-egg for the next release - that would add at least one useful feature authored by Kacper Muda - https://github.com/apache/airflow/pull/48941 but even without it it's not broken, just missing that feature.
czw., 10 kwi 2025 o 21:49 Jarek Potiuk <ja...@potiuk.com> napisał(a): > Accidentally It turned out - with today's investigation of why RC images > are not building that it is VERY needed. It seems that `pip` resolution > works currently with our meta-package in a very bad way. If in our > pyproject.toml we have just > "amazon' = "apache-airflow-providers-amazon", "google" = > "apache-airflow-providers-google" .... > > then > > apache-airflow[google,amazon] - will lead pip to download and try to > resolve ALL VERSIONS or apparently ALL providers that have ever been > released. That means we have 100 amazon, 100 google - and if you have more > extras - 10s or 100s of other providers to consider. And essentially what > pip tries to do is - try all combinations of those to find out which ones > are good. Or at least very, very big subset of those providers. Which > actually explains the "Resolution Too Deep" problem we tried to diagnose > and investigate over the last few weeks. > > So summarising: > * resolution too deep was actually "real" issue > * it was caused by no lower-bounds in providers in meta package > * we will have to set those lower-binds to something reasonable (and I > might start later today with simply latest released versions if i have > internet on my flight back from the US) > * and we will have to keep those min versions updated and bumped > periodically > > The last case is interesting - because it will finally force us to > introduce some policy on how old providers we support when we release > airflow. With Airflow 3 I think we are good with basically "latest" > versions (and maybe get a few versions back for crucial providers). But we > can discuss it later. > > J. > > > > > > On Thu, Apr 10, 2025 at 2:49 PM Vincent Beck <vincb...@apache.org> wrote: > > > I assume this is not necessary when the constraint is already set on the > > provider side? For example, FAB provider has a dependency on Airflow 3 > > (`apache-airflow>=3.0.0` in `providers/fab/pyproject.toml`) > > > > On 2025/04/10 17:22:13 Jarek Potiuk wrote: > > > Hello, > > > > > > TL;DR; Do people who work on some providers think that Airflow 3 should > > > have minimum version of those? > > > > > > While discussing a question about chicken-egg providers with Kaxil, I > > > realized that we can finally have min versions of providers easily > added > > to > > > "apache-airflow" meta-package. > > > > > > Simply speaking - we can say that for example > > > > > > apache-airflow[openlineage] -> > > apache-airflow-providers-openlineage>=2.1.3 > > > > > > Previously it was not that easy or straightforward, but now we can do > it > > > easily - just changing: > > > > > > "openlineage" = [ > > > "apache-airflow-providers-openlineage" > > > ] > > > into > > > > > > "openlineage" = [ > > > "apache-airflow-providers-openlineage>=2.1.3" > > > ] > > > > > > We will likely have to change the tooling a bit to adapt to RC / Dev > > > packaging versions for chicken-egg-providers - but this should be > rather > > > easy. > > > > > > And I think this is mostly about openlineage, standard, common and all > > the > > > other kinds of providers that are "special" in some ways. > > > > > > Are we aware of some minimum versions we **SHOULD** put on those ? > > > > > > J. > > > > > > > > > > > > So the question is. > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > >
