There absolutely is a downside to the users — “forcing" users to upgrade providers along with Airflow version makes upgrades a much more daunting experience.
> On 10 Apr 2025, at 21:31, Maciej Obuchowski <mobuchow...@apache.org> wrote: > > For OpenLineage we'd definitely like something like this. > IMO there is no downside to restricting the providers to the latest > released version. > We could go with the chicken-egg for the next release - that would add at > least one useful > feature authored by Kacper Muda - > https://github.com/apache/airflow/pull/48941 > but even without it it's not broken, just missing that feature. > > > czw., 10 kwi 2025 o 21:49 Jarek Potiuk <ja...@potiuk.com> napisał(a): > >> Accidentally It turned out - with today's investigation of why RC images >> are not building that it is VERY needed. It seems that `pip` resolution >> works currently with our meta-package in a very bad way. If in our >> pyproject.toml we have just >> "amazon' = "apache-airflow-providers-amazon", "google" = >> "apache-airflow-providers-google" .... >> >> then >> >> apache-airflow[google,amazon] - will lead pip to download and try to >> resolve ALL VERSIONS or apparently ALL providers that have ever been >> released. That means we have 100 amazon, 100 google - and if you have more >> extras - 10s or 100s of other providers to consider. And essentially what >> pip tries to do is - try all combinations of those to find out which ones >> are good. Or at least very, very big subset of those providers. Which >> actually explains the "Resolution Too Deep" problem we tried to diagnose >> and investigate over the last few weeks. >> >> So summarising: >> * resolution too deep was actually "real" issue >> * it was caused by no lower-bounds in providers in meta package >> * we will have to set those lower-binds to something reasonable (and I >> might start later today with simply latest released versions if i have >> internet on my flight back from the US) >> * and we will have to keep those min versions updated and bumped >> periodically >> >> The last case is interesting - because it will finally force us to >> introduce some policy on how old providers we support when we release >> airflow. With Airflow 3 I think we are good with basically "latest" >> versions (and maybe get a few versions back for crucial providers). But we >> can discuss it later. >> >> J. >> >> >> >> >> >> On Thu, Apr 10, 2025 at 2:49 PM Vincent Beck <vincb...@apache.org> wrote: >> >>> I assume this is not necessary when the constraint is already set on the >>> provider side? For example, FAB provider has a dependency on Airflow 3 >>> (`apache-airflow>=3.0.0` in `providers/fab/pyproject.toml`) >>> >>> On 2025/04/10 17:22:13 Jarek Potiuk wrote: >>>> Hello, >>>> >>>> TL;DR; Do people who work on some providers think that Airflow 3 should >>>> have minimum version of those? >>>> >>>> While discussing a question about chicken-egg providers with Kaxil, I >>>> realized that we can finally have min versions of providers easily >> added >>> to >>>> "apache-airflow" meta-package. >>>> >>>> Simply speaking - we can say that for example >>>> >>>> apache-airflow[openlineage] -> >>> apache-airflow-providers-openlineage>=2.1.3 >>>> >>>> Previously it was not that easy or straightforward, but now we can do >> it >>>> easily - just changing: >>>> >>>> "openlineage" = [ >>>> "apache-airflow-providers-openlineage" >>>> ] >>>> into >>>> >>>> "openlineage" = [ >>>> "apache-airflow-providers-openlineage>=2.1.3" >>>> ] >>>> >>>> We will likely have to change the tooling a bit to adapt to RC / Dev >>>> packaging versions for chicken-egg-providers - but this should be >> rather >>>> easy. >>>> >>>> And I think this is mostly about openlineage, standard, common and all >>> the >>>> other kinds of providers that are "special" in some ways. >>>> >>>> Are we aware of some minimum versions we **SHOULD** put on those ? >>>> >>>> J. >>>> >>>> >>>> >>>> So the question is. >>>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org >>> For additional commands, e-mail: dev-h...@airflow.apache.org >>> >>> >> --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org For additional commands, e-mail: dev-h...@airflow.apache.org
