Actually what I see when I attempted to do it - is that even if we
introduce it - it is not going to be a hard requirement. It is just a soft
limit when installing full apache-airflow[with extras].
It will be more of a default behaviour to limit the providers when extras
are used but will be easy to override and it will not apply when you
install airflow-core /task-sdk/ providers directly. So yes - I think no
real drawback
This limits will only be on extras of the 'apache-airflow' and in case
someone used the extras so far - the were not able to control the version
of provider that will be used anyway - and this only used a lower version
of provider than latest I'd some other package was installed at the same
time that was conflicting with latest provider..
Thoss limits are not going to be persistent, they are going to simply be
taken into account when apache-airflow{provider} is used and only for the
time of the installation. It will not prevent the user from
upgrading/downgrading the provider later. And also user still be able to
remove extra and use the provider directly in the same `pip install`
command
I have a draft change that sets the lower bound to now() -6 months + manual
overrides if we need to get some new provider limited to latest version
this way for example. I am still coming back from US and will be home later
today and try it and I think some form of it will be good to have - 6
months seems reasonable and it will have the 'self-maintenance' capability.
J.
czw., 10 kwi 2025, 22:46 użytkownik Ash Berlin-Taylor <a...@apache.org>
napisał:
> There absolutely is a downside to the users — “forcing" users to upgrade
> providers along with Airflow version makes upgrades a much more daunting
> experience.
>
> > On 10 Apr 2025, at 21:31, Maciej Obuchowski <mobuchow...@apache.org>
> wrote:
> >
> > For OpenLineage we'd definitely like something like this.
> > IMO there is no downside to restricting the providers to the latest
> > released version.
> > We could go with the chicken-egg for the next release - that would add at
> > least one useful
> > feature authored by Kacper Muda -
> > https://github.com/apache/airflow/pull/48941
> > but even without it it's not broken, just missing that feature.
> >
> >
> > czw., 10 kwi 2025 o 21:49 Jarek Potiuk <ja...@potiuk.com> napisał(a):
> >
> >> Accidentally It turned out - with today's investigation of why RC images
> >> are not building that it is VERY needed. It seems that `pip` resolution
> >> works currently with our meta-package in a very bad way. If in our
> >> pyproject.toml we have just
> >> "amazon' = "apache-airflow-providers-amazon", "google" =
> >> "apache-airflow-providers-google" ....
> >>
> >> then
> >>
> >> apache-airflow[google,amazon] - will lead pip to download and try to
> >> resolve ALL VERSIONS or apparently ALL providers that have ever been
> >> released. That means we have 100 amazon, 100 google - and if you have
> more
> >> extras - 10s or 100s of other providers to consider. And essentially
> what
> >> pip tries to do is - try all combinations of those to find out which
> ones
> >> are good. Or at least very, very big subset of those providers. Which
> >> actually explains the "Resolution Too Deep" problem we tried to diagnose
> >> and investigate over the last few weeks.
> >>
> >> So summarising:
> >> * resolution too deep was actually "real" issue
> >> * it was caused by no lower-bounds in providers in meta package
> >> * we will have to set those lower-binds to something reasonable (and I
> >> might start later today with simply latest released versions if i have
> >> internet on my flight back from the US)
> >> * and we will have to keep those min versions updated and bumped
> >> periodically
> >>
> >> The last case is interesting - because it will finally force us to
> >> introduce some policy on how old providers we support when we release
> >> airflow. With Airflow 3 I think we are good with basically "latest"
> >> versions (and maybe get a few versions back for crucial providers). But
> we
> >> can discuss it later.
> >>
> >> J.
> >>
> >>
> >>
> >>
> >>
> >> On Thu, Apr 10, 2025 at 2:49 PM Vincent Beck <vincb...@apache.org>
> wrote:
> >>
> >>> I assume this is not necessary when the constraint is already set on
> the
> >>> provider side? For example, FAB provider has a dependency on Airflow 3
> >>> (`apache-airflow>=3.0.0` in `providers/fab/pyproject.toml`)
> >>>
> >>> On 2025/04/10 17:22:13 Jarek Potiuk wrote:
> >>>> Hello,
> >>>>
> >>>> TL;DR; Do people who work on some providers think that Airflow 3
> should
> >>>> have minimum version of those?
> >>>>
> >>>> While discussing a question about chicken-egg providers with Kaxil, I
> >>>> realized that we can finally have min versions of providers easily
> >> added
> >>> to
> >>>> "apache-airflow" meta-package.
> >>>>
> >>>> Simply speaking - we can say that for example
> >>>>
> >>>> apache-airflow[openlineage] ->
> >>> apache-airflow-providers-openlineage>=2.1.3
> >>>>
> >>>> Previously it was not that easy or straightforward, but now we can do
> >> it
> >>>> easily - just changing:
> >>>>
> >>>> "openlineage" = [
> >>>> "apache-airflow-providers-openlineage"
> >>>> ]
> >>>> into
> >>>>
> >>>> "openlineage" = [
> >>>> "apache-airflow-providers-openlineage>=2.1.3"
> >>>> ]
> >>>>
> >>>> We will likely have to change the tooling a bit to adapt to RC / Dev
> >>>> packaging versions for chicken-egg-providers - but this should be
> >> rather
> >>>> easy.
> >>>>
> >>>> And I think this is mostly about openlineage, standard, common and all
> >>> the
> >>>> other kinds of providers that are "special" in some ways.
> >>>>
> >>>> Are we aware of some minimum versions we **SHOULD** put on those ?
> >>>>
> >>>> J.
> >>>>
> >>>>
> >>>>
> >>>> So the question is.
> >>>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
> >>> For additional commands, e-mail: dev-h...@airflow.apache.org
> >>>
> >>>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
> For additional commands, e-mail: dev-h...@airflow.apache.org
>
>
