A draft PR here (i will split it as there are few unrelated changes)
https://github.com/apache/airflow/pull/49103 -> but it shows how we can
automatically set the min versions in "apache-airflow" meta-package (with
6-months-old provider versions).
That will quite likely help to get the "Resolution too deep" error solved,
and I think it has a nice property - that we can treat it as "recommended"
range of versions for providers - while giving all the flexibility to the
user to keep their old versions or downgrade.
J.
On Fri, Apr 11, 2025 at 7:01 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> Actually what I see when I attempted to do it - is that even if we
> introduce it - it is not going to be a hard requirement. It is just a soft
> limit when installing full apache-airflow[with extras].
>
> It will be more of a default behaviour to limit the providers when extras
> are used but will be easy to override and it will not apply when you
> install airflow-core /task-sdk/ providers directly. So yes - I think no
> real drawback
>
> This limits will only be on extras of the 'apache-airflow' and in case
> someone used the extras so far - the were not able to control the version
> of provider that will be used anyway - and this only used a lower version
> of provider than latest I'd some other package was installed at the same
> time that was conflicting with latest provider..
>
> Thoss limits are not going to be persistent, they are going to simply be
> taken into account when apache-airflow{provider} is used and only for the
> time of the installation. It will not prevent the user from
> upgrading/downgrading the provider later. And also user still be able to
> remove extra and use the provider directly in the same `pip install`
> command
>
> I have a draft change that sets the lower bound to now() -6 months +
> manual overrides if we need to get some new provider limited to latest
> version this way for example. I am still coming back from US and will be
> home later today and try it and I think some form of it will be good to
> have - 6 months seems reasonable and it will have the 'self-maintenance'
> capability.
>
> J.
>
> czw., 10 kwi 2025, 22:46 użytkownik Ash Berlin-Taylor <a...@apache.org>
> napisał:
>
>> There absolutely is a downside to the users — “forcing" users to upgrade
>> providers along with Airflow version makes upgrades a much more daunting
>> experience.
>>
>> > On 10 Apr 2025, at 21:31, Maciej Obuchowski <mobuchow...@apache.org>
>> wrote:
>> >
>> > For OpenLineage we'd definitely like something like this.
>> > IMO there is no downside to restricting the providers to the latest
>> > released version.
>> > We could go with the chicken-egg for the next release - that would add
>> at
>> > least one useful
>> > feature authored by Kacper Muda -
>> > https://github.com/apache/airflow/pull/48941
>> > but even without it it's not broken, just missing that feature.
>> >
>> >
>> > czw., 10 kwi 2025 o 21:49 Jarek Potiuk <ja...@potiuk.com> napisał(a):
>> >
>> >> Accidentally It turned out - with today's investigation of why RC
>> images
>> >> are not building that it is VERY needed. It seems that `pip` resolution
>> >> works currently with our meta-package in a very bad way. If in our
>> >> pyproject.toml we have just
>> >> "amazon' = "apache-airflow-providers-amazon", "google" =
>> >> "apache-airflow-providers-google" ....
>> >>
>> >> then
>> >>
>> >> apache-airflow[google,amazon] - will lead pip to download and try to
>> >> resolve ALL VERSIONS or apparently ALL providers that have ever been
>> >> released. That means we have 100 amazon, 100 google - and if you have
>> more
>> >> extras - 10s or 100s of other providers to consider. And essentially
>> what
>> >> pip tries to do is - try all combinations of those to find out which
>> ones
>> >> are good. Or at least very, very big subset of those providers. Which
>> >> actually explains the "Resolution Too Deep" problem we tried to
>> diagnose
>> >> and investigate over the last few weeks.
>> >>
>> >> So summarising:
>> >> * resolution too deep was actually "real" issue
>> >> * it was caused by no lower-bounds in providers in meta package
>> >> * we will have to set those lower-binds to something reasonable (and I
>> >> might start later today with simply latest released versions if i have
>> >> internet on my flight back from the US)
>> >> * and we will have to keep those min versions updated and bumped
>> >> periodically
>> >>
>> >> The last case is interesting - because it will finally force us to
>> >> introduce some policy on how old providers we support when we release
>> >> airflow. With Airflow 3 I think we are good with basically "latest"
>> >> versions (and maybe get a few versions back for crucial providers).
>> But we
>> >> can discuss it later.
>> >>
>> >> J.
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> On Thu, Apr 10, 2025 at 2:49 PM Vincent Beck <vincb...@apache.org>
>> wrote:
>> >>
>> >>> I assume this is not necessary when the constraint is already set on
>> the
>> >>> provider side? For example, FAB provider has a dependency on Airflow 3
>> >>> (`apache-airflow>=3.0.0` in `providers/fab/pyproject.toml`)
>> >>>
>> >>> On 2025/04/10 17:22:13 Jarek Potiuk wrote:
>> >>>> Hello,
>> >>>>
>> >>>> TL;DR; Do people who work on some providers think that Airflow 3
>> should
>> >>>> have minimum version of those?
>> >>>>
>> >>>> While discussing a question about chicken-egg providers with Kaxil, I
>> >>>> realized that we can finally have min versions of providers easily
>> >> added
>> >>> to
>> >>>> "apache-airflow" meta-package.
>> >>>>
>> >>>> Simply speaking - we can say that for example
>> >>>>
>> >>>> apache-airflow[openlineage] ->
>> >>> apache-airflow-providers-openlineage>=2.1.3
>> >>>>
>> >>>> Previously it was not that easy or straightforward, but now we can do
>> >> it
>> >>>> easily - just changing:
>> >>>>
>> >>>> "openlineage" = [
>> >>>> "apache-airflow-providers-openlineage"
>> >>>> ]
>> >>>> into
>> >>>>
>> >>>> "openlineage" = [
>> >>>> "apache-airflow-providers-openlineage>=2.1.3"
>> >>>> ]
>> >>>>
>> >>>> We will likely have to change the tooling a bit to adapt to RC / Dev
>> >>>> packaging versions for chicken-egg-providers - but this should be
>> >> rather
>> >>>> easy.
>> >>>>
>> >>>> And I think this is mostly about openlineage, standard, common and
>> all
>> >>> the
>> >>>> other kinds of providers that are "special" in some ways.
>> >>>>
>> >>>> Are we aware of some minimum versions we **SHOULD** put on those ?
>> >>>>
>> >>>> J.
>> >>>>
>> >>>>
>> >>>>
>> >>>> So the question is.
>> >>>>
>> >>>
>> >>> ---------------------------------------------------------------------
>> >>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
>> >>> For additional commands, e-mail: dev-h...@airflow.apache.org
>> >>>
>> >>>
>> >>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
>> For additional commands, e-mail: dev-h...@airflow.apache.org
>>
>>
