That's a bit of a different topic :)  - I was only asking about the minimum
provider versions.

And I think no-one wants to add new extras - but  we do have a bunch of
them already. What we have now is that (see pyproject-toml files in both) :

* `apache-airflow-core` has a few of those historically to add "features"
to the core , but some (say kerberos) - does not have providers.
airflow-core currently has NO provider extras - they are only defined in
the meta-package
* `apache-airflow`  meta- on the other hand - has "pass-through" core
extras (same as "airflow-core") for back compatibility + it has provider
extras

Possibly we should also remove some of the "core" extras (for example I
think async, cgroups and a few others might not make sense there. But I
think this is another discussion . I will start it separately

J.


On Fri, Apr 11, 2025 at 12:30 PM Kaxil Naik <kaxiln...@gmail.com> wrote:

> Agreed on not having extras apart from providers.
>
> On Fri, 11 Apr 2025 at 13:36, Ash Berlin-Taylor <a...@apache.org> wrote:
>
> > Just checked, no this was in 2.10 releases. NM then.
> >
> > Still, I think we should not add new extras to `apache-airflow` in
> > general, and definitely not for anything other than direct provider
> > “short-names”.
> >
> >
> > -ash
> >
> > > On 11 Apr 2025, at 09:02, Ash Berlin-Taylor <a...@apache.org> wrote:
> > >
> > > Oh hang on wait a moment.
> > >
> > > Is aiobotocore a new extra that is not available in 2.10.5 or any early
> > versions?
> > >
> > > If that is the case then I’d instead vote that we don’t add new extras
> > to `apache-airflow`, and we instead remove it. I grudgingly accept that
> > having `apache-airflow[amazon]` make sense, but `aiobotocore` isn’t the
> > name of the provider, and you need to know what it means anyway, so I
> don’t
> > think we should have it as an extra anywhere other than on the provider.
> > >
> > > -ash
> > >
> > >> On 11 Apr 2025, at 08:40, Jarek Potiuk <ja...@potiuk.com> wrote:
> > >>
> > >> A draft PR here (i will split it as there are few unrelated changes)
> > >> https://github.com/apache/airflow/pull/49103 -> but it shows how we
> can
> > >> automatically set the min versions in "apache-airflow" meta-package
> > (with
> > >> 6-months-old provider versions).
> > >>
> > >> That will quite likely help to get the "Resolution too deep" error
> > solved,
> > >> and I think it has a nice property - that we can treat it as
> > "recommended"
> > >> range of versions for providers - while giving all the flexibility to
> > the
> > >> user to keep their old versions or downgrade.
> > >>
> > >>
> > >> J.
> > >>
> > >> On Fri, Apr 11, 2025 at 7:01 AM Jarek Potiuk <ja...@potiuk.com>
> wrote:
> > >>
> > >>> Actually what I see when I attempted to do it - is that even if we
> > >>> introduce it - it is not going to be a hard requirement. It is just a
> > soft
> > >>> limit when installing full apache-airflow[with extras].
> > >>>
> > >>> It will be more of a default behaviour to limit the providers when
> > extras
> > >>> are used but will be easy to override and it will not apply when you
> > >>> install airflow-core /task-sdk/ providers directly. So yes - I think
> no
> > >>> real drawback
> > >>>
> > >>> This limits will only be on extras of the 'apache-airflow' and in
> case
> > >>> someone used the extras so far - the were not able to control the
> > version
> > >>> of provider that will be used anyway - and this only used a lower
> > version
> > >>> of provider than latest I'd some other package was installed at the
> > same
> > >>> time that was conflicting with latest provider..
> > >>>
> > >>> Thoss limits are not going to be persistent, they are going to simply
> > be
> > >>> taken into account when apache-airflow{provider} is used and only for
> > the
> > >>> time of the installation. It will not prevent the user from
> > >>> upgrading/downgrading the provider later. And also user still be able
> > to
> > >>> remove extra and use the provider directly in the same `pip install`
> > >>> command
> > >>>
> > >>> I have a draft change that sets the lower bound to now() -6 months +
> > >>> manual overrides if we need to get some new provider limited to
> latest
> > >>> version this way for example. I am still coming back from US and will
> > be
> > >>> home later today and try it and I think some form of it will be good
> to
> > >>> have  - 6 months seems reasonable and it will have the
> > 'self-maintenance'
> > >>> capability.
> > >>>
> > >>> J.
> > >>>
> > >>> czw., 10 kwi 2025, 22:46 użytkownik Ash Berlin-Taylor <
> a...@apache.org>
> > >>> napisał:
> > >>>
> > >>>> There absolutely is a downside to the users — “forcing" users to
> > upgrade
> > >>>> providers along with Airflow version makes upgrades a much more
> > daunting
> > >>>> experience.
> > >>>>
> > >>>>> On 10 Apr 2025, at 21:31, Maciej Obuchowski <
> mobuchow...@apache.org>
> > >>>> wrote:
> > >>>>>
> > >>>>> For OpenLineage we'd definitely like something like this.
> > >>>>> IMO there is no downside to restricting the providers to the latest
> > >>>>> released version.
> > >>>>> We could go with the chicken-egg for the next release - that would
> > add
> > >>>> at
> > >>>>> least one useful
> > >>>>> feature authored by Kacper Muda -
> > >>>>> https://github.com/apache/airflow/pull/48941
> > >>>>> but even without it it's not broken, just missing that feature.
> > >>>>>
> > >>>>>
> > >>>>> czw., 10 kwi 2025 o 21:49 Jarek Potiuk <ja...@potiuk.com>
> > napisał(a):
> > >>>>>
> > >>>>>> Accidentally It turned out - with today's investigation of why RC
> > >>>> images
> > >>>>>> are not building that it is VERY needed. It seems that `pip`
> > resolution
> > >>>>>> works currently with our meta-package in a very bad way. If in our
> > >>>>>> pyproject.toml we have just
> > >>>>>> "amazon' =  "apache-airflow-providers-amazon", "google" =
> > >>>>>> "apache-airflow-providers-google" ....
> > >>>>>>
> > >>>>>> then
> > >>>>>>
> > >>>>>> apache-airflow[google,amazon] - will lead pip to download and try
> to
> > >>>>>> resolve ALL VERSIONS or apparently ALL providers that have ever
> been
> > >>>>>> released. That means we have 100 amazon, 100 google - and if you
> > have
> > >>>> more
> > >>>>>> extras - 10s or 100s of other providers to consider. And
> essentially
> > >>>> what
> > >>>>>> pip tries to do is - try all combinations of those to find out
> which
> > >>>> ones
> > >>>>>> are good. Or at least very, very big subset of those providers.
> > Which
> > >>>>>> actually explains the "Resolution Too Deep" problem we tried to
> > >>>> diagnose
> > >>>>>> and investigate over the last few weeks.
> > >>>>>>
> > >>>>>> So summarising:
> > >>>>>> * resolution too deep was actually "real" issue
> > >>>>>> * it was caused by no lower-bounds in providers in meta package
> > >>>>>> * we will have to set those lower-binds to something reasonable
> > (and I
> > >>>>>> might start later today with simply latest released versions if i
> > have
> > >>>>>> internet on my flight back from the US)
> > >>>>>> * and we will have to keep those min versions updated and bumped
> > >>>>>> periodically
> > >>>>>>
> > >>>>>> The last case is interesting - because it will finally force us to
> > >>>>>> introduce some policy on how old providers we support when we
> > release
> > >>>>>> airflow. With Airflow 3 I think we are good with basically
> "latest"
> > >>>>>> versions (and maybe get a few versions back for crucial
> providers).
> > >>>> But we
> > >>>>>> can discuss it later.
> > >>>>>>
> > >>>>>> J.
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> On Thu, Apr 10, 2025 at 2:49 PM Vincent Beck <vincb...@apache.org
> >
> > >>>> wrote:
> > >>>>>>
> > >>>>>>> I assume this is not necessary when the constraint is already set
> > on
> > >>>> the
> > >>>>>>> provider side? For example, FAB provider has a dependency on
> > Airflow 3
> > >>>>>>> (`apache-airflow>=3.0.0` in `providers/fab/pyproject.toml`)
> > >>>>>>>
> > >>>>>>> On 2025/04/10 17:22:13 Jarek Potiuk wrote:
> > >>>>>>>> Hello,
> > >>>>>>>>
> > >>>>>>>> TL;DR; Do people who work on some providers think that Airflow 3
> > >>>> should
> > >>>>>>>> have minimum version of those?
> > >>>>>>>>
> > >>>>>>>> While discussing a question about chicken-egg providers with
> > Kaxil, I
> > >>>>>>>> realized that we can finally have min versions of providers
> easily
> > >>>>>> added
> > >>>>>>> to
> > >>>>>>>> "apache-airflow" meta-package.
> > >>>>>>>>
> > >>>>>>>> Simply speaking - we can say that for example
> > >>>>>>>>
> > >>>>>>>> apache-airflow[openlineage] ->
> > >>>>>>> apache-airflow-providers-openlineage>=2.1.3
> > >>>>>>>>
> > >>>>>>>> Previously it was not that easy or straightforward, but now we
> > can do
> > >>>>>> it
> > >>>>>>>> easily - just changing:
> > >>>>>>>>
> > >>>>>>>> "openlineage" = [
> > >>>>>>>>  "apache-airflow-providers-openlineage"
> > >>>>>>>> ]
> > >>>>>>>> into
> > >>>>>>>>
> > >>>>>>>> "openlineage" = [
> > >>>>>>>>  "apache-airflow-providers-openlineage>=2.1.3"
> > >>>>>>>> ]
> > >>>>>>>>
> > >>>>>>>> We will likely have to change the tooling a bit to adapt to RC /
> > Dev
> > >>>>>>>> packaging versions for chicken-egg-providers - but this should
> be
> > >>>>>> rather
> > >>>>>>>> easy.
> > >>>>>>>>
> > >>>>>>>> And I think this is mostly about openlineage, standard, common
> and
> > >>>> all
> > >>>>>>> the
> > >>>>>>>> other kinds of providers that are "special" in some ways.
> > >>>>>>>>
> > >>>>>>>> Are we aware of some minimum versions we **SHOULD** put on
> those ?
> > >>>>>>>>
> > >>>>>>>> J.
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> So the question is.
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>>
> > ---------------------------------------------------------------------
> > >>>>>>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
> > >>>>>>> For additional commands, e-mail: dev-h...@airflow.apache.org
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>>>
> > >>>>
> ---------------------------------------------------------------------
> > >>>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
> > >>>> For additional commands, e-mail: dev-h...@airflow.apache.org
> > >>>>
> > >>>>
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
> > > For additional commands, e-mail: dev-h...@airflow.apache.org
> > >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
> > For additional commands, e-mail: dev-h...@airflow.apache.org
> >
> >
>

Reply via email to