Sounds good On Fri, 11 Apr 2025 at 16:16, Jarek Potiuk <ja...@potiuk.com> wrote:
> That's a bit of a different topic :) - I was only asking about the minimum > provider versions. > > And I think no-one wants to add new extras - but we do have a bunch of > them already. What we have now is that (see pyproject-toml files in both) : > > * `apache-airflow-core` has a few of those historically to add "features" > to the core , but some (say kerberos) - does not have providers. > airflow-core currently has NO provider extras - they are only defined in > the meta-package > * `apache-airflow` meta- on the other hand - has "pass-through" core > extras (same as "airflow-core") for back compatibility + it has provider > extras > > Possibly we should also remove some of the "core" extras (for example I > think async, cgroups and a few others might not make sense there. But I > think this is another discussion . I will start it separately > > J. > > > On Fri, Apr 11, 2025 at 12:30 PM Kaxil Naik <kaxiln...@gmail.com> wrote: > > > Agreed on not having extras apart from providers. > > > > On Fri, 11 Apr 2025 at 13:36, Ash Berlin-Taylor <a...@apache.org> wrote: > > > > > Just checked, no this was in 2.10 releases. NM then. > > > > > > Still, I think we should not add new extras to `apache-airflow` in > > > general, and definitely not for anything other than direct provider > > > “short-names”. > > > > > > > > > -ash > > > > > > > On 11 Apr 2025, at 09:02, Ash Berlin-Taylor <a...@apache.org> wrote: > > > > > > > > Oh hang on wait a moment. > > > > > > > > Is aiobotocore a new extra that is not available in 2.10.5 or any > early > > > versions? > > > > > > > > If that is the case then I’d instead vote that we don’t add new > extras > > > to `apache-airflow`, and we instead remove it. I grudgingly accept that > > > having `apache-airflow[amazon]` make sense, but `aiobotocore` isn’t the > > > name of the provider, and you need to know what it means anyway, so I > > don’t > > > think we should have it as an extra anywhere other than on the > provider. > > > > > > > > -ash > > > > > > > >> On 11 Apr 2025, at 08:40, Jarek Potiuk <ja...@potiuk.com> wrote: > > > >> > > > >> A draft PR here (i will split it as there are few unrelated changes) > > > >> https://github.com/apache/airflow/pull/49103 -> but it shows how we > > can > > > >> automatically set the min versions in "apache-airflow" meta-package > > > (with > > > >> 6-months-old provider versions). > > > >> > > > >> That will quite likely help to get the "Resolution too deep" error > > > solved, > > > >> and I think it has a nice property - that we can treat it as > > > "recommended" > > > >> range of versions for providers - while giving all the flexibility > to > > > the > > > >> user to keep their old versions or downgrade. > > > >> > > > >> > > > >> J. > > > >> > > > >> On Fri, Apr 11, 2025 at 7:01 AM Jarek Potiuk <ja...@potiuk.com> > > wrote: > > > >> > > > >>> Actually what I see when I attempted to do it - is that even if we > > > >>> introduce it - it is not going to be a hard requirement. It is > just a > > > soft > > > >>> limit when installing full apache-airflow[with extras]. > > > >>> > > > >>> It will be more of a default behaviour to limit the providers when > > > extras > > > >>> are used but will be easy to override and it will not apply when > you > > > >>> install airflow-core /task-sdk/ providers directly. So yes - I > think > > no > > > >>> real drawback > > > >>> > > > >>> This limits will only be on extras of the 'apache-airflow' and in > > case > > > >>> someone used the extras so far - the were not able to control the > > > version > > > >>> of provider that will be used anyway - and this only used a lower > > > version > > > >>> of provider than latest I'd some other package was installed at the > > > same > > > >>> time that was conflicting with latest provider.. > > > >>> > > > >>> Thoss limits are not going to be persistent, they are going to > simply > > > be > > > >>> taken into account when apache-airflow{provider} is used and only > for > > > the > > > >>> time of the installation. It will not prevent the user from > > > >>> upgrading/downgrading the provider later. And also user still be > able > > > to > > > >>> remove extra and use the provider directly in the same `pip > install` > > > >>> command > > > >>> > > > >>> I have a draft change that sets the lower bound to now() -6 months > + > > > >>> manual overrides if we need to get some new provider limited to > > latest > > > >>> version this way for example. I am still coming back from US and > will > > > be > > > >>> home later today and try it and I think some form of it will be > good > > to > > > >>> have - 6 months seems reasonable and it will have the > > > 'self-maintenance' > > > >>> capability. > > > >>> > > > >>> J. > > > >>> > > > >>> czw., 10 kwi 2025, 22:46 użytkownik Ash Berlin-Taylor < > > a...@apache.org> > > > >>> napisał: > > > >>> > > > >>>> There absolutely is a downside to the users — “forcing" users to > > > upgrade > > > >>>> providers along with Airflow version makes upgrades a much more > > > daunting > > > >>>> experience. > > > >>>> > > > >>>>> On 10 Apr 2025, at 21:31, Maciej Obuchowski < > > mobuchow...@apache.org> > > > >>>> wrote: > > > >>>>> > > > >>>>> For OpenLineage we'd definitely like something like this. > > > >>>>> IMO there is no downside to restricting the providers to the > latest > > > >>>>> released version. > > > >>>>> We could go with the chicken-egg for the next release - that > would > > > add > > > >>>> at > > > >>>>> least one useful > > > >>>>> feature authored by Kacper Muda - > > > >>>>> https://github.com/apache/airflow/pull/48941 > > > >>>>> but even without it it's not broken, just missing that feature. > > > >>>>> > > > >>>>> > > > >>>>> czw., 10 kwi 2025 o 21:49 Jarek Potiuk <ja...@potiuk.com> > > > napisał(a): > > > >>>>> > > > >>>>>> Accidentally It turned out - with today's investigation of why > RC > > > >>>> images > > > >>>>>> are not building that it is VERY needed. It seems that `pip` > > > resolution > > > >>>>>> works currently with our meta-package in a very bad way. If in > our > > > >>>>>> pyproject.toml we have just > > > >>>>>> "amazon' = "apache-airflow-providers-amazon", "google" = > > > >>>>>> "apache-airflow-providers-google" .... > > > >>>>>> > > > >>>>>> then > > > >>>>>> > > > >>>>>> apache-airflow[google,amazon] - will lead pip to download and > try > > to > > > >>>>>> resolve ALL VERSIONS or apparently ALL providers that have ever > > been > > > >>>>>> released. That means we have 100 amazon, 100 google - and if you > > > have > > > >>>> more > > > >>>>>> extras - 10s or 100s of other providers to consider. And > > essentially > > > >>>> what > > > >>>>>> pip tries to do is - try all combinations of those to find out > > which > > > >>>> ones > > > >>>>>> are good. Or at least very, very big subset of those providers. > > > Which > > > >>>>>> actually explains the "Resolution Too Deep" problem we tried to > > > >>>> diagnose > > > >>>>>> and investigate over the last few weeks. > > > >>>>>> > > > >>>>>> So summarising: > > > >>>>>> * resolution too deep was actually "real" issue > > > >>>>>> * it was caused by no lower-bounds in providers in meta package > > > >>>>>> * we will have to set those lower-binds to something reasonable > > > (and I > > > >>>>>> might start later today with simply latest released versions if > i > > > have > > > >>>>>> internet on my flight back from the US) > > > >>>>>> * and we will have to keep those min versions updated and bumped > > > >>>>>> periodically > > > >>>>>> > > > >>>>>> The last case is interesting - because it will finally force us > to > > > >>>>>> introduce some policy on how old providers we support when we > > > release > > > >>>>>> airflow. With Airflow 3 I think we are good with basically > > "latest" > > > >>>>>> versions (and maybe get a few versions back for crucial > > providers). > > > >>>> But we > > > >>>>>> can discuss it later. > > > >>>>>> > > > >>>>>> J. > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> On Thu, Apr 10, 2025 at 2:49 PM Vincent Beck < > vincb...@apache.org > > > > > > >>>> wrote: > > > >>>>>> > > > >>>>>>> I assume this is not necessary when the constraint is already > set > > > on > > > >>>> the > > > >>>>>>> provider side? For example, FAB provider has a dependency on > > > Airflow 3 > > > >>>>>>> (`apache-airflow>=3.0.0` in `providers/fab/pyproject.toml`) > > > >>>>>>> > > > >>>>>>> On 2025/04/10 17:22:13 Jarek Potiuk wrote: > > > >>>>>>>> Hello, > > > >>>>>>>> > > > >>>>>>>> TL;DR; Do people who work on some providers think that > Airflow 3 > > > >>>> should > > > >>>>>>>> have minimum version of those? > > > >>>>>>>> > > > >>>>>>>> While discussing a question about chicken-egg providers with > > > Kaxil, I > > > >>>>>>>> realized that we can finally have min versions of providers > > easily > > > >>>>>> added > > > >>>>>>> to > > > >>>>>>>> "apache-airflow" meta-package. > > > >>>>>>>> > > > >>>>>>>> Simply speaking - we can say that for example > > > >>>>>>>> > > > >>>>>>>> apache-airflow[openlineage] -> > > > >>>>>>> apache-airflow-providers-openlineage>=2.1.3 > > > >>>>>>>> > > > >>>>>>>> Previously it was not that easy or straightforward, but now we > > > can do > > > >>>>>> it > > > >>>>>>>> easily - just changing: > > > >>>>>>>> > > > >>>>>>>> "openlineage" = [ > > > >>>>>>>> "apache-airflow-providers-openlineage" > > > >>>>>>>> ] > > > >>>>>>>> into > > > >>>>>>>> > > > >>>>>>>> "openlineage" = [ > > > >>>>>>>> "apache-airflow-providers-openlineage>=2.1.3" > > > >>>>>>>> ] > > > >>>>>>>> > > > >>>>>>>> We will likely have to change the tooling a bit to adapt to > RC / > > > Dev > > > >>>>>>>> packaging versions for chicken-egg-providers - but this should > > be > > > >>>>>> rather > > > >>>>>>>> easy. > > > >>>>>>>> > > > >>>>>>>> And I think this is mostly about openlineage, standard, common > > and > > > >>>> all > > > >>>>>>> the > > > >>>>>>>> other kinds of providers that are "special" in some ways. > > > >>>>>>>> > > > >>>>>>>> Are we aware of some minimum versions we **SHOULD** put on > > those ? > > > >>>>>>>> > > > >>>>>>>> J. > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> So the question is. > > > >>>>>>>> > > > >>>>>>> > > > >>>>>>> > > > --------------------------------------------------------------------- > > > >>>>>>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > >>>>>>> For additional commands, e-mail: dev-h...@airflow.apache.org > > > >>>>>>> > > > >>>>>>> > > > >>>>>> > > > >>>> > > > >>>> > > > >>>> > > --------------------------------------------------------------------- > > > >>>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > >>>> For additional commands, e-mail: dev-h...@airflow.apache.org > > > >>>> > > > >>>> > > > > > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > > > > > >
