Oh hang on wait a moment. Is aiobotocore a new extra that is not available in 2.10.5 or any early versions?
If that is the case then I’d instead vote that we don’t add new extras to `apache-airflow`, and we instead remove it. I grudgingly accept that having `apache-airflow[amazon]` make sense, but `aiobotocore` isn’t the name of the provider, and you need to know what it means anyway, so I don’t think we should have it as an extra anywhere other than on the provider. -ash > On 11 Apr 2025, at 08:40, Jarek Potiuk <ja...@potiuk.com> wrote: > > A draft PR here (i will split it as there are few unrelated changes) > https://github.com/apache/airflow/pull/49103 -> but it shows how we can > automatically set the min versions in "apache-airflow" meta-package (with > 6-months-old provider versions). > > That will quite likely help to get the "Resolution too deep" error solved, > and I think it has a nice property - that we can treat it as "recommended" > range of versions for providers - while giving all the flexibility to the > user to keep their old versions or downgrade. > > > J. > > On Fri, Apr 11, 2025 at 7:01 AM Jarek Potiuk <ja...@potiuk.com> wrote: > >> Actually what I see when I attempted to do it - is that even if we >> introduce it - it is not going to be a hard requirement. It is just a soft >> limit when installing full apache-airflow[with extras]. >> >> It will be more of a default behaviour to limit the providers when extras >> are used but will be easy to override and it will not apply when you >> install airflow-core /task-sdk/ providers directly. So yes - I think no >> real drawback >> >> This limits will only be on extras of the 'apache-airflow' and in case >> someone used the extras so far - the were not able to control the version >> of provider that will be used anyway - and this only used a lower version >> of provider than latest I'd some other package was installed at the same >> time that was conflicting with latest provider.. >> >> Thoss limits are not going to be persistent, they are going to simply be >> taken into account when apache-airflow{provider} is used and only for the >> time of the installation. It will not prevent the user from >> upgrading/downgrading the provider later. And also user still be able to >> remove extra and use the provider directly in the same `pip install` >> command >> >> I have a draft change that sets the lower bound to now() -6 months + >> manual overrides if we need to get some new provider limited to latest >> version this way for example. I am still coming back from US and will be >> home later today and try it and I think some form of it will be good to >> have - 6 months seems reasonable and it will have the 'self-maintenance' >> capability. >> >> J. >> >> czw., 10 kwi 2025, 22:46 użytkownik Ash Berlin-Taylor <a...@apache.org> >> napisał: >> >>> There absolutely is a downside to the users — “forcing" users to upgrade >>> providers along with Airflow version makes upgrades a much more daunting >>> experience. >>> >>>> On 10 Apr 2025, at 21:31, Maciej Obuchowski <mobuchow...@apache.org> >>> wrote: >>>> >>>> For OpenLineage we'd definitely like something like this. >>>> IMO there is no downside to restricting the providers to the latest >>>> released version. >>>> We could go with the chicken-egg for the next release - that would add >>> at >>>> least one useful >>>> feature authored by Kacper Muda - >>>> https://github.com/apache/airflow/pull/48941 >>>> but even without it it's not broken, just missing that feature. >>>> >>>> >>>> czw., 10 kwi 2025 o 21:49 Jarek Potiuk <ja...@potiuk.com> napisał(a): >>>> >>>>> Accidentally It turned out - with today's investigation of why RC >>> images >>>>> are not building that it is VERY needed. It seems that `pip` resolution >>>>> works currently with our meta-package in a very bad way. If in our >>>>> pyproject.toml we have just >>>>> "amazon' = "apache-airflow-providers-amazon", "google" = >>>>> "apache-airflow-providers-google" .... >>>>> >>>>> then >>>>> >>>>> apache-airflow[google,amazon] - will lead pip to download and try to >>>>> resolve ALL VERSIONS or apparently ALL providers that have ever been >>>>> released. That means we have 100 amazon, 100 google - and if you have >>> more >>>>> extras - 10s or 100s of other providers to consider. And essentially >>> what >>>>> pip tries to do is - try all combinations of those to find out which >>> ones >>>>> are good. Or at least very, very big subset of those providers. Which >>>>> actually explains the "Resolution Too Deep" problem we tried to >>> diagnose >>>>> and investigate over the last few weeks. >>>>> >>>>> So summarising: >>>>> * resolution too deep was actually "real" issue >>>>> * it was caused by no lower-bounds in providers in meta package >>>>> * we will have to set those lower-binds to something reasonable (and I >>>>> might start later today with simply latest released versions if i have >>>>> internet on my flight back from the US) >>>>> * and we will have to keep those min versions updated and bumped >>>>> periodically >>>>> >>>>> The last case is interesting - because it will finally force us to >>>>> introduce some policy on how old providers we support when we release >>>>> airflow. With Airflow 3 I think we are good with basically "latest" >>>>> versions (and maybe get a few versions back for crucial providers). >>> But we >>>>> can discuss it later. >>>>> >>>>> J. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Thu, Apr 10, 2025 at 2:49 PM Vincent Beck <vincb...@apache.org> >>> wrote: >>>>> >>>>>> I assume this is not necessary when the constraint is already set on >>> the >>>>>> provider side? For example, FAB provider has a dependency on Airflow 3 >>>>>> (`apache-airflow>=3.0.0` in `providers/fab/pyproject.toml`) >>>>>> >>>>>> On 2025/04/10 17:22:13 Jarek Potiuk wrote: >>>>>>> Hello, >>>>>>> >>>>>>> TL;DR; Do people who work on some providers think that Airflow 3 >>> should >>>>>>> have minimum version of those? >>>>>>> >>>>>>> While discussing a question about chicken-egg providers with Kaxil, I >>>>>>> realized that we can finally have min versions of providers easily >>>>> added >>>>>> to >>>>>>> "apache-airflow" meta-package. >>>>>>> >>>>>>> Simply speaking - we can say that for example >>>>>>> >>>>>>> apache-airflow[openlineage] -> >>>>>> apache-airflow-providers-openlineage>=2.1.3 >>>>>>> >>>>>>> Previously it was not that easy or straightforward, but now we can do >>>>> it >>>>>>> easily - just changing: >>>>>>> >>>>>>> "openlineage" = [ >>>>>>> "apache-airflow-providers-openlineage" >>>>>>> ] >>>>>>> into >>>>>>> >>>>>>> "openlineage" = [ >>>>>>> "apache-airflow-providers-openlineage>=2.1.3" >>>>>>> ] >>>>>>> >>>>>>> We will likely have to change the tooling a bit to adapt to RC / Dev >>>>>>> packaging versions for chicken-egg-providers - but this should be >>>>> rather >>>>>>> easy. >>>>>>> >>>>>>> And I think this is mostly about openlineage, standard, common and >>> all >>>>>> the >>>>>>> other kinds of providers that are "special" in some ways. >>>>>>> >>>>>>> Are we aware of some minimum versions we **SHOULD** put on those ? >>>>>>> >>>>>>> J. >>>>>>> >>>>>>> >>>>>>> >>>>>>> So the question is. >>>>>>> >>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org >>>>>> For additional commands, e-mail: dev-h...@airflow.apache.org >>>>>> >>>>>> >>>>> >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org >>> For additional commands, e-mail: dev-h...@airflow.apache.org >>> >>> --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org For additional commands, e-mail: dev-h...@airflow.apache.org
