PR here: https://github.com/apache/airflow/pull/30874
On Wed, Apr 26, 2023 at 2:08 AM Jarek Potiuk <[email protected]> wrote: > -1: Pankaj is right. We need to fix the dependencies. The aiobotocore was > mistakenly added as the "required" dependency 2 days ago in: > https://github.com/apache/airflow/pull/30032 > > However for the Amazon provider we have it as an optional one. > > Currently in provider,yaml we have both: > > dependencies: > .... > - aiobotocore[boto3]>=2.2.0 > > > and: > > additional-extras: > - name: pandas > dependencies: > - pandas>=0.17.1 > # There is conflict between boto3 and aiobotocore dependency botocore. > # TODO: We can remove it once boto3 and aiobotocore both have compatible > botocore version or > # boto3 have native async support and we move away from aio aiobotocore > - name: aiobotocore > dependencies: > - aiobotocore>=2.1.1 > > > The first one should be removed and the second one updated. PR is coming. > > J. > > > > > > On Wed, Apr 26, 2023 at 12:26 AM Mehta, Shubham <[email protected]> > wrote: > >> Hey Pankaj, >> >> I have a few clarifying questions: >> 1. Can you specify the Airflow version used during testing? Based on the >> botocore version constraint, it appears to be v2.5.3. >> 2. Did you use a modified constraint file to install the Amazon provider >> v8.0.0rc2 with Airflow v2.5.3? If so, what were the modifications? >> >> As you know, we added the _aiobotocore_ dependency to support deferrable >> operators in the Amazon provider package (AMPP) because boto3 does not >> offer async support by default. This decision has some implications, which >> we discussed in detail within the related GitHub issues: >> 1. _aiobotocore_ is included as a dependency in the constraints file >> starting from Airflow v2.6.0 and onwards. Consequently, users will not >> encounter version conflict issues for any Airflow version equal to or >> greater than 2.6.0. >> 2. Since _aiobotocore_ was not added as a dependency for older Airflow >> versions, conflicts may arise if you attempt to use newer AMPP versions by >> selectively modifying the constraint file for the AMPP version alone, >> without adjusting botocore and boto3 versions. >> >> In summary, for older Airflow versions, you must modify the constraints >> for AMPP, botocore, and boto3. For newer Airflow versions, the process will >> remain smooth, as it was before. Although this may result in a slightly >> worse user experience, it is the only alternative we have in order to >> support deferrable operators in AMPP. >> >> Thanks >> Shubham >> >> On 2023-04-25, 11:57 AM, "Pankaj Singh" <[email protected] >> <mailto:[email protected]>> wrote: >> >> >> CAUTION: This email originated from outside of the organization. Do not >> click links or open attachments unless you can confirm the sender and know >> the content is safe. >> >> >> >> >> >> >> -1 (non-binding) >> >> >> We won't be able to use the amazon provider for airflow version less than >> equal to 2.5.3 because conflict cause by aibotocore >> >> >> >> >> ERROR: Cannot install aiobotocore[boto3]==2.2.0, >> aiobotocore[boto3]==2.3.0, >> aiobotocore[boto3]==2.3.1, aiobotocore[boto3]==2.3.2, >> aiobotocore[boto3]==2.3.3, aiobotocore[boto3]==2.3.4, >> aiobotocore[boto3]==2.4.0, aiobotocore[boto3]==2.4.1, >> aiobotocore[boto3]==2.4.2 and aiobotocore[boto3]==2.5.0 because these >> package versions have conflicting dependencies. >> >> >> The conflict is caused by: >> aiobotocore[boto3] 2.5.0 depends on botocore<1.29.77 and >=1.29.76 >> aiobotocore[boto3] 2.4.2 depends on botocore<1.27.60 and >=1.27.59 >> aiobotocore[boto3] 2.4.1 depends on botocore<1.27.60 and >=1.27.59 >> aiobotocore[boto3] 2.4.0 depends on botocore<1.27.60 and >=1.27.59 >> aiobotocore[boto3] 2.3.4 depends on botocore<1.24.22 and >=1.24.21 >> aiobotocore[boto3] 2.3.3 depends on botocore<1.24.22 and >=1.24.21 >> aiobotocore[boto3] 2.3.2 depends on botocore<1.24.22 and >=1.24.21 >> aiobotocore[boto3] 2.3.1 depends on botocore<1.24.22 and >=1.24.21 >> aiobotocore[boto3] 2.3.0 depends on botocore<1.24.22 and >=1.24.21 >> aiobotocore[boto3] 2.2.0 depends on botocore<1.24.22 and >=1.24.21 >> The user requested (constraint) botocore==1.29.102 >> >> >> On Tue, Apr 25, 2023 at 8:35 PM Jed Cunningham <[email protected] >> <mailto:[email protected]>> >> wrote: >> >> >> > +1 (binding) >> > >> > Checked signatures, checksums, licences. >> > >> >> >> >>
