Thank Jarek, good catch! You're right, it's better to keep aiobotocore as additional dependency for now. If you attempt to install apache-airflow-providers-amazon[aiobotocore] using constraints for older Airflow versions, like Pankaj, you will still encounter conflicts. However, the impact would be less severe, as the use of Amazon's deferrable functionality would be minimal in the beginning.
On 2023-04-25, 5:15 PM, "Jarek Potiuk" <[email protected] <mailto:[email protected]>> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. PR here: https://github.com/apache/airflow/pull/30874 <https://github.com/apache/airflow/pull/30874> On Wed, Apr 26, 2023 at 2:08 AM Jarek Potiuk <[email protected] <mailto:[email protected]>> wrote: > -1: Pankaj is right. We need to fix the dependencies. The aiobotocore was > mistakenly added as the "required" dependency 2 days ago in: > https://github.com/apache/airflow/pull/30032 > <https://github.com/apache/airflow/pull/30032> > > However for the Amazon provider we have it as an optional one. > > Currently in provider,yaml we have both: > > dependencies: > .... > - aiobotocore[boto3]>=2.2.0 > > > and: > > additional-extras: > - name: pandas > dependencies: > - pandas>=0.17.1 > # There is conflict between boto3 and aiobotocore dependency botocore. > # TODO: We can remove it once boto3 and aiobotocore both have compatible > botocore version or > # boto3 have native async support and we move away from aio aiobotocore > - name: aiobotocore > dependencies: > - aiobotocore>=2.1.1 > > > The first one should be removed and the second one updated. PR is coming. > > J. > > > > > > On Wed, Apr 26, 2023 at 12:26 AM Mehta, Shubham <[email protected] > <mailto:[email protected]>lid> > wrote: > >> Hey Pankaj, >> >> I have a few clarifying questions: >> 1. Can you specify the Airflow version used during testing? Based on the >> botocore version constraint, it appears to be v2.5.3. >> 2. Did you use a modified constraint file to install the Amazon provider >> v8.0.0rc2 with Airflow v2.5.3? If so, what were the modifications? >> >> As you know, we added the _aiobotocore_ dependency to support deferrable >> operators in the Amazon provider package (AMPP) because boto3 does not >> offer async support by default. This decision has some implications, which >> we discussed in detail within the related GitHub issues: >> 1. _aiobotocore_ is included as a dependency in the constraints file >> starting from Airflow v2.6.0 and onwards. Consequently, users will not >> encounter version conflict issues for any Airflow version equal to or >> greater than 2.6.0. >> 2. Since _aiobotocore_ was not added as a dependency for older Airflow >> versions, conflicts may arise if you attempt to use newer AMPP versions by >> selectively modifying the constraint file for the AMPP version alone, >> without adjusting botocore and boto3 versions. >> >> In summary, for older Airflow versions, you must modify the constraints >> for AMPP, botocore, and boto3. For newer Airflow versions, the process will >> remain smooth, as it was before. Although this may result in a slightly >> worse user experience, it is the only alternative we have in order to >> support deferrable operators in AMPP. >> >> Thanks >> Shubham >> >> On 2023-04-25, 11:57 AM, "Pankaj Singh" <[email protected] >> <mailto:[email protected]> >> <mailto:[email protected] <mailto:[email protected]>>> wrote: >> >> >> CAUTION: This email originated from outside of the organization. Do not >> click links or open attachments unless you can confirm the sender and know >> the content is safe. >> >> >> >> >> >> >> -1 (non-binding) >> >> >> We won't be able to use the amazon provider for airflow version less than >> equal to 2.5.3 because conflict cause by aibotocore >> >> >> >> >> ERROR: Cannot install aiobotocore[boto3]==2.2.0, >> aiobotocore[boto3]==2.3.0, >> aiobotocore[boto3]==2.3.1, aiobotocore[boto3]==2.3.2, >> aiobotocore[boto3]==2.3.3, aiobotocore[boto3]==2.3.4, >> aiobotocore[boto3]==2.4.0, aiobotocore[boto3]==2.4.1, >> aiobotocore[boto3]==2.4.2 and aiobotocore[boto3]==2.5.0 because these >> package versions have conflicting dependencies. >> >> >> The conflict is caused by: >> aiobotocore[boto3] 2.5.0 depends on botocore<1.29.77 and >=1.29.76 >> aiobotocore[boto3] 2.4.2 depends on botocore<1.27.60 and >=1.27.59 >> aiobotocore[boto3] 2.4.1 depends on botocore<1.27.60 and >=1.27.59 >> aiobotocore[boto3] 2.4.0 depends on botocore<1.27.60 and >=1.27.59 >> aiobotocore[boto3] 2.3.4 depends on botocore<1.24.22 and >=1.24.21 >> aiobotocore[boto3] 2.3.3 depends on botocore<1.24.22 and >=1.24.21 >> aiobotocore[boto3] 2.3.2 depends on botocore<1.24.22 and >=1.24.21 >> aiobotocore[boto3] 2.3.1 depends on botocore<1.24.22 and >=1.24.21 >> aiobotocore[boto3] 2.3.0 depends on botocore<1.24.22 and >=1.24.21 >> aiobotocore[boto3] 2.2.0 depends on botocore<1.24.22 and >=1.24.21 >> The user requested (constraint) botocore==1.29.102 >> >> >> On Tue, Apr 25, 2023 at 8:35 PM Jed Cunningham <[email protected] >> <mailto:[email protected]> >> <mailto:[email protected] <mailto:[email protected]>>> >> wrote: >> >> >> > +1 (binding) >> > >> > Checked signatures, checksums, licences. >> > >> >> >> >> --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
