Hey Pankaj, I have a few clarifying questions: 1. Can you specify the Airflow version used during testing? Based on the botocore version constraint, it appears to be v2.5.3. 2. Did you use a modified constraint file to install the Amazon provider v8.0.0rc2 with Airflow v2.5.3? If so, what were the modifications?
As you know, we added the _aiobotocore_ dependency to support deferrable operators in the Amazon provider package (AMPP) because boto3 does not offer async support by default. This decision has some implications, which we discussed in detail within the related GitHub issues: 1. _aiobotocore_ is included as a dependency in the constraints file starting from Airflow v2.6.0 and onwards. Consequently, users will not encounter version conflict issues for any Airflow version equal to or greater than 2.6.0. 2. Since _aiobotocore_ was not added as a dependency for older Airflow versions, conflicts may arise if you attempt to use newer AMPP versions by selectively modifying the constraint file for the AMPP version alone, without adjusting botocore and boto3 versions. In summary, for older Airflow versions, you must modify the constraints for AMPP, botocore, and boto3. For newer Airflow versions, the process will remain smooth, as it was before. Although this may result in a slightly worse user experience, it is the only alternative we have in order to support deferrable operators in AMPP. Thanks Shubham On 2023-04-25, 11:57 AM, "Pankaj Singh" <[email protected] <mailto:[email protected]>> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. -1 (non-binding) We won't be able to use the amazon provider for airflow version less than equal to 2.5.3 because conflict cause by aibotocore ERROR: Cannot install aiobotocore[boto3]==2.2.0, aiobotocore[boto3]==2.3.0, aiobotocore[boto3]==2.3.1, aiobotocore[boto3]==2.3.2, aiobotocore[boto3]==2.3.3, aiobotocore[boto3]==2.3.4, aiobotocore[boto3]==2.4.0, aiobotocore[boto3]==2.4.1, aiobotocore[boto3]==2.4.2 and aiobotocore[boto3]==2.5.0 because these package versions have conflicting dependencies. The conflict is caused by: aiobotocore[boto3] 2.5.0 depends on botocore<1.29.77 and >=1.29.76 aiobotocore[boto3] 2.4.2 depends on botocore<1.27.60 and >=1.27.59 aiobotocore[boto3] 2.4.1 depends on botocore<1.27.60 and >=1.27.59 aiobotocore[boto3] 2.4.0 depends on botocore<1.27.60 and >=1.27.59 aiobotocore[boto3] 2.3.4 depends on botocore<1.24.22 and >=1.24.21 aiobotocore[boto3] 2.3.3 depends on botocore<1.24.22 and >=1.24.21 aiobotocore[boto3] 2.3.2 depends on botocore<1.24.22 and >=1.24.21 aiobotocore[boto3] 2.3.1 depends on botocore<1.24.22 and >=1.24.21 aiobotocore[boto3] 2.3.0 depends on botocore<1.24.22 and >=1.24.21 aiobotocore[boto3] 2.2.0 depends on botocore<1.24.22 and >=1.24.21 The user requested (constraint) botocore==1.29.102 On Tue, Apr 25, 2023 at 8:35 PM Jed Cunningham <[email protected] <mailto:[email protected]>> wrote: > +1 (binding) > > Checked signatures, checksums, licences. >
