-1: Pankaj is right. We need to fix the dependencies. The aiobotocore was
mistakenly added as the "required" dependency 2 days ago in:
https://github.com/apache/airflow/pull/30032
However for the Amazon provider we have it as an optional one.
Currently in provider,yaml we have both:
dependencies:
....
- aiobotocore[boto3]>=2.2.0
and:
additional-extras:
- name: pandas
dependencies:
- pandas>=0.17.1
# There is conflict between boto3 and aiobotocore dependency botocore.
# TODO: We can remove it once boto3 and aiobotocore both have
compatible botocore version or
# boto3 have native async support and we move away from aio aiobotocore
- name: aiobotocore
dependencies:
- aiobotocore>=2.1.1
The first one should be removed and the second one updated. PR is coming.
J.
On Wed, Apr 26, 2023 at 12:26 AM Mehta, Shubham <[email protected]>
wrote:
> Hey Pankaj,
>
> I have a few clarifying questions:
> 1. Can you specify the Airflow version used during testing? Based on the
> botocore version constraint, it appears to be v2.5.3.
> 2. Did you use a modified constraint file to install the Amazon provider
> v8.0.0rc2 with Airflow v2.5.3? If so, what were the modifications?
>
> As you know, we added the _aiobotocore_ dependency to support deferrable
> operators in the Amazon provider package (AMPP) because boto3 does not
> offer async support by default. This decision has some implications, which
> we discussed in detail within the related GitHub issues:
> 1. _aiobotocore_ is included as a dependency in the constraints file
> starting from Airflow v2.6.0 and onwards. Consequently, users will not
> encounter version conflict issues for any Airflow version equal to or
> greater than 2.6.0.
> 2. Since _aiobotocore_ was not added as a dependency for older Airflow
> versions, conflicts may arise if you attempt to use newer AMPP versions by
> selectively modifying the constraint file for the AMPP version alone,
> without adjusting botocore and boto3 versions.
>
> In summary, for older Airflow versions, you must modify the constraints
> for AMPP, botocore, and boto3. For newer Airflow versions, the process will
> remain smooth, as it was before. Although this may result in a slightly
> worse user experience, it is the only alternative we have in order to
> support deferrable operators in AMPP.
>
> Thanks
> Shubham
>
> On 2023-04-25, 11:57 AM, "Pankaj Singh" <[email protected]
> <mailto:[email protected]>> wrote:
>
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
>
>
>
> -1 (non-binding)
>
>
> We won't be able to use the amazon provider for airflow version less than
> equal to 2.5.3 because conflict cause by aibotocore
>
>
>
>
> ERROR: Cannot install aiobotocore[boto3]==2.2.0, aiobotocore[boto3]==2.3.0,
> aiobotocore[boto3]==2.3.1, aiobotocore[boto3]==2.3.2,
> aiobotocore[boto3]==2.3.3, aiobotocore[boto3]==2.3.4,
> aiobotocore[boto3]==2.4.0, aiobotocore[boto3]==2.4.1,
> aiobotocore[boto3]==2.4.2 and aiobotocore[boto3]==2.5.0 because these
> package versions have conflicting dependencies.
>
>
> The conflict is caused by:
> aiobotocore[boto3] 2.5.0 depends on botocore<1.29.77 and >=1.29.76
> aiobotocore[boto3] 2.4.2 depends on botocore<1.27.60 and >=1.27.59
> aiobotocore[boto3] 2.4.1 depends on botocore<1.27.60 and >=1.27.59
> aiobotocore[boto3] 2.4.0 depends on botocore<1.27.60 and >=1.27.59
> aiobotocore[boto3] 2.3.4 depends on botocore<1.24.22 and >=1.24.21
> aiobotocore[boto3] 2.3.3 depends on botocore<1.24.22 and >=1.24.21
> aiobotocore[boto3] 2.3.2 depends on botocore<1.24.22 and >=1.24.21
> aiobotocore[boto3] 2.3.1 depends on botocore<1.24.22 and >=1.24.21
> aiobotocore[boto3] 2.3.0 depends on botocore<1.24.22 and >=1.24.21
> aiobotocore[boto3] 2.2.0 depends on botocore<1.24.22 and >=1.24.21
> The user requested (constraint) botocore==1.29.102
>
>
> On Tue, Apr 25, 2023 at 8:35 PM Jed Cunningham <[email protected]
> <mailto:[email protected]>>
> wrote:
>
>
> > +1 (binding)
> >
> > Checked signatures, checksums, licences.
> >
>
>
>
>
