potiuk commented on PR #30539:
URL: https://github.com/apache/airflow/pull/30539#issuecomment-1500902806

   I think we do not need to do anything. This is quite expected to be honest.
   
   The error is really about some dependencies "friction" (I would not even say 
it's conflict) when you install all latest providers and airflow 2.3 together. 
We will never get it 100% as this is already really difficult to keep those 
dependencies in check for "latest" version, but it's not really problematic - 
because installing "all latest providers" on "old airrflow" is not going to 
happenm and it's very likely that the same "friction" will not appear when you 
install only a subset of providers. Also as a deployment manager it's a bit 
your responsibility to make sure your dependencies are not in conflict (pip and 
install-requires will prevent some of the serious conflicts, but not all.
   
   This particular example (and it is somewhat related to a change @ash added 
to requests and we were lobbying for it for a long time until we managed (with 
a LOT of struggle) the requests maintainers to make chardet optional because of 
LGPL licence it used (you can see an epic history of the issue here 
https://github.com/psf/requests/pull/5797) :
   
   * This only happens on our latest CI image, when we remove latest airflow, 
replace it with 2.3 and install all latest providers on top of it (this is 
mainly done to see if there are any imports from latest airflow in latest 
providers that make the top-level imports fail with 2.3 and prevent it).
   
   * while this is done, requests library 2.28.2 (released in Jan 2023) gets 
replaced with 2.27.1 (released in Jan 2022). Why exactly this happens and which 
dependency does it - it's hard to say. This is `pip` dependency resolver in 
play, which makes the decision which dependencies needs to be brought down and 
which brouhgt up to get "best match".
   
   * also we have chardet 5.1.0 (which is currently installed in latest CI) and 
it remains untouched by pip - the 2.27.1. even though it has <  5 limit in 
`requests` (but the limit is only enforced for Python 2.7, because chardet - 
due to the issue above has been turned into optional dependency, and with pip 
you cannot "enforrce" limits on optional dependencies. But this limit has been 
pretty "artifficial" - because version 6 of chardet did not cause any troubles 
for requests. This 
https://github.com/psf/requests/commit/73793ce43e3d940a0f8d6b9b8ff125617828d8a1 
is a change that bumped the upper limit to 6 - it merely changed the limits, 
did not fix anything. Genearlly chardet is really an optional feature of 
requests that almost no-one uses (it detects charset based on the content of an 
HTTP requests) and this is a rather obscure feature that requests maintainers 
want to drop as soon as they bump major version of requests (we discussed it 
extensively).
   
   * We have more similar "soft" conflicts in this 2.3 environment (`pip is 
happy to install them and select those versions when resolving but reports them 
as incompatible)  - and we cannot do much about it:
   
   ```
   types-paramiko 3.0.0.7 requires cryptography>=37.0.0, but you have 
cryptography 36.0.2 which is incompatible.
   sentry-sdk 1.19.1 requires urllib3>=1.26.11; python_version >= "3.6", but 
you have urllib3 1.26.9 which is incompatible.
   pytest-httpx 0.21.3 requires httpx==0.23.*, but you have httpx 0.22.0 which 
is incompatible.
   openapi-spec-validator 0.5.6 requires importlib-resources<6.0.0,>=5.8.0; 
python_version < "3.9", but you have importlib-resources 5.7.1 which is 
incompatible.
   keyring 23.13.1 requires importlib-metadata>=4.11.4; python_version < 
"3.12", but you have importlib-metadata 4.11.3 which is incompatible.
   jsonschema-spec 0.1.4 requires typing-extensions<5.0.0,>=4.3.0, but you have 
typing-extensions 4.2.0 which is incompatible.
   google-cloud-dataform 0.2.0 requires google-api-core[grpc]<3.0.0dev,>=2.8.0, 
but you have google-api-core 2.7.2 which is incompatible.
   google-cloud-dataflow-client 0.5.4 requires 
google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0,
 but you have google-api-core 2.7.2 which is incompatible.
   gevent 22.10.2 requires greenlet>=2.0.0; platform_python_implementation == 
"CPython", but you have greenlet 1.1.2 which is incompatible.
   flask-limiter 3.3.0 requires Flask>=2, but you have flask 1.1.2 which is 
incompatible.
   azure-storage-file-datalake 12.10.1 requires azure-core<2.0.0,>=1.26.0, but 
you have azure-core 1.23.1 which is incompatible.
   azure-storage-file-datalake 12.10.1 requires 
azure-storage-blob<13.0.0,>=12.15.0, but you have azure-storage-blob 12.8.1 
which is incompatible.
   aws-sam-translator 1.64.0 requires typing-extensions<5,>=4.4, but you have 
typing-extensions 4.2.0 which is incompatible.
   aiobotocore 2.5.0 requires botocore<1.29.77,>=1.29.76, but you have botocore 
1.25.0 which is incompatible.
   ```
   
   Some of them are a bit less, some of them a bit more worrying, but I am 
afraid we cannot do much, those incompatibilities get worse over time when we 
install old version of Airflow.
    
   EXCEPT ONE THING THAT WE CAN AND WILL DO:
   
   * Next month we bump min-airflow version to 2.4 for all providers. And the 
main reason for that is because we are aware those "jpint" dependencies with 
old versions of airflow will "deteriorate" over time and it will be harder and 
harder to install new provider on old airflow. This is the main reason we have 
the current policy of supporting min-airlfow version in providers 12 months 
after first version from 2.N line has been released - to account for such 
crawling dependency deterioration.
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to