potiuk commented on code in PR #33881:
URL: https://github.com/apache/airflow/pull/33881#discussion_r1308776313


##########
airflow/providers/amazon/aws/hooks/appflow.py:
##########
@@ -24,7 +24,6 @@
 
 if TYPE_CHECKING:
     from mypy_boto3_appflow.client import AppflowClient
-    from mypy_boto3_appflow.type_defs import TaskOutputTypeDef, TaskTypeDef
 

Review Comment:
   We actually (sort of) do. The packages get upgraded automatically as the 
boto3 - which means that `--eager-upgrade` of ours will update them 
automatically when they are released. We certainly do not (and cannot) pin and 
update those packages manually (unless someone volunteers for updating it every 
few days). And to add the complexity we are also limited by aiobotocore 
releases - which automatically pin boto3 to specific versions.
   
   And it ain't so easy as having the same version. Currently (due to 
aiobotocore) our "base CI image" has 
   
   * boto3==1.28.17 => we have not upgraded to latest 1.28.36 because 
aiobotocore holds us at 1.28.17 - we will upgrade automatically once 
aiobotocore releases newer version. Latest aiobotocore has:
   ```
   Requires-Dist: boto3 <1.28.18,>=1.28.17 ; extra == 'boto3'
   ```
   
   * mypy-boto3-appflow==1.28.16 -> simply because there is no 1.28.17 and 
1.28.36 (which is next version) apparently has been released today
   
   * the mypy-boto3-rds   1.28.36 has been released today as well and we will 
update to 1.28.36 likely (unless it has some requirements preventing from doing 
it).
   
   same redshift-data
   
   There is no consistency (for example some all the mypy libs had version 
betwen and some did not). And "eventually consistent" approach is I think the 
best we can do.
   
   Of course we could try to somehow manyally manage it and put manual limits 
and bump them when we see we need to. But with 650 dependencies we have, that's 
far too much work - even if we would have a dependabot attempting to do it for 
us we often have 5-10 packages updated a day - so this automation we have to 
attempt to upgrade everything, let pypi figure out the latest "eager upgraded" 
set of dependencies that are not conflicting and passing the tests (only after 
they pass the tests they make it into latest constraints") is probably the best 
we can do.
   
   At the expense of course that sometimes our main and PRs that change 
dependencies (which are few and far between) will get broken by unrelated 
dependency changes.
   
   I have not figured a better way of doing it, but if somone has any ideas, I 
am all ears :).
   
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to