potiuk commented on code in PR #33881:
URL: https://github.com/apache/airflow/pull/33881#discussion_r1308776313
##########
airflow/providers/amazon/aws/hooks/appflow.py:
##########
@@ -24,7 +24,6 @@
if TYPE_CHECKING:
from mypy_boto3_appflow.client import AppflowClient
- from mypy_boto3_appflow.type_defs import TaskOutputTypeDef, TaskTypeDef
Review Comment:
We actually (sort of) do. The packages get upgraded automatically as the
boto3 - which means that `--eager-upgrade` of ours will update them
automatically when they are released. We certainly do not (and cannot) pin and
update those packages manually (unless someone volunteers for updating it every
few days). And to add the complexity we are also limited by aiobotocore
releases - which automatically pin boto3 to specific versions.
And it ain't so easy as having the same version. Currently (due to
aiobotocore) our "base CI image" has
* boto3==1.28.17 => we have not upgraded to latest 1.28.36 because
aiobotocore holds us at 1.28.17 - we will upgrade automatically once
aiobotocore releases newer version. Latest aiobotocore has:
```
Requires-Dist: boto3 <1.28.18,>=1.28.17 ; extra == 'boto3'
```
* mypy-boto3-appflow==1.28.16 -> simply because there is no 1.28.17 and
1.28.36 (which is next version) apparently has been released today
* the mypy-boto3-rds 1.28.36 has been released today as well and we will
update to 1.28.36 likely (unless it has some requirements preventing from doing
it).
same redshift-data
There is no consistency (for example some all the mypy libs had version
betwen and some did not). And "eventually consistent" approach is I think the
best we can do.
Of course we could try to somehow manyally manage it and put manual limits
and bump them when we see we need to. But with 650 dependencies we have, that's
far too much work - even if we would have a dependabot attempting to do it for
us we often have 5-10 packages updated a day - so this automation we have to
attempt to upgrade everything, let pypi figure out the latest "eager upgraded"
set of dependencies that are not conflicting and passing the tests (only after
they pass the tests they make it into latest constraints") is probably the best
we can do.
At the expense of course that sometimes our main and PRs that change
dependencies (which are few and far between) will get broken by unrelated
dependency changes.
I have not figured a better way of doing it, but if somone has any ideas, I
am all ears :).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]