potiuk edited a comment on issue #17453: URL: https://github.com/apache/airflow/issues/17453#issuecomment-894623948
Also just one point - conflicting dependencies are not and will never be as huge problem as you might think. I made sure of it when designing the whole system of constraints and automated upgrades. I thought and experimented a lot with that over the last few years and my experiences from last few years are rather positive here. The 'general' approach of almost all active direct dependencies we use is that they are updating dependencies pretty fast and by simply updating to latest versions of dependent packages solves most of the problems. We also advise all our users to update to latest versions of providers when they can (also it is kind of given when we release image and constraints we always use latest released versions of providers and dependencies that work). We are continuously bumping the constraints with 'eager' upgrade (they are updated after all tests pass). And we have mechanisms (which i utilize) to handle exceptions. If you look at Dockerfile.ci and Dockerfile - there are a few hard -limited dependencies in it that handle few cases that are otherwise not easy to handle. But there are just a few of those. Those are helping in making the automated upgrades work (and that's why it is important to have constraints, ci, tests. Dockerfile together - we can only do that because all of those pieces are connected and they are helping each other - Dockerfile use, constraints to build, the tests are using resulting images, the constraints use results of those tests and docker image to upgrade itself - it is all nicely connected and work in continuous circles of build-test-upgrade. Also the way we approach our dependencies in setup.py makes it quite difficult to get into conflict situation when you look closer - we rarely update minimum versions (mainly when we handle a CVE or incompatible change in implementing certain APIs). We also usually do not add upper-bounds for our dependencies - unless we know they are breaking something. On one hand it is risky (but our constraints and the fact that we only upgrade after successful tests mitigate the risk) but also handle the situations when even major upgrades of dependencies work without any fixes. Not everyone uses SemVer, we cannot rely on that, so we actually will never know if things are going to break. Constraints and making them essential while installing airflow solve the problem very nicely. This actually makes it pretty possible to keep it all working and conflicts are far and few between (and usually can be solved with proper constraints use). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
