[GitHub] [airflow] potiuk edited a comment on issue #17453: SQLAlchemy constraint for apache-airflow-snowflake-provider installation
potiuk edited a comment on issue #17453: URL: https://github.com/apache/airflow/issues/17453#issuecomment-894623948 Also just one point - conflicting dependencies are not and will never be as huge problem as you might think. I made sure of it when designing the whole system of constraints and automated upgrades. I thought and experimented a lot with that over the last few years and my experiences from last few years are rather positive here. The 'general' approach of almost all active direct dependencies we use is that they are updating dependencies pretty fast and by simply updating to latest versions of dependent packages solves most of the problems. We also advise all our users to update to latest versions of providers when they can (also it is kind of given when we release image and constraints we always use latest released versions of providers and dependencies that work). We are continuously bumping the constraints with 'eager' upgrade (they are updated after all tests pass). And we have mechanisms (which i utilize) to handle exceptions. If you look at Dockerfile.ci and Dockerfile - there are a few hard -limited dependencies in them that handle few cases that are otherwise not easy to handle. But there are just a few of those. Those are helping in making the automated upgrades work. That's why it is important to have constraints, ci, tests, Dockerfile, Dockerfile.ci together in one monorepo. We can only do that because all of those pieces are connected and they are helping each other - Dockerfile and Dockerfile.ci use constraints to build itself, the tests are using resulting images, we can run those images in eager upgrade mode later and re-run the tests, the constraints Are updated after the tests are succesful, they are pushed to the repo, and both docker images arerebuilt with those new upgraded constraints - it is all nicely connected and work in continuous circles of build-test-upgrade. Also the way we approach our dependencies in setup.py makes it quite difficult to get into conflict situation when you look closer - we rarely update minimum versions (mainly when we handle a CVE or incompatible change in implementing certain APIs). We also usually do not add upper-bounds for our dependencies - unless we know they are breaking something. On one hand it is risky (but our constraints and the fact that we only upgrade after successful tests mitigate the risk) but also handle the situations when even major upgrades of dependencies work without any fixes. Not everyone uses SemVer, we cannot rely on that, so we actually will never know if things are going to break. Constraints and making them essential while installing airflow solve the problem very nicely. This actually makes it pretty possible to keep it all working and conflicts are far and few between (and usually can be solved with proper constraints use). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #17453: SQLAlchemy constraint for apache-airflow-snowflake-provider installation
potiuk edited a comment on issue #17453: URL: https://github.com/apache/airflow/issues/17453#issuecomment-894623948 Also just one point - conflicting dependencies are not and will never be as huge problem as you might think. I made sure of it when designing the whole system of constraints and automated upgrades. I thought and experimented a lot with that over the last few years and my experiences from last few years are rather positive here. The 'general' approach of almost all active direct dependencies we use is that they are updating dependencies pretty fast and by simply updating to latest versions of dependent packages solves most of the problems. We also advise all our users to update to latest versions of providers when they can (also it is kind of given when we release image and constraints we always use latest released versions of providers and dependencies that work). We are continuously bumping the constraints with 'eager' upgrade (they are updated after all tests pass). And we have mechanisms (which i utilize) to handle exceptions. If you look at Dockerfile.ci and Dockerfile - there are a few hard -limited dependencies in it that handle few cases that are otherwise not easy to handle. But there are just a few of those. Those are helping in making the automated upgrades work. That's why it is important to have constraints, ci, tests, Dockerfile, Dockerfile.ci together in one monorepo. We can only do that because all of those pieces are connected and they are helping each other - Dockerfile and Dockerfile.ci use constraints to build itself, the tests are using resulting images, we can run those images in eager upgrade mode later and re-run the tests, the constraints Are updated after the tests are succesful, they are pushed to the repo, and both docker images arerebuilt with those new upgraded constraints - it is all nicely connected and work in continuous circles of build-test-upgrade. Also the way we approach our dependencies in setup.py makes it quite difficult to get into conflict situation when you look closer - we rarely update minimum versions (mainly when we handle a CVE or incompatible change in implementing certain APIs). We also usually do not add upper-bounds for our dependencies - unless we know they are breaking something. On one hand it is risky (but our constraints and the fact that we only upgrade after successful tests mitigate the risk) but also handle the situations when even major upgrades of dependencies work without any fixes. Not everyone uses SemVer, we cannot rely on that, so we actually will never know if things are going to break. Constraints and making them essential while installing airflow solve the problem very nicely. This actually makes it pretty possible to keep it all working and conflicts are far and few between (and usually can be solved with proper constraints use). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #17453: SQLAlchemy constraint for apache-airflow-snowflake-provider installation
potiuk edited a comment on issue #17453: URL: https://github.com/apache/airflow/issues/17453#issuecomment-894623948 Also just one point - conflicting dependencies are not and will never be as huge problem as you might think. I made sure of it when designing the whole system of constraints and automated upgrades. I thought and experimented a lot with that over the last few years and my experiences from last few years are rather positive here. The 'general' approach of almost all active direct dependencies we use is that they are updating dependencies pretty fast and by simply updating to latest versions of dependent packages solves most of the problems. We also advise all our users to update to latest versions of providers when they can (also it is kind of given when we release image and constraints we always use latest released versions of providers and dependencies that work). We are continuously bumping the constraints with 'eager' upgrade (they are updated after all tests pass). And we have mechanisms (which i utilize) to handle exceptions. If you look at Dockerfile.ci and Dockerfile - there are a few hard -limited dependencies in it that handle few cases that are otherwise not easy to handle. But there are just a few of those. Those are helping in making the automated upgrades work. That's why it is important to have constraints, ci, tests, Dockerfile, Dockerfile.ci together in one monorepo. We can only do that because all of those pieces are connected and they are helping each other - Dockerfile use, constraints to build, the tests are using resulting images, the constraints use results of those tests and docker image to upgrade itself - it is all nicely connected and work in continuous circles of build-test-upgrade. Also the way we approach our dependencies in setup.py makes it quite difficult to get into conflict situation when you look closer - we rarely update minimum versions (mainly when we handle a CVE or incompatible change in implementing certain APIs). We also usually do not add upper-bounds for our dependencies - unless we know they are breaking something. On one hand it is risky (but our constraints and the fact that we only upgrade after successful tests mitigate the risk) but also handle the situations when even major upgrades of dependencies work without any fixes. Not everyone uses SemVer, we cannot rely on that, so we actually will never know if things are going to break. Constraints and making them essential while installing airflow solve the problem very nicely. This actually makes it pretty possible to keep it all working and conflicts are far and few between (and usually can be solved with proper constraints use). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #17453: SQLAlchemy constraint for apache-airflow-snowflake-provider installation
potiuk edited a comment on issue #17453: URL: https://github.com/apache/airflow/issues/17453#issuecomment-894623948 Also just one point - conflicting dependencies are not and will never be as huge problem as you might think. I made sure of it when designing the whole system of constraints and automated upgrades. I thought and experimented a lot with that over the last few years and my experiences from last few years are rather positive here. The 'general' approach of almost all active direct dependencies we use is that they are updating dependencies pretty fast and by simply updating to latest versions of dependent packages solves most of the problems. We also advise all our users to update to latest versions of providers when they can (also it is kind of given when we release image and constraints we always use latest released versions of providers and dependencies that work). We are continuously bumping the constraints with 'eager' upgrade (they are updated after all tests pass). And we have mechanisms (which i utilize) to handle exceptions. If you look at Dockerfile.ci and Dockerfile - there are a few hard -limited dependencies in it that handle few cases that are otherwise not easy to handle. But there are just a few of those. Those are helping in making the automated upgrades work. That's why it is important to have constraints, ci, tests. Dockerfile together - we can only do that because all of those pieces are connected and they are helping each other - Dockerfile use, constraints to build, the tests are using resulting images, the constraints use results of those tests and docker image to upgrade itself - it is all nicely connected and work in continuous circles of build-test-upgrade. Also the way we approach our dependencies in setup.py makes it quite difficult to get into conflict situation when you look closer - we rarely update minimum versions (mainly when we handle a CVE or incompatible change in implementing certain APIs). We also usually do not add upper-bounds for our dependencies - unless we know they are breaking something. On one hand it is risky (but our constraints and the fact that we only upgrade after successful tests mitigate the risk) but also handle the situations when even major upgrades of dependencies work without any fixes. Not everyone uses SemVer, we cannot rely on that, so we actually will never know if things are going to break. Constraints and making them essential while installing airflow solve the problem very nicely. This actually makes it pretty possible to keep it all working and conflicts are far and few between (and usually can be solved with proper constraints use). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #17453: SQLAlchemy constraint for apache-airflow-snowflake-provider installation
potiuk edited a comment on issue #17453: URL: https://github.com/apache/airflow/issues/17453#issuecomment-894623948 Also just one point - conflicting dependencies are not and will never be as huge problem as you might think. I made sure of it when designing the whole system of constraints and automated upgrades. I thought and experimented a lot with that over the last few years and my experiences from last few years are rather positive here. The 'general' approach of almost all active direct dependencies we use is that they are updating dependencies pretty fast and by simply updating to latest versions of dependent packages solves most of the problems. We also advise all our users to update to latest versions of providers when they can (also it is kind of given when we release image and constraints we always use latest released versions of providers and dependencies that work). We are continuously bumping the constraints with 'eager' upgrade (they are updated after all tests pass). And we have mechanisms (which i utilize) to handle exceptions. If you look at Dockerfile.ci and Dockerfile - there are a few hard -limited dependencies in it that handle few cases that are otherwise not easy to handle. But there are just a few of those. Those are helping in making the automated upgrades work (and that's why it is important to have constraints, ci, tests. Dockerfile together - we can only do that because all of those pieces are connected and they are helping each other - Dockerfile use, constraints to build, the tests are using resulting images, the constraints use results of those tests and docker image to upgrade itself - it is all nicely connected and work in continuous circles of build-test-upgrade. Also the way we approach our dependencies in setup.py makes it quite difficult to get into conflict situation when you look closer - we rarely update minimum versions (mainly when we handle a CVE or incompatible change in implementing certain APIs). We also usually do not add upper-bounds for our dependencies - unless we know they are breaking something. On one hand it is risky (but our constraints and the fact that we only upgrade after successful tests mitigate the risk) but also handle the situations when even major upgrades of dependencies work without any fixes. Not everyone uses SemVer, we cannot rely on that, so we actually will never know if things are going to break. Constraints and making them essential while installing airflow solve the problem very nicely. This actually makes it pretty possible to keep it all working and conflicts are far and few between (and usually can be solved with proper constraints use). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #17453: SQLAlchemy constraint for apache-airflow-snowflake-provider installation
potiuk edited a comment on issue #17453: URL: https://github.com/apache/airflow/issues/17453#issuecomment-894623948 Also just one point - conflicting dependencies are not and will never be as huge problem as you might think. I made sure of it when designing the whole system of constraints and automated upgrades. I thought and experimented a lot with that over the last few years and my experiences from last few years are rather positive here. The 'general' approach of almost all active direct dependencies we use is that they are updating dependencies pretty fast and by simply updating to latest versions of dependent packages solves most of the problems. We also advise all our users to update to latest versions of providers when they can (also it is kind of given when we release image and constraints we always use latest released versions of providers and dependencies that work). We are continuously bumping the constraints with 'eager' upgrade (they are updated after all tests pass). Also the way we approach our dependencies in setup.py makes it quite difficult to get into conflict situation when you look closer - we rarely update minimum versions (mainly when we handle a CVE or incompatible change in implementing certain APIs). We also usually do not add upper-bounds for our dependencies - unless we know they are breaking something. On one hand it is risky (but our constraints and the fact that we only upgrade after successful tests mitigate the risk) but also handle the situations when even major upgrades of dependencies work without any fixes. Not everyone uses SemVer, we cannot rely on that, so we actually will never know if things are going to break. Constraints and making them essential while installing airflow solve the problem very nicely. This actually makes it pretty possible to keep it all working and conflicts are far and few between (and usually can be solved with proper constraints use). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #17453: SQLAlchemy constraint for apache-airflow-snowflake-provider installation
potiuk edited a comment on issue #17453: URL: https://github.com/apache/airflow/issues/17453#issuecomment-894623948 Also just one point - conflicting constraint are not and will never be as huge problem as you might think. I made sure of it when designing the whole system of constraints and automated upgrades. I thought and experimented a lot with that over the last few years and my experiences from last few years are rather positive here. The 'general' approach of almost all active direct dependencies we use is that they are updating dependencies pretty fast and by simply updating to latest versions of dependent packages solves most of the problems. We also advise all our users to update to latest versions of providers when they can (also it is kind of given when we release image and constraints we always use latest released versions of providers and dependencies that work). We are continuously bumping the constraints with 'eager' upgrade (they are updated after all tests pass). Also the way we approach our dependencies in setup.py makes it quite difficult to get into conflict situation when you look closer - we rarely update minimum versions (mainly when we handle a CVE or incompatible change in implementing certain APIs). We also usually do not add upper-bounds for our dependencies - unless we know they are breaking something. On one hand it is risky (but our constraints and the fact that we only upgrade after successful tests mitigate the risk) but also handle the situations when even major upgrades of dependencies work without any fixes. Not everyone uses SemVer, we cannot rely on that, so we actually will never know if things are going to break. Constraints and making them essential while installing airflow solve the problem very nicely. This actually makes it pretty possible to keep it all working and conflicts are far and few between (and usually can be solved with proper constraints use). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #17453: SQLAlchemy constraint for apache-airflow-snowflake-provider installation
potiuk edited a comment on issue #17453: URL: https://github.com/apache/airflow/issues/17453#issuecomment-894623948 Also just one point - conflicting constraint are not and will never be as huge problem as you might think. I made sure of it when designing the whole system of constraints and automated upgrades. I thought and experimented a lot with that over the last few years and my experiences from last few years are rather positive here. The 'general' approach of almost all active direct dependencies we use is that they are updating dependencies pretty fast and by simply updating to latest versions of dependent packages solves most of the problem. We also advise all our users to update to latest versions of providers when they can (also it is kind of given when we release image and constraints we always use latest released versions of providers and dependencies that work). We are continuously bumping the constraints with 'eager' upgrade (they are updated after all tests pass). Also the way we approach our dependencies in setup.py makes it quite difficult to get into conflict situation when you look closer - we rarely update minimum versions (mainly when we handle a CVE or incompatible change in implementing certain APIs). We also usually do not add upper-bounds for our dependencies - unless we know they are breaking something. On one hand it is risky (but our constraints and the fact that we only upgrade after successful tests mitigate the risk) but also handle the situations when even major upgrades of dependencies work without any fixes. Not everyone uses SemVer, we cannot rely on that, so we actually will never know if things are going to break. Constraints and making them essential while installing airflow solve the problem very nicely. This actually makes it pretty possible to keep it all working and conflict are far and few between (and usually can be solved with proper constraints use). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #17453: SQLAlchemy constraint for apache-airflow-snowflake-provider installation
potiuk edited a comment on issue #17453: URL: https://github.com/apache/airflow/issues/17453#issuecomment-894184031 Also I am a bit surprised that this happened. By default, when `pip install ` is run and the installed version of dependency falls in the limits of the dependencies, it should NOT upgrade it to later version (you need to specify `eager` version of upgrade in order that it happens by PIP flags). @wolfier -> could you please describe exactly what you have done and how you instlaled the newer version of snowflake provider? Did you have airflow already installed ? which version? what was the command you used to upgrade the snowflake provider? Which version of PIP you used? (or was it a different way of installing - with poetry or pip-tools maybe?) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
