Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
vincbeck commented on issue #28723: URL: https://github.com/apache/airflow/issues/28723#issuecomment-3711088427 Completed by #59218 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
vincbeck closed issue #28723: Fix all deprecations for SQLAlchemy 2.0 URL: https://github.com/apache/airflow/issues/28723 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
vincbeck commented on issue #28723: URL: https://github.com/apache/airflow/issues/28723#issuecomment-3340329705 Yes, this PR is open to use flask-appbuilder in FAB provider: https://github.com/apache/airflow/pull/50960. We are close :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
taranarmo commented on issue #28723: URL: https://github.com/apache/airflow/issues/28723#issuecomment-3340321901 BTW FAB 5.0 was released recently introducing support for SQLA 2.x https://github.com/dpgaspar/Flask-AppBuilder/releases/tag/v5.0.0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
anton-bedrock commented on issue #28723: URL: https://github.com/apache/airflow/issues/28723#issuecomment-3339793840 What is the plan for SQL Alchemy 2? Seems Airflow 3.1 still missing it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
kaxil commented on issue #28723: URL: https://github.com/apache/airflow/issues/28723#issuecomment-3339850209 Yeah, Python 3.13: https://github.com/apache/airflow/blob/constraints-3-1/constraints-3.13.txt#L56 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
Dev-iL commented on issue #28723: URL: https://github.com/apache/airflow/issues/28723#issuecomment-3339813524 @anton-bedrock I think one _can_ use SQLA2 **but only when**: on python 3.13 and not using the Flask Auth Manager (fab provider). Otherwise - see plan [here](https://github.com/apache/airflow/pull/55954#issuecomment-3334107487). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
Dev-iL commented on issue #28723: URL: https://github.com/apache/airflow/issues/28723#issuecomment-3339780998 Another PR related to this epic: https://github.com/apache/airflow/pull/55954 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
Dev-iL commented on issue #28723: URL: https://github.com/apache/airflow/issues/28723#issuecomment-3004935545 If anyone wants to help move this issue along, you might want to base your work off of my branch mentioned in the PR right above this comment - which adds a CI job that runs all tests with SQLA v2. Locally, these tests can be run using ```bash breeze testing core-tests --upgrade-sqlalchemy --maxfail=100 # Expect many failures initially ``` Currently, most errors are of the below sort: > ERROR airflow-core/tests/unit/always/test_secrets.py - sqlalchemy.orm.exc.MappedAnnotationError: Type annotation for "TaskInstance.dag_model" can't be correctly interpreted for Annotated Declarative Table form. ORM annotations should normally make use of the ``Mapped[]`` generic type, or other ORM-compatible generic type, as a container for the actual type, which indicates the intent that the attribute is mapped. Class variables that are not intended to be mapped by the ORM should use ClassVar[]. To allow Annotated Declarative to disregard legacy annotations which don't use Mapped[] to pass, set "__allow_unmapped__ = True" on the class or a superclass this class. (Background on this error at: https://sqlalche.me/e/20/zlpr) Here's an overview of this problem and potential solutions by Claude: [sqlalchemy-upgrade-guide.md](https://github.com/user-attachments/files/20905444/sqlalchemy-upgrade-guide.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
anton-bedrock commented on issue #28723: URL: https://github.com/apache/airflow/issues/28723#issuecomment-2959746081 Seems that https://github.com/apache/airflow/issues/39593 is solved and Airflow 3.x released. When can we expect that SQL Alchemy will migrate to 2.x version? What is blocking such a migration? Thx -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
ultra2207 commented on issue #28723: URL: https://github.com/apache/airflow/issues/28723#issuecomment-2958950658 any updates on this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
laudiacay commented on issue #28723: URL: https://github.com/apache/airflow/issues/28723#issuecomment-2859063745 downgrading unfortunately causes unresolvable dependency conflicts with other dependencies I'm using :| it's ok, I just went back to pydantic + sqlalchemy -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
Dev-iL commented on issue #28723: URL: https://github.com/apache/airflow/issues/28723#issuecomment-2858827982 @laudiacay you can still use SQLModel - just not the latest version... something like 0.11 should work IIRC. We use a venv operator to make sure all the dependencies have the right versions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
eladkal commented on issue #28723: URL: https://github.com/apache/airflow/issues/28723#issuecomment-2817109305 I think now we can revisit this issue as fab is not part of airflow-core any more -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
hamidelmaazouz commented on issue #28723: URL: https://github.com/apache/airflow/issues/28723#issuecomment-2344312151 Any update on this ? ``` Because no versions of apache-airflow match >2.10.0,<2.10.1 || >2.10.1,<3.0.0 and apache-airflow[celery] (2.10.0) depends on sqlalchemy (>=1.4.36,<2.0), apache-airflow[celery] (>=2.10.0,<2.10.1 || >2.10.1,<3.0.0) requires sqlalchemy (>=1.4.36,<2.0). And because apache-airflow[celery] (2.10.1) depends on sqlalchemy (>=1.4.36,<2.0) ... version solving failed ``` Any workarounds I can try to mitigate above error ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
veinkr commented on issue #28723: URL: https://github.com/apache/airflow/issues/28723#issuecomment-2241469969 Hi, @Taragolis , just curious if we can disable the usage of [FAB](https://airflow.apache.org/docs/apache-airflow-providers-fab/stable/index.html), and then update the sqlalchemy-2.0? Cause kindly I run airflow in local area network I can accept that don't use [FAB](https://airflow.apache.org/docs/apache-airflow-providers-fab/stable/index.html). Or is there any keyword that you can suggest for me to make this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
Taragolis commented on issue #28723: URL: https://github.com/apache/airflow/issues/28723#issuecomment-2094752721 The strategic, maybe even since Airflow 2.0, is get rid of FAB in Airflow at all. It moved into the separate provider since 2.9, however all Authorisation still happen go through the [FAB](https://airflow.apache.org/docs/apache-airflow-providers-fab/stable/index.html). There is no ETA when it happen, more possible that in [Airflow 3.0](https://lists.apache.org/thread/3chvg9964zvh15mtrbl073f4oj3nlzp2) but maybe sooner if someone could propose how we could get rid off FAB "Here and Now" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
veinkr commented on issue #28723: URL: https://github.com/apache/airflow/issues/28723#issuecomment-2094746444 @Taragolis , sorry to bother, I jsut checked the [Flask-AppBuilder](https://pypi.org/project/Flask-AppBuilder/) repo, it haven't updated since two months ago, and it seems that it blocked by https://github.com/dpgaspar/Flask-AppBuilder/issues/2038#issuecomment-1549577483 so that it can't update to `flask-sqlalchemy >=3`. Seems that we can't upgrade to sqlalchemy to 2.0 in a short time, do we have other plan for it? I can see that in previous PR, the pandas have been limited to `pandas<2.2`, which is a important library for data engineer, if we keep sqlalchemy<2.0, then we will behind too much things. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
veinkr commented on issue #28723: URL: https://github.com/apache/airflow/issues/28723#issuecomment-2094701420 Hi, @dondaum , I saw you said that all `RemovedIn20Warning `should be fixed in PR [https://github.com/apache/airflow/pull/39299](https://github.com/apache/airflow/pull/39299) , does that mean we can upgrade the sqlalchemy to 2.0 version safely? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
dondaum commented on issue #28723: URL: https://github.com/apache/airflow/issues/28723#issuecomment-2067633235 Hi. I would like to support. To clarify is it correct that all warnings can be found in this issue detail section from potiuk? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
Taragolis commented on issue #28723: URL: https://github.com/apache/airflow/issues/28723#issuecomment-2066283481 I've add collect this warning during the CI step, however our warning collection system collect warnings only during run tests, and errors might happen during initial configurations, so this might be other incompatibilities, i would plan to extend our plugin and collect in the other steps but it required some time. I have add all currently found warnings in Issue detail, so fill free to fix it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
moiseenkov commented on issue #28723: URL: https://github.com/apache/airflow/issues/28723#issuecomment-2058451579 Excluding providers (FAB and openlineage) I managed to catch two warnings in the core: ``` airflow/www/views.py:845: [W0513(warning), ] The "columns" argument to Select.with_only_columns(), when referring to a sequence of items, is now passed as a series of positional elements, rather than as a list. (Background on SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9) airflow/models/taskinstance.py:1870: [W0513(warning), ] Using strings to indicate column or relationship paths in loader options is deprecated and will be removed in SQLAlchemy 2.0. Please use the class-bound attribute directly. (Background on SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
VladaZakharova commented on issue #28723: URL: https://github.com/apache/airflow/issues/28723#issuecomment-2051458814 Hi Team! How is the progress going in this one? :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
Dev-iL commented on issue #28723: URL: https://github.com/apache/airflow/issues/28723#issuecomment-1951460606 @potiuk Sure thing, will ping when ready for review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
potiuk commented on issue #28723: URL: https://github.com/apache/airflow/issues/28723#issuecomment-1951454071 Why don you make a PR for that @Dev-iL ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
Dev-iL commented on issue #28723: URL: https://github.com/apache/airflow/issues/28723#issuecomment-1951118883 Not a deprecation but probably related to this effort: `DagBag` cannot be imported with SQLA 2: ```python >>> from airflow.models import DagBag Traceback (most recent call last): File "", line 1, in File "/home/.../.venv/lib64/python3.9/site-packages/airflow/__init__.py", line 68, in settings.initialize() File "/home/.../.venv/lib64/python3.9/site-packages/airflow/settings.py", line 544, in initialize configure_orm() File "/home/.../.venv/lib64/python3.9/site-packages/airflow/settings.py", line 242, in configure_orm engine = create_engine(SQL_ALCHEMY_CONN, connect_args=connect_args, **engine_args, future=True) File "", line 2, in create_engine File "/home/.../.venv/lib64/python3.9/site-packages/sqlalchemy/util/deprecations.py", line 281, in warned return fn(*args, **kwargs) # type: ignore[no-any-return] File "/home/.../.venv/lib64/python3.9/site-packages/sqlalchemy/engine/create.py", line 686, in create_engine raise TypeError( TypeError: Invalid argument(s) 'encoding' sent to create_engine(), using configuration SQLiteDialect_pysqlite/QueuePool/Engine. Please check that the keyword arguments are appropriate for this combination of components. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
infohash commented on issue #28723: URL: https://github.com/apache/airflow/issues/28723#issuecomment-1857972226 This is why I forked SQLAlchemy 2.0 and put it in my project's source directory instead of installing it as a project dependency. Once airflow ecosystem completely moves to SQLA 2.0, I will just stop using my own fork. All I had to do was change the package name of SQLA 2.0. You can also fork it and rebuild SQLA 2.0 wheel with a different package name. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
Taragolis commented on issue #28723: URL: https://github.com/apache/airflow/issues/28723#issuecomment-1855694165 Just for the record, fix all stuff do not automatically granted ability to run Airflow with SQLAlchemy 2.0, due to upper bound limitation of Airflow and Providers dependencies: ```console root@4c41f46e319c:/opt/airflow# pipdeptree --packages sqlalchemy -r SQLAlchemy==1.4.50 ├── alembic==1.13.0 [requires: SQLAlchemy>=1.3.0] │ ├── apache-airflow==2.9.0.dev0 [requires: alembic>=1.6.3,<2.0] │ ├── databricks-sql-connector==2.9.3 [requires: alembic>=1.0.11,<2.0.0] │ └── sqlalchemy-spanner==1.6.2 [requires: alembic] ├── apache-airflow==2.9.0.dev0 [requires: SQLAlchemy>=1.4.28,<2.0] ├── databricks-sql-connector==2.9.3 [requires: SQLAlchemy>=1.3.24,<2.0.0] ├── eralchemy2==1.3.8 [requires: SQLAlchemy>=1.4] ├── Flask-AppBuilder==4.3.10 [requires: SQLAlchemy<1.5] │ └── apache-airflow==2.9.0.dev0 [requires: Flask-AppBuilder==4.3.10] ├── Flask-SQLAlchemy==2.5.1 [requires: SQLAlchemy>=0.8.0] │ └── Flask-AppBuilder==4.3.10 [requires: Flask-SQLAlchemy>=2.4,<3] │ └── apache-airflow==2.9.0.dev0 [requires: Flask-AppBuilder==4.3.10] ├── marshmallow-sqlalchemy==0.26.1 [requires: SQLAlchemy>=1.2.0] │ └── Flask-AppBuilder==4.3.10 [requires: marshmallow-sqlalchemy>=0.22.0,<0.27.0] │ └── apache-airflow==2.9.0.dev0 [requires: Flask-AppBuilder==4.3.10] ├── snowflake-sqlalchemy==1.5.1 [requires: SQLAlchemy>=1.4.0,<2.0.0] ├── sqlalchemy-bigquery==1.9.0 [requires: SQLAlchemy>=1.2.0,<2.0.0dev] ├── sqlalchemy-drill==1.1.4 [requires: SQLAlchemy] ├── SQLAlchemy-JSONField==1.0.2 [requires: SQLAlchemy] │ └── apache-airflow==2.9.0.dev0 [requires: SQLAlchemy-JSONField>=1.0] ├── sqlalchemy-redshift==0.8.14 [requires: SQLAlchemy>=0.9.2,<2.0.0] ├── sqlalchemy-spanner==1.6.2 [requires: SQLAlchemy>=1.1.13] └── SQLAlchemy-Utils==0.41.1 [requires: SQLAlchemy>=1.3] └── Flask-AppBuilder==4.3.10 [requires: SQLAlchemy-Utils>=0.32.21,<1] └── apache-airflow==2.9.0.dev0 [requires: Flask-AppBuilder==4.3.10] ``` Also required that this packages also supports SA 2.0: - [Flask-AppBuilder](https://pypi.org/project/Flask-AppBuilder/) - Doesn't support SA 2.0 yet - [databricks-sql-connector](https://pypi.org/project/databricks-sql-connector/) - Support SA 2.0 from 3.0.0, and drop support 1.4 - [snowflake-sqlalchemy](https://pypi.org/project/snowflake-sqlalchemy/) - Doesn't support SA 2.0 yet - [sqlalchemy-bigquery](https://pypi.org/project/sqlalchemy-bigquery/) - Doesn't support SA 2.0 yet - [sqlalchemy-redshift](https://pypi.org/project/sqlalchemy-redshift/) - Doesn't support SA 2.0 yet -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
VladaZakharova commented on issue #28723: URL: https://github.com/apache/airflow/issues/28723#issuecomment-1855569830 Hi Team! Is there some progress on this one? Are we actually close to close it? :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
moiseenkov commented on issue #28723: URL: https://github.com/apache/airflow/issues/28723#issuecomment-1768132556 > Not yet @moiseenkov , we are close but still have couple of files which we are working on. I will keep you posted when all the codebase has been refactored ! Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
phanikumv commented on issue #28723: URL: https://github.com/apache/airflow/issues/28723#issuecomment-1768102141 Not yet @moiseenkov , we are close but still have couple of files which we are working on. I will keep you posted when all the codebase has been refactored ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
