kaxil opened a new pull request, #77166: URL: https://github.com/apache/airflow/pull/77166
An explicit `[celery] result_backend` with a driverless URL such as `db+postgresql://user:pass@host/db` goes to Celery unchanged, and Celery passes it to SQLAlchemy's `create_engine`. SQLAlchemy 2.0 picks psycopg2 for a driverless `postgresql://` URL, so when psycopg2 is not installed the scheduler and every Celery worker crash with `ModuleNotFoundError: No module named 'psycopg2'`, even though psycopg (v3) is installed and the metadata database already uses it. The derived case already handles this. When `result_backend` is not set, the provider builds it from `sql_alchemy_conn` and turns a driverless `postgresql://` into `postgresql+psycopg://`, or `postgresql+psycopg2://` without psycopg and SQLAlchemy 2.0 ([default_celery.py](https://github.com/apache/airflow/blob/e306ae6529f2e1e725b69300c2620580ff2d532f/providers/celery/src/airflow/providers/celery/executors/default_celery.py#L156-L167), from #69526). Core does the same for a driverless `[database] sql_alchemy_conn` ([configuration.py](https://github.com/apache/airflow/blob/e306ae6529f2e1e725b69300c2620580ff2d532f/airflow-core/src/airflow/configuration.py#L376-L393)). The explicit setting never got that treatment, so the same URL works as `sql_alchemy_conn` and crashes as `result_backend`. The Helm chart writes its result backend as `db+<protocol>://` from `data.resultBackendConnection` into `AIRFLOW__CELERY__RESULT_BACKEND` ([template](https://github.com/apache/airflow/blob/e306ae65 29f2e1e725b69300c2620580ff2d532f/chart/templates/secrets/result-backend-connection-secret.yaml#L49)), and its example uses `protocol: postgresql`, so chart deployments without psycopg2 hit this too. The fix moves the existing rewrite into one helper that both paths go through. A URL that names a driver is left alone. The rewrite runs before the engine options are built, so an explicit driverless URL now also gets the `prepare_threshold=None` default for psycopg from #74841. **With both drivers installed, an explicit driverless result backend moves from psycopg2 to psycopg.** That matches what the metadata database and the derived result backend already use; set `db+postgresql+psycopg2://` to keep psycopg2. The #69526 changelog note said an explicit `result_backend` was unaffected, so this adds a note for the next release, and the option's description now says how the driver is chosen. On Airflow 2.11 with SQLAlchemy 1.4 nothing changes: the rewrite produces `+psycopg2`, which SQLAlchemy 1.4 picked anyway. Run with CeleryExecutor on Postgres, Python 3.12, SQLAlchemy 2.0.57, celery 5.6.3, psycopg 3.3.8, psycopg2 uninstalled, and `[celery] result_backend = db+postgresql://...`: | | main | this PR | |---|---|---| | Result backend engine | `ModuleNotFoundError` | driver `psycopg`, `select 1` works | | Celery worker | exits at startup | `celery@... ready.` | | Scheduler | crashes once the executor touches the backend | runs, no tracebacks | | Dag run with 4 tasks | never completes, 0 tasks run | success, 4 tasks succeeded | Scheduler on main (it also crashed from `executor.heartbeat()` in another run): ``` File "airflow/providers/celery/executors/celery_executor.py", line 348, in try_adopt_task_instances File "celery/backends/database/session.py", line 56, in get_engine File "sqlalchemy/engine/create.py", line 617, in create_engine File "sqlalchemy/dialects/postgresql/psycopg2.py", line 697, in import_dbapi ModuleNotFoundError: No module named 'psycopg2' ``` The `explicit-driverless` test cases fail on main with `'psycopg2' != 'psycopg'` and pass here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
