uplsh580 opened a new issue, #71395: URL: https://github.com/apache/airflow/issues/71395
### Under which category would you file this issue? Providers ### Apache Airflow version 3.2.2 ### What happened and how to reproduce it? With MySQL as the metadata DB and the FAB auth manager, after the api-server sits idle longer than MySQL's `wait_timeout`, the **first** authenticated request (any UI or REST endpoint) fails with HTTP 500. The exception is an `OperationalError` (MySQL error 4031, "The client was disconnected by the server because of inactivity") raised from `RevokedToken.is_revoked` inside `BaseAuthManager.get_user_from_token`: https://github.com/apache/airflow/blob/3.2.2/airflow-core/src/airflow/api_fastapi/auth/managers/base_auth_manager.py#L152 The immediately following request succeeds, because the failed query poisons and effectively resets the pooled connection. In production this shows up as intermittent 500s on the first request after overnight idle periods. <details> <summary>Traceback (trimmed)</summary> ``` File ".../airflow/api_fastapi/auth/managers/base_auth_manager.py", line 152, in get_user_from_token if (jti := payload.get("jti")) and RevokedToken.is_revoked(jti): File ".../airflow/utils/session.py", line 100, in wrapper return func(*args, session=session, **kwargs) File ".../airflow/models/revoked_token.py", line 61, in is_revoked return bool(session.scalar(select(exists().where(cls.jti == jti)))) ... sqlalchemy.exc.OperationalError: (MySQLdb.OperationalError) (4031, 'The client was disconnected by the server because of inactivity. See wait_timeout and interactive_timeout for configuring this behavior.') ``` </details> ### How to reproduce 1. Run Airflow 3.2.x with MySQL as the metadata DB and the FAB auth manager. 2. Lower the idle timeout to make the repro fast: `SET GLOBAL wait_timeout = 60;` 3. Make any authenticated API request (this warms up the pooled connection). 4. Wait longer than `wait_timeout` (e.g. 90 s) without touching the api-server. 5. Repeat the same request → HTTP 500 with `OperationalError` 4031 raised from `RevokedToken.is_revoked`. Repeat once more → succeeds. ### What you think should happen instead? #### Root cause 1. The FAB auth manager shares core's scoped `settings.Session`. `FabAuthManager.deserialize_user` leaves a transaction open after the request is served, so the connection stays checked out across requests. Once it idles past MySQL's `wait_timeout`, the server drops it (error 4031). Because the connection is never returned to the pool, there is no checkout event — `pool_pre_ping` / `pool_recycle` cannot help. 2. This exact failure mode was reported in #62903 and fixed by #62919, which added a discard-and-retry recovery to `FabAuthManager.deserialize_user`. 3. However, 3.2.0 introduced the JWT revocation check (#61339 / AIP-84): `RevokedToken.is_revoked` in `get_user_from_token` now runs **before** `deserialize_user`, so it is the first thing to touch the poisoned scoped session — and it has no recovery logic. The 500 that #62919 fixed is back, just raised one step earlier in the auth path. ### What you think should happen instead? The first request after an idle disconnect should recover transparently, as it does in `deserialize_user` since #62919. The revoked-token check should apply the same recovery: on `SQLAlchemyError`, discard the scoped session (`settings.Session.remove()`) and retry once on a fresh connection. Something like: ```python if jti := payload.get("jti"): try: revoked = RevokedToken.is_revoked(jti) except SQLAlchemyError: log.warning( "Revoked-token check failed on a stale DB session; " "discarding the session and retrying once.", exc_info=True, ) with suppress(Exception): settings.Session.remove() revoked = RevokedToken.is_revoked(jti) if revoked: raise InvalidTokenError("Token has been revoked") ``` Alternatively, the recovery could be hoisted so it covers the whole `get_user_from_token` path instead of being duplicated per call site. ### Operating System _No response_ ### Deployment None ### Apache Airflow Provider(s) _No response_ ### Versions of Apache Airflow Providers apache-airflow-providers-fab==3.6.4 (already includes the #62919 fix — it is bypassed, not missing) ### Official Helm Chart version 1.22.0 (latest released) ### Kubernetes Version 1.25 ### Helm Chart configuration _No response_ ### Docker Image customizations _No response_ ### Anything else? - Occurs deterministically whenever the first request after an idle period touches a connection older than `wait_timeout`; looks intermittent in production. - Related prior work on the same class of failure: - #62903 / #62919 — same disconnect, recovery added to `deserialize_user` (this check runs earlier and bypasses it) - #62335 / #62336 — `Session.remove()` hardening in FAB's cleanup middleware - #66493 / #66494 — same call site, different concern (per-request DB roundtrip / pool exhaustion) - We have been running the retry patch above in production for several weeks (applied at image build time) and it eliminated these 500s entirely. ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
