shahar1 commented on issue #70222: URL: https://github.com/apache/airflow/issues/70222#issuecomment-5077237265
**Triage proposal** The signature here — `ExpiredSignatureError` on `/execution/...` for tasks that have not been running anywhere near the token lifetime, several failing in the same window — matches a mid-request expiry race in `JWTReissueMiddleware`, not a missing backport. Two corrections to the report's premises first: - **The two-token fix is already in 3.2.2.** [apache/airflow#60108](https://github.com/apache/airflow/pull/60108) was backported as [apache/airflow#66989](https://github.com/apache/airflow/pull/66989) (`f00b3c948e`, an ancestor of the `3.2.2` tag). That fix covers tokens expiring while a task waits in an *executor queue*, which is a different failure mode — so its presence neither contradicts nor explains this report. - **`[execution_api] jwt_expiration_time` defaults to `600`, not `3600`** ([`config.yml` @ 3.2.2](https://github.com/apache/airflow/blob/3.2.2/airflow-core/src/airflow/config_templates/config.yml)). The `3600` quoted in the report is `[api_auth] jwt_cli_expiration_time`. Unless it has been overridden, the Execution API token lifetime in play is 10 minutes — which is consistent with "the affected tasks are not necessarily running for one hour". On the mechanism: `JWTBearer` validates the bearer token and caches the claims on the ASGI request scope ([`security.py` @ 3.2.2](https://github.com/apache/airflow/blob/3.2.2/airflow-core/src/airflow/api_fastapi/execution_api/security.py#L88-L139)); after the handler returns, [`JWTReissueMiddleware`](https://github.com/apache/airflow/blob/3.2.2/airflow-core/src/airflow/api_fastapi/execution_api/app.py#L130-L164) re-validates the *same* bearer string. If the token crosses `exp` during the request, that second validation raises, the exception is swallowed, and no `Refreshed-API-Token` is returned — so the client keeps a dead token, the next call 403s, and after enough failed heartbeats the task is killed. That is intermittent by nature and can take out several tasks at once, which fits the report. **Proposed disposition: DUPLICATE of [apache/airflow#67939](https://github.com/apache/airflow/issues/67939).** Same root cause, same code path. [apache/airflow#67939](https://github.com/apache/airflow/issues/67939) is older (filed 2026-06-03), on the same 3.2.2, and carries the reliable reproduction this report is missing (long-running `BashOperator` under `LocalExecutor`). Worth noting: this race now has **two open PRs making the same change** — [apache/airflow#68499](https://github.com/apache/airflow/pull/68499) against #67939, and [apache/airflow#70227](https://github.com/apache/airflow/pull/70227) against this issue. Both reuse the claims already cached in the request scope instead of re-validating. The reissue middleware is byte-identical between the `3.2.2` tag and current `main`, so this is not fixed on `main` either. Action items: 1. Confirm the merge direction — fold this into #67939 as canonical, or keep both open. 2. Reconcile the two open PRs. They fix the same race the same way; carrying both is reviewer noise whichever issue survives. 3. From the reporter, to confirm the mechanism: which executor is in use, and the decoded `iat`/`exp` of one failing token next to the API-server timestamp of the rejection. Flagging this as a **low-confidence merge direction** — this report's "several tasks fail simultaneously" is a broader signature than #67939's single long-running task, so an argument for keeping them separate is reasonable. Please challenge. My read on severity is medium-to-high (it kills running tasks under default config), but scoring is the team's call. --- Drafted-by: Claude Code (Opus 5); reviewed by @shahar1 before posting -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
