noamst-monday opened a new issue, #72469: URL: https://github.com/apache/airflow/issues/72469
### Apache Airflow version 3.2.2 ### What happened The two-token mechanism introduced in #60108 (fixing #53713) hard-couples the workload-scoped JWT's lifetime to `[scheduler] task_queued_timeout` (see `airflow-core/docs/security/jwt_token_authentication.rst`, "Token scopes (Execution API)": *"Lifetime equals `[scheduler] task_queued_timeout`... so tuning `task_queued_timeout` also widens the window a task can wait in a backed-up queue before its workload token expires."*). This means `task_queued_timeout` now serves two unrelated purposes at once: 1. How long a workload JWT stays valid while a task waits in an executor queue. 2. How long the scheduler waits before declaring a `queued` task orphaned/stuck and failing+retrying it (the queued-task-timeout reaper in `airflow-core/src/airflow/jobs/scheduler_job_runner.py`, around the `check_workers`/queued-timeout logic). An operator who legitimately needs a long workload-token lifetime — e.g. to survive real queue backlog during a cron-wave burst, or to tolerate their own scheduler restarting mid-dispatch — is forced to also raise the orphaned-queued-task detection timeout to the same large value. We had to set ours to 36000s (10h) for exactly this reason. The cost: a task that gets stuck in `queued` state for an *unrelated* reason (e.g. the scheduler gets OOMKilled in the narrow window between flipping a TI to `queued` in the DB and actually dispatching a pod/worker for it — no workload token has even been issued yet in that case) has no separate, shorter safety net. It can sit invisible and un-retried for up to 10 hours, blocking downstream tasks and the whole DAG run, because the only detection mechanism (`task_queued_timeout`) has been deliberately widened for an unrelated concern. We checked #53713's thread: a maintainer (@amoghrajesh) already flagged this exact coupling and suggested "increase `jwt_expiration_time`" as a workaround, but that config (`[execution_api] jwt_expiration_time`) only governs the *post-`/run`* execution token — it does not affect the workload token's lifetime, so it doesn't help here. Searched for an existing issue proposing to decouple these two configs and didn't find one. ### What you think should happen instead Add a separate, independently-tunable config for workload-token TTL (e.g. `[execution_api] workload_token_expiration_time`), decoupled from `[scheduler] task_queued_timeout`, so operators can tune: - how long a task can legitimately wait in a backed-up queue before its dispatch token expires, and - how quickly the scheduler detects and recovers a genuinely orphaned `queued` task ...independently of each other. ### How to reproduce 1. Set `[scheduler] task_queued_timeout` high (e.g. 36000s) to tolerate legitimate queue backlog / avoid workload-token expiry under load. 2. Have the scheduler crash/restart in the window after a TI is marked `queued` in the DB but before a pod/worker is actually dispatched for it (e.g. via OOMKill during a burst of concurrent dispatches). 3. Observe the TI remains `queued` with no pod, no heartbeat, and is not reaped until `task_queued_timeout` elapses — up to 10h in our config. ### Operating System N/A (config/design issue, not OS-specific) ### Versions of Apache Airflow Providers N/A ### Deployment Other Docker-based deployment ### Deployment details KubernetesExecutor + CeleryExecutor (multi-executor), Helm chart 1.21.0, self-hosted on EKS. ### Anything else Related: #53713, #60108 ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
