wolvery opened a new pull request, #70931:
URL: https://github.com/apache/airflow/pull/70931
## Summary
This is a draft/RFC vertical slice that prevents **expected** stale executor
launches from surfacing as opaque `/run` `404` failures. It introduces durable
executor launch records and a typed `409 stale_executor_launch` response, so
that a launch attempt from an executor whose task instance row has already been
superseded (reset, rescheduled, cleared, or re-adopted) can be distinguished
from a genuinely unknown task.
Refs #69760.
## Motivation
When a task instance transitions out of its original launch (scheduler
failover, adoption, stuck-queued reschedule, orphan reset, or a manual clear),
the executor that was originally assigned may still attempt to `/run` the task.
Because the launch identity lived only on the mutable `TaskInstance` row, that
late launch produced an opaque `404 not_found`, which is indistinguishable from
a truly unknown task and consumes a retry / blocks clean requeueing.
## Approach
- **Durable launch records.** New `TaskInstanceLaunch` ORM persists an
immutable executor launch token independently of the mutable/deleted
`TaskInstance` row, with an `active -> consumed | superseded` lifecycle,
guarded state transitions, lookup helpers, indexes, and a state check
constraint.
- **Atomic creation.** A launch record is written during the `SCHEDULED ->
QUEUED` transition in the scheduler.
- **Supersession.** Reschedule of stuck-queued tasks, failed adoption,
orphan reset, and TaskInstance clear/next-try all supersede any active launch
so an old token cannot be reused if the old executor comes back. Successful
adoption preserves the token.
- **Typed conflict.** `/run` validates the launch token:
- Known **terminal** token (consumed/superseded) -> `409
stale_executor_launch`.
- Unknown token -> remains `404 not_found` (legacy tokens predating this
change still return `404`).
- Successful `/run` guarded-consumes the launch; duplicate-running remains
idempotent.
- **Worker behavior.** New `TaskInstanceSupersededError` in the Task SDK; a
superseded worker logs once and exits successfully instead of failing.
- **Executor-agnostic.** Works across Kubernetes, Celery, and Edge
executors. No authorization changes and no fallback by logical task key.
## Changes
- New model `airflow/models/task_instance_launch.py` + migration
`3c5f8e9a1d2b` (down-revision `7a98f1b7dbd3`), registered with DB cleanup.
- Scheduler: launch creation on queueing; supersession on reschedule /
failed adoption / orphan reset.
- `TaskInstance` clear/next-try: supersede active launch.
- Execution API `/run`: durable-token validation + typed `409`; Cadwyn
version adds optional `external_executor_id`.
- Task SDK: propagate token on start, map `409 stale_executor_launch` to
`TaskInstanceSupersededError`; supervisor exits `0` on supersession.
- Tests: model, migration, scheduler, execution API, SDK client, and
supervisor.
## Rollout / compatibility
- Additive migration; no backfill required.
- Unknown legacy tokens (launched before this change) still return `404` —
behavior only changes for tokens that have a durable record.
## Status
Draft / RFC for review of the approach before hardening. Feedback welcome on
the launch-record lifecycle and the `409` contract.
> Note: two pre-existing Task SDK supervisor tests (`test_run_simple_dag`,
`test_supervise_handles_deferred_task`) fail on a clean checkout in this
environment and are unrelated to this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
