DominikBalga opened a new issue, #65708: URL: https://github.com/apache/airflow/issues/65708
### Under which category would you file this issue? Task SDK ### Apache Airflow version 3.2.0. & 3.2.1 ### What happened and how to reproduce it? **Issue Description** The observed behavior starts after upgrading to **Airflow 3.2.0** (and persists in **3.2.1**) and disappears when downgrading to **3.1.8**. The failure manifests on Celery workers during the post-execution "finish" step (Task SDK / execution API), where the worker attempts to PATCH the TaskInstance state and receives: - **HTTP 409** - `reason=invalid_state` - `message=TI was not in the running state so it cannot be updated` - `previous_state=success` This results in a worker-side exception (`ServerResponseError`) and causes the Celery workload to be treated as failed even when the underlying task logic has completed. - Airflow version(s) affected: **3.2.0**, **3.2.1** - Airflow version known-good: **3.1.8** - Executor: **CeleryExecutor** - Deployment: Kubernetes - Python: 3.12 (from stack traces) - Database: Postgres Typical worker log sequence: - Worker starts and loads secrets backend - Worker begins Task SDK API calls (often with retries) - API returns 409 invalid_state with `previous_state=success` - Supervisor raises `ServerResponseError` and the Celery task fails Example: - `Secrets backends loaded for worker ...` - `Starting call to 'airflow.sdk.api.client.Client.request', this is the 1st time calling it.` - `Starting call to ... this is the 2nd time calling it.` - `API server error ... status_code=409 ... previous_state='success'` - `airflow.sdk.api.client.ServerResponseError: Server returned error` **Steps to reproduce** The issue appears intermittently; higher concurrency tends to increase the frequency. ### Prerequisites - Airflow **3.2.0+** - **CeleryExecutor** - Execution API enabled (Task SDK in use) 1. Deploy Airflow 3.2.0 or 3.2.1. 2. Trigger the DAG multiple times (or run multiple manual runs concurrently). 3. Observe worker logs for `ServerResponseError` with HTTP 409 `invalid_state` and `previous_state=success`. ### What you think should happen instead? Airflow workers should be able to finalise a task idempotently; if the API server has already moved the TaskInstance to a terminal state (e.g. success), the worker’s PATCH /execution/task-instances/<id>/state should not return a fatal 409 or fail the task. ### Operating System _No response_ ### Deployment Official Apache Airflow Helm Chart ### Apache Airflow Provider(s) _No response_ ### Versions of Apache Airflow Providers _No response_ ### Official Helm Chart version Not Applicable ### Kubernetes Version _No response_ ### Helm Chart configuration _No response_ ### Docker Image customizations _No response_ ### Anything else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
