Andrushika commented on issue #64212:
URL: https://github.com/apache/airflow/issues/64212#issuecomment-4937168795
I found it hard to reproduce this locally, but I have read the traceback and
the related PRs. I wanted to share my current understanding of the failure
chain, and please correct me if this reading is off!
A role-based view of the chain looks roughly like this:
```mermaid
flowchart LR
subgraph R["Task runtime"]
R1["1. Task calls xcom_pull()<br/>and sends a request via
SUPERVISOR_COMMS"]
R2["4. Task waits for response frame<br/>in _read_frame() /
socket.recv(4)"]
R3["5. execution_timeout fires<br/>SIGALRM interrupts the blocked
recv"]
R4["6. handle_timeout() runs<br/>and tries to log first"]
R5["8. AirflowTaskTimeout is masked<br/>by the logging failure"]
R6["9. Generic error path / cleanup runs"]
end
subgraph S["Supervisor"]
S1["2. Supervisor receives the task request"]
S2["3. Supervisor handles the request<br/>and calls the Execution
API / webserver"]
S3["3b. Transport/non-HTTP failure<br/>may prevent a normal
response"]
end
subgraph A["Execution API / webserver"]
A1["3a. API is slow/unavailable<br/>ReadTimeout / retries / possible
OOM context"]
end
subgraph L["Logging channel"]
L1["7. Timeout handler writes a log record<br/>through the task
logging channel"]
L2["7b. Logging channel is unavailable<br/>BrokenPipeError /
OSError"]
end
subgraph C["Cleanup"]
C1["10. Cleanup may hit unset state<br/>and raise UnboundLocalError"]
end
R1 --> S1
S1 --> S2
S2 --> A1
A1 -. timeout / transport failure .-> S3
S3 -. no or late response .-> R2
S3 -. supervisor/request failure may leave<br/>logging channel
unavailable .-> L2
R2 --> R3
R3 --> R4
R4 --> L1
L1 --> L2
L2 --> R5
R5 --> R6
R6 --> C1
```
The related PRs seem to cut different failure or masking points in this
chain:
* **#66572 → steps 3 / 3b**
If supervisor request handling hits a transport or non-HTTP exception, it
now returns an `ErrorResponse` and keeps the request loop alive. This cuts one
path where the task could otherwise keep waiting without a usable response.
* **#67955 → step 10**
If `state` was never assigned because an earlier error path failed,
cleanup should not raise `UnboundLocalError` while emitting `ti.finish` stats.
* **#69713 → steps 6 / 7 / 8 (opened as a follow-up, not merged yet)**
`handle_timeout()` should treat timeout logging as best-effort. If the log
write fails with `OSError`, it should still raise `AirflowTaskTimeout` instead
of letting the logging failure mask the timeout.
I may still be missing details around the exact supervisor/log socket
lifecycle, since I have not reproduced this locally. But from the traceback, it
looks like several secondary failures can stack on top of the original timeout,
making the final visible exception misleading.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
