GitHub user seanmuth edited a comment on the discussion: Proposal: supervisor-level liveness enforcement for tasks with blocking native calls (was: JVM process isolation)
Second, independent case for the same blind spot: OpenSSL fork-orphaned lock ([#71707](https://github.com/apache/airflow/issues/71707), fix merged as [#72164](https://github.com/apache/airflow/pull/72164), shipping 3.3.2/3.4.0). Task SDK's supervisor forks the task process via a bare `os.fork()`; if any supervisor thread (OTel/Sentry export, google-auth refresh, secrets-backend polling — confirmed across three separate customer incidents) is mid-critical-section on OpenSSL 3.x's provider-store lock at that instant, the lock is orphaned into the child permanently — no thread survives fork to release it. Same blind spot you both already established for JPype: `execution_timeout`'s SIGALRM never fires (the thread's stuck in native code, never returns to the interpreter loop), and heartbeating stays healthy regardless. Real-world result: task pods alive, heartbeating, and frozen for 3+ days before manual intervention. +1 on making the supervisor enforce `execution_timeout` as a backstop (#53337) — it's the right default-on mechanism and this case is more corroborating evidence for it. Where I'd push for keeping the no-progress/low-CPU angle alive as a second, complementary layer rather than closing it out once #53337 lands: `execution_timeout` only bounds a wedge to whatever the DAG author declared, and two things are true in practice for this specific failure mode. Plenty of deployments don't set it at all — Moloco's hadn't. And where it is set, it's commonly hours, not minutes, sized for the task's legitimate runtime rather than for detecting a hang. The OpenSSL deadlock happens at the instant of fork, before the task has done any real work — so a task wedged this way is knowably stuck almost immediately, and a multi-hour `execution_timeout` still leaves it sitting, heartbeating, doing nothing, for most of that window. The two backstops catch different things: `execution_timeout` catches "ran longer than the DAG author's declared bound," which is correct and sufficient for most hangs; a no-progress/low-CPU check catches "this specific process has been doing zero work since shortly after it started," independent of whatever timeout is or isn't configured. A genuinely wedged process here shows a CPU flatline from very early on, not just "slow" — confirmed directly against a live wedge in this investigation — which is a different and earlier-available signal than elapsed wall-clock time alone. Not proposing this as a replacement for #53337 or as fully specified yet — just think it's worth continued exploration alongside it, since the two are complementary defense-in-depth rather than competing designs. --- Drafted-by: Claude Sonnet 5; reviewed by @seanmuth before posting GitHub link: https://github.com/apache/airflow/discussions/70055#discussioncomment-18475241 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
