GitHub user seanmuth added a comment to the discussion: Proposal: opt-in process isolation for JVM-backed providers (JDBC / jaydebeapi)
@uranusjr @ashb following up on both of your points, and reframing this proposal based on where the investigation actually led. The original framing (JDBC-specific process isolation) missed the real gap. Ash's question about the process tree was the right challenge: each task already runs in its own forked process, via `StandardTaskRunner` in AF2 and the Task SDK's `WatchedSubprocess`/`supervise_task` in AF3. That per-task isolation is already there and already works for the clean-crash case. What isn't there is liveness enforcement *within* that isolation. The supervisor currently only checks whether the child process has exited. It has no way to tell "alive and making progress" from "alive and permanently wedged." A native fault that corrupts a thread but doesn't kill the process (confirmed for the JDBC/JPype case: JPype never releases the GIL around a JNI call, so if that specific call hangs, nothing else in the process, including any same-process watchdog thread, can run either) leaves the task heartbeating and Running indefinitely. That's a supervisor-level gap, not a process-isolation gap, and it's why isolation alone, however it's implemented, doesn't close it. So the concrete proposal, in the spirit of "should be default" rather than JDBC-specific: **Add a generic in-flight deadline primitive that any task can arm, enforced by the supervisor.** Shape: - A shared value, created by the supervisor before it forks the child (so it's naturally inherited in AF2, or riding on AF3's existing pre-fork socketpairs, no new IPC setup needed), representing "the deadline by which the current unit of work should have returned control." - A hook/operator author arms it before a risky blocking call (e.g. a JDBC query) and disarms it immediately after the call returns, success or a normal exception either way. - The supervisor's existing poll loop, the same one that already checks whether the child has exited, gains one more check: if the deadline is armed and expired while the child is still alive, `SIGKILL` it. - Killing the child this way lands in the exact same failure/retry path Airflow already uses for any other child death (a normal exit, an exception, a kernel OOM-kill), so there's no new correctness surface on the retry side, only the *decision to kill* is new. Why this has to live at the supervisor layer and not in a provider package or a same-process thread: only the supervisor is structurally guaranteed to be a separate process with its own GIL that never touches whatever native resource the task is using. This generalizes past JDBC too, any native-extension-backed hook (pymongo, paramiko/cryptography-backed SSH, other JNI bridges) has the same class of risk, a native call that hangs without releasing the GIL is invisible to anything running inside that same process. The honest cost: this needs coordinated changes across the task-runner layer (owns the fork, needs to set up the shared primitive first) and the supervisor's own poll loop (needs the extra check), so it's a core change, not something a provider can ship alone. Scope-wise this generalizes cleanly: task_runner is orthogonal to executor choice, so it applies to Celery, Kubernetes, Local, and Sequential executors alike, in both AF2 and AF3, since they all funnel through the same shared dispatch. It does not apply to `DebugExecutor`, which intentionally bypasses this for in-process debugging, or to executors with a fundamentally different worker architecture that doesn't route through the standard task-runner/supervisor layer at all. This doesn't replace the daemon-isolation idea from the original post, that's still the stronger fix for the specific JVM-in-process case. But this is a smaller, more general first step that gets most of the value (a wedged task becomes a detected, retryable failure instead of a silent multi-hour hang) without touching provider or isolation architecture at all, and it's the piece that actually makes "isolation as default" mean something operationally, since isolation without liveness enforcement doesn't catch this failure mode either way. 🤖 Drafted with Claude Code (Claude Sonnet 5) GitHub link: https://github.com/apache/airflow/discussions/70055#discussioncomment-17706013 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
