GitHub user seanmuth edited a discussion: Proposal: supervisor-level liveness enforcement for tasks with blocking native calls (was: JVM process isolation)
## Summary The JDBC provider (and any provider that embeds a JVM via JPype/jaydebeapi) runs the JVM *inside* the worker's task process. This makes a whole class of native faults impossible to handle gracefully: a native crash or hang in the driver corrupts or wedges the task process, and there is no in-process remedy. I'd like to propose an opt-in process-isolation execution mode for these providers and gather input on the right shape. ## Why this can't be solved in-process today - A `SIGSEGV` (or other native abort) in the JDBC driver / JVM is not a Java exception. It never surfaces through JPype as something `try/except` in a task or in the hook can catch. It corrupts the process itself. - JPype's JVM is a per-process singleton: started once, shared by every task in that process, and not restartable after it crashes. One native fault leaves a poisoned JVM that every subsequent task in that process inherits. - It is not fork-safe, so naive `multiprocessing` workarounds are unreliable. The provider already carries scar tissue from this in-process model: `get_conn` had to be made threadsafe to avoid "JVM is already started" (#44718), Java exceptions are suppressed with warnings around autocommit (#43786), and there is recurring JPype packaging fragility (#23847, #65532). These are tactical patches around a structural issue: the JVM has no isolation from the task process. ## Observed real-world impact A production user running Sybase via the jConnect JDBC driver hit repeated `SIGSEGV`s in the JVM. Because the segfault killed a native thread but the parent Python task survived and kept heartbeating, the task never transitioned to a terminal state. Airflow's zombie detection is heartbeat-based, so it never reaped these tasks. They sat "Running" for 13+ hours, blocking their workload's cadence, and only recovered when manually marked failed. A native fault that should have been a single retryable task failure instead became a silent, process-poisoning hang. ## Proposal Add an opt-in isolation mode for JVM-backed execution, so a native fault is contained to a disposable process and converted into a normal, retryable task failure. Two candidate shapes: 1. **Per-call subprocess.** Simple, and already partially achievable today via `ExternalPythonOperator`, but each call pays full JVM cold-start (1-2s+), which is prohibitive for high-frequency, short-running queries. 2. **Persistent isolated JVM worker (daemon).** A long-lived sidecar process hosts the JVM and services queries over IPC. This preserves warm-JVM/connection reuse while giving crash containment: if the daemon dies, the parent sees it, fails the task cleanly, and respawns a fresh JVM for the next run. More work, but the version that actually fits latency-sensitive workloads. ## Open questions - Is this better as an execution mode on the existing hook/operator (e.g. `execution_isolation="daemon"`) or a distinct operator? An execution mode keeps users on the operator they already know and could generalize to any JPype/native-backed hook. - The Task Execution Interface (AIP-72 / Task SDK, accepted and shipped in Airflow 3) already moves task execution behind a defined, language-agnostic interface with a supervisor over a task subprocess. That supervisor catches a full task-process crash, but not this case: a native-thread `SIGSEGV` leaves the task subprocess alive and heartbeating, and the JVM still lives inside that subprocess. Does a JVM-isolation boundary belong as an extension of that interface, or is it better kept local to the JVM-backed providers? - Appetite for a reference implementation of the daemon model, or is per-call subprocess isolation an acceptable first step? Would appreciate maintainer input on the shape before anyone invests in an implementation. 🤖 Drafted with Claude Code (Claude Opus 4.8, 1M context) GitHub link: https://github.com/apache/airflow/discussions/70055 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
