slice-soupam commented on issue #71707: URL: https://github.com/apache/airflow/issues/71707#issuecomment-5590309664
<p style="white-space: pre-wrap; margin-top: 0.1em; margin-bottom: 0.2em; unicode-bidi: plaintext; color: rgb(204, 204, 204); font-family: -apple-system, "system-ui", "Segoe UI", Roboto, sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(54, 39, 18); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">The machinery to fix this already exists on <code style="font-family: monospace; color: rgb(215, 186, 125); background-color: rgba(255, 255, 255, 0.1); padding: 2px 4px; border-radius: 3px; word-break: break-word; font-size: 0.9em;">main</code> — it's just gated off for Linux:</p><div class="codeBlockWrapper_-a7MRw" style="position: relative; margin: 8px 0px; color : rgb(204, 204, 204); font-family: -apple-system, "system-ui", "Segoe UI", Roboto, sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(54, 39, 18); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><button class="copyButton_CEmTFw copyButton_-a7MRw" title="Copy code" aria-label="Copy code to clipboard" style="color: rgb(204, 204, 204); font-family: -apple-system, "system-ui", "Segoe UI", Roboto, sans-serif; font-size: 13px; background: none 0% 0% / auto repeat scroll padding-box border-box rgb(34, 26, 15); border-color: rgba(127, 127, 127, 0.4); border-style: solid; border-width: 1px; border-image: none 100% / 1 / 0 stretch; cursor: pointer; opacity: 0; display: flex; border-radius: 4px; justify-content: center; align-items: center; padding: 4px; transition: opacity 0.15s, background 0.15s; position: absolute; top: 4px; right: 4px;"><svg xmlns="http://www.w3.org/2000/svg"; viewBox="0 0 20 20" fill="currentColor" aria-hidden="true" data-slot="icon" class="copyIcon_CEmTFw"><path fill-rule="evenodd" d="M15.988 3.012A2.25 2.25 0 0 1 18 5.25v6.5A2.25 2.25 0 0 1 15.75 14H13.5v-3.379a3 3 0 0 0-.879-2.121l-3.12-3.121a3 3 0 0 0-1.402-.791 2.252 2.252 0 0 1 1.913-1.576A2.25 2.25 0 0 1 12.25 1h1.5a2.25 2.25 0 0 1 2.238 2.012ZM11.5 3.25a.75.75 0 0 1 .75-.75h1.5a.75.75 0 0 1 .75.75v.25h-3v-.25Z" clip-rule="evenodd"></path><path d="M3.5 6A1.5 1.5 0 0 0 2 7.5v9A1.5 1.5 0 0 0 3.5 18h7a1.5 1.5 0 0 0 1.5-1.5v-5.879a1.5 1.5 0 0 0-.44-1.06L8.44 6.439A1.5 1.5 0 0 0 7.378 6H3.5Z"></path></svg></button><pre style="overflow-x: auto; white-space: pre; box-sizing: border-box; border-radius: 4px; max-width: 100%; margin: 0px; pad ding: 8px;"><code class="language-python" style="font-family: monospace; color: rgb(215, 186, 125); background-color: rgba(255, 255, 255, 0.1); padding: 0px; border-radius: 3px; word-break: break-word; font-size: 0.9em;">_FORK_EXEC_PLATFORMS = {"darwin"} # task-sdk/.../execution_time/supervisor.py </code></pre></div><p style="white-space: pre-wrap; margin-top: 0.1em; margin-bottom: 0.2em; unicode-bidi: plaintext; color: rgb(204, 204, 204); font-family: -apple-system, "system-ui", "Segoe UI", Roboto, sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(54, 39, 18); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><code style="font-family: monospace; color: rgb(215, 186, 125); background-color: rgba(255, 255, 255, 0.1); padding: 2px 4px; border-radius: 3px; word-break: break-word; font-size: 0.9em;">start()</code> can <code style="font-family: monospace; color: rgb(215, 186, 125); background-color: rgba(255, 255, 255, 0.1); padding: 2px 4px; border-radius: 3 px; word-break: break-word; font-size: 0.9em;">os.execv</code> a fresh interpreter right after <code style="font-family: monospace; color: rgb(215, 186, 125); background-color: rgba(255, 255, 255, 0.1); padding: 2px 4px; border-radius: 3px; word-break: break-word; font-size: 0.9em;">os.fork()</code>. It was added for macOS's ObjC fork-unsafety (#65691), but it's the right remedy here too: <code style="font-family: monospace; color: rgb(215, 186, 125); background-color: rgba(255, 255, 255, 0.1); padding: 2px 4px; border-radius: 3px; word-break: break-word; font-size: 0.9em;">execv</code> replaces the child's address space, so an OpenSSL mutex left locked by a thread that doesn't exist in the child can't survive into it. And it's not new code — it already carries the task runner, DAG processor and triggerer on macOS, socketpair/FD handoff and process groups included.</p><p style="white-space: pre-wrap; margin-top: 0.1em; margin-bottom: 0.2em; unicode-bidi: plaintext; color: rgb(204, 204, 204); font-family: -apple-system, "system-ui", "Segoe UI", Roboto, sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(54, 39, 18); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">So: reuse it on Linux rather than add a second spawn mechanism. The catch is that <code style="font-family: monospace; color: rgb(215, 186, 125); background-color: rgba(255, 255, 255, 0.1); padding: 2px 4px; border-radius: 3px; word-break: break-word; font-size: 0.9em;">exec</code> forfeits copy-on-write sharing of the parent's imported Airflow, so each child pays a fresh interpreter start plus full re-import — in both latency (~1-3s) and RSS, since nothing is shared with the pa rent any more. That cost is per-fork, so it scales with fork rate, and the fork rate differs by three orders of magnitude across components:</p><p style="white-space: pre-wrap; margin-top: 0.1em; margin-bottom: 0.2em; unicode-bidi: plaintext; color: rgb(204, 204, 204); font-family: -apple-system, "system-ui", "Segoe UI", Roboto, sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(54, 39, 18); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><strong>High concurrent task load.</strong> Worst case. Every task instance pays the re-import, and because COW sharing is gone, each concurrent child now holds its own full copy of Airflow — so worker RSS grows with concurre ncy instead of staying near-flat. Short tasks are hit hardest, since the overhead is a fixed cost against a small payload. Note the perverse incentive: deadlock probability rises with parent thread activity, so the deployments that most need this fix are exactly the ones that pay most for it.</p><p style="white-space: pre-wrap; margin-top: 0.1em; margin-bottom: 0.2em; unicode-bidi: plaintext; color: rgb(204, 204, 204); font-family: -apple-system, "system-ui", "Segoe UI", Roboto, sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(54, 39, 18); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><strong>High DAG file count.</strong> Also significant, and I want to corr ect something I'd assumed: the DAG processor forks once <em>per file per parse cycle</em> (<code style="font-family: monospace; color: rgb(215, 186, 125); background-color: rgba(255, 255, 255, 0.1); padding: 2px 4px; border-radius: 3px; word-break: break-word; font-size: 0.9em;">_create_process</code>, bounded by <code style="font-family: monospace; color: rgb(215, 186, 125); background-color: rgba(255, 255, 255, 0.1); padding: 2px 4px; border-radius: 3px; word-break: break-word; font-size: 0.9em;">parsing_processes</code>). With thousands of files, adding an interpreter start to every parse stretches the whole loop, and longer loops mean staler DAGs and later scheduling. Re-import is only cheap relative to a <em>slow</em> parse; across many fast ones it dominates.</p><p style="white-space: pre-wrap; margin-top: 0.1em; margin-bottom: 0.2em; unicode-bidi: plaintext; color: rgb(204, 204, 204); font-family: -apple-system, "system-ui", "Segoe UI", Roboto, sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(54, 39, 18); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><strong>Triggerer.</strong> Forks once at startup. Overhead is unmeasurable.</p><p style="white-space: pre-wrap; margin-top: 0.1em; margin-bottom: 0.2em; unicode-bidi: plaintext; color: rgb(204, 204, 204); font-family: -apple-system, "system-ui", "Segoe UI", Roboto, sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: r gb(54, 39, 18); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Which suggests gating by fork rate rather than by platform alone:</p> Component | Fork rate | Proposal -- | -- | -- Triggerer | once | enable on Linux unconditionally DAG processor | per file, per cycle | config-gated, default off Task execution | per task instance | config-gated, default off <div class="codeBlockWrapper_-a7MRw" style="position: relative; margin: 8px 0px; color: rgb(204, 204, 204); font-family: -apple-system, "system-ui", "Segoe UI", Roboto, sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(54, 39, 18); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><button class="copyButton_CEmTFw copyButton_-a7MRw" title="Copy code" aria-label="Copy code to clipboard" style="color: rgb(204, 204, 204); font-family: -apple-system, "system-ui", "Segoe UI", Roboto, sans-serif; font-size: 13px; background: none 0% 0% / auto repeat scroll padding-box border-box rgb(34, 26, 15); border-color: rgba(127, 127, 127 , 0.4); border-style: solid; border-width: 1px; border-image: none 100% / 1 / 0 stretch; cursor: pointer; opacity: 0; display: flex; border-radius: 4px; justify-content: center; align-items: center; padding: 4px; transition: opacity 0.15s, background 0.15s; position: absolute; top: 4px; right: 4px;"><svg xmlns="http://www.w3.org/2000/svg"; viewBox="0 0 20 20" fill="currentColor" aria-hidden="true" data-slot="icon" class="copyIcon_CEmTFw"><path fill-rule="evenodd" d="M15.988 3.012A2.25 2.25 0 0 1 18 5.25v6.5A2.25 2.25 0 0 1 15.75 14H13.5v-3.379a3 3 0 0 0-.879-2.121l-3.12-3.121a3 3 0 0 0-1.402-.791 2.252 2.252 0 0 1 1.913-1.576A2.25 2.25 0 0 1 12.25 1h1.5a2.25 2.25 0 0 1 2.238 2.012ZM11.5 3.25a.75.75 0 0 1 .75-.75h1.5a.75.75 0 0 1 .75.75v.25h-3v-.25Z" clip-rule="evenodd"></path><path d="M3.5 6A1.5 1.5 0 0 0 2 7.5v9A1.5 1.5 0 0 0 3.5 18h7a1.5 1.5 0 0 0 1.5-1.5v-5.879a1.5 1.5 0 0 0-.44-1.06L8.44 6.439A1.5 1.5 0 0 0 7.378 6H3.5Z"></path></svg></button><pre style="overflow-x: auto; white-s pace: pre; box-sizing: border-box; border-radius: 4px; max-width: 100%; margin: 0px; padding: 8px;"><code class="language-python" style="font-family: monospace; color: rgb(215, 186, 125); background-color: rgba(255, 255, 255, 0.1); padding: 0px; border-radius: 3px; word-break: break-word; font-size: 0.9em;">_FORK_EXEC_PLATFORMS = {"darwin", "linux"} # darwin: unconditional (correctness). linux: unconditional for the triggerer, # conf.getboolean("workers", "fork_exec_task_subprocess", fallback=False) and the # dag_processor equivalent elsewhere. See #71707. </code></pre></div><p style="white-space: pre-wrap; margin-top: 0.1em; margin-bottom: 0.2em; unicode-bidi: plaintext; color: rgb(204, 204, 204); font-family: -apple-system, "system-ui", "Segoe UI", Roboto, sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(54, 39, 18); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><code style="font-family: monospace; color: rgb(215, 186, 125); background-color: rgba(255, 255, 255, 0.1); padding: 2px 4px; border-radius: 3px; word-break: break-word; font-size: 0.9em;">airflow.sdk.configuration.conf</code> is already the SDK-side accessor and <code style="font-family: monospace; color: rgb(215, 186, 125); background-color: rgba (255, 255, 255, 0.1); padding: 2px 4px; border-radius: 3px; word-break: break-word; font-size: 0.9em;">[workers]</code> an existing SDK-visible section, so no new config plumbing. Follows the <code style="font-family: monospace; color: rgb(215, 186, 125); background-color: rgba(255, 255, 255, 0.1); padding: 2px 4px; border-radius: 3px; word-break: break-word; font-size: 0.9em;">mp_forkserver_preload</code> precedent of exposing a start-method tradeoff rather than hardcoding it.</p><p style="white-space: pre-wrap; margin-top: 0.1em; margin-bottom: 0.2em; unicode-bidi: plaintext; color: rgb(204, 204, 204); font-family: -apple-system, "system-ui", "Segoe UI", Roboto, sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb (54, 39, 18); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><strong>Pros</strong></p><ul style="padding-inline-start: 2em; color: rgb(204, 204, 204); font-family: -apple-system, "system-ui", "Segoe UI", Roboto, sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(54, 39, 18); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><li style="unicode-bidi: plaintext;">Reuses a shipped, exercised path; prevents the deadlock rather than detecting it (unlike<span> </span><code style="font-family: monospace; color: rgb(215, 186, 125); background-color: rgba(255, 255, 255, 0.1); padding: 2px 4px; borde r-radius: 3px; word-break: break-word; font-size: 0.9em;">execution_timeout</code>, which only kills an already-lost task and can't cover the pre-<code style="font-family: monospace; color: rgb(215, 186, 125); background-color: rgba(255, 255, 255, 0.1); padding: 2px 4px; border-radius: 3px; word-break: break-word; font-size: 0.9em;">execute()</code><span> </span>window).</li><li style="unicode-bidi: plaintext;">Default-off leaves unaffected deployments bit-for-bit unchanged — safe to backport, affected users flip one flag.</li><li style="unicode-bidi: plaintext;">Fixes the whole inherited-state class in the child, not just OpenSSL.</li></ul><p style="white-space: pre-wrap; margin-top: 0.1em; margin-bottom: 0.2em; unicode-bidi: plaintext; color: rgb(204, 204, 204); font-family: -apple-system, "system-ui", "Segoe UI", Roboto, sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spaci ng: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(54, 39, 18); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><strong>Cons</strong></p><ul style="padding-inline-start: 2em; color: rgb(204, 204, 204); font-family: -apple-system, "system-ui", "Segoe UI", Roboto, sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(54, 39, 18); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><li style="unicode-bidi: plaintext;">Opt-in means you hit the deadlock before you find the flag. Worth do cumenting the symptom (stuck in<span> </span><code style="font-family: monospace; color: rgb(215, 186, 125); background-color: rgba(255, 255, 255, 0.1); padding: 2px 4px; border-radius: 3px; word-break: break-word; font-size: 0.9em;">running</code>, no logs, no heartbeat progress) in the option description.</li><li style="unicode-bidi: plaintext;">When enabled at high concurrency the cost is real, in RSS as much as latency — the tradeoff is genuinely per-deployment, which is the argument for a flag over a default.</li><li style="unicode-bidi: plaintext;">Two more platform-conditional branches to keep tested.</li></ul><p style="white-space: pre-wrap; margin-top: 0.1em; margin-bottom: 0.2em; unicode-bidi: plaintext; color: rgb(204, 204, 204); font-family: -apple-system, "system-ui", "Segoe UI", Roboto, sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(54, 39, 18); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><strong>Longer term</strong> the cleaner answer is to not fork from a threaded parent at all — a <code style="font-family: monospace; color: rgb(215, 186, 125); background-color: rgba(255, 255, 255, 0.1); padding: 2px 4px; border-radius: 3px; word-break: break-word; font-size: 0.9em;">forkserver</code> forked before any threads start removes the entire inherited-lock class and amortises the import across children, which would address the high-concurrency cost too. Bigger change; shouldn't block the mitigation above.</p><p style="white-space: pre-wrap; margin-top: 0.1em; margin-bottom: 0.2em; unicode-bidi: plaintext; color: rgb(204, 204, 204); font-family: -apple-system, "system-ui", "Segoe UI", Roboto, sans-s erif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(54, 39, 18); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Happy to put up a PR for the split above if the direction seems right.</p> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
