gang-zh commented on code in PR #72164:
URL: https://github.com/apache/airflow/pull/72164#discussion_r3981941402
##########
task-sdk/src/airflow/sdk/execution_time/supervisor.py:
##########
@@ -531,15 +547,37 @@ def _resolve_child_target(dotted: str) -> Callable[[],
None]:
return pkgutil.resolve_name(dotted)
+# Runs in the exec'd child before anything else. execve reset PR_SET_DUMPABLE
(4 in
+# <linux/prctl.h>); restore it before the Airflow import so the window in
which a same-UID
+# sibling can open /proc/<pid>/mem or ptrace-attach is interpreter start only
(a descriptor
+# or attach taken in that window survives a later prctl -- the kernel checks
once, at open).
+# _child_exec_main() repeats the call as the logged fallback.
+_CHILD_EXEC_PRELUDE = """\
+import sys
+if sys.platform == "linux":
+ import ctypes
Review Comment:
Fixed in 3f40f69749 — the import is inside the `try` now, matching
`_make_process_nondumpable()`;
`test_prelude_survives_an_interpreter_without_ctypes` runs the prelude with
`sys.modules['_ctypes'] = None`.
##########
task-sdk/src/airflow/sdk/execution_time/supervisor.py:
##########
@@ -531,15 +547,37 @@ def _resolve_child_target(dotted: str) -> Callable[[],
None]:
return pkgutil.resolve_name(dotted)
+# Runs in the exec'd child before anything else. execve reset PR_SET_DUMPABLE
(4 in
+# <linux/prctl.h>); restore it before the Airflow import so the window in
which a same-UID
+# sibling can open /proc/<pid>/mem or ptrace-attach is interpreter start only
(a descriptor
+# or attach taken in that window survives a later prctl -- the kernel checks
once, at open).
+# _child_exec_main() repeats the call as the logged fallback.
+_CHILD_EXEC_PRELUDE = """\
+import sys
+if sys.platform == "linux":
+ import ctypes
+ try:
+ ctypes.CDLL(None, use_errno=True).prctl(4, 0, 0, 0, 0)
+ except Exception:
+ pass
+"""
+_CHILD_EXEC_BOOTSTRAP = _CHILD_EXEC_PRELUDE + (
Review Comment:
Added in 3f40f69749:
`test_fork_exec_bootstrap_runs_an_importable_target_end_to_end` drives
`ActivitySubprocess.start` through the real `os.execv` — fresh interpreter runs
`_CHILD_EXEC_BOOTSTRAP`, FDs 0-3 rebuilt, target rehydrated by name, stdout
asserted through the supervisor — plus the `startswith(_CHILD_EXEC_PRELUDE)`
floor. It uses an importable probe standing in for `_subprocess_main` instead
of a `test_run_simple_dag` row: the suite stubs `_get_plugins` in-process
(task-sdk conftest), which a bare-forked child inherits and a fresh interpreter
does not, so the exec'd task runner fails with 'Plugins folder is not set'
(reproduced outside pytest in both fork modes; it is also why
`test_run_simple_dag` fails on macOS today).
##########
airflow-core/docs/security/workload.rst:
##########
@@ -67,8 +67,11 @@ Worker process memory protection (Linux)
''''''''''''''''''''''''''''''''''''''''
On Linux, the supervisor process calls ``prctl(PR_SET_DUMPABLE, 0)`` at the
start of
-``supervise_task()`` before forking the task process. This flag is inherited
by the forked
-child. Marking processes as non-dumpable prevents same-UID sibling processes
from reading
+``supervise_task()`` before forking the task process. A bare-forked child
inherits the flag;
+a child started through ``exec`` (macOS, or ``[core]
execute_tasks_new_python_interpreter``)
+restores it in its bootstrap, before importing Airflow, because ``execve``
resets it; the remaining
+window is interpreter start, which ``kernel.yama.ptrace_scope >= 1`` covers.
Marking processes as
Review Comment:
Thanks — both docs now say it covers `/proc/<pid>/mem` and `ptrace` attach
for that window (bb3553efad).
##########
airflow-core/newsfragments/72164.significant.rst:
##########
@@ -0,0 +1,22 @@
+``[core] execute_tasks_new_python_interpreter`` now applies to Airflow 3 task
processes
+
+On Airflow 3 the option had no effect on task execution (only the Edge worker
read it). When set to
+``True``, the task supervisor now ``exec``\ s a fresh interpreter right after
forking the task process,
+which prevents the fork from inheriting a lock held by a supervisor thread (a
permanent hang at the
+task's first TLS call). Deployments that kept the option ``True`` from Airflow
2 get this behaviour,
+and its per-task interpreter start-up cost, on upgrade without further action;
set it to ``False`` to
+keep bare fork. Edge workers with the option ``True`` already start a fresh
interpreter for the
+supervisor and will now start a second one for the task. The Dag processor,
the triggerer and task
Review Comment:
Added in bb3553efad: config text and newsfragment say the task process reads
the global value (Edge resolves it per team, so a team-scoped `True` alone does
not turn it on). Forwarding the resolved value in `_launch_job_subprocess`
sounds right as an edge3 follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
