kaxil commented on issue #67515:
URL: https://github.com/apache/airflow/issues/67515#issuecomment-4550501544
Each task in Airflow 3 runs in its own `os.fork()` from the Task SDK
supervisor. The fork parses the DAG file, runs the task, then `os._exit()`s.
Memory is released to the kernel on exit; Python GC doesn't unload
`sys.modules` entries.
For a 2-worker x 16-process Celery cluster:
| Process | Accumulates the heavy SDKs being discussed (neo4j / snowflake /
google.cloud)? | Lazy-load benefit |
|---|---|---|
| Celery daemon + 32 prefork children | No -- they import the celery
executor framework, not the third-party SDKs that #67515 targets | 0 |
| Per-task fork (ephemeral) | Yes, but the fork dies after the task | ~60 MB
peak per fork (Neo4j case), released on `os._exit()` |
| **Dag-processor parsing process(es)** | **Yes, for the lifetime of the
parsing process** | Bounded by `parsing_processes` x per-process saving |
So in the 12-task mixed-DAG case, "11 of 12 tasks save 60 MB each" is 11
transient peak reductions in ephemeral forks that die minutes later. Nothing
persists in the worker tier.
The only long-lived processes that benefit are the dag-processor parsing
workers. The saving scales with `parsing_processes` x per-process saving,
typically landing somewhere in the single-digit percent range of total worker
RAM on a properly-sized cluster.
For deployments that hit OOM in workers, the dominant consumers are data,
not imports: XComs, intermediate task results, cursor rows, BigQuery result
sets, pandas dataframes pulled into the worker.
Frameworks that do lazy-load successfully do it architecturally, not by
hand-moving imports. **boto3** imports the top-level package eagerly, but
`Session.client('s3')` is a dispatch point that lazy-loads per-service models
from JSON specs in `botocore.loaders` -- the laziness is in the service-model
layer, not the import graph. **Django** populates an app registry at startup
and uses `apps.get_model(label)` for string-based lookup after that, avoiding
direct `from x.y import Model` ordering. Both are deliberate architectural
patterns, not "move every heavy import into a method body." Inline imports for
performance is not a pattern that scales -- it's a smell.
And that's why Python 3.15's PEP 810 (`lazy import` keyword +
`__lazy_modules__` polyfill) is the right shape of what we'd want here imo.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
