rahulchheda opened a new pull request, #65370: URL: https://github.com/apache/airflow/pull/65370
## What Service IPC sockets in a background thread during `_refresh_dag_bundles()` so child parsing processes don't block on `Variable.get()` / `Connection.get()` calls. ## Why In `_run_parsing_loop()`, IPC is only serviced once per iteration via `_service_processor_sockets()`. While the main loop executes `_refresh_dag_bundles()` and other operations, child processes that call `Variable.get()` block on `socket.recv()` waiting for the parent to respond. With many DAG files and multiple parsing processes, this turns sub-second parses into multi-minute waits. This is a regression from Airflow 2.x where `Variable.get()` was a direct DB query with no IPC dependency. For comparison, `WatchedSubprocess._monitor_subprocess()` in the task execution path has a tight loop that continuously services IPC — the dag-processor lacks this. ## How - Added a `_ipc_service_thread()` context manager that runs `_service_processor_sockets()` in a daemon thread with a 100ms poll interval - Wrapped `_refresh_dag_bundles()` with this context manager — the longest-running phase where child processes are likely to be blocked - Thread is automatically stopped when the context exits, before the main loop continues with its own `_service_processor_sockets()` call The change is minimal and non-invasive — existing IPC servicing in the main loop is untouched. Closes #65369 --- ^ Add meaningful description above Read the **[Pull Request Contribution Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst)** for more information. In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html). In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{pr_number}.significant.md`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
