richevanscybermyte opened a new pull request, #13737: URL: https://github.com/apache/cloudstack/pull/13737
### Description This PR fixes #13735. The router health check scripts run shell commands with `Popen(cmd, shell=True, stdout=PIPE)` and then call `pout.wait()` before reading the pipe. When the command output exceeds the OS pipe capacity (64 KiB on Linux), the child process blocks writing to the full pipe and the parent blocks in `wait()` forever. This is the deadlock warned about in the Python `subprocess` documentation. In `iptables_check.py` the command is `iptables-save | grep <destIp>`. On routers with large port-forwarding rule sets the output exceeds the pipe buffer and the check hangs on every run. Because the advanced health check fires every `router.health.checks.advanced.interval` (10 minutes by default), a new stuck process chain (`monitorServices.py advanced` -> `sh` -> `iptables_check.py`) accumulates every interval, about 13 MB RSS each. On a 512 MB router, RAM and swap are exhausted after roughly 9 hours, keepalived's heartbeat track script can no longer fork within its timeout, the router enters FAULT, demotes, wipes its keepalived config and reboots. Redundant pairs fail near-simultaneously because both routers are created together, so the exhaustion clocks are synchronized. See the issue for full evidence, including a control group of routers with small rule sets running the same check cleanly for 152 days. The fix drains the pipe with `communicate()` and checks `returncode`, instead of calling `wait()` with an unread pipe. The same latent pattern is fixed in `memory_usage_check.py`, `cpu_usage_check.py` and `gateways_check.py`; their outputs are normally tiny so they do not hang today, but they share the same hazard. ### Types of changes - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] New feature (non-breaking change which adds functionality) - [x] Bug fix (non-breaking change which fixes an issue) - [ ] Enhancement (improves an existing feature and functionality) - [ ] Cleanup (Code refactoring and cleanup, that may add test cases) - [ ] Build/CI ### Bug Severity - [ ] BLOCKER - [x] Critical - [ ] Major - [ ] Minor - [ ] Trivial ### How Has This Been Tested? Reproduced on CloudStack 4.22.0.0/4.22.1.0 (KVM, redundant VPC, systemvm template 4.22.0) with a VPC whose `iptables-save` output is 77,029 bytes. Before the fix, every advanced health check run left a permanently stuck `iptables_check.py` chain (verified via `ps`, both processes in `do_wait`) and router memory climbed about 87 MB/h until the router self-rebooted at about 9 hours of uptime. With the fixed script logic (equivalent change validated on the affected router), the check completes in under a second against the same 77 KB rule set, no processes accumulate, and the router has run past the previous failure window with flat memory usage. Routers with small rule sets (1-9 KB) behave identically before and after, since their output never filled the pipe. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
