(And I hope smeone else will come along and "just fix" this package :))
Hi all,
Okay, I had a stab at this. I looked at IPC-Shareable-1.19/t/47-seg_size.t.My background is rather in C, and I have never written any Perl, so for the patch, I will happily leave that to the Perl devs here. Therefore, I am afraid, no "just fix" solution; but still the cause and a write-up for a patch.
## The Root CauseThe test has a "beyond RAM limits" block that passes `limit => 0` to bypass internal module checks, and intentionally requests an "impossible" 999,999,999,999-byte (~1 TB) segment, expecting the OS to refuse it.
- On non-Linux systems (macOS, BSD), this works, because they enforce small, strict default SHMMAX ceilings (e.g., 4 MB to 512 MB). - On Linux, the default `kernel.shmmax` is effectively unlimited (`ULONG_MAX - 2**24` - check `/proc/sys/kernel/shmmax`). Therefore, `shmget()` falls through to the standard overcommit heuristic in `__vm_enough_memory()`, which only rejects segments that exceed MemTotal + SwapTotal. - On large infrastructure nodes where RAM + swap exceeds 1 TB, the kernel grants the segment lazily. When `SharedMem.pm` subsequently performs a `shmread()` to inspect the segment, it forces two massive physical allocations simultaneously: a 1 TB Perl scalar heap allocation for the read buffer, and the physical materialisation of the 1 TB zero-filled shared memory segment via page faults. This results in an immediate 2 TB memory footprint spike, causing the test runner to be killed.
Paul's numbers confirm this.Paul reported a virtual size of 1,953,147,760 KiB for `/usr/bin/perl t/65-seg_size.t` on line 601. When converted, that is exactly 2,000,023,306,240 bytes. Against the 1 TB target request, this maps almost cleanly to a 2.0000x multiplier. The remaining ~22 MB is also to be found in Paul's message: `/usr/bin/perl /usr/bin/prove` on line 463 shows a virtual size of 22240 KiB = 22.19 MiB.
## Proposed Fix ArchitectureRather than relying on a hardcoded "impossible" size that modern server hardware can easily accommodate, the test script (t/47-seg_size.t) should calculate a dynamic refusal ceiling at runtime:
1. Keep a baseline fallback: Define a static constant of 999999999999 bytes (~1 TB) as the default choice for non-Linux architectures. 2. Handle a lowered Linux shmmax (if defined): On Linux, check `/proc/sys/kernel/shmmax`. If a system administrator has manually set it to a value lower than 1 TB, keep the 1 TB test size, as the OS will already safely reject it. 3. Account for Overcommit Mode 1: Check `/proc/sys/vm/overcommit_memory`. If it is set to 1 (always overcommit), the kernel will never reject an allocation at creation time. In this case, cleanly skip the two test assertions to prevent a guaranteed out-of-memory crash. 4. Compute a dynamic ceiling via `meminfo`: If overcommit is in standard mode, parse `/proc/meminfo` to read MemTotal and SwapTotal. Calculate a new target size equivalent to MemTotal + SwapTotal + 1 GB (adding 1 GB to safely clear page-alignment rounding bounds). Use this computed size for the tie operation to guarantee an immediate application-layer or kernel-level refusal without initialising any real memory blocks. 5. Enforce defensive container fallbacks: Ensure that all file-read operations on `/proc` files fail gracefully (returning the 1 TB baseline fallback) if executed inside locked-down sandbox environments, Distrobox environments, or strict container runtimes where `/proc` access is restricted.
This runtime logic completely avoids hardcoding architecture-specific blacklists, cleanly accommodates modern high-memory hardware configurations, and prevents sudden worker terminations.
Edmund
--
Edmund Lodewijks <[email protected]>
TZ: UTC+2 / GMT+2
OpenPGP_signature.asc
Description: OpenPGP digital signature
