May be you hit memory shortage in a particular NUMA node only, for example.
# numactl --hardware # numastat -m Or go hard way - trace kernel where exactly do we get -ENOMEM: trace the kernel function cgroup_mkdir() using /sys/kernel/debug/tracing/ with function_graph tracer. https://lwn.net/Articles/370423/ -- Best regards, Konstantin Khorenko, Virtuozzo Linux Kernel Team On 01/28/2021 12:43 PM, Joe Dougherty wrote:
I checked that, doesn't appear to be the case. *# pwd* */sys/fs/cgroup/memory* *# cat *limit_in_bytes* *9223372036854771712* *9223372036854767616* *2251799813685247* *2251799813685247* *9223372036854771712* *9223372036854771712* *9223372036854771712* *# cat *failcnt* *0* *0* *0* *0* *0* # pwd /sys/fs/cgroup/memory/machine.slice *# cat *limit_in_bytes* *9223372036854771712* *9223372036854767616* *9223372036854771712* *9223372036854771712* *9223372036854771712* *9223372036854771712* *9223372036854771712* *# cat *failcnt* *0* *0* *0* *0* *0* On Thu, Jan 28, 2021 at 2:47 AM Konstantin Khorenko <[email protected] <mailto:[email protected]>> wrote: Hi Joe, i'd suggest to check memory limits for root and "machine.slice" memory cgroups /sys/fs/cgroup/memory/*limit_in_bytes /sys/fs/cgroup/memory/machine.slice/*limit_in_bytes All of them should be unlimited. If not - search who limit them. -- Best regards, Konstantin Khorenko, Virtuozzo Linux Kernel Team On 01/27/2021 10:28 PM, Joe Dougherty wrote:I'm running into an issue on only 1 of my OpenVZ 7 nodes where it's unable to create a directory on /sys/fs/cgroup/memory/machine.slice due to "Cannot allocate memory" whenever I try to start a new container or restart and existing one. I've been trying to research this but I'm unable to find any concrete info on what could cause this. It appears to be memory related because sometimes if I issue "echo 1 /proc/sys/vm/drop_caches" it allows me to start a container (this only works sometimes) but my RAM usage is extremely low with no swapping (swappiness even set to 0 for testing). Thank you in advance for your help. Example: *# vzctl start 9499* *Starting Container ...* *Mount image: /vz/private/9499/root.hdd* *Container is mounted* *Can't create directory /sys/fs/cgroup/memory/machine.slice/9499: Cannot allocate memory* *Unmount image: /vz/private/9499/root.hdd (190)* *Container is unmounted* *Failed to start the Container* Node Info: *Uptime: 10 days* *OS: Virtuozzo 7.0.15* *Kernel: 3.10.0-1127.18.2.vz7.163.46 GNU/Linux* *System Load: 3.1* */vz Usage: 56% of 37T* *Swap Usage: 0%* *RAM Free: 84% of 94.2GB* * * *# free -m* * total used free shared buff/cache available* *Mem: 96502 14259 49940 413 32303 80990* *Swap: 32767 93 32674* _______________________________________________ Users mailing list [email protected] <mailto:[email protected]> https://lists.openvz.org/mailman/listinfo/users_______________________________________________ Users mailing list [email protected] <mailto:[email protected]> https://lists.openvz.org/mailman/listinfo/users -- */-Joe Dougherty/* */Chief Operating Officer/* /***Secure Dragon LLC */ */www.SecureDragon.net <http://www.SecureDragon.net>/* _______________________________________________ Users mailing list [email protected] https://lists.openvz.org/mailman/listinfo/users
_______________________________________________ Users mailing list [email protected] https://lists.openvz.org/mailman/listinfo/users
