Tomas, Marcy, Thanks. I copied and pasted cmmflush and it seems to work nicely:
# free -m total used free shared buffers cached Mem: 492 162 329 0 29 83 -/+ buffers/cache: 49 442 Swap: 898 0 898 # cmmflush 11:16:17 Currently free 328MB, dropping cache... 11:16:18 Now free 422MB, released 93MB 11:16:18 CMM base is 0MB, target is 396MB 11:16:19 CMM currently at 396MB... 11:16:19 Done! CMM base restored to 0MB 11:16:19 Released 396 MB of memory # free -m total used free shared buffers cached Mem: 492 69 423 0 0 19 -/+ buffers/cache: 49 442 Swap: 898 0 898 Rob, thanks for the contribution. -Mike On Thu, Jul 9, 2015 at 11:06 AM, Pavelka, Tomas <tomas.pave...@ca.com> wrote: > This is a really ugly problem that I don't have a solution for. But let me > give you a bit of info if you want to do your own digging: > > The way I found this is that I was adding NICs to a Linux on the fly. > Sometimes this would fail, saying page allocation in syslog. The discussion > on this list is here: > > http://www.mail-archive.com/linux-390%40vm.marist.edu/msg65371.html > > What I found later is that the NIC driver needs 64k of memory in kernel > space. This means the memory needs to be continuous. The kernel keeps > memory in structures called slabs, and keeps pools of these. If you do > > cat /proc/buddyinfo > Node 0, zone DMA 9078 10398 3135 838 164 14 0 > 0 2 > > Another way to get memory report is to run "echo m > /proc/sysrq-trigger" > and look into syslog for a report about kernel memory usage. > > You will see how many slabs of each order you have. 9078 of order 1 slabs > (4kb), 10398 of order 2 slabs (8kb) ... 2 order 9 slabs (1MB). If a slab of > lower order is needed it may split a higher order one (e.g. if the kernel > wants a 4k slab it may split an 8k slab into two). Lots of kernel > allocations and you may run out of the higher order slabs. What worked for > me for trigerring this condition was moving a lot of data to the Linux over > SCP. There may be other causes. > > Now the significance of 32k is that this is where Linux stops retrying to > rearrange memory to find larger slabs. I don't remember the details, but if > you want to investigate look at the kernel sources, namely mm/page_alloc.c > and mm/vmscan.c > > So the bottom line is, anytime you have an operation that needs a large > buffer in kernel (chccwdev of a NIC, vmcp with --buffer, DIAG from Linux) > it may fail at unexpected times. I have not found a good way to get around > this but I will be interested if you find anything. > > In the case of VMCP what may help is if it allocated a buffer at kernel > startup. At the moment it allocates it for every call, see > http://lxr.free-electrons.com/source/drivers/s390/char/vmcp.c#L105 > > Tomas > > ---------------------------------------------------------------------- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 ---------------------------------------------------------------------- For more information on Linux on System z, visit http://wiki.linuxvm.org/