Tomas, Marcy,
Thanks. I copied and pasted cmmflush and it seems to work nicely:
# free -m
total used free shared buffers cached
Mem: 492 162 329 0 29 83
-/+ buffers/cache: 49 442
Swap: 898 0 898
# cmmflush
11:16:17 Currently free 328MB, dropping cache...
11:16:18 Now free 422MB, released 93MB
11:16:18 CMM base is 0MB, target is 396MB
11:16:19 CMM currently at 396MB...
11:16:19 Done! CMM base restored to 0MB
11:16:19 Released 396 MB of memory
# free -m
total used free shared buffers cached
Mem: 492 69 423 0 0 19
-/+ buffers/cache: 49 442
Swap: 898 0 898
Rob, thanks for the contribution.
-Mike
On Thu, Jul 9, 2015 at 11:06 AM, Pavelka, Tomas <tomas.pave...@ca.com>
wrote:
> This is a really ugly problem that I don't have a solution for. But let me
> give you a bit of info if you want to do your own digging:
>
> The way I found this is that I was adding NICs to a Linux on the fly.
> Sometimes this would fail, saying page allocation in syslog. The discussion
> on this list is here:
>
> http://www.mail-archive.com/linux-390%40vm.marist.edu/msg65371.html
>
> What I found later is that the NIC driver needs 64k of memory in kernel
> space. This means the memory needs to be continuous. The kernel keeps
> memory in structures called slabs, and keeps pools of these. If you do
>
> cat /proc/buddyinfo
> Node 0, zone DMA 9078 10398 3135 838 164 14 0
> 0 2
>
> Another way to get memory report is to run "echo m > /proc/sysrq-trigger"
> and look into syslog for a report about kernel memory usage.
>
> You will see how many slabs of each order you have. 9078 of order 1 slabs
> (4kb), 10398 of order 2 slabs (8kb) ... 2 order 9 slabs (1MB). If a slab of
> lower order is needed it may split a higher order one (e.g. if the kernel
> wants a 4k slab it may split an 8k slab into two). Lots of kernel
> allocations and you may run out of the higher order slabs. What worked for
> me for trigerring this condition was moving a lot of data to the Linux over
> SCP. There may be other causes.
>
> Now the significance of 32k is that this is where Linux stops retrying to
> rearrange memory to find larger slabs. I don't remember the details, but if
> you want to investigate look at the kernel sources, namely mm/page_alloc.c
> and mm/vmscan.c
>
> So the bottom line is, anytime you have an operation that needs a large
> buffer in kernel (chccwdev of a NIC, vmcp with --buffer, DIAG from Linux)
> it may fail at unexpected times. I have not found a good way to get around
> this but I will be interested if you find anything.
>
> In the case of VMCP what may help is if it allocated a buffer at kernel
> startup. At the moment it allocates it for every call, see
> http://lxr.free-electrons.com/source/drivers/s390/char/vmcp.c#L105
>
> Tomas
>
>
----------------------------------------------------------------------
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
----------------------------------------------------------------------
For more information on Linux on System z, visit
http://wiki.linuxvm.org/
