I am having problems with GPFS servers running out of memory. We have an open PMR for this, however if anyone has seen this or has any ideas I would be grateful for a heads up. Servers have 128 Gbytes f RAM, kernel 2.6.32-573.18.1.el6.x86_64, GPFS version 4.2.3.4
In the latest incident the free memory went to below 1Gbyte, and we started to have processes killed, including our monitoring setup. I shut down GPFS on that server and /proc/meminfo still shows: Slab: 111192296 kB SReclaimable: 29020 kB SUnreclaim: 111163276 kB Am I barking up a wrong tree here and pointing the finger at GPFS? Something is causing the scsi_data_buffer slab memory usage (see below). One thing I did yesterday was to change the disk scheduler for each disk from cfq to dealine (as recommended in the tuning guide) However the server was already in short memory at that point. Slabtop shows Active / Total Objects (% used) : -306803185 / -306722574 (100.0%) Active / Total Slabs (% used) : 27749714 / 27749719 (100.0%) Active / Total Caches (% used) : 115 / 198 (58.1%) Active / Total Size (% used) : 93857848.58K / 93872319.47K (100.0%) Minimum / Average / Maximum Object : 0.02K / 0.02K / 4096.00K OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME 3987822096 3987821817 0% 0.02K 27693209 144 110772836K scsi_data_buffer 91155 64448 70% 0.06K 1545 59 6180K size-64 36064 32035 88% 0.03K 322 112 1288K size-32 35505 34334 96% 0.25K 2367 15 9468K skbuff_head_cache 33876 33874 99% 8.00K 33876 1 271008K size-8192 33804 33615 99% 0.14K 1252 27 5008K sysfs_dir_cache -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt.
_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
