Hi all,
Recently I found out one server generating alot of load on our storage.
I did a few steps to narrow down what is the cause and what might be a solution.
As first I used iostat to get what device is using most IO:
Linux 2.6.18-274.7.1.el5 (hostname) 05/14/2012
avg-cpu: %user %nice %system %iowait %steal %idle
11.99 0.00 8.53 2.85 0.00 76.63
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 952.75 5.54 14090.70 42708891 108632689706
sda1 0.00 0.00 0.00 6988 1802
sda2 0.51 0.17 8.29 1322786 63885168
sda3 952.24 5.37 14082.42 41378693 108568802736
sdb 8.29 1.01 739.31 7791930 5699710024
dm-0 2.14 1.48 17.03 11382146 131278120
dm-1 0.76 0.78 5.47 6030578 42202752
dm-2 8.69 3.08 68.72 23737626 529796056
dm-3 1748.90 0.03 13991.19 226754 107865527240
dm-4 0.00 0.00 0.00 1192 216
dm-5 92.49 1.01 739.31 7790146 5699710024
Using ls -lh /dev/mapper I found out that dm-3 corresponds with the mountpoint
for /tmp (note that the system is using LVM).
One thing that seems strange to me is that there is a huge load of blocks
written however almost none get red.
Then I used lsof | grep tmp:
httpd 25142 apache DEL REG 253,3
13 /tmp/apc.Fm8SAS
httpd 25386 apache DEL REG 253,3
13 /tmp/apc.Fm8SAS
httpd 25429 apache DEL REG 253,3
13 /tmp/apc.Fm8SAS
httpd 25431 apache DEL REG 253,3
13 /tmp/apc.Fm8SAS
httpd 25454 apache DEL REG 253,3
13 /tmp/apc.Fm8SAS
httpd 25622 apache DEL REG 253,3
13 /tmp/apc.Fm8SAS
httpd 25792 apache DEL REG 253,3
13 /tmp/apc.Fm8SAS
httpd 25794 apache DEL REG 253,3
13 /tmp/apc.Fm8SAS
httpd 25846 apache DEL REG 253,3
13 /tmp/apc.Fm8SAS
httpd 25849 apache DEL REG 253,3
13 /tmp/apc.Fm8SAS
...
The only other is for sshd and another for vmware tools.
If I try to do ls -lh /tmp/apc.Fm8SAS I get no such file or directory.
Using the access time I would see if this file is accessed over and over
again.
Am I missing something here? Is this apache doing a fstat() call for example?
Or is this file created and deleted over and over again at a very fast
pace that I can't access it (this seems less likely to me).
Has anyone seen such behaviour in httpd? Or does someone knows a little more of
the inner workings of httpd?
The name does me think that APC (sort of php module iirc) is having something
to do with this:
apc.ini:apc.mmap_file_mask=/tmp/apc.XXXXXX
The config file seems to confirm this.
Anyone has an idea how to fix this or to go further investigate?
Thanks alot in advance!
