On 16 Jun 2008, at 18:48, Oliver Fromme wrote:
Stut <[EMAIL PROTECTED]> wrote:
I'm having a serious problem with one of my FreeBSD servers. It runs
FreeBSD 6.2-RELEASE, Apache 2.2.8 and PHP 5.2.1. I've checked the
hardware and it's all OK. There's approximately 400 Apache processes
running and a 2GB memcached instance.
Up until last weekend this server was working perfectly at a level
of HTTP traffic higher than it's currently getting. Last week our
database server (separate server) died and had to be rebuilt. While
this was being done this server hosted the database. This has now
been completed and the PHP app is pointing back at the dedicated DB
server.
Top shows the following...
last pid: 26838; load averages: 10.22, 14.06, 13.55 up
2+00:34:47 18:03:43
619 processes: 1 running, 618 sleeping
CPU states: 4.9% user, 0.0% nice, 24.8% system, 0.4% interrupt,
70.0% idle
Mem: 2241M Active, 2718M Inact, 462M Wired, 394M Cache, 214M Buf,
1747M Free
Swap: 8192M Total, 124K Used, 8192M Free
PID UID THR PRI NICE SIZE RES STATE C TIME WCPU
COMMAND
26807 80 1 -4 0 79892K 9996K devfs 0 0:00 2.81%
httpd
26797 80 1 -4 0 82376K 12592K devfs 1 0:00 2.19%
httpd
26791 80 1 -4 0 82376K 12636K devfs 0 0:00 1.85%
httpd
26783 80 1 -4 0 82392K 12640K devfs 3 0:00 1.84%
httpd
[...]
Please let "vmstat 5" run for a minute ... Anything
that looks unusual?
procs memory page disks
faults cpu
r b w avm fre flt re pi po fr sr da0 pa0 in sy
cs us sy id
1 412 2 8910132 2743568 2230 3 1 0 2106 93 0 0 1098 4733
3025 4 17 79
1 412 0 8841792 2760548 1630 0 0 0 2276 0 21 0 658 3970
6374 3 26 71
0 422 0 8804744 2765076 1349 0 0 0 1394 0 12 0 576 3454
5131 3 26 71
1 421 0 8756808 2778660 1095 0 0 0 1581 0 48 0 574 3076
4415 3 25 72
0 420 0 8684932 2800128 2049 0 0 0 2927 0 45 0 505 2998
3770 3 24 73
1 380 0 8603676 2838452 1149 0 0 0 2842 0 22 0 505 3497
4339 2 26 72
1 21 0 8280260 3097392 7630 0 0 0 19969 0 29 0 1241 8895
6950 10 30 60
1 4 0 8195828 3131992 9113 0 0 0 9742 0 78 0 996 9405
2830 7 19 74
0 20 0 8169288 3121688 2899 0 0 0 2098 0 47 0 1193 7740
2733 4 23 73
11 13 0 8123476 3128688 1947 0 0 0 2160 0 52 0 1161 6231
2617 4 26 70
1 14 0 8067304 3143984 2298 0 0 0 2885 0 57 0 1806 5572
3373 4 25 70
2 17 1 8015924 3156168 2702 0 0 0 3164 0 23 0 1384 8243
2770 6 25 69
1 22 0 7956476 3176376 2013 0 0 0 2879 0 23 0 917 7063
2484 6 25 69
0 35 0 7944760 3150452 4274 0 0 0 2806 0 21 0 1591 8281
3399 7 25 68
1 67 3 7903160 3158776 2043 0 0 0 2442 0 20 0 1095 6405
3605 6 25 70
44 69 0 7872504 3147192 2569 0 0 0 1712 0 81 0 1137 5773
4998 6 26 69
1 146 0 7849632 3135388 2095 0 0 0 1274 0 19 0 869 5550
5466 5 26 69
3 195 2 7825932 3122116 2482 0 0 0 1586 0 15 0 863 5558
6135 5 26 69
1 244 3 7798148 3111624 1609 0 0 0 1226 0 10 0 849 4027
6477 4 27 69
2 273 3 7776772 3102948 2080 0 0 0 1310 0 14 0 604 4376
6527 4 26 70
1 312 2 7768232 3079012 3148 0 0 0 1800 0 12 0 1066 6088
10242 6 28 66
1 340 0 7742680 3068244 2020 0 0 0 1421 0 74 0 729 4407
8204 4 26 70
2 366 0 7740324 3068068 1612 0 0 0 1553 0 11 0 613 3526
6728 3 26 70
1 397 1 7928688 3059900 1886 0 0 0 1344 0 12 0 515 3081
6864 3 27 70
1 400 0 8074560 3008988 4771 0 0 0 1457 0 14 0 950 5309
8996 5 27 68
Lots of processes blocked - which I guess is what the devfs state
indicates.
Have you checked dmesg?
Yes. The only odd thing I can see is the following message, but from
what I've read it's not critical until you get 5 and it's only in
there once.
"collecting pv entries -- suggest increasing PMAP_SHPGPERPROC|
Is this FreeBSD i386 (32bit) or amd64 (64bit)?
FreeBSD harold.freeads.co.uk 6.2-RELEASE FreeBSD 6.2-RELEASE #0: Fri
Jan 12 08:43:30 UTC 2007 [EMAIL PROTECTED]:/usr/obj/usr/src/
sys/SMP amd64
Have you considered updating? 6.2-RELEASE isn't the
freshest anymore. You might even consider going to
7-stable and using the new ULE scheduler which copes
better with SMP servers.
I have, but I'd rather understand what's happening.
As you can see there's plenty of free memory and the CPU is 70% idle
yet the load is sky high.
Well, load 10 isn't that much for a 4-way SMP system.
A couple of weeks ago this server was fairly fast, load never really
going beyond 3 and everything was reasonably responsive. Now it
regularly goes up to and beyond a load of 10 (more often than not at
the moment) and everything is slow. This means our website users are
getting a very poor experience which is reflected in our traffic
levels which have dropped by about 25% since this started happening.
Thanks for your help. Any other ideas?
-Stut
_______________________________________________
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
