Re: sudden peak in load average
Zbigniew Szalbot wrote: Hello, From the MRTG data I can see that suddenly on a currently not very busy machine the load averege went over 15 or more. This happened around 10 in the morning. Not many entries in httpd access log, smtp server was not too much loaded (at that time it generally produces a load average of about 1). But this time: 2008-02-27 10:03:36 1JUICO-0002Bs-23 no immediate delivery: load average 15.88 Where can I look for more information about what happened. The server was rebooted early in the morning so its swap consumption was almost non-existent. Around 10 it suddenly consumed probably around 200MB of swap, now it stands at 135MB which is pretty normal with this machine (it only has 512 RAM). SWAP usage is normal with this machine but it usually is a matter of 2-3 days before it gradually takes over 10-13% of swap space. I tried by looking at: http access log auth.log maillog messages Nothing in there indicating an outburst of sudden activities. heh - I now think I may be wrong. The load average did not necessarily produce so much swap consumption so fast. At 10:13 I run a cron job optimising all mysql tables. So maybe in fact the swap was used by mysql operations, although I am still interested to now what casued load average to go above 15 at 10:03. 10:00AM up 3:14, 0 users, load averages: 0.40, 0.13, 0.05 10:01AM up 3:15, 0 users, load averages: 0.23, 0.14, 0.05 10:02AM up 3:16, 0 users, load averages: 0.08, 0.11, 0.05 10:03AM up 3:17, 0 users, load averages: 11.54, 3.09, 1.15 10:04AM up 3:18, 0 users, load averages: 13.26, 5.69, 2.28 10:05AM up 3:19, 0 users, load averages: 4.98, 4.69, 2.14 10:06AM up 3:20, 0 users, load averages: 1.79, 3.80, 1.99 Thanks for any pointers. No way to know. You'll have to set up more detailed logging, e.g. a script that runs ps if load is over a certain limit. Kris ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: sudden peak in load average
Hello, 2008/2/27, Kris Kennaway [EMAIL PROTECTED]: Zbigniew Szalbot wrote: Hello, From the MRTG data I can see that suddenly on a currently not very busy machine the load averege went over 15 or more. This happened around 10 in the morning. Not many entries in httpd access log, smtp server was not too much loaded (at that time it generally produces a load average of about 1). But this time: 2008-02-27 10:03:36 1JUICO-0002Bs-23 no immediate delivery: load average 15.88 Where can I look for more information about what happened. The server was rebooted early in the morning so its swap consumption was almost non-existent. Around 10 it suddenly consumed probably around 200MB of swap, now it stands at 135MB which is pretty normal with this machine (it only has 512 RAM). SWAP usage is normal with this machine but it usually is a matter of 2-3 days before it gradually takes over 10-13% of swap space. I tried by looking at: http access log auth.log maillog messages Nothing in there indicating an outburst of sudden activities. heh - I now think I may be wrong. The load average did not necessarily produce so much swap consumption so fast. At 10:13 I run a cron job optimising all mysql tables. So maybe in fact the swap was used by mysql operations, although I am still interested to now what casued load average to go above 15 at 10:03. 10:00AM up 3:14, 0 users, load averages: 0.40, 0.13, 0.05 10:01AM up 3:15, 0 users, load averages: 0.23, 0.14, 0.05 10:02AM up 3:16, 0 users, load averages: 0.08, 0.11, 0.05 10:03AM up 3:17, 0 users, load averages: 11.54, 3.09, 1.15 10:04AM up 3:18, 0 users, load averages: 13.26, 5.69, 2.28 10:05AM up 3:19, 0 users, load averages: 4.98, 4.69, 2.14 10:06AM up 3:20, 0 users, load averages: 1.79, 3.80, 1.99 Thanks for any pointers. No way to know. You'll have to set up more detailed logging, e.g. a script that runs ps if load is over a certain limit. Thank you Kris! Is anybody willing to share such a script (if there is one)? Thanks! -- Zbigniew Szalbot ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: sudden peak in load average
From the MRTG data I can see that suddenly on a currently not very busy machine the load averege went over 15 or more. This happened around 10 in the morning. Not many entries in httpd access log, smtp server was not too much loaded (at that time it generally produces a load average of about 1). By any chance, do you run SpamAssassin? I have seen load average bursts with SA. It seems to me that spam sites are bursting spam to attempt to bring down the anti-spam filters. As mentioned by others a ps (or I prefer pstree) list will help solve the issue. --Mark Tinguely ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: sudden peak in load average
Hello, 2008/2/27, Mark Tinguely [EMAIL PROTECTED]: From the MRTG data I can see that suddenly on a currently not very busy machine the load averege went over 15 or more. This happened around 10 in the morning. Not many entries in httpd access log, smtp server was not too much loaded (at that time it generally produces a load average of about 1). By any chance, do you run SpamAssassin? I have seen load average bursts with SA. It seems to me that spam sites are bursting spam to attempt to bring down the anti-spam filters. As mentioned by others a ps (or I prefer pstree) list will help solve the issue. Thanks. I do have SA but I studied the logs carefully and no outside connections, except for one with a vary small message desitined for mailman subscription arrived at around that time. -- Zbigniew Szalbot ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: sudden peak in load average
Zbigniew Szalbot wrote: Hello, 2008/2/27, Mark Tinguely [EMAIL PROTECTED]: From the MRTG data I can see that suddenly on a currently not very busy machine the load averege went over 15 or more. This happened around 10 in the morning. Not many entries in httpd access log, smtp server was not too much loaded (at that time it generally produces a load average of about 1). By any chance, do you run SpamAssassin? I have seen load average bursts with SA. It seems to me that spam sites are bursting spam to attempt to bring down the anti-spam filters. As mentioned by others a ps (or I prefer pstree) list will help solve the issue. Thanks. I do have SA but I studied the logs carefully and no outside connections, except for one with a vary small message desitined for mailman subscription arrived at around that time. Is it possible that there's a message in your queue that's *being processed*, so it may have arrived earlier than near that time and causes the spike? What tests is SA running? Or something with your DNS settings that is holding up SA while trying to look something up? ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: sudden peak in load average
Is it possible that there's a message in your queue that's *being processed*, so it may have arrived earlier than near that time and causes the spike? Bart is correct that the SA processing occurs before sendmail log entry. Lately, I have had problems with the latest spamass-milter. Occasionally, something is forking off another spamass-milter and the original one is in some tight loop eating processor time. I am not sure if it is the newer spamass-milter or the fact that I also added the dkim-milter into the mix. FYI: I sent to the original questioner a crude C program to monitor his current loadaverage. This monitor will save the output of the command ps -aux to a timestamped temporary file when the current loadaverage exceeds a defined amount (15.0). --Mark Tinguely ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: sudden peak in load average
Mark Tinguely wrote: Is it possible that there's a message in your queue that's *being processed*, so it may have arrived earlier than near that time and causes the spike? Bart is correct that the SA processing occurs before sendmail log entry. Lately, I have had problems with the latest spamass-milter. Occasionally, something is forking off another spamass-milter and the original one is in some tight loop eating processor time. I am not sure if it is the newer spamass-milter or the fact that I also added the dkim-milter into the mix. FYI: I sent to the original questioner a crude C program to monitor his current loadaverage. This monitor will save the output of the command ps -aux to a timestamped temporary file when the current loadaverage exceeds a defined amount (15.0). Another thing to look at would be the output of something like lsof, so that if it is spamassassin, maybe there's a possibility that it could be narrowed down to a particular temporary file unless there's another way to see if there's a particular message chewing away on SA's analysis? It doesn't take a big message to skew SA asunder if it has the right bit of information in it... ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
