Leon,
You beat me to it. I would be surprised if it is not a garbage
collection problem. You should be able to get stats on the java apps to
see if this is true. If it is you can play with the app settings to
hopefully solve the problem.
cheers,
ski
On 11/12/2013 09:07 PM, Leon Towns-von Stauber wrote:
Random-seeming resource usage spikes on a Java application server?
I'd suspect garbage collection.
- Leon
On Nov 12, 2013, at 7:57 PM, Chris Picton <[email protected]>
wrote:
Hi all
I have a set of servers running asterisk and some java apps which
have (so far) unexplained spikes in load average.
A typical spike which occurs at "random" times would see the 1
minute load average load go from around 4 to upwards of 50,
sometime approaching 200, within one second.
From proc manpage, the 1 min load average is "number of jobs in
the run queue (state R) or waiting for disk I/O (state D) averaged
over 1 minute"
I am collecting many different stats from proc every second, but
nothing I have found can correlate with the spike in load average.
The counts of process numbers from /proc/stat and /prov/loadavg do
not match up to the sudden spike. I have looked at memory paging,
irqs, number of threads, cpu states(intr/iowait/etc), network
traffic, disk io, etc but no metric I have yet found indicates it
is changing behaviour at the same time as the load average spikes
As I am writing this, I have realized that I am not actually
tracking the numbers which would be the direct cause of the load
average, which would be to loop through all processes, extract the
process state from /proc/<pid>/stat, and add up the various types.
This would provide (hopefully) a match so I could see that the load
average numbers are "correct", and may indicate a cause (many
processes waiting for IO, or lots of the same process (asterisk or
java) being scheduled to run at the same time)
While I do that, would anyone have some other idea of how to
troubleshoot the cause of very high load spikes?
Regards
Chris
_______________________________________________ Tech mailing list
[email protected]
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech This list
provided by the League of Professional System Administrators
http://lopsa.org/
_______________________________________________ Tech mailing list
[email protected]
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech This list
provided by the League of Professional System Administrators
http://lopsa.org/
--
"When we try to pick out anything by itself, we find it
connected to the entire universe" John Muir
Chris "Ski" Kacoroski, Director of LOPSA, [email protected],
206-501-9803 or ski98033 on most IM services
_______________________________________________
Tech mailing list
[email protected]
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
