Dear Khan, as Vladimir said, "System CPU" is spent in the kernel on I/O, Interrupts, memory management. Just out of curiosity: what Linux are you (is your customer) running, which kernel version and what is the uptime?
I ask, because I recently was facing a similar issue on Servers running SLES11/SP2 (kernel 3.0.58-0.6.2-default). Those were used for Tomcat (Java) processes, not HPC. They started to really max out all CPUs 100% with 75% solid "red". But that happened only after some days of uptime It turned out that in our situation turning of the half-baked (at least in that kernel) "Transparent Huge Pages" feature off (or to voluntary mode) solved the problem: # echo madvise > /sys/kernel/mm/transparent_hugepage/enabled # echo madvise > /sys/kernel/mm/transparent_hugepage/defrag # cat /sys/kernel/mm/transparent_hugepage/{enabled,defrag} always [madvise] never always [madvise] never Doing that is pretty much without risk and can be done/reverted at any time. It may cost a bit of performance in systems with lots of memory, but I personally think it is overrated for general usage. As I said, not sure it applies to your situation, but comes from a real world high throughput environment. Cheers Martin On Tue, Oct 13, 2015 at 7:49 PM, Kamran Khan <kam...@pssclabs.com> wrote: > Hi All, > > > This isn't a problem with Ganglia, but I was hoping I might get a little > advice on what I am seeing. I have a customer who is running ls-dyna > applications, and he is noticing something odd. He is noticing his jobs > being bogged down and not running at their full capacity. He looked at the > Ganglia web interface and saw that "System CPU" was at 100%, while "User > CPU" was at like 20%. What processes does the "System CPU" refer to? What > tools can I use to track what might be pushing the "System CPU" to 100%? > There are times when the "User CPU" goes up to 100%, which is what he > wants, but then at times it spikes down to 20% ish and the "System CPU" > stays up around 100%. > > > Any advice is greatly appreciated. If you need me to send output, I > certainly can. Just let me know what to run. > > > Please let me know. > > > Thanks. > -- > Kamran Khan > PSSC Labs > HPC Software / Technical Engineer > > > ------------------------------------------------------------------------------ > > _______________________________________________ > Ganglia-general mailing list > Ganglia-general@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/ganglia-general > > -- ------------------------------------------------------ Martin Knoblauch email: k n o b i AT knobisoft DOT de www: http://www.knobisoft.de
------------------------------------------------------------------------------
_______________________________________________ Ganglia-general mailing list Ganglia-general@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-general