On Niagara one immediately notices, how misleading mpstat output can be: 4 strands share a core. mpstat shows every strand as behaving individually, but the 4 strands share the same pool of core cycles.
Two extreme situations: Take a simple program with an endless loop incrementing a counter. mpstat on an N CPU niagara will show one CPU 100% busy and 4*N-1 CPUs idle. But cpustat will tell you, that in fact one core is 100% busy, because it is executing an instruction on every clock cycle. Now start N of these programs, then in reality the CPU ressources are used 100%, but mpstat shows N CPUs busy and 3*N CPUs idle, so a total busyness of 25%. It's also possible to be misled in the other direction. Start 3*N programs making heavy use of floating point or having a huge working set (memory). mpstat will show you 3*N CPUs busy, so 75% total usage. In reality 90% or more of your core cycles could still be used by an additional better fitting workload, so from that point of view your system load could also be called 10% busy. I know, that there is no easy solution, but since admins are used to working with mpstat and don't really care about cpustat, it might be possible to add a switch, that shows instruction count percentage per core. -- This messages posted from opensolaris.org
