On Sun, Nov 9, 2008 at 7:54 PM, Jeff Victor <[EMAIL PROTECTED]> wrote: > It has become clear that there is a need to monitor resource > consumption of workloads in zones, and an easy method to compare > consumption to resource controls. In order to understand how a > software tool could fulfill this need, I created an OpenSolaris > project and a prototype to get started. If this sounds interesting, > you can find the project and Perl script at: > http://opensolaris.org/os/project/zonestat/ . > > If you have any comments, or suggestions for improvement, please let > me know on this e-mail list or via private e-mail.
I've had such needs for a while and have developed some tools to help my organization with that. Unfortunately, I'm not able to share that code. I am able to share suggestions... I am in a habit of: #! /usr/bin/perl -w use strict; That catches a lot of mistakes that may be masked by: close STDERR; which I never do. :) Please do not use /etc/release as a test of kernel functionality. Those that patch to an equivalent level as the update release have a similar level of functionality. A better mechanism would be to check for specific kernel patches. See http://blogs.sun.com/patch/entry/solaris_10_kernel_patchid_progression for kernel patch ID's through Update 6. The blog entry was posted just before U5 shipped, so the U6 info should be checked for accuracy. # Get amount and cap of memory locked by processes in each zone. # "kstat -p caps:*:lockedmem_zone_*" conveniently summarizes all zones for us. # open(KSTAT, "/usr/bin/kstat -p caps:*:lockedmem_zone_* |"); while (<KSTAT>) { You could just use Sun::Solaris::Kstat rather than forking another perl script. My feeling on capped memory is that if it becomes an issue and capped swap is not really close to capped memory, the over-consumptive zone has too high of a chance of causing horrible I/O problems for all zones. That is, the cap is likely to do more harm than good. This may change if swap can go onto solid state disk. I only mention this, because I don't see a purpose in capping RSS, rather I cap swap. FWIW, I tend to use the term "reserved memory" instead of "swap" because that is less confusing to most people. For CPU related stats, take a look at a discussion I started a while back: http://mail.opensolaris.org/pipermail/perf-discuss/2005-November/002048.html One project I would like to kick off sometime is doing per user, per project, and per zone microstate accounting. Presumably this data would be available through kstat. The tricky part here is to not introduce a big load on the system in the process of doing this. The above discussion and/or others in a similar vein have led me to think that collecting stats as processes exit and periodically through a kernel thread would be the way to go. This approach won't be accurate to subsecond intervals, but generally speaking you don't need better data than per minute. Such a thread should have no more impact on the system than a single user running prstat or top with a similar interval. Further, it would be good data for prstat (e.g. -a) to use. A follow-on to that would be to have a way to track usage of kernel taskq work. As more in-kernel functionality comes into existence, it becomes harder to see where the utilization is. For example, a kstat that counted the relative amount of time in zfs crypto versus zfs gzip9 operations would be helpful to the support person that is trying to answer the call "why does vmstat say my system is pegged while prstat shows no processes consuming CPU?" Sure, dtrace can get that information - but it is hard for the typical person to write and pretty expensive to run as regular monitoring. But, now I'm a bit off topic. The use of mdb rules out use by most users. I dislike tools that make users (e.g. application owners) ask me for root access. I haven't looked closely to see which, if any, of the other commands also require some elevated privileges. Most (all?) other commands have the full path set but mdb doesn't. Perhaps $ENV{PATH} = ... would be a good thing to add. I didn't have a chance to check logic closely or run it on a test system. I'll offer more feedback if needed when I get a chance to test it. It is a great start and I can't wait to see it progress. -- Mike Gerdts http://mgerdts.blogspot.com/ _______________________________________________ zones-discuss mailing list zones-discuss@opensolaris.org
