Hey Peter,
Indeed, the vmstat output is very unhelpful in this context...
There are a bunch of things you can do that's really really simple though.
On the global zone, kick off a prstat -s size and a prstat -s rss
get them logging to files, and interrogate them when things go belly up.
Now: If the memory grab happens in a few seconds, that might be
problematic, as the default interval of prstat is 5 seconds. It might
happen and disappear before you even get to see it. If this is the case,
you could (at the expense of some CPU) have prstat's interval set to 1s.
But, I'd suspect this will be enough to help you see it.
Other than this, you have some other options using DTrace, but it's kind
of like swatting a fly with a bus... And it'll be very much non-trivial.
;) Think per brk() operation tracing, buffer sizes and all that sort of
stuff... Nasty.
Other things to consider doing (just in case the memory utilisation is
NOT in userspace) is to kick off something like:
while sleep 5
do
echo ::memstat
done | mdb -k
or something to that effect, so you can see if it is indeed kernel (and
if you have a late enough kernel patch, if it's ZFS's ARC.
Hope this helps!
Nathan.
Peter Keating wrote:
> All,
> I am trying to isolate a performance issue and was wondering if
> anyone could help. Below is some "vmstat -p" output from a M4000
> (64GB, 4x Sparc64 VII, S10_u7 - 139555-08) running 5 local zones (each
> running Oracle, one with grid control and one with SAP). It appears
> that at various times, something suddenly increases it's real memory
> requirements. When this occurs the system is somewhat unresponsive, so
> isolating the process(es) is problematic, it's often over before we hear
> about it.
>
> I was thinking that dtrace(or something else) might be able to identify
> process.
>
> Any ideas?
> Peter
>
>
>
> #date time interval swap free re mf fr de sr
> epi epo epf api apo apf fpi fpo fpf
> 2010/03/04 14:08 60 17794448 1577664 1071 4927 1 0 0 0 0 0 134 0 0 0 1 1
> 2010/03/04 14:09 60 18106176 1909744 1988 11769 2 0 0 0 0 0 204 0 0 0 2 2
> 2010/03/04 14:10 60 17775872 1462440 1686 9215 1 0 0 0 0 0 46 0 0 0 1 1
> 2010/03/04 14:11 60 17764936 1439960 1312 6824 0 0 0 0 0 0 39 0 0 0 0 0
> 2010/03/04 14:12 60 18185640 1816304 1396 8137 1 0 0 0 0 0 332 0 0 0 1 1
> 2010/03/04 14:13 60 17528096 1197368 1336 7575 0 3728 0 0 0 0 106 0 0 1 0 0
> 2010/03/04 14:14 60 17622328 1280640 1728 13445 4347 1026760 8174 0 0 1
> 167 3571 3840 0 5 506
> 2010/03/04 14:15 60 16889408 1282600 1427 9631 3393 13032 4778 1 0 1 225
> 2948 3128 0 0 264
> 2010/03/04 14:16 60 17258256 1445952 1038 5746 1 0 0 1 0 0 36 0 0 0 1 1
> #date time interval swap free re mf fr
> de sr epi epo epf api apo apf fpi fpo fpf
> 2010/03/04 14:17 60 17065032 1330008 673 4888 14682 1026760 63296 0 0
> 48 222 9858 11364 247 6 3270
> 2010/03/04 14:18 60 15389112 1688640 324 1659 30164 2424 8792 2 0
> 12 20 29690 29415 1 2 737
> 2010/03/04 14:19 60 14553768 1703528 275 1432 1065 0 0 8 0 0 104 1380
> 1065 2 0 0
> 2010/03/04 14:20 60 16028776 3364696 2439 7346 11422 0 0 27 0 0 863
> 13294 11421 15 1 1
> 2010/03/04 14:21 60 16672000 3968296 1944 6821 0 0 0 14 0 0 2081 0 0 3 0 0
> 2010/03/04 14:22 60 16201328 3181632 1746 9441 1 0 0 1 0 0 1540 0 0 5 1 1
> 2010/03/04 14:23 60 15626224 2233352 1332 6039 0 0 0 1 0 0 301 0 0 2 0 0
> 2010/03/04 14:24 60 15253440 1808192 1398 5951 0 0 0 24 0 0 269 0 0 12 0 0
> 2010/03/04 14:25 60 15241024 1777448 1472 5677 0 0 0 3 0 0 142 0 0 2 0 0
> 2010/03/04 14:26 60 15280624 1785088 1102 5276 0 0 0 2 0 0 221 0 0 0 0 0
> 2010/03/04 14:27 60 15208128 1683320 1483 6312 0 0 0 0 0 0 172 0 0 0 0 0
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> ug-msosug mailing list
> ug-msosug at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/ug-msosug
