First, I confess that I'm pretty uncomfortable answering Oracle
support questions here. Oracle has very aggressively monetized the
operating system with the notion that its support was of tremendous
value; that Oracle has to essentially rely on the broader community
for that support speaks volumes to the actual value that Oracle is
provided its customers in this regard. Put more bluntly: let no one
question the vitality of illumos when it is we in the illumos
community who are fielding a support call about your operating
system...
Now, on to the meat here. The problem here is a bug in the script,
albeit one that is due to some subtle semantics: threads that exit
will fire sched:::on-cpu but never fire sched:::off-cpu. (That is, a
thread exits while on the CPU; it does not give up the CPU to die.)
This can, of course, be easily confirmed with DTrace itself; try this
experiment:
# dtrace -n sched:::on-cpu,sched:::off-cpu,proc:::lwp-exit'/pid ==
$target/{printf("%d: %s\n", timestamp, probename)}' -q -c
"/usr/bin/sleep 1" | sort -n
When I run this on my illumos machine, I see:
153116965615211: on-cpu
153116965637283: off-cpu
153116965876279: on-cpu
153116966272791: off-cpu
153116966908637: on-cpu
153116966979203: off-cpu
153116967125718: on-cpu
153116967215732: off-cpu
153116967377948: on-cpu
153116967447287: off-cpu
153116967573339: on-cpu
153116968106208: off-cpu
153116968166568: on-cpu
153116968181498: off-cpu
153116968347277: on-cpu
153116968367427: off-cpu
153116980796598: on-cpu
153116980810590: off-cpu
153116980969177: on-cpu
153116981580498: off-cpu
153116981746760: on-cpu
153116981761285: off-cpu
153116981909940: on-cpu
153116981931360: off-cpu
153116996876523: on-cpu
153116996890628: off-cpu
153116997047335: on-cpu
153116999245574: off-cpu
153117999387773: on-cpu
153117999472949: lwp-exit
So the script is leaking a dynamic variable whenever a thread that has
been off CPU exits -- and it's not at all surprising that dynamic
variable drops are only seen after 24 hours and only under heavy load.
To fix this, the customer should add a clause for proc:::lwp-exit
that zeroes out their thread-local variable....
- Bryan
_______________________________________________
dtrace-discuss mailing list
dtrace-discuss@opensolaris.org
