Re: [dtrace-discuss] DTrace in Perl: What probes should we have?
On Fri, Jan 11, 2008 at 06:12:57PM -0800, Roman Shaposhnik wrote: My main beef with stack helpers is that they have to be executed inside the kernel and are subject to the usual DTrace restrictions. I would like to see a possibility of offloading more work to user space in a generic fashion. For example, we have a libcollector.so facility that is capable of reconstructing stacks under the most challenging of situations (inside the function preamble, etc.) it is way more accurate in what it reports back than ustack() it is also capable of more stack unwinding than jstack(). Of course, it runs in user space. All respect to libcollector.so, but the conditions under which DTrace must gather a stack are far more constrained than those of a user-land program. The DTrace code from probe context can't do I/O or block -- it can only load and store. Even looping is a bit of a stretch. Adam -- Adam Leventhal, FishWorkshttp://blogs.sun.com/ahl
Re: [dtrace-discuss] DTrace in Perl: What probes should we have?
On Jan 12, 2008, at 12:26 PM, Adam Leventhal wrote: On Fri, Jan 11, 2008 at 06:12:57PM -0800, Roman Shaposhnik wrote: My main beef with stack helpers is that they have to be executed inside the kernel and are subject to the usual DTrace restrictions. I would like to see a possibility of offloading more work to user space in a generic fashion. For example, we have a libcollector.so facility that is capable of reconstructing stacks under the most challenging of situations (inside the function preamble, etc.) it is way more accurate in what it reports back than ustack() it is also capable of more stack unwinding than jstack(). Of course, it runs in user space. All respect to libcollector.so, but the conditions under which DTrace must gather a stack are far more constrained than those of a user-land program. The DTrace code from probe context can't do I/O or block -- it can only load and store. Even looping is a bit of a stretch. I don't disagree at all! And I'm not trying to say that *stack() should be replaced. I'm merely trying to explore whether the best of both worlds approach is doable. What we've done for Project D-Light clearly shows that for some applications it is doable. At this point I really would like to explore whether something like that could be generalized. See my earlier email to Eric. I understand his argument perfectly well. But I'm still not convinced that it makes the idea void. If convincing me is perceived as a waste of time -- that's ok. I guess I should get back to you when I earn my Solaris kernel guru degree :-) If on the other hand, I'm not explaining what I want clearly -- I can try again. Thanks, Roman.
Re: [dtrace-discuss] DTrace in Perl: What probes should we have?
excellent stuff Andy! To start with the obvious :) Due to the lack of ustack helper, I am hoping to add a stack probe - I really could do with being able to write a dtrace that shows me a perl stack trace on file system access - yesterday I spent a borish day using inotifywait and some perl tracing on linux :/ Another other thing I'm missing is the arguments to ops. Oh, and can I trace exception throws and catches? Sven Andy Armstrong wrote: As of patch 32953 dtrace support is in bleadperl (5.11.0). The probes are based on Alan Burlinson's original blog post on the subject: http://blogs.sun.com/alanbur/date/20050909 By guarding the probes with PERL_SUB_*_ENABLED the performance hit is unmeasurable. All the necessary bits already existed in the wild. I just assembled them and made the necessary changes to Perl's Configure script. The patched Perl works with the examples in DTraceToolkit. Currently we have probes on subroutine entry and return. Next I'd like to solicit input about what other probes would be useful. I have Sven Dowideit's patch that adds probes on new__sv, del__sv (creation and deletion of values) and, main__enter, main__exit (interpreter lifecycle) and load__module (require, use etc). Where else might we usefully probe? What would people find useful? -- notes -- I'm crossposting this to what I hope are a few interested lists. If anyone reading this needs a primer on dtrace this video is highly recommended: http://video.google.com/videoplay?docid=-8002801113289007228 It's 90 minutes of your life but it'll be time well spent :) If anyone would like to try bleadperl it is available here: rsync -avz --exclude .svn/ --delete \ rsync://ftp.linux.activestate.com/perl-current/ bleadperl You can configure it like this: ./Configure -de -Dusedevel -Dinc_version_list=none \ -Dprefix=$install -Dldflags=-Dman3ext=3pm \ -Duseithreads -Duseshrplib -Uversiononly \ -Dusedtrace where $install is your preferred install base. Right now ./Configure displays a harmless error about '!' being an unknown command just after the questions. There's a patch in the pipeline to fix that. -- Professional Wiki Innovation and Support Sven Dowideit - http://DistributedINFORMATION.com A WikiRing Partner http://wikiring.com
Re: [dtrace-discuss] DTrace in Perl: What probes should we have?
On Fri, 2008-01-11 at 17:32 -0800, Bryan Cantrill wrote: On Fri, Jan 11, 2008 at 04:39:25PM -0800, Roman Shaposhnik wrote: On Sat, 2008-01-12 at 11:21 +1100, Sven Dowideit wrote: excellent stuff Andy! To start with the obvious :) Due to the lack of ustack helper, I am hoping to add a stack probe Two things: first of all DTrace could really benefit from a general purpose mechanism of hooking up custom stack helpers with the rest of the system. Frankly, the way jstack is done doesn't strike me as too generic a mechanism. Do you actually understand how it's implemented? Not fully, but I've spent a great deal of time staring at jhelper.d ;-) Seriously, though -- my experience comes mostly from reading the source. And the source code for DTrace can get pretty tricky. If you have any suggestion for me as to what would be a good intro into stack helpers -- I'd appreciate that. It's actually quite general -- witness the presence of ustack helpers for both Python and PHP. (Indeed, the underlying mechanism is _so_ general that for some time we weren't sure if we had fallen prey to false generality.) I guess we are talking about two different kinds of generality here. What I have in mind is the generality that would allow DTrace to be able to offload some of the heavylifting to the userspace. I DO understand the ramifications of that and I know full well that pushing DTrace into userspace might be perceived as the deviation from the original design. To some extent it might even reduce the generality that you seem to be referring to. Second, I would like to encourage those who do DTrace integration into Perl to take a look at what JavaScript is doing. They don't have a custom stack helper either but the set of probes they offer makes stack synthesis a trivial (albeit costly) matter. While the JavaScript approach is better than nothing, it is _not_ preferable to the ustack approach Agreed. Still it is WAY better than nothing ;-) Thanks, Roman.
Re: [dtrace-discuss] DTrace in Perl: What probes should we have?
On Fri, Jan 11, 2008 at 06:23:14PM -0800, Roman Shaposhnik wrote: I guess we are talking about two different kinds of generality here. What I have in mind is the generality that would allow DTrace to be able to offload some of the heavylifting to the userspace. I DO understand the ramifications of that and I know full well that pushing DTrace into userspace might be perceived as the deviation from the original design. To some extent it might even reduce the generality that you seem to be referring to. The issue is not one of design, but of the constraints on the problem. DTrace actions must be executed in probe context. Heavy lifting can be done via post-processing in userland (symbol translation, system() actions, etc), but the data gathering must be done in arbitrary context. How would you call arbitrary userspace code from, say, an interrupt handler? - Eric -- Eric Schrock, FishWorkshttp://blogs.sun.com/eschrock
Re: [dtrace-discuss] DTrace in Perl: What probes should we have?
On Sat, Jan 12, 2008 at 02:03:42AM +, Andy Armstrong wrote: http://blogs.sun.com/levon/entry/python_and_dtrace_in_build I read that Bryan. I got the impression - maybe incorrectly - that it relies on the Python interpreter recursing when it enters a Python function - so that a walk up the C stack visits all entries in the Python stack. Did I get that wrong? Nope, that's right. Perl doesn't recurse in C when it calls a Perl subroutine so walking up the stack won't yield a Perl call chain. I always got the impression the difficulty was in walking the stack for the relevant Perl thread at all, not just the C vs. Perl stack issue. Whilst the C stack issue is a problem, it's not necessarily a major one. Presuming (and I really don't know) that it's easy to get to the top of the Perl stack from the available initial state, and it's feasible to traverse the relevant structures, in theory we could extend the ustack() implementation with a yield()-style approach: that is, the helper is called multiple times and each time yields a relevant string until done. I'm sure Bryan will jump and tell me I'm crazy. BTW this has been discussed before: http://mail.opensolaris.org/pipermail/dtrace-discuss/2007-September/004572.html regards john
Re: [dtrace-discuss] DTrace in Perl: What probes should we have?
On Sat, 2008-01-12 at 01:30 +, John Levon wrote: On Fri, Jan 11, 2008 at 04:39:25PM -0800, Roman Shaposhnik wrote: Due to the lack of ustack helper, I am hoping to add a stack probe Two things: first of all DTrace could really benefit from a general purpose mechanism of hooking up custom stack helpers with the rest of the system. Frankly, the way jstack is done doesn't strike me as too generic a mechanism. Do you have something in mind? Discalimer: I can't really claim to be an expert in DTrace, my main familiarity with it comes from working on a tool that sometimes pushes DTrace to its limits and that way I'm forced to explore how these limits could be overcome. If what I have to say below is a wrong perception of the technology I'd love to be educated by the subject experts. My main beef with stack helpers is that they have to be executed inside the kernel and are subject to the usual DTrace restrictions. I would like to see a possibility of offloading more work to user space in a generic fashion. For example, we have a libcollector.so facility that is capable of reconstructing stacks under the most challenging of situations (inside the function preamble, etc.) it is way more accurate in what it reports back than ustack() it is also capable of more stack unwinding than jstack(). Of course, it runs in user space. We do have some preliminary ideas on how such functionality can be hooked to the rest of DTrace machinery. If there's a significant portion of the DTrace community that is interested in extending DTrace functionality in a way that would make it more reliant on userspace -- perhaps it is a good thing to have on the DTrace conference agenda. Thanks, Roman.
Re: [dtrace-discuss] DTrace in Perl: What probes should we have?
On Jan 11, 2008, at 6:26 PM, Eric Schrock wrote:
On Fri, Jan 11, 2008 at 06:23:14PM -0800, Roman Shaposhnik wrote:
I guess we are talking about two different kinds of generality
here.
What I have in mind is the generality that would allow DTrace to be
able to offload some of the heavylifting to the userspace. I DO
understand the ramifications of that and I know full well that
pushing
DTrace into userspace might be perceived as the deviation from the
original design. To some extent it might even reduce the generality
that you seem to be referring to.
The issue is not one of design, but of the constraints on the problem.
If we are talking about *u*stack, than I disagree with the way you're
setting up constraints. The *u*stack is by definition a userspace
related
operation. Not only that, but calling ustack() is declared void for a
reason.
Calling it changes the output of the DTrace but has no way of affecting
the rest of the DTrace script's internal logic.
DTrace actions must be executed in probe context.
Please don't redefine the problem. We are not talking about arbitrary
actions here.
Heavy lifting can be
done via post-processing in userland (symbol translation, system()
actions, etc), but the data gathering must be done in arbitrary
context.
What data gathering? Could you, please, be more specific? What data
there is that can't be leveraged from the userspace? And again, I'm
talking specifically about ustack() here.
How would you call arbitrary userspace code from, say, an interrupt
handler?
I'm not suggesting that at all. And in fact for Project D-Light we
devised
a mechanism for getting much more accurate stacktraces from userspace
compared to what DTrace is capable of. E.g.:
self uint32_t stkIdx;
collector$1:::ustack
{
printf(0 %d %d %u\n, tid, self-stkIdx++, arg032|
(arg10x));
}
syscall::read:return
/pid == $1/
{
fname = (self-fd == 0 ) ? stdin : fnames[self-fd];
printf(1 %d %d %d %d \%s\ %d %d\n,
cpu,
tid,
timestamp,
0,
fname,
arg1,
self-stkIdx);
raise(29);
}
The awkwardness here comes from the fact that we have to:
* make libcollector available in the address space of the process
* use a raise(29) in order to poke userspace instead of
something more
DTrace specific.
Otherwise it works very nice. We get all the benefits of userspace
code:
we can match dlopen/dlclose events and such and our Java stack are
better than jstack().
Thanks,
Roman.
Re: [dtrace-discuss] DTrace in Perl: What probes should we have?
On Fri, Jan 11, 2008 at 04:39:25PM -0800, Roman Shaposhnik wrote: Due to the lack of ustack helper, I am hoping to add a stack probe Two things: first of all DTrace could really benefit from a general purpose mechanism of hooking up custom stack helpers with the rest of the system. Frankly, the way jstack is done doesn't strike me as too generic a mechanism. Do you have something in mind? regards john
Re: [dtrace-discuss] DTrace in Perl: What probes should we have?
Roman Shaposhnik wrote: On Sat, 2008-01-12 at 11:21 +1100, Sven Dowideit wrote: excellent stuff Andy! To start with the obvious :) Due to the lack of ustack helper, I am hoping to add a stack probe Two things: first of all DTrace could really benefit from a general purpose mechanism of hooking up custom stack helpers with the rest of the system. Frankly, the way jstack is done doesn't strike me as too generic a mechanism. Second, I would like to encourage those who do DTrace integration into Perl to take a look at what JavaScript is doing. They don't have a custom stack helper either but the set of probes they offer makes stack synthesis a trivial (albeit costly) matter. Thanks, Roman. Yeah, I'm pondering (naively) the idea of activating the perl debugger using isenabled. -- Professional Wiki Innovation and Support Sven Dowideit - http://DistributedINFORMATION.com A WikiRing Partner http://wikiring.com
