To me, utracer was always a prototype effort whose main goal was to gain familiarity with writing kernel modules and with using the utrace APIs in a module. It served that purpose well, and we can send it to bed content.
I do have many ideas about what we want in a new interface (both vague and specific, and all fungible). I've been thinking about it a lot for around three years now, so I tend to forget how many things about this I take for granted that aren't obvious to everyone else. I think what's been germinating in my head does encompass all the concerns that everyone has raised in this thread. You have been concerned primarily about the transport. That is, the means by which the userland application and the utrace-based facility exchange information and wake each other up. This is obviously the crux of what it means to have a user-level interface, and its details are a particular well-known pain point for the userland debugger implementors. But to me it is really the least interesting part of the equation. A transport is a transport. There will be some way the tracing engine gets control messages from userland and some way it delivers data to userland, with asynchronous messages both ways and perhaps also synchronous control operations initiated from userland. We are frankly quite far away from microoptimizing the transport paths being a sensible priority. What we need is some transport, any transport, that is easy to implement and easy to use, to get us off the ground. We will not be forever tied to whatever we use here, it's just the way the bits get into the guts. Fortunately, what's easy to implement is also what's easy to use. Some of Chris's impressions notwithstanding, there is a very broad consensus for an fd-based interface where the standard select/poll and/or nonblocking fd modes etc are the means of synchronization/wakeup, standard read/write (and perhaps ioctl) are fine for exchanging messages/data. On the kernel side, it is easy for a kernel module to implement a device, magic /proc or /sys file, or magicked-up socket, that can support read/write/ioctl and wake up select/poll. I would like to see us just get one going, whatever works, and table the discussion about merits of transport details for now. (It's a meaningful discussion to have at some point, but it's really not a blocker issue for getting somewhere--it's just a tangent that slows down the progress that matters at this stage.) I asked you to look into systemtap for a few reasons. Firstly, some very worthwhile discussion about utrace issues has happened with systemtap developers. Those conversations are in the context of solving systemtap problems, and often use systemtap terms or examples of systemtap scripts or pseudo-script syntax to illustrate the underlying issues. So to be able to follow everything in those threads that contributes to the bigger picture, it helps to be moderately well-versed in the fundamentals of systemtap use and script-writing going in. (In particular, the thread at http://sourceware.org/ml/systemtap/2008-q2/msg00459.html is the first stab I've made at laying the groundwork for where I think we'll be going. I'll start that particular conversation explicitly on this list shortly.) Second, I'm mostly looking at systemtap not as a user of systemtap but as an implementor seeing what support systemtap needs from below to do its job. You and I need to follow what they are trying to build up from the low level picture to the meaningul user semantics, and see all the commonalities with what we want to provide in a user-level interface. I also know a lot about what debuggers and other traditional user tracing applications want from an interface. There is a great deal of overlap in their needs, though they are indeed quite different contexts in which to address those needs. For most of what needs hard work done to make lots of cool new things possible, fast and easy, those differences are really only skin-deep. We'll serve them both with a lot of the same code, though one thing they don't share is needing us to provide a transport or a user/"wire" interface as such at all. Third, was to consider possibly using systemtap as a prototyping tool. This is a subtle idea and I know my messages get long and dense, so I won't go into what I mean by this now. I don't think it's the most important subject to get clarity on first. I have numerous more specific thoughts about interface features. Some time back, I almost got started towards my preferred directions in the "ntrace" tree, but not quite. I think I have a good sense of all the "moving parts" going into the interface/fancy-debugger-module picture. In the next day or two, I'll start a new thread here about each facet of the problem as I see it and what I've had in mind; for lack of a better name, under the heading "ntrace". Thanks, Roland
