Re: [RFC] [PATCH 1/7] User Space Breakpoint Assistance Layer (UBP)
On Thu, 2010-01-14 at 11:46 -0800, Jim Keniston wrote: discussed elsewhere. Thanks for the pointer...
Re: [RFC] [PATCH 4/7] Uprobes Implementation
On Thu, 2010-01-14 at 14:49 -0800, Jim Keniston wrote: On Thu, 2010-01-14 at 12:09 +0100, Peter Zijlstra wrote: On Mon, 2010-01-11 at 17:55 +0530, Srikar Dronamraju wrote: Uprobes Infrastructure enables user to dynamically establish probepoints in user applications and collect information by executing a handler functions when the probepoints are hit. Please refer Documentation/uprobes.txt for more details. This patch provides the core implementation of uprobes. This patch builds on utrace infrastructure. You need to follow this up with the uprobes patch for your architecture. So all this is basically some glue between what you call ubp (the real userspace breakpoint stuff) and utrace? Or does it do more? My reply in http://lkml.indiana.edu/hypermail/linux/kernel/1001.1/02483.html addresses this. Right, so all that need be done is add the multiple probe stuff to UBP and its a sane interface to use on its own, at which point I'd be inclined to call that uprobes (UBP really is an crap name). Then we can ditch the whole utrace muck as I see no reason to want to use that, whereas the ubp (given a sane name) looks interesting.
Re: [RFC] [PATCH 4/7] Uprobes Implementation
Peter Zijlstra pet...@infradead.org writes: [...] Right, so all that need be done is add the multiple probe stuff to UBP and its a sane interface to use on its own, at which point I'd be inclined to call that uprobes (UBP really is an crap name). At one point ubp+uprobes were one piece. They were separated on the suspicion that lkml would like them that way. Then we can ditch the whole utrace muck as I see no reason to want to use that, whereas the ubp (given a sane name) looks interesting. Assuming you meant what you write, perhaps you misunderstand the layering relationship of these pieces. utrace underlies uprobes and other process manipulation functionality, present and future. - FChE
Re: [RFC] [PATCH 4/7] Uprobes Implementation
On Fri, 2010-01-15 at 04:26 -0500, Frank Ch. Eigler wrote: Peter Zijlstra pet...@infradead.org writes: [...] Right, so all that need be done is add the multiple probe stuff to UBP and its a sane interface to use on its own, at which point I'd be inclined to call that uprobes (UBP really is an crap name). At one point ubp+uprobes were one piece. They were separated on the suspicion that lkml would like them that way. Right, good thinking, that way we can use ubp without having to use utrace ;-) Then we can ditch the whole utrace muck as I see no reason to want to use that, whereas the ubp (given a sane name) looks interesting. Assuming you meant what you write, perhaps you misunderstand the layering relationship of these pieces. utrace underlies uprobes and other process manipulation functionality, present and future. Why, utrace doesn't at all look to bring a fundamental contribution to all that. If there's a proper kernel interface to install probes on userspace code (ubp seems to be mostly that) then I can use perf/ftrace to do the rest of the state management, no need to use utrace there. You can hardly force me to use utrace there, can you?
Re: [RFC] [PATCH 1/7] User Space Breakpoint Assistance Layer (UBP)
On Fri, Jan 15, 2010 at 10:03:48AM +0100, Peter Zijlstra wrote: On Thu, 2010-01-14 at 11:46 -0800, Jim Keniston wrote: discussed elsewhere. Thanks for the pointer... :-) Peter, I think Jim was referring to http://sources.redhat.com/ml/systemtap/2007-q1/msg00571.html Ananth
Re: [RFC] [PATCH 1/7] User Space Breakpoint Assistance Layer (UBP)
On Fri, 2010-01-15 at 15:08 +0530, Ananth N Mavinakayanahalli wrote: On Fri, Jan 15, 2010 at 10:03:48AM +0100, Peter Zijlstra wrote: On Thu, 2010-01-14 at 11:46 -0800, Jim Keniston wrote: discussed elsewhere. Thanks for the pointer... :-) Peter, I think Jim was referring to http://sources.redhat.com/ml/systemtap/2007-q1/msg00571.html That's a 2007 email from some obscure list... that's hardly something that can be referenced to without link. As previously stated, I think poking at a process's address space is an utter no-go.
Re: [RFC] [PATCH 1/7] User Space Breakpoint Assistance Layer (UBP)
On Fri, 2010-01-15 at 15:40 +0530, Ananth N Mavinakayanahalli wrote: Ideas? emulate the one instruction?
Re: [RFC] [PATCH 1/7] User Space Breakpoint Assistance Layer (UBP)
On Fri, Jan 15, 2010 at 11:13:32AM +0100, Peter Zijlstra wrote: On Fri, 2010-01-15 at 15:40 +0530, Ananth N Mavinakayanahalli wrote: Ideas? emulate the one instruction? In kernel? Generically? Don't think its that easy for userspace -- you have the full gamut of instructions to emulate (fp, vector, etc); further, the instruction could itself cause a page fault and the like.
Re: [RFC] [PATCH 4/7] Uprobes Implementation
Hi Peter, My reply in http://lkml.indiana.edu/hypermail/linux/kernel/1001.1/02483.html addresses this. Right, so all that need be done is add the multiple probe stuff to UBP and its a sane interface to use on its own, at which point I'd be inclined to call that uprobes (UBP really is an crap name). I am fine with renaming ubp to a suggested name. The reason for splitting uprobes to two layers was to allow others (currently none) to reuse the current ubp layer. It was felt that there could be multiple clients for ubp who could co-exist. However ubp alone is not enough to provide the userspace tracing. Currently it wouldn't understand synchronization between different threads of a process, process life time issues, context in which the handler has to be run. As pointed out by Jim earlier, we have segregrated that layer which takes care of the above issues into the uprobes layer. For example, while inserting a breakpoint, one of the threads of a process could be running at the same place where we are trying to place a breakpoint. Or there could be two threads that could be racing to insert/delete a breakpoint. These synchronization issues are all handled by the Uprobes layer. Uprobes layer would need to be notified of process life-time events like fork/clone/exec/exit. It also needs to know - when a breakpoint is hit - stop and resume a thread. Uprobes layer uses utrace to be notified of the process life time events and the signal handling part. -- Thanks and Regards Srikar Then we can ditch the whole utrace muck as I see no reason to want to use that, whereas the ubp (given a sane name) looks interesting.
Re: [RFC] [PATCH 4/7] Uprobes Implementation
On Fri, 2010-01-15 at 15:56 +0530, Srikar Dronamraju wrote: Hi Peter, Or there could be two threads that could be racing to insert/delete a breakpoint. These synchronization issues are all handled by the Uprobes layer. Shouldn't be hard to put that in the ubp layer, right? Uprobes layer would need to be notified of process life-time events like fork/clone/exec/exit. No so much the process lifetimes as the vma life times are interesting, placing a hook in the vm code to track that isn't too hard, It also needs to know - when a breakpoint is hit - stop and resume a thread. A simple hook in the trap code is done quickly enough, and no reason to stop the thread, its not going anywhere when it traps.
Re: [RFC] [PATCH 1/7] User Space Breakpoint Assistance Layer (UBP)
On Fri, 2010-01-15 at 15:52 +0530, Ananth N Mavinakayanahalli wrote: On Fri, Jan 15, 2010 at 11:13:32AM +0100, Peter Zijlstra wrote: On Fri, 2010-01-15 at 15:40 +0530, Ananth N Mavinakayanahalli wrote: Ideas? emulate the one instruction? In kernel? Generically? Don't think its that easy for userspace -- you have the full gamut of instructions to emulate (fp, vector, etc); further, Can't you jit a piece of code that wraps the one instruction, save the full cpu state, set the userspace segments, have it load pt_regs (except for the IP) execute the one ins, save the results, restore the full state? Then replace pt_regs with the saved result and advance the stored IP by the length of that one instruction and return to userspace? All you need to take care of are the priv insns, but doesn't something like kvm already have code to deal with that? the instruction could itself cause a page fault and the like. Faults aren't a problem, we take faults from kernel space all the time.
Re: [RFC] [PATCH 4/7] Uprobes Implementation
On Fri, Jan 15, 2010 at 11:33:27AM +0100, Peter Zijlstra wrote: On Fri, 2010-01-15 at 15:56 +0530, Srikar Dronamraju wrote: Hi Peter, Or there could be two threads that could be racing to insert/delete a breakpoint. These synchronization issues are all handled by the Uprobes layer. Shouldn't be hard to put that in the ubp layer, right? Uprobes layer would need to be notified of process life-time events like fork/clone/exec/exit. No so much the process lifetimes as the vma life times are interesting, placing a hook in the vm code to track that isn't too hard, I think similar hooks were given thumbs down in the previous incarnation of uprobes (which was implemented without utrace). http://lkml.indiana.edu/hypermail/linux/kernel/0603.2/1254.html Thanks Maneesh -- Maneesh Soni Linux Technology Center IBM India Systems and Technology Lab, Bangalore, India.
Re: [RFC] [PATCH 3/7] Execution out of line (XOL)
* Peter Zijlstra pet...@infradead.org [2010-01-15 10:07:35]: On Thu, 2010-01-14 at 14:43 -0800, Jim Keniston wrote: Yeah, there's not a lot of context there. I hope it will make more sense if you read section 1.1 of Documentation/uprobes.txt (patch #6). Or look at get_insn_slot() in kprobes, and understand that we're trying to do something similar in uprobes, where the instruction copies have to reside in the user address space of the probed process. That's not the point, changelogs shoulnd not be this cryptic. They should be stand alone and descriptive of what, why and how. If you can't be bothered writing such for something you want reviewed for inclusion then I might not be bothered looking at them at all. Okay shall add to the Changelog with the information providing the context for this patch.
Re: [RFC] [PATCH 4/7] Uprobes Implementation
On Fri, 2010-01-15 at 16:35 +0530, Maneesh Soni wrote: On Fri, Jan 15, 2010 at 11:33:27AM +0100, Peter Zijlstra wrote: On Fri, 2010-01-15 at 15:56 +0530, Srikar Dronamraju wrote: Hi Peter, Or there could be two threads that could be racing to insert/delete a breakpoint. These synchronization issues are all handled by the Uprobes layer. Shouldn't be hard to put that in the ubp layer, right? Uprobes layer would need to be notified of process life-time events like fork/clone/exec/exit. No so much the process lifetimes as the vma life times are interesting, placing a hook in the vm code to track that isn't too hard, I think similar hooks were given thumbs down in the previous incarnation of uprobes (which was implemented without utrace). http://lkml.indiana.edu/hypermail/linux/kernel/0603.2/1254.html I wasn't at all proposing to mess with a_ops, nor do you really need to, I was more thinking of adding a callback like perf_event_mmap() and a corresponding unmap(), that way you can track mapping life-times and add/remove probes accordingly. Adding the probe uses the fact that (most) executable mappings are MAP_PRIVATE and CoWs a private copy of the page with the modified ins, right? What does it do for MAP_SHARED|MAP_EXECUTABLE sections -- simply fail to add the probe?
Re: [RFC] [PATCH 4/7] Uprobes Implementation
On Fri, 2010-01-15 at 12:12 +0100, Peter Zijlstra wrote: On Fri, 2010-01-15 at 16:35 +0530, Maneesh Soni wrote: On Fri, Jan 15, 2010 at 11:33:27AM +0100, Peter Zijlstra wrote: On Fri, 2010-01-15 at 15:56 +0530, Srikar Dronamraju wrote: Hi Peter, Or there could be two threads that could be racing to insert/delete a breakpoint. These synchronization issues are all handled by the Uprobes layer. Shouldn't be hard to put that in the ubp layer, right? Uprobes layer would need to be notified of process life-time events like fork/clone/exec/exit. No so much the process lifetimes as the vma life times are interesting, placing a hook in the vm code to track that isn't too hard, I think similar hooks were given thumbs down in the previous incarnation of uprobes (which was implemented without utrace). http://lkml.indiana.edu/hypermail/linux/kernel/0603.2/1254.html I wasn't at all proposing to mess with a_ops, nor do you really need to, I was more thinking of adding a callback like perf_event_mmap() and a corresponding unmap(), that way you can track mapping life-times and add/remove probes accordingly. Adding the probe uses the fact that (most) executable mappings are MAP_PRIVATE and CoWs a private copy of the page with the modified ins, right? Does it clean up the CoW'ed page on removing the probe? Does that account for userspace having made other changes in between installing and removing the probe (for PROT_WRITE mappings obviously)?
Re: [RFC] [PATCH 4/7] Uprobes Implementation
* Peter Zijlstra pet...@infradead.org [2010-01-15 11:33:27]: Uprobes layer would need to be notified of process life-time events like fork/clone/exec/exit. No so much the process lifetimes as the vma life times are interesting, placing a hook in the vm code to track that isn't too hard, It also needs to know - when a breakpoint is hit - stop and resume a thread. A simple hook in the trap code is done quickly enough, and no reason to stop the thread, its not going anywhere when it traps. Some of the threads could be executing in the vicinity of the breakpoint when it is getting inserted or deleted. Wont we need to stop/quiesce those threads?
Re: [RFC] [PATCH 4/7] Uprobes Implementation
Hi - Then we can ditch the whole utrace muck as I see no reason to want to use that, whereas the ubp (given a sane name) looks interesting. Assuming you meant what you write, perhaps you misunderstand the layering relationship of these pieces. utrace underlies uprobes and other process manipulation functionality, present and future. Why, utrace doesn't at all look to bring a fundamental contribution to all that. If there's a proper kernel interface to install probes on userspace code (ubp seems to be mostly that) then I can use perf/ftrace to do the rest of the state management, no need to use utrace there. You can hardly force me to use utrace there, can you? utrace is not a form of punishment inflicted upon the undeserving. It is a service layer that uprobes et alii are built upon. You as a potential uprobes client need not also talk directly to it, if you wish to reimplement task-finder-like services some other way. - FChE
Re: [RFC] [PATCH 4/7] Uprobes Implementation
On Fri, 2010-01-15 at 19:50 +0530, Srikar Dronamraju wrote: Srikar seemed to suggest it needed stop/resume. If process traps, We dont need to stop/resume other threads. uprobes needs threads to quiesce when inserting/deleting the breakpoint. Right, I don't think we need to at all. See the CoW thing from previous emails.
Re: [RFC] [PATCH 4/7] Uprobes Implementation
On Fri, 2010-01-15 at 09:22 -0500, Frank Ch. Eigler wrote: Hi - Well, I'm not in a position to argue line by line about the necessity or the cost of utrace low level guts, but this may represent the most practical engineering balance between functionality / modularity / undesirably intrusive modifications. How intrusive and non-modular is installing a DIE_INT3 notifier? I'm not sure about all the reasons pro/con, but it looks like installing such a systemwide hook would force every userspace breakpoint or kprobe event machine wide to pass through the hypothetical uprobes layer, whether or not applicable to a current task. Well, we'll have to pass through the global die notifier anyway, but a quick per task filter sounds like a good idea, we can do that by keeping a per-task count of the number of uprobes in use. Then the uprobe code can avoid the lookup if there are no task users and no global users. The advantage of this construct is that is easily allows for global users, whereas a utrace based one doesn't.
Re: [RFC] [PATCH 3/7] Execution out of line (XOL)
On Fri, 2010-01-15 at 10:07 +0100, Peter Zijlstra wrote: On Thu, 2010-01-14 at 14:43 -0800, Jim Keniston wrote: Yeah, there's not a lot of context there. I hope it will make more sense if you read section 1.1 of Documentation/uprobes.txt (patch #6). Or look at get_insn_slot() in kprobes, and understand that we're trying to do something similar in uprobes, where the instruction copies have to reside in the user address space of the probed process. That's not the point, changelogs shoulnd not be this cryptic. They should be stand alone and descriptive of what, why and how. Point taken. If you can't be bothered writing such for something you want reviewed for inclusion then I might not be bothered looking at them at all. We appreciate your persistence wrt this patch set. :-} Jim
Re: [RFC] [PATCH 1/7] User Space Breakpoint Assistance Layer (UBP)
On Fri, 2010-01-15 at 10:02 +0100, Peter Zijlstra wrote: On Thu, 2010-01-14 at 11:46 -0800, Jim Keniston wrote: +Instruction copies to be single-stepped are stored in a per-process +single-step out of line (XOL) area, which is a little VM area +created by Uprobes in each probed process's address space. I think tinkering with the probed process's address space is a no-no. Have you ran this by the linux mm folks? Sort of. Back in 2007 (!), we were getting ready to post uprobes (which was then essentially uprobes+xol+upb) to LKML, pondering XOL alternatives and waiting for utrace to get pulled back into the -mm tree. (It turned out to be a long wait.) I emailed Andrew Morton, inquiring about the prospects for utrace and giving him a preview of utrace-based uprobes. He expressed openness to the idea of allocating a piece of the user address space for the XOL area, a la the vdso page. With advice and review from Dave Hansen, we implemented an XOL page, set up for every process (probed or not) along the same lines as the vdso page. About that time, Roland McGrath suggested using do_mmap_pgoff() to create a separate vma on demand. This was the seed of the current implementation. It had the advantages of being architecture-independent, affecting only probed processes, and allowing the allocation of more XOL slots. (Uprobes can make do with a fixed number of XOL slots -- allowing one probepoint to steal another's slot -- but it isn't pretty.) As I recall, Dave preferred the other idea (1 XOL page for every process, probed or not) -- mostly because he didn't like the idea of a new vma popping into existence when the process gets probed -- but was OK with us going ahead with Roland's idea. (I'm not a VM guy; pardon any imprecision in my language.) Jim I'd be inclined to NAK this straight out.
Re: [RFC] [PATCH 1/7] User Space Breakpoint Assistance Layer (UBP)
On Fri, 2010-01-15 at 13:07 -0800, Jim Keniston wrote: On Fri, 2010-01-15 at 10:02 +0100, Peter Zijlstra wrote: On Thu, 2010-01-14 at 11:46 -0800, Jim Keniston wrote: +Instruction copies to be single-stepped are stored in a per-process +single-step out of line (XOL) area, which is a little VM area +created by Uprobes in each probed process's address space. I think tinkering with the probed process's address space is a no-no. Have you ran this by the linux mm folks? Sort of. Back in 2007 (!), we were getting ready to post uprobes (which was then essentially uprobes+xol+upb) to LKML, pondering XOL alternatives and waiting for utrace to get pulled back into the -mm tree. (It turned out to be a long wait.) I emailed Andrew Morton, inquiring about the prospects for utrace and giving him a preview of utrace-based uprobes. He expressed openness to the idea of allocating a piece of the user address space for the XOL area, a la the vdso page. With advice and review from Dave Hansen, we implemented an XOL page, set up for every process (probed or not) along the same lines as the vdso page. About that time, Roland McGrath suggested using do_mmap_pgoff() to create a separate vma on demand. This was the seed of the current implementation. It had the advantages of being architecture-independent, affecting only probed processes, and allowing the allocation of more XOL slots. (Uprobes can make do with a fixed number of XOL slots -- allowing one probepoint to steal another's slot -- but it isn't pretty.) As I recall, Dave preferred the other idea (1 XOL page for every process, probed or not) -- mostly because he didn't like the idea of a new vma popping into existence when the process gets probed -- but was OK with us going ahead with Roland's idea. Well, I think its all very gross, I would really like people to try and 'emulate' or plain execute those original instructions from kernel space. As to the privileged instructions, I think qemu/kvm like projects should have pretty much all of that covered. Nor do I think we need utrace at all to make user space probes useful. Even stronger, I think the focus on utrace made you get some fundamentals wrong. Its not mainly about task state, but like said, its about text mappings, which is something utrace knows nothing about. That is not to say you cannot build a useful interface from uprobes and utrace, but its not at all required or natural.
Re: [RFC] [PATCH 4/7] Uprobes Implementation
On Fri, 2010-01-15 at 12:18 +0100, Peter Zijlstra wrote: On Fri, 2010-01-15 at 12:12 +0100, Peter Zijlstra wrote: ... Adding the probe uses the fact that (most) executable mappings are MAP_PRIVATE and CoWs a private copy of the page with the modified ins, right? We've just used access_process_vm() to insert the breakpoint instruction. (If there are situations where that's not appropriate, please advise.) Does it clean up the CoW'ed page on removing the probe? If I understand your question, the answer is no. We make no attempt to reclaim COW'ed pages, even after all the probes have been removed. In fact, once the first probe is hit and the XOL vma is created, the XOL vma hangs around for the life of the process. Does that account for userspace having made other changes in between installing and removing the probe (for PROT_WRITE mappings obviously)? We don't attempt the aforementioned cleanup, so I think the answer is N/A. Jim
Re: [RFC] [PATCH 4/7] Uprobes Implementation
On Fri, 2010-01-15 at 19:50 +0530, Srikar Dronamraju wrote: Furthermore it requires stopping and resuming tasks and nonsense like that, that's unwanted in many cases, just run stuff from the trap site and you're done. I don't know what you mean exactly. A trap already stopped task. utrace merely allows various clients to inspect/manipulate the state of the task at that moment. It does not add any context switches or spurious stop/resumue operations. Srikar seemed to suggest it needed stop/resume. If process traps, We dont need to stop/resume other threads. uprobes needs threads to quiesce when inserting/deleting the breakpoint. Years ago, we had pre-utrace versions of uprobes where the uprobes breakpoint-handler code was dispatched from the die_notifier, before the int3 turned into a SIGTRAP. I believe that's what Peter is recommending. On my old Pentium M... - a pre-utrace uprobe hit cost about 1 usec; - a utrace-based uprobe hit cost about 3 usec; - and an unboosted kprobe hit cost 0.57 usec. So yeah, learning about the int3 via utrace after the SIGTRAP gets created adds some overhead to uprobes. But as previously discussed in this thread -- e.g., http://lkml.indiana.edu/hypermail/linux/kernel/1001.1/02969.html -- there are ways to avoid the 2nd (single-step) trap, which should cut overhead in half. Jim
Re: [RFC] [PATCH 4/7] Uprobes Implementation
On Fri, 2010-01-15 at 12:12 +0100, Peter Zijlstra wrote: ... Adding the probe uses the fact that (most) executable mappings are MAP_PRIVATE and CoWs a private copy of the page with the modified ins, right? What does it do for MAP_SHARED|MAP_EXECUTABLE sections -- simply fail to add the probe? If the vma containing the instruction to be probed has the VM_EXEC flag set (and it's not in the XOL area) we go ahead and try to probe it. I'm not familar with the implications of MAP_SHARED|MAP_EXECUTABLE -- how you would get such a combo, or what access_process_vm() would do with it. Jim
