Re: [RFC] [PATCH 1/7] User Space Breakpoint Assistance Layer (UBP)
On 01/19/2010 12:15 AM, Jim Keniston wrote: I don't like the idea but if the performance benefits are real (are they?), Based on what seems to be the closest thing to an apples-to-apples comparison -- counting the number of calls to a specified function -- uprobes is 6-7 times faster than the ptrace-based equivalent, ltrace -c. And of course, uprobes provides much, much more flexibility, appears to scale better, and works with multithreaded apps. Likewise, FWIW, utrace is more than 10x faster than strace -c in counting system calls. This is still with a kernel entry, yes? Do you have plans for a variant that's completely in userspace? -- error compiling committee.c: too many arguments to function
Re: [RFC] [PATCH 1/7] User Space Breakpoint Assistance Layer (UBP)
On Tue, 2010-01-19 at 10:07 +0200, Avi Kivity wrote: On 01/19/2010 12:15 AM, Jim Keniston wrote: I don't like the idea but if the performance benefits are real (are they?), Based on what seems to be the closest thing to an apples-to-apples comparison -- counting the number of calls to a specified function -- uprobes is 6-7 times faster than the ptrace-based equivalent, ltrace -c. And of course, uprobes provides much, much more flexibility, appears to scale better, and works with multithreaded apps. Likewise, FWIW, utrace is more than 10x faster than strace -c in counting system calls. This is still with a kernel entry, yes? Yes, this involves setting a breakpoint and trapping into the kernel when it's hit. The 6-7x figure is with the current 2-trap approach (breakpoint, single-step). Boosting could presumably make that more like 12-14x. Do you have plans for a variant that's completely in userspace? I don't know of any such plans, but I'd be interested to read more of your thoughts here. As I understand it, you've suggested replacing the probed instruction with a jump into an instrumentation vma (the XOL area, or something similar). Masami has demonstrated -- through his djprobes enhancement to kprobes -- that this can be done for many x86 instructions. What does the code in the jumped-to vma do? Is the instrumentation code that corresponds to the uprobe handlers encoded in an ad hoc .so? BTW, when some people say completely in userspace, they mean something like ptrace, where the kernel is still heavily involved but the instrumentation code runs in user space. The ubp layer is intended to support that model as well. In our various implementations of the XOL vma/address area, however, the XOL area is either created on exec or created/expanded only by the probed process. Jim
Re: [RFC] [PATCH 1/7] User Space Breakpoint Assistance Layer (UBP)
On Tue, Jan 19, 2010 at 09:47:45AM -0800, Jim Keniston wrote: Do you have plans for a variant that's completely in userspace? I don't know of any such plans, but I'd be interested to read more of your thoughts here. As I understand it, you've suggested replacing the probed instruction with a jump into an instrumentation vma (the XOL area, or something similar). Masami has demonstrated -- through his djprobes enhancement to kprobes -- that this can be done for many x86 instructions. What does the code in the jumped-to vma do? Is the instrumentation code that corresponds to the uprobe handlers encoded in an ad hoc .so? Once the instrumentation is requested by a process that is not the instrumented one, this looks impossible to set a uprobe without a minimal voluntary collaboration from the instrumented process (events sent through IPC or whatever). So that looks too limited, this is not anymore a true dynamic uprobe.
linux-next: add utrace tree
Hi - Having been reviewed a couple of times, and we hope being a good candidate for merging next time, please start pulling git://git.kernel.org/pub/scm/linux/kernel/git/frob/linux-2.6-utrace.git branch master This repo contains frequent merges from Linus' tree. If you'd prefer a cleaner rebase-based branch to pull from, we can make one of those too. Thanks! - FChE
Re: linux-next: add utrace tree
Hi Frank, On Tue, 19 Jan 2010 16:16:46 -0500 Frank Ch. Eigler f...@redhat.com wrote: Having been reviewed a couple of times, and we hope being a good candidate for merging next time, please start pulling git://git.kernel.org/pub/scm/linux/kernel/git/frob/linux-2.6-utrace.git branch master I have added this from today with you and utrace-devel as the contacts. I have cc'd the wider community on this email so that people are aware that this has been included. This repo contains frequent merges from Linus' tree. If you'd prefer a cleaner rebase-based branch to pull from, we can make one of those too. For now it is OK, but you might like to ask Linus if he would like it cleaned up before submission since it seems to have history right back to 2.6.29 and (as you say) lots of merges with his tree. You should also add a commit with an entry in MAINTAINERS. [Standard boilerplate] Thanks for adding your subsystem tree as a participant of linux-next. As you may know, this is not a judgment of your code. The purpose of linux-next is for integration testing and to lower the impact of conflicts between subsystems in the next merge window. You will need to ensure that the patches/commits in your tree/series have been: * submitted under GPL v2 (or later) and include the Contributor's Signed-off-by, * posted to the relevant mailing list, * reviewed by you (or another maintainer of your subsystem tree), * successfully unit tested, and * destined for the current or next Linux merge window. Basically, this should be just what you would send to Linus (or ask him to fetch). It is allowed to be rebased if you deem it necessary. -- Cheers, Stephen Rothwell s...@canb.auug.org.au Legal Stuff: By participating in linux-next, your subsystem tree contributions are public and will be included in the linux-next trees. You may be sent e-mail messages indicating errors or other issues when the patches/commits from your subsystem tree are merged and tested in linux-next. These messages may also be cross-posted to the linux-next mailing list, the linux-kernel mailing list, etc. The linux-next tree project and IBM (my employer) make no warranties regarding the linux-next project, the testing procedures, the results, the e-mails, etc. If you don't agree to these ground rules, let me know and I'll remove your tree from participation in linux-next. pgpLabqIDVtHS.pgp Description: PGP signature
linux-next: manual merge of the utrace tree with the fsnotify tree
Hi all, Today's linux-next merge of the utrace tree got a conflict in kernel/Makefile between commit 9878914352df8ccfbad1307d51ca05706d50cae4 (Audit: split audit watch Kconfig) from the fsnotify tree and commit f357a74067bc548772166a4817d5f2c32005a449 (utrace core) from the utrace tree. Just context changes. I fixed it up (see below) and can carry the fix as necessary. -- Cheers, Stephen Rothwells...@canb.auug.org.au diff --cc kernel/Makefile index 702260a,8a0185e..000 --- a/kernel/Makefile +++ b/kernel/Makefile @@@ -69,13 -69,13 +69,14 @@@ obj-$(CONFIG_IKCONFIG) += configs. obj-$(CONFIG_RESOURCE_COUNTERS) += res_counter.o obj-$(CONFIG_STOP_MACHINE) += stop_machine.o obj-$(CONFIG_KPROBES_SANITY_TEST) += test_kprobes.o + obj-$(CONFIG_UTRACE) += utrace.o -obj-$(CONFIG_AUDIT) += audit.o auditfilter.o audit_watch.o +obj-$(CONFIG_AUDIT) += audit.o auditfilter.o obj-$(CONFIG_AUDITSYSCALL) += auditsc.o -obj-$(CONFIG_GCOV_KERNEL) += gcov/ +obj-$(CONFIG_AUDIT_WATCH) += audit_watch.o obj-$(CONFIG_AUDIT_TREE) += audit_tree.o +obj-$(CONFIG_GCOV_KERNEL) += gcov/ obj-$(CONFIG_KPROBES) += kprobes.o -obj-$(CONFIG_KGDB) += kgdb.o +obj-$(CONFIG_KGDB) += debug/ obj-$(CONFIG_DETECT_SOFTLOCKUP) += softlockup.o obj-$(CONFIG_DETECT_HUNG_TASK) += hung_task.o obj-$(CONFIG_GENERIC_HARDIRQS) += irq/
Re: linux-next: add utrace tree
On Wed, Jan 20, 2010 at 06:49:50AM +0100, Ingo Molnar wrote: Ingo, Note, i'm not yet convinced that this (and the rest: uprobes and systemtap, etc.) can go uptream in its present form. Agreed, uprobes is still not upstream ready -- it was an RFC. We are working through the comments there to get it ready for merger. IMHO the far more important thing to address beyond formalities and workflow cleanliness are the (many) technical observations and objections offered by Peter Zijstra on lkml. Not just the git history but also the abstractions and concepts are messy and should be reworked IMO, and also good and working perf events integration should be achieved, etc. I think Oleg addressed most of Peter's concerns on utrace when the ptrace/utrace patchset was reposted. Perf integration with uprobes will be done and discussions have started with Masami and Frederic. There are a couple of fundamental technical aspects (XOL vma vs. emulation; breakpoint insertion through CoW and not through quiesce) that need resolution. The fact that there's a well established upstream workflow for instrumentation patches, which is being routed around by the utrace/uprobes/systemtap code here is not a good sign in terms of reaching a good upstream solution. Lets hope it works out well though. Agreed. On the other hand, having ptrace/utrace in the -next tree will give it a lot more testing, while any outstanding technical issues are being addressed. Stephen, To exercise ptrace/utrace, it would be very useful if you pulled in git://git.kernel.org/pub/scm/linux/kernel/git/frob/linux-2.6-utrace.git branch utrace-ptrace instead of 'master'. Thanks, Ananth
Re: [RFC] [PATCH 1/7] User Space Breakpoint Assistance Layer (UBP)
* Frederic Weisbecker fweis...@gmail.com [2010-01-19 19:06:12]: On Tue, Jan 19, 2010 at 09:47:45AM -0800, Jim Keniston wrote: What does the code in the jumped-to vma do? Is the instrumentation code that corresponds to the uprobe handlers encoded in an ad hoc .so? Once the instrumentation is requested by a process that is not the instrumented one, this looks impossible to set a uprobe without a minimal voluntary collaboration from the instrumented process (events sent through IPC or whatever). So that looks too limited, this is not anymore a true dynamic uprobe. I dont see a case where the thread being debugged refuses to place a probe unless the process is exiting. The traced process doesnt decide if it wants to be probed or not. There could be a slight delay from the time the tracer requested to the time the probe is placed. But this delay in only affecting the tracer and the tracee. This is in contract to say stop_machine where the threads of other applications are also affected.
Re: linux-next: add utrace tree
Hi Frank, On Wed, 20 Jan 2010 07:28:34 +0100 Ingo Molnar mi...@elte.hu wrote: Including experimental code that is RFC and which is not certain to go upstream is certainly not the purpose of linux-next though. Ingo is correct in what he says here. See the boilerplate: * destined for the current or next Linux merge window. Basically, this should be just what you would send to Linus (or ask him to fetch). I will remove this tree from linux-next tomorrow and wait until it is more ready for mainline inclusion. -- Cheers, Stephen Rothwells...@canb.auug.org.au http://www.canb.auug.org.au/~sfr/ pgp45X43xbpbG.pgp Description: PGP signature
Re: linux-next: add utrace tree
* Ingo Molnar mi...@elte.hu wrote: * Ananth N Mavinakayanahalli ana...@in.ibm.com wrote: On Wed, Jan 20, 2010 at 06:49:50AM +0100, Ingo Molnar wrote: Ingo, Note, i'm not yet convinced that this (and the rest: uprobes and systemtap, etc.) can go uptream in its present form. Agreed, uprobes is still not upstream ready -- it was an RFC. We are working through the comments there to get it ready for merger. IMHO the far more important thing to address beyond formalities and workflow cleanliness are the (many) technical observations and objections offered by Peter Zijstra on lkml. Not just the git history but also the abstractions and concepts are messy and should be reworked IMO, and also good and working perf events integration should be achieved, etc. I think Oleg addressed most of Peter's concerns on utrace when the ptrace/utrace patchset was reposted. Peter is Cc:-ed and he might want to chime in. Perf integration with uprobes will be done and discussions have started with Masami and Frederic. There are a couple of fundamental technical aspects (XOL vma vs. emulation; breakpoint insertion through CoW and not through quiesce) that need resolution. The fact that there's a well established upstream workflow for instrumentation patches, which is being routed around by the utrace/uprobes/systemtap code here is not a good sign in terms of reaching a good upstream solution. Lets hope it works out well though. Agreed. On the other hand, having ptrace/utrace in the -next tree will give it a lot more testing, while any outstanding technical issues are being addressed. Including experimental code that is RFC and which is not certain to go upstream is certainly not the purpose of linux-next though. It will cause conflicts with various other trees and increases the overhead all around. It also causes us to trust linux-next bugreports less - as it's not the 'next Linux' anymore. Also, there's virtually no high-level technical review done in linux-next: the trees are implicitly trusted (because they are pushed by maintainers), bugs and conflicts are reported but otherwise it's a neutral tree that includes pretty much any commit indiscriminately. If you need review and testing there's a number of trees you can get inclusion into. Btw., the utrace code has lived in -mm for quite some time - that's an excellent route as Andrew does thorough review and testing. If Andrew agrees with this particular tree as-is and wants these bits to live in linux-next and have it in -mm that way then that's a fair approach obviously and i have no objections ... The point is to have at least one relevant maintainer request and track it and then supervise the completion of it (which includes the resolution of all outstanding objections) and then push it to Linus. Ingo
