Re: s390 user_enable_single_step() (Was: odd utrace testing results on s390x)
On Thu, 21 Jan 2010 21:51:13 +0100 Oleg Nesterov o...@redhat.com wrote: On 01/07, Roland McGrath wrote: I am confused as well. Yes, I thought about regs-psw.mask change too, but I don't understand why it helps.. [...] But. Acoording to the testing I did (unless I did something wrong again) this patch doesn't make any difference in this particular case. 6580807da14c423f0d0a708108e6df6ebc8bc83d does. Those results are quite mysterious to me. I think we'll have to get Martin to sort it out definitively. Finally nailed that one. Grrmpf.. the special case in the program check handler for single stepped svcs clobbers the argument registers. With our test case this affects the clone() system call. Funny things happen when the clone_flags argument is more or less random .. The following patch fixes the problem for me. -- Subject: [PATCH] fix single stepped svcs with TRACE_IRQFLAGS=y From: Martin Schwidefsky schwidef...@de.ibm.com If irq flags tracing is enabled the TRACE_IRQS_ON macros expands to a function call which clobbers registers %r0-%r5. The macro is used in the code path for single stepped system calls. The argument registers %r2-%r6 need to be restored from the stack before the system call function is called. Cc: sta...@kernel.org Signed-off-by: Martin Schwidefsky schwidef...@de.ibm.com --- arch/s390/kernel/entry.S |1 + arch/s390/kernel/entry64.S |1 + 2 files changed, 2 insertions(+) diff -urpN linux-2.6/arch/s390/kernel/entry64.S linux-2.6-patched/arch/s390/kernel/entry64.S --- linux-2.6/arch/s390/kernel/entry64.S2009-12-03 04:51:21.0 +0100 +++ linux-2.6-patched/arch/s390/kernel/entry64.S2010-01-26 14:04:58.0 +0100 @@ -549,6 +549,7 @@ pgm_svcper: mvc __THREAD_per+__PER_access_id(1,%r8),__LC_PER_ACCESS_ID oi __TI_flags+7(%r9),_TIF_SINGLE_STEP # set TIF_SINGLE_STEP TRACE_IRQS_ON + lmg %r2,%r6,SP_R2(%r15) # load svc arguments stosm __SF_EMPTY(%r15),0x03 # reenable interrupts j sysc_do_svc diff -urpN linux-2.6/arch/s390/kernel/entry.S linux-2.6-patched/arch/s390/kernel/entry.S --- linux-2.6/arch/s390/kernel/entry.S 2009-12-03 04:51:21.0 +0100 +++ linux-2.6-patched/arch/s390/kernel/entry.S 2010-01-26 14:04:58.0 +0100 @@ -571,6 +571,7 @@ pgm_svcper: mvc __THREAD_per+__PER_access_id(1,%r8),__LC_PER_ACCESS_ID oi __TI_flags+3(%r9),_TIF_SINGLE_STEP # set TIF_SINGLE_STEP TRACE_IRQS_ON + lm %r2,%r6,SP_R2(%r15) # load svc arguments stosm __SF_EMPTY(%r15),0x03 # reenable interrupts b BASED(sysc_do_svc) -- blue skies, Martin. Reality continues to ruin my life. - Calvin.
Re: linux-next: add utrace tree
On Fri 2010-01-22 08:43:18, valdis.kletni...@vt.edu wrote: On Fri, 22 Jan 2010 10:51:39 +0530, Ananth N Mavinakayanahalli said: FWIW, Oleg's implementation of ptrace over utrace is 100% compatible with legacy ptrace; gdb testsuite indicates that (http://lkml.org/lkml/2009/12/21/98). No, that only proves it's compatible enough for gdb to not care. The problem is all those *other* packages that abuse ptrace in totally crackhead ways. (No, I can't name them - but ptrace is the sort of interface that almost encourages its use for things somewhere between crackhead and mad-scientist, so they're almost certainly out there.. WAY out there.. :) strace, subterfugue, ltrace, ...? Plus various homegrown sandboxing tools... Pavel -- (english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Re: linux-next: add utrace tree
On Mon, Jan 25, 2010 at 01:41:57PM -0800, Linus Torvalds wrote: On Mon, 25 Jan 2010, Tom Tromey wrote: ... * Support displaced stepping in the kernel; I think this would improve performance when debugging in non-stop mode. Don't we already do that at least on x86? Just doing a single-step should work on an instruction even if it has a breakpoint on it, because we set the TF bit. Or maybe I'm not understanding what displaced stepping means to you. If Tom is referring to supporting single-stepping out of line, ie., not putting back the original instruction at the bp location, yes, we already support it on various architectures for kernel breakpoints, through the kprobes infrastructure. For userspace, there are more complications to take care of. We are reworking a prototype based on community comments (see the long UBP/XOL thread on lkml from a few days ago). Hopefully the userspace breakpoint assistance layer will be generic enough for gdb to also take advantage of, though the interface details need to be hashed out. Ananth
Re: linux-next: add utrace tree
Hi - On Mon, Jan 25, 2010 at 02:05:54PM -0700, Tom Tromey wrote: [...] Nevertheless, if the Linux kernel were to present a new user-space API, and if it had an advantage over ptrace, then we would port GDB to use it. There are other platforms where, IIRC, we now use some /proc thing instead of ptrace. There are definitely things we would like from such an API. Here's a few I can think of immediately, there are probably others. * Use an fd, not SIGCHLD+wait, to report inferior state changes to gdb. [...] Relatedly, don't mess with the inferior's parentage. This is satisfied by the gdbstub prototype. * Support displaced stepping in the kernel [...] I believe this is tantamount to hardware breakpoint support, which is already present (via optional uprobes). * Support some kind of breakpoint expression in the kernel; this would improve performance of conditional breakpoints. Perhaps the existing gdb agent expressions could be used. This is in the todo list. And that KILLER FEATURE of running strace plus gdb on the same process? It *already works* with the gdbstub, and unmodified strace + gdb, thanks to utrace multiplexing process control. It is still artificially restricted in many ways, but this sort of thing is ready for testing: % process [1] % strace -o FILE -p % gdb process (gdb) target remote /proc//gdb (gdb) backtrace (gdb) cont (gdb) ^D % [process continues] % cat FILE [...] % kill - FChE
Re: linux-next: add utrace tree
On Mon, Jan 25, 2010 at 04:07:21PM -0800, Linus Torvalds wrote: On Tue, 26 Jan 2010, Renzo Davoli wrote: The solution is that everybody can code his/her optimized kernel/user interface for tracing in his/her kernel module, i.e. utrace. I don't think people understand. That is simply not a solution. That is a PROBLEM. The thing you describe is an absolute disaster. Which is exactly why I rant against it. The last thing we want to have is here, take this, and make your own kernel module mess around it optimized for your particular crazy scenario. But every SINGLE post in this thread that has argued for utrace has argued exactly this way. I haven't followed much of the utrace discussions, but my impression was that utrace primarily is a cleanup effort, replacing don't change it, you might break it code with a clean, well defined (and even documented) implementation. To make it easier for people not familiar with the low-level architecture details to experiment with debugging stuff. Two points to consider: 1. If you'd merge utrace + ptrace-on-utrace, but never anything else which uses the utrace API, wouldn't it still be an improvement? 2. A well defined utrace API makes debugging code more hackable, thus more likely that someone might come up with a brilliant killer debug feature in the future. (This might sound lame, but there are already a few people doing crazy things with utrace while I'm not aware that people have done such experiments based on the current ptrace impl.) BTW, the ptrace improvements discussed elsewhere in this thread (like using an fd intead of signals/wait) are orthogonal to utrace, no? IMHO it's a seperate discussion. Johannes
Re: linux-next: add utrace tree
On Tue, 26 Jan 2010, Johannes Stezenbach wrote: 1. If you'd merge utrace + ptrace-on-utrace, but never anything else which uses the utrace API, wouldn't it still be an improvement? I already said earlier that I'd be perfectly happy to merge utrace code, as long as it was clear that I'm not merging a platform for crazy work. IOW, the end result might be merging 99% of the code, but I want to set peoples _expectations_ right. I'm not at all interested in merging stuff that has various exported helper functions for people doing random things, but I could happily merge stuff that cleans up internal implementation. 2. A well defined utrace API makes debugging code more hackable, thus more likely that someone might come up with a brilliant killer debug feature in the future. I don't really agree. Clean code makes things easier to improve, and maybe utrace cleans thigns up. But defining new API's makes me very worried, and quite frankly, the last thing I ever want to see is a new interface that out-of-tree modules starr using for random hacking. So I'd be much happier without the whole utrace kernel interface and callbacks, and very much would want to avoid the whole issue of plugins. I'd like to see ptrace improvements - not something else. In other words, I'd much much rather keep the utrace thing _internal_ to ptrace. If people have performance complaints about ptrace, let's look at fixing those _as_such_, rather than look at new modules etc. BTW, the ptrace improvements discussed elsewhere in this thread (like using an fd intead of signals/wait) are orthogonal to utrace, no? IMHO it's a seperate discussion. Largely, yes. Tied together to some degree of course, but the whole issue of code cleanup can be seen as a reasonably independent first step (while moving to a fd-based interface should probably not be done without some cleanup first, so they _are_ somewhat tied together). Linus
Re: linux-next: add utrace tree
Tom Tromey tro...@redhat.com writes: * Use an fd, not SIGCHLD+wait, to report inferior state changes to gdb. Internally we're already using a self-pipe to integrate this into gdb's main loop. Relatedly, don't mess with the inferior's parentage. How would having a kernel based solution be better over your user space simulation? BTW there's the new signalfd() system call that might do it (haven't checked if it works for SIGCHLD) * Support displaced stepping in the kernel; I think this would improve performance when debugging in non-stop mode. Not sure what displaced stepping is exactly, but it sounds like the branch tracing extensions that got added a few releases ago? On modern Intel chips they give you a branch buffer in memory. -Andi -- a...@linux.intel.com -- Speaking for myself only.
Re: linux-next: add utrace tree
On Tue, 26 Jan 2010, Andi Kleen wrote: Tom Tromey tro...@redhat.com writes: * Use an fd, not SIGCHLD+wait, to report inferior state changes to gdb. Internally we're already using a self-pipe to integrate this into gdb's main loop. Relatedly, don't mess with the inferior's parentage. How would having a kernel based solution be better over your user space simulation? Oh, the reason we should do something in the kernel is that you really can't do certain things with the ptrace() interface. For example, think about how Wine and UML use ptrace - and then realize that that makes it impossible to attach a debugger from the outside. That's a real deficiency in ptrace - much more so than the fact that there are some odd details (ie the whole read/write a word at a time is just a quirky detail in comparison - not a fundamental problem). BTW there's the new signalfd() system call that might do it (haven't checked if it works for SIGCHLD) No, you miss the point. The problem isn't that you want to turn signals into a file descriptor just because you like file descriptors. The problem is that anything that is based on reparenting and signals is fundamentally a one parent only kind of interface. See? So the reason I think using an fd is a good idea is _not_ because gdb already uses an fd internally, but because it gives you a connection between the debugger and debuggee that is not fundamentally limited to a single controller. (It doesn't have to be a file descriptor, of course, but could be any kind of other model that allows multiple connections. It's just that in unix terms, using a file descriptor as the cookie for the connection is a very natural model. So the important part isn't the file descriptor itself, it's the model you could build). Linus
Re: linux-next: add utrace tree
The problem is that anything that is based on reparenting and signals is fundamentally a one parent only kind of interface. See? I was actually thinking about that before I wrote the email. But when I did that i couldn't come up with a good scenario where multiple debuggers actually make sense. In a sense being a debugger is really a very intimate thing for process. Do you really want to have multiple of them messing with each other? If yes how would they know what to touch and what not? The only thing I could think of was user space virtualization (like old UML) together with a real debugger, but frankly these solutions all seemed like big race conditions to me anyways and should be better done in the kernel or below it, so I have a hard time taking them seriously. Can you think of any scenario where multiple debuggers on a process make sense? -Andi
Re: linux-next: add utrace tree
On 01/26, Linus Torvalds wrote: The problem is that anything that is based on reparenting and signals is fundamentally a one parent only kind of interface. See? Indeed. signals + do_wait() is the horrible model. So the reason I think using an fd is a good idea is _not_ because gdb already uses an fd internally, but because it gives you a connection between the debugger and debuggee that is not fundamentally limited to a single controller. (It doesn't have to be a file descriptor, of course, but could be any kind of other model that allows multiple connections. Yes. But then we need something which represents this connection in kernel: utrace_engine. Then we need something which allows multiple tracers to cooperate. Just for example, one tracer wants to resume the tracee, another tracer wants the tracee to be stopped. Utrace does this. And, since we should preserve the current ptrace, the tracers should cooperate with ptrace too. IOW, this quickly leads to the new abstraction layer, I think. And of course it is possible to implement this new model on top of utrace. Yes, utrace itself comes with utrace_engine_ops vector to implement whatever you like, perhaps you dislike this part. Oleg.
Re: linux-next: add utrace tree
On 01/26, Andi Kleen wrote: But when I did that i couldn't come up with a good scenario where multiple debuggers actually make sense. In a sense being a debugger is really a very intimate thing for process. Do you really want to have multiple of them messing with each other? If yes how would they know what to touch and what not? Yes, multiple debuggers can confuse each other if they change the state of debuggee simultaneously. The user should do this ;) Can you think of any scenario where multiple debuggers on a process make sense? Simple example. Try to debug/strace strace ot gdb itself. Not trivial, you can't attach to strace's tracees. Recently I spent 2 days trying to understand why strace -f hangs. I was able to attach to strace, but I wasn't able to see what its tracees do. And, it was not possible to even trace strace until it hangs, with ptrace the tracee (strace) must stop to report the event and this shadowed the race. Oleg.
Re: linux-next: add utrace tree
Simple example. Try to debug/strace strace ot gdb itself. Not trivial, you can't attach to strace's tracees. Recently I spent 2 days trying to understand why strace -f hangs. I was able to attach to strace, but I wasn't able to see what its tracees do. But what would the semantics be inside the tracees even if you could? And, it was not possible to even trace strace until it hangs, with ptrace the tracee (strace) must stop to report the event and this shadowed the race. Shadowing the race was the second surname of strace I thought anyways @) Basically if you care about races never use strace in the first place. -Andi -- a...@linux.intel.com -- Speaking for myself only.
Re: linux-next: add utrace tree
Linus == Linus Torvalds torva...@linux-foundation.org writes: Tom * Support displaced stepping in the kernel; I think this would improve Tom performance when debugging in non-stop mode. Linus Don't we already do that at least on x86? I don't know. If it does, and gdb does not yet use that, then that would be worth changing. Linus Or maybe I'm not understanding what displaced stepping means to you. In non-stop mode (where you can stop one thread but leave the others running), gdb wants to have the breakpoints always inserted. So, something must emulate the displaced instruction. Tom
Re: linux-next: add utrace tree
Tom * Use an fd, not SIGCHLD+wait, to report inferior state changes to gdb. Tom Internally we're already using a self-pipe to integrate this into Tom gdb's main loop. Relatedly, don't mess with the inferior's parentage. Andi How would having a kernel based solution be better over your Andi user space simulation? Signals and wait are a pain because if we want to use some random library in gdb, there might be conflicts. This is true even if we use signalfd. An fd-for-debugging does not have this problem. This matters more now that we're letting people script gdb in python. Tom
Re: linux-next: add utrace tree
On 01/26, Andi Kleen wrote: Simple example. Try to debug/strace strace ot gdb itself. Not trivial, you can't attach to strace's tracees. Recently I spent 2 days trying to understand why strace -f hangs. I was able to attach to strace, but I wasn't able to see what its tracees do. But what would the semantics be inside the tracees even if you could? In this particular case, all I need was something like gdb -p to attach to the tracee, see the backtrace and detach. And, it was not possible to even trace strace until it hangs, with ptrace the tracee (strace) must stop to report the event and this shadowed the race. Shadowing the race was the second surname of strace I thought anyways @) Basically if you care about races never use strace in the first place. Yes. And utrace doesn't require the tracee to be stopped to report the event ;) Yes, yes, utrace can't fix strace in this sense automatically, but still. Oleg.
Re: linux-next: add utrace tree
On Tue, 26 Jan 2010, Tom Tromey wrote: In non-stop mode (where you can stop one thread but leave the others running), gdb wants to have the breakpoints always inserted. So, something must emulate the displaced instruction. I'm almost totally uninterested in breakpoints that actually re-write instructions. It's impossible to do that efficiently and well, especially in threaded environments. So if you do instruction rewriting, I can only say that's your problem. But using the hardware breakpoints should automatically DTRT, both wrt threads _and_ wrt restarting. Sure, there's onyl a limited number of them, so if somebody wants more than that they are kind of screwed, but that's just how life is. Linus
Re: linux-next: add utrace tree
tromey wrote: [...] In non-stop mode (where you can stop one thread but leave the others running), gdb wants to have the breakpoints always inserted. So, something must emulate the displaced instruction. This sounds like the sort of thing that kernel kprobes do, which the uprobes patch does for userspace. The gdbstub prototype can use uprobes for such displaced breakpoints, and single-step-out-of-line to execute them on a few platforms like x86-*. This is already prototyped / working. (gdbstub currently restricts itself to single-threaded programs only, but that's another todo.) - FChE
