Re: linux-next: add utrace tree
On Sun, 24 Jan 2010, Kyle Moffett wrote: The point that's being missed is that there is a chicken-and-egg problem here. The chicken is a replacement or extension to the debugger interface that would make it possible for me to do things like GDB a process while it's being strace'd or vice versa. The egg is the utrace bits, an unstable but somewhat arch-generic ABI that abstracts out ptrace() to make it possible to stack both in-kernel and userspace debuggers/tracers/etc and have multiple simultaneous users. Quite frankly, as far as I'm concerned, I'd be a whole lot more interested in utrace if it's _only_ stated (and implied) goal was to do exactly this. The thing I object to is the whole dessert topping _and_ floor wax thing, with kernel interfaces for random other users. If somebody extended ptrace in good ways, that's a totally different thing. But I think utrace has been over-designed, possibly as a result of others coming in and saying hey, I'd like to use that too for xyz. Do one thing, and do it well. I'd not mind somebody improving ptrace (including extending its semantics - I do agree that the whole SIGSTOP thing makes it hard to have multiple debuggers). That said, I also suspect that people should still look seriously at simply just improving ptrace. For example, I suspect that the biggest problem with ptrace is really just the signalling, and that creating a new extension for JUST THAT, and then having a model where you can choose - at PTRACE_ATTACH time - how to wait for events would be a good thing. But as long as it is I want to solve all problems, I'm not very impressed. Maybe somebody would be interested in trying to take the utrace improvements, and scaling down what they promise, and ignoring all input except for I want to strace and gdb at the same time. So stop the crazy new kernel interfaces crap. Stop the crazy maybe we can use it for ftrace and generic user event tracing too. Stop the crazy. Linus
Re: linux-next: add utrace tree
Hi - On Mon, Jan 25, 2010 at 08:52:41AM -0800, Linus Torvalds wrote: [...] If somebody extended ptrace in good ways, that's a totally different thing. But I think utrace has been over-designed, possibly as a result of others coming in and saying hey, I'd like to use that too for xyz. [...] Earlier, you said that you haven't followed utrace at all. Upon what real information do you infer that it has been over-designed? - FChE
Re: linux-next: add utrace tree
On Mon, 25 Jan 2010, Frank Ch. Eigler wrote: Earlier, you said that you haven't followed utrace at all. Upon what real information do you infer that it has been over-designed? Upon the information that people are talking about magic new kernel interfaces to do fancy things. And talking about doing things with it that are simply not relevant for ptrace/strace. In fact, in this very thread I've been informed that there are no user interfaces to utrace at all, which to me says that it's been TOTALLY MISDESIGNED FROM THE VERY START, and has nothing to do with making ptrace work for strace/gdb at the same time. In other words, I may not have followed utrace development, but I sure as hell can read. And everything I read about it just makes me less inclined to want to merge it. The people who argue for it are actually screwing themselves by arguing for all the wrong things, and making me convinced I don't want to touch it with a ten-foot pole. If somebody were to argue that this is a simple series of patches to clean up ptrace and make it possible to strace a debugged process, then that would have been different. That's not what you or others have been doing. You've been pushing exactly the _reverse_ of that, namely how great it is for some random totally new features that I'm convinced aren't even used by a lot of people. So give me a populist argument that makes sense for tons of actual users, not some f*cking here's a cool infrastructure that developers can do random crazy out-of-tree crap with. Because I'm not interested in crazy developers. Linus
Re: linux-next: add utrace tree
On Mon, 25 Jan 2010, Linus Torvalds wrote: So give me a populist argument that makes sense for tons of actual users, not some f*cking here's a cool infrastructure that developers can do random crazy out-of-tree crap with. Because I'm not interested in crazy developers. In other words, give me the killer feature. The thing I've asked for all the time. The thing that you seem to continually NOT EVEN UNDERSTAND. Linus
Re: linux-next: add utrace tree
On Mon, 2010-01-25 at 09:36 -0800, Linus Torvalds wrote: Because I'm not interested in crazy developers. Linus Uh oh, that's not good for us real-time folks. http://lwn.net/Articles/357800/ And, according to Linus, the realtime people are crazy, so they can be left to deal with the weird stuff. -- Steve (Sorry, I just couldn't resist)
Re: linux-next: add utrace tree
Uh oh, that's not good for us real-time folks. http://lwn.net/Articles/357800/ And, according to Linus, the realtime people are crazy, so they can be left to deal with the weird stuff. I'd prefer the trees to be separate for testing purposes: it doens't make much sense to have SMP support as a normal kernel feature when most people won't have SMP anyway -- Linus Torvalds Use cases got that into the tree pretty easily, I am sure RT ones will do the same.
Re: linux-next: add utrace tree
On Mon, 25 Jan 2010, Steven Rostedt wrote: Uh oh, that's not good for us real-time folks. http://lwn.net/Articles/357800/ And, according to Linus, the realtime people are crazy, so they can be left to deal with the weird stuff. The RT people have actually been pretty good at slipping their stuff in, in small increments, and always with good reasons for why they aren't crazy. Yeah, it's taken them years, and they still have out-of-tree stuff. And yeah, they had to change some things to make them more palatable to the mainline kernel - the whole fundamental raw spinlock change is just the most recent example of that. But on the whole, I think it's actually worked out pretty well for them. I think the mainline kernel has improved in the process, but I also suspect that _their_ RT patches have also improved thanks to having to make the work more palatable to people like me who don't care all that deeply about their particular flavor of crazy. And yeah, I still think the hard-RT people are mostly crazy. So I can work with crazy people, that's not the problem. They just need to _sell_ their crazy stuff to me using non-crazy arguments, and in small and well-defined pieces. When I ask for killer features, I want them to lull me into a safe and cozy world where the stuff they are pushing is actually useful to mainline people _first_. In other words, every new crazy feature should be hidden in a nice solid Trojan Horse gift: something that looks _obviously_ good at first sight. The fact that it may contain the germs for future features should be hidden so well that not only is it not used as an argument (Hey, look at all those soldiers in that horse, imagine what you could do with them), it should also not be obvious from the source code (Look at all those hooks I sprinkled around, which aren't actually used by anything, but just imagine what you could do with them). Linus
Re: linux-next: add utrace tree
On Mon, 2010-01-25 at 10:12 -0800, Linus Torvalds wrote: But on the whole, I think it's actually worked out pretty well for them. I think the mainline kernel has improved in the process, but I also suspect that _their_ RT patches have also improved thanks to having to make the work more palatable to people like me who don't care all that deeply about their particular flavor of crazy. Actually this is an understatement. Every feature (and I do mean _every_) that went from -rt into mainline, undertook 3 or more rewrites before it was acceptable for mainline. And every time, the end result made the -rt patch set better as a whole. Not to mention, that a lot of the early stuff also cleaned up mainline. You can't have Real-Time without having a clean kernel. And as you stated, a lot of those patches to clean up the kernel, no one even knew that the real reason was to help the -rt patch set. They were well disguised Trojan horses. Darn, it looks like you are onto our scheme. -- Steve
Re: linux-next: add utrace tree
On Mon, 25 Jan 2010, Steven Rostedt wrote: On Mon, 2010-01-25 at 10:12 -0800, Linus Torvalds wrote: But on the whole, I think it's actually worked out pretty well for them. I think the mainline kernel has improved in the process, but I also suspect that _their_ RT patches have also improved thanks to having to make the work more palatable to people like me who don't care all that deeply about their particular flavor of crazy. Actually this is an understatement. Every feature (and I do mean _every_) that went from -rt into mainline, undertook 3 or more rewrites before it was acceptable for mainline. And every time, the end result made the -rt patch set better as a whole. Not to mention, that a lot of the early stuff also cleaned up mainline. You can't have Real-Time without having a clean kernel. And as you stated, a lot of those patches to clean up the kernel, no one even knew that the real reason was to help the -rt patch set. They were well disguised Trojan horses. Tsss. Never admit such things. Darn, it looks like you are onto our scheme. Which scheme ? The only Trojan horses in the kernel tree are in drivers/char/drivers/char/tty_io.c which put Linus himself into Linux-0.98.2 :) tglx
Re: linux-next: add utrace tree
* Thomas Gleixner t...@linutronix.de wrote: On Mon, 25 Jan 2010, Steven Rostedt wrote: On Mon, 2010-01-25 at 10:12 -0800, Linus Torvalds wrote: But on the whole, I think it's actually worked out pretty well for them. I think the mainline kernel has improved in the process, but I also suspect that _their_ RT patches have also improved thanks to having to make the work more palatable to people like me who don't care all that deeply about their particular flavor of crazy. Actually this is an understatement. Every feature (and I do mean _every_) that went from -rt into mainline, undertook 3 or more rewrites before it was acceptable for mainline. And every time, the end result made the -rt patch set better as a whole. Not to mention, that a lot of the early stuff also cleaned up mainline. You can't have Real-Time without having a clean kernel. And as you stated, a lot of those patches to clean up the kernel, no one even knew that the real reason was to help the -rt patch set. They were well disguised Trojan horses. Tsss. Never admit such things. Here's four examples of recent kernel features: - lockdep [1] - ftrace [2] - new-style generic mutexes and spin-mutexes [3] - the new arch/x86 tree[4] I suspect few would guess that all of these features were motivated by the -rt kernel originally: [1] lockdep started out as the 'track irqs-off sections' patches in -rt [2] ftrace started out as -rt's latency tracer and logdev [3] mutex.c was motivated by rtmutex.c [4] arch-x86 was motivated by annoyance with needless porting of -rt features from 32-bit to 64-bit x86 and back. [ Nor would you normally guess that Linux itself was motivated by a guy wanting to toy around with 32-bit x86 assembly ;-) ] Various forms of craziness that motivate us dont really hurt, as long as the process is rooted in reality. We can 'wish' for the crazier future stuff and can help it indirectly, and sometimes it might even happen down the road - but reality and common-sense utility is what controls. And note that there's nothing dishonest about doing multi-purpose patches, as long as the mainstream purpose isnt really just a decoy. When we decouple a feature from -rt we usually forget its -rt purpose and the intermediate for-mainstream forms arent even useful for -rt - back-integration into -rt comes at a later stage. This makes it doubly sure that it's all formed by mainstream's need, not -rt's needs. In the few cases where the -rt role is prominent for some weird reason we declare it as such. It's the exception to the rule really - few useful kernel features are single purpose. ( When they are then we are likely doing something wrong. -rt _is_ a special case. ) Ingo
Re: linux-next: add utrace tree
On Mon, 25 Jan 2010, Mark Wielaard wrote: And all these users have wishes to extend the current ptrace interface mess. But nobody dares to extend ptrace in any direction because fixing/cleaning up one of these use cases might break the others in subtle and not so subtle ways. Which is why the utrace series of patches is cleaning up all this stuff first. I call bullshit. You can clean up ptrace without introducing odd new interfaces and trying to sell it as some revolutionary new kernel interface that can do anything. I also call bullshit on the ptrace() is so horribly nasty argument. Yes, I've seen the code that uses ptrace in user space, and yes, it's nasty, but it's invariably _not_ nasty so much because ptrace itself is nasty, but because it's full of #ifdef so-and-so-os/so-and-so-arch, and the code is never cleaned up. There are a couple of obvious cases of ptrace being uglier-than-it-needs- to-be. Like the traditional ptrace read/write interface being purely word at a time, and that clearly is not pretty. Several architectures already do copy range kind of versions on it, though, so that's just a detail, and if anybody wanted to clean it up, they could have. The more fundamental problem is the use of signals (while at the same time wanting to _trap_ non-ptrace signals), without any model for a connection state, which is why you can have only one tracer. But again, that's largely a user interface issue, and apparently utrace does _nothing_ for that problem at all. So I do agree that ptrace is not a great interface. However: repeating that statement over and over in _no_ way excuses some totally unrelated code that doesn't have anything what-so-ever to do with the actual problems of ptrace. Linus
Re: linux-next: add utrace tree
Linus == Linus Torvalds torva...@linux-foundation.org writes: Linus No. There is absolutely _no_ reason to believe that gdb et al would ever Linus delete the ptrace interfaces anyway. Yes, in GDB we approximately never delete anything. Nevertheless, if the Linux kernel were to present a new user-space API, and if it had an advantage over ptrace, then we would port GDB to use it. There are other platforms where, IIRC, we now use some /proc thing instead of ptrace. There are definitely things we would like from such an API. Here's a few I can think of immediately, there are probably others. * Use an fd, not SIGCHLD+wait, to report inferior state changes to gdb. Internally we're already using a self-pipe to integrate this into gdb's main loop. Relatedly, don't mess with the inferior's parentage. * Support displaced stepping in the kernel; I think this would improve performance when debugging in non-stop mode. * Support some kind of breakpoint expression in the kernel; this would improve performance of conditional breakpoints. Perhaps the existing gdb agent expressions could be used. Tom
Re: linux-next: add utrace tree
On Mon, 25 Jan 2010, Tom Tromey wrote: There are definitely things we would like from such an API. Here's a few I can think of immediately, there are probably others. * Use an fd, not SIGCHLD+wait, to report inferior state changes to gdb. Internally we're already using a self-pipe to integrate this into gdb's main loop. Relatedly, don't mess with the inferior's parentage. As I kind of alluded to elsewhere, I heartily agree with this. The really major design mistake of ptrace (as opposed to just various ugly corners) is how it has no connection information, and that ends up being one of the main reasons why you can't have two ptracers working on the same thing. (There are other things that complicate that too, of course, like simply just trying to manage various per-thread state like debug registers etc, but that's a separate class of complications). * Support displaced stepping in the kernel; I think this would improve performance when debugging in non-stop mode. Don't we already do that at least on x86? Just doing a single-step should work on an instruction even if it has a breakpoint on it, because we set the TF bit. Or maybe I'm not understanding what displaced stepping means to you. * Support some kind of breakpoint expression in the kernel; this would improve performance of conditional breakpoints. Perhaps the existing gdb agent expressions could be used. I suspect it might be reasonable to do simple expressions on breakpoints, but not the kind of things gdb exports to users. IOW, maybe you could have a single conditional on a single value (register or memory) associated with an expression. Regardless, internally to the kernel your two later issues are details. The how to connect to the debuggee is a much more fundamental issue, and has the biggest design/interface impact. The other would likely just be new ptrace command extensions that somebody would have to just implement the grotty details on. Linus
Re: linux-next: add utrace tree
Let me add my two euro-cents to this discussion. Mark Wielaard m...@redhat.com: Unfortunately ptrace does all that magic already (badly). People don't just use it for (s)tracing syscalls, but also for tracing signals, for single step debugging and poking at memory, register state, for process jailing and virtualization (uml) through syscall emulation. So when they are talking about these fancy things that is because that is what ptrace gives them currently. And they hate it, because the ptrace interface is such a pain to work with. And all these things don't really work together. You cannot trace, emulate, debug, jail at the same time. I support Mark's words. I don't use ptrace for debugging/tracing and I have experienced severe limitations of ptrace interface. (I have tried to post some extensions for ptrace to overcome some constraints see my posts on ptrace_vm or ptrace_multi on LKML). Oleg Nesterov, writing to Andrew Morton said: First of all, utrace makes other things possible. gdbstub, nondestructive core dump, uprobes, kmview, hopefully more. I didn't look at these projects closely, perhaps other people can tell more. As for their merge status, until utrace itself is merged it is very hard to develop them out of tree. In the list above there is also kmview, which is a creature of mines. umview and kmview are partial virtual machines, processes running in a [uk]mview machine can have their own view for the file system, networking support, user-id, system-name, etc. A [uk]mview machine virtualizes just what the user need: the filesystem or just a subtree/some subtrees or networking or define one/some virtual devices, etc. The view provided by a [uk]mview machine can be a composition of real resources (provided by the Linux kernel) and virtual resources. Each system call request gets hijacked to a module of [uk]mview when it refers to a virtual resource. The request is forwarded to the kernel otherwise. umview is based on ptrace, kmview uses a kernel module based on utrace. (umview is included in debian lenny (to sid), tutorial and manuals in wiki.virtualsquare.org) IMHO utrace is better than ptrace (or an optimized version of it): 1 - Frank Ch. Eigler wrote: At least one reason is that ptrace is single-usage-only, so for example you cannot concurrently debug strace the same program. - exactly. utrace allows multiple tracing engines, this means that kmview machines can be nested (in a natural way, no extra code is needed for this feature). In the same way strace/gdb can run on virtualized processes, too. 2 - kmview kernel module implements several optimizations to minimize the number of requests forwarded to the kmview process (the virtual machine monitor). kmview is just a module using the utrace interface, prior attempts of optimized umview required kernel patches. Like kmview any other service requiring process tracing can include specific optimizations in its own kernel module. On the other hand, all these services could use the standardized utrace interface for their optimizations, instead asking for messy patches to change code all around the kernel source. 3 - ptrace takes SIGSTOP/SIGCONT for its own management. Strace/gdb and umview cannot be transparent for programs using these signals. Oleg Nesterov talking about Ptrace said: Of course they can't use other interfaces, we don't have them. And without the new abstraction layer we will never have, I think. I agree. THe following list includes the execution times I got in a recent test (make vde-2, see http://www.cs.unibo.it/~renzo/view-os-lk2009.pdf) plain kernel 22.7s, kmview (no modules) 23.9s (+5.5%), full kmview (modules loaded, all syscall virtualized) 38.5s (+70%) optimized umview 51.0 (+124%), umview on vanilla kernel 75.7s (+233%). utrace can be used to speedup virtualization (at least in my case it worked in this way). Performance can be useful for debugging but it is a main issue for virtualization. Kmview module provides optimizations to select the system call requests depending on the syscall number, the pathnames or the file descriptors. http://wiki.virtualsquare.org/index.php/KMview_module_interface_specifications Trying to add all the optimizations needed by different projects to ptrace is a never-ending nightmare: the LKML will continue to receive patch proposals for ptrace... The solution is that everybody can code his/her optimized kernel/user interface for tracing in his/her kernel module, i.e. utrace. renzo
Re: linux-next: add utrace tree
On Tue, 26 Jan 2010, Renzo Davoli wrote: The solution is that everybody can code his/her optimized kernel/user interface for tracing in his/her kernel module, i.e. utrace. I don't think people understand. That is simply not a solution. That is a PROBLEM. The thing you describe is an absolute disaster. Which is exactly why I rant against it. The last thing we want to have is here, take this, and make your own kernel module mess around it optimized for your particular crazy scenario. But every SINGLE post in this thread that has argued for utrace has argued exactly this way. Linus
