Re: linux-next: add utrace tree
On Sat, Jan 23, 2010 at 09:04:56PM -0800, Linus Torvalds wrote: The killer app for this will be the ability to delete thousands of lines of code from GDB, strace, and all the various other tools that have to painfully work around the major interface gotchas of ptrace(), while at the same time making their handling of complex processes much more robust. No. There is absolutely _no_ reason to believe that gdb et al would ever delete the ptrace interfaces anyway. More to the point, gdb *couldn't* use utrace, because utrace only exports a kernel API; not a syscall interface. And if the Red Hat Toolchain folks are thinking about encouraging gdb to start creating out-of-tree kernel modules, so that (a) gdb requires root privs, and (b) gdb is as (un)stable as SystemTap with respect to development kernels by making it dependent on internal kernel API's, the Red Hat Toolchain group needs to be smacked upside the head... - Ted
Re: linux-next: add utrace tree
Hi - On Sun, Jan 24, 2010 at 05:25:13AM -0500, ty...@mit.edu wrote: [...] The killer app for this will be the ability to delete thousands of lines of code from GDB, strace, and all the various other tools that have to painfully work around the major interface gotchas of ptrace(), while at the same time making their handling of complex processes much more robust. No. There is absolutely _no_ reason to believe that gdb et al would ever delete the ptrace interfaces anyway. More to the point, gdb *couldn't* use utrace, because utrace only exports a kernel API; not a syscall interface. Yes, this might explain why Kyle wrote: [...] I believe that utrace is the kernel side of that API. [...] And if the Red Hat Toolchain folks are thinking about encouraging gdb to start creating out-of-tree kernel modules [...] the Red Hat Toolchain group needs to be smacked upside the head... Those keeping up will note that an ordinary in-tree, non-modular, non-root-only, already-works-with-standard-gdb, potentially-better-than-ptrace debugger interface has already been prototyped posted on lkml as an RFC. - FChE
Re: linux-next: add utrace tree
On Sat, 23 Jan 2010, Frank Ch. Eigler wrote: On Sat, Jan 23, 2010 at 07:04:01AM +0100, Ingo Molnar wrote: [...] Also, if any systemtap person is interested in helping us create a more generic filter engine out of the current ftrace filter engine (which is really a precursor of a safe, sandboxed in-kernel script engine), that would be excellent as well. [...] Thank you for the invitation. More could be done - a simple C-like set of function perhaps - some minimal per probe local variable state, etc. (perhaps even looping as well, with a limit on number of predicament executions per filter invocation.) Yes, at some point when such bytecode intepreter gets rich enough, one may not need the translated-to-C means of running scripts. ( _Such_ a facility, could then perhaps be used to allow applications access to safe syscall sandboxing techniques: i.e. a programmable seccomp concept in essence, controlled via ASCII space filter expressions [...] IMHO that would be a superior concept for security modules too [...] [...] specific functionality with an immediately visible upside, with no need for opaque hooks. This OTOH seem like rather a stretch. If one claims that opaque hooks are bad, so instead have hooks that jump not to auditable C code but an bytecode interpreter? And have the bytecodes be uploaded from userspace? How is this supposed to produce transparency from the kernel/hook point of view? Simply because the kernel controls which byte code is executed and has control over the functionality behind it. That makes the hooks well defined and transparent. Thanks, tglx
Re: linux-next: add utrace tree
Hi - tytso wrote: [...] Let me see if I can paraphrase those of your concerns that were substantive: 1) That if utrace is merged, and systemtap keeps on using it, there may be some sort of chilling effect on kernel developers that would impede utrace's future development. This might sound plausible to an outsider, but luckily we're not stuck with having to speculate: one can examine history. Systemtap has been around, working roughly the same way, for about *five years*. Systemtap modules use more than a handful of mainstream module-accessible kernel services. During all this time, how many examples have there been when when systemtap developers have pleaded with lkml to avoid changing some prior interface? How many of those successfully? (That last one is a trick question, since both numbers are really close to *zero*.) How much real impediment to change has our mere existence caused? 2) That systemtap is not portable to all kernel versions. Problems do periodically occur. However, one can again refer to historical facts to assess whether in fact they warrant long term grudges. In every release note, we list the range of kernel versions we test against. We may have one of the broadest ranges of support, 2.6.9 through to many current -rc*s and non-linus trees. We have several mechanisms which let us easily adapt to most changes. It may interest readers to find out that the number of systemtap changes we have had to add on account of kernel changes is on the order of a *few per year*. The usual turnaround, once reported, is on the order of a *few days*. 3) That systemtap users will complain to kernel developers if systemtap becomes incompatible. Let's go to the historical record again. How many such complaints have actually been seen in inappropriate fora such as lkml? How difficult were they to diagnose / redirect to the proper venue? Have they constituted a loss of face for kernel developers? 4) That systemtap is almost but not quite as evil as nvidia. It seems factors like ... - always being completely open source project - keeping in regular contact with lkml and other constituencies - not being related to essential hardware enablement, so users not wanting it don't have to touch it - the compile-to-C approach being technologically necessary since there was no alternative plausible way at the time (and still now) - repeatedly offering infrastructure code with non-stap uses ... all add up to a mere nudge away from entirely evil. If so, I wonder if your sort of grossly bimodal view of ethical virtue is going to foster the right sorts of change in the linux kernel community. - FChE
Re: linux-next: add utrace tree
On 01/24/10 13:01, Frank Ch. Eigler wrote: ... all add up to a mere nudge away from entirely evil. If so, I wonder if your sort of grossly bimodal view of ethical virtue is going to foster the right sorts of change in the linux kernel community. Nothing like a good religious debate to liven up your Sunday... - FChE
Re: [RFC] [PATCH 1/7] User Space Breakpoint Assistance Layer (UBP)
On Sun 2010-01-17 16:01:46, Peter Zijlstra wrote: On Sun, 2010-01-17 at 16:56 +0200, Avi Kivity wrote: On 01/17/2010 04:52 PM, Peter Zijlstra wrote: Also, if its fixed size you're imposing artificial limits on the number of possible probes. Obviously we'll need a limit, a uprobe will also take kernel memory, we can't allow people to exhaust it. Only if its unprivilidged, kernel and root should be able to place as many probes until the machine keels over. Well, it is address space that limits you in both cases... -- (english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Re: linux-next: add utrace tree
On Sat, Jan 23, 2010 at 14:48, ty...@mit.edu wrote: The fundamental issue which Ingo is trying to say (and which you apparently don't seem to be understanding) is that utrace doesn't export a syscall (which is an ABI that we are willing to promise will be stable), but rather a set of kernel API's (which we never promise to be stable), The point that's being missed is that there is a chicken-and-egg problem here. The chicken is a replacement or extension to the debugger interface that would make it possible for me to do things like GDB a process while it's being strace'd or vice versa. The egg is the utrace bits, an unstable but somewhat arch-generic ABI that abstracts out ptrace() to make it possible to stack both in-kernel and userspace debuggers/tracers/etc and have multiple simultaneous users. and the fact that there will be out-of-tree programs that are going to be trying to depend on that interface (much like Systemtap does today when it creates kernel modules) is something that is considered on par with Nvidia trying to ship proprietary video drivers. Ugh... perhaps we should derive a variation of Godwin's law for this: As an LKML discussion grows longer, the probability of an unfavorable comparison involving nVidia or Microsoft approaches 1. If you want to try to slide utrace in, such that we're able to ignore the fact that there will be this external house that will be built on quicksand, pointing at how nice the external house will be isn't going to be helpful. Nor is pointing at the ability that other people will be able to build other really nice houses on the aforementioned quicksand (i.e., out-of-tree kernel modules that depend on kernel API's). Personally I don't give a flying about SystemTap; I'm interested in things like the ability to stack gdb with strace, the RFC gdb-stub posted a week ago, etc. None of those abilities would be out-of-tree modules at all, and therefore the quicksand analogy is specious. A simple code cleanup argument is not carrying the day (Look! We can cleanup the ptree code!). It's going to have to be a **really** cool in-tree kernel funtionality that provides a killer feature (in Linus's words), enough so that people are willing to overlook the fact that there's this monster external out-of-tree project that wants to be depend on API's that may not be stable, and which, even if the developers don't grump at us, users will grump at us when we change API's that we had never guaranteed will be stable, and then Systemtap breaks. I would be willing to guess that something like 95% of the people using SystemTap or other tools are doing so on Red Hat Enterprise Linux or other enterprise supported platforms, and so when something breaks they go whinge at Red Hat, etc. If I recall correctly Red Hat and many of the other vendors already heavily fiddle with kernel patches they apply to provide some amount of binary module compatibility. This is probably why Ingo invited you to think about ways of doing some kind of safe in-kernel bytecode approach. That has the advantage of doing away with external kernel modules, with all of their many downsides: its dependency on unstable kernel API's, the fact that many financial customers have security policies that prohibit C compilers on production machines, the inherent security risk of allowing external random kernel modules to be delivered and loaded into a system, etc. There are substantial non-SystemTap uses for utrace that would *not* be satisfied by an in-kernel bytecode approach, starting with stacking debuggers and tracers. Furthermore, let's say they did go off and build the in-kernel bytecode interpreter. I can pretty much guarantee that people would say the hooks into the rest of the kernel are too invasive and they should be abstracted out into an API. *This is that API!* Cheers, Kyle Moffett
Re: linux-next: add utrace tree
On Sat, Jan 23, 2010 at 12:23:33PM +0100, Ingo Molnar wrote: * Kyle Moffett k...@moffetthome.net wrote: On Fri, Jan 22, 2010 at 19:22, Linus Torvalds torva...@linux-foundation.org wrote: ... In that sense it might be better to fix/enhance ptrace, if there's interest. I've written a handful of ptrace extensions in the past (none of them went upstream tho), it can be done in a useful manner and the code is pretty hackable. There are basic problems left to be solved: for example why is there still no 'memory block copy' call, why are we _still_ limited to one word per system call PTRACE_PEEK* memory copies? It's ridiculous. SparcLinux has PTRACE_WRITE*/READ* support that implements this, but none of the other architectures have it so it's essentially unused. Or another possible direction would be to extend the perf events syscall with interception capabilities. It's far more performant at extracting application state without scheduling than any ptrace method - and interception/injection would be a natural next step - if there's interest. This certainly is now a chicken and egg problem. Everybody agrees that Linux needs something better than ptrace; legacy ptrace will continue to live, so will utilities written to it (strace, etc). But should that limit what Linux can offer? What's the way out? - Enhance ptrace: At least one ptrace maintainer (Roland) had publically stated he doesn't prefer enhancing legacy ptrace -- that its already a beast to maintain, and adding more complexity to it does it no good. - Extend perf; would perf then use utrace underneath? Or would one have to redo some of what utrace already does for thread level control? - Give utrace a syscall and make it the primary way for users to interact with the layer. There are benefits to this if there is agreement on the utrace layer itself, maybe with less fexibility than what it currently offers? If yes, what should it look like? Any new debug facility will have to incorporate some or most learnings from what utrace tried to address. It would be sad to just dump utrace and redo everything from scratch or band-aid existing interfaces. Ananth
brine ll xylop hone horse power s
agree able finge rstal l calen dar leuco cyte caden ce
Re: linux-next: add utrace tree
On Sun, Jan 24, 2010 at 08:42:13PM -0500, Kyle Moffett wrote: Personally I don't give a flying about SystemTap; I'm interested in things like the ability to stack gdb with strace, the RFC gdb-stub posted a week ago, etc. None of those abilities would be out-of-tree modules at all, and therefore the quicksand analogy is specious. Great. So what should be reviewed is utrace *plus* these other userland interfaces, which may get critiqued and improved, and utrace patches can be reviewed in light of these new features. But be warned if it turns out that only 30% of utrace is only needed to support gdb stacking with strace, etc., the other 70% will likely get ejected and the utrace patches streamlined to support these in-tree users. But since you don't give a flying about SystemTap, presumably you won't mind, right? I would be willing to guess that something like 95% of the people using SystemTap or other tools are doing so on Red Hat Enterprise Linux or other enterprise supported platforms, and so when something breaks they go whinge at Red Hat, etc. If I recall correctly Red Hat and many of the other vendors already heavily fiddle with kernel patches they apply to provide some amount of binary module compatibility. Sure, but as out-of-tree modules, the best they can expect is that most kernel developers will pretend that they don't exist. Which is OK, when I tried using SystemTap most of the concerns which I expressed as being critical for kernel developers were largely ignored (as near as I could tell) because the target market was RHEL corporate customers, and they prioritized their resourcing accordingly --- so they shouldn't mind if kernel developers return the favor. But that means that we should only merge those portions of utrace that are needed for these alleged killer new features, and only if these new features are cool enough that they justify the new code on their own merits. At least, IMNSHO. - Ted
