Re: [RFC] [PATCH] In-kernel gdbstub based on utrace Infrastructure.
Ingo Molnar mi...@elte.hu writes: [...] Since the speed of development in this area is truly glacial at the moment and the practical advantages that i can experience personally (directly as a Linux user and indirectly as a maintainer) are miniscule so far, caution is warranted IMO. [...] If the caution you suggest is operationally equivalent to discouraging even miniscule improvements, is it any wonder that progress is glacial? The gdbstub prototype was constructed for two reasons: to demonstrate utrace usage now, and in the future to be incrementally useful (over ptrace, by moving into fast kernel-space operations like multithreading control, gdb-tracepoint support, other stuff). #1 is about done. With respect to #2, we can certainly commit to ongoing work on improvements, provided the community shows interest and goodwill. - FChE
Re: [RFC] [PATCH] In-kernel gdbstub based on utrace Infrastructure.
Hi - On Thu, Dec 10, 2009 at 07:16:38PM +0100, Ingo Molnar wrote: [...] The gdbstub prototype was constructed for two reasons: to demonstrate utrace usage now, and in the future to be incrementally useful (over ptrace, by moving into fast kernel-space operations like multithreading control, gdb-tracepoint support, other stuff). [...] What i'd like to see is measurable benefits to users, developers and maintainers. OK, it's clear that in the gdb-stub's case, some positive motivation beyond it being an api-educational example would be appropriate before merging. Note that it was only an RFC at the time. I'd like to see the same for SystemTap too btw. systemtap's benefits are beyond question to its users. (Others are worried about systemtap problems, but that wasn't your question.) In what form do you expect to see such measurements? It would help if you could point out prior examples of such measurable benefit analyses that perchance accompanied other then-new kernel features, say in the tracing/debugging area. - FChE
Re: [RFC] [PATCH] In-kernel gdbstub based on utrace Infrastructure.
* Frank Ch. Eigler f...@redhat.com wrote: [...] If the in-kernel gdb stub replaced kgdb functionally you'd hear no complaints from me. Let's leave it as an idea for the future. We came a full circle - that's the argument. We say overlap, duplication and incomplete implementation in this area is a problem. Since the speed of development in this area is truly glacial at the moment and the practical advantages that i can experience personally (directly as a Linux user and indirectly as a maintainer) are miniscule so far, caution is warranted IMO. You say there is no problem. We'll have to agree to disagree i guess. Ingo
Re: [RFC] [PATCH] In-kernel gdbstub based on utrace Infrastructure.
Hi - Help me out here: by kgdb extension do you imagine something new that an unprivileged user can use to debug his own process? Or do you imagine a new userspace facility that single-steps the kernel? Is this a trick question? Single-stepping the kernel on the same system [especially if it's an UP system] would certainly be a challenge ;-) What i mean is what i said: if you provide a new framework (especially if it's user visible - which both kgdb and the gdb stub is) you should either fully replace existing functionality or extend it. Overlapping it in an incomplete way is not useful to anyone. But there is no overlap beyond the name. The functional scope of the two interfaces is totally non-overlapping, and are consistent with the current chasms between kernel- and user-side debugging. Sure, in the future, it may make sense to teach the kernel-side (kgdb serial console) interface to manipulate userspace. But that will require a gdb extension. And it would not satisfy an unprivileged user's need to debug pure userspace (in a better way than current ptrace can). This is why I keep asking for specificity as to this new framework you imagine. Just sharing definitions such as kgdb_arch/kgdb_io but otherwise completely disconnected (separate channels)? Extending kgdb to allow the use of it as if we used gdb locally would certainly be interesting - and then you could drop into the kernel anytime as well. (Is this a restatement of the trick question idea?) We dont want to separate facilities for the same conceptual thing: examining application state (be that in user-space and kernel-space). This seems like a shallow sort of consistency. kgdb was added after ptrace existed -- why not extend ptrace instead to target the kernel? After all, it's examining application state. The answer is that it doesn't make a heck of a lot of sense. kgdb simply used gdb's preferred way of remote debugging. That's certainly the ugliest bit of it btw - but it's an externality to kgdb. Had it extended ptrace it wouldnt have gdb compatibility. So, because of a constraint for gdb compatibility, you built a separate interface for kgdb vs. ptrace. Fine. Do you accept that, even if a hypothetical single channel existed for which kernel- and user-space debugging could occur, current gdb is not compatible with this? So by your own reasoning, such a facility should not be mandated as a necessary first step. [...] perf replaces oprofile functionally. (I'm told that it's not a strict superset from a functional point of view, FWIW, something about a larger selection of low level hardware counters.) If the in-kernel gdb stub replaced kgdb functionally you'd hear no complaints from me. Let's leave it as an idea for the future. - FChE
Re: [RFC] [PATCH] In-kernel gdbstub based on utrace Infrastructure.
Hi - On Tue, Dec 01, 2009 at 05:11:32PM +0100, Ingo Molnar wrote: Those facilities are not overlapping with kgdb though so my point doesnt apply to them. An in-kernel gdb server sure overlaps/extends kgdb though. Only in name. One is highly invasive, for debugging the kernel across serial consoles. The other is highly noninvasive, for debugging user processes across normal userspace channels. They both happen to talk to gdb, but that's the end of the natural overlap. Even if kgdb was extended to be able to manage userspace, and if gdb itself was extended to be able to use that same single channel, this would still not duplicate the use scenario for an ordinary user debugging his own processes. (Plus, in the future where at least gdb is applied toward kernel+user debugging, it is unlikely to be the case that this would need to be done *over a single channel*. A separate channel for kernel and separate channels for userspace programs are no less likely.) Btw., perf does meet that definition: it functionally replaces all facilities that it overlaps/extends - such as Oprofile. [...] (And they currently separately coexist.) - FChE
Re: [RFC] [PATCH] In-kernel gdbstub based on utrace Infrastructure.
* Frank Ch. Eigler f...@redhat.com wrote: Hi - Only in name. One is highly invasive, for debugging the kernel across serial consoles. The other is highly noninvasive, for debugging user processes across normal userspace channels. They both happen to talk to gdb, but that's the end of the natural overlap. [...] Well nothing that you mention here changes our obvious suggestion that an in-kernel gdb stub should obviously either be a kgdb extension, or a replacement of it. Help me out here: by kgdb extension do you imagine something new that an unprivileged user can use to debug his own process? Or do you imagine a new userspace facility that single-steps the kernel? Is this a trick question? Single-stepping the kernel on the same system [especially if it's an UP system] would certainly be a challenge ;-) What i mean is what i said: if you provide a new framework (especially if it's user visible - which both kgdb and the gdb stub is) you should either fully replace existing functionality or extend it. Overlapping it in an incomplete way is not useful to anyone. Extending kgdb to allow the use of it as if we used gdb locally would certainly be interesting - and then you could drop into the kernel anytime as well. But i'm not siding with any particular solution - i'm just seconding Peter's point that there's very clear overlap and inconsistency, and that ought to be resolved one way or another. We dont want to separate facilities for the same conceptual thing: examining application state (be that in user-space and kernel-space). This seems like a shallow sort of consistency. kgdb was added after ptrace existed -- why not extend ptrace instead to target the kernel? After all, it's examining application state. The answer is that it doesn't make a heck of a lot of sense. kgdb simply used gdb's preferred way of remote debugging. That's certainly the ugliest bit of it btw - but it's an externality to kgdb. Had it extended ptrace it wouldnt have gdb compatibility. So i think this example of yours is inapposite as well. Having said all that, i certainly subscribe to the view that neither kgdb nor ptrace is particularly cleanly done. So i wouldnt mind if something new existed that had a modern, flexible, extensible and generally pleasant interface and implementation. If you are heading in that direction, please let me know. Btw., perf does meet that definition: it functionally replaces all facilities that it overlaps/extends - such as Oprofile. [...] (And they currently separately coexist.) You didnt get my point apparently. Keeping the overlapped facility for compatibility (and general user inertia) is fine. Creating a new facility that doesnt do everything that the existing facility does, and not integrating it either, is not fine. oprofile and perfctr are closer in concept than kgdb and ptrace, yet AFAIK perfctr doesn't interface to oprofile, except perhaps to the extent of resolving contention over the underlying physical resources. In any case this is not a great analogy. (FYI, 'perfctr' is a different project that has existed for years, i suspect you meant perf events?) perf replaces oprofile functionally. If the in-kernel gdb stub replaced kgdb functionally you'd hear no complaints from me. Ingo
Re: [RFC] [PATCH] In-kernel gdbstub based on utrace Infrastructure.
On Mon, 2009-11-30 at 17:33 +0530, Srikar Dronamraju wrote: This patch implements an in-kernel gdb stub. It provides an interface between gdb and Linux Kernel by implementing the remote serial protocol. This gdbstub uses utrace infrastructure. This patch provides register set access, signal mapping, process event handling, input/output operations. /proc/pid/gdb was chosen as file for gdb to interact with the process through remote serial protocol. Hence users would have to use target remote /proc/pid/gdb command on gdb prompt to start using this infrastructure. For Breakpointing support, gdbstub needs User space breakpointing layer and uprobes layer which will be posted later. How does this compare to kgdb and related efforts?
Re: [RFC] [PATCH] In-kernel gdbstub based on utrace Infrastructure.
* Peter Zijlstra pet...@infradead.org [2009-11-30 13:09:12]: On Mon, 2009-11-30 at 17:33 +0530, Srikar Dronamraju wrote: This patch implements an in-kernel gdb stub. It provides an interface between gdb and Linux Kernel by implementing the remote serial protocol. This gdbstub uses utrace infrastructure. This patch provides register set access, signal mapping, process event handling, input/output operations. /proc/pid/gdb was chosen as file for gdb to interact with the process through remote serial protocol. Hence users would have to use target remote /proc/pid/gdb command on gdb prompt to start using this infrastructure. For Breakpointing support, gdbstub needs User space breakpointing layer and uprobes layer which will be posted later. How does this compare to kgdb and related efforts? This is a In-kernel gdbstub to debug user space programs. This stub doesnt help in debugging kernel. Hence I am not sure how to compare kgdb gdbstub with this gdbstub. Can you please provide more pointers on what you were referring to? -- Thanks and Regards Srikar
Re: [RFC] [PATCH] In-kernel gdbstub based on utrace Infrastructure.
On Mon, 2009-11-30 at 18:02 +0530, Srikar Dronamraju wrote: * Peter Zijlstra pet...@infradead.org [2009-11-30 13:09:12]: On Mon, 2009-11-30 at 17:33 +0530, Srikar Dronamraju wrote: This patch implements an in-kernel gdb stub. It provides an interface between gdb and Linux Kernel by implementing the remote serial protocol. This gdbstub uses utrace infrastructure. This patch provides register set access, signal mapping, process event handling, input/output operations. /proc/pid/gdb was chosen as file for gdb to interact with the process through remote serial protocol. Hence users would have to use target remote /proc/pid/gdb command on gdb prompt to start using this infrastructure. For Breakpointing support, gdbstub needs User space breakpointing layer and uprobes layer which will be posted later. How does this compare to kgdb and related efforts? This is a In-kernel gdbstub to debug user space programs. This stub doesnt help in debugging kernel. Hence I am not sure how to compare kgdb gdbstub with this gdbstub. Can you please provide more pointers on what you were referring to? Well, not even that much was clear from your changelog, so I wasn't really sure wtf I was looking at. All it says was an in-kernel gdb stub, what other than to debug the kernel would you need in-kernel stubs for? So now my question is, what do you need an in-kernel stub to debug userspace for? In general, tell me about this patch thing, what does it do, why does it do it, and how does it improve on the current situation. Your changelog doesn't address any of those things, so wth are we supposed to think?
Re: [RFC] [PATCH] In-kernel gdbstub based on utrace Infrastructure.
* Peter Zijlstra pet...@infradead.org [2009-11-30 13:41:47]: This is a In-kernel gdbstub to debug user space programs. This stub doesnt help in debugging kernel. Hence I am not sure how to compare kgdb gdbstub with this gdbstub. Can you please provide more pointers on what you were referring to? Well, not even that much was clear from your changelog, so I wasn't really sure wtf I was looking at. All it says was an in-kernel gdb stub, what other than to debug the kernel would you need in-kernel stubs for? So now my question is, what do you need an in-kernel stub to debug userspace for? This stub would allow users use features provided by utrace but through a gdb interface. This idea was brought up in this year's Tracing roundtable at the Linux Foundation Collaboration summit, April 8-10 in San Francisco. Here is the link to the minutes of the meeting sent by Christoph Hellwig. http://www.mail-archive.com/utrace-devel@redhat.com/msg00830.html In general, tell me about this patch thing, what does it do, why does it do it, and how does it improve on the current situation. This is suppose to be one of the interfaces to use utrace, i.e Allow gdb to use utrace features without having to change gdb itself. Though there are not enough features in this patch, intentions include support multi-threaded debugging, concurrent debugger support for same process, syscall tracing. For Breakpoint support(not yet submitted to LKML), it would use execution out of line rather than the conventional inline-single stepping. I guess Christoph, Roland and Frank would be able to explain in a better fashion the rational and advantages of this stub over convential gdb. Your changelog doesn't address any of those things, so wth are we supposed to think? Thanks for pointing out. -- Thanks and Regards Srikar
Re: [RFC] [PATCH] In-kernel gdbstub based on utrace Infrastructure.
On Mon, 2009-11-30 at 18:49 +0530, Srikar Dronamraju wrote: This is suppose to be one of the interfaces to use utrace, i.e Allow gdb to use utrace features without having to change gdb itself. Though there are not enough features in this patch, intentions include support multi-threaded debugging, concurrent debugger support for same process, syscall tracing. For Breakpoint support(not yet submitted to LKML), it would use execution out of line rather than the conventional inline-single stepping. I guess Christoph, Roland and Frank would be able to explain in a better fashion the rational and advantages of this stub over convential gdb. Hmm,. wouldn't it make much more sense to extend the current kgdb stub to include userspace debugging, providing an all-in-one solution? I think it would be much more powerful to be able to observe the full software stack and not be limited by this user-kernel barrier. (Provided the user has sufficient privileges of course).
Re: [RFC] [PATCH] In-kernel gdbstub based on utrace Infrastructure.
I guess Christoph, Roland and Frank would be able to explain in a better fashion the rational and advantages of this stub over convential gdb. Hmm,. wouldn't it make much more sense to extend the current kgdb stub to include userspace debugging, providing an all-in-one solution? I see two limitations but I guess there could be ways to get over it. 1. gdb requiring file that needs to be debugged. I always thought it can either be a user program or a vmlinux file. gdb makes most of the information(i.e registers) from the remote protocol to display the backtrace, variable values and the like by reading this file. 2. Also I am not sure if gdb has a way to tell the remote to switch the context and provide information(registers) pertaining to the user mode/ kernel mode. There could be other limitations too that I may not be aware. I think it would be much more powerful to be able to observe the full software stack and not be limited by this user-kernel barrier. (Provided the user has sufficient privileges of course). In this implementation, the current user can debug his/her own processes. May be if we can debug both the context from same gdb then we might have to place restrictions.. -- Thanks and Regards Srikar
Re: [RFC] [PATCH] In-kernel gdbstub based on utrace Infrastructure.
peterz wrote: Hmm,. wouldn't it make much more sense to extend the current kgdb stub to include userspace debugging, providing an all-in-one solution? I think it would be much more powerful to be able to observe the full software stack and not be limited by this user-kernel barrier. There exist other tools for this broad a scope (systemtap being one), and present gdb is not well suited for this. That makes this idea an exciting potential for the future, but not a practical short-term goal. - FChE
Re: [RFC] [PATCH] In-kernel gdbstub based on utrace Infrastructure.
* Frank Ch. Eigler f...@redhat.com wrote: peterz wrote: Hmm,. wouldn't it make much more sense to extend the current kgdb stub to include userspace debugging, providing an all-in-one solution? I think it would be much more powerful to be able to observe the full software stack and not be limited by this user-kernel barrier. There exist other tools for this broad a scope (systemtap being one), and present gdb is not well suited for this. That makes this idea an exciting potential for the future, but not a practical short-term goal. Well, but Peter's suggestion is the obvious next step - or even a necessary first step in my view. kgdb exists here and today in the kernel and you cannot just build a facility that doesnt replace it and doesnt integrate well with it. So if a unified user/kernel model for debugging is a 'long term' feature in your view then perhaps this framework (which introduces _extensive_ hooks all around the kernel) is not designed/approached in the right way and should not be merged in this form. Concentrating on 'other tools' just generates extensive dependencies on something that is lacking - making it even harder to implement unified debugging down the line. Thanks, Ingo
Re: [RFC] [PATCH] In-kernel gdbstub based on utrace Infrastructure.
On Mon, Nov 30, 2009 at 04:16:50PM +0100, Ingo Molnar wrote: [...] kgdb exists here and today in the kernel and you cannot just build a facility that doesnt replace it and doesnt integrate well with it. Surely you don't mean that: every non-kgdb facility in the kernel meets that definition, even all debugging-related facilities such as perf and ftrace. So if a unified user/kernel model for debugging is a 'long term' feature in your view then perhaps this framework (which introduces _extensive_ hooks all around the kernel) is not designed/approached in the right way and should not be merged in this form. Which this framework are you talking about? Please clarify what exactly you're trying to say. - FChE
