Re: [PATCH v1] dynamic_debug: add support for logs destination
śr., 18 paź 2023 o 05:08 napisał(a): > > adding in Jason, not sure how he got missed. > > On Mon, Oct 16, 2023 at 9:13 AM Łukasz Bartosik wrote: > > > > czw., 12 paź 2023 o 20:48 napisał(a): > > > > > > > If you want the kernel to keep separate flight recorders I guess we > > > > could > > > > add that, but I don't think it currently exists for the dyndbg stuff at > > > > least. Maybe a flight recorder v2 feature, once the basics are in. > > > > > > > > > > dyndbg has +pwrites to syslog > > > +T would separately independently write the same to global trace > > > > > > This would allow graceful switchover to tracefs, > > > without removing logging from dmesg, where most folks > > > (and any monitor tools) would expect it. > > > > > > Lukas (iiuc) wants to steer each site to just 1 destination. > > > Or maybe (in addition to +p > syslog) one trace destination, > > > either global via events, or a separate tracebuf > > > > > > Im ambivalent, but thinking the smooth rollover from syslog to trace > > > might be worth having to ease migration / weaning off syslog. > > > > > > And we have a 4 byte hole in struct _ddebug we could just use. > > > > I'm glad you brought that up. This means we can leave class_id field > > untouched, have separate +T in flags (for consistency) > > and dst_id can be easily 8 bits wide. > > > > Also can you point me to the latest version of writing debug logs to > > trace events (+T option). > > I would like to base trace instances work on that because both are > > closely related. > > > > Ive got 3 branches, all stacked up on rc6 > (last I checked, they were clean on drm-tip) > > https://github.com/jimc/linux/tree/dd-fix-7c > the regression fix, reposting this version next. > it also grabs another flag bit: _DPRINTK_FLAGS_INCL_LOOKUP > > > https://github.com/jimc/linux/tree/dd-shrink-3b > split modname,filename,function to new __dyndbg_sites section > loads the 3 column values and their intervals into 3 maple trees > and implements 3 accessor functions to retrieve the vals > from the descriptor addresses. > it "works" but doesnt erase intervals properly on rmmod > it also claims another flag - _DPRINTK_FLAGS_PREFIX_CACHED > and uses it to mark cached prefix fragment that get created for enabled calls. > > https://github.com/jimc/linux/tree/dd-kitchen-sink > this adds the +T flag stuff. > its still a little fuzzy, esp around the descriptor arg - > using it to render the prefix str at buffer-render time > got UNSAFE warnings - likely due to loadable module > descriptors which could or did go away between > capture and render (after rmmod) > Great, I will use commits related to adding +T from this tree as a baseline for adding trace instances. > > > > > > Unless the align 8 is optional on 32-bits, > > > > I verified with "gcc -g -m32 ..." that the align(8) is honored on 32 bits. > > > > this is sorta the opposite of what I was probing, but I think result > is the same. > istm align(8) is only for JUMP_LABEL, nothing else in the struct > appears to need it > so moving it to the top trades the hole for padding. > Ahh I see what you meant now. Although I have to admit that looking at the jump label code and static key usage cases around the kernel code I don't see where align(8) requirement for jump label comes from. > > > > > I think we're never gonna close the hole anywhere. > > > > > > > :) > > > > > is align 8 a generic expression of an architectural simplifying > > > constraint ? > > > or a need for 1-7 ptr offsets ? > > > > > > > > > > > > > > > > > That's my idea of it. It is interesting to see how far the > > > > > requirements > > > > > can be reasonably realised. > > > > > > > > I think aside from the "make it available directly to unpriviledged > > > > userspace" everything sounds reasonable and doable. > > > > > > > > More on the process side of things, I think Jim is very much looking for > > > > acks and tested-by by people who are interested in better drm logging > > > > infra. That should help that things are moving in a direction that's > > > > actually useful, even when it's not yet entirely complete. > > > > > > > > > > yes, please. Now posted at > > > > > > https://lore.kernel.org/lkml/20231012172137.3286566-1-jim.cro...@gmail.com/T/#t > > > > > > Lukas, I managed to miss your email in the send phase. > > > please consider yourself a direct recipient :-) > > > > > > thanks everyone > > > > > > > Cheers, Sima > > > > -- > > > > Daniel Vetter > > > > Software Engineer, Intel Corporation > > > > http://blog.ffwll.ch
Re: [PATCH v1] dynamic_debug: add support for logs destination
adding in Jason, not sure how he got missed. On Mon, Oct 16, 2023 at 9:13 AM Łukasz Bartosik wrote: > > czw., 12 paź 2023 o 20:48 napisał(a): > > > > > If you want the kernel to keep separate flight recorders I guess we could > > > add that, but I don't think it currently exists for the dyndbg stuff at > > > least. Maybe a flight recorder v2 feature, once the basics are in. > > > > > > > dyndbg has +pwrites to syslog > > +T would separately independently write the same to global trace > > > > This would allow graceful switchover to tracefs, > > without removing logging from dmesg, where most folks > > (and any monitor tools) would expect it. > > > > Lukas (iiuc) wants to steer each site to just 1 destination. > > Or maybe (in addition to +p > syslog) one trace destination, > > either global via events, or a separate tracebuf > > > > Im ambivalent, but thinking the smooth rollover from syslog to trace > > might be worth having to ease migration / weaning off syslog. > > > > And we have a 4 byte hole in struct _ddebug we could just use. > > I'm glad you brought that up. This means we can leave class_id field > untouched, have separate +T in flags (for consistency) > and dst_id can be easily 8 bits wide. > > Also can you point me to the latest version of writing debug logs to > trace events (+T option). > I would like to base trace instances work on that because both are > closely related. > Ive got 3 branches, all stacked up on rc6 (last I checked, they were clean on drm-tip) https://github.com/jimc/linux/tree/dd-fix-7c the regression fix, reposting this version next. it also grabs another flag bit: _DPRINTK_FLAGS_INCL_LOOKUP https://github.com/jimc/linux/tree/dd-shrink-3b split modname,filename,function to new __dyndbg_sites section loads the 3 column values and their intervals into 3 maple trees and implements 3 accessor functions to retrieve the vals from the descriptor addresses. it "works" but doesnt erase intervals properly on rmmod it also claims another flag - _DPRINTK_FLAGS_PREFIX_CACHED and uses it to mark cached prefix fragment that get created for enabled calls. https://github.com/jimc/linux/tree/dd-kitchen-sink this adds the +T flag stuff. its still a little fuzzy, esp around the descriptor arg - using it to render the prefix str at buffer-render time got UNSAFE warnings - likely due to loadable module descriptors which could or did go away between capture and render (after rmmod) > > Unless the align 8 is optional on 32-bits, > > I verified with "gcc -g -m32 ..." that the align(8) is honored on 32 bits. > this is sorta the opposite of what I was probing, but I think result is the same. istm align(8) is only for JUMP_LABEL, nothing else in the struct appears to need it so moving it to the top trades the hole for padding. > > I think we're never gonna close the hole anywhere. > > > > :) > > > is align 8 a generic expression of an architectural simplifying constraint ? > > or a need for 1-7 ptr offsets ? > > > > > > > > > > > > That's my idea of it. It is interesting to see how far the requirements > > > > can be reasonably realised. > > > > > > I think aside from the "make it available directly to unpriviledged > > > userspace" everything sounds reasonable and doable. > > > > > > More on the process side of things, I think Jim is very much looking for > > > acks and tested-by by people who are interested in better drm logging > > > infra. That should help that things are moving in a direction that's > > > actually useful, even when it's not yet entirely complete. > > > > > > > yes, please. Now posted at > > > > https://lore.kernel.org/lkml/20231012172137.3286566-1-jim.cro...@gmail.com/T/#t > > > > Lukas, I managed to miss your email in the send phase. > > please consider yourself a direct recipient :-) > > > > thanks everyone > > > > > Cheers, Sima > > > -- > > > Daniel Vetter > > > Software Engineer, Intel Corporation > > > http://blog.ffwll.ch
Re: [PATCH v1] dynamic_debug: add support for logs destination
czw., 12 paź 2023 o 20:48 napisał(a): > > > If you want the kernel to keep separate flight recorders I guess we could > > add that, but I don't think it currently exists for the dyndbg stuff at > > least. Maybe a flight recorder v2 feature, once the basics are in. > > > > dyndbg has +pwrites to syslog > +T would separately independently write the same to global trace > > This would allow graceful switchover to tracefs, > without removing logging from dmesg, where most folks > (and any monitor tools) would expect it. > > Lukas (iiuc) wants to steer each site to just 1 destination. > Or maybe (in addition to +p > syslog) one trace destination, > either global via events, or a separate tracebuf > > Im ambivalent, but thinking the smooth rollover from syslog to trace > might be worth having to ease migration / weaning off syslog. > > And we have a 4 byte hole in struct _ddebug we could just use. I'm glad you brought that up. This means we can leave class_id field untouched, have separate +T in flags (for consistency) and dst_id can be easily 8 bits wide. Also can you point me to the latest version of writing debug logs to trace events (+T option). I would like to base trace instances work on that because both are closely related. > Unless the align 8 is optional on 32-bits, I verified with "gcc -g -m32 ..." that the align(8) is honored on 32 bits. > I think we're never gonna close the hole anywhere. > :) > is align 8 a generic expression of an architectural simplifying constraint ? > or a need for 1-7 ptr offsets ? > > > > > > > That's my idea of it. It is interesting to see how far the requirements > > > can be reasonably realised. > > > > I think aside from the "make it available directly to unpriviledged > > userspace" everything sounds reasonable and doable. > > > > More on the process side of things, I think Jim is very much looking for > > acks and tested-by by people who are interested in better drm logging > > infra. That should help that things are moving in a direction that's > > actually useful, even when it's not yet entirely complete. > > > > yes, please. Now posted at > > https://lore.kernel.org/lkml/20231012172137.3286566-1-jim.cro...@gmail.com/T/#t > > Lukas, I managed to miss your email in the send phase. > please consider yourself a direct recipient :-) > > thanks everyone > > > Cheers, Sima > > -- > > Daniel Vetter > > Software Engineer, Intel Corporation > > http://blog.ffwll.ch
Re: [PATCH v1] dynamic_debug: add support for logs destination
On Thu, 12 Oct 2023 11:53:52 +0200 Daniel Vetter wrote: > > You said that turning the kernel ring buffer contents into strings is a > > very heavy operation, so it is not possible to push this scope > > separation to userspace, right? > > I think it's the kernel that does the formatting, but honestly not sure > how this works with perf traces. Might be that it's actually userspace > doing the formatting later on so that it doesn't incur the overhead while > recording. perf and trace-cmd do the formatting in user space via libtraceevent: git://git.kernel.org/pub/scm/libs/libtrace/libtraceevent.git It reads the format files of the events: /sys/kernel/tracing/events/*/*/format and uses that to read the raw data saved from the kernel into human readable output. Note, this means that addresses coming from kernel trace events are not hashed! -- Steve
Re: [PATCH v1] dynamic_debug: add support for logs destination
> If you want the kernel to keep separate flight recorders I guess we could > add that, but I don't think it currently exists for the dyndbg stuff at > least. Maybe a flight recorder v2 feature, once the basics are in. > dyndbg has +pwrites to syslog +T would separately independently write the same to global trace This would allow graceful switchover to tracefs, without removing logging from dmesg, where most folks (and any monitor tools) would expect it. Lukas (iiuc) wants to steer each site to just 1 destination. Or maybe (in addition to +p > syslog) one trace destination, either global via events, or a separate tracebuf Im ambivalent, but thinking the smooth rollover from syslog to trace might be worth having to ease migration / weaning off syslog. And we have a 4 byte hole in struct _ddebug we could just use. Unless the align 8 is optional on 32-bits, I think we're never gonna close the hole anywhere. is align 8 a generic expression of an architectural simplifying constraint ? or a need for 1-7 ptr offsets ? > > That's my idea of it. It is interesting to see how far the requirements > > can be reasonably realised. > > I think aside from the "make it available directly to unpriviledged > userspace" everything sounds reasonable and doable. > > More on the process side of things, I think Jim is very much looking for > acks and tested-by by people who are interested in better drm logging > infra. That should help that things are moving in a direction that's > actually useful, even when it's not yet entirely complete. > yes, please. Now posted at https://lore.kernel.org/lkml/20231012172137.3286566-1-jim.cro...@gmail.com/T/#t Lukas, I managed to miss your email in the send phase. please consider yourself a direct recipient :-) thanks everyone > Cheers, Sima > -- > Daniel Vetter > Software Engineer, Intel Corporation > http://blog.ffwll.ch
Re: [PATCH v1] dynamic_debug: add support for logs destination
On Thu, Oct 12, 2023 at 01:39:44PM +0300, Pekka Paalanen wrote: > On Thu, 12 Oct 2023 11:53:52 +0200 > Daniel Vetter wrote: > > > On Thu, Oct 12, 2023 at 11:55:48AM +0300, Pekka Paalanen wrote: > > > On Wed, 11 Oct 2023 11:42:24 +0200 > > > Daniel Vetter wrote: > > > > > > > On Wed, Oct 11, 2023 at 11:48:16AM +0300, Pekka Paalanen wrote: > > ... > > > > > > - all selections tailored separately for each userspace subscriber > > > > > (- per open device file description selection of messages) > > > > > > > > Again this feels like a userspace problem. Sessions could register what > > > > kind of info they need for their session, and something like journald > > > > can > > > > figure out how to record it all. > > > > > > Only if the kernel actually attaches all the required information to > > > the debug messages *in machine readable form* so that userspace > > > actually can do the filtering. And that makes *that* information UABI. > > > Maybe that's fine? I wouldn't know. > > > > Well if you configure the filters to go into separate ringbuffers for each > > session (or whatever you want to split) it also becomes uapi. > > It's a different UAPI: filter configuration vs. message structure. I > don't mind which it is, I just suspect one is easier to maintain and > extend than the other. > > > Also I'd say that for the first cut just getting the logs out on demand > > should be good enough, multi-gpu (or multi-compositor) systems are a step > > further. We can figure those out when we get there. > > This reminds me of what you recently said in IRC about a very different > topic: > >swick[m], tell this past me roughly 10 years ago, would > have been easy to add into the design back when there was no > driver code yet > > I just want to mention today everything I can see as useful. It's up to > the people doing the actual work to decide what they include and how. I actually pondered this a bit more today, and I think even with hindsight the atomic design we ended up with was probably rather close to optimal. Sure there's a bunch of things that would have been nice to include, but another very hard requirement of atomic was that it's feasible to convert current drivers over to it. And I think going full free-standing state structures with unlimited (at least at the design level) queue depth would have been a bridge too far. The hacks and conversion helpers are all gone by now, but "you can just peek at the object struct to get your state" was a huge help in reducing the conversion churn. But it definitely resulted in a big price we're still paying. tldr I don't think getting somewhere useful, even if somewhat deficient, is bad. -Sima -- Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch
Re: [PATCH v1] dynamic_debug: add support for logs destination
On Thu, 12 Oct 2023 11:53:52 +0200 Daniel Vetter wrote: > On Thu, Oct 12, 2023 at 11:55:48AM +0300, Pekka Paalanen wrote: > > On Wed, 11 Oct 2023 11:42:24 +0200 > > Daniel Vetter wrote: > > > > > On Wed, Oct 11, 2023 at 11:48:16AM +0300, Pekka Paalanen wrote: ... > > > > - all selections tailored separately for each userspace subscriber > > > > (- per open device file description selection of messages) > > > > > > Again this feels like a userspace problem. Sessions could register what > > > kind of info they need for their session, and something like journald can > > > figure out how to record it all. > > > > Only if the kernel actually attaches all the required information to > > the debug messages *in machine readable form* so that userspace > > actually can do the filtering. And that makes *that* information UABI. > > Maybe that's fine? I wouldn't know. > > Well if you configure the filters to go into separate ringbuffers for each > session (or whatever you want to split) it also becomes uapi. It's a different UAPI: filter configuration vs. message structure. I don't mind which it is, I just suspect one is easier to maintain and extend than the other. > Also I'd say that for the first cut just getting the logs out on demand > should be good enough, multi-gpu (or multi-compositor) systems are a step > further. We can figure those out when we get there. This reminds me of what you recently said in IRC about a very different topic: swick[m], tell this past me roughly 10 years ago, would have been easy to add into the design back when there was no driver code yet I just want to mention today everything I can see as useful. It's up to the people doing the actual work to decide what they include and how. Thanks, pq pgpGben6913p0.pgp Description: OpenPGP digital signature
Re: [PATCH v1] dynamic_debug: add support for logs destination
On Thu, Oct 12, 2023 at 11:55:48AM +0300, Pekka Paalanen wrote: > On Wed, 11 Oct 2023 11:42:24 +0200 > Daniel Vetter wrote: > > > On Wed, Oct 11, 2023 at 11:48:16AM +0300, Pekka Paalanen wrote: > > > On Tue, 10 Oct 2023 10:06:02 -0600 > > > jim.cro...@gmail.com wrote: > > > > > > > since I name-dropped you all, > > > > > > Hi everyone, > > > > > > I'm really happy to see this topic being developed! I've practically > > > forgot about it myself, but the need for it has not diminished at all. > > > > > > I didn't understand much of the conversation, so I'll just reiterate > > > what I would use it for, as a Wayland compositor developer. > > > > > > I added a few more cc's to get better coverage of DRM and Wayland > > > compositor developers. > > > > > > > On Tue, Oct 10, 2023 at 10:01 AM wrote: > > > > > > > > > > On Mon, Oct 9, 2023 at 4:47 PM Łukasz Bartosik > > > > > wrote: > > > > > > ... > > > > > > > > > I don't have a real life use case to configure different trace > > > > > > instance for each callsite. > > > > > > I just tried to be as much flexible as possible. > > > > > > > > > > > > > > > > Ive come around to agree - I looked back at some old threads > > > > > (that I was a part of, and barely remembered :-} > > > > > > > > > > At least Sean Paul, Lyude, Simon Ser, Pekka Paalanen > > > > > have expressed a desire for a "flight-recorder" > > > > > it'd be hard to say now that 2-3 such buffers would always be enough, > > > > > esp as theres a performance reason for having your own. > > > > > > A Wayland compositor has roughly three important things where the kernel > > > debugs might come in handy: > > > - input > > > - DRM KMS > > > - DRM GPU rendering > > > > > > DRM KMS is the one I've been thinking of in the flight recorder context > > > the most, because KMS hardware varies a lot, and there is plenty of > > > room for both KMS drivers and KMS userspace to go wrong. The usual > > > result is your display doesn't work, so the system is practically > > > unusable to the end user. In the wild, the simplest or maybe the only > > > way out of that may be a reboot, maybe an automated one (e.g. digital > > > signage). In order to debug such problems, we would need both > > > compositor logs and the relevant kernel debug messages. > > > > > > For example, Weston already has a flight recorder framework of its own, > > > so we have the compositor debug logs. It would be useful to get the > > > selected kernel debug logs in the same place. It could be used for > > > automated or semi-manual bug reporting, for example, making the > > > administrator or end user life much easier reporting issues. > > > > > > Since this is usually a production environment, and the Wayland > > > compositor runs without root privileges, we need something that works > > > with that. We would likely want the kernel debug messages in the > > > compositor to combine and order them properly with the compositor debug > > > messages. > > > > > > It's quite likely that developers would like to pick and choose which > > > kernel debug messages might be interesting enough to record, to avoid > > > excessive log flooding. The flight recorder in Weston is fixed size to > > > avoid running out of memory or disk space. I can also see that Weston > > > could have debugging options that affect which kernel debug messages it > > > subscribes to. We can have a reasonable default setup that allows us to > > > pinpoint the problem area and figure out most problems, and if needed, > > > we could ask the administrator pass another debug option to Weston. It > > > helps if there is just one place to configure everything about the > > > compositor. > > > > > > This implies that it would be really nice to have userspace subscriber > > > specific debug message streams from the kernel, or a good way to filter > > > the messages we want. A Wayland compositor would not be interested in > > > file system or wireless debugs for example, but another system > > > component might be. There is also a security aspect of which component is > > > allowed to see which messages in case they could contain anything > > > sensitive (input debug could contain typed passwords). > > > > > > Configuring the kernel debug message selection for our debug message > > > stream can and probably should require elevated privileges, and we can > > > likely solve that in userspace with a daemon or such, to allow the > > > Wayland compositor to run as a regular user. > > > > > > Thinking of desktop systems, and especially physically multi-seat systems: > > > - there can be multiple different Wayland compositors running > > > simultaneously > > > - each of them may want debug messages only from a specific DRM KMS > > > device instance, and not from others > > > - each of them may have a different idea of which debug messages are > > > important > > > - because DRM KMS leasing is a thing, different compositor instances > > > could be using
Re: [PATCH v1] dynamic_debug: add support for logs destination
On Wed, 11 Oct 2023 11:42:24 +0200 Daniel Vetter wrote: > On Wed, Oct 11, 2023 at 11:48:16AM +0300, Pekka Paalanen wrote: > > On Tue, 10 Oct 2023 10:06:02 -0600 > > jim.cro...@gmail.com wrote: > > > > > since I name-dropped you all, > > > > Hi everyone, > > > > I'm really happy to see this topic being developed! I've practically > > forgot about it myself, but the need for it has not diminished at all. > > > > I didn't understand much of the conversation, so I'll just reiterate > > what I would use it for, as a Wayland compositor developer. > > > > I added a few more cc's to get better coverage of DRM and Wayland > > compositor developers. > > > > > On Tue, Oct 10, 2023 at 10:01 AM wrote: > > > > > > > > On Mon, Oct 9, 2023 at 4:47 PM Łukasz Bartosik > > > > wrote: > > > > ... > > > > > > > I don't have a real life use case to configure different trace > > > > > instance for each callsite. > > > > > I just tried to be as much flexible as possible. > > > > > > > > > > > > > Ive come around to agree - I looked back at some old threads > > > > (that I was a part of, and barely remembered :-} > > > > > > > > At least Sean Paul, Lyude, Simon Ser, Pekka Paalanen > > > > have expressed a desire for a "flight-recorder" > > > > it'd be hard to say now that 2-3 such buffers would always be enough, > > > > esp as theres a performance reason for having your own. > > > > A Wayland compositor has roughly three important things where the kernel > > debugs might come in handy: > > - input > > - DRM KMS > > - DRM GPU rendering > > > > DRM KMS is the one I've been thinking of in the flight recorder context > > the most, because KMS hardware varies a lot, and there is plenty of > > room for both KMS drivers and KMS userspace to go wrong. The usual > > result is your display doesn't work, so the system is practically > > unusable to the end user. In the wild, the simplest or maybe the only > > way out of that may be a reboot, maybe an automated one (e.g. digital > > signage). In order to debug such problems, we would need both > > compositor logs and the relevant kernel debug messages. > > > > For example, Weston already has a flight recorder framework of its own, > > so we have the compositor debug logs. It would be useful to get the > > selected kernel debug logs in the same place. It could be used for > > automated or semi-manual bug reporting, for example, making the > > administrator or end user life much easier reporting issues. > > > > Since this is usually a production environment, and the Wayland > > compositor runs without root privileges, we need something that works > > with that. We would likely want the kernel debug messages in the > > compositor to combine and order them properly with the compositor debug > > messages. > > > > It's quite likely that developers would like to pick and choose which > > kernel debug messages might be interesting enough to record, to avoid > > excessive log flooding. The flight recorder in Weston is fixed size to > > avoid running out of memory or disk space. I can also see that Weston > > could have debugging options that affect which kernel debug messages it > > subscribes to. We can have a reasonable default setup that allows us to > > pinpoint the problem area and figure out most problems, and if needed, > > we could ask the administrator pass another debug option to Weston. It > > helps if there is just one place to configure everything about the > > compositor. > > > > This implies that it would be really nice to have userspace subscriber > > specific debug message streams from the kernel, or a good way to filter > > the messages we want. A Wayland compositor would not be interested in > > file system or wireless debugs for example, but another system > > component might be. There is also a security aspect of which component is > > allowed to see which messages in case they could contain anything > > sensitive (input debug could contain typed passwords). > > > > Configuring the kernel debug message selection for our debug message > > stream can and probably should require elevated privileges, and we can > > likely solve that in userspace with a daemon or such, to allow the > > Wayland compositor to run as a regular user. > > > > Thinking of desktop systems, and especially physically multi-seat systems: > > - there can be multiple different Wayland compositors running simultaneously > > - each of them may want debug messages only from a specific DRM KMS > > device instance, and not from others > > - each of them may have a different idea of which debug messages are > > important > > - because DRM KMS leasing is a thing, different compositor instances > > could be using the same DRM KMS device instance simultaneously; since > > this is specific to DRM KMS, and it should be harmless to get a > > little too much DRM KMS debug (that is, from the whole device instead > > of just the leased parts), it may not be worth to consider sp
Re: [PATCH v1] dynamic_debug: add support for logs destination
śr., 11 paź 2023 o 11:42 Daniel Vetter napisał(a): > > On Wed, Oct 11, 2023 at 11:48:16AM +0300, Pekka Paalanen wrote: > > On Tue, 10 Oct 2023 10:06:02 -0600 > > jim.cro...@gmail.com wrote: > > > > > since I name-dropped you all, > > > > Hi everyone, > > > > I'm really happy to see this topic being developed! I've practically > > forgot about it myself, but the need for it has not diminished at all. > > It's good to hear you guys are also interested in this feature. > > I didn't understand much of the conversation, so I'll just reiterate > > what I would use it for, as a Wayland compositor developer. > > > > I added a few more cc's to get better coverage of DRM and Wayland > > compositor developers. > > > > > On Tue, Oct 10, 2023 at 10:01 AM wrote: > > > > > > > > On Mon, Oct 9, 2023 at 4:47 PM Łukasz Bartosik > > > > wrote: > > > > ... > > > > > > > I don't have a real life use case to configure different trace > > > > > instance for each callsite. > > > > > I just tried to be as much flexible as possible. > > > > > > > > > > > > > Ive come around to agree - I looked back at some old threads > > > > (that I was a part of, and barely remembered :-} > > > > > > > > At least Sean Paul, Lyude, Simon Ser, Pekka Paalanen > > > > have expressed a desire for a "flight-recorder" > > > > it'd be hard to say now that 2-3 such buffers would always be enough, > > > > esp as theres a performance reason for having your own. > > > > A Wayland compositor has roughly three important things where the kernel > > debugs might come in handy: > > - input > > - DRM KMS > > - DRM GPU rendering > > > > DRM KMS is the one I've been thinking of in the flight recorder context > > the most, because KMS hardware varies a lot, and there is plenty of > > room for both KMS drivers and KMS userspace to go wrong. The usual > > result is your display doesn't work, so the system is practically > > unusable to the end user. In the wild, the simplest or maybe the only > > way out of that may be a reboot, maybe an automated one (e.g. digital > > signage). In order to debug such problems, we would need both > > compositor logs and the relevant kernel debug messages. > > > > For example, Weston already has a flight recorder framework of its own, > > so we have the compositor debug logs. It would be useful to get the > > selected kernel debug logs in the same place. It could be used for > > automated or semi-manual bug reporting, for example, making the > > administrator or end user life much easier reporting issues. > > > > Since this is usually a production environment, and the Wayland > > compositor runs without root privileges, we need something that works > > with that. We would likely want the kernel debug messages in the > > compositor to combine and order them properly with the compositor debug > > messages. > > > > It's quite likely that developers would like to pick and choose which > > kernel debug messages might be interesting enough to record, to avoid > > excessive log flooding. The flight recorder in Weston is fixed size to > > avoid running out of memory or disk space. I can also see that Weston > > could have debugging options that affect which kernel debug messages it > > subscribes to. We can have a reasonable default setup that allows us to > > pinpoint the problem area and figure out most problems, and if needed, > > we could ask the administrator pass another debug option to Weston. It > > helps if there is just one place to configure everything about the > > compositor. > > > > This implies that it would be really nice to have userspace subscriber > > specific debug message streams from the kernel, or a good way to filter > > the messages we want. A Wayland compositor would not be interested in > > file system or wireless debugs for example, but another system > > component might be. There is also a security aspect of which component is > > allowed to see which messages in case they could contain anything > > sensitive (input debug could contain typed passwords). > > > > Configuring the kernel debug message selection for our debug message > > stream can and probably should require elevated privileges, and we can > > likely solve that in userspace with a daemon or such, to allow the > > Wayland compositor to run as a regular user. > > > > Thinking of desktop systems, and especially physically multi-seat systems: > > - there can be multiple different Wayland compositors running simultaneously > > - each of them may want debug messages only from a specific DRM KMS > > device instance, and not from others > > - each of them may have a different idea of which debug messages are > > important > > - because DRM KMS leasing is a thing, different compositor instances > > could be using the same DRM KMS device instance simultaneously; since > > this is specific to DRM KMS, and it should be harmless to get a > > little too much DRM KMS debug (that is, from the whole device instead > > of just the leased parts), it may n
Re: [PATCH v1] dynamic_debug: add support for logs destination
On Wed, Oct 11, 2023 at 11:48:16AM +0300, Pekka Paalanen wrote: > On Tue, 10 Oct 2023 10:06:02 -0600 > jim.cro...@gmail.com wrote: > > > since I name-dropped you all, > > Hi everyone, > > I'm really happy to see this topic being developed! I've practically > forgot about it myself, but the need for it has not diminished at all. > > I didn't understand much of the conversation, so I'll just reiterate > what I would use it for, as a Wayland compositor developer. > > I added a few more cc's to get better coverage of DRM and Wayland > compositor developers. > > > On Tue, Oct 10, 2023 at 10:01 AM wrote: > > > > > > On Mon, Oct 9, 2023 at 4:47 PM Łukasz Bartosik wrote: > > > > > ... > > > > > I don't have a real life use case to configure different trace > > > > instance for each callsite. > > > > I just tried to be as much flexible as possible. > > > > > > > > > > Ive come around to agree - I looked back at some old threads > > > (that I was a part of, and barely remembered :-} > > > > > > At least Sean Paul, Lyude, Simon Ser, Pekka Paalanen > > > have expressed a desire for a "flight-recorder" > > > it'd be hard to say now that 2-3 such buffers would always be enough, > > > esp as theres a performance reason for having your own. > > A Wayland compositor has roughly three important things where the kernel > debugs might come in handy: > - input > - DRM KMS > - DRM GPU rendering > > DRM KMS is the one I've been thinking of in the flight recorder context > the most, because KMS hardware varies a lot, and there is plenty of > room for both KMS drivers and KMS userspace to go wrong. The usual > result is your display doesn't work, so the system is practically > unusable to the end user. In the wild, the simplest or maybe the only > way out of that may be a reboot, maybe an automated one (e.g. digital > signage). In order to debug such problems, we would need both > compositor logs and the relevant kernel debug messages. > > For example, Weston already has a flight recorder framework of its own, > so we have the compositor debug logs. It would be useful to get the > selected kernel debug logs in the same place. It could be used for > automated or semi-manual bug reporting, for example, making the > administrator or end user life much easier reporting issues. > > Since this is usually a production environment, and the Wayland > compositor runs without root privileges, we need something that works > with that. We would likely want the kernel debug messages in the > compositor to combine and order them properly with the compositor debug > messages. > > It's quite likely that developers would like to pick and choose which > kernel debug messages might be interesting enough to record, to avoid > excessive log flooding. The flight recorder in Weston is fixed size to > avoid running out of memory or disk space. I can also see that Weston > could have debugging options that affect which kernel debug messages it > subscribes to. We can have a reasonable default setup that allows us to > pinpoint the problem area and figure out most problems, and if needed, > we could ask the administrator pass another debug option to Weston. It > helps if there is just one place to configure everything about the > compositor. > > This implies that it would be really nice to have userspace subscriber > specific debug message streams from the kernel, or a good way to filter > the messages we want. A Wayland compositor would not be interested in > file system or wireless debugs for example, but another system > component might be. There is also a security aspect of which component is > allowed to see which messages in case they could contain anything > sensitive (input debug could contain typed passwords). > > Configuring the kernel debug message selection for our debug message > stream can and probably should require elevated privileges, and we can > likely solve that in userspace with a daemon or such, to allow the > Wayland compositor to run as a regular user. > > Thinking of desktop systems, and especially physically multi-seat systems: > - there can be multiple different Wayland compositors running simultaneously > - each of them may want debug messages only from a specific DRM KMS > device instance, and not from others > - each of them may have a different idea of which debug messages are important > - because DRM KMS leasing is a thing, different compositor instances > could be using the same DRM KMS device instance simultaneously; since > this is specific to DRM KMS, and it should be harmless to get a > little too much DRM KMS debug (that is, from the whole device instead > of just the leased parts), it may not be worth to consider splitting > debug message streams this far. > > If userspace is offered some standardised fields in kernel debug > structures, then userspace could do some filtering on its own too, but I > guess it would be better to filter at the source and not need that. > > There is al
Re: [PATCH v1] dynamic_debug: add support for logs destination
On Tue, 10 Oct 2023 10:06:02 -0600 jim.cro...@gmail.com wrote: > since I name-dropped you all, Hi everyone, I'm really happy to see this topic being developed! I've practically forgot about it myself, but the need for it has not diminished at all. I didn't understand much of the conversation, so I'll just reiterate what I would use it for, as a Wayland compositor developer. I added a few more cc's to get better coverage of DRM and Wayland compositor developers. > On Tue, Oct 10, 2023 at 10:01 AM wrote: > > > > On Mon, Oct 9, 2023 at 4:47 PM Łukasz Bartosik wrote: ... > > > I don't have a real life use case to configure different trace > > > instance for each callsite. > > > I just tried to be as much flexible as possible. > > > > > > > Ive come around to agree - I looked back at some old threads > > (that I was a part of, and barely remembered :-} > > > > At least Sean Paul, Lyude, Simon Ser, Pekka Paalanen > > have expressed a desire for a "flight-recorder" > > it'd be hard to say now that 2-3 such buffers would always be enough, > > esp as theres a performance reason for having your own. A Wayland compositor has roughly three important things where the kernel debugs might come in handy: - input - DRM KMS - DRM GPU rendering DRM KMS is the one I've been thinking of in the flight recorder context the most, because KMS hardware varies a lot, and there is plenty of room for both KMS drivers and KMS userspace to go wrong. The usual result is your display doesn't work, so the system is practically unusable to the end user. In the wild, the simplest or maybe the only way out of that may be a reboot, maybe an automated one (e.g. digital signage). In order to debug such problems, we would need both compositor logs and the relevant kernel debug messages. For example, Weston already has a flight recorder framework of its own, so we have the compositor debug logs. It would be useful to get the selected kernel debug logs in the same place. It could be used for automated or semi-manual bug reporting, for example, making the administrator or end user life much easier reporting issues. Since this is usually a production environment, and the Wayland compositor runs without root privileges, we need something that works with that. We would likely want the kernel debug messages in the compositor to combine and order them properly with the compositor debug messages. It's quite likely that developers would like to pick and choose which kernel debug messages might be interesting enough to record, to avoid excessive log flooding. The flight recorder in Weston is fixed size to avoid running out of memory or disk space. I can also see that Weston could have debugging options that affect which kernel debug messages it subscribes to. We can have a reasonable default setup that allows us to pinpoint the problem area and figure out most problems, and if needed, we could ask the administrator pass another debug option to Weston. It helps if there is just one place to configure everything about the compositor. This implies that it would be really nice to have userspace subscriber specific debug message streams from the kernel, or a good way to filter the messages we want. A Wayland compositor would not be interested in file system or wireless debugs for example, but another system component might be. There is also a security aspect of which component is allowed to see which messages in case they could contain anything sensitive (input debug could contain typed passwords). Configuring the kernel debug message selection for our debug message stream can and probably should require elevated privileges, and we can likely solve that in userspace with a daemon or such, to allow the Wayland compositor to run as a regular user. Thinking of desktop systems, and especially physically multi-seat systems: - there can be multiple different Wayland compositors running simultaneously - each of them may want debug messages only from a specific DRM KMS device instance, and not from others - each of them may have a different idea of which debug messages are important - because DRM KMS leasing is a thing, different compositor instances could be using the same DRM KMS device instance simultaneously; since this is specific to DRM KMS, and it should be harmless to get a little too much DRM KMS debug (that is, from the whole device instead of just the leased parts), it may not be worth to consider splitting debug message streams this far. If userspace is offered some standardised fields in kernel debug structures, then userspace could do some filtering on its own too, but I guess it would be better to filter at the source and not need that. There is also an anti-goal. The kernel debug message contents are specifically not machine-parsable. I very much do not want to impose debug strings as ABI, they are for human (and AI?) readers only. As a summary, here are the most important requirements first: - usable in production a
