Re: [PATCH] Re: relayfs documentation sucks?
On Mon, Jul 25, 2005 at 07:47:45PM -0400, Karim Yaghmour wrote: > Now if only I could remember what I talked about after I left the Black > Thorn at 2h45am and the guy in the elevator at Les Suites pressed on a > button and said "'M' for more beer" ... I bet in involved 'M' for more markers, Karim :-) -- http://www.PowerDNS.com Open source, database driven DNS Software http://netherlabs.nl Open and Closed source services - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] Re: relayfs documentation sucks?
Christoph Hellwig wrote: > That beein said I wish LTT folks would make a little more progress so > we could actually include it. We're working on it. On the topic of revamping LTT, 3 different people came up with 3 different implementations. Following your feedback on the patch I sent a few weeks back, I headed out asking myself "what is the bare-minimum tracing functionality that will actually fly while still being flexible enough to add to it?" I spent some time at the OLS comparing notes with others interested in this area, and I think we've got something that should fit the bill. We should be able to post something sooner rather than later. Now if only I could remember what I talked about after I left the Black Thorn at 2h45am and the guy in the elevator at Les Suites pressed on a button and said "'M' for more beer" ... Thanks, Karim -- Author, Speaker, Developer, Consultant Pushing Embedded and Real-Time Linux Systems Beyond the Limits http://www.opersys.com || [EMAIL PROTECTED] || 1-866-677-4546 - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] Re: relayfs documentation sucks?
On Fri, Jul 22, 2005 at 01:01:32PM -0700, Paul Jackson wrote: > Another vote in favor of relayfs here ... > > I am reminded by my good colleagues at SGI that relayfs is a key > to the Linux Trace Toolkit (LTT), which is in turn an important > technology for some product(s) on which SGI is working. I don't think anyone cares for product plans of particular companies. That beein said I wish LTT folks would make a little more progress so we could actually include it. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] Re: relayfs documentation sucks?
Another vote in favor of relayfs here ... I am reminded by my good colleagues at SGI that relayfs is a key to the Linux Trace Toolkit (LTT), which is in turn an important technology for some product(s) on which SGI is working. It is uses such as this which speak to the value of including relayfs in the kernel. -- I won't rest till it's the best ... Programmer, Linux Scalability Paul Jackson <[EMAIL PROTECTED]> 1.925.600.0401 - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] Re: relayfs documentation sucks?
Bert wrote: > the diskstat tools require relayfs That way might lay the real value of relayfs, as a common technology basis for specific tools that are developed and maintained on top of relayfs. -- I won't rest till it's the best ... Programmer, Linux Scalability Paul Jackson <[EMAIL PROTECTED]> 1.925.600.0401 - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] Re: relayfs documentation sucks?
> > When I'm debugging something requiring detailed tracing, I don't want > to have to think about whether the tracing tool has the particular > behaviour, performance, data loss, and other such characteristics > needed for my immediate needs. It is easier to code up some little > ad hoc mechanism than it is to try to figure out whether some general > purpose mechanism is suitable and how to use the generic mechanism. You can do lots of modes with relayfs already - no ping-pong buffer, n-buffer, lossy, not lossy etc etc. I currently use it in 'flight-recorder' mode where new messages overwrite old ones. It might be good to document different possible ways of using relayfs. > If there are enough specific purposes for relayfs, fine. But beware > of over generalizing its potential usefulness. There is always the > risk of over designing it, adding additional flexibility and options > in an effort to gain customers, at the expense of making it less and > less obviously useful in a trivial way for any specific purpose. It's currently pretty limited - but you can add more features on top of it, in a modular fashion. I tend not to use the complex stuff, but you can layer it if you want. It'd be nice if we had some basic relaying infrastructure available that'd cover most needs successfully. Advanced users can do advanced things if they want. Btw, the diskstat tools (http://ds9a.nl/diskstat) require relayfs. It'll be released this Friday or so. -- http://www.PowerDNS.com Open source, database driven DNS Software http://netherlabs.nl Open and Closed source services - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] Re: relayfs documentation sucks?
Steve wrote: > The reason I would like to see this merged, so kernel hackers don't need > to constantly write there own logging buffers everytime you need to > debug a complex area of the kernel. But I doubt that relayfs, or anything resembling it, will accomplish this purpose, at least for some of us, in many such situations. When I'm debugging something requiring detailed tracing, I don't want to have to think about whether the tracing tool has the particular behaviour, performance, data loss, and other such characteristics needed for my immediate needs. It is easier to code up some little ad hoc mechanism than it is to try to figure out whether some general purpose mechanism is suitable and how to use the generic mechanism. Invariably in any particular situation, there is some almost trivial way to hack in something adequate, for very little effort, doing things that would be utterly useless in some other case. Such tracing mechanisms work to obtain major subsystem isolation, by exposing the flow of data and control back and forth across a major boundary, such as using strace for the initial isolation of a problem that might be in user space, or might be in the kernel. But for detailed work within a subsystem, the corners that one can cut with ad hoc tools often make them vastly superior to general purpose tools. Even the best equipped of carpenters sometimes throw together some temporary scaffolding using rough cut 2x4's (2 inch by 4 inch cross section lumbar; I don't know what they're called in metric nations.) If there are enough specific purposes for relayfs, fine. But beware of over generalizing its potential usefulness. There is always the risk of over designing it, adding additional flexibility and options in an effort to gain customers, at the expense of making it less and less obviously useful in a trivial way for any specific purpose. -- I won't rest till it's the best ... Programmer, Linux Scalability Paul Jackson <[EMAIL PROTECTED]> 1.925.600.0401 - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] Re: relayfs documentation sucks?
On Sun, 2005-07-17 at 21:45 +0200, bert hubert wrote: > On Sun, Jul 17, 2005 at 10:43:40AM -0500, Tom Zanussi wrote: > > > It is racey - in this mode, there's nothing to keep the kernel from > > writing as much as it wants before the user side has a chance to read > > any of it. The only way this can be used safely is to make sure the > > kernel side isn't writing anything when the client is reading. This > > would be typical of a flight-recording usage i.e. kernel writes a > > bunch of data continuously, then stops and allows the client to read > > whatever's in there. > > Or by numbering entries written out, when in flight-recording mode you > wouldn't want to block the kernel. Exactly! I've written a logging device to record data in the kernel that a printk can't help with. I've used this in debugging inturrupts, the scheduler, and high speed network packets. Where a printk to a serial would just slow things down, and going to the network is too expensive, and complex if you happen to be debugging the network. This tool is called logdev (http://www.kihontech.com/logdev) and uses a ring buffer that is like the relayfs overwrite mode. It can do printk like records and when something goes wrong, I dump the buffer to the serial. Or I have a user space program reading it from a device. I don't care about anything that happened earlier, I want to only know what happened up to the point I dumped the buffer. Lately, I've been usuing this with Ingo's RT patch, and when the system locks up, I dump the buffer, and it shows quite nicely where the lockup occurred, and why. With Tom's help, I also have a version that uses relayfs as a backend in overwrite mode. It's still a work in progress (so no web site yet!) since there's some issues of using a singe buffer for multiple CPUs. This helps in debugging race conditions since you need to see how events interleave. > > > > In fact, it appears this might even happen in non-overwrite mode. > > > > It shouldn't ever be able to happen in non-overwrite mode - if it > > did, it would be a bug. Can you be more specific as to how you see > > this happening in this mode? > > Yeah - you're right. The misunderstanding is because in both cases > (overwrite and non-overwrite) data is lost, except that in one case you lose > old data, and in the other new data. > > It might be a good idea to document this as well. > > Btw, I've already uncovered interesting things using relayfs, but I still > don't see the case for having it merged :-) The reason I would like to see this merged, so kernel hackers don't need to constantly write there own logging buffers everytime you need to debug a complex area of the kernel. -- Steve - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] Re: relayfs documentation sucks?
bert hubert writes: > On Sun, Jul 17, 2005 at 10:43:40AM -0500, Tom Zanussi wrote: > > > It is racey - in this mode, there's nothing to keep the kernel from > > writing as much as it wants before the user side has a chance to read > > any of it. The only way this can be used safely is to make sure the > > kernel side isn't writing anything when the client is reading. This > > would be typical of a flight-recording usage i.e. kernel writes a > > bunch of data continuously, then stops and allows the client to read > > whatever's in there. > > Or by numbering entries written out, when in flight-recording mode you > wouldn't want to block the kernel. > > > > In fact, it appears this might even happen in non-overwrite mode. > > > > It shouldn't ever be able to happen in non-overwrite mode - if it > > did, it would be a bug. Can you be more specific as to how you see > > this happening in this mode? > > Yeah - you're right. The misunderstanding is because in both cases > (overwrite and non-overwrite) data is lost, except that in one case you lose > old data, and in the other new data. Just to clarify - in either mode, if you don't have a consumer or the consumer can't keep up with the amount of data being written by the kernel, you will of course lose data at some point. Normally you wouldn't want to lose data; by using non-overwrite mode you're implicitly letting relayfs know this i.e. if at any point all the sub-buffers remain unread and the kernel is still trying to write into them, let the client know (via the buffer-full callback) that this has happened. Presumably you would then increase the buffer size or have the kernel write less etc. > > It might be a good idea to document this as well. > Yes, I'll make it more explicit in the documentation. > Btw, I've already uncovered interesting things using relayfs, but I still > don't see the case for having it merged :-) Glad to hear it. Can you say what if anything would convince you it should be merged? > > Thanks for your answers, I think I get it all now. No problem, and thanks for patch and other suggestions. Tom - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] Re: relayfs documentation sucks?
On Sun, Jul 17, 2005 at 10:43:40AM -0500, Tom Zanussi wrote: > It is racey - in this mode, there's nothing to keep the kernel from > writing as much as it wants before the user side has a chance to read > any of it. The only way this can be used safely is to make sure the > kernel side isn't writing anything when the client is reading. This > would be typical of a flight-recording usage i.e. kernel writes a > bunch of data continuously, then stops and allows the client to read > whatever's in there. Or by numbering entries written out, when in flight-recording mode you wouldn't want to block the kernel. > > In fact, it appears this might even happen in non-overwrite mode. > > It shouldn't ever be able to happen in non-overwrite mode - if it > did, it would be a bug. Can you be more specific as to how you see > this happening in this mode? Yeah - you're right. The misunderstanding is because in both cases (overwrite and non-overwrite) data is lost, except that in one case you lose old data, and in the other new data. It might be a good idea to document this as well. Btw, I've already uncovered interesting things using relayfs, but I still don't see the case for having it merged :-) Thanks for your answers, I think I get it all now. -- http://www.PowerDNS.com Open source, database driven DNS Software http://netherlabs.nl Open and Closed source services - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] Re: relayfs documentation sucks?
bert hubert writes:
> On Sat, Jul 16, 2005 at 06:13:55PM -0500, Tom Zanussi wrote:
>
> > relayfs itself only provides the buffering and file operations along
> > with the kernel API for clients as documented in
> > Documentation/filesystems/relayfs.txt. Applications still need some
> > kind of communication between the kernel and user space in order to
> > know when data is ready and how much is ready - the relay-apps stuff
> > tries to make this easy to do by allowing clients to ignore all those
> > details. It happens to use netlink for this, but clients can use
> > whatever they want to do this communication.
>
> Ok - that is good to know. What is missing from relayfs.txt is a demarcation
> of which system does what.
>
> As I see it there are three things currently:
>
> 1) Basic relayfs facilities, which only stuff data into N sub-buffers per
> CPU, but also offer a set of functions that could be called via userspace
> over some sort of communication channel.
>
> 2) klog which is a thin wrapper over relay_write
>
> 3) relay-app.h which lives in the kernel and communicates with librelay.c in
> user space, providing that communication.
>
> Is this correct?
Yes.
>
> > Then just run the kleak app, and when it finishes, you should have a
> > set of files, cpu0l...cpuX in your current directory containing all
> > the data you've logged.
>
> I've changed the fprintf(stderr, "netlink send error") to perror("netlink
> send error") and now it prints 'Connection refused', which makes heaps of
> sense since I did not use relay-app.h, but wrote directly to the
channel.
Right - you need to insmod kleak.ko in order for the netlink socket to
be created in the kernel.
>
> > > 2) What kind of messages do I need to send/receive?
> >
> > Basically, the daemon needs to know, for a given per-cpu buffer, how
> > many sub-buffers have been produced and consumed, in order to know
> > which sections of the mmapped buffer to read. It also needs to notify
>
> I currently just write away without any userspace component, except that I
> mmap the entire relayfs file in which I see the four configured sub-buffers.
> I guess that in override mode that would work?
Right - this sounds exactly like what overwrite mode is meant for -
flight-recording types of applications, where you don't have an active
reader in userspace and you're interested in the most recent data. If
you don't have an active reader and use no-overwrite mode, the buffer
will become full when it wraps around the first time, and subsequent
events will be lost (the buffer-full callback will tell you when this
happens).
>
> > The format is whatever the client writes into it - relayfs itself
> > doesn't impose any format at all. The client doesn't need librelay.c
> > to read the data itself - librelay.c is for managing the daemon side
> > of the application and writing ready data to disk as it becomes
> > available. It doesn't know anything about the actual data being written.
>
> Ok - so there is nothing in there except n stretches of data, and some
> padding? Each write is either IN a sub-buffer or not at all, it doesn't span
> sub-buffers?
Right, a write will never be split across sub-buffers.
>
> > > 4) What are the semantics for reading from that file?
> >
> > The file is a buffer broken up into sub-buffers. The client reads the
> > sub-buffers it knows are ready directly from the mmapped buffer.
> > The file can only be mmap()ed - there is no read() available.
>
> Indeed. So the idea is to wait for a ringbuffer to become 'full', read it,
> and wait for the next one to become full?
Right, as sub-buffers become full, the userspace part of the client
should read them, update the kernel part with how many it just
consumed, and wait around for more.
>
> > BTW, there's also documentation in relay-app.h, don't know if you saw
> > that.
>
> Yes - but it only makes sense after the 'separation of powers' within
> relayfs is clear. relayfs.txt talks rather cavalierly of 'clients' and
> 'calls' but does not make clear this client lives in userspace and can't
> just call kernel functions.
>
> Please consider the patch below. I'm not 100% sure if everything is correct,
> but I'd love to know.
Yes, on first reading, it all looks correct, and does a nice job of
clarifying things - thanks for taking the time to do this. :-)
>
> I'm wondering how relayfs could be operated safely in overwrite mode, btw -
> who's to say the kernel might not have zoomed past my sub-buffer once I'm
> notified of the crossing? The padding data I receive might be outdated by
> then. Sounds racey.
It is racey - in this mode, there's nothing to keep the kernel from
writing as much as it wants before the user side has a chance to read
any of it. The only way this can be used safely is to make sure the
kernel side isn't writing anything when the client is reading. This
would be typical of
[PATCH] Re: relayfs documentation sucks?
On Sat, Jul 16, 2005 at 06:13:55PM -0500, Tom Zanussi wrote:
> relayfs itself only provides the buffering and file operations along
> with the kernel API for clients as documented in
> Documentation/filesystems/relayfs.txt. Applications still need some
> kind of communication between the kernel and user space in order to
> know when data is ready and how much is ready - the relay-apps stuff
> tries to make this easy to do by allowing clients to ignore all those
> details. It happens to use netlink for this, but clients can use
> whatever they want to do this communication.
Ok - that is good to know. What is missing from relayfs.txt is a demarcation
of which system does what.
As I see it there are three things currently:
1) Basic relayfs facilities, which only stuff data into N sub-buffers per
CPU, but also offer a set of functions that could be called via userspace
over some sort of communication channel.
2) klog which is a thin wrapper over relay_write
3) relay-app.h which lives in the kernel and communicates with librelay.c in
user space, providing that communication.
Is this correct?
> Then just run the kleak app, and when it finishes, you should have a
> set of files, cpu0l...cpuX in your current directory containing all
> the data you've logged.
I've changed the fprintf(stderr, "netlink send error") to perror("netlink
send error") and now it prints 'Connection refused', which makes heaps of
sense since I did not use relay-app.h, but wrote directly to the channel.
> > 2) What kind of messages do I need to send/receive?
>
> Basically, the daemon needs to know, for a given per-cpu buffer, how
> many sub-buffers have been produced and consumed, in order to know
> which sections of the mmapped buffer to read. It also needs to notify
I currently just write away without any userspace component, except that I
mmap the entire relayfs file in which I see the four configured sub-buffers.
I guess that in override mode that would work?
> The format is whatever the client writes into it - relayfs itself
> doesn't impose any format at all. The client doesn't need librelay.c
> to read the data itself - librelay.c is for managing the daemon side
> of the application and writing ready data to disk as it becomes
> available. It doesn't know anything about the actual data being written.
Ok - so there is nothing in there except n stretches of data, and some
padding? Each write is either IN a sub-buffer or not at all, it doesn't span
sub-buffers?
> > 4) What are the semantics for reading from that file?
>
> The file is a buffer broken up into sub-buffers. The client reads the
> sub-buffers it knows are ready directly from the mmapped buffer.
> The file can only be mmap()ed - there is no read() available.
Indeed. So the idea is to wait for a ringbuffer to become 'full', read it,
and wait for the next one to become full?
> BTW, there's also documentation in relay-app.h, don't know if you saw
> that.
Yes - but it only makes sense after the 'separation of powers' within
relayfs is clear. relayfs.txt talks rather cavalierly of 'clients' and
'calls' but does not make clear this client lives in userspace and can't
just call kernel functions.
Please consider the patch below. I'm not 100% sure if everything is correct,
but I'd love to know.
I'm wondering how relayfs could be operated safely in overwrite mode, btw -
who's to say the kernel might not have zoomed past my sub-buffer once I'm
notified of the crossing? The padding data I receive might be outdated by
then. Sounds racey.
In fact, it appears this might even happen in non-overwrite mode.
diff -urBb -X linux-2.6.13-rc3-mm1/Documentation/dontdiff
linux-2.6.13-rc3-mm1/Documentation/filesystems/relayfs.txt
linux-2.6.13-rc3-mm1-ahu/Documentation/filesystems/relayfs.txt
--- linux-2.6.13-rc3-mm1/Documentation/filesystems/relayfs.txt 2005-07-17
11:00:48.000638680 +0200
+++ linux-2.6.13-rc3-mm1-ahu/Documentation/filesystems/relayfs.txt
2005-07-17 10:58:21.634889656 +0200
@@ -23,6 +23,46 @@
the function parameters are documented along with the functions in the
filesystem code - please see that for details.
+Semantics
+=
+
+Each relayfs channel has one buffer per CPU, each buffer has one or
+more sub-buffers. Messages are written to the first sub-buffer until it
+is too full to contain a new message, in which case it it is written to
+the next (if available). At this point, userspace can be notified so it
+empties the first ringbuffer, while the kernel continues writing to the
+next.
+
+If notified that a sub-buffer is full, the kernel knows how many bytes
+of it are padding, ie, unused. Userspace can use this knowledge to copy
+only valid data.
+
+After copying, userspace can notify the kernel that a sub-channel has
+been consumed.
+
+relayfs can operate in a mode where it will overwrite data not yet
+collected by userspace, and not wait for it to consume it.
+
+relayfs itself does not provide for communication of such data between
