Re: Creating a logging library and have questions
I second Rainer. A logging system should commit (at least) error messages immediately, particularly if the application has multiple threads. Otherwise it is going to make debugging a crashing system a nightmare. When I do it I just stick 'synchronized' in front of the statement that does the write. Yes, I fully understand that and in the current single-threaded version I have, that's exactly what happens: all error and fatal messages cause the buffer to flush. What I'm looking for is the best way to handle having a daemon writer. My biggest concern here is multi-threaded applications. Granted, it would be nice to not have the writing in the same thread as the rest of the code to try to keep file I/O from affecting performance, but that's secondary. Here's what I know: a variable of type OutputStream cannot be shared. I did not try using __gshared, but regardless while this would work and I could easily synchronize the writes, I can see a lot of contention if multiple threads trying to write a lot of data to a log file. (E.g. trace statements that capture variable states to enhance debugging) Granted, this shouldn't be the case in production code, but if I can find a better way to do this, I'd love to. This is why I thought that the new threading model with message passing would be good, but I again have concerns if there are a lot of messages trying to be written. I know in Erlang you can fill up the message buffer if you're not careful. Casey
Re: Creating a logging library and have questions
sybrandy Wrote: > On 02/03/2010 12:03 AM, Rainer Deyke wrote: > > sybrandy wrote: > >> 1) I use the current core.thread library and put all my messages in a > >> buffer that the thread checks periodically, pulls off a new message, and > >> then writes it to a file. There will be some work to make sure nothing > >> collides with each other, but I think I can manage it. > > > > Wouldn't this allow logging messages to be lost before a hard crash? To > > me, that would greatly reduce the utility of the logging system. > > If done improperly, yes, it would. My hope is that if I did go with > this method, I would try to find a way to ensure no data was lost. > Oddly enough, I'm currently having that problem with the single-threaded > version for some reason. The test program will stop and some of the > logging statements never make it to the file. > > > >> 3) Something else. I really don't have much experience with threading, > >> so I'm being very careful and really want to understand it. This > >> library looks to be a good way to learn, however if it's not the best > >> way to do things, then what would be? > > > > Global mutex associated with the logging system. Lock, output, unlock. > > There are scalability issues with that approach, but I don't see why it > > wouldn't work in this situation. (Plus, if you have a message queue, > > you probably need to protect that queue with a mutex anyway.) > > > > > > Understood. My goal is that if I do put the writing in another thread, > I do my best to ensure it will scale. I have a tendency to put a lot of > logging statements in code when I'm trying to debug something, so I > don't want to slow things down too much nor do I want to lose anything. > In short: I want the log writing to be as out of the way as possible. > > Casey I second Rainer. A logging system should commit (at least) error messages immediately, particularly if the application has multiple threads. Otherwise it is going to make debugging a crashing system a nightmare. When I do it I just stick 'synchronized' in front of the statement that does the write.
Re: Creating a logging library and have questions
On 02/03/2010 12:03 AM, Rainer Deyke wrote: sybrandy wrote: 1) I use the current core.thread library and put all my messages in a buffer that the thread checks periodically, pulls off a new message, and then writes it to a file. There will be some work to make sure nothing collides with each other, but I think I can manage it. Wouldn't this allow logging messages to be lost before a hard crash? To me, that would greatly reduce the utility of the logging system. If done improperly, yes, it would. My hope is that if I did go with this method, I would try to find a way to ensure no data was lost. Oddly enough, I'm currently having that problem with the single-threaded version for some reason. The test program will stop and some of the logging statements never make it to the file. 3) Something else. I really don't have much experience with threading, so I'm being very careful and really want to understand it. This library looks to be a good way to learn, however if it's not the best way to do things, then what would be? Global mutex associated with the logging system. Lock, output, unlock. There are scalability issues with that approach, but I don't see why it wouldn't work in this situation. (Plus, if you have a message queue, you probably need to protect that queue with a mutex anyway.) Understood. My goal is that if I do put the writing in another thread, I do my best to ensure it will scale. I have a tendency to put a lot of logging statements in code when I'm trying to debug something, so I don't want to slow things down too much nor do I want to lose anything. In short: I want the log writing to be as out of the way as possible. Casey
Re: Creating a logging library and have questions
sybrandy wrote: > 1) I use the current core.thread library and put all my messages in a > buffer that the thread checks periodically, pulls off a new message, and > then writes it to a file. There will be some work to make sure nothing > collides with each other, but I think I can manage it. Wouldn't this allow logging messages to be lost before a hard crash? To me, that would greatly reduce the utility of the logging system. > 3) Something else. I really don't have much experience with threading, > so I'm being very careful and really want to understand it. This > library looks to be a good way to learn, however if it's not the best > way to do things, then what would be? Global mutex associated with the logging system. Lock, output, unlock. There are scalability issues with that approach, but I don't see why it wouldn't work in this situation. (Plus, if you have a message queue, you probably need to protect that queue with a mutex anyway.) -- Rainer Deyke - rain...@eldwood.com
