On Tue, 2008-11-04 at 14:58 -0600, David Teigland wrote: > On Thu, Oct 30, 2008 at 11:26:14PM -0700, Steven Dake wrote: > > There are two types of messages. Those intended for users/admins and > > those intended for developers. > > > > Both of these message types should always be recorded *somewhere*. The > > entire concept of "LOG_LEVEL_DEBUG" is dubious to me. If you want to > > stick with that symanetic and definition that is fine, but really a > > LOG_LEVEL_DEBUG means "this message is for the developer". These > > messages should be recorded and stored when a process segfaults, aborts > > due to assertion, or at administrative request. Since the frequency of > > these messages is high there is no other option for recording them since > > they must _always_ be recorded for the purposes of debugging a field > > failure. Recording to disk or syslog has significant performance > > impact. > > > > The only solution for these types of messages is to record them into a > > flight recorder buffer which can be dumped: > > 1) at segv > > 2) at sigabt > > 3) at administrative request > > > > This is a fundamental difference in how we have approached logging > > debugging messages in the past but will lead to the ability to ensure we > > _always_ have debug trace data available instead of telling the > > user/admin "Go turn on debug and hope you can reproduce that error and > > btw since 100000k messages are logged your disk will fill up with > > irrelevant debug messages and your system will perform like mud". > > > > Logging these in memory is the only solution that I see as suitable and > > in all cases they should be filtered from any output source such as > > stderr, file, or syslog. > > There's a difference between high volume trace debug data stored in > memory, and low volume informational debug data that can be easily written > to a file. Both kinds of data can be useful. > > My programs are simple enough that low volume informational debug data is > enough for me to identify and fix a problem. So, low volume informational > data is all I produce. It can be useful to write this data to a file. > > Your program is complex enough that high volume trace debug data is > usually needed for you to identify and fix a problem. So, high volume > trace data is all you produce. This is too much data to write to a file > (by the running program). > > So, we're using "DEBUG" to refer to different things. We need to define > two different levels (just for clarity in this discussion): > . DEBUGLO is low volume informational data like I use > . DEBUGHI is high volume trace data like you use > > DEBUGHI messages wouldn't ever be logged to files by the program while > running. DEBUGLO messages could be, though, if the user configured it. > So, circling back around, how should a user configure DEBUGLO messages to > appear in syslog or a logfile? In particular, what would they enter in > the cluster.conf <logging/> section? My suggestion is: > > syslog_level=foo > logfile_level=bar > > where foo and bar are one of the standard priority names in syslog.h. > So, if a user wanted DEBUGLO messages to appear in daemon.log, they'd set > > logging/<daemon>/logfile_level=debug > > and if they wanted DEBUGLO messages to appear in /var/log/messages, > > logging/<daemon>/syslog_level=debug > > (Note that "debug" means DEBUGLO here because DEBUGHI messages are only > saved in memory, not to files by a running program.) > > There's another separate question I have about corosync, and that's > whether you could identify some limited number of messages that would be > appropriate for DEBUGLO? They would be used by non-experts to do some > rough debugging of problems, and by experts to narrow down a problem > before digging into the high volume trace data. I'd suggest that a good > starting point for DEBUGLO would be the data that openais has historically > put in /var/log/messages. Data that helps you quickly triage a problem > (or verify that things are happening correctly) without stepping through > all the trace data. >
I thought about this a few days and this seems to make sense. The priority (ie: specifically LOG_LEVEL_DEBUG messages) would be selectable per output medium (syslog, file, or stderr) and the high performance tracing would use the trace macros or log_rec and only ever go to memory. Each output medium if configured would filter based upon a filtering priority which is LOG_LEVEL_DEBUG or others. This does require further API changes that Fabio would have to manage with. Regards -steve _______________________________________________ Openais mailing list Openais@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/openais