Re: [Tutor] python logging module: two handlers writing to the same file - okay?
Hans Fangohr wrote: > FYI: The reason for wanting to use two different file formats is this: > we have a somewhat larger project (http://nmag.soton.ac.uk) where we > combine high-level Python code with low-level Objective Caml code (and > a number of libraries). We would like to use the Python-logging module > to log all sorts of messages coming from various parts of the code. In > particular, we find having the line number (and function name if the > python version provides this) useful information in the log messages > (which can bedone for Python code issueing log messages). However, > this information cannot be obtained from the OCaml code; thus we do > not want to waste space in the log messages, and therefore we'd like > to use different formatters. Perhaps you could write a formatter that holds two different format strings and selects between them based on whether the line number is available. Kent ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] python logging module: two handlers writing to the same file - okay?
HI Riccardo, >>> >>> As far as I can see, the only reason in your example program to >>> open the >>> same file twice is to use two different formatters (i.e. two >>> different >>> type of lines) in the same log, >> Absolutely right. >> >>> if you'd drop the requirement for two >>> different format of line in the same log (which is itself somewhat >>> questionable) then you could use the same handler for both >>> loggers, thus >>> opening only one file. >> >> True. >> >> FYI: The reason for wanting to use two different file formats is >> this: >> we have a somewhat larger project (http://nmag.soton.ac.uk) where we >> combine high-level Python code with low-level Objective Caml code >> (and >> a number of libraries). We would like to use the Python-logging >> module >> to log all sorts of messages coming from various parts of the >> code. In >> particular, we find having the line number (and function name if the >> python version provides this) useful information in the log messages >> (which can bedone for Python code issueing log messages). However, >> this information cannot be obtained from the OCaml code; thus we do >> not want to waste space in the log messages, and therefore we'd like >> to use different formatters. >> >> I'll reply to Ken's email in a minute, and I think that closes my >> query. >> >> Best wishes, >> >> Hans >> > > One last thing Hans. If you are concerned about the space the log > files > will use maybe it would be interesting to check out the zlib module > which could be used to compress your log strings before sending > them to > the logging module, then a utility to read the modules would close the > cycle (caveat: I've never done this, just a crazy idea, but... what if > it works?). Apologies, I wasn't clear: just wanted to be nice to the user and avoid getting "(unknown)" entries in the log messages for the line number and the function name which pushes the useful message further to the right. If we are talking compression, I guess you could inherit from the RotatingFileHandler and modify such that you gzip the rotated log files. > Best of luck. > Many thanks, Hans ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] python logging module: two handlers writing to the same file - okay?
Hans Fangohr wrote: Ricardo Aráoz wrote: > Kent Johnson wrote: >> I don't know the answer, but it has nothing to do with the logging >> module. The question is, can the same file reliably be opened >> twice for >> writing in the same module. > > Well, the question would actually be if the logging module is smart > enough to find out that both your filehandlers are referring to the > same > file and open it only once. A quick glance at the logging module shows that it is not that smart. It just opens the file twice; hence my original question. >>> >>> To summarise the situation: the logging module is not clever enough to >>> detect that I am opening the same file twice. If I use 'w' as the >>> opening mode, then I know (and have an example program for this) that >>> the logging 'fails': I suppose this can be tracked to the first part of >>> the log file being overriden when the second filehandler attempts to log >>> its first message, and thus opens the file (again) in 'w' mode. >>> >>> Using 'a' (append) or 'a+' (?) *seems* to solve this problem but we are >>> still short of knowing as to whether this is guaranteed to work (on all >>> platforms...) or whether it just happened to be okay on Mac OS X and >>> (Debian) Linux (on which I have tested this). >>> >>> I think my example raises another question (in particular in context >>> with openening the files in 'w' mode): would it be worth contacting the >>> developers of the logging module to suggest that it >>> - tracks which files it writes to and >>> - warns if the same file is opened twice (or more ofter) using 'w'? >>> >>> Alternatively, one could enhance the internal logic of the logging >>> module so that it is aware of the same file being used twice (or more >>> times) so that it only opens the file once, but I suppose this raises >>> some other questions (if the two filehandlers have been given different >>> opening modes, say 'a' and 'w', which one should it take when opening >>> the file only once?). Therefore, just issueing a warning may be the >>> better solution: should be easy to implement, doesn't change the >>> interface, and prevents (naive) users (like myself) from wasting time >>> trying to track down the problem of the beginning of the log file >>> missing. >>> >> >> >> As far as I can see, the only reason in your example program to open the >> same file twice is to use two different formatters (i.e. two different >> type of lines) in the same log, > Absolutely right. > >> if you'd drop the requirement for two >> different format of line in the same log (which is itself somewhat >> questionable) then you could use the same handler for both loggers, thus >> opening only one file. > > True. > > FYI: The reason for wanting to use two different file formats is this: > we have a somewhat larger project (http://nmag.soton.ac.uk) where we > combine high-level Python code with low-level Objective Caml code (and > a number of libraries). We would like to use the Python-logging module > to log all sorts of messages coming from various parts of the code. In > particular, we find having the line number (and function name if the > python version provides this) useful information in the log messages > (which can bedone for Python code issueing log messages). However, > this information cannot be obtained from the OCaml code; thus we do > not want to waste space in the log messages, and therefore we'd like > to use different formatters. > > I'll reply to Ken's email in a minute, and I think that closes my query. > > Best wishes, > > Hans > One last thing Hans. If you are concerned about the space the log files will use maybe it would be interesting to check out the zlib module which could be used to compress your log strings before sending them to the logging module, then a utility to read the modules would close the cycle (caveat: I've never done this, just a crazy idea, but... what if it works?). Best of luck. Ricardo ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] python logging module: two handlers writing to the same file - okay?
Ricardo Aráoz wrote: Why should it be all or nothing. Couldn't the programmer indicate that both handlers use the same file? It would be very easy to do this using StreamHandlers instead of FileHandlers. It would also be easy to make a FileHandler subclass that keeps a map from file name to file objects so if the same file is given to two instances it reuses the first one. This could be specified as the handler in the config file even. This is probably just the right suggestion (subclassing FileHandler, including a map of files and overriding the open() method). I realise that our requirements are somewhat special (needing different formatters for one file), and that it maybe unwise trying to 'protect' the programmer from himself. On the positive side: while the logging module is not clever enough to provide the discussed functionality, it is easy to understand (and extend). Python has 'batteries included' but sometimes you have to add some wiring ;-) Thanks to everybody who contributed to this; I think we have found a good solution. Regards, Hans Kent ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] python logging module: two handlers writing to the same file - okay?
Ricardo Aráoz wrote: Kent Johnson wrote: I don't know the answer, but it has nothing to do with the logging module. The question is, can the same file reliably be opened twice for writing in the same module. Well, the question would actually be if the logging module is smart enough to find out that both your filehandlers are referring to the same file and open it only once. A quick glance at the logging module shows that it is not that smart. It just opens the file twice; hence my original question. To summarise the situation: the logging module is not clever enough to detect that I am opening the same file twice. If I use 'w' as the opening mode, then I know (and have an example program for this) that the logging 'fails': I suppose this can be tracked to the first part of the log file being overriden when the second filehandler attempts to log its first message, and thus opens the file (again) in 'w' mode. Using 'a' (append) or 'a+' (?) *seems* to solve this problem but we are still short of knowing as to whether this is guaranteed to work (on all platforms...) or whether it just happened to be okay on Mac OS X and (Debian) Linux (on which I have tested this). I think my example raises another question (in particular in context with openening the files in 'w' mode): would it be worth contacting the developers of the logging module to suggest that it - tracks which files it writes to and - warns if the same file is opened twice (or more ofter) using 'w'? Alternatively, one could enhance the internal logic of the logging module so that it is aware of the same file being used twice (or more times) so that it only opens the file once, but I suppose this raises some other questions (if the two filehandlers have been given different opening modes, say 'a' and 'w', which one should it take when opening the file only once?). Therefore, just issueing a warning may be the better solution: should be easy to implement, doesn't change the interface, and prevents (naive) users (like myself) from wasting time trying to track down the problem of the beginning of the log file missing. As far as I can see, the only reason in your example program to open the same file twice is to use two different formatters (i.e. two different type of lines) in the same log, Absolutely right. if you'd drop the requirement for two different format of line in the same log (which is itself somewhat questionable) then you could use the same handler for both loggers, thus opening only one file. True. FYI: The reason for wanting to use two different file formats is this: we have a somewhat larger project (http://nmag.soton.ac.uk) where we combine high-level Python code with low-level Objective Caml code (and a number of libraries). We would like to use the Python-logging module to log all sorts of messages coming from various parts of the code. In particular, we find having the line number (and function name if the python version provides this) useful information in the log messages (which can bedone for Python code issueing log messages). However, this information cannot be obtained from the OCaml code; thus we do not want to waste space in the log messages, and therefore we'd like to use different formatters. I'll reply to Ken's email in a minute, and I think that closes my query. Best wishes, Hans ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] python logging module: two handlers writing to the same file - okay?
Ricardo Aráoz wrote: > Why should it be all or nothing. Couldn't the programmer indicate that > both handlers use the same file? It would be very easy to do this using StreamHandlers instead of FileHandlers. It would also be easy to make a FileHandler subclass that keeps a map from file name to file objects so if the same file is given to two instances it reuses the first one. This could be specified as the handler in the config file even. Python has 'batteries included' but sometimes you have to add some wiring ;-) Kent ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] python logging module: two handlers writing to the same file - okay?
Kent Johnson wrote: > Hans Fangohr wrote: >> Dear all, >> >> thanks to everybody who replied to my question. >> >> On 15 Dec 2007, at 16:34, Kent Johnson wrote: >> >>> Ricardo Aráoz wrote: > >> What is the recommended method to make such a suggestion to the python >> team, or the people who look after the logging module? > > You could post a feature request (possibly including a patch or > suggested implementation) to the Python bug tracker: > http://www.python.org/dev/patches/ > > You could contact the developer directly via the feedback link here: > http://www.red-dove.com/python_logging.html > > You could start a discussion on comp.lang.python. > > Note: when I was researching my original answer I found a thread about a > similar problem with shelve. The discussion was not sympathetic to the > OP for two reasons: > - in general it's difficult to determine if the same file is being > opened twice > - it is trying too hard to protect the programmer against himself. > http://groups.google.com/group/comp.lang.python/browse_frm/thread/4aafd9a8192cfbe7/5803ea195210a28f?hl=en&lnk=gst&q=open+file+twice#5803ea195210a28f Why should it be all or nothing. Couldn't the programmer indicate that both handlers use the same file? For example if instead of pointing to a file the formatter points to another formatter then it is using the same file of the pointed formatter. Then both your points would be answered and the module would be a little bit better (I don't think I'll ever need that feature as I believe that a log should have all it's lines in the same format). ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] python logging module: two handlers writing to the same file - okay?
Hans Fangohr wrote: > Dear all, > > thanks to everybody who replied to my question. > > On 15 Dec 2007, at 16:34, Kent Johnson wrote: > >> Ricardo Aráoz wrote: >>> Kent Johnson wrote: I don't know the answer, but it has nothing to do with the logging module. The question is, can the same file reliably be opened twice for writing in the same module. >>> >>> Well, the question would actually be if the logging module is smart >>> enough to find out that both your filehandlers are referring to the same >>> file and open it only once. >> >> A quick glance at the logging module shows that it is not that smart. It >> just opens the file twice; hence my original question. >> > > To summarise the situation: the logging module is not clever enough to > detect that I am opening the same file twice. If I use 'w' as the > opening mode, then I know (and have an example program for this) that > the logging 'fails': I suppose this can be tracked to the first part of > the log file being overriden when the second filehandler attempts to log > its first message, and thus opens the file (again) in 'w' mode. > > Using 'a' (append) or 'a+' (?) *seems* to solve this problem but we are > still short of knowing as to whether this is guaranteed to work (on all > platforms...) or whether it just happened to be okay on Mac OS X and > (Debian) Linux (on which I have tested this). > > I think my example raises another question (in particular in context > with openening the files in 'w' mode): would it be worth contacting the > developers of the logging module to suggest that it > - tracks which files it writes to and > - warns if the same file is opened twice (or more ofter) using 'w'? > > Alternatively, one could enhance the internal logic of the logging > module so that it is aware of the same file being used twice (or more > times) so that it only opens the file once, but I suppose this raises > some other questions (if the two filehandlers have been given different > opening modes, say 'a' and 'w', which one should it take when opening > the file only once?). Therefore, just issueing a warning may be the > better solution: should be easy to implement, doesn't change the > interface, and prevents (naive) users (like myself) from wasting time > trying to track down the problem of the beginning of the log file missing. > As far as I can see, the only reason in your example program to open the same file twice is to use two different formatters (i.e. two different type of lines) in the same log, if you'd drop the requirement for two different format of line in the same log (which is itself somewhat questionable) then you could use the same handler for both loggers, thus opening only one file. ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] python logging module: two handlers writing to the same file - okay?
Hans Fangohr wrote: > Dear all, > > thanks to everybody who replied to my question. > > On 15 Dec 2007, at 16:34, Kent Johnson wrote: > >> Ricardo Aráoz wrote: > What is the recommended method to make such a suggestion to the python > team, or the people who look after the logging module? You could post a feature request (possibly including a patch or suggested implementation) to the Python bug tracker: http://www.python.org/dev/patches/ You could contact the developer directly via the feedback link here: http://www.red-dove.com/python_logging.html You could start a discussion on comp.lang.python. Note: when I was researching my original answer I found a thread about a similar problem with shelve. The discussion was not sympathetic to the OP for two reasons: - in general it's difficult to determine if the same file is being opened twice - it is trying too hard to protect the programmer against himself. http://groups.google.com/group/comp.lang.python/browse_frm/thread/4aafd9a8192cfbe7/5803ea195210a28f?hl=en&lnk=gst&q=open+file+twice#5803ea195210a28f ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] python logging module: two handlers writing to the same file - okay?
Dear all, thanks to everybody who replied to my question. On 15 Dec 2007, at 16:34, Kent Johnson wrote: > Ricardo Aráoz wrote: >> Kent Johnson wrote: >>> I don't know the answer, but it has nothing to do with the logging >>> module. The question is, can the same file reliably be opened >>> twice for >>> writing in the same module. >> >> Well, the question would actually be if the logging module is smart >> enough to find out that both your filehandlers are referring to >> the same >> file and open it only once. > > A quick glance at the logging module shows that it is not that > smart. It > just opens the file twice; hence my original question. > To summarise the situation: the logging module is not clever enough to detect that I am opening the same file twice. If I use 'w' as the opening mode, then I know (and have an example program for this) that the logging 'fails': I suppose this can be tracked to the first part of the log file being overriden when the second filehandler attempts to log its first message, and thus opens the file (again) in 'w' mode. Using 'a' (append) or 'a+' (?) *seems* to solve this problem but we are still short of knowing as to whether this is guaranteed to work (on all platforms...) or whether it just happened to be okay on Mac OS X and (Debian) Linux (on which I have tested this). I think my example raises another question (in particular in context with openening the files in 'w' mode): would it be worth contacting the developers of the logging module to suggest that it - tracks which files it writes to and - warns if the same file is opened twice (or more ofter) using 'w'? Alternatively, one could enhance the internal logic of the logging module so that it is aware of the same file being used twice (or more times) so that it only opens the file once, but I suppose this raises some other questions (if the two filehandlers have been given different opening modes, say 'a' and 'w', which one should it take when opening the file only once?). Therefore, just issueing a warning may be the better solution: should be easy to implement, doesn't change the interface, and prevents (naive) users (like myself) from wasting time trying to track down the problem of the beginning of the log file missing. What is the recommended method to make such a suggestion to the python team, or the people who look after the logging module? Many thanks, Hans > Kent > ___ > Tutor maillist - Tutor@python.org > http://mail.python.org/mailman/listinfo/tutor > > > -- Hans Fangohr School of Engineering Sciences University of Southampton Phone: +44 (0) 238059 8345 Email: [EMAIL PROTECTED] http://www.soton.ac.uk/~fangohr ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] python logging module: two handlers writing to the same file - okay?
Ricardo Aráoz wrote: > Kent Johnson wrote: >> I don't know the answer, but it has nothing to do with the logging >> module. The question is, can the same file reliably be opened twice for >> writing in the same module. > > Well, the question would actually be if the logging module is smart > enough to find out that both your filehandlers are referring to the same > file and open it only once. A quick glance at the logging module shows that it is not that smart. It just opens the file twice; hence my original question. Kent ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] python logging module: two handlers writing to the same file - okay?
Kent Johnson wrote: > Hans Fangohr wrote: > >> (i) is this (as in the log.conf file) the right use of the logging >> module to achieve what I need? > > I think you understand the module correctly. > >> (ii) in particular, it appears we have two filehandlers that write to >> the same file (in mode 'a+'). While this seems to work fine in the >> examples I have tested, I'd like some independent advice on whether >> this is 'legal' (or whether it works by chance). > > I don't know the answer, but it has nothing to do with the logging > module. The question is, can the same file reliably be opened twice for > writing in the same module. > > Another option: If you configure logging in code, you could create two > StreamHandlers that log to the same file - open the file yourself and > pass it to both handlers. If you do this you will have to close the file > yourself somehow. > > Kent > Well, the question would actually be if the logging module is smart enough to find out that both your filehandlers are referring to the same file and open it only once. ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] python logging module: two handlers writing to the same file - okay?
Hans Fangohr wrote: > (i) is this (as in the log.conf file) the right use of the logging > module to achieve what I need? I think you understand the module correctly. > (ii) in particular, it appears we have two filehandlers that write to > the same file (in mode 'a+'). While this seems to work fine in the > examples I have tested, I'd like some independent advice on whether > this is 'legal' (or whether it works by chance). I don't know the answer, but it has nothing to do with the logging module. The question is, can the same file reliably be opened twice for writing in the same module. Another option: If you configure logging in code, you could create two StreamHandlers that log to the same file - open the file yourself and pass it to both handlers. If you do this you will have to close the file yourself somehow. Kent ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
