Log rotation issue with runit
Hello! I am Debian maintainer of `runit' supervision suite. Recently, I received bug report [^1] about stray .u logfiles. After some investigation I found code, that caused issue, but it seems that it was written with some purpose, yet I fail to understand that purpose. Author Gerrit Pape directed me here. Any clarification would be welcome! Please keep bug in thread. [^1] https://bugs.debian.org/916230
Re: Log rotation issue with runit
Hi! On Tue, Dec 25, 2018 at 01:39:17PM +, Dmitry Bogatov wrote: > Recently, I received bug report [^1] about stray .u logfiles. After > some investigation I found code, that caused issue, but it seems that it > was written with some purpose, yet I fail to understand that purpose. > > [^1] https://bugs.debian.org/916230 .u files are created after svlogd crash (actually, when svlogd starts again after crash). I don't think I've ever seen actual crash of svlogd, so usually it means system was rebooted because of kernel OOPS or pressing reset button or power loss or someone has killed svlogd with SIGKILL. I believe this isn't a bug, but this behaviour isn't documented in svlogd man page. To me it looks useful to be able to detect such crash-reboots and what happens just before by looking for/in .u files. -- WBR, Alex.
Re: Log rotation issue with runit
El mar., 25 dic. 2018 a las 10:39, Dmitry Bogatov escribió: > > Hello! > > I am Debian maintainer of `runit' supervision suite. Recently, I > received bug report [^1] about stray .u logfiles. After some > investigation I found code, that caused issue, but it seems that it was > written with some purpose, yet I fail to understand that purpose. > > Author Gerrit Pape directed me here. Any clarification would be welcome! > Please keep bug in thread. > > [^1] https://bugs.debian.org/916230 It was likely written on purpose. A 'current' file that doesn't have the execute by owner permission set is not considered to completely processed, and becomes a .u file instead of being appended to. As has been said, it can happen if svlogd does not exit cleanly. Although it doesn't seem to be mentioned in svlogd's man page, this comes from daemontools' multilog program, which inspired svlogd: * https://cr.yp.to/daemontools/multilog.html (section "Automatically rotated logs") On the other hand, it seems to me that the patch used for resolution of bug #878476 prevents all .u files from being deleted, so they would indeed accumulate indefinitely. Both .u and .s files are supposed to be deleted when they are the oldest file in the logging directory, to honor 'u' lines in 'config' files, and this is only a problem when the one about to be fed to a processor happens to be the one chosen for deletion because of a wonky clock. This issue was brought up earlier in this mailing list, and a different patch was suggested: * https://www.mail-archive.com/supervision@list.skarnet.org/msg00667.html This one seems to do the right check, maybe it should have been applied instead? G.
Re: Log rotation issue with runit
> […] Both .u and .s files are supposed to > be deleted when they are the oldest file in the logging directory, to > honor 'u' lines in 'config' files, […] Sorry, that should have read: to honor 'n' lines in 'config' files. G.
Re: Log rotation issue with runit
Hi! I'm not sure is it good idea to include .u files in usual rotation process at all, and especially handle them just like .s files. If several crashes happens for some reason in a short period of time this will result in deletion of all log files except last (say) 10 .u files, which are usually small and may contain just one line. Replacing 10MB of last logs with 10 last log lines doesn't sounds like a nice idea. Possible "right" solution will be to keep same amount of last .u files as configured for .s files, i.e. if we've configured to keep last 10 files then we may have at most double amount (10 .u files and 10 .s files). (I didn't checked mentioned patches, so maybe they already works this way.) -- WBR, Alex.
Re: Log rotation issue with runit
On Tue, 25 Dec 2018 13:39:17 + Dmitry Bogatov wrote: > Hello! > > I am Debian maintainer of `runit' supervision suite. Recently, I > received bug report [^1] about stray .u logfiles. After some > investigation I found code, that caused issue, but it seems that it > was written with some purpose, yet I fail to understand that purpose. > > Author Gerrit Pape directed me here. Any clarification would be > welcome! Please keep bug in thread. > > [^1] https://bugs.debian.org/916230 It's vital that no code be changed until the exact nature of the problem is understood and reproducible at will. The extra files reported in the bug aren't nearly especially inconvenient when compared to possible problems when modifying code that the code's author isn't maintaining on a regular basis. It would be interesting to see the dates and contents of the files: That might shed some light on what's happening. One person in this thread mentioned that .u files shouldn't be rotated and deleted at all. If that's the case, a fairly simple cron job could be made to delete all .u files older than a certain age except the current/latest .u file. That would probably fix the inconvenience endured by the bug reporter, without applying any premature fixes. This seems to be an intermittent problem (not reproducible at will), and very hard to reproduce by anyone but the bug reporter. If the preceding sentence is true, it's possible it will never be solved. In such a situation, a workaround becomes a legitimate tactic. SteveT Steve Litt December 2018 featured book: Rapid Learning for the 21st Century http://www.troubleshooters.com/rl21
