tl;dr: rsyslog sees 512,000,000 bytes and tells logrotate "time to rotate!" and logrotate sees less than 512MB and say "nah, not big enough" and rsyslog never writes the logs anymore because the file is too big.
On Mon, Sep 15, 2014 at 10:07 AM, Wayne Witzel <wayne.wit...@canonical.com> wrote: > When this was being implemented and during the review process the actual > size of the rotated files got adjusted. I tested very thoroughly the actual > rotation script and also tested the triggering of the rotation script. > Enough that I was happy landing the changes. > > When the sizes were adjusted the same amount of diligence was not applied > in testing. This is what introduced the bug. > > The size mismatch is problem. rsyslogd will stop logging to the file and > attempted to execute the script in the outchannel every time it gets a new > log message, which runs logrotate, which does nothing because the file > isn't big enough. > > Here is the rsyslog docs for implementation of sized based rotation. > http://www.rsyslog.com/doc/log_rotation_fix_size.html > > I first attempted use the mv command, but with our rsyslogd configuration, > when I performed a mv, rsyslogd would stop logging to the file until I > actually did a reload on the process. Also this didn't easily support > things like compression or adding more archives if we decided to keep more > than one rotation around. So the choice to use logrotate is was made. > > As for the actual act of reloading rsyslog. You can use copytruncate to > avoid it all together or you can use postrotate to reload rsyslogd. In both > cases there is a small window of possible data loss. Though with > copyandtruncate it is more likley to happen when the system is under load. > So changing to a postrotate that reloads rsyslogd is probably a good idea. > > The rotate setting is just the number of files to keep around. I asked > some people after the size update if 1 rotation, a total of 1GB was enough > to keep around. Most people thought it was fine, since the original purpose > of the sized based long rotation was to address a ticket that was > complaining about the juju log folder taking up 3GB+ of space. > > I will get a ticket created to address the size mismatch issue and switch > the copyandtruncate approach to a postrotate of rsyslogd. > > Wayne > > On Mon, Sep 15, 2014 at 5:30 AM, John Meinel <j...@arbash-meinel.com> > wrote: > >> So we are using rsyslog.conf to have it figure out when rotation needs to >> be done with: >> # Maximum size for the log on this outchannel is 512MB >> # The command to execute when an outchannel as reached its size limit >> cannot accept any arguments >> # that is why we have created the helper script for executing logrotate. >> $outchannel >> logRotation,{{logDir}}/all-machines.log,512000000,{{logrotateHelperPath}} >> >> I would think that would not require SIGHUP along with having it run the >> script at our request. >> >> John >> =:-> >> >> On Mon, Sep 15, 2014 at 12:13 PM, Stuart Bishop < >> stuart.bis...@canonical.com> wrote: >> >>> On 15 September 2014 12:38, John Meinel <j...@arbash-meinel.com> wrote: >>> >>> > 7) "copytruncate" seems the wrong setting for interactive with >>> rsyslog. I >>> > believe rsyslog is already aware that the file needs to be rotated, >>> and thus >>> >>> It is only aware if you sent it a HUP signal. >>> >>> > it shouldn't be trying to write to the same file handle (and thus we >>> don't >>> > need to truncate in place). I'm not 100% sure on the interactions >>> here, but >>> > "copytruncate" seems to have an inherent likelyhood of dropping data >>> (while >>> > you are copying, if any data gets written then you'll miss those last >>> few >>> > bytes when you go to truncate, right?) >>> >>> -- >>> Stuart Bishop <stuart.bis...@canonical.com> >>> >> >> >> -- >> Juju-dev mailing list >> Juju-dev@lists.ubuntu.com >> Modify settings or unsubscribe at: >> https://lists.ubuntu.com/mailman/listinfo/juju-dev >> >> > > > -- > Wayne Witzel III > wayne.wit...@canonical.com > > -- > Juju-dev mailing list > Juju-dev@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/juju-dev > >
-- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
