Thanks all! WRT Ralph's comments:
The compressed active log can be read as it is being written with no problems using e.g. zcat/zless. This can read up to the end of the data despite the GZIP not being closed. There is a concern about the application dying and not closing the GZIP compression. If we start logging again to a file damaged in this way then we will not be able to read the new data even with zless, as it stops reading at the unexpected new GZIP header. This could however be alleviated with an onStartupTriggeringPolicy for rollover - the worst that could happen in this situation is that one of the rolled over logs would be missing a GZIP trailer and therefore would not work with gunzip, but would still have all its data readable through zless. No log data would be lost. WRT Volkan's comment: This enhancement required changes to several areas of logging-log4j2 (e.g. compression actions, file managers) - I'm not sure if it could easily be split off into its own appender. -----Original Message----- From: Volkan Yazıcı <vol...@yazi.ci> Sent: Thursday, May 5, 2022 3:11 PM To: Apache Logging Developers List <dev@logging.apache.org> Subject: Re: RollingFileAppender enhancement for file compression I agree with Ralph here. My suggestion would be to publish this custom appender as a separate F/OSS project first. It can certainly be contributed back to Log4j at a later stage depending on its reception by the community. On Thu, May 5, 2022 at 9:27 AM Ralph Goers <ralph.go...@dslextreme.com> wrote: > While useful I have concerns as to what the state of the file would be > if the application dies. Also, is the file readable as it is being written? > The RollingAppenders are already overly complicated so I am reluctant > to add new features without doing something to simplify them. > > Ralph > > > On May 5, 2022, at 12:34 AM, Matt Sicker <boa...@gmail.com> wrote: > > > > That sounds pretty useful to me. Using the existing compression > > actions could make it so you can more easily support other > > compression formats. > > > > On Wed, May 4, 2022 at 4:44 PM Alexander Lambert (HE/HIM) > > <lambe...@microsoft.com.invalid> wrote: > >> > >> Hi all, > >> > >> We have a local modification to logging-log4j2/log4j-core that we > >> are > hoping to contribute back to open source. We modify the > RollingFileAppender, adding a config option to compress the file as it > is originally written instead of on rollover. The value of this is in > reducing the volume of data required to be written to disk, improving > performance. > >> > >> The summary of this change is as follows: > >> > >> * A new Boolean config option "compressOnWrite" for the > RollingFileAppender. > >> * If true, the log is written to e.g. a GZIPOutputStream instead of > a FileOutputStream. > >> * Any rollover compression action implied by the filePattern > extension is skipped as the file is already compressed, and so only a > rename is done. > >> > >> Right now our change only applies to the ".gz" compression type, > >> but it > would be possible to extend this functionality to the other > compression methods - the file extension from the fileName could be > used to detect this. The GZIP format is however especially appropriate for > this feature: > >> > >> * No need to enforce an OnStartupTriggeringPolicy for the rollover, > as it is valid to append new GZIP data to a closed GZIP file. > >> > >> Is there any reason that this enhancement would not be accepted > >> into > log4j? > >> > >> Many thanks, > >> Alex > >
