Hi We have unfortunately had an incident where NiFi doing a weekend is filling up the disk with logs because a process is failing and produce hundreds of error messages per seconds. We have changed the rollingPolicy to use daily rollover with a maxHistory of 30 and maxFileSize at 100MB. When the daily logfile is going to be bigger than maxFileSize the rollingpolicy will create incrementing "subfiles" as .# for that day. But the maxHistory does not count for subfiles. So with a process producing hundreds of error messages per seconds you can have a situation where you will end up with thousands of subfiles for each day filling up the disk.
There are an attribute called "totalSizeCap" which has been asked for in JIRA: https://issues.apache.org/jira/browse/NIFI-2203 https://issues.apache.org/jira/browse/NIFI-4315 This attribute is already working, but is by default not included in the logback.xml file nor in the new stateless-logback.xml file. Example: <appender name="APP_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender"> <file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-stateless.log</file> <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy"> <!-- For daily rollover, use 'app_%d.log'. For hourly rollover, use 'app_%d{yyyy-MM-dd_HH}.log'. To GZIP rolled files, replace '.log' with '.log.gz'. To ZIP rolled files, replace '.log' with '.log.zip'. --> <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-stateless_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern> <maxFileSize>100MB</maxFileSize> <!-- keep 30 log files worth of history --> <maxHistory>30</maxHistory> </rollingPolicy> * <totalSizeCap>10GB</totalSizeCap>* <immediateFlush>true</immediateFlush> <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder"> <pattern>%date %level [%thread] %logger{40} %msg%n</pattern> </encoder> </appender> Please add this as a new default parameter kind regards Jens M. Kofoed