It is worth noting that the totalSizeCap can be added at any time to an existing NiFi deployment, it is not necessary to wait for an updated release.
Also related, the nifi-deprecation.log default setting includes a 100 MB totalSizeCap to avoid excessive repetition of deprecation messages, and that has worked well for several releases. Regards, David Handermann On Sun, Jul 9, 2023 at 3:26 PM Mike Thomsen <mikerthom...@gmail.com> wrote: > The totalcapsize feature will help a lot from what I’ve seen. > > Sent from my iPhone > > On Jul 8, 2023, at 9:26 AM, Lars Winderling <lars.winderl...@posteo.de> > wrote: > > > Hi Mike, > > thanks for the advice. Our NiFi instances are running for week, if not > months. Often times until the next update, so the startup option will bring > much benefit, I fear, or am I mistaken. But looking forward to 1.23! > > > On 8 July 2023 13:40:15 CEST, Mike Thomsen <mikerthom...@gmail.com> wrote: > >> Lars, >> >> You should also experiment with cleanHistoryOnStart. I did some >> experimentation this morning where I set the maxHistory to 1 (1 day vs the >> default of 30 which is 30 days), created a few fake log files from previous >> days and NiFi immediately cleared out those "old files" on startup. I have >> a Jira ticket up to fix this for 1.x and 2.x and will likely have it up >> today. Should definitely be ready for 1.23 >> >> On Sat, Jul 8, 2023 at 4:17 AM Lars Winderling <lars.winderl...@posteo.de> >> wrote: >> >>> Dear NiFiers, we have been bugged so much by overflowing logfiles, and >>> nothing has ever helped. I thought it was just my lack of >>> skills...especially when NiFi has some issues and keeps on spilling >>> stacktraces with high frequency to disk, it eats up space quickly. I have >>> created cronjobs that rotate logs every minute iff required, and when >>> almost no space is left, it simply deletes old files. Will try totalCapSize >>> etc. Thank you for the pointers! Best, Lars >>> >>> >>> On 8 July 2023 09:33:41 CEST, "Jens M. Kofoed" <jmkofoed....@gmail.com> >>> wrote: >>> >>>> Hi >>>> >>>> Please have a look at this old jira: >>>> https://issues.apache.org/jira/browse/NIFI-2203 >>>> I have had issues where a processor create a log message ever 10ms >>>> resulting in the disk is being full. For me it seems like the maxHistory >>>> settings only effect how many files defined by the rolling patten to be >>>> kept. If you have defined it like this: >>>> >>>> <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app%d{yyyy-MM-dd}.%i.log</fileNamePattern> >>>> MaxHistory only effect the days not the increments file %i per day. So >>>> you can stille have thousands of files in one day. >>>> The totalSizeCap will delete the oldes files if the total size hits >>>> the cap settings. >>>> >>>> The totalSizeCap have been added in the logback.xml file for >>>> nifi-registry where it has been added inside the rollingPolicy section. I >>>> cound not get it to work inside the rollingPolicy section in nifi but just >>>> added in appender section. See my comment in the jira: >>>> https://issues.apache.org/jira/browse/NIFI-2203 >>>> >>>> Kind regards >>>> Jens M. Kofoed >>>> >>>> Den lør. 8. jul. 2023 kl. 04.27 skrev Mike Thomsen < >>>> mikerthom...@gmail.com>: >>>> >>>>> Yeah, I'm working through some of it where I have time. I plan to have >>>>> a Jira up this weekend. I'm wondering, though, if we shouldn't consider a >>>>> spike for switching to log4j2 in 2.X because I saw a lot of complaints >>>>> about logback being inconsistent in honoring its settings. >>>>> >>>>> On Fri, Jul 7, 2023 at 10:19 PM Joe Witt <joe.w...@gmail.com> wrote: >>>>> >>>>>> Hmmmm. Interesting. Can you capture these bits of fun in a jira? >>>>>> >>>>>> Thanks >>>>>> >>>>>> On Fri, Jul 7, 2023 at 7:17 PM Mike Thomsen <mikerthom...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> After doing some research, it appears that <maxFileSize/> is a wonky >>>>>>> setting WRT how well it's honored by logback. I let a GenerateFlowFile > >>>>>>> LogAttribute flow run for a long time, and it just kept filling up. >>>>>>> When I >>>>>>> added <totalSizeCap/> that appeared to force expected behavior on total >>>>>>> log >>>>>>> size. We might want to add the following: >>>>>>> >>>>>>> <cleanHistoryOnStart>true</cleanHistoryOnStart> >>>>>>> <totalSizeCap>50GB</totalSizeCap> >>>>>>> >>>>>>> On Fri, Jul 7, 2023 at 11:33 AM Michael Moser <moser...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Mike, >>>>>>>> >>>>>>>> You aren't alone in experiencing this. I think logback uses a >>>>>>>> pattern matcher on filename to discover files to delete. If >>>>>>>> "something" >>>>>>>> happens which causes a gap in the date pattern, then the matcher will >>>>>>>> then >>>>>>>> fail to pick up and delete files on the other side of that gap. >>>>>>>> >>>>>>>> Regards, >>>>>>>> -- Mike M >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Jul 6, 2023 at 10:28 AM Mike Thomsen < >>>>>>>> mikerthom...@gmail.com> wrote: >>>>>>>> >>>>>>>>> We are using the stock configuration, and have noticed that we >>>>>>>>> have a lot of nifi-app* logs that are well beyond the historic data >>>>>>>>> cap of >>>>>>>>> 30 days in logback.xml; some of those logs go back to April. We also >>>>>>>>> have a >>>>>>>>> bunch of 0 byte nifi-user logs and some of the other logs are 0 bytes >>>>>>>>> as >>>>>>>>> well. It looks like logback is rotating based on time, but isn't >>>>>>>>> cleaning >>>>>>>>> up. Is this expected behavior or a problem with the configuration? >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Mike >>>>>>>>> >>>>>>>>
