Re: NiFi not rolling logs

Lars Winderling Mon, 10 Jul 2023 07:02:17 -0700

David,
great recommendations, thanks!
Best, Lars

On 23-07-10 15:59, David Handermann wrote:

It is worth noting that the totalSizeCap can be added at any time to an existing NiFi deployment, it is not necessary to wait for an updated release.

Also related, the nifi-deprecation.log default setting includes a 100 MB totalSizeCap to avoid excessive repetition of deprecation messages, and that has worked well for several releases.


Regards,
David Handermann

On Sun, Jul 9, 2023 at 3:26 PM Mike Thomsen <mikerthom...@gmail.com> wrote:


    The totalcapsize feature will help a lot from what I’ve seen.

    Sent from my iPhone

On Jul 8, 2023, at 9:26 AM, Lars Winderling
<lars.winderl...@posteo.de> wrote:

Hi Mike,

thanks for the advice. Our NiFi instances are running for week,
if not months. Often times until the next update, so the startup
option will bring much benefit, I fear, or am I mistaken. But
looking forward to 1.23!

On 8 July 2023 13:40:15 CEST, Mike Thomsen
<mikerthom...@gmail.com> wrote:

Lars,

You should also experiment with cleanHistoryOnStart. I did
some experimentation this morning where I set the maxHistory
to 1 (1 day vs the default of 30 which is 30 days), created a
few fake log files from previous days and NiFi immediately
cleared out those "old files" on startup. I have a Jira
ticket up to fix this for 1.x and 2.x and will likely have it
up today. Should definitely be ready for 1.23

On Sat, Jul 8, 2023 at 4:17 AM Lars Winderling
<lars.winderl...@posteo.de> wrote:

Dear NiFiers, we have been bugged so much by overflowing
logfiles, and nothing has ever helped. I thought it was
just my lack of skills...especially when NiFi has some
issues and keeps on spilling stacktraces with high
frequency to disk, it eats up space quickly. I have
created cronjobs that rotate logs every minute iff
required, and when almost no space is left, it simply
deletes old files. Will try totalCapSize etc. Thank you
for the pointers! Best, Lars

On 8 July 2023 09:33:41 CEST, "Jens M. Kofoed"
<jmkofoed....@gmail.com> wrote:

Please have a look at this old jira:
https://issues.apache.org/jira/browse/NIFI-2203
I have had issues where a processor create a log
message ever 10ms resulting in the disk is being
full. For me it seems like the maxHistory settings
only effect how many files defined by the rolling
patten to be kept. If you have defined it like this:

<fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app%d{yyyy-MM-dd}.%i.log</fileNamePattern>
MaxHistory only effect the days not the increments
file %i per day. So you can stille have thousands of
files in one day.
The totalSizeCap will delete the oldes files if the
total size hits the cap settings.

The totalSizeCap have been added in the logback.xml
file for nifi-registry where it has been added inside
the rollingPolicy section. I cound not get it to work
inside the rollingPolicy section in nifi but just
added in appender section. See my comment in the
jira: https://issues.apache.org/jira/browse/NIFI-2203

Kind regards
Jens M. Kofoed

Den lør. 8. jul. 2023 kl. 04.27 skrev Mike Thomsen
<mikerthom...@gmail.com>:

Yeah, I'm working through some of it where I have
time. I plan to have a Jira up this weekend. I'm
wondering, though, if we shouldn't consider a
spike for switching to log4j2 in 2.X because I
saw a lot of complaints about logback being
inconsistent in honoring its settings.

On Fri, Jul 7, 2023 at 10:19 PM Joe Witt
<joe.w...@gmail.com> wrote:

Hmmmm. Interesting. Can you capture these
bits of fun in a jira?

Thanks

On Fri, Jul 7, 2023 at 7:17 PM Mike Thomsen
<mikerthom...@gmail.com> wrote:

After doing some research, it appears
that <maxFileSize/> is a wonky setting
WRT how well it's honored by logback. I
let a GenerateFlowFile > LogAttribute
flow run for a long time, and it just
kept filling up. When I added
<totalSizeCap/> that appeared to force
expected behavior on total log size. We
might want to add the following:

On Fri, Jul 7, 2023 at 11:33 AM Michael
Moser <moser...@gmail.com> wrote:

Hi Mike,

You aren't alone in experiencing
this. I think logback uses a pattern
matcher on filename to discover files
to delete. If "something" happens
which causes a gap in the date
pattern, then the matcher will then
fail to pick up and delete files on
the other side of that gap.

Regards,
-- Mike M

On Thu, Jul 6, 2023 at 10:28 AM Mike
Thomsen <mikerthom...@gmail.com> wrote:

We are using the stock
configuration, and have noticed
that we have a lot of nifi-app*
logs that are well beyond the
historic data cap of 30 days in
logback.xml; some of those logs
go back to April. We also have a
bunch of 0 byte nifi-user logs
and some of the other logs are 0
bytes as well. It looks like
logback is rotating based on
time, but isn't cleaning up. Is
this expected behavior or a
problem with the configuration?

Thanks,

Mike

OpenPGP_signature
Description: OpenPGP digital signature

Re: NiFi not rolling logs

Reply via email to