Re: Demuxing NiFi logs

2018-09-07 Thread James Srinivasan
You can already correlate logs per processor using the id, but that's a bit clunky especially with multi line messages (eg stack traces). It might be possible to add the processor group id to the logging context I guess. Another use case is monitoring production flows for intermittent errors when

Re: Demuxing NiFi logs

2018-09-07 Thread Joe Witt
Yeah makes sense. We could either make the process group specific logs an option, or make tag based logs an option, or we could perhaps make bulletins something we store/make accessible for a longer period of time. With the first two it is probably not a lot of code changes but then raw logs the

Re: Demuxing NiFi logs

2018-09-07 Thread James Srinivasan
Sure, we have several data flow managers who are each responsible for a number of separate data flows, each of which lives in a separate processor group. When debugging, being able to see what's happening in your processor group (including full stack traces) is essential, but the noise from all

Re: Demuxing NiFi logs

2018-09-07 Thread Joe Witt
James, One log per process group is not currently supported. Through things like the message diagnostic context it might be pretty doable. That said, the logs aren't really meant to be the true historical record of value. The data provenance entries are. Can you share more about what you're

Demuxing NiFi logs

2018-09-07 Thread James Srinivasan
As we add more and more different processor groups doing separate things, our nifi-app log is getting rather unmanageable. I'm probably missing something obvious, but is one log per group possible? Thanks