I'm working on a project in NiFi that needs to handle a variety of CSV Formats
with varying delimiters, headers, escape characters etc. These files will be
converted to AVRO and streamed into their respective Hive tables. I'd rather
not have hundreds of CSVReader services and I'm wondering if th
As we add more and more different processor groups doing separate things,
our nifi-app log is getting rather unmanageable. I'm probably missing
something obvious, but is one log per group possible?
Thanks
James,
One log per process group is not currently supported. Through things
like the message diagnostic context it might be pretty doable.
That said, the logs aren't really meant to be the true historical
record of value. The data provenance entries are. Can you share more
about what you're try
Sure, we have several data flow managers who are each responsible for a
number of separate data flows, each of which lives in a separate processor
group. When debugging, being able to see what's happening in your processor
group (including full stack traces) is essential, but the noise from all
the
Yeah makes sense. We could either make the process group specific
logs an option, or make tag based logs an option, or we could perhaps
make bulletins something we store/make accessible for a longer period
of time. With the first two it is probably not a lot of code changes
but then raw logs the
You can already correlate logs per processor using the id, but that's a bit
clunky especially with multi line messages (eg stack traces). It might be
possible to add the processor group id to the logging context I guess.
Another use case is monitoring production flows for intermittent errors
when