Re: FlowFile Logging

[James Farrington]

ah that sounds perfect for what I need to do. Thank you!

On Tue, Aug 8, 2017 at 7:20 PM, Andy LoPresto  wrote:

> You can add an appender in the conf/logback.xml file which handles
> “org.apache.nifi” and anything of ERROR level and writes to a separate log
> file, which you can then monitor/parse/send wherever you like. This will
> filter out the normal operations logging from the failures you are
> interested in. Unfortunately, by default the full stacktraces and some
> other error information will be included here as well.
>
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Aug 8, 2017, at 1:09 PM, James Farrington  wrote:
>
> Hi Andy,
>
> I am trying to catch any error that happens from any processor in my flow.
> So adding a PutFile for each and every processor would not be ideal. And I
> don't need the data to be transformed into a usable type. I am passing this
> data to Logstash as a csv file (so raw data is just fine). Any thoughts?
>
> Thank you,
> James
>
> On Mon, Aug 7, 2017 at 11:35 PM, Andy LoPresto 
> wrote:
>
> Hi James,
>
> The app log will definitely contain a lot of relevant information about
> flowfile failure, but you can also make this easier for yourself by routing
> the failure connection of the relevant processor to a PutFile/PutEmail
> processor which outputs the flowfile UUID and content claim size to a
> special error destination. This means you won’t have to parse the app log
> and filter the non-error information. You may also want to look at the
> SiteToSiteBulletinReportingTask and SiteToSiteProvenanceReportingTask to
> transform the metadata (errors = bulletins, provenance for failure events)
> into raw data that you can then operate on directly.
>
> Obviously, you can also have that failure connection go to other
> processors, back to the original to retry, etc.
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Aug 7, 2017, at 12:09 PM, James Farrington  wrote:
>
> Hi everyone,
>
> Whenever there is an error in a flow and some flow file is not processed
> properly, I need to retrieve the Flow File UUID and the size of that file.
> I checked the nifi-app.log file and it seemed that most (if not all) of the
> time this information was being sent there on a processor error. Does
> anyone know if this will always be the case? Or if there is a better way to
> capture all flow files that failed to process?
>
> Any help would be great!
>
> Thank you.
>
>
>

Re: FlowFile Logging

[Andy LoPresto]

You can add an appender in the conf/logback.xml file which handles 
“org.apache.nifi” and anything of ERROR level and writes to a separate log 
file, which you can then monitor/parse/send wherever you like. This will filter 
out the normal operations logging from the failures you are interested in. 
Unfortunately, by default the full stacktraces and some other error information 
will be included here as well.


Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Aug 8, 2017, at 1:09 PM, James Farrington  wrote:
> 
> Hi Andy,
> 
> I am trying to catch any error that happens from any processor in my flow.
> So adding a PutFile for each and every processor would not be ideal. And I
> don't need the data to be transformed into a usable type. I am passing this
> data to Logstash as a csv file (so raw data is just fine). Any thoughts?
> 
> Thank you,
> James
> 
> On Mon, Aug 7, 2017 at 11:35 PM, Andy LoPresto  > wrote:
> 
>> Hi James,
>> 
>> The app log will definitely contain a lot of relevant information about
>> flowfile failure, but you can also make this easier for yourself by routing
>> the failure connection of the relevant processor to a PutFile/PutEmail
>> processor which outputs the flowfile UUID and content claim size to a
>> special error destination. This means you won’t have to parse the app log
>> and filter the non-error information. You may also want to look at the
>> SiteToSiteBulletinReportingTask and SiteToSiteProvenanceReportingTask to
>> transform the metadata (errors = bulletins, provenance for failure events)
>> into raw data that you can then operate on directly.
>> 
>> Obviously, you can also have that failure connection go to other
>> processors, back to the original to retry, etc.
>> 
>> Andy LoPresto
>> alopre...@apache.org
>> *alopresto.apa...@gmail.com  
>> mailto:alopresto.apa...@gmail.com>>*
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>> 
>> On Aug 7, 2017, at 12:09 PM, James Farrington  wrote:
>> 
>> Hi everyone,
>> 
>> Whenever there is an error in a flow and some flow file is not processed
>> properly, I need to retrieve the Flow File UUID and the size of that file.
>> I checked the nifi-app.log file and it seemed that most (if not all) of the
>> time this information was being sent there on a processor error. Does
>> anyone know if this will always be the case? Or if there is a better way to
>> capture all flow files that failed to process?
>> 
>> Any help would be great!
>> 
>> Thank you.



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: FlowFile Logging

[James Farrington]

Hi Andy,

I am trying to catch any error that happens from any processor in my flow.
So adding a PutFile for each and every processor would not be ideal. And I
don't need the data to be transformed into a usable type. I am passing this
data to Logstash as a csv file (so raw data is just fine). Any thoughts?

Thank you,
James

On Mon, Aug 7, 2017 at 11:35 PM, Andy LoPresto  wrote:

> Hi James,
>
> The app log will definitely contain a lot of relevant information about
> flowfile failure, but you can also make this easier for yourself by routing
> the failure connection of the relevant processor to a PutFile/PutEmail
> processor which outputs the flowfile UUID and content claim size to a
> special error destination. This means you won’t have to parse the app log
> and filter the non-error information. You may also want to look at the
> SiteToSiteBulletinReportingTask and SiteToSiteProvenanceReportingTask to
> transform the metadata (errors = bulletins, provenance for failure events)
> into raw data that you can then operate on directly.
>
> Obviously, you can also have that failure connection go to other
> processors, back to the original to retry, etc.
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Aug 7, 2017, at 12:09 PM, James Farrington  wrote:
>
> Hi everyone,
>
> Whenever there is an error in a flow and some flow file is not processed
> properly, I need to retrieve the Flow File UUID and the size of that file.
> I checked the nifi-app.log file and it seemed that most (if not all) of the
> time this information was being sent there on a processor error. Does
> anyone know if this will always be the case? Or if there is a better way to
> capture all flow files that failed to process?
>
> Any help would be great!
>
> Thank you.
>
>
>

Re: FlowFile Logging

[Andy LoPresto]

Hi James,

The app log will definitely contain a lot of relevant information about 
flowfile failure, but you can also make this easier for yourself by routing the 
failure connection of the relevant processor to a PutFile/PutEmail processor 
which outputs the flowfile UUID and content claim size to a special error 
destination. This means you won’t have to parse the app log and filter the 
non-error information. You may also want to look at the 
SiteToSiteBulletinReportingTask and SiteToSiteProvenanceReportingTask to 
transform the metadata (errors = bulletins, provenance for failure events) into 
raw data that you can then operate on directly.

Obviously, you can also have that failure connection go to other processors, 
back to the original to retry, etc.

Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Aug 7, 2017, at 12:09 PM, James Farrington  wrote:
> 
> Hi everyone,
> 
> Whenever there is an error in a flow and some flow file is not processed
> properly, I need to retrieve the Flow File UUID and the size of that file.
> I checked the nifi-app.log file and it seemed that most (if not all) of the
> time this information was being sent there on a processor error. Does
> anyone know if this will always be the case? Or if there is a better way to
> capture all flow files that failed to process?
> 
> Any help would be great!
> 
> Thank you.



signature.asc
Description: Message signed with OpenPGP using GPGMail

FlowFile Logging

[James Farrington]

Hi everyone,

Whenever there is an error in a flow and some flow file is not processed
properly, I need to retrieve the Flow File UUID and the size of that file.
I checked the nifi-app.log file and it seemed that most (if not all) of the
time this information was being sent there on a processor error. Does
anyone know if this will always be the case? Or if there is a better way to
capture all flow files that failed to process?

Any help would be great!

Thank you.