subject:"NiFi logging"

Re: How to control NiFi logging

2016-06-08 Thread Andre

Stephane,

Please upload both your bootstrap and logback configuration files to a gist
or pastebin so we can have a look.

Cheers

On Wed, Jun 8, 2016 at 11:50 AM, Stéphane Maarek 
wrote:

> Hi Andrew,
>
> Thanks for the quick response. Sorry stuff isn't obvious just yet. Please
> bear with me
> I copied the file you indicated to my logback.xml and didn't change a thing
>
> In bootstrap.conf, I added the following:
> org.apache.nifi.bootstrap.config.log.dir=/mnt/xvdf/logs
>
> Now on start of NiFi, the logging still doesn't work, and I'm getting the
> following folder in my nifi root:
> org.apache.nifi.bootstrap.config.log.dir_IS_UNDEFINED
>
> Can you please help?
>
> regards,
> Stephane
>
> On Wed, Jun 8, 2016 at 11:07 AM Andre  wrote:
>
>> Stephane,
>>
>> There are two settings controlling each individual log. The sections are
>> clearly commented within logback settings
>>
>> You can read the instance in your logback.xml or read the one available
>> online
>>
>>
>> https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/conf/logback.xml
>>
>> Cheers
>>
>> On Wed, Jun 8, 2016 at 10:22 AM, Stéphane Maarek <
>> stephane.maa...@gmail.com> wrote:
>>
>>> Hi Andrew,
>>>
>>> It looks like the settings don't work. only the main logs go to the
>>> indicated folder, the other dated files still go to my internal log
>>> folder...
>>> ls /mnt/xvdf/logs
>>> nifi-app.log  nifi-bootstrap.log  nifi-user.log
>>>
>>> ls ~/nifi/logs/
>>> nifi-app_2016-06-06_16.0.log   nifi-app_2016-06-07_22.1.log
>>> nifi-app_2016-06-06_17.0.log   nifi-app_2016-06-07_22.20.log
>>> nifi-app_2016-06-06_18.0.log   nifi-app_2016-06-07_22.21.log
>>> nifi-app_2016-06-06_19.0.log   nifi-app_2016-06-07_22.22.log
>>> nifi-app_2016-06-06_20.0.log   nifi-app_2016-06-07_22.23.log
>>> nifi-app_2016-06-06_21.0.log   nifi-app_2016-06-07_22.24.log
>>> nifi-app_2016-06-06_22.0.log   nifi-app_2016-06-07_22.25.log
>>> nifi-app_2016-06-06_23.0.log   nifi-app_2016-06-07_22.26.log
>>> nifi-app_2016-06-07_00.0.log   nifi-app_2016-06-07_22.27.log
>>> nifi-app_2016-06-07_01.0.log   nifi-app_2016-06-07_22.28.log
>>> nifi-app_2016-06-07_01.1.log   nifi-app_2016-06-07_22.29.log
>>> nifi-app_2016-06-07_02.0.log   nifi-app_2016-06-07_22.2.log
>>> nifi-app_2016-06-07_03.0.log   nifi-app_2016-06-07_22.30.log
>>> nifi-app_2016-06-07_04.0.log   nifi-app_2016-06-07_22.31.log
>>> nifi-app_2016-06-07_05.0.log   nifi-app_2016-06-07_22.32.log
>>> 
>>> nifi-app_2016-06-07_22.17.log  nifi-bootstrap_2016-06-06.log
>>> nifi-app_2016-06-07_22.18.log  nifi-user_2016-06-06.log
>>> nifi-app_2016-06-07_22.19.log
>>>
>>>
>>> Can you please check on your end and let me know what I need to change?
>>>
>>> Thanks,
>>> Stephane
>>>
>>> On Tue, Jun 7, 2016 at 9:49 AM Pat Trainor 
>>> wrote:
>>>
 No wonder it looked like log4j! That's some pretty slick stuff, right
 there!

 On Jun 6 2016, at 8:06 am, Andrew Psaltis 
 wrote:

> Hi Pat,
> It is all standard logback, described here: http://logback.qos.ch/
>

>>

Re: How to control NiFi logging

2016-06-07 Thread Stéphane Maarek

Hi Andrew,

Thanks for the quick response. Sorry stuff isn't obvious just yet. Please
bear with me
I copied the file you indicated to my logback.xml and didn't change a thing

In bootstrap.conf, I added the following:
org.apache.nifi.bootstrap.config.log.dir=/mnt/xvdf/logs

Now on start of NiFi, the logging still doesn't work, and I'm getting the
following folder in my nifi root:
org.apache.nifi.bootstrap.config.log.dir_IS_UNDEFINED

Can you please help?

regards,
Stephane

On Wed, Jun 8, 2016 at 11:07 AM Andre  wrote:

> Stephane,
>
> There are two settings controlling each individual log. The sections are
> clearly commented within logback settings
>
> You can read the instance in your logback.xml or read the one available
> online
>
>
> https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/conf/logback.xml
>
> Cheers
>
> On Wed, Jun 8, 2016 at 10:22 AM, Stéphane Maarek <
> stephane.maa...@gmail.com> wrote:
>
>> Hi Andrew,
>>
>> It looks like the settings don't work. only the main logs go to the
>> indicated folder, the other dated files still go to my internal log
>> folder...
>> ls /mnt/xvdf/logs
>> nifi-app.log  nifi-bootstrap.log  nifi-user.log
>>
>> ls ~/nifi/logs/
>> nifi-app_2016-06-06_16.0.log   nifi-app_2016-06-07_22.1.log
>> nifi-app_2016-06-06_17.0.log   nifi-app_2016-06-07_22.20.log
>> nifi-app_2016-06-06_18.0.log   nifi-app_2016-06-07_22.21.log
>> nifi-app_2016-06-06_19.0.log   nifi-app_2016-06-07_22.22.log
>> nifi-app_2016-06-06_20.0.log   nifi-app_2016-06-07_22.23.log
>> nifi-app_2016-06-06_21.0.log   nifi-app_2016-06-07_22.24.log
>> nifi-app_2016-06-06_22.0.log   nifi-app_2016-06-07_22.25.log
>> nifi-app_2016-06-06_23.0.log   nifi-app_2016-06-07_22.26.log
>> nifi-app_2016-06-07_00.0.log   nifi-app_2016-06-07_22.27.log
>> nifi-app_2016-06-07_01.0.log   nifi-app_2016-06-07_22.28.log
>> nifi-app_2016-06-07_01.1.log   nifi-app_2016-06-07_22.29.log
>> nifi-app_2016-06-07_02.0.log   nifi-app_2016-06-07_22.2.log
>> nifi-app_2016-06-07_03.0.log   nifi-app_2016-06-07_22.30.log
>> nifi-app_2016-06-07_04.0.log   nifi-app_2016-06-07_22.31.log
>> nifi-app_2016-06-07_05.0.log   nifi-app_2016-06-07_22.32.log
>> 
>> nifi-app_2016-06-07_22.17.log  nifi-bootstrap_2016-06-06.log
>> nifi-app_2016-06-07_22.18.log  nifi-user_2016-06-06.log
>> nifi-app_2016-06-07_22.19.log
>>
>>
>> Can you please check on your end and let me know what I need to change?
>>
>> Thanks,
>> Stephane
>>
>> On Tue, Jun 7, 2016 at 9:49 AM Pat Trainor  wrote:
>>
>>> No wonder it looked like log4j! That's some pretty slick stuff, right
>>> there!
>>>
>>> On Jun 6 2016, at 8:06 am, Andrew Psaltis 
>>> wrote:
>>>
 Hi Pat,
 It is all standard logback, described here: http://logback.qos.ch/

>>>
>

Re: How to control NiFi logging

2016-06-07 Thread Andre

Stephane,

There are two settings controlling each individual log. The sections are
clearly commented within logback settings

You can read the instance in your logback.xml or read the one available
online

https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/conf/logback.xml

Cheers

On Wed, Jun 8, 2016 at 10:22 AM, Stéphane Maarek 
wrote:

> Hi Andrew,
>
> It looks like the settings don't work. only the main logs go to the
> indicated folder, the other dated files still go to my internal log
> folder...
> ls /mnt/xvdf/logs
> nifi-app.log  nifi-bootstrap.log  nifi-user.log
>
> ls ~/nifi/logs/
> nifi-app_2016-06-06_16.0.log   nifi-app_2016-06-07_22.1.log
> nifi-app_2016-06-06_17.0.log   nifi-app_2016-06-07_22.20.log
> nifi-app_2016-06-06_18.0.log   nifi-app_2016-06-07_22.21.log
> nifi-app_2016-06-06_19.0.log   nifi-app_2016-06-07_22.22.log
> nifi-app_2016-06-06_20.0.log   nifi-app_2016-06-07_22.23.log
> nifi-app_2016-06-06_21.0.log   nifi-app_2016-06-07_22.24.log
> nifi-app_2016-06-06_22.0.log   nifi-app_2016-06-07_22.25.log
> nifi-app_2016-06-06_23.0.log   nifi-app_2016-06-07_22.26.log
> nifi-app_2016-06-07_00.0.log   nifi-app_2016-06-07_22.27.log
> nifi-app_2016-06-07_01.0.log   nifi-app_2016-06-07_22.28.log
> nifi-app_2016-06-07_01.1.log   nifi-app_2016-06-07_22.29.log
> nifi-app_2016-06-07_02.0.log   nifi-app_2016-06-07_22.2.log
> nifi-app_2016-06-07_03.0.log   nifi-app_2016-06-07_22.30.log
> nifi-app_2016-06-07_04.0.log   nifi-app_2016-06-07_22.31.log
> nifi-app_2016-06-07_05.0.log   nifi-app_2016-06-07_22.32.log
> 
> nifi-app_2016-06-07_22.17.log  nifi-bootstrap_2016-06-06.log
> nifi-app_2016-06-07_22.18.log  nifi-user_2016-06-06.log
> nifi-app_2016-06-07_22.19.log
>
>
> Can you please check on your end and let me know what I need to change?
>
> Thanks,
> Stephane
>
> On Tue, Jun 7, 2016 at 9:49 AM Pat Trainor  wrote:
>
>> No wonder it looked like log4j! That's some pretty slick stuff, right
>> there!
>>
>> On Jun 6 2016, at 8:06 am, Andrew Psaltis 
>> wrote:
>>
>>> Hi Pat,
>>> It is all standard logback, described here: http://logback.qos.ch/
>>>
>>

Re: How to control NiFi logging

2016-06-07 Thread Stéphane Maarek

Hi Andrew,

It looks like the settings don't work. only the main logs go to the
indicated folder, the other dated files still go to my internal log
folder...
ls /mnt/xvdf/logs
nifi-app.log  nifi-bootstrap.log  nifi-user.log

ls ~/nifi/logs/
nifi-app_2016-06-06_16.0.log   nifi-app_2016-06-07_22.1.log
nifi-app_2016-06-06_17.0.log   nifi-app_2016-06-07_22.20.log
nifi-app_2016-06-06_18.0.log   nifi-app_2016-06-07_22.21.log
nifi-app_2016-06-06_19.0.log   nifi-app_2016-06-07_22.22.log
nifi-app_2016-06-06_20.0.log   nifi-app_2016-06-07_22.23.log
nifi-app_2016-06-06_21.0.log   nifi-app_2016-06-07_22.24.log
nifi-app_2016-06-06_22.0.log   nifi-app_2016-06-07_22.25.log
nifi-app_2016-06-06_23.0.log   nifi-app_2016-06-07_22.26.log
nifi-app_2016-06-07_00.0.log   nifi-app_2016-06-07_22.27.log
nifi-app_2016-06-07_01.0.log   nifi-app_2016-06-07_22.28.log
nifi-app_2016-06-07_01.1.log   nifi-app_2016-06-07_22.29.log
nifi-app_2016-06-07_02.0.log   nifi-app_2016-06-07_22.2.log
nifi-app_2016-06-07_03.0.log   nifi-app_2016-06-07_22.30.log
nifi-app_2016-06-07_04.0.log   nifi-app_2016-06-07_22.31.log
nifi-app_2016-06-07_05.0.log   nifi-app_2016-06-07_22.32.log

nifi-app_2016-06-07_22.17.log  nifi-bootstrap_2016-06-06.log
nifi-app_2016-06-07_22.18.log  nifi-user_2016-06-06.log
nifi-app_2016-06-07_22.19.log


Can you please check on your end and let me know what I need to change?

Thanks,
Stephane

On Tue, Jun 7, 2016 at 9:49 AM Pat Trainor  wrote:

> No wonder it looked like log4j! That's some pretty slick stuff, right
> there!
>
> On Jun 6 2016, at 8:06 am, Andrew Psaltis 
> wrote:
>
>> Hi Pat,
>> It is all standard logback, described here: http://logback.qos.ch/
>>
>

Re: How to control NiFi logging

2016-06-06 Thread Pat Trainor

No wonder it looked like log4j! That's some pretty slick stuff, right there!

  

On Jun 6 2016, at 8:06 am, Andrew Psaltis 
wrote:  

> Hi Pat,

>

> It is all standard logback, described here:

Re: How to control NiFi logging

2016-06-06 Thread Andrew Psaltis

Hi Pat,
It is all standard logback, described here: http://logback.qos.ch/

On Mon, Jun 6, 2016 at 12:18 PM, Pat Trainor  wrote:

> Andrew,
> Is any of this not standard log4j, and nifi-specific?
> On Jun 6, 2016 2:34 AM, "Andrew Psaltis"  wrote:
>
>> You are correct, sorry about that Stephane!
>>
>> On Mon, Jun 6, 2016 at 8:23 AM, Stéphane Maarek <
>> stephane.maa...@gmail.com> wrote:
>>
>>> Actually, it seems I only needed to change one line (versus two)
>>>
>>> >> class="ch.qos.logback.core.rolling.RollingFileAppender">
>>> /mnt/xvdf/logs/nifi-bootstrap.log
>>> >> class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
>>> 
>>>
>>> ./logs/nifi-bootstrap_%d.log
>>>
>>> On Mon, Jun 6, 2016 at 4:20 PM Stéphane Maarek <
>>> stephane.maa...@gmail.com> wrote:
>>>
 Hi Andrew,

 After changing the file I'm getting the following error. I changed the
 lines you indicated.

 Failed to auto configure default logger context
 Reported exception:
 ch.qos.logback.core.joran.spi.JoranException: Problem parsing XML
 document. See previously reported errors.
 at
 ch.qos.logback.core.joran.event.SaxEventRecorder.recordEvents(SaxEventRecorder.java:67)
 at
 ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:134)
 at
 ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:99)
 at
 ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:49)
 at
 ch.qos.logback.classic.util.ContextInitializer.configureByResource(ContextInitializer.java:77)
 at
 ch.qos.logback.classic.util.ContextInitializer.autoConfig(ContextInitializer.java:152)
 at
 org.slf4j.impl.StaticLoggerBinder.init(StaticLoggerBinder.java:85)
 at
 org.slf4j.impl.StaticLoggerBinder.(StaticLoggerBinder.java:55)
 at org.slf4j.LoggerFactory.bind(LoggerFactory.java:141)
 at
 org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:120)
 at
 org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:331)
 at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:283)
 at org.apache.nifi.bootstrap.RunNiFi.(RunNiFi.java:117)
 at org.apache.nifi.bootstrap.RunNiFi.main(RunNiFi.java:199)
 Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber:
 1; Premature end of file.
 at
 com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239)
 at
 com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:642)
 at
 com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:326)
 at
 ch.qos.logback.core.joran.event.SaxEventRecorder.recordEvents(SaxEventRecorder.java:61)
 ... 13 more
 06:16:11,055 |-INFO in ch.qos.logback.classic.LoggerContext[default] -
 Could NOT find resource [logback.groovy]
 06:16:11,055 |-INFO in ch.qos.logback.classic.LoggerContext[default] -
 Could NOT find resource [logback-test.xml]
 06:16:11,055 |-INFO in ch.qos.logback.classic.LoggerContext[default] -
 Found resource [logback.xml] at [file:/home/ec2-user/nifi/conf/logback.xml]
 06:16:11,127 |-ERROR in
 ch.qos.logback.core.joran.event.SaxEventRecorder@682ea4b1 -
 XML_PARSING - Parsing fatal error on line 1 and column 1
 org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Premature
 end of file.
 at org.xml.sax.SAXParseException: Premature end of file.
 at  at
 com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:198)
 at  at
 com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
 at  at
 com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:400)
 at  at
 com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:327)
 at  at
 com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1438)
 at  at
 com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:1019)
 at  at
 com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
 at  at
 com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:118)
 at  at
 com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
 at  at
 com.sun.org.apache.xerces.internal.

Re: How to control NiFi logging

2016-06-06 Thread Pat Trainor

Andrew,
Is any of this not standard log4j, and nifi-specific?
On Jun 6, 2016 2:34 AM, "Andrew Psaltis"  wrote:

> You are correct, sorry about that Stephane!
>
> On Mon, Jun 6, 2016 at 8:23 AM, Stéphane Maarek  > wrote:
>
>> Actually, it seems I only needed to change one line (versus two)
>>
>> > class="ch.qos.logback.core.rolling.RollingFileAppender">
>> /mnt/xvdf/logs/nifi-bootstrap.log
>> > class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
>> 
>>
>> ./logs/nifi-bootstrap_%d.log
>>
>> On Mon, Jun 6, 2016 at 4:20 PM Stéphane Maarek 
>> wrote:
>>
>>> Hi Andrew,
>>>
>>> After changing the file I'm getting the following error. I changed the
>>> lines you indicated.
>>>
>>> Failed to auto configure default logger context
>>> Reported exception:
>>> ch.qos.logback.core.joran.spi.JoranException: Problem parsing XML
>>> document. See previously reported errors.
>>> at
>>> ch.qos.logback.core.joran.event.SaxEventRecorder.recordEvents(SaxEventRecorder.java:67)
>>> at
>>> ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:134)
>>> at
>>> ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:99)
>>> at
>>> ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:49)
>>> at
>>> ch.qos.logback.classic.util.ContextInitializer.configureByResource(ContextInitializer.java:77)
>>> at
>>> ch.qos.logback.classic.util.ContextInitializer.autoConfig(ContextInitializer.java:152)
>>> at
>>> org.slf4j.impl.StaticLoggerBinder.init(StaticLoggerBinder.java:85)
>>> at
>>> org.slf4j.impl.StaticLoggerBinder.(StaticLoggerBinder.java:55)
>>> at org.slf4j.LoggerFactory.bind(LoggerFactory.java:141)
>>> at
>>> org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:120)
>>> at
>>> org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:331)
>>> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:283)
>>> at org.apache.nifi.bootstrap.RunNiFi.(RunNiFi.java:117)
>>> at org.apache.nifi.bootstrap.RunNiFi.main(RunNiFi.java:199)
>>> Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber:
>>> 1; Premature end of file.
>>> at
>>> com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239)
>>> at
>>> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:642)
>>> at
>>> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:326)
>>> at
>>> ch.qos.logback.core.joran.event.SaxEventRecorder.recordEvents(SaxEventRecorder.java:61)
>>> ... 13 more
>>> 06:16:11,055 |-INFO in ch.qos.logback.classic.LoggerContext[default] -
>>> Could NOT find resource [logback.groovy]
>>> 06:16:11,055 |-INFO in ch.qos.logback.classic.LoggerContext[default] -
>>> Could NOT find resource [logback-test.xml]
>>> 06:16:11,055 |-INFO in ch.qos.logback.classic.LoggerContext[default] -
>>> Found resource [logback.xml] at [file:/home/ec2-user/nifi/conf/logback.xml]
>>> 06:16:11,127 |-ERROR in
>>> ch.qos.logback.core.joran.event.SaxEventRecorder@682ea4b1 - XML_PARSING
>>> - Parsing fatal error on line 1 and column 1 org.xml.sax.SAXParseException;
>>> lineNumber: 1; columnNumber: 1; Premature end of file.
>>> at org.xml.sax.SAXParseException: Premature end of file.
>>> at  at
>>> com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:198)
>>> at  at
>>> com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
>>> at  at
>>> com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:400)
>>> at  at
>>> com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:327)
>>> at  at
>>> com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1438)
>>> at  at
>>> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:1019)
>>> at  at
>>> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
>>> at  at
>>> com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:118)
>>> at  at
>>> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
>>> at  at
>>> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
>>> at  at
>>> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
>>> at  at
>>> com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)

Re: How to control NiFi logging

2016-06-05 Thread Andrew Psaltis

You are correct, sorry about that Stephane!

On Mon, Jun 6, 2016 at 8:23 AM, Stéphane Maarek 
wrote:

> Actually, it seems I only needed to change one line (versus two)
>
>  class="ch.qos.logback.core.rolling.RollingFileAppender">
> /mnt/xvdf/logs/nifi-bootstrap.log
>  class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
> 
> ./logs/nifi-bootstrap_%d.log
>
> On Mon, Jun 6, 2016 at 4:20 PM Stéphane Maarek 
> wrote:
>
>> Hi Andrew,
>>
>> After changing the file I'm getting the following error. I changed the
>> lines you indicated.
>>
>> Failed to auto configure default logger context
>> Reported exception:
>> ch.qos.logback.core.joran.spi.JoranException: Problem parsing XML
>> document. See previously reported errors.
>> at
>> ch.qos.logback.core.joran.event.SaxEventRecorder.recordEvents(SaxEventRecorder.java:67)
>> at
>> ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:134)
>> at
>> ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:99)
>> at
>> ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:49)
>> at
>> ch.qos.logback.classic.util.ContextInitializer.configureByResource(ContextInitializer.java:77)
>> at
>> ch.qos.logback.classic.util.ContextInitializer.autoConfig(ContextInitializer.java:152)
>> at
>> org.slf4j.impl.StaticLoggerBinder.init(StaticLoggerBinder.java:85)
>> at
>> org.slf4j.impl.StaticLoggerBinder.(StaticLoggerBinder.java:55)
>> at org.slf4j.LoggerFactory.bind(LoggerFactory.java:141)
>> at
>> org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:120)
>> at
>> org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:331)
>> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:283)
>> at org.apache.nifi.bootstrap.RunNiFi.(RunNiFi.java:117)
>> at org.apache.nifi.bootstrap.RunNiFi.main(RunNiFi.java:199)
>> Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1;
>> Premature end of file.
>> at
>> com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239)
>> at
>> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:642)
>> at
>> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:326)
>> at
>> ch.qos.logback.core.joran.event.SaxEventRecorder.recordEvents(SaxEventRecorder.java:61)
>> ... 13 more
>> 06:16:11,055 |-INFO in ch.qos.logback.classic.LoggerContext[default] -
>> Could NOT find resource [logback.groovy]
>> 06:16:11,055 |-INFO in ch.qos.logback.classic.LoggerContext[default] -
>> Could NOT find resource [logback-test.xml]
>> 06:16:11,055 |-INFO in ch.qos.logback.classic.LoggerContext[default] -
>> Found resource [logback.xml] at [file:/home/ec2-user/nifi/conf/logback.xml]
>> 06:16:11,127 |-ERROR in
>> ch.qos.logback.core.joran.event.SaxEventRecorder@682ea4b1 - XML_PARSING
>> - Parsing fatal error on line 1 and column 1 org.xml.sax.SAXParseException;
>> lineNumber: 1; columnNumber: 1; Premature end of file.
>> at org.xml.sax.SAXParseException: Premature end of file.
>> at  at
>> com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:198)
>> at  at
>> com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
>> at  at
>> com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:400)
>> at  at
>> com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:327)
>> at  at
>> com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1438)
>> at  at
>> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:1019)
>> at  at
>> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
>> at  at
>> com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:118)
>> at  at
>> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
>> at  at
>> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
>> at  at
>> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
>> at  at
>> com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
>> at  at
>> com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
>> at  at
>> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAX

Re: How to control NiFi logging

2016-06-05 Thread Stéphane Maarek

Actually, it seems I only needed to change one line (versus two)


/mnt/xvdf/logs/nifi-bootstrap.log


./logs/nifi-bootstrap_%d.log

On Mon, Jun 6, 2016 at 4:20 PM Stéphane Maarek 
wrote:

> Hi Andrew,
>
> After changing the file I'm getting the following error. I changed the
> lines you indicated.
>
> Failed to auto configure default logger context
> Reported exception:
> ch.qos.logback.core.joran.spi.JoranException: Problem parsing XML
> document. See previously reported errors.
> at
> ch.qos.logback.core.joran.event.SaxEventRecorder.recordEvents(SaxEventRecorder.java:67)
> at
> ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:134)
> at
> ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:99)
> at
> ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:49)
> at
> ch.qos.logback.classic.util.ContextInitializer.configureByResource(ContextInitializer.java:77)
> at
> ch.qos.logback.classic.util.ContextInitializer.autoConfig(ContextInitializer.java:152)
> at
> org.slf4j.impl.StaticLoggerBinder.init(StaticLoggerBinder.java:85)
> at
> org.slf4j.impl.StaticLoggerBinder.(StaticLoggerBinder.java:55)
> at org.slf4j.LoggerFactory.bind(LoggerFactory.java:141)
> at
> org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:120)
> at
> org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:331)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:283)
> at org.apache.nifi.bootstrap.RunNiFi.(RunNiFi.java:117)
> at org.apache.nifi.bootstrap.RunNiFi.main(RunNiFi.java:199)
> Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1;
> Premature end of file.
> at
> com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239)
> at
> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:642)
> at
> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:326)
> at
> ch.qos.logback.core.joran.event.SaxEventRecorder.recordEvents(SaxEventRecorder.java:61)
> ... 13 more
> 06:16:11,055 |-INFO in ch.qos.logback.classic.LoggerContext[default] -
> Could NOT find resource [logback.groovy]
> 06:16:11,055 |-INFO in ch.qos.logback.classic.LoggerContext[default] -
> Could NOT find resource [logback-test.xml]
> 06:16:11,055 |-INFO in ch.qos.logback.classic.LoggerContext[default] -
> Found resource [logback.xml] at [file:/home/ec2-user/nifi/conf/logback.xml]
> 06:16:11,127 |-ERROR in
> ch.qos.logback.core.joran.event.SaxEventRecorder@682ea4b1 - XML_PARSING -
> Parsing fatal error on line 1 and column 1 org.xml.sax.SAXParseException;
> lineNumber: 1; columnNumber: 1; Premature end of file.
> at org.xml.sax.SAXParseException: Premature end of file.
> at  at
> com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:198)
> at  at
> com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
> at  at
> com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:400)
> at  at
> com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:327)
> at  at
> com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1438)
> at  at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:1019)
> at  at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
> at  at
> com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:118)
> at  at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
> at  at
> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
> at  at
> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
> at  at
> com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
> at  at
> com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
> at  at
> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:642)
> at  at
> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:326)
> at  at
> ch.qos.logback.core.joran.event.SaxEventRecorder.recordEvents(SaxEventRecorder.java:61)
> at  at
> ch.qos.logback.core.joran.GenericConfig

Re: How to control NiFi logging

2016-06-05 Thread Stéphane Maarek

Hi Andrew,

After changing the file I'm getting the following error. I changed the
lines you indicated.

Failed to auto configure default logger context
Reported exception:
ch.qos.logback.core.joran.spi.JoranException: Problem parsing XML document.
See previously reported errors.
at
ch.qos.logback.core.joran.event.SaxEventRecorder.recordEvents(SaxEventRecorder.java:67)
at
ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:134)
at
ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:99)
at
ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:49)
at
ch.qos.logback.classic.util.ContextInitializer.configureByResource(ContextInitializer.java:77)
at
ch.qos.logback.classic.util.ContextInitializer.autoConfig(ContextInitializer.java:152)
at
org.slf4j.impl.StaticLoggerBinder.init(StaticLoggerBinder.java:85)
at
org.slf4j.impl.StaticLoggerBinder.(StaticLoggerBinder.java:55)
at org.slf4j.LoggerFactory.bind(LoggerFactory.java:141)
at
org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:120)
at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:331)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:283)
at org.apache.nifi.bootstrap.RunNiFi.(RunNiFi.java:117)
at org.apache.nifi.bootstrap.RunNiFi.main(RunNiFi.java:199)
Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1;
Premature end of file.
at
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239)
at
com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:642)
at
com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:326)
at
ch.qos.logback.core.joran.event.SaxEventRecorder.recordEvents(SaxEventRecorder.java:61)
... 13 more
06:16:11,055 |-INFO in ch.qos.logback.classic.LoggerContext[default] -
Could NOT find resource [logback.groovy]
06:16:11,055 |-INFO in ch.qos.logback.classic.LoggerContext[default] -
Could NOT find resource [logback-test.xml]
06:16:11,055 |-INFO in ch.qos.logback.classic.LoggerContext[default] -
Found resource [logback.xml] at [file:/home/ec2-user/nifi/conf/logback.xml]
06:16:11,127 |-ERROR in
ch.qos.logback.core.joran.event.SaxEventRecorder@682ea4b1 - XML_PARSING -
Parsing fatal error on line 1 and column 1 org.xml.sax.SAXParseException;
lineNumber: 1; columnNumber: 1; Premature end of file.
at org.xml.sax.SAXParseException: Premature end of file.
at  at
com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:198)
at  at
com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
at  at
com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:400)
at  at
com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:327)
at  at
com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1438)
at  at
com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:1019)
at  at
com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
at  at
com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:118)
at  at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
at  at
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
at  at
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
at  at
com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
at  at
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
at  at
com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:642)
at  at
com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:326)
at  at
ch.qos.logback.core.joran.event.SaxEventRecorder.recordEvents(SaxEventRecorder.java:61)
at  at
ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:134)
at  at
ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:99)
at  at
ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:49)
at  at
ch.qos.logback.classic.util.ContextInitializer.configureByResource(ContextInitializer.java:77)
at  at
ch.qos.logback.

Re: How to control NiFi logging

2016-06-05 Thread Andrew Psaltis

Hi Stephane,
In the conf directory of your nifi install there is a logback.xml file that
needs to be edited.  This configuration file controls the internal logging
done by nifi. For each of the appenders you will see an XML elements that
reference the "logs" directory. For example here is the config for the
app.log

*logs/nifi-app.log <--- This should be changed*

*
./logs/nifi-app_%d{-MM-dd_HH}.%i.log
<--This should be changed*

100MB

30

%date %level [%thread] %logger{40} %msg%n
true

You would want to change the references above in to your attached volume. I
have marked them with bold above. In the logback config file you will see
two other appenders as well, these same changes should be done there as
well.

Thanks,
Andrew

On Mon, Jun 6, 2016 at 6:56 AM, Stéphane Maarek 
wrote:

> Hi,
>
> I'm running NiFi on a t1.micro in AWS. I have attached a 100GB EBS volume
> at /mnt/xvdf and all the NiFi databases are there. I have a problem though.
> It seems that NiFi keeps an internal log and I can't change the position of
> it, so it's quickly clogging my t1.micro disk. See below. How can I improve
> / change that? And what's the "work" folder for?
>
> [ec2-user@ ~]$ du -s *
> 5271404 nifi
> [ec2-user@ ~]$ cd nifi
> [ec2-user@ nifi]$ du -s *
> 28  bin
> 756 conf
> 8412docs
> 427468  lib
> 64  LICENSE
> 4331244 logs
> 44  NOTICE
> 8   README
> 144 state
> 503232  work
>
>
> Thanks!
> Stephane
>

-- 
Thanks,
Andrew

Subscribe to my book: Streaming Data 

twiiter: @itmdata

How to control NiFi logging

2016-06-05 Thread Stéphane Maarek

Hi,

I'm running NiFi on a t1.micro in AWS. I have attached a 100GB EBS volume
at /mnt/xvdf and all the NiFi databases are there. I have a problem though.
It seems that NiFi keeps an internal log and I can't change the position of
it, so it's quickly clogging my t1.micro disk. See below. How can I improve
/ change that? And what's the "work" folder for?

[ec2-user@ ~]$ du -s *
5271404 nifi
[ec2-user@ ~]$ cd nifi
[ec2-user@ nifi]$ du -s *
28  bin
756 conf
8412docs
427468  lib
64  LICENSE
4331244 logs
44  NOTICE
8   README
144 state
503232  work


Thanks!
Stephane

Re: NiFi logging

2016-02-11 Thread Andrew Grande

Take a look at ReportingTask, this sounds like a good fit. Much higher level 
than explicitly putting log steps in the flow or reconfiguring the logging 
system to forward a category elsewhere.

HTH,
Andrew

[1] 
https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#reporting-tasks

From: sudeep mishra mailto:sudeepshekh...@gmail.com>>
Reply-To: "users@nifi.apache.org<mailto:users@nifi.apache.org>" 
mailto:users@nifi.apache.org>>
Date: Wednesday, February 10, 2016 at 11:59 PM
To: "users@nifi.apache.org<mailto:users@nifi.apache.org>" 
mailto:users@nifi.apache.org>>
Subject: Re: NiFi logging

Thanks for your feedback Andy.

I am looking to log the general events taking place in data flow to track my 
data at a later stage. Say I have below flow:

Flow 1 - User sending XML data over HTTP POST --> NiFi HTTPListener --> NiFi 
PutKafka processor
Flow 2 - NiFi GetKafka processor --> NiFi UpdateAttribute processor  --> NiFi RouteOnAttribute processor --> NiFi PutHDFS processor

For each such XML posted by a user I want to log the time and some attributes 
for the XML received at each flow. Such as

  Received  XML at Listener
 < NiFi PutKafka processor>  XML moved to topic 
 < NiFi GettKafka processor> Received  XML moved from topic 

..
 < NiFi PutHDFS processor> Moved  XML moved to HDFS


On Thu, Feb 11, 2016 at 10:13 AM, Andy LoPresto 
mailto:alopresto.apa...@gmail.com>> wrote:
Hi Sudeep.

SLF4J Loggers are available throughout the code. Are you trying to log details 
from inside a processor? Within the framework, there are various 
implementations of `ComponentLog` [1], and `ProcessorLog` (just a marker 
interface), which is implemented by `SimpleProcessLogger` [2]. From within a 
processor which extends `AbstractProcessor` [3], you can call `getLogger()` to 
return an instance of `ProcessorLog` and then write to it as you would any 
SLF4J logger instance.

```java


String key = "someKey";
String value = "someValue";
final ProcessorLog logger = getLogger();
logger.warn("This is a warning message");
logger.info<http://logger.info>("This is a message with some interpolation 
key={}, value={}", new String[]{key, value});

```

This will result in the output:

```
[pool-2-thread-1] WARN org.apache.nifi.processors.standard.EncryptContent - 
EncryptContent[id=1c569c19-d37b-4d30-aec7-410b318c360d] This is a warning 
message
[pool-2-thread-1] INFO org.apache.nifi.processors.standard.EncryptContent - 
EncryptContent[id=1c569c19-d37b-4d30-aec7-410b318c360d] This is a message with 
some interpolation key=someKey, value=someValue
```

We recommend using SLF4J, but if you have a requirement to use Log4j, you can 
use an adapter as described here [4].

[1] 
https://github.com/apache/nifi/blob/3a7ddc6a35f143c1d4a735226fe7467a83d04583/nifi-api/src/main/java/org/apache/nifi/logging/ComponentLog.java
[2] 
https://github.com/apache/nifi/blob/807eb8de6d7054cef1fac91cbf2e2a1eb37d30ed/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/processor/SimpleProcessLogger.java
[3] 
https://github.com/apache/nifi/blob/31fba6b3332978ca2f6a1d693f6053d719fb9daa/nifi-api/src/main/java/org/apache/nifi/processor/AbstractProcessor.java
[4] http://www.slf4j.org/legacy.html


Andy LoPresto
alopresto.apa...@gmail.com<mailto:alopresto.apa...@gmail.com>
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Feb 10, 2016, at 7:54 PM, sudeep mishra 
mailto:sudeepshekh...@gmail.com>> wrote:

Hi,

What is the preferred practice for logging details for NiFi data flow? How can 
I use my own logging using log4j to log custom details for NiFi data flow.


Thanks & Regards,

Sudeep




--
Thanks & Regards,

Sudeep

Re: NiFi logging

2016-02-11 Thread Joe Witt

There is also the provenance data itself which captures this and there
is the LogAttributes processor.

Multiple options to get the information about the flow out.
Thanks
Joe

On Thu, Feb 11, 2016 at 10:34 AM, Andrew Grande  wrote:
> Take a look at ReportingTask, this sounds like a good fit. Much higher level
> than explicitly putting log steps in the flow or reconfiguring the logging
> system to forward a category elsewhere.
>
> HTH,
> Andrew
>
> [1]
> https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#reporting-tasks
>
> From: sudeep mishra 
> Reply-To: "users@nifi.apache.org" 
> Date: Wednesday, February 10, 2016 at 11:59 PM
> To: "users@nifi.apache.org" 
> Subject: Re: NiFi logging
>
> Thanks for your feedback Andy.
>
> I am looking to log the general events taking place in data flow to track my
> data at a later stage. Say I have below flow:
>
> Flow 1 - User sending XML data over HTTP POST --> NiFi HTTPListener --> NiFi
> PutKafka processor
> Flow 2 - NiFi GetKafka processor --> NiFi UpdateAttribute processor  some metaadta> --> NiFi RouteOnAttribute processor --> NiFi PutHDFS
> processor
>
> For each such XML posted by a user I want to log the time and some
> attributes for the XML received at each flow. Such as
>
>   Received  XML at Listener
>  < NiFi PutKafka processor>  XML moved to topic 
>  < NiFi GettKafka processor> Received  XML moved from topic
> 
> ..
>  < NiFi PutHDFS processor> Moved  XML moved to HDFS
>
>
> On Thu, Feb 11, 2016 at 10:13 AM, Andy LoPresto 
> wrote:
>>
>> Hi Sudeep.
>>
>> SLF4J Loggers are available throughout the code. Are you trying to log
>> details from inside a processor? Within the framework, there are various
>> implementations of `ComponentLog` [1], and `ProcessorLog` (just a marker
>> interface), which is implemented by `SimpleProcessLogger` [2]. From within a
>> processor which extends `AbstractProcessor` [3], you can call `getLogger()`
>> to return an instance of `ProcessorLog` and then write to it as you would
>> any SLF4J logger instance.
>>
>> ```java
>>
>> String key = "someKey";
>> String value = "someValue";
>> final ProcessorLog logger = getLogger();
>> logger.warn("This is a warning message");
>> logger.info("This is a message with some interpolation key={}, value={}",
>> new String[]{key, value});
>>
>> ```
>>
>> This will result in the output:
>>
>> ```
>> [pool-2-thread-1] WARN org.apache.nifi.processors.standard.EncryptContent
>> - EncryptContent[id=1c569c19-d37b-4d30-aec7-410b318c360d] This is a warning
>> message
>> [pool-2-thread-1] INFO org.apache.nifi.processors.standard.EncryptContent
>> - EncryptContent[id=1c569c19-d37b-4d30-aec7-410b318c360d] This is a message
>> with some interpolation key=someKey, value=someValue
>> ```
>>
>> We recommend using SLF4J, but if you have a requirement to use Log4j, you
>> can use an adapter as described here [4].
>>
>> [1]
>> https://github.com/apache/nifi/blob/3a7ddc6a35f143c1d4a735226fe7467a83d04583/nifi-api/src/main/java/org/apache/nifi/logging/ComponentLog.java
>> [2]
>> https://github.com/apache/nifi/blob/807eb8de6d7054cef1fac91cbf2e2a1eb37d30ed/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/processor/SimpleProcessLogger.java
>> [3]
>> https://github.com/apache/nifi/blob/31fba6b3332978ca2f6a1d693f6053d719fb9daa/nifi-api/src/main/java/org/apache/nifi/processor/AbstractProcessor.java
>> [4] http://www.slf4j.org/legacy.html
>>
>>
>> Andy LoPresto
>> alopresto.apa...@gmail.com
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>
>> On Feb 10, 2016, at 7:54 PM, sudeep mishra 
>> wrote:
>>
>> Hi,
>>
>> What is the preferred practice for logging details for NiFi data flow? How
>> can I use my own logging using log4j to log custom details for NiFi data
>> flow.
>>
>>
>> Thanks & Regards,
>>
>> Sudeep
>>
>>
>
>
>
> --
> Thanks & Regards,
>
> Sudeep

Re: NiFi logging

2016-02-10 Thread sudeep mishra

Thanks for your feedback Andy.

I am looking to log the general events taking place in data flow to track
my data at a later stage. Say I have below flow:

Flow 1 - User sending XML data over HTTP POST --> NiFi HTTPListener -->
NiFi PutKafka processor
Flow 2 - NiFi GetKafka processor --> NiFi UpdateAttribute processor
 --> NiFi RouteOnAttribute processor --> NiFi
PutHDFS processor

For each such XML posted by a user I want to log the time and some
attributes for the XML received at each flow. Such as

  Received  XML at Listener
 < NiFi PutKafka processor>  XML moved to topic 
 < NiFi GettKafka processor> Received  XML moved from topic

..
 < NiFi PutHDFS processor> Moved  XML moved to HDFS


On Thu, Feb 11, 2016 at 10:13 AM, Andy LoPresto 
wrote:

> Hi Sudeep.
>
> SLF4J Loggers are available throughout the code. Are you trying to log
> details from inside a processor? Within the framework, there are various
> implementations of `ComponentLog` [1], and `ProcessorLog` (just a marker
> interface), which is implemented by `SimpleProcessLogger` [2]. From within
> a processor which extends `AbstractProcessor` [3], you can call
> `getLogger()` to return an instance of `ProcessorLog` and then write to it
> as you would any SLF4J logger instance.
>
> ```java
>
> String key = "someKey";
> String value = "someValue";
> final ProcessorLog logger = getLogger();
> logger.warn("This is a warning message");
> logger.info("This is a message with some interpolation key={}, value={}", new 
> String[]{key, value});
>
> ```
>
> This will result in the output:
>
> ```
> [pool-2-thread-1] WARN org.apache.nifi.processors.standard.EncryptContent
> - EncryptContent[id=1c569c19-d37b-4d30-aec7-410b318c360d] This is a warning
> message
> [pool-2-thread-1] INFO org.apache.nifi.processors.standard.EncryptContent
> - EncryptContent[id=1c569c19-d37b-4d30-aec7-410b318c360d] This is a message
> with some interpolation key=someKey, value=someValue
> ```
>
> We recommend using SLF4J, but if you have a requirement to use Log4j, you
> can use an adapter as described here [4].
>
> [1]
> https://github.com/apache/nifi/blob/3a7ddc6a35f143c1d4a735226fe7467a83d04583/nifi-api/src/main/java/org/apache/nifi/logging/ComponentLog.java
> [2]
> https://github.com/apache/nifi/blob/807eb8de6d7054cef1fac91cbf2e2a1eb37d30ed/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/processor/SimpleProcessLogger.java
> [3]
> https://github.com/apache/nifi/blob/31fba6b3332978ca2f6a1d693f6053d719fb9daa/nifi-api/src/main/java/org/apache/nifi/processor/AbstractProcessor.java
> [4] http://www.slf4j.org/legacy.html
>
>
> Andy LoPresto
> alopresto.apa...@gmail.com
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Feb 10, 2016, at 7:54 PM, sudeep mishra 
> wrote:
>
> Hi,
>
> What is the preferred practice for logging details for NiFi data flow? How
> can I use my own logging using log4j to log custom details for NiFi data
> flow.
>
>
> Thanks & Regards,
>
> Sudeep
>
>
>


-- 
Thanks & Regards,

Sudeep

Re: NiFi logging

2016-02-10 Thread Andy LoPresto

Hi Sudeep.

SLF4J Loggers are available throughout the code. Are you trying to log details 
from inside a processor? Within the framework, there are various 
implementations of `ComponentLog` [1], and `ProcessorLog` (just a marker 
interface), which is implemented by `SimpleProcessLogger` [2]. From within a 
processor which extends `AbstractProcessor` [3], you can call `getLogger()` to 
return an instance of `ProcessorLog` and then write to it as you would any 
SLF4J logger instance.

```java

String key = "someKey";
String value = "someValue";
final ProcessorLog logger = getLogger();
logger.warn("This is a warning message");
logger.info("This is a message with some interpolation key={}, value={}", new 
String[]{key, value});
```

This will result in the output:

```
[pool-2-thread-1] WARN org.apache.nifi.processors.standard.EncryptContent - 
EncryptContent[id=1c569c19-d37b-4d30-aec7-410b318c360d] This is a warning 
message
[pool-2-thread-1] INFO org.apache.nifi.processors.standard.EncryptContent - 
EncryptContent[id=1c569c19-d37b-4d30-aec7-410b318c360d] This is a message with 
some interpolation key=someKey, value=someValue
```

We recommend using SLF4J, but if you have a requirement to use Log4j, you can 
use an adapter as described here [4].

[1] 
https://github.com/apache/nifi/blob/3a7ddc6a35f143c1d4a735226fe7467a83d04583/nifi-api/src/main/java/org/apache/nifi/logging/ComponentLog.java
[2] 
https://github.com/apache/nifi/blob/807eb8de6d7054cef1fac91cbf2e2a1eb37d30ed/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/processor/SimpleProcessLogger.java
[3] 
https://github.com/apache/nifi/blob/31fba6b3332978ca2f6a1d693f6053d719fb9daa/nifi-api/src/main/java/org/apache/nifi/processor/AbstractProcessor.java
[4] http://www.slf4j.org/legacy.html


Andy LoPresto
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Feb 10, 2016, at 7:54 PM, sudeep mishra  wrote:
> 
> Hi,
> 
> What is the preferred practice for logging details for NiFi data flow? How 
> can I use my own logging using log4j to log custom details for NiFi data flow.
> 
> 
> Thanks & Regards,
> 
> Sudeep



signature.asc
Description: Message signed with OpenPGP using GPGMail

NiFi logging

2016-02-10 Thread sudeep mishra

Hi,

What is the preferred practice for logging details for NiFi data flow? How
can I use my own logging using log4j to log custom details for NiFi data
flow.


Thanks & Regards,

Sudeep

Re: How to control NiFi logging

Re: How to control NiFi logging

Re: How to control NiFi logging

Re: How to control NiFi logging

Re: How to control NiFi logging

Re: How to control NiFi logging

Re: How to control NiFi logging

Re: How to control NiFi logging

Re: How to control NiFi logging

Re: How to control NiFi logging

Re: How to control NiFi logging

How to control NiFi logging

Re: NiFi logging

Re: NiFi logging

Re: NiFi logging

Re: NiFi logging

NiFi logging

17 matches

Site Navigation

Mail list logo

Footer information