that said even if it is due to crashes or even disk full cases we should figure out what happened and make it not possible. We must always work to eliminate the possibility of corruption causing events and work to recover well in the face of corruption...
On Fri, Nov 11, 2016 at 2:57 PM, Oleg Zhurakousky <ozhurakou...@hortonworks.com> wrote: > Conrad > > Is it possible that you may be dealing with corrupted repositories (swap, > flow file etc.) due to your upgrades or may be even possible crashes? > > Cheers > Oleg > > On Nov 11, 2016, at 3:11 AM, Conrad Crampton <conrad.cramp...@secdata.com> > wrote: > > Hi, > This is the flow. The incoming flow is basically a syslog message which is > parsed, enriched then saved to HDFS > 1. Parse (extracttext) > 2. Assign matching parts to attributes > 3. Enrich ip address location > 4. Assign attributes with geoenrichment > 5. Execute python script to parse useragent > 6. Create json from attributes > 7. Convert to avro (all strings) > 8. Convert to target avro schema (had to do 7 & 8 due to bug(?) where > couldn’t go directly from json to avro with integers/longs) > 9. Merge into bins (see props below) > 10. Append ‘.avro’ to filenames (for reading in Spark subsequently) > 11. Save to HDFS > > Does this help at all? > If you need anything else just shout. > Regards > Conrad > > <image001.png> > > > <image002.png> > additional out of shot > · compression level : 1 > · Keep Path : false > > > From: Oleg Zhurakousky <ozhurakou...@hortonworks.com> > Reply-To: "users@nifi.apache.org" <users@nifi.apache.org> > Date: Thursday, 10 November 2016 at 18:40 > To: "users@nifi.apache.org" <users@nifi.apache.org> > Subject: Re: NPE MergeContent processor > > Conrad > > Any chance you an provide a bit more info about your flow? > I was able to find a condition when something like this can happen, but it > would have to be with some legacy NiFi distribution, so it’s a bit puzzling, > but i really want o see if we can close the loop on this. > In any event I think it is safe to raise JIRA on this one > > Cheers > Oleg > > > On Nov 10, 2016, at 10:06 AM, Conrad Crampton <conrad.cramp...@secdata.com> > wrote: > > Hi, > The processor continues to write (to HDFS – the next processor in flow) and > doesn’t block any others coming into this processor (MergeContent), so not > quite the same observed behaviour as NIFI-2015. > If there is anything else you would like me to do to help with this more > than happy to help. > Regards > Conrad > > From: Bryan Bende <bbe...@gmail.com> > Reply-To: "users@nifi.apache.org" <users@nifi.apache.org> > Date: Thursday, 10 November 2016 at 14:59 > To: "users@nifi.apache.org" <users@nifi.apache.org> > Subject: Re: NPE MergeContent processor > > Conrad, > > Thanks for reporting this. I wonder if this is also related to: > > https://issues.apache.org/jira/browse/NIFI-2015 > > Seems like there is some case where the UUID is ending up as null. > > -Bryan > > > On Wed, Nov 9, 2016 at 11:57 AM, Conrad Crampton > <conrad.cramp...@secdata.com> wrote: > > Hi, > I saw this error after I upgraded to 1.0.0 but thought it was maybe due to > the issues I had with that upgrade (entirely my fault it turns out!), but I > have seen it a number of times since so I turned debugging on to get a > better stacktrace. Relevant log section as below. > Nothing out of the ordinary, and I never saw this in v0.6.1 or below. > I would have raised a Jira issue, but after logging in to Jira it only let > me create a service desk request (which didn’t sound right). > Regards > Conrad > > 2016-11-09 16:43:46,413 DEBUG [Timer-Driven Process Thread-5] > o.a.n.processors.standard.MergeContent > MergeContent[id=12c0bec7-68b7-3b60-a020-afcc7b4599e7] has chosen to yield > its resources; will not be scheduled to run again for 1000 milliseconds > 2016-11-09 16:43:46,414 DEBUG [Timer-Driven Process Thread-5] > o.a.n.processors.standard.MergeContent > MergeContent[id=8db3bb68-0354-3116-96c5-dc80854ef116] Binned 42 FlowFiles > 2016-11-09 16:43:46,418 INFO [Timer-Driven Process Thread-5] > o.a.n.processors.standard.MergeContent > MergeContent[id=8db3bb68-0354-3116-96c5-dc80854ef116] Merged > [StandardFlowFileRecord[uuid=5e846136-0a7a-46fb-be96-8200d5cdd33d,claim=StandardContentClaim > [resourceClaim=StandardResourceClaim[id=1475059643340-275849, > container=default, section=393], offset=567158, > length=2337],offset=0,name=17453303363322987,size=2337], > StandardFlowFileRecord[uuid=a5f4bd55-82e3-40cb-9fa9-86b9e6816f67,claim=StandardContentClaim > [resourceClaim=StandardResourceClaim[id=1475059643340-275849, > container=default, section=393], offset=573643, > length=2279],offset=0,name=17453303351196175,size=2279], > StandardFlowFileRecord[uuid=c1ca745b-660a-49cd-82e5-fa8b9a2f4165,claim=StandardContentClaim > [resourceClaim=StandardResourceClaim[id=1475059643340-275849, > container=default, section=393], offset=583957, > length=2223],offset=0,name=17453303531879367,size=2223], > StandardFlowFileRecord[uuid=<null>,claim=StandardContentClaim > [resourceClaim=StandardResourceClaim[id=1475059643340-275849, > container=default, section=393], offset=595617, > length=2356],offset=0,name=<null>,size=2356], > StandardFlowFileRecord[uuid=<null>,claim=StandardContentClaim > [resourceClaim=StandardResourceClaim[id=1475059643340-275849, > container=default, section=393], offset=705637, > length=2317],offset=0,name=<null>,size=2317], > StandardFlowFileRecord[uuid=<null>,claim=StandardContentClaim > [resourceClaim=StandardResourceClaim[id=1475059643340-275849, > container=default, section=393], offset=725376, > length=2333],offset=0,name=<null>,size=2333], > StandardFlowFileRecord[uuid=<null>,claim=StandardContentClaim > [resourceClaim=StandardResourceClaim[id=1475059643340-275849, > container=default, section=393], offset=728703, > length=2377],offset=0,name=<null>,size=2377]] into > StandardFlowFileRecord[uuid=1ef3e5a0-f8db-49eb-935d-ed3c991fd631,claim=StandardContentClaim > [resourceClaim=StandardResourceClaim[id=1478709819819-416, > container=default, section=416], offset=982498, > length=4576],offset=0,name=3649103647775837,size=4576] > 2016-11-09 16:43:46,418 ERROR [Timer-Driven Process Thread-5] > o.a.n.processors.standard.MergeContent > MergeContent[id=8db3bb68-0354-3116-96c5-dc80854ef116] > MergeContent[id=8db3bb68-0354-3116-96c5-dc80854ef116] failed to process > session due to java.lang.NullPointerException: > java.lang.NullPointerException > 2016-11-09 16:43:46,422 ERROR [Timer-Driven Process Thread-5] > o.a.n.processors.standard.MergeContent > java.lang.NullPointerException: null > at > org.apache.nifi.stream.io.DataOutputStream.writeUTF(DataOutputStream.java:300) > ~[nifi-utils-1.0.0.jar:1.0.0] > at > org.apache.nifi.stream.io.DataOutputStream.writeUTF(DataOutputStream.java:281) > ~[nifi-utils-1.0.0.jar:1.0.0] > at > org.apache.nifi.provenance.StandardRecordWriter.writeUUID(StandardRecordWriter.java:257) > ~[na:na] > at > org.apache.nifi.provenance.StandardRecordWriter.writeUUIDs(StandardRecordWriter.java:266) > ~[na:na] > at > org.apache.nifi.provenance.StandardRecordWriter.writeRecord(StandardRecordWriter.java:232) > ~[na:na] > at > org.apache.nifi.provenance.PersistentProvenanceRepository.persistRecord(PersistentProvenanceRepository.java:766) > ~[na:na] > at > org.apache.nifi.provenance.PersistentProvenanceRepository.registerEvents(PersistentProvenanceRepository.java:432) > ~[na:na] > at > org.apache.nifi.controller.repository.StandardProcessSession.updateProvenanceRepo(StandardProcessSession.java:713) > ~[nifi-framework-core-1.0.0.jar:1.0.0] > at > org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:311) > ~[nifi-framework-core-1.0.0.jar:1.0.0] > at > org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:299) > ~[nifi-framework-core-1.0.0.jar:1.0.0] > at > org.apache.nifi.processor.util.bin.BinFiles.processBins(BinFiles.java:256) > ~[nifi-processor-utils-1.0.0.jar:1.0.0] > at > org.apache.nifi.processor.util.bin.BinFiles.onTrigger(BinFiles.java:190) > ~[nifi-processor-utils-1.0.0.jar:1.0.0] > at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1064) > ~[nifi-framework-core-1.0.0.jar:1.0.0] > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) > [nifi-framework-core-1.0.0.jar:1.0.0] > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) > [nifi-framework-core-1.0.0.jar:1.0.0] > at > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) > [nifi-framework-core-1.0.0.jar:1.0.0] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_51] > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > [na:1.8.0_51] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > [na:1.8.0_51] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > [na:1.8.0_51] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_51] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_51] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51] > 2016-11-09 16:43:46,422 WARN [Timer-Driven Process Thread-5] > o.a.n.processors.standard.MergeContent > MergeContent[id=8db3bb68-0354-3116-96c5-dc80854ef116] Processor > Administratively Yielded for 1 sec due to processing failure > 2016-11-09 16:43:46,422 WARN [Timer-Driven Process Thread-5] > o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding > MergeContent[id=8db3bb68-0354-3116-96c5-dc80854ef116] due to uncaught > Exception: java.lang.NullPointerException > 2016-11-09 16:43:46,423 WARN [Timer-Driven Process Thread-5] > o.a.n.c.t.ContinuallyRunProcessorTask > java.lang.NullPointerException: null > at > org.apache.nifi.stream.io.DataOutputStream.writeUTF(DataOutputStream.java:300) > ~[nifi-utils-1.0.0.jar:1.0.0] > at > org.apache.nifi.stream.io.DataOutputStream.writeUTF(DataOutputStream.java:281) > ~[nifi-utils-1.0.0.jar:1.0.0] > at > org.apache.nifi.provenance.StandardRecordWriter.writeUUID(StandardRecordWriter.java:257) > ~[na:na] > at > org.apache.nifi.provenance.StandardRecordWriter.writeUUIDs(StandardRecordWriter.java:266) > ~[na:na] > at > org.apache.nifi.provenance.StandardRecordWriter.writeRecord(StandardRecordWriter.java:232) > ~[na:na] > at > org.apache.nifi.provenance.PersistentProvenanceRepository.persistRecord(PersistentProvenanceRepository.java:766) > ~[na:na] > at > org.apache.nifi.provenance.PersistentProvenanceRepository.registerEvents(PersistentProvenanceRepository.java:432) > ~[na:na] > at > org.apache.nifi.controller.repository.StandardProcessSession.updateProvenanceRepo(StandardProcessSession.java:713) > ~[nifi-framework-core-1.0.0.jar:1.0.0] > at > org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:311) > ~[nifi-framework-core-1.0.0.jar:1.0.0] > at > org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:299) > ~[nifi-framework-core-1.0.0.jar:1.0.0] > at > org.apache.nifi.processor.util.bin.BinFiles.processBins(BinFiles.java:256) > ~[nifi-processor-utils-1.0.0.jar:1.0.0] > at > org.apache.nifi.processor.util.bin.BinFiles.onTrigger(BinFiles.java:190) > ~[nifi-processor-utils-1.0.0.jar:1.0.0] > at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1064) > ~[nifi-framework-core-1.0.0.jar:1.0.0] > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) > [nifi-framework-core-1.0.0.jar:1.0.0] > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) > [nifi-framework-core-1.0.0.jar:1.0.0] > at > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) > [nifi-framework-core-1.0.0.jar:1.0.0] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_51] > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > [na:1.8.0_51] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > [na:1.8.0_51] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > [na:1.8.0_51] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_51] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_51] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51] > > > > SecureData, combating cyber threats > > ________________________________ > The information contained in this message or any of its attachments may be > privileged and confidential and intended for the exclusive use of the > intended recipient. If you are not the intended recipient any disclosure, > reproduction, distribution or other dissemination or use of this > communications is strictly prohibited. The views expressed in this email are > those of the individual and not necessarily of SecureData Europe Ltd. Any > prices quoted are only valid if followed up by a formal written quote. > SecureData Europe Limited. Registered in England & Wales 04365896. > Registered Address: SecureData House, Hermitage Court, Hermitage Lane, > Maidstone, Kent, ME16 9NT > > > > > > ***This email originated outside SecureData*** > Click here to report this email as spam. > > > >
