Re: Unable to access flowfile content

2019-12-21 Thread Joe Witt 
Hello

Based on the exception stack provided the processor was expecting to read a
X bytes from the stream and it reached end of file "EOF" before that
point.  This suggests that somehow those bytes were not written truly
written to disk or became corrupted/modified from what the flowfile
repository expected.  This would be possible under a hard restart with
default properties (allowing write delay) or in the event of a bug in core
framework code.  The NiFi 1.10.0 release has a pretty massive number of
bugs/features included so it may be worth looking through to see if any
sound relevant to your case.

Thanks

On Sat, Dec 21, 2019 at 10:36 AM Purushotham Pushpavanthar <
pushpavant...@gmail.com> wrote:

> Hi Lei,
> Could you please help me understand what caused this problem? and how to
> avoid it.
>
> Regards,
> Purushotham Pushpavanth
>
>
>
> On Thu, 19 Dec 2019 at 16:55, wangl...@geekplus.com.cn <
> wangl...@geekplus.com.cn> wrote:
>
> > Hi Purushotham，
> >
> > Since you are using cluster mode， just delete the flow.xml.gz file and
> > restart the node, the flow file will be synced from the other two nodes.
> >
> > Regards,
> > Lei
> >
> >
> >
> > wangl...@geekplus.com.cn
> >
> > From: Purushotham Pushpavanthar
> > Date: 2019-12-19 17:05
> > To: dev
> > Subject: Unable to access flowfile content
> > Hi,
> >
> > We've have 3 node production cluster running seamlessly for almost 8
> month
> > with manageable ups and downs. However, yesterday we ran into an issue in
> > one of the processors due to which CPU shot up and node went down. On
> > restart, the contents of few enqueued flowfiles went missing all of
> sudden
> > (I was unable to view content from the content viewer in UI). This also
> > resulted in below exception, when was blocking downstream processor from
> > processing any flowfile.
> > We are using version 1.9.2. It would be very helpful if you can help me
> > debug this issue.
> > 2019-12-19 07:05:03,653 ERROR [Timer-Driven Process Thread-4]
> > o.apache.nifi.processors.hive.PutHiveQL
> > PutHiveQL[id=c820350d-d6fd-183d-a3d5-006a2b14d10a]
> > PutHiveQL[id=c820350d-d6fd-183d-a3d5-006a2b14d10a] failed to process
> > session due to java.lang.RuntimeException: Failed to execute due to
> > org.apache.nifi.processor.exception.FlowFileAccessException: Could not
> read
> > from
> >
> >
> StandardFlowFileRecord[uuid=253e1652-6e3f-49c3-b190-3788fcbc1480,claim=StandardContentClaim
> > [resourceClaim=StandardResourceClaim[id=1576648697457-40,
> > container=default, section=40], offset=10977,
> > length=83],offset=0,name=hid_1004.ejuserstruct2.2019121100.sql,size=83];
> > Processor Administratively Yielded for 1 sec: java.lang.RuntimeException:
> > Failed to execute due to
> > org.apache.nifi.processor.exception.FlowFileAccessException: Could not
> read
> > from
> >
> >
> StandardFlowFileRecord[uuid=253e1652-6e3f-49c3-b190-3788fcbc1480,claim=StandardContentClaim
> > [resourceClaim=StandardResourceClaim[id=1576648697457-40,
> > container=default, section=40], offset=10977,
> > length=83],offset=0,name=hid_1004.ejuserstruct2.2019121100.sql,size=83]
> > java.lang.RuntimeException: Failed to execute due to
> > org.apache.nifi.processor.exception.FlowFileAccessException: Could not
> read
> > from
> >
> >
> StandardFlowFileRecord[uuid=253e1652-6e3f-49c3-b190-3788fcbc1480,claim=StandardContentClaim
> > [resourceClaim=StandardResourceClaim[id=1576648697457-40,
> > container=default, section=40], offset=10977,
> > length=83],offset=0,name=hid_1004.ejuserstruct2.2019121100.sql,size=83]
> >   at org.apache.nifi.processor.util.pattern.Put.onTrigger(Put.java:145)
> >   at
> >
> >
> org.apache.nifi.processors.hive.PutHiveQL.lambda$onTrigger$6(PutHiveQL.java:295)
> >   at
> >
> >
> org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114)
> >   at
> >
> >
> org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(RollbackOnFailure.java:184)
> >   at
> > org.apache.nifi.processors.hive.PutHiveQL.onTrigger(PutHiveQL.java:295)
> >   at
> >
> >
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162)
> >   at
> >
> >
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:209)
> >   at
> >
> >
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> >   at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> >   at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> >   at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> >   at
> >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> >   at
> >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> >   at
> >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> >   at
> >
> >
> java.util.concurr

Re: Unable to access flowfile content

2019-12-21 Thread Purushotham Pushpavanthar 
Hi Lei,
Could you please help me understand what caused this problem? and how to
avoid it.

Regards,
Purushotham Pushpavanth



On Thu, 19 Dec 2019 at 16:55, wangl...@geekplus.com.cn <
wangl...@geekplus.com.cn> wrote:

> Hi Purushotham，
>
> Since you are using cluster mode， just delete the flow.xml.gz file and
> restart the node, the flow file will be synced from the other two nodes.
>
> Regards,
> Lei
>
>
>
> wangl...@geekplus.com.cn
>
> From: Purushotham Pushpavanthar
> Date: 2019-12-19 17:05
> To: dev
> Subject: Unable to access flowfile content
> Hi,
>
> We've have 3 node production cluster running seamlessly for almost 8 month
> with manageable ups and downs. However, yesterday we ran into an issue in
> one of the processors due to which CPU shot up and node went down. On
> restart, the contents of few enqueued flowfiles went missing all of sudden
> (I was unable to view content from the content viewer in UI). This also
> resulted in below exception, when was blocking downstream processor from
> processing any flowfile.
> We are using version 1.9.2. It would be very helpful if you can help me
> debug this issue.
> 2019-12-19 07:05:03,653 ERROR [Timer-Driven Process Thread-4]
> o.apache.nifi.processors.hive.PutHiveQL
> PutHiveQL[id=c820350d-d6fd-183d-a3d5-006a2b14d10a]
> PutHiveQL[id=c820350d-d6fd-183d-a3d5-006a2b14d10a] failed to process
> session due to java.lang.RuntimeException: Failed to execute due to
> org.apache.nifi.processor.exception.FlowFileAccessException: Could not read
> from
>
> StandardFlowFileRecord[uuid=253e1652-6e3f-49c3-b190-3788fcbc1480,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1576648697457-40,
> container=default, section=40], offset=10977,
> length=83],offset=0,name=hid_1004.ejuserstruct2.2019121100.sql,size=83];
> Processor Administratively Yielded for 1 sec: java.lang.RuntimeException:
> Failed to execute due to
> org.apache.nifi.processor.exception.FlowFileAccessException: Could not read
> from
>
> StandardFlowFileRecord[uuid=253e1652-6e3f-49c3-b190-3788fcbc1480,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1576648697457-40,
> container=default, section=40], offset=10977,
> length=83],offset=0,name=hid_1004.ejuserstruct2.2019121100.sql,size=83]
> java.lang.RuntimeException: Failed to execute due to
> org.apache.nifi.processor.exception.FlowFileAccessException: Could not read
> from
>
> StandardFlowFileRecord[uuid=253e1652-6e3f-49c3-b190-3788fcbc1480,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1576648697457-40,
> container=default, section=40], offset=10977,
> length=83],offset=0,name=hid_1004.ejuserstruct2.2019121100.sql,size=83]
>   at org.apache.nifi.processor.util.pattern.Put.onTrigger(Put.java:145)
>   at
>
> org.apache.nifi.processors.hive.PutHiveQL.lambda$onTrigger$6(PutHiveQL.java:295)
>   at
>
> org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114)
>   at
>
> org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(RollbackOnFailure.java:184)
>   at
> org.apache.nifi.processors.hive.PutHiveQL.onTrigger(PutHiveQL.java:295)
>   at
>
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162)
>   at
>
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:209)
>   at
>
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
>   at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
>   at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>   at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>   at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>   at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.nifi.processor.exception.FlowFileAccessException:
> Could not read from
>
> StandardFlowFileRecord[uuid=253e1652-6e3f-49c3-b190-3788fcbc1480,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1576648697457-40,
> container=default, section=40], offset=10977,
> length=83],offset=0,name=hid_1004.ejuserstruct2.2019121100.sql,size=83]
>   at
> org.apache.nifi.controller.repository.io
> .FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:93)
>   at
> org.apache.nifi.controller.repository.io
> .TaskTerminationInputStream.read(TaskTerminationInputStream.java:68)
>   at org.apache.nifi.stream.io.StreamUtils.fillBuffer(StreamUtils.java:89)
>   at org.apache.nifi.stream.io.StreamUtils.fillBuffer(StreamUtils.java:72)
>   at
>
> org.apache.nifi.processors.hive.AbstractHiveQLProcessor$1.process(

Re: Unable to access flowfile content

2019-12-19 Thread wangl...@geekplus.com.cn 
Hi Purushotham， 

Since you are using cluster mode， just delete the flow.xml.gz file and restart 
the node, the flow file will be synced from the other two nodes.

Regards,
Lei



wangl...@geekplus.com.cn
 
From: Purushotham Pushpavanthar
Date: 2019-12-19 17:05
To: dev
Subject: Unable to access flowfile content
Hi,
 
We've have 3 node production cluster running seamlessly for almost 8 month
with manageable ups and downs. However, yesterday we ran into an issue in
one of the processors due to which CPU shot up and node went down. On
restart, the contents of few enqueued flowfiles went missing all of sudden
(I was unable to view content from the content viewer in UI). This also
resulted in below exception, when was blocking downstream processor from
processing any flowfile.
We are using version 1.9.2. It would be very helpful if you can help me
debug this issue.
2019-12-19 07:05:03,653 ERROR [Timer-Driven Process Thread-4]
o.apache.nifi.processors.hive.PutHiveQL
PutHiveQL[id=c820350d-d6fd-183d-a3d5-006a2b14d10a]
PutHiveQL[id=c820350d-d6fd-183d-a3d5-006a2b14d10a] failed to process
session due to java.lang.RuntimeException: Failed to execute due to
org.apache.nifi.processor.exception.FlowFileAccessException: Could not read
from
StandardFlowFileRecord[uuid=253e1652-6e3f-49c3-b190-3788fcbc1480,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1576648697457-40,
container=default, section=40], offset=10977,
length=83],offset=0,name=hid_1004.ejuserstruct2.2019121100.sql,size=83];
Processor Administratively Yielded for 1 sec: java.lang.RuntimeException:
Failed to execute due to
org.apache.nifi.processor.exception.FlowFileAccessException: Could not read
from
StandardFlowFileRecord[uuid=253e1652-6e3f-49c3-b190-3788fcbc1480,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1576648697457-40,
container=default, section=40], offset=10977,
length=83],offset=0,name=hid_1004.ejuserstruct2.2019121100.sql,size=83]
java.lang.RuntimeException: Failed to execute due to
org.apache.nifi.processor.exception.FlowFileAccessException: Could not read
from
StandardFlowFileRecord[uuid=253e1652-6e3f-49c3-b190-3788fcbc1480,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1576648697457-40,
container=default, section=40], offset=10977,
length=83],offset=0,name=hid_1004.ejuserstruct2.2019121100.sql,size=83]
  at org.apache.nifi.processor.util.pattern.Put.onTrigger(Put.java:145)
  at
org.apache.nifi.processors.hive.PutHiveQL.lambda$onTrigger$6(PutHiveQL.java:295)
  at
org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114)
  at
org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(RollbackOnFailure.java:184)
  at org.apache.nifi.processors.hive.PutHiveQL.onTrigger(PutHiveQL.java:295)
  at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162)
  at
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:209)
  at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
  at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
  at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.nifi.processor.exception.FlowFileAccessException:
Could not read from
StandardFlowFileRecord[uuid=253e1652-6e3f-49c3-b190-3788fcbc1480,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1576648697457-40,
container=default, section=40], offset=10977,
length=83],offset=0,name=hid_1004.ejuserstruct2.2019121100.sql,size=83]
  at
org.apache.nifi.controller.repository.io.FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:93)
  at
org.apache.nifi.controller.repository.io.TaskTerminationInputStream.read(TaskTerminationInputStream.java:68)
  at org.apache.nifi.stream.io.StreamUtils.fillBuffer(StreamUtils.java:89)
  at org.apache.nifi.stream.io.StreamUtils.fillBuffer(StreamUtils.java:72)
  at
org.apache.nifi.processors.hive.AbstractHiveQLProcessor$1.process(AbstractHiveQLProcessor.java:92)
  at
org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2212)
  at
org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2180)
  at
org.apache.nifi.processors.hive.AbstractHiveQLProcessor.getHiveQL(AbstractHiveQLProcessor.java:89)
  at
org.apache.nifi.processors.hive.PutHiveQL.lambda$new$4(PutHiveQL.java:220)
  at or