Hi Lei, Could you please help me understand what caused this problem? and how to avoid it.
Regards, Purushotham Pushpavanth On Thu, 19 Dec 2019 at 16:55, wangl...@geekplus.com.cn < wangl...@geekplus.com.cn> wrote: > Hi Purushotham， > > Since you are using cluster mode， just delete the flow.xml.gz file and > restart the node, the flow file will be synced from the other two nodes. > > Regards, > Lei > > > > wangl...@geekplus.com.cn > > From: Purushotham Pushpavanthar > Date: 2019-12-19 17:05 > To: dev > Subject: Unable to access flowfile content > Hi, > > We've have 3 node production cluster running seamlessly for almost 8 month > with manageable ups and downs. However, yesterday we ran into an issue in > one of the processors due to which CPU shot up and node went down. On > restart, the contents of few enqueued flowfiles went missing all of sudden > (I was unable to view content from the content viewer in UI). This also > resulted in below exception, when was blocking downstream processor from > processing any flowfile. > We are using version 1.9.2. It would be very helpful if you can help me > debug this issue. > 2019-12-19 07:05:03,653 ERROR [Timer-Driven Process Thread-4] > o.apache.nifi.processors.hive.PutHiveQL > PutHiveQL[id=c820350d-d6fd-183d-a3d5-006a2b14d10a] > PutHiveQL[id=c820350d-d6fd-183d-a3d5-006a2b14d10a] failed to process > session due to java.lang.RuntimeException: Failed to execute due to > org.apache.nifi.processor.exception.FlowFileAccessException: Could not read > from > > StandardFlowFileRecord[uuid=253e1652-6e3f-49c3-b190-3788fcbc1480,claim=StandardContentClaim > [resourceClaim=StandardResourceClaim[id=1576648697457-40, > container=default, section=40], offset=10977, > length=83],offset=0,name=hid_1004.ejuserstruct2.2019121100.sql,size=83]; > Processor Administratively Yielded for 1 sec: java.lang.RuntimeException: > Failed to execute due to > org.apache.nifi.processor.exception.FlowFileAccessException: Could not read > from > > StandardFlowFileRecord[uuid=253e1652-6e3f-49c3-b190-3788fcbc1480,claim=StandardContentClaim > [resourceClaim=StandardResourceClaim[id=1576648697457-40, > container=default, section=40], offset=10977, > length=83],offset=0,name=hid_1004.ejuserstruct2.2019121100.sql,size=83] > java.lang.RuntimeException: Failed to execute due to > org.apache.nifi.processor.exception.FlowFileAccessException: Could not read > from > > StandardFlowFileRecord[uuid=253e1652-6e3f-49c3-b190-3788fcbc1480,claim=StandardContentClaim > [resourceClaim=StandardResourceClaim[id=1576648697457-40, > container=default, section=40], offset=10977, > length=83],offset=0,name=hid_1004.ejuserstruct2.2019121100.sql,size=83] > at org.apache.nifi.processor.util.pattern.Put.onTrigger(Put.java:145) > at > > org.apache.nifi.processors.hive.PutHiveQL.lambda$onTrigger$6(PutHiveQL.java:295) > at > > org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114) > at > > org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(RollbackOnFailure.java:184) > at > org.apache.nifi.processors.hive.PutHiveQL.onTrigger(PutHiveQL.java:295) > at > > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162) > at > > org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:209) > at > > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) > at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > at > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > at > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.nifi.processor.exception.FlowFileAccessException: > Could not read from > > StandardFlowFileRecord[uuid=253e1652-6e3f-49c3-b190-3788fcbc1480,claim=StandardContentClaim > [resourceClaim=StandardResourceClaim[id=1576648697457-40, > container=default, section=40], offset=10977, > length=83],offset=0,name=hid_1004.ejuserstruct2.2019121100.sql,size=83] > at > org.apache.nifi.controller.repository.io > .FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:93) > at > org.apache.nifi.controller.repository.io > .TaskTerminationInputStream.read(TaskTerminationInputStream.java:68) > at org.apache.nifi.stream.io.StreamUtils.fillBuffer(StreamUtils.java:89) > at org.apache.nifi.stream.io.StreamUtils.fillBuffer(StreamUtils.java:72) > at > > org.apache.nifi.processors.hive.AbstractHiveQLProcessor$1.process(AbstractHiveQLProcessor.java:92) > at > > org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2212) > at > > org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2180) > at > > org.apache.nifi.processors.hive.AbstractHiveQLProcessor.getHiveQL(AbstractHiveQLProcessor.java:89) > at > org.apache.nifi.processors.hive.PutHiveQL.lambda$new$4(PutHiveQL.java:220) > at org.apache.nifi.processor.util.pattern.Put.putFlowFiles(Put.java:59) > at org.apache.nifi.processor.util.pattern.Put.onTrigger(Put.java:102) > ... 15 common frames omitted > Caused by: java.io.EOFException: null > at org.apache.nifi.stream.io.StreamUtils.skip(StreamUtils.java:270) > at > > org.apache.nifi.controller.repository.FileSystemRepository.read(FileSystemRepository.java:859) > at > org.apache.nifi.controller.repository.io > .ContentClaimInputStream.formDelegate(ContentClaimInputStream.java:154) > at > org.apache.nifi.controller.repository.io > .ContentClaimInputStream.getDelegate(ContentClaimInputStream.java:51) > at > org.apache.nifi.controller.repository.io > .ContentClaimInputStream.read(ContentClaimInputStream.java:89) > at > org.apache.nifi.controller.repository.io > .DisableOnCloseInputStream.read(DisableOnCloseInputStream.java:49) > at > org.apache.nifi.controller.repository.io > .LimitedInputStream.read(LimitedInputStream.java:86) > at > org.apache.nifi.controller.repository.io > .DisableOnCloseInputStream.read(DisableOnCloseInputStream.java:49) > at > org.apache.nifi.stream.io > .ByteCountingInputStream.read(ByteCountingInputStream.java:51) > at java.io.FilterInputStream.read(FilterInputStream.java:133) > at > org.apache.nifi.controller.repository.io > .FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:82) > ... 25 common frames omitted > 2019-12-19 07:05:03,654 WARN [Timer-Driven Process Thread-4] > o.a.n.controller.tasks.ConnectableTask Administratively Yielding > PutHiveQL[id=c820350d-d6fd-183d-a3d5-006a2b14d10a] due to uncaught > Exception: java.lang.RuntimeException: Failed to execute due to > org.apache.nifi.processor.exception.FlowFileAccessException: Could not read > from > > StandardFlowFileRecord[uuid=253e1652-6e3f-49c3-b190-3788fcbc1480,claim=StandardContentClaim > [resourceClaim=StandardResourceClaim[id=1576648697457-40, > container=default, section=40], offset=10977, > length=83],offset=0,name=hid_1004.ejuserstruct2.2019121100.sql,size=83] > java.lang.RuntimeException: Failed to execute due to > org.apache.nifi.processor.exception.FlowFileAccessException: Could not read > from > > StandardFlowFileRecord[uuid=253e1652-6e3f-49c3-b190-3788fcbc1480,claim=StandardContentClaim > [resourceClaim=StandardResourceClaim[id=1576648697457-40, > container=default, section=40], offset=10977, > length=83],offset=0,name=hid_1004.ejuserstruct2.2019121100.sql,size=83] > at org.apache.nifi.processor.util.pattern.Put.onTrigger(Put.java:145) > at > > org.apache.nifi.processors.hive.PutHiveQL.lambda$onTrigger$6(PutHiveQL.java:295) > at > > org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114) > at > > org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(RollbackOnFailure.java:184) > at > org.apache.nifi.processors.hive.PutHiveQL.onTrigger(PutHiveQL.java:295) > at > > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162) > at > > org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:209) > at > > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) > at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > at > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > at > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.nifi.processor.exception.FlowFileAccessException: > Could not read from > > StandardFlowFileRecord[uuid=253e1652-6e3f-49c3-b190-3788fcbc1480,claim=StandardContentClaim > [resourceClaim=StandardResourceClaim[id=1576648697457-40, > container=default, section=40], offset=10977, > length=83],offset=0,name=hid_1004.ejuserstruct2.2019121100.sql,size=83] > at > org.apache.nifi.controller.repository.io > .FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:93) > at > org.apache.nifi.controller.repository.io > .TaskTerminationInputStream.read(TaskTerminationInputStream.java:68) > at org.apache.nifi.stream.io.StreamUtils.fillBuffer(StreamUtils.java:89) > at org.apache.nifi.stream.io.StreamUtils.fillBuffer(StreamUtils.java:72) > at > > org.apache.nifi.processors.hive.AbstractHiveQLProcessor$1.process(AbstractHiveQLProcessor.java:92) > at > > org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2212) > at > > org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2180) > at > > org.apache.nifi.processors.hive.AbstractHiveQLProcessor.getHiveQL(AbstractHiveQLProcessor.java:89) > at > org.apache.nifi.processors.hive.PutHiveQL.lambda$new$4(PutHiveQL.java:220) > at org.apache.nifi.processor.util.pattern.Put.putFlowFiles(Put.java:59) > at org.apache.nifi.processor.util.pattern.Put.onTrigger(Put.java:102) > ... 15 common frames omitted > Caused by: java.io.EOFException: null > at org.apache.nifi.stream.io.StreamUtils.skip(StreamUtils.java:270) > at > > org.apache.nifi.controller.repository.FileSystemRepository.read(FileSystemRepository.java:859) > at > org.apache.nifi.controller.repository.io > .ContentClaimInputStream.formDelegate(ContentClaimInputStream.java:154) > at > org.apache.nifi.controller.repository.io > .ContentClaimInputStream.getDelegate(ContentClaimInputStream.java:51) > at > org.apache.nifi.controller.repository.io > .ContentClaimInputStream.read(ContentClaimInputStream.java:89) > at > org.apache.nifi.controller.repository.io > .DisableOnCloseInputStream.read(DisableOnCloseInputStream.java:49) > at > org.apache.nifi.controller.repository.io > .LimitedInputStream.read(LimitedInputStream.java:86) > at > org.apache.nifi.controller.repository.io > .DisableOnCloseInputStream.read(DisableOnCloseInputStream.java:49) > at > org.apache.nifi.stream.io > .ByteCountingInputStream.read(ByteCountingInputStream.java:51) > at java.io.FilterInputStream.read(FilterInputStream.java:133) > at > org.apache.nifi.controller.repository.io > .FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:82) > ... 25 common frames omitted > > > Regards, > Purushotham Pushpavanth >
