Manish When it is in the stuck state can you please run bin/nifi.sh dump. If you can then share the nifi-bootstrap.log that would aid us in narrowing in on a possible cause.
Thanks Joe On Thu, Dec 1, 2016 at 8:44 AM, Manish G <[email protected]> wrote: > Hi Joe, > > Here is what I can see in the App Log: > > 2016-12-01 09:28:52,004 ERROR [Timer-Driven Process Thread-4] > testing.nifi.processor.hdfs.testingPutHDFS "" > org.apache.hadoop.fs.azure.AzureException: java.util.NoSuchElementException: > An error occurred while enumerating the result, check the original exception > for details. > at > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1930) > ~[hadoop-azure-2.7.2.jar:na] > at > org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:1592) > ~[hadoop-azure-2.7.2.jar:na] > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424) > ~[hadoop-common-2.7.2.jar:na] > at > testing.nifi.processor.hdfs.testingPutHDFS.onTrigger(testingPutHDFS.java:260) > ~[testing.nifi.processor-1.0.0.nar-unpacked/:na] > at > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) > [nifi-api-0.7.0.jar:0.7.0] > at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1054) > [nifi-framework-core-0.7.0.jar:0.7.0] > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) > [nifi-framework-core-0.7.0.jar:0.7.0] > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) > [nifi-framework-core-0.7.0.jar:0.7.0] > at > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127) > [nifi-framework-core-0.7.0.jar:0.7.0] > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) > [na:1.8.0_101] > at java.util.concurrent.FutureTask.runAndReset(Unknown Source) > [na:1.8.0_101] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown > Source) [na:1.8.0_101] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown > Source) [na:1.8.0_101] > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > [na:1.8.0_101] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > [na:1.8.0_101] > at java.lang.Thread.run(Unknown Source) [na:1.8.0_101] > Caused by: java.util.NoSuchElementException: An error occurred while > enumerating the result, check the original exception for details. > at > com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:113) > ~[azure-storage-2.0.0.jar:na] > at > org.apache.hadoop.fs.azure.StorageInterfaceImpl$WrappingIterator.hasNext(StorageInterfaceImpl.java:128) > ~[hadoop-azure-2.7.2.jar:na] > at > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1909) > ~[hadoop-azure-2.7.2.jar:na] > ... 15 common frames omitted > Caused by: com.microsoft.azure.storage.StorageException: Forbidden > at > com.microsoft.azure.storage.StorageException.translateFromHttpStatus(StorageException.java:202) > ~[azure-storage-2.0.0.jar:na] > at > com.microsoft.azure.storage.StorageException.translateException(StorageException.java:172) > ~[azure-storage-2.0.0.jar:na] > at > com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:273) > ~[azure-storage-2.0.0.jar:na] > at > com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:109) > ~[azure-storage-2.0.0.jar:na] > ... 17 common frames omitted > Caused by: java.lang.NullPointerException: null > at > com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:181) > ~[azure-storage-2.0.0.jar:na] > ... 18 common frames omitted > > Regards, > Manish > > On Thu, Dec 1, 2016 at 10:36 AM, Joe Witt <[email protected]> wrote: >> >> Manish >> >> Please produce and share the thread dump I mentioned. >> >> Thanks >> Joe >> >> On Dec 1, 2016 7:23 AM, "Manish G" <[email protected]> wrote: >>> >>> Hi, >>> >>> I don't know why, but this is happening now more frequently. Where should >>> I look into to find the root cause? >>> >>> Thanks, >>> Manish >>> >>> On Wed, Nov 30, 2016 at 9:20 PM, Manish G <[email protected]> wrote: >>>> >>>> Hi Joe, >>>> >>>> Thanks for the quick reply. Yes, the processor keeps running on a single >>>> thread (even after stopping). And the number remains there even after >>>> stopping. >>>> Today, it happened on my customized putHDFS processor. Only thing >>>> different in this processor is - I have added an additional attribute that >>>> tells if the processor created the directory while loading the file on >>>> HDFS. >>>> I don't think this should be the issue though. >>>> >>>> Regards, >>>> Manish >>>> >>>> >>>> On Wed, Nov 30, 2016 at 7:05 PM, Joe Witt <[email protected]> wrote: >>>>> >>>>> Manish >>>>> >>>>> When it is stuck do you see a number in the top right corner of the >>>>> processor? When you stop it does the number remain? That number is >>>>> telling you how many threads are still executing. Which processor are >>>>> we talking about? When it is in the stuck state can you please run >>>>> bin/nifi.sh dump. If you can then share the nifi-bootstrap.log that >>>>> would aid us in narrowing in on a possible cause. >>>>> >>>>> Thanks >>>>> Joe >>>>> >>>>> On Wed, Nov 30, 2016 at 7:02 PM, Manish G <[email protected]> >>>>> wrote: >>>>> > >>>>> > Hi, >>>>> > >>>>> > I have noticed that sometime a flow file gets stuck on a processor >>>>> > for a >>>>> > very long time for no reason and then I can not even stop the >>>>> > processor to >>>>> > look at the flow flow file from queue. If I click on stop, then >>>>> > processor >>>>> > goes into a state where I cannot start/stop the processor. >>>>> > >>>>> > On restarting the NiFi, the file gets processed successfully and >>>>> > routed to >>>>> > success queue. I checked in App log, but everything seems to be >>>>> > normal for >>>>> > the flow file. I don't see anything mysterious in provenance too >>>>> > (except >>>>> > that queue time is in hours). >>>>> > >>>>> > Has anyone else faced a similar issue? What else should I check to >>>>> > identify >>>>> > the root cause for this? >>>>> > >>>>> > Thanks, >>>>> > Manish >>>> >>>> >>>> >>>> >>>> -- >>>> >>>> >>>> With Warm Regards, >>>> Manish >>> >>> >>> >>> >>> -- >>> >>> >>> With Warm Regards, >>> Manish > > > > > -- > > > With Warm Regards, > Manish
