Hello, Would you be able to share your code for PutFileAzureDLStore so we can help identify if there is a concurrency problem?
-Bryan On Fri, Jun 3, 2016 at 3:39 PM, Kumiko Yada <kumiko.y...@ds-iq.com> wrote: > Hello, > > > > I wrote the following custom service control and processor. When the > custom processor is running concurrently, it’s failing often with several > different errors. Are there any special handlings for concurrently that I > need to add in the custom processor? I wrote the sample Java program which > does the same thing as the custom processor (authenticate every time the > file is created/create a file, create 2 threads and run concurrently), it’s > working fine. The custom processor also fine when this is not running > concurrently. > > > > *Custom service control – set the properties for the Microsoft Azure > Datalake Store* > > *Custom processor – authenticate, then create a file in Microsoft Azure > Datalake Store* > > > > Error1: > > 2016-06-03 12:29:31,942 INFO [pool-2815-thread-1] > c.m.aad.adal4j.AuthenticationAuthority [Correlation ID: > 64c89876-2f7b-4fbb-8e6f-fb395a47aa87] Instance discovery was successful > > 2016-06-03 12:29:31,946 ERROR [Timer-Driven Process Thread-10] > n.a.d.processors.PutFileAzureDLStore > > java.util.ConcurrentModificationException: null > > at > java.util.HashMap$HashIterator.nextNode(HashMap.java:1429) ~[na:1.8.0_77] > > at java.util.HashMap$EntryIterator.next(HashMap.java:1463) > ~[na:1.8.0_77] > > at java.util.HashMap$EntryIterator.next(HashMap.java:1461) > ~[na:1.8.0_77] > > at > com.microsoft.rest.AutoRestBaseUrl.url(AutoRestBaseUrl.java:28) ~[na:na] > > at retrofit2.RequestFactory.create(RequestFactory.java:50) > ~[na:na] > > at retrofit2.OkHttpCall.createRawCall(OkHttpCall.java:181) > ~[na:na] > > at retrofit2.OkHttpCall.execute(OkHttpCall.java:165) > ~[na:na] > > at > com.microsoft.azure.management.datalake.store.FileSystemOperationsImpl.create(FileSystemOperationsImpl.java:1432) > ~[na:na] > > at > nifi.azure.dlstore.processors.PutFileAzureDLStore.CreateFile(PutFileAzureDLStore.java:252) > ~[na:na] > > at > nifi.azure.dlstore.processors.PutFileAzureDLStore.onTrigger(PutFileAzureDLStore.java:218) > ~[na:na] > > at > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) > ~[nifi-api-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] > > at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1057) > [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] > > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) > [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] > > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) > [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] > > at > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:123) > [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] > > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_77] > > at > java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > [na:1.8.0_77] > > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > [na:1.8.0_77] > > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > [na:1.8.0_77] > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_77] > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_77] > > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77] > > > > Error2: > > 2016-06-03 12:29:24,913 WARN [Timer-Driven Process Thread-5] > o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding > PutFileAzureDLStore[id=2133fdad-54b7-46a5-9bcf-958ddd8cfac0] due to > uncaught Exception: com.microsoft.rest.ServiceException: Invalid status > code 403 > > 2016-06-03 12:29:24,915 WARN [Timer-Driven Process Thread-5] > o.a.n.c.t.ContinuallyRunProcessorTask > > com.microsoft.rest.ServiceException: Invalid status code 403 > > at > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > ~[na:1.8.0_77] > > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > ~[na:1.8.0_77] > > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > ~[na:1.8.0_77] > > at > java.lang.reflect.Constructor.newInstance(Constructor.java:423) > ~[na:1.8.0_77] > > at > com.microsoft.rest.ServiceResponseBuilder.build(ServiceResponseBuilder.java:147) > ~[na:na] > > at > com.microsoft.azure.management.datalake.store.FileSystemOperationsImpl.createDelegate(FileSystemOperationsImpl.java:1491) > ~[na:na] > > at > com.microsoft.azure.management.datalake.store.FileSystemOperationsImpl.create(FileSystemOperationsImpl.java:1432) > ~[na:na] > > at > nifi.azure.dlstore.processors.PutFileAzureDLStore.CreateFile(PutFileAzureDLStore.java:252) > ~[na:na] > > at > nifi.azure.dlstore.processors.PutFileAzureDLStore.onTrigger(PutFileAzureDLStore.java:218) > ~[na:na] > > at > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) > ~[nifi-api-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] > > at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1057) > ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] > > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) > [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] > > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) > [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] > > at > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:123) > [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] > > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_77] > > at > java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > [na:1.8.0_77] > > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > [na:1.8.0_77] > > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > [na:1.8.0_77] > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_77] > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_77] > > > > Thanks > > Kumiko > > > > > > > > > > > > >
