Hello, Thanks for reporting this, we definitely want to figure out what is going on here.
Was this flow working fine before using Apache NiFi 1.1.x (or some earlier version) and then stopped working when upgrading to 1.2.0? Thanks, Bryan On Wed, May 17, 2017 at 4:49 AM, Arnaud G <greatpat...@gmail.com> wrote: > Hi! > > We are currently facing an issue and we have not yet find a potential > solution. > > We are running 1.2 and we have are facing an interaction between the Kafka > 10.2 and PutHDFS processors. > > We are connecting to an external Kafka server that require SASL/PLAIN > authentication. To enable this we are using the new Kafka processor with a > JAAS configuration in the processor to authenticate, and this is working > fine. > > However on an unrelated flow the PutHDFS processor stopped to work as the > processor is now using the JAAS configuration and ignoring the Kerberos > configuration of the processor. (the setup are completely unrelated) > > We tried a lot of configuration but we have not yet find a way to allow Nifi > to authenticate to both Kafka 10.2 and HDFS at the same time, it's either > one or the other. > > Are we missing something? Is there a way to ensure that both processor keep > their own configuration? > > Thanks! > > > For reference PutHDFS failure log: > > 2017-05-16 15:27:01,199 ERROR [StandardProcessScheduler Thread-8] > o.apache.nifi.processors.hadoop.PutHDFS > PutHDFS[id=b6464c58-fee9-341c-8428-932218f7d239] > PutHDFS[id=b6464c58-fee9-341c-8428-932218f7d239] failed to invoke > @OnScheduled method due to java.lang.RuntimeException: Failed while > executing one of processor's OnScheduled task.; processor will not be > scheduled to run for 30 seconds: java.lang.RuntimeException: Failed while > executing one of processor's OnScheduled task. > > java.lang.RuntimeException: Failed while executing one of processor's > OnScheduled task. > > at > org.apache.nifi.controller.StandardProcessorNode.invokeTaskAsCancelableFuture(StandardProcessorNode.java:1480) > > at > org.apache.nifi.controller.StandardProcessorNode.access$000(StandardProcessorNode.java:100) > > at > org.apache.nifi.controller.StandardProcessorNode$1.run(StandardProcessorNode.java:1301) > > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > > at java.lang.Thread.run(Thread.java:745) > > Caused by: java.util.concurrent.ExecutionException: > java.lang.reflect.InvocationTargetException > > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > > at java.util.concurrent.FutureTask.get(FutureTask.java:206) > > at > org.apache.nifi.controller.StandardProcessorNode.invokeTaskAsCancelableFuture(StandardProcessorNode.java:1463) > > ... 9 common frames omitted > > Caused by: java.lang.reflect.InvocationTargetException: null > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:498) > > at > org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:137) > > at > org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:125) > > at > org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:70) > > at > org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:47) > > at > org.apache.nifi.controller.StandardProcessorNode$1$1.call(StandardProcessorNode.java:1305) > > at > org.apache.nifi.controller.StandardProcessorNode$1$1.call(StandardProcessorNode.java:1301) > > ... 6 common frames omitted > > Caused by: java.lang.NullPointerException: null > > at > org.apache.kafka.common.security.plain.PlainSaslServer$PlainSaslServerFactory.getMechanismNames(PlainSaslServer.java:162) > > at > org.apache.hadoop.security.SaslRpcServer$FastSaslServerFactory.<init>(SaslRpcServer.java:380) > > at > org.apache.hadoop.security.SaslRpcServer.init(SaslRpcServer.java:184) > > at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:577) > > at > org.apache.hadoop.hdfs.NameNodeProxies.createNNProxyWithClientProtocol(NameNodeProxies.java:418) > > at > org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:314) > > at > org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.getProxy(ConfiguredFailoverProxyProvider.java:124) > > at > org.apache.hadoop.io.retry.RetryInvocationHandler.<init>(RetryInvocationHandler.java:73) > > at > org.apache.hadoop.io.retry.RetryInvocationHandler.<init>(RetryInvocationHandler.java:64) > > at org.apache.hadoop.io.retry.RetryProxy.create(RetryProxy.java:59) > > at > org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:181) > > at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:678) > > at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619) > > at > org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:149) > > at > org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669) > > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370) > > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:172) > > at > org.apache.nifi.processors.hadoop.AbstractHadoopProcessor$1.run(AbstractHadoopProcessor.java:304) > > at > org.apache.nifi.processors.hadoop.AbstractHadoopProcessor$1.run(AbstractHadoopProcessor.java:301) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAs(Subject.java:422) > > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) > > at > org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.getFileSystemAsUser(AbstractHadoopProcessor.java:301) > > at > org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.resetHDFSResources(AbstractHadoopProcessor.java:268) > > at > org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.abstractOnScheduled(AbstractHadoopProcessor.java:200) > > at > org.apache.nifi.processors.hadoop.PutHDFS.onScheduled(PutHDFS.java:191) > > ... 16 common frames omitted > >
