Hello,
I would like to report a potential bug that affects NiFi 1.16.1. Our current
setup consists of a cluster of 3 nodes and the aim is to build a workflow that
can write to both HDFS and Apache Ozone. To this end, we have configured two
distinct PutHDFS processors. The configuration of these two processors is such
that one points to HDFS and the other one to Ozone using two different
core-sites configuration files. The PutHDFS processor that points to Ozone uses
OFS as protocol.
When we try running the workflow, however, we run into problems. Essentially,
this seems to be caused by conflicting libraries that are instantiated by the
two processors, thus generating an error.
We have found a way to circumvent this problem but this required us to build a
custom PutOzone processor that better encapsulates the way the libraries are
instantiated.
We are not sure whether this is a known issue. Please find attached the
relevant log lines.
Regards,
Giorgio
Sent with [Proton Mail](https://proton.me/) secure email.
2022-05-20 08:44:55,992 ERROR [Timer-Driven Process Thread-1]
o.apache.nifi.processors.hadoop.ListHDFS
ListHDFS[id=b7224f6f-0180-1000-8861-c322c2b577e8] Failed to properly initialize
Processor. If still scheduled to run, NiFi will attempt to initialize and run
the Processor again after the 'Administrative Yield Duration' has elapsed.
Failure is due to java.lang.ExceptionInInitializerError
java.lang.ExceptionInInitializerError: null
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at
org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2602)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2567)
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2663)
at org.apache.hadoop.ipc.RPC.getProtocolEngine(RPC.java:213)
at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:624)
at
org.apache.hadoop.hdfs.NameNodeProxiesClient.createProxyWithAlignmentContext(NameNodeProxiesClient.java:370)
at
org.apache.hadoop.hdfs.NameNodeProxiesClient.createNonHAProxyWithClientProtocol(NameNodeProxiesClient.java:348)
at
org.apache.hadoop.hdfs.server.namenode.ha.ClientHAProxyFactory.createProxy(ClientHAProxyFactory.java:46)
at
org.apache.hadoop.hdfs.server.namenode.ha.AbstractNNFailoverProxyProvider.createProxyIfNeeded(AbstractNNFailoverProxyProvider.java:155)
at
org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.getProxy(ConfiguredFailoverProxyProvider.java:60)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$ProxyDescriptor.<init>(RetryInvocationHandler.java:197)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.<init>(RetryInvocationHandler.java:328)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.<init>(RetryInvocationHandler.java:322)
at org.apache.hadoop.io.retry.RetryProxy.create(RetryProxy.java:59)
at
org.apache.hadoop.hdfs.NameNodeProxiesClient.createHAProxy(NameNodeProxiesClient.java:326)
at
org.apache.hadoop.hdfs.NameNodeProxiesClient.createProxyWithClientProtocol(NameNodeProxiesClient.java:144)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:374)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:308)
at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:184)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3414)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:521)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:272)
at
org.apache.nifi.processors.hadoop.AbstractHadoopProcessor$1.run(AbstractHadoopProcessor.java:546)
at
org.apache.nifi.processors.hadoop.AbstractHadoopProcessor$1.run(AbstractHadoopProcessor.java:543)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845)
at
org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.getFileSystemAsUser(AbstractHadoopProcessor.java:543)
at
org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.resetHDFSResources(AbstractHadoopProcessor.java:476)
at
org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.abstractOnScheduled(AbstractHadoopProcessor.java:352)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:145)
at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:133)
at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:78)
at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:55)
at
org.apache.nifi.controller.StandardProcessorNode.lambda$initiateStart$6(StandardProcessorNode.java:1662)
at org.apache.nifi.engine.FlowEngine$3.call(FlowEngine.java:123)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: ReRegistration of rpcKind:
RPC_PROTOCOL_BUFFER
at org.apache.hadoop.ipc.Server.registerProtocolEngine(Server.java:290)
at
org.apache.hadoop.ipc.ProtobufRpcEngine2.<clinit>(ProtobufRpcEngine2.java:64)
... 48 common frames omitted
