Hi all - I'm running a NiFi cluster in kubernetes - version 1.25.0, and
when it's under load, I get these errors:
2024-03-06 22:22:58,081 ERROR [Load-Balanced Client Thread-4]
o.a.n.c.q.c.c.a.n.NioAsyncLoadBalanceClient Unable to connect to
nifi-0.nifi-headless.nifi-jos.svc.cluster.local:8443 for load balancing
java.net.UnknownHostException: null
at java.base/sun.nio.ch.Net.translateException(Unknown Source)
at java.base/sun.nio.ch.SocketAdaptor.connect(Unknown Source)
at java.base/sun.nio.ch.SocketAdaptor.connect(Unknown Source)
at
org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.createChannel(NioAsyncLoadBalanceClient.java:497)
at
org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.establishConnection(NioAsyncLoadBalanceClient.java:440)
at
org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.communicate(NioAsyncLoadBalanceClient.java:234)
at
org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClientTask.run(NioAsyncLoadBalanceClientTask.java:81)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown
Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
Source)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
The pod then exits, and Nifi restarts, and resumes, but this can happen
many times per day.
Any ideas?
I also see:
2024-03-06 22:22:42,022 ERROR [Load-Balanced Client Thread-1]
o.a.n.c.q.c.c.a.n.NioAsyncLoadBalanceClient Unable to connect to
nifi-0.nifi-headless.nifi-jos.svc.cluster.local:8443 for load balancing
java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.Net.connect0(Native Method)
at java.base/sun.nio.ch.Net.connect(Unknown Source)
at java.base/sun.nio.ch.Net.connect(Unknown Source)
at java.base/sun.nio.ch.SocketChannelImpl.connect(Unknown Source)
at java.base/sun.nio.ch.SocketAdaptor.connect(Unknown Source)
at java.base/sun.nio.ch.SocketAdaptor.connect(Unknown Source)
at
org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.createChannel(NioAsyncLoadBalanceClient.java:497)
at
org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.establishConnection(NioAsyncLoadBalanceClient.java:440)
at
org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.communicate(NioAsyncLoadBalanceClient.java:234)
at
org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClientTask.run(NioAsyncLoadBalanceClientTask.java:81)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown
Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
Source)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
It various from node to node - sometimes it's nifi0, sometimes nifi1,
and sometimes nifi2. Having a hard time tracking down the cause. When
the system is lightly loaded, it's no problem, but when queue size gets
large, I start getting these restarts.
-Joe
--
This email has been checked for viruses by AVG antivirus software.
www.avg.com
