Hi
I have a 3 node test cluster running v. 1.16.1. Which has been working
fine, with no errors. But i doesn't do much, since it is my test cluster.
But now I am struggling with load balance connection refuse between nodes.
Both node 2 and 3 are refusing load balancing connections, even after
stopping the cluster. I have deleted all files at node 2 and 3, but they
still refuse connection.
I have not made any change to the configuration, but I have created a new
flow which got node 1 to die.
The new flow is using an ExecuteStreamCommand using stdin/stdout to a shell
command to manipulate with some data (1 GB files). I set the "Concurrent
Tasks" to high so node 1 ran out of memory and stopped. It continues to
stop if I tried to start it again. So I deleted the flow.gz file and the
run, state and work folders and started the node. Now node 1 was running
again. But after this node 2 and 3 are refusing load balancing connections.
I can't see what this flow have to do with this, but I have now tried to
remove all files at node 2 and 3 to get clean nodes. But they still doesn't
work
Does any one have an idea how to debug further?
>From the log:
2022-05-19 13:22:46,268 ERROR [Load-Balanced Client Thread-2]
o.a.n.c.q.c.c.a.n.NioAsyncLoadBalanceClient Unable to connect to
nifi-n03:8443 for load balancing
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:482)
at sun.nio.ch.Net.connect(Net.java:474)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:647)
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:107)
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:92)
at
org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.createChannel(NioAsyncLoadBalanceClient.java:497)
at
org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.establishConnection(NioAsyncLoadBalanceClient.java:440)
at
org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.communicate(NioAsyncLoadBalanceClient.java:234)
at
org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClientTask.run(NioAsyncLoadBalanceClientTask.java:81)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
kind regards
Jens
