Jens, So that would tell us that the hostname or the port is wrong, or that NiFi is not running/listening for load balanced connections. I would recommend the following:
- Check nifi.properties on all nodes to make sure that the nifi.cluster.load.balance.host property is set to the node’s hostname. - You could try deleting the state/ directory on the nodes and restarting. Perhaps somehow it’s got an old value cached and is trying to communicate with the wrong port? Thanks -Mark > On May 19, 2022, at 7:42 AM, Jens M. Kofoed <[email protected]> wrote: > > Hi > > I have a 3 node test cluster running v. 1.16.1. Which has been working fine, > with no errors. But i doesn't do much, since it is my test cluster. > But now I am struggling with load balance connection refuse between nodes. > Both node 2 and 3 are refusing load balancing connections, even after > stopping the cluster. I have deleted all files at node 2 and 3, but they > still refuse connection. > I have not made any change to the configuration, but I have created a new > flow which got node 1 to die. > The new flow is using an ExecuteStreamCommand using stdin/stdout to a shell > command to manipulate with some data (1 GB files). I set the "Concurrent > Tasks" to high so node 1 ran out of memory and stopped. It continues to stop > if I tried to start it again. So I deleted the flow.gz file and the run, > state and work folders and started the node. Now node 1 was running again. > But after this node 2 and 3 are refusing load balancing connections. > I can't see what this flow have to do with this, but I have now tried to > remove all files at node 2 and 3 to get clean nodes. But they still doesn't > work > > Does any one have an idea how to debug further? > > From the log: > 2022-05-19 13:22:46,268 ERROR [Load-Balanced Client Thread-2] > o.a.n.c.q.c.c.a.n.NioAsyncLoadBalanceClient Unable to connect to > nifi-n03:8443 for load balancing > java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) > at sun.nio.ch.Net.connect(Net.java:482) > at sun.nio.ch.Net.connect(Net.java:474) > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:647) > at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:107) > at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:92) > at > org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.createChannel(NioAsyncLoadBalanceClient.java:497) > at > org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.establishConnection(NioAsyncLoadBalanceClient.java:440) > at > org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.communicate(NioAsyncLoadBalanceClient.java:234) > at > org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClientTask.run(NioAsyncLoadBalanceClientTask.java:81) > at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > kind regards > Jens
