Hi, I got the below errors on my client node. CLIENT=TRUE I can see that it cannot communicate but why is it trying to communicate to 127.0.0.1? This is running inside a docker container and the cache nodes are running on dedicated VMs.
{"appTimestamp":"2020-06-08T19:26:46.209+00:00","threadName":"sys-stripe-2-#3%foo%","level":"WARN","loggerName":"org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi","message":"Connect timed out (consider increasing 'failureDetectionTimeout' configuration property) [addr=/xxx.xxx.xxx.7:47100, failureDetectionTimeout=10000]"} {"appTimestamp":"2020-06-08T19:26:46.218+00:00","threadName":"sys-stripe-2-#3%foo%","level":"ERROR","loggerName":"org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi","message":"Failed to send message to remote node [node=TcpDiscoveryNode [id=379b18e9-a03b-4956-9184-8c514924af2a, addrs=[127.0.0.1, xxx.xxx.xxx.7], sockAddrs=[/127.0.0.1:0, /xxx.xxx.xxx.7:0], discPort=0, order=1184, intOrder=597, lastExchangeTime=1591643806376, loc=false, ver=2.7.0#20181130-sha1:256ae401, isClient=true], msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridDhtAtomicNearResponse [partId=522, futId=2, primaryId=6113a59a-b087-4e70-8541-d050eea8c934, errs=null, flags=]]]","stackTrace":"org.apache.ignite.IgniteCheckedException: Failed to connect to node (is node still alive?). Make sure that each ComputeTask and cache Transaction has a timeout set in order to prevent parties from waiting forever in case of network issues [nodeId=379b18e9-a03b-4956-9184-8c514924af2a, addrs=[/xxx.xxx.xxx.7:47100, /127.0.0.1:47100]]\n\tat org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3459)\n\tat org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2987)\n\tat org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2870)\n\tat org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2713)\n\tat org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2672)\n\tat org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1656)\n\tat org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:1731)\n\tat org.apache.ignite.internal.processors.cache.GridCacheIoManager.send(GridCacheIoManager.java:1170)\n\tat org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.sendDhtNearResponse(GridDhtAtomicCache.java:3492)\n\tat org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processDhtAtomicUpdateRequest(GridDhtAtomicCache.java:3364)\n\tat org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:135)\n\tat org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:309)\n\tat org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:304)\n\tat org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)\n\tat org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)\n\tat org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)\n\tat org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)\n\tat org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)\n\tat org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)\n\tat org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)\n\tat org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)\n\tat org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)\n\tat org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)\n\tat org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)\n\tat org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)\n\tat java.lang.Thread.run(Thread.java:748)\n\tSuppressed: org.apache.ignite.IgniteCheckedException: Failed to connect to address [addr=/xxx.xxx.xxx.7:47100, err=Host is unreachable]\n\t\tat org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3462)\n\t\t... 25 common frames omitted\n\tCaused by: java.net.NoRouteToHostException: Host is unreachable\n\t\tat sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)\n\t\tat sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)\n\t\tat sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:111)\n\t\tat org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3299)\n\t\t... 25 common frames omitted\n"} {"appTimestamp":"2020-06-08T19:26:49.209+00:00","threadName":"sys-stripe-2-#3%foo%","level":"ERROR","loggerName":"org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi","message":"Failed to send message to remote node [node=TcpDiscoveryNode [id=379b18e9-a03b-4956-9184-8c514924af2a, addrs=[127.0.0.1, xxx.xxx.xxx.7], sockAddrs=[/127.0.0.1:0, /xxx.xxx.xxx.7:0], discPort=0, order=1184, intOrder=597, lastExchangeTime=1591643806376, loc=false, ver=2.7.0#20181130-sha1:256ae401, isClient=true], msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridDhtAtomicNearResponse [partId=522, futId=2, primaryId=6113a59a-b087-4e70-8541-d050eea8c934, errs=null, flags=]]]","stackTrace":"org.apache.ignite.IgniteCheckedException: Failed to connect to node (is node still alive?). Make sure that each ComputeTask and cache Transaction has a timeout set in order to prevent parties from waiting forever in case of network issues [nodeId=379b18e9-a03b-4956-9184-8c514924af2a, addrs=[/xxx.xxx.xxx.7:47100, /127.0.0.1:47100]]\n\tat org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3459)\n\tat org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2987)\n\tat org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2870)\n\tat org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2713)\n\tat org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2672)\n\tat org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1656)\n\tat org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:1731)\n\tat org.apache.ignite.internal.processors.cache.GridCacheIoManager.send(GridCacheIoManager.java:1170)\n\tat org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.sendDhtNearResponse(GridDhtAtomicCache.java:3492)\n\tat org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processDhtAtomicUpdateRequest(GridDhtAtomicCache.java:3364)\n\tat org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:135)\n\tat org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:309)\n\tat org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:304)\n\tat org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)\n\tat org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)\n\tat org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)\n\tat org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)\n\tat org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)\n\tat org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)\n\tat org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)\n\tat org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)\n\tat org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)\n\tat org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)\n\tat org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)\n\tat org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)\n\tat java.lang.Thread.run(Thread.java:748)\n\tSuppressed: org.apache.ignite.IgniteCheckedException: Failed to connect to address [addr=/xxx.xxx.xxx.7:47100, err=Host is unreachable]\n\t\tat org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3462)\n\t\t... 25 common frames omitted\n\tCaused by: java.net.NoRouteToHostException: Host is unreachable\n\t\tat sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)\n\t\tat sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)\n\t\tat sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:111)\n\t\tat org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3299)\n\t\t... 25 common frames omitted\n"} {"appTimestamp":"2020-06-08T19:26:52.214+00:00","threadName":"sys-stripe-2-#3%foo%","level":"ERROR","loggerName":"org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi","message":"Failed to send message to remote node [node=TcpDiscoveryNode [id=379b18e9-a03b-4956-9184-8c514924af2a, addrs=[127.0.0.1, xxx.xxx.xxx.7], sockAddrs=[/127.0.0.1:0, /xxx.xxx.xxx.7:0], discPort=0, order=1184, intOrder=597, lastExchangeTime=1591643806376, loc=false, ver=2.7.0#20181130-sha1:256ae401, isClient=true], msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridDhtAtomicNearResponse [partId=522, futId=2, primaryId=6113a59a-b087-4e70-8541-d050eea8c934, errs=null, flags=]]]","stackTrace":"org.apache.ignite.IgniteCheckedException: Failed to connect to node (is node still alive?). Make sure that each ComputeTask and cache Transaction has a timeout set in order to prevent parties from waiting forever in case of network issues [nodeId=379b18e9-a03b-4956-9184-8c514924af2a, addrs=[/xxx.xxx.xxx.7:47100, /127.0.0.1:47100]]\n\tat org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3459)\n\tat org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2987)\n\tat org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2870)\n\tat org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2713)\n\tat org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2672)\n\tat org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1656)\n\tat org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:1731)\n\tat org.apache.ignite.internal.processors.cache.GridCacheIoManager.send(GridCacheIoManager.java:1170)\n\tat org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.sendDhtNearResponse(GridDhtAtomicCache.java:3492)\n\tat org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processDhtAtomicUpdateRequest(GridDhtAtomicCache.java:3364)\n\tat org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:135)\n\tat org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:309)\n\tat org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:304)\n\tat org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)\n\tat org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)\n\tat org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)\n\tat org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)\n\tat org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)\n\tat org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)\n\tat org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)\n\tat org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)\n\tat org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)\n\tat org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)\n\tat org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)\n\tat org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)\n\tat java.lang.Thread.run(Thread.java:748)\n\tSuppressed: org.apache.ignite.IgniteCheckedException: Failed to connect to address [addr=/xxx.xxx.xxx.7:47100, err=Host is unreachable]\n\t\tat org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3462)\n\t\t... 25 common frames omitted\n\tCaused by: java.net.NoRouteToHostException: Host is unreachable\n\t\tat sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)\n\t\tat sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)\n\t\tat sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:111)\n\t\tat org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3299)\n\t\t... 25 common frames omitted\n"} {"appTimestamp":"2020-06-08T19:26:52.275+00:00","threadName":"sys-stripe-2-#3%foo%","level":"ERROR","loggerName":"org.apache.ignite.cache.msg.atomic","message":"Failed to send DHT near response [futId=16392, nearFutId=2, node=379b18e9-a03b-4956-9184-8c514924af2a, res=GridDhtAtomicNearResponse [partId=522, futId=2, primaryId=6113a59a-b087-4e70-8541-d050eea8c934, errs=null, flags=]]","stackTrace":"org.apache.ignite.IgniteCheckedException: Failed to send message (node may have left the grid or TCP connection cannot be established due to firewall issues) [node=TcpDiscoveryNode [id=379b18e9-a03b-4956-9184-8c514924af2a, addrs=[127.0.0.1, xxx.xxx.xxx.7], sockAddrs=[/127.0.0.1:0, /xxx.xxx.xxx.7:0], discPort=0, order=1184, intOrder=597, lastExchangeTime=1591643806376, loc=false, ver=2.7.0#20181130-sha1:256ae401, isClient=true], topic=TOPIC_CACHE, msg=GridDhtAtomicNearResponse [partId=522, futId=2, primaryId=6113a59a-b087-4e70-8541-d050eea8c934, errs=null, flags=], policy=2]\n\tat org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1667)\n\tat org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:1731)\n\tat org.apache.ignite.internal.processors.cache.GridCacheIoManager.send(GridCacheIoManager.java:1170)\n\tat org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.sendDhtNearResponse(GridDhtAtomicCache.java:3492)\n\tat org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processDhtAtomicUpdateRequest(GridDhtAtomicCache.java:3364)\n\tat org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:135)\n\tat org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:309)\n\tat org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:304)\n\tat org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)\n\tat org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)\n\tat org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)\n\tat org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)\n\tat org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)\n\tat org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)\n\tat org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)\n\tat org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)\n\tat org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)\n\tat org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)\n\tat org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)\n\tat org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)\n\tat java.lang.Thread.run(Thread.java:748)\nCaused by: org.apache.ignite.spi.IgniteSpiException: Failed to send message to remote node: TcpDiscoveryNode [id=379b18e9-a03b-4956-9184-8c514924af2a, addrs=[127.0.0.1, xxx.xxx.xxx.7], sockAddrs=[/127.0.0.1:0, /xxx.xxx.xxx.7:0], discPort=0, order=1184, intOrder=597, lastExchangeTime=1591643806376, loc=false, ver=2.7.0#20181130-sha1:256ae401, isClient=true]\n\tat org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2747)\n\tat org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2672)\n\tat org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1656)\n\t... 20 common frames omitted\nCaused by: org.apache.ignite.IgniteCheckedException: Failed to connect to node (is node still alive?). Make sure that each ComputeTask and cache Transaction has a timeout set in order to prevent parties from waiting forever in case of network issues [nodeId=379b18e9-a03b-4956-9184-8c514924af2a, addrs=[/xxx.xxx.xxx.7:47100, /127.0.0.1:47100]]\n\tat org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3459)\n\tat org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2987)\n\tat org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2870)\n\tat org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2713)\n\t... 22 common frames omitted\n\tSuppressed: org.apache.ignite.IgniteCheckedException: Failed to connect to address [addr=/xxx.xxx.xxx.7:47100, err=Host is unreachable]\n\t\tat org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3462)\n\t\t... 25 common frames omitted\n\tCaused by: java.net.NoRouteToHostException: Host is unreachable\n\t\tat sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)\n\t\tat sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)\n\t\tat sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:111)\n\t\tat org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3299)\n\t\t... 25 common frames omitted\n"}