Sergey Karpushin created IGNITE-10388: -----------------------------------------
Summary: Prevent Ignite from "dying" if first attempt to join the cluster failed Key: IGNITE-10388 URL: https://issues.apache.org/jira/browse/IGNITE-10388 Project: Ignite Issue Type: Bug Components: general Affects Versions: 2.6 Environment: * Linux OS (I'm using Manjaro 1.18) * Jdk 1.8 * Docker-compose 3.7 Reporter: Sergey Karpushin I'm creating this issue as suggested on [stackoverflow|https://stackoverflow.com/questions/53433749/how-to-prevent-ignite-from-dying-if-first-attempt-to-join-the-cluster-failed] I've also created a simple [github project|https://github.com/skarpushin/issue20181122] to help reproduce the issue (please check README file before running the project). I'm using {{docker-compose}} to start several containers with java web application with embedded Ignite 2.6. I configured Ignite so it will use SSL. If I start node1 and wait it to fully start, and then start node2, they will join cluster no problem. But if I start them simultaneously (that's how it works by default with docker-compose) they both will fail with exception and will never retry or get back. {{{{class org.apache.ignite.IgniteException: Unable to establish secure connection. Was remote cluster configured with SSL? [rmtAddr=/10.9.9.1:47501, errMsg="Remote host closed connection during handshake"] at org.apache.ignite.spi.discovery.tcp.ServerImpl.sendMessageDirectly(ServerImpl.java:1293) at org.apache.ignite.spi.discovery.tcp.ServerImpl.sendJoinRequestMessage(ServerImpl.java:1046) at org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:890) at org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:373) at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:1948) at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297) at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:915) at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1721) at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1028) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2014) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1723) at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1151) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:649) at org.apache.ignite.IgniteSpring.start(IgniteSpring.java:66) at org.apache.ignite.IgniteSpringBean.afterSingletonsInstantiated(IgniteSpringBean.java:172) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:777) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:869) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550) at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:409) at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:291) at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:103) at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4853) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5314) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:753) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:729) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:717) at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:976) at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1853) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1002) at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385) at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:757) at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123) at java.io.OutputStream.write(OutputStream.java:75) at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.writeToSocket(TcpDiscoverySpi.java:1501) at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1452) at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1413) at org.apache.ignite.spi.discovery.tcp.ServerImpl.sendMessageDirectly(ServerImpl.java:1199) ... 33 more Caused by: java.io.EOFException: SSL peer shut down incorrectly at sun.security.ssl.InputRecord.read(InputRecord.java:505) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983) ... 41 more 2018-11-22 14:47:06.727 INFO localhost-startStop-1 Log4JLogger Command protocol successfully stopped: TCP binary 2018-11-22 14:47:11.733 INFO localhost-startStop-1 Log4JLogger No verification for local node leave has been received from coordinator (will stop node anyway).}}}} -- This message was sent by Atlassian JIRA (v7.6.3#76005)