[ https://issues.apache.org/jira/browse/IGNITE-9206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pavel Kovalenko updated IGNITE-9206: ------------------------------------ Ignite Flags: (was: Docs Required) > Node can't join to ring if all existing nodes have stopped and another new > node joined ahead > -------------------------------------------------------------------------------------------- > > Key: IGNITE-9206 > URL: https://issues.apache.org/jira/browse/IGNITE-9206 > Project: Ignite > Issue Type: Bug > Components: cache > Affects Versions: 2.5 > Reporter: Pavel Kovalenko > Priority: Major > Fix For: 2.7 > > > TcpDiscovery SPI problem. > Situation: > Existing cluster with nodes 1 and 2. Nodes 1 and 2 are stopping. > 1) Node 3 joins to cluster and sends JoinMessage to node 2. > 2) Node 2 is stopping and unable to handle JoinMessage from node 3. Node 3 > choose node 1 as the next node to send the message. > 3) Node 3 sends JoinMessage to node 1. > 4) Node 4 joins to cluster. > 5) Node 1 is stopping and unable to handle JoinMessage from node 3. > 6) Node 4 sees that there are no alive nodes in the ring at the time and > become the first node in the topology. > 7) Node 3 sends JoinMessage to Node 4 and this process repeats again and > again without any success. > At Node 4 logs we can see that remote connection from Node 3 is established > but no active actions have performed. Node 3 leaves in CONNECTING state > forever. At the same time Node 4 thinks that Node 3 is already in the ring. > Failed test: > GridCacheReplicatedDataStructuresFailoverSelfTest#testAtomicSequenceConstantTopologyChange > Link to TC: > https://ci.ignite.apache.org/viewLog.html?buildId=1594376&tab=buildResultsDiv&buildTypeId=IgniteTests24Java8_DataStructures > Shrinked log: > {code:java} > [00:09:13] : [Step 3/4] [2018-08-04 21:09:13,733][INFO ][main][root] >>> > Stopping grid > [name=replicated.GridCacheReplicatedDataStructuresFailoverSelfTest0, > id=3e2c94bd-8e98-4dd9-8d1a-befbfe000000] > [00:09:13] : [Step 3/4] [2018-08-04 21:09:13,739][INFO > ][thread-replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7][root] > Start node: replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7 > [00:09:13] : [Step 3/4] [2018-08-04 21:09:13,740][INFO > ][tcp-disco-msg-worker-#2146%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest6%][TcpDiscoverySpi] > New next node [newNext=TcpDiscoveryNode > [id=3e2c94bd-8e98-4dd9-8d1a-befbfe000000, addrs=ArrayList [127.0.0.1], > sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, > lastExchangeTime=1533416953738, loc=false, ver=2.7.0#20180803-sha1:3ab8bbad, > isClient=false]] > [00:09:13] : [Step 3/4] [2018-08-04 21:09:13,741][INFO > ][tcp-disco-srvr-#2100%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest0%][TcpDiscoverySpi] > TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, > rmtPort=50099] > [00:09:13] : [Step 3/4] [2018-08-04 21:09:13,741][INFO > ][tcp-disco-srvr-#2100%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest0%][TcpDiscoverySpi] > TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, > rmtPort=50099] > [00:09:13] : [Step 3/4] [2018-08-04 21:09:13,743][INFO > ][tcp-disco-sock-reader-#2151%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest0%][TcpDiscoverySpi] > Started serving remote node connection [rmtAddr=/127.0.0.1:50099, > rmtPort=50099] > [00:09:13] : [Step 3/4] [2018-08-04 21:09:13,746][INFO > ][thread-replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7][GridCacheReplicatedDataStructuresFailoverSelfTest7] > > [00:09:13] : [Step 3/4] > [00:09:13] : [Step 3/4] >>> __________ ________________ > [00:09:13] : [Step 3/4] >>> / _/ ___/ |/ / _/_ __/ __/ > [00:09:13] : [Step 3/4] >>> _/ // (7 7 // / / / / _/ > [00:09:13] : [Step 3/4] >>> /___/\___/_/|_/___/ /_/ /___/ > [00:09:13] : [Step 3/4] >>> > [00:09:13] : [Step 3/4] >>> ver. 2.7.0-SNAPSHOT#20180803-sha1:3ab8bbad > [00:09:13] : [Step 3/4] >>> 2018 Copyright(C) Apache Software Foundation > [00:09:13] : [Step 3/4] >>> > [00:09:13] : [Step 3/4] >>> Ignite documentation: http://ignite.apache.org > [00:09:13] : [Step 3/4] > [00:09:13] : [Step 3/4] [2018-08-04 21:09:13,746][INFO > ][thread-replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7][GridCacheReplicatedDataStructuresFailoverSelfTest7] > Config URL: n/a > [00:09:13] : [Step 3/4] [2018-08-04 21:09:13,747][INFO > ][thread-replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7][GridCacheReplicatedDataStructuresFailoverSelfTest7] > IgniteConfiguration > [igniteInstanceName=replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7, > pubPoolSize=8, svcPoolSize=8, callbackPoolSize=8, stripedPoolSize=8, > sysPoolSize=8, mgmtPoolSize=4, igfsPoolSize=5, dataStreamerPoolSize=8, > utilityCachePoolSize=8, utilityCacheKeepAliveTime=60000, p2pPoolSize=2, > qryPoolSize=8, igniteHome=/data/teamcity/work/9198da4c51c3e112, > igniteWorkDir=/data/teamcity/work/9198da4c51c3e112/work, > mbeanSrv=com.sun.jmx.mbeanserver.JmxMBeanServer@13fed1ec, > nodeId=fe9e7ca7-c0fa-4b51-8a87-1255f8c00007, marsh=BinaryMarshaller [], > marshLocJobs=false, daemon=false, p2pEnabled=true, netTimeout=10000, > sndRetryDelay=1000, sndRetryCnt=3, metricsHistSize=10000, > metricsUpdateFreq=1000, metricsExpTime=9223372036854775807, > discoSpi=TcpDiscoverySpi [addrRslvr=null, sockTimeout=0, ackTimeout=0, > marsh=null, reconCnt=10, reconDelay=2000, maxAckTimeout=600000, > forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null], > segPlc=STOP, segResolveAttempts=2, waitForSegOnStart=true, > allResolversPassReq=true, segChkFreq=10000, commSpi=TcpCommunicationSpi > [connectGate=null, > connPlc=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$FirstConnectionPolicy@514987e3, > enableForcibleNodeKill=false, enableTroubleshootingLog=false, locAddr=null, > locHost=null, locPort=45170, locPortRange=100, shmemPort=-1, directBuf=true, > directSndBuf=false, idleConnTimeout=600000, connTimeout=5000, > maxConnTimeout=600000, reconCnt=10, sockSndBuf=32768, sockRcvBuf=32768, > msgQueueLimit=0, slowClientQueueLimit=0, nioSrvr=null, shmemSrv=null, > usePairedConnections=false, connectionsPerNode=1, tcpNoDelay=true, > filterReachableAddresses=false, ackSndThreshold=32, unackedMsgsBufSize=0, > sockWriteTimeout=2000, boundTcpPort=-1, boundTcpShmemPort=-1, selectorsCnt=4, > selectorSpins=0, addrRslvr=null, > ctxInitLatch=java.util.concurrent.CountDownLatch@65788f9a[Count = 1], > stopping=false], evtSpi=MemoryEventStorageSpi > [expireAgeMs=9223372036854775807, expireCnt=10000, filter=null], > colSpi=NoopCollisionSpi [], deploySpi=LocalDeploymentSpi [], > indexingSpi=org.apache.ignite.spi.indexing.noop.NoopIndexingSpi@4fb8a27b, > addrRslvr=null, clientMode=false, rebalanceThreadPoolSize=1, > txCfg=TransactionConfiguration [txSerEnabled=false, > dfltIsolation=REPEATABLE_READ, dfltConcurrency=PESSIMISTIC, dfltTxTimeout=0, > txTimeoutOnPartitionMapExchange=0, pessimisticTxLogSize=0, > pessimisticTxLogLinger=10000, tmLookupClsName=null, txManagerFactory=null, > useJtaSync=false], cacheSanityCheckEnabled=true, discoStartupDelay=60000, > deployMode=SHARED, p2pMissedCacheSize=100, locHost=127.0.0.1, > timeSrvPortBase=31100, timeSrvPortRange=100, failureDetectionTimeout=10000, > clientFailureDetectionTimeout=10000, metricsLogFreq=0, hadoopCfg=null, > connectorCfg=null, odbcCfg=null, warmupClos=null, > atomicCfg=AtomicConfiguration [seqReserveSize=1000, cacheMode=REPLICATED, > backups=0, aff=null, grpName=null], classLdr=null, sslCtxFactory=null, > platformCfg=null, binaryCfg=null, memCfg=null, pstCfg=null, > dsCfg=DataStorageConfiguration [sysRegionInitSize=41943040, > sysRegionMaxSize=104857600, pageSize=0, concLvl=0, > dfltDataRegConf=DataRegionConfiguration [name=default, maxSize=20268895436, > initSize=268435456, swapPath=null, pageEvictionMode=DISABLED, > evictionThreshold=0.9, emptyPagesPoolSize=100, metricsEnabled=false, > metricsSubIntervalCount=5, metricsRateTimeInterval=60000, > persistenceEnabled=false, checkpointPageBufSize=0], dataRegions=null, > storagePath=null, checkpointFreq=180000, lockWaitTime=10000, > checkpointThreads=4, checkpointWriteOrder=SEQUENTIAL, walHistSize=20, > walSegments=10, walSegmentSize=67108864, walPath=db/wal, > walArchivePath=db/wal/archive, metricsEnabled=false, walMode=LOG_ONLY, > walTlbSize=131072, walBuffSize=0, walFlushFreq=2000, walFsyncDelay=1000, > walRecordIterBuffSize=67108864, alwaysWriteFullPages=false, > fileIOFactory=org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory@a63cd72, > metricsSubIntervalCnt=5, metricsRateTimeInterval=60000, > walAutoArchiveAfterInactivity=-1, writeThrottlingEnabled=false, > walCompactionEnabled=false], activeOnStart=true, autoActivation=true, > longQryWarnTimeout=3000, sqlConnCfg=null, > cliConnCfg=ClientConnectorConfiguration [host=null, port=10800, > portRange=100, sockSndBufSize=0, sockRcvBufSize=0, tcpNoDelay=true, > maxOpenCursorsPerConn=128, threadPoolSize=8, idleTimeout=0, jdbcEnabled=true, > odbcEnabled=true, thinCliEnabled=true, sslEnabled=false, > useIgniteSslCtxFactory=true, sslClientAuth=false, sslCtxFactory=null], > authEnabled=false, failureHnd=NoOpFailureHandler [], commFailureRslvr=null] > ... > [00:09:13] : [Step 3/4] [2018-08-04 21:09:13,750][INFO > ][tcp-disco-sock-reader-#2151%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest0%][TcpDiscoverySpi] > Finished serving remote node connection [rmtAddr=/127.0.0.1:50099, > rmtPort=50099 > [00:09:13] : [Step 3/4] [2018-08-04 21:09:13,750][INFO > ][tcp-disco-sock-reader-#2145%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest0%][TcpDiscoverySpi] > Finished serving remote node connection [rmtAddr=/127.0.0.1:39170, > rmtPort=39170 > [00:09:13]W: [org.apache.ignite:ignite-core] [2018-08-04 > 21:09:13,842][WARN > ][tcp-disco-msg-worker-#2146%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest6%][TcpDiscoverySpi] > Local node has detected failed nodes and started cluster-wide procedure. To > speed up failure detection please see 'Failure Detection' section under > javadoc for 'TcpDiscoverySpi' > [00:09:13] : [Step 3/4] [2018-08-04 21:09:13,749][INFO > ][tcp-disco-sock-reader-#2139%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest0%][TcpDiscoverySpi] > Finished serving remote node connection [rmtAddr=/127.0.0.1:33299, > rmtPort=33299 > ... > [00:09:13] : [Step 3/4] [2018-08-04 21:09:13,875][INFO > ][main][GridCacheReplicatedDataStructuresFailoverSelfTest0] > [00:09:13] : [Step 3/4] > [00:09:13] : [Step 3/4] >>> > +------------------------------------------------------------------------------------------+ > [00:09:13] : [Step 3/4] >>> Ignite ver. > 2.7.0-SNAPSHOT#20180803-sha1:3ab8bbadc80446e53261da784bfbed48b6ab5322 stopped > OK > [00:09:13] : [Step 3/4] >>> > +------------------------------------------------------------------------------------------+ > [00:09:13] : [Step 3/4] >>> Ignite instance name: > replicated.GridCacheReplicatedDataStructuresFailoverSelfTest0 > [00:09:13] : [Step 3/4] >>> Grid uptime: 00:00:02.811 > [00:09:13] : [Step 3/4] > [00:09:13] : [Step 3/4] > [00:09:13] : [Step 3/4] [2018-08-04 21:09:13,876][INFO > ][thread-replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7][TcpCommunicationSpi] > Successfully bound communication NIO server to TCP port [port=45170, > locHost=/127.0.0.1, selectorsCnt=4, selectorSpins=0, pairedConn=false] > [00:09:13]W: [org.apache.ignite:ignite-core] [2018-08-04 > 21:09:13,876][WARN > ][thread-replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7][TcpCommunicationSpi] > Message queue limit is set to 0 which may lead to potential OOMEs when > running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to message > queues growth on sender and receiver sides. > [00:09:13]W: [org.apache.ignite:ignite-core] [2018-08-04 > 21:09:13,903][WARN > ][thread-replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7][GridCollisionManager] > Collision resolution is disabled (all jobs will be activated upon arrival). > [00:09:13] : [Step 3/4] [2018-08-04 21:09:13,903][INFO > ][thread-replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7][GridCacheReplicatedDataStructuresFailoverSelfTest7] > Security status [authentication=off, tls/ssl=off] > [00:09:13] : [Step 3/4] [2018-08-04 21:09:13,912][INFO > ][thread-replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7][ClientListenerProcessor] > Client connector processor has started on TCP port 10800 > [00:09:13] : [Step 3/4] [2018-08-04 21:09:13,916][INFO > ][thread-replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7][GridCacheReplicatedDataStructuresFailoverSelfTest7] > Non-loopback local IPs: 172.17.0.6 > [00:09:13] : [Step 3/4] [2018-08-04 21:09:13,916][INFO > ][thread-replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7][GridCacheReplicatedDataStructuresFailoverSelfTest7] > Enabled local MACs: 0242AC110006 > [00:09:13] : [Step 3/4] [2018-08-04 21:09:13,917][INFO > ][thread-replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7][TcpDiscoverySpi] > Connection check threshold is calculated: 10000 > [00:09:13] : [Step 3/4] [2018-08-04 21:09:13,918][INFO > ][thread-replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7][TcpDiscoverySpi] > Successfully bound to TCP port [port=47500, localHost=/127.0.0.1, > locNodeId=fe9e7ca7-c0fa-4b51-8a87-1255f8c00007] > [00:09:13] : [Step 3/4] [2018-08-04 21:09:13,926][INFO > ][tcp-disco-srvr-#2147%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest6%][TcpDiscoverySpi] > TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, > rmtPort=35863] > [00:09:13] : [Step 3/4] [2018-08-04 21:09:13,926][INFO > ][tcp-disco-srvr-#2147%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest6%][TcpDiscoverySpi] > TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, > rmtPort=35863] > [00:09:13] : [Step 3/4] [2018-08-04 21:09:13,926][INFO > ][tcp-disco-sock-reader-#2156%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest6%][TcpDiscoverySpi] > Started serving remote node connection [rmtAddr=/127.0.0.1:35863, > rmtPort=35863] > [00:09:13] : [Step 3/4] [2018-08-04 21:09:13,933][INFO > ][tcp-disco-sock-reader-#2156%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest6%][TcpDiscoverySpi] > Finished serving remote node connection [rmtAddr=/127.0.0.1:35863, > rmtPort=35863 > [00:09:15] : [Step 3/4] [2018-08-04 21:09:15,938][INFO > ][tcp-disco-srvr-#2147%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest6%][TcpDiscoverySpi] > TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, > rmtPort=38642] > [00:09:15] : [Step 3/4] [2018-08-04 21:09:15,938][INFO > ][tcp-disco-srvr-#2147%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest6%][TcpDiscoverySpi] > TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, > rmtPort=38642] > [00:09:15] : [Step 3/4] [2018-08-04 21:09:15,939][INFO > ][tcp-disco-sock-reader-#2157%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest6%][TcpDiscoverySpi] > Started serving remote node connection [rmtAddr=/127.0.0.1:38642, > rmtPort=38642] > [00:09:15] : [Step 3/4] [2018-08-04 21:09:15,942][INFO > ][tcp-disco-sock-reader-#2157%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest6%][TcpDiscoverySpi] > Finished serving remote node connection [rmtAddr=/127.0.0.1:38642, > rmtPort=38642 > [00:09:17] : [Step 3/4] [2018-08-04 21:09:17,945][INFO > ][tcp-disco-srvr-#2147%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest6%][TcpDiscoverySpi] > TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, > rmtPort=41001] > [00:09:17] : [Step 3/4] [2018-08-04 21:09:17,945][INFO > ][tcp-disco-srvr-#2147%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest6%][TcpDiscoverySpi] > TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, > rmtPort=41001] > [00:09:20] : [Step 3/4] [2018-08-04 21:09:20,744][INFO > ][tcp-disco-msg-worker-#2154%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7%][TcpDiscoverySpi] > New next node [newNext=TcpDiscoveryNode > [id=79ab4ba3-98df-488b-95b6-68e0e6a00006, addrs=ArrayList [127.0.0.1], > sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=0, intOrder=2, > lastExchangeTime=1533416960734, loc=false, ver=2.7.0#20180803-sha1:3ab8bbad, > isClient=false]] > [00:09:20] : [Step 3/4] [2018-08-04 21:09:20,745][INFO > ][tcp-disco-srvr-#2147%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest6%][TcpDiscoverySpi] > TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, > rmtPort=60760] > [00:09:20] : [Step 3/4] [2018-08-04 21:09:20,745][INFO > ][tcp-disco-srvr-#2147%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest6%][TcpDiscoverySpi] > TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, > rmtPort=60760] > [00:09:20] : [Step 3/4] [2018-08-04 21:09:20,745][INFO > ][tcp-disco-sock-reader-#2163%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest6%][TcpDiscoverySpi] > Started serving remote node connection [rmtAddr=/127.0.0.1:60760, > rmtPort=60760] > [00:09:25] : [Step 3/4] [2018-08-04 21:09:25,745][INFO > ][tcp-disco-srvr-#2155%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7%][TcpDiscoverySpi] > TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, > rmtPort=58381] > [00:09:25] : [Step 3/4] [2018-08-04 21:09:25,746][INFO > ][tcp-disco-srvr-#2155%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7%][TcpDiscoverySpi] > TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, > rmtPort=58381] > ... > [00:14:16] : [Step 3/4] [2018-08-04 21:14:16,143][INFO > ][tcp-disco-sock-reader-#2222%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7%][TcpDiscoverySpi] > Started serving remote node connection [rmtAddr=/127.0.0.1:39278, > rmtPort=39278] > [00:14:16] : [Step 3/4] [2018-08-04 21:14:16,145][INFO > ][tcp-disco-sock-reader-#2222%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7%][TcpDiscoverySpi] > Finished serving remote node connection [rmtAddr=/127.0.0.1:39278, > rmtPort=39278 > [00:14:21]W: [org.apache.ignite:ignite-core] [2018-08-04 > 21:14:21,145][WARN > ][thread-replicated.GridCacheReplicatedDataStructuresFailoverSelfTest6][TcpDiscoverySpi] > Node has not been connected to topology and will repeat join process. Check > remote nodes logs for possible error messages. Note that large topology may > require significant time to start. Increase 'TcpDiscoverySpi.networkTimeout' > configuration property if getting this message on the starting nodes > [networkTimeout=5000] > [00:14:21] : [Step 3/4] [2018-08-04 21:14:21,146][INFO > ][tcp-disco-srvr-#2155%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7%][TcpDiscoverySpi] > TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, > rmtPort=48489] > [00:14:21] : [Step 3/4] [2018-08-04 21:14:21,147][INFO > ][tcp-disco-srvr-#2155%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7%][TcpDiscoverySpi] > TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, > rmtPort=48489] > ... > [00:19:16] : [Step 3/4] [2018-08-04 21:19:16,639][INFO > ][tcp-disco-sock-reader-#2282%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7%][TcpDiscoverySpi] > Started serving remote node connection [rmtAddr=/127.0.0.1:39305, > rmtPort=39305] > [00:19:16] : [Step 3/4] [2018-08-04 21:19:16,643][INFO > ][tcp-disco-sock-reader-#2282%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7%][TcpDiscoverySpi] > Finished serving remote node connection [rmtAddr=/127.0.0.1:39305, > rmtPort=39305 > [00:19:21]W: [org.apache.ignite:ignite-core] [2018-08-04 > 21:19:21,645][WARN > ][thread-replicated.GridCacheReplicatedDataStructuresFailoverSelfTest6][TcpDiscoverySpi] > Node has not been connected to topology and will repeat join process. Check > remote nodes logs for possible error messages. Note that large topology may > require significant time to start. Increase 'TcpDiscoverySpi.networkTimeout' > configuration property if getting this message on the starting nodes > [networkTimeout=5000] > [00:19:21] : [Step 3/4] [2018-08-04 21:19:21,646][INFO > ][tcp-disco-srvr-#2155%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7%][TcpDiscoverySpi] > TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, > rmtPort=41240] > [00:19:21] : [Step 3/4] [2018-08-04 21:19:21,647][INFO > ][tcp-disco-srvr-#2155%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7%][TcpDiscoverySpi] > TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, > rmtPort=41240] > ... > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)