Hi all, I have trouble to reconnect client node to cluster. Scenario I am running is
1. Ignite version 2.5 and Java 1.8 2. Two servers nodes running same machine with default GC set up 3. Client node runs on the same machine that servers do 4. Client node connects to cluster creates cache name=TEST_CACHE then destroy it and then successfully completes then close client by calling ignite.close(); See client node log starting from line - [08:20:22,277][INFO][Test worker][GridDiscoveryManager] Topology snapshot [ver=5, servers=2, clients=1, CPUs=2, offheap=8.0GB, heap=1.8GB] 5. Client connects to cluster again to execute next sample operation but stucks starting from [08:20:35,074][WARNING][exchange-worker-#212][diagnostic] Failed to wait for partition map exchange [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], node=bd8d192b-bea3-4660-88c1-9949488f3249]. Consider changing TransactionConfiguration.txTimeoutOnPartitionMapSynchronization to non default value to avoid this message. Dumping pending objects that might be the cause: 6. Wait time for client to finally connect to cluster can be different(from 10 secs up 10 mins) and for sure depends on GC settings. Also if connect to client JVM when it keeps complete partition exchange with jconsole and explicitly trigger System.gc this fix the problem. Please advise possible solutions to fix subsequent client connect issues problem. Please let me know if more details needed to about the issue Thanks in advance, Oleksandr Shevchenko
ignite-client-conf.xml
Description: ignite-client-conf.xml
ignite-server-conf.xml
Description: ignite-server-conf.xml
>>> __________ ________________ >>> / _/ ___/ |/ / _/_ __/ __/ >>> _/ // (7 7 // / / / / _/ >>> /___/\___/_/|_/___/ /_/ /___/ >>> >>> ver. 2.5.0#20180523-sha1:86e110c7 >>> 2018 Copyright(C) Apache Software Foundation >>> >>> Ignite documentation: http://ignite.apache.org [08:20:17,129][INFO][Test worker][IgniteKernal] Config URL: file:/C:/Users/X217195/IdeaProjects/calculation-library/ignite-impl/build/resources/integrationTest/ignite-client-conf.xml [08:20:17,140][INFO][Test worker][IgniteKernal] IgniteConfiguration [igniteInstanceName=null, pubPoolSize=8, svcPoolSize=8, callbackPoolSize=8, stripedPoolSize=8, sysPoolSize=8, mgmtPoolSize=4, igfsPoolSize=2, dataStreamerPoolSize=8, utilityCachePoolSize=8, utilityCacheKeepAliveTime=60000, p2pPoolSize=2, qryPoolSize=8, igniteHome=H:\apache-ignite-fabric-2.4.0-bin, igniteWorkDir=H:\apache-ignite-fabric-2.4.0-bin\work, mbeanSrv=com.sun.jmx.mbeanserver.JmxMBeanServer@763335f9, nodeId=22efeb8f-176c-455a-b86a-9435f0bf0736, marsh=org.apache.ignite.internal.binary.BinaryMarshaller@521ec289, marshLocJobs=false, daemon=false, p2pEnabled=true, netTimeout=5000, sndRetryDelay=1000, sndRetryCnt=3, metricsHistSize=10000, metricsUpdateFreq=2000, metricsExpTime=9223372036854775807, discoSpi=TcpDiscoverySpi [addrRslvr=null, sockTimeout=0, ackTimeout=0, marsh=null, reconCnt=10, reconDelay=2000, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null], segPlc=STOP, segResolveAttempts=2, waitForSegOnStart=true, allResolversPassReq=true, segChkFreq=10000, commSpi=TcpCommunicationSpi [connectGate=null, connPlc=null, enableForcibleNodeKill=false, enableTroubleshootingLog=false, srvLsnr=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$2@713e5835, locAddr=null, locHost=null, locPort=47100, locPortRange=100, shmemPort=-1, directBuf=true, directSndBuf=false, idleConnTimeout=600000, connTimeout=5000, maxConnTimeout=600000, reconCnt=10, sockSndBuf=32768, sockRcvBuf=32768, msgQueueLimit=0, slowClientQueueLimit=0, nioSrvr=null, shmemSrv=null, usePairedConnections=false, connectionsPerNode=1, tcpNoDelay=true, filterReachableAddresses=false, ackSndThreshold=32, unackedMsgsBufSize=0, sockWriteTimeout=2000, lsnr=null, boundTcpPort=-1, boundTcpShmemPort=-1, selectorsCnt=4, selectorSpins=0, addrRslvr=null, ctxInitLatch=java.util.concurrent.CountDownLatch@ec81a8f[Count = 1], stopping=false, metricsLsnr=org.apache.ignite.spi.communication.tcp.TcpCommunicationMetricsListener@7132fdff], evtSpi=org.apache.ignite.spi.eventstorage.NoopEventStorageSpi@559c63da, colSpi=NoopCollisionSpi [], deploySpi=LocalDeploymentSpi [lsnr=null], indexingSpi=org.apache.ignite.spi.indexing.noop.NoopIndexingSpi@566066ac, addrRslvr=null, clientMode=true, rebalanceThreadPoolSize=1, txCfg=org.apache.ignite.configuration.TransactionConfiguration@1832128d, cacheSanityCheckEnabled=true, discoStartupDelay=60000, deployMode=SHARED, p2pMissedCacheSize=100, locHost=null, timeSrvPortBase=31100, timeSrvPortRange=100, failureDetectionTimeout=10000, clientFailureDetectionTimeout=30000, metricsLogFreq=60000, hadoopCfg=null, connectorCfg=org.apache.ignite.configuration.ConnectorConfiguration@7e89c971, odbcCfg=null, warmupClos=null, atomicCfg=AtomicConfiguration [seqReserveSize=1000, cacheMode=PARTITIONED, backups=1, aff=null, grpName=null], classLdr=null, sslCtxFactory=null, platformCfg=null, binaryCfg=null, memCfg=null, pstCfg=null, dsCfg=null, activeOnStart=true, autoActivation=true, longQryWarnTimeout=3000, sqlConnCfg=null, cliConnCfg=ClientConnectorConfiguration [host=null, port=10800, portRange=100, sockSndBufSize=0, sockRcvBufSize=0, tcpNoDelay=true, maxOpenCursorsPerConn=128, threadPoolSize=8, idleTimeout=0, jdbcEnabled=true, odbcEnabled=true, thinCliEnabled=true, sslEnabled=false, useIgniteSslCtxFactory=true, sslClientAuth=false, sslCtxFactory=null], authEnabled=false, failureHnd=null, commFailureRslvr=null] [08:20:17,142][INFO][Test worker][IgniteKernal] Daemon mode: off [08:20:17,143][INFO][Test worker][IgniteKernal] OS: Windows 7 6.1 amd64 [08:20:17,207][INFO][Test worker][IgniteKernal] OS user: X217195 [08:20:17,275][INFO][Test worker][IgniteKernal] PID: 26328 [08:20:17,332][INFO][Test worker][IgniteKernal] Language runtime: Java Platform API Specification ver. 1.8 [08:20:17,377][INFO][Test worker][IgniteKernal] VM information: Java(TM) SE Runtime Environment 1.8.0_144-b01 Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.144-b01 [08:20:17,606][INFO][Test worker][IgniteKernal] VM total memory: 0.89GB [08:20:17,608][INFO][Test worker][IgniteKernal] Remote Management [restart: off, REST: off, JMX (remote: off)] [08:20:17,609][INFO][Test worker][IgniteKernal] Logger: JavaLogger [quiet=false, config=null] [08:20:17,611][INFO][Test worker][IgniteKernal] IGNITE_HOME=H:\apache-ignite-fabric-2.4.0-bin [08:20:17,612][INFO][Test worker][IgniteKernal] VM arguments: [-DIGNITE_QUIET=false, -Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager, -Dorg.gradle.native=false, -Xms512m, -Xmx1024m, -Dfile.encoding=windows-1252, -Duser.country=US, -Duser.language=en, -Duser.variant, -ea] [08:20:17,641][INFO][Test worker][IgniteKernal] Configured caches [in 'sysMemPlc' dataRegion: ['ignite-sys-cache']] [08:20:17,746][WARNING][Test worker][IgniteKernal] Peer class loading is enabled (disable it in production for performance and deployment consistency reasons) [08:20:17,750][INFO][Test worker][IgniteKernal] 3-rd party licenses can be found at: H:\apache-ignite-fabric-2.4.0-bin\libs\licenses [08:20:17,759][INFO][Test worker][IgnitePluginProcessor] Configured plugins: [08:20:17,760][INFO][Test worker][IgnitePluginProcessor] ^-- None [08:20:17,761][INFO][Test worker][IgnitePluginProcessor] [08:20:17,767][INFO][Test worker][FailureProcessor] Configured failure handler: [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0]] [08:20:18,015][INFO][Test worker][TcpCommunicationSpi] Successfully bound communication NIO server to TCP port [port=47100, locHost=0.0.0.0/0.0.0.0, selectorsCnt=4, selectorSpins=0, pairedConn=false] [08:20:18,105][WARNING][Test worker][TcpCommunicationSpi] Message queue limit is set to 0 which may lead to potential OOMEs when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender and receiver sides. [08:20:18,444][WARNING][Test worker][NoopCheckpointSpi] Checkpoints are disabled (to enable configure any GridCheckpointSpi implementation) [08:20:18,553][WARNING][Test worker][GridCollisionManager] Collision resolution is disabled (all jobs will be activated upon arrival). [08:20:18,558][INFO][Test worker][IgniteKernal] Security status [authentication=off, tls/ssl=off] [08:20:18,592][WARNING][Test worker][IgniteH2Indexing] Custom H2 serialization is already configured, will override. [08:20:18,599][INFO][Test worker][ClientListenerProcessor] Client connector processor has started on TCP port 10800 [08:20:18,720][INFO][Test worker][GridRestProcessor] REST protocols do not start on client node. To start the protocols on client node set '-DIGNITE_REST_START_ON_CLIENT=true' system property. [08:20:20,424][INFO][Test worker][IgniteKernal] Non-loopback local IPs: 48.159.12.168 [08:20:20,425][INFO][Test worker][IgniteKernal] Enabled local MACs: 005056BD2662 [08:20:21,846][WARNING][Test worker][IgniteKernal] Nodes started on local machine require more than 20% of physical RAM what can lead to significant slowdown due to swapping (please decrease JVM heap size, data region size or checkpoint buffer size) [required=10213MB, available=8191MB] [08:20:22,005][INFO][exchange-worker-#171][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], crd=false, evt=NODE_JOINED, evtNode=22efeb8f-176c-455a-b86a-9435f0bf0736, customEvt=null, allowMerge=true] [08:20:22,049][INFO][exchange-worker-#171][GridCacheProcessor] Started cache [name=ignite-sys-cache, id=-2100569601, memoryPolicyName=sysMemPlc, mode=REPLICATED, atomicity=TRANSACTIONAL, backups=2147483647] [08:20:22,138][INFO][grid-nio-worker-tcp-comm-0-#161][TcpCommunicationSpi] Established outgoing communication connection [locAddr=/127.0.0.1:52974, rmtAddr=/127.0.0.1:47101] [08:20:22,140][INFO][exchange-worker-#171][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], crd=false] [08:20:22,166][INFO][sys-#172][GridDhtPartitionsExchangeFuture] Received full message, will finish exchange [node=09007637-a72e-4a68-940a-10392309eb64, resVer=AffinityTopologyVersion [topVer=5, minorTopVer=0]] [08:20:22,168][INFO][sys-#172][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], err=null] [08:20:22,190][INFO][Test worker][IgniteKernal] Performance suggestions for grid (fix if possible) [08:20:22,218][INFO][Test worker][IgniteKernal] To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true [08:20:22,219][INFO][Test worker][IgniteKernal] ^-- Enable G1 Garbage Collector (add '-XX:+UseG1GC' to JVM options) [08:20:22,222][INFO][Test worker][IgniteKernal] ^-- Set max direct memory size if getting 'OOME: Direct buffer memory' (add '-XX:MaxDirectMemorySize=<size>[g|G|m|M|k|K]' to JVM options) [08:20:22,230][INFO][Test worker][IgniteKernal] ^-- Disable processing of calls to System.gc() (add '-XX:+DisableExplicitGC' to JVM options) [08:20:22,238][INFO][Test worker][IgniteKernal] ^-- Disable assertions (remove '-ea' from JVM options) [08:20:22,241][INFO][Test worker][IgniteKernal] Refer to this page for more performance suggestions: https://apacheignite.readme.io/docs/jvm-and-system-tuning [08:20:22,256][INFO][Test worker][IgniteKernal] [08:20:22,258][INFO][Test worker][IgniteKernal] To start Console Management & Monitoring run ignitevisorcmd.{sh|bat} [08:20:22,258][INFO][Test worker][IgniteKernal] [08:20:22,269][INFO][Test worker][IgniteKernal] >>> +----------------------------------------------------------------------+ >>> Ignite ver. 2.5.0#20180523-sha1:86e110c750a340dc9be2d396415f0b80d7ed8813 >>> +----------------------------------------------------------------------+ >>> OS name: Windows 7 6.1 amd64 >>> CPU(s): 2 >>> Heap: 0.89GB >>> VM name: 26328@V7ABDMZP101404 >>> Local node [ID=22EFEB8F-176C-455A-B86A-9435F0BF0736, order=5, >>> clientMode=true] >>> Local node addresses: [V7ABDMZP101404.prudential.com/0:0:0:0:0:0:0:1, >>> /127.0.0.1, /48.159.12.168] >>> Local ports: TCP:10800 TCP:47100 [08:20:22,277][INFO][Test worker][GridDiscoveryManager] Topology snapshot [ver=5, servers=2, clients=1, CPUs=2, offheap=8.0GB, heap=1.8GB] [08:20:22,287][INFO][Test worker][GridDiscoveryManager] ^-- Node [id=22EFEB8F-176C-455A-B86A-9435F0BF0736, clusterState=ACTIVE] 08:20:22.290 [Test worker] INFO c.prudential.ignite.impl.IgniteUtil - Ignite start: 5.305 seconds. [08:20:22,368][INFO][exchange-worker-#171][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=1], crd=false, evt=DISCOVERY_CUSTOM_EVT, evtNode=22efeb8f-176c-455a-b86a-9435f0bf0736, customEvt=DynamicCacheChangeBatch [id=230d306a661-7ea6ad1c-8aad-432c-a1cd-701403a1f212, reqs=[DynamicCacheChangeRequest [cacheName=TEST_CACHE, hasCfg=true, nodeId=22efeb8f-176c-455a-b86a-9435f0bf0736, clientStartOnly=false, stop=false, destroy=false, disabledAfterStartfalse]], exchangeActions=ExchangeActions [startCaches=[TEST_CACHE], stopCaches=null, startGrps=[TEST_CACHE], stopGrps=[], resetParts=null, stateChangeRequest=null], startCaches=false], allowMerge=false] [08:20:22,532][INFO][exchange-worker-#171][GridCacheProcessor] Started cache [name=TEST_CACHE, id=-838655627, memoryPolicyName=null, mode=PARTITIONED, atomicity=ATOMIC, backups=0] [08:20:22,628][INFO][exchange-worker-#171][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=1], crd=false] [08:20:23,069][INFO][sys-#177][GridDhtPartitionsExchangeFuture] Received full message, will finish exchange [node=09007637-a72e-4a68-940a-10392309eb64, resVer=AffinityTopologyVersion [topVer=5, minorTopVer=1]] [08:20:23,072][INFO][sys-#177][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=5, minorTopVer=1], resVer=AffinityTopologyVersion [topVer=5, minorTopVer=1], err=null] [08:20:23,157][INFO][Test worker][GridCacheProcessor] Stopped cache [cacheName=ignite-sys-cache] [08:20:23,168][INFO][Test worker][GridCacheProcessor] Stopped cache [cacheName=TEST_CACHE] [08:20:23,192][INFO][Test worker][GridDeploymentLocalStore] Removed undeployed class: GridDeployment [ts=1540383622163, depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@73d16e93, clsLdrId=a20d306a661-22efeb8f-176c-455a-b86a-9435f0bf0736, userVer=0, loc=true, sampleClsName=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionFullMap, pendingUndeploy=false, undeployed=true, usage=0] [08:20:23,229][INFO][Test worker][IgniteKernal] >>> +---------------------------------------------------------------------------------+ >>> Ignite ver. 2.5.0#20180523-sha1:86e110c750a340dc9be2d396415f0b80d7ed8813 >>> stopped OK >>> +---------------------------------------------------------------------------------+ >>> Grid uptime: 00:00:01.036 08:20:23.258 [Test worker] WARN c.prudential.ignite.impl.IgniteUtil - No ignite lib configuration found. Returning empty properties. [08:20:23,357][INFO][Test worker][IgniteKernal] >>> __________ ________________ >>> / _/ ___/ |/ / _/_ __/ __/ >>> _/ // (7 7 // / / / / _/ >>> /___/\___/_/|_/___/ /_/ /___/ >>> >>> ver. 2.5.0#20180523-sha1:86e110c7 >>> 2018 Copyright(C) Apache Software Foundation >>> >>> Ignite documentation: http://ignite.apache.org [08:20:23,401][INFO][Test worker][IgniteKernal] Config URL: file:/C:/Users/X217195/IdeaProjects/calculation-library/ignite-impl/build/resources/integrationTest/ignite-client-conf.xml [08:20:23,431][INFO][Test worker][IgniteKernal] IgniteConfiguration [igniteInstanceName=null, pubPoolSize=8, svcPoolSize=8, callbackPoolSize=8, stripedPoolSize=8, sysPoolSize=8, mgmtPoolSize=4, igfsPoolSize=2, dataStreamerPoolSize=8, utilityCachePoolSize=8, utilityCacheKeepAliveTime=60000, p2pPoolSize=2, qryPoolSize=8, igniteHome=H:\apache-ignite-fabric-2.4.0-bin, igniteWorkDir=H:\apache-ignite-fabric-2.4.0-bin\work, mbeanSrv=com.sun.jmx.mbeanserver.JmxMBeanServer@763335f9, nodeId=bd8d192b-bea3-4660-88c1-9949488f3249, marsh=org.apache.ignite.internal.binary.BinaryMarshaller@59fd2d95, marshLocJobs=false, daemon=false, p2pEnabled=true, netTimeout=5000, sndRetryDelay=1000, sndRetryCnt=3, metricsHistSize=10000, metricsUpdateFreq=2000, metricsExpTime=9223372036854775807, discoSpi=TcpDiscoverySpi [addrRslvr=null, sockTimeout=0, ackTimeout=0, marsh=null, reconCnt=10, reconDelay=2000, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null], segPlc=STOP, segResolveAttempts=2, waitForSegOnStart=true, allResolversPassReq=true, segChkFreq=10000, commSpi=TcpCommunicationSpi [connectGate=null, connPlc=null, enableForcibleNodeKill=false, enableTroubleshootingLog=false, srvLsnr=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$2@7e5bbef1, locAddr=null, locHost=null, locPort=47100, locPortRange=100, shmemPort=-1, directBuf=true, directSndBuf=false, idleConnTimeout=600000, connTimeout=5000, maxConnTimeout=600000, reconCnt=10, sockSndBuf=32768, sockRcvBuf=32768, msgQueueLimit=0, slowClientQueueLimit=0, nioSrvr=null, shmemSrv=null, usePairedConnections=false, connectionsPerNode=1, tcpNoDelay=true, filterReachableAddresses=false, ackSndThreshold=32, unackedMsgsBufSize=0, sockWriteTimeout=2000, lsnr=null, boundTcpPort=-1, boundTcpShmemPort=-1, selectorsCnt=4, selectorSpins=0, addrRslvr=null, ctxInitLatch=java.util.concurrent.CountDownLatch@6fb88ed4[Count = 1], stopping=false, metricsLsnr=org.apache.ignite.spi.communication.tcp.TcpCommunicationMetricsListener@30b306c4], evtSpi=org.apache.ignite.spi.eventstorage.NoopEventStorageSpi@1204c2, colSpi=NoopCollisionSpi [], deploySpi=LocalDeploymentSpi [lsnr=null], indexingSpi=org.apache.ignite.spi.indexing.noop.NoopIndexingSpi@231d8e07, addrRslvr=null, clientMode=true, rebalanceThreadPoolSize=1, txCfg=org.apache.ignite.configuration.TransactionConfiguration@8786a9e, cacheSanityCheckEnabled=true, discoStartupDelay=60000, deployMode=SHARED, p2pMissedCacheSize=100, locHost=null, timeSrvPortBase=31100, timeSrvPortRange=100, failureDetectionTimeout=10000, clientFailureDetectionTimeout=30000, metricsLogFreq=60000, hadoopCfg=null, connectorCfg=org.apache.ignite.configuration.ConnectorConfiguration@20d5d873, odbcCfg=null, warmupClos=null, atomicCfg=AtomicConfiguration [seqReserveSize=1000, cacheMode=PARTITIONED, backups=1, aff=null, grpName=null], classLdr=null, sslCtxFactory=null, platformCfg=null, binaryCfg=null, memCfg=null, pstCfg=null, dsCfg=null, activeOnStart=true, autoActivation=true, longQryWarnTimeout=3000, sqlConnCfg=null, cliConnCfg=ClientConnectorConfiguration [host=null, port=10800, portRange=100, sockSndBufSize=0, sockRcvBufSize=0, tcpNoDelay=true, maxOpenCursorsPerConn=128, threadPoolSize=8, idleTimeout=0, jdbcEnabled=true, odbcEnabled=true, thinCliEnabled=true, sslEnabled=false, useIgniteSslCtxFactory=true, sslClientAuth=false, sslCtxFactory=null], authEnabled=false, failureHnd=null, commFailureRslvr=null] [08:20:23,457][INFO][Test worker][IgniteKernal] Daemon mode: off [08:20:23,475][INFO][Test worker][IgniteKernal] OS: Windows 7 6.1 amd64 [08:20:23,476][INFO][Test worker][IgniteKernal] OS user: X217195 [08:20:23,477][INFO][Test worker][IgniteKernal] PID: 26328 [08:20:23,479][INFO][Test worker][IgniteKernal] Language runtime: Java Platform API Specification ver. 1.8 [08:20:23,481][INFO][Test worker][IgniteKernal] VM information: Java(TM) SE Runtime Environment 1.8.0_144-b01 Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.144-b01 [08:20:23,482][INFO][Test worker][IgniteKernal] VM total memory: 0.89GB [08:20:23,482][INFO][Test worker][IgniteKernal] Remote Management [restart: off, REST: off, JMX (remote: off)] [08:20:23,485][INFO][Test worker][IgniteKernal] Logger: JavaLogger [quiet=false, config=null] [08:20:23,487][INFO][Test worker][IgniteKernal] IGNITE_HOME=H:\apache-ignite-fabric-2.4.0-bin [08:20:23,488][INFO][Test worker][IgniteKernal] VM arguments: [-DIGNITE_QUIET=false, -Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager, -Dorg.gradle.native=false, -Xms512m, -Xmx1024m, -Dfile.encoding=windows-1252, -Duser.country=US, -Duser.language=en, -Duser.variant, -ea] [08:20:23,488][INFO][Test worker][IgniteKernal] Configured caches [in 'sysMemPlc' dataRegion: ['ignite-sys-cache']] [08:20:23,489][WARNING][Test worker][IgniteKernal] Peer class loading is enabled (disable it in production for performance and deployment consistency reasons) [08:20:23,512][INFO][Test worker][IgniteKernal] 3-rd party licenses can be found at: H:\apache-ignite-fabric-2.4.0-bin\libs\licenses [08:20:23,566][INFO][Test worker][IgnitePluginProcessor] Configured plugins: [08:20:23,628][INFO][Test worker][IgnitePluginProcessor] ^-- None [08:20:23,633][INFO][Test worker][IgnitePluginProcessor] [08:20:23,637][INFO][Test worker][FailureProcessor] Configured failure handler: [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0]] [08:20:23,651][INFO][Test worker][TcpCommunicationSpi] Successfully bound communication NIO server to TCP port [port=47100, locHost=0.0.0.0/0.0.0.0, selectorsCnt=4, selectorSpins=0, pairedConn=false] [08:20:23,657][WARNING][Test worker][TcpCommunicationSpi] Message queue limit is set to 0 which may lead to potential OOMEs when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender and receiver sides. [08:20:23,663][WARNING][Test worker][NoopCheckpointSpi] Checkpoints are disabled (to enable configure any GridCheckpointSpi implementation) [08:20:23,674][WARNING][Test worker][GridCollisionManager] Collision resolution is disabled (all jobs will be activated upon arrival). [08:20:23,678][INFO][Test worker][IgniteKernal] Security status [authentication=off, tls/ssl=off] [08:20:23,705][WARNING][Test worker][IgniteH2Indexing] Custom H2 serialization is already configured, will override. [08:20:23,730][INFO][Test worker][ClientListenerProcessor] Client connector processor has started on TCP port 10800 [08:20:23,753][INFO][Test worker][GridRestProcessor] REST protocols do not start on client node. To start the protocols on client node set '-DIGNITE_REST_START_ON_CLIENT=true' system property. [08:20:23,899][INFO][Test worker][IgniteKernal] Non-loopback local IPs: 48.159.12.168 [08:20:23,905][INFO][Test worker][IgniteKernal] Enabled local MACs: 005056BD2662 [08:20:24,993][WARNING][Test worker][IgniteKernal] Nodes started on local machine require more than 20% of physical RAM what can lead to significant slowdown due to swapping (please decrease JVM heap size, data region size or checkpoint buffer size) [required=10213MB, available=8191MB] [08:20:25,029][INFO][exchange-worker-#212][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], crd=false, evt=NODE_JOINED, evtNode=bd8d192b-bea3-4660-88c1-9949488f3249, customEvt=null, allowMerge=true] [08:20:25,047][INFO][exchange-worker-#212][GridCacheProcessor] Started cache [name=ignite-sys-cache, id=-2100569601, memoryPolicyName=sysMemPlc, mode=REPLICATED, atomicity=TRANSACTIONAL, backups=2147483647] [08:20:25,063][INFO][grid-nio-worker-tcp-comm-0-#202][TcpCommunicationSpi] Established outgoing communication connection [locAddr=/127.0.0.1:52997, rmtAddr=/127.0.0.1:47101] [08:20:25,068][INFO][exchange-worker-#212][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], crd=false] [08:20:35,074][WARNING][exchange-worker-#212][diagnostic] Failed to wait for partition map exchange [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], node=bd8d192b-bea3-4660-88c1-9949488f3249]. Consider changing TransactionConfiguration.txTimeoutOnPartitionMapSynchronization to non default value to avoid this message. Dumping pending objects that might be the cause: [08:20:35,075][WARNING][exchange-worker-#212][diagnostic] Ready affinity version: AffinityTopologyVersion [topVer=-1, minorTopVer=0] [08:20:35,082][WARNING][exchange-worker-#212][diagnostic] Last exchange future: GridDhtPartitionsExchangeFuture [firstDiscoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=bd8d192b-bea3-4660-88c1-9949488f3249, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=7, intOrder=0, lastExchangeTime=1540383623902, loc=true, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], topVer=7, nodeId8=bd8d192b, msg=null, type=NODE_JOINED, tstamp=1540383624974], crd=TcpDiscoveryNode [id=09007637-a72e-4a68-940a-10392309eb64, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[V7ABDMZP101404.prudential.com/48.159.12.168:47501, /0:0:0:0:0:0:0:1:47501, /127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1540383624962, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=false], exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=bd8d192b-bea3-4660-88c1-9949488f3249, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=7, intOrder=0, lastExchangeTime=1540383623902, loc=true, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], topVer=7, nodeId8=bd8d192b, msg=null, type=NODE_JOINED, tstamp=1540383624974], nodeId=bd8d192b, evt=NODE_JOINED], added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=true, hash=1924369423], init=true, lastVer=null, partReleaseFut=null, exchActions=ExchangeActions [startCaches=null, stopCaches=null, startGrps=[], stopGrps=[], resetParts=null, stateChangeRequest=null], affChangeMsg=null, initTs=1540383625027, centralizedAff=false, forceAffReassignment=false, changeGlobalStateE=null, done=false, state=CLIENT, evtLatch=0, remaining=[1943a94c-caa2-4082-a04b-a06913489a60, 09007637-a72e-4a68-940a-10392309eb64], super=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=529982688]] [08:20:35,083][WARNING][exchange-worker-#212][GridCachePartitionExchangeManager] First 10 pending exchange futures [total=0] [08:20:35,092][WARNING][exchange-worker-#212][diagnostic] Last 10 exchange futures (total: 1): [08:20:35,094][WARNING][exchange-worker-#212][diagnostic] >>> GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=bd8d192b-bea3-4660-88c1-9949488f3249, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=7, intOrder=0, lastExchangeTime=1540383623902, loc=true, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], done=false] [08:20:35,101][WARNING][exchange-worker-#212][diagnostic] Latch manager state: ExchangeLatchManager [serverLatches={}, clientLatches={}] [08:20:35,104][WARNING][exchange-worker-#212][diagnostic] Pending transactions: [08:20:35,106][WARNING][exchange-worker-#212][diagnostic] Pending explicit locks: [08:20:35,106][WARNING][exchange-worker-#212][diagnostic] Pending cache futures: [08:20:35,107][WARNING][exchange-worker-#212][diagnostic] Pending atomic cache futures: [08:20:35,108][WARNING][exchange-worker-#212][diagnostic] Pending data streamer futures: [08:20:35,109][WARNING][exchange-worker-#212][diagnostic] Pending transaction deadlock detection futures: [08:20:35,155][INFO][grid-nio-worker-tcp-comm-0-#202][diagnostic] Exchange future waiting for coordinator response [crd=09007637-a72e-4a68-940a-10392309eb64, topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0]] Remote node information: General node info [id=09007637-a72e-4a68-940a-10392309eb64, client=false, discoTopVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], time=08:20:35.125] Partitions exchange info [readyVer=AffinityTopologyVersion [topVer=6, minorTopVer=0]] Last initialized exchange future: GridDhtPartitionsExchangeFuture [firstDiscoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=22efeb8f-176c-455a-b86a-9435f0bf0736, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=5, intOrder=4, lastExchangeTime=1540383621546, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], topVer=6, nodeId8=09007637, msg=Node left: TcpDiscoveryNode [id=22efeb8f-176c-455a-b86a-9435f0bf0736, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=5, intOrder=4, lastExchangeTime=1540383621546, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], type=NODE_LEFT, tstamp=1540383623258], crd=TcpDiscoveryNode [id=09007637-a72e-4a68-940a-10392309eb64, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[V7ABDMZP101404.prudential.com/48.159.12.168:47501, /0:0:0:0:0:0:0:1:47501, /127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1540383634895, loc=true, ver=2.5.0#20180523-sha1:86e110c7, isClient=false], exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=22efeb8f-176c-455a-b86a-9435f0bf0736, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=5, intOrder=4, lastExchangeTime=1540383621546, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], topVer=6, nodeId8=09007637, msg=Node left: TcpDiscoveryNode [id=22efeb8f-176c-455a-b86a-9435f0bf0736, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=5, intOrder=4, lastExchangeTime=1540383621546, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], type=NODE_LEFT, tstamp=1540383623258], nodeId=22efeb8f, evt=NODE_LEFT], added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=true, hash=760946415], init=false, lastVer=null, partReleaseFut=null, exchActions=null, affChangeMsg=null, initTs=1540383623279, centralizedAff=false, forceAffReassignment=false, changeGlobalStateE=null, done=true, state=CRD, evtLatch=0, remaining=[1943a94c-caa2-4082-a04b-a06913489a60], super=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=AffinityTopologyVersion [topVer=6, minorTopVer=0], hash=605301579]] Communication SPI statistics [rmtNode=bd8d192b-bea3-4660-88c1-9949488f3249] Communication SPI recovery descriptors: [key=ConnectionKey [nodeId=bd8d192b-bea3-4660-88c1-9949488f3249, idx=0, connCnt=0, dummy=false], msgsSent=0, msgsAckedByRmt=0, msgsRcvd=2, lastAcked=0, reserveCnt=1, descIdHash=689424991] Communication SPI clients: [node=bd8d192b-bea3-4660-88c1-9949488f3249, client=GridTcpNioCommunicationClient [ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=0, bytesRcvd=987, bytesSent=426, bytesRcvd0=853, bytesSent0=0, select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-0, igniteInstanceName=integration_tests, finished=false, hashCode=1400285321, interrupted=false, runner=grid-nio-worker-tcp-comm-0-#25%integration_tests%]]], writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], inRecovery=GridNioRecoveryDescriptor [acked=0, resendCnt=0, rcvCnt=2, sentCnt=0, reserved=true, lastAck=0, nodeLeft=false, node=TcpDiscoveryNode [id=bd8d192b-bea3-4660-88c1-9949488f3249, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=7, intOrder=5, lastExchangeTime=1540383624919, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], connected=true, connectCnt=0, queueLimit=4096, reserveCnt=1, pairedConnections=false], outRecovery=GridNioRecoveryDescriptor [acked=0, resendCnt=0, rcvCnt=2, sentCnt=0, reserved=true, lastAck=0, nodeLeft=false, node=TcpDiscoveryNode [id=bd8d192b-bea3-4660-88c1-9949488f3249, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=7, intOrder=5, lastExchangeTime=1540383624919, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], connected=true, connectCnt=0, queueLimit=4096, reserveCnt=1, pairedConnections=false], super=GridNioSessionImpl [locAddr=/127.0.0.1:47101, rmtAddr=/127.0.0.1:52997, createTime=1540383625057, closeTime=0, bytesSent=28, bytesRcvd=963, bytesSent0=0, bytesRcvd0=853, sndSchedTime=1540383625057, lastSndTime=1540383625057, lastRcvTime=1540383635111, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=org.apache.ignite.internal.util.nio.GridDirectParser@1fff65b2, directMode=true], GridConnectionBytesVerifyFilter], accepted=true]], super=GridAbstractCommunicationClient [lastUsed=1540383625057, closed=false, connIdx=0]]] NIO sessions statistics: >> Selector info [idx=0, keysCnt=1, bytesRcvd=987, bytesRcvd0=853, >> bytesSent=426, bytesSent0=0] Connection info [in=true, rmtAddr=/127.0.0.1:52997, locAddr=/127.0.0.1:47101, msgsSent=0, msgsAckedByRmt=0, descIdHash=689424991, msgsRcvd=2, lastAcked=0, descIdHash=689424991, bytesRcvd=963, bytesRcvd0=853, bytesSent=28, bytesSent0=0, opQueueSize=0] Exchange future: GridDhtPartitionsExchangeFuture [firstDiscoEvt=null, crd=null, exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], discoEvt=null, nodeId=bd8d192b, evt=NODE_JOINED], added=false, initFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1896978384], init=false, lastVer=null, partReleaseFut=null, exchActions=null, affChangeMsg=null, initTs=0, centralizedAff=false, forceAffReassignment=false, changeGlobalStateE=null, done=false, state=null, evtLatch=1, remaining=[], super=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=155967337]] Local communication statistics: Communication SPI statistics [rmtNode=09007637-a72e-4a68-940a-10392309eb64] Communication SPI recovery descriptors: [key=ConnectionKey [nodeId=09007637-a72e-4a68-940a-10392309eb64, idx=0, connCnt=-1, dummy=false], msgsSent=2, msgsAckedByRmt=0, msgsRcvd=1, lastAcked=0, reserveCnt=1, descIdHash=472467749] Communication SPI clients: [node=09007637-a72e-4a68-940a-10392309eb64, client=GridTcpNioCommunicationClient [ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=0, bytesRcvd=6542, bytesSent=921, bytesRcvd0=6542, bytesSent0=853, select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-0, igniteInstanceName=null, finished=false, hashCode=2080814698, interrupted=false, runner=grid-nio-worker-tcp-comm-0-#202]]], writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], inRecovery=GridNioRecoveryDescriptor [acked=0, resendCnt=0, rcvCnt=1, sentCnt=2, reserved=true, lastAck=0, nodeLeft=false, node=TcpDiscoveryNode [id=09007637-a72e-4a68-940a-10392309eb64, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[V7ABDMZP101404.prudential.com/48.159.12.168:47501, /0:0:0:0:0:0:0:1:47501, /127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1540383624962, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=false], connected=false, connectCnt=1, queueLimit=4096, reserveCnt=1, pairedConnections=false], outRecovery=GridNioRecoveryDescriptor [acked=0, resendCnt=0, rcvCnt=1, sentCnt=2, reserved=true, lastAck=0, nodeLeft=false, node=TcpDiscoveryNode [id=09007637-a72e-4a68-940a-10392309eb64, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[V7ABDMZP101404.prudential.com/48.159.12.168:47501, /0:0:0:0:0:0:0:1:47501, /127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1540383624962, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=false], connected=false, connectCnt=1, queueLimit=4096, reserveCnt=1, pairedConnections=false], super=GridNioSessionImpl [locAddr=/127.0.0.1:52997, rmtAddr=/127.0.0.1:47101, createTime=1540383625060, closeTime=0, bytesSent=921, bytesRcvd=6542, bytesSent0=853, bytesRcvd0=6542, sndSchedTime=1540383625060, lastSndTime=1540383635111, lastRcvTime=1540383635135, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=org.apache.ignite.internal.util.nio.GridDirectParser@2ab0a278, directMode=true], GridConnectionBytesVerifyFilter], accepted=false]], super=GridAbstractCommunicationClient [lastUsed=1540383625060, closed=false, connIdx=0]]] NIO sessions statistics: >> Selector info [idx=0, keysCnt=1, bytesRcvd=6542, bytesRcvd0=6542, >> bytesSent=921, bytesSent0=853] Connection info [in=false, rmtAddr=/127.0.0.1:47101, locAddr=/127.0.0.1:52997, msgsSent=2, msgsAckedByRmt=0, descIdHash=472467749, unackedMsgs=[GridDhtPartitionsSingleMessage, IgniteDiagnosticMessage], msgsRcvd=1, lastAcked=0, descIdHash=472467749, bytesRcvd=6542, bytesRcvd0=6542, bytesSent=921, bytesSent0=853, opQueueSize=0] [08:20:37,769][INFO][ignite-update-notifier-timer][GridUpdateNotifier] Update status is not available. [08:20:45,029][WARNING][Test worker][GridCachePartitionExchangeManager] Failed to wait for initial partition map exchange. Possible reasons are: ^-- Transactions in deadlock. ^-- Long running transactions (ignore if this is the case). ^-- Unreleased explicit locks. [08:20:52,225][INFO][ignite-update-notifier-timer][GridUpdateNotifier] Update status is not available. [08:20:55,143][WARNING][exchange-worker-#212][diagnostic] Failed to wait for partition map exchange [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], node=bd8d192b-bea3-4660-88c1-9949488f3249]. Consider changing TransactionConfiguration.txTimeoutOnPartitionMapSynchronization to non default value to avoid this message. Dumping pending objects that might be the cause: [08:20:55,192][WARNING][exchange-worker-#212][diagnostic] Ready affinity version: AffinityTopologyVersion [topVer=-1, minorTopVer=0] [08:20:55,227][WARNING][exchange-worker-#212][diagnostic] Last exchange future: GridDhtPartitionsExchangeFuture [firstDiscoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=bd8d192b-bea3-4660-88c1-9949488f3249, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=7, intOrder=0, lastExchangeTime=1540383623902, loc=true, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], topVer=7, nodeId8=bd8d192b, msg=null, type=NODE_JOINED, tstamp=1540383624974], crd=TcpDiscoveryNode [id=09007637-a72e-4a68-940a-10392309eb64, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[V7ABDMZP101404.prudential.com/48.159.12.168:47501, /0:0:0:0:0:0:0:1:47501, /127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1540383624962, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=false], exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=bd8d192b-bea3-4660-88c1-9949488f3249, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=7, intOrder=0, lastExchangeTime=1540383623902, loc=true, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], topVer=7, nodeId8=bd8d192b, msg=null, type=NODE_JOINED, tstamp=1540383624974], nodeId=bd8d192b, evt=NODE_JOINED], added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=true, hash=1924369423], init=true, lastVer=null, partReleaseFut=null, exchActions=ExchangeActions [startCaches=null, stopCaches=null, startGrps=[], stopGrps=[], resetParts=null, stateChangeRequest=null], affChangeMsg=null, initTs=1540383625027, centralizedAff=false, forceAffReassignment=false, changeGlobalStateE=null, done=false, state=CLIENT, evtLatch=0, remaining=[1943a94c-caa2-4082-a04b-a06913489a60, 09007637-a72e-4a68-940a-10392309eb64], super=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=529982688]] [08:20:55,240][WARNING][exchange-worker-#212][GridCachePartitionExchangeManager] First 10 pending exchange futures [total=0] [08:20:55,244][WARNING][exchange-worker-#212][diagnostic] Last 10 exchange futures (total: 1): [08:20:55,251][WARNING][exchange-worker-#212][diagnostic] >>> GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=bd8d192b-bea3-4660-88c1-9949488f3249, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=7, intOrder=0, lastExchangeTime=1540383623902, loc=true, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], done=false] [08:20:55,255][WARNING][exchange-worker-#212][diagnostic] Latch manager state: ExchangeLatchManager [serverLatches={}, clientLatches={}] [08:20:55,256][WARNING][exchange-worker-#212][diagnostic] Pending transactions: [08:20:55,257][WARNING][exchange-worker-#212][diagnostic] Pending explicit locks: [08:20:55,266][WARNING][exchange-worker-#212][diagnostic] Pending cache futures: [08:20:55,271][WARNING][exchange-worker-#212][diagnostic] Pending atomic cache futures: [08:20:55,272][WARNING][exchange-worker-#212][diagnostic] Pending data streamer futures: [08:20:55,273][WARNING][exchange-worker-#212][diagnostic] Pending transaction deadlock detection futures: [08:20:55,306][INFO][grid-nio-worker-tcp-comm-3-#205][diagnostic] Exchange future waiting for coordinator response [crd=09007637-a72e-4a68-940a-10392309eb64, topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0]] Remote node information: General node info [id=09007637-a72e-4a68-940a-10392309eb64, client=false, discoTopVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], time=08:20:55.266] Partitions exchange info [readyVer=AffinityTopologyVersion [topVer=6, minorTopVer=0]] Last initialized exchange future: GridDhtPartitionsExchangeFuture [firstDiscoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=22efeb8f-176c-455a-b86a-9435f0bf0736, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=5, intOrder=4, lastExchangeTime=1540383621546, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], topVer=6, nodeId8=09007637, msg=Node left: TcpDiscoveryNode [id=22efeb8f-176c-455a-b86a-9435f0bf0736, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=5, intOrder=4, lastExchangeTime=1540383621546, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], type=NODE_LEFT, tstamp=1540383623258], crd=TcpDiscoveryNode [id=09007637-a72e-4a68-940a-10392309eb64, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[V7ABDMZP101404.prudential.com/48.159.12.168:47501, /0:0:0:0:0:0:0:1:47501, /127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1540383654976, loc=true, ver=2.5.0#20180523-sha1:86e110c7, isClient=false], exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=22efeb8f-176c-455a-b86a-9435f0bf0736, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=5, intOrder=4, lastExchangeTime=1540383621546, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], topVer=6, nodeId8=09007637, msg=Node left: TcpDiscoveryNode [id=22efeb8f-176c-455a-b86a-9435f0bf0736, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=5, intOrder=4, lastExchangeTime=1540383621546, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], type=NODE_LEFT, tstamp=1540383623258], nodeId=22efeb8f, evt=NODE_LEFT], added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=true, hash=760946415], init=false, lastVer=null, partReleaseFut=null, exchActions=null, affChangeMsg=null, initTs=1540383623279, centralizedAff=false, forceAffReassignment=false, changeGlobalStateE=null, done=true, state=CRD, evtLatch=0, remaining=[1943a94c-caa2-4082-a04b-a06913489a60], super=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=AffinityTopologyVersion [topVer=6, minorTopVer=0], hash=605301579]] Communication SPI statistics [rmtNode=bd8d192b-bea3-4660-88c1-9949488f3249] Communication SPI recovery descriptors: [key=ConnectionKey [nodeId=bd8d192b-bea3-4660-88c1-9949488f3249, idx=0, connCnt=0, dummy=false], msgsSent=1, msgsAckedByRmt=0, msgsRcvd=3, lastAcked=0, reserveCnt=1, descIdHash=689424991] Communication SPI clients: [node=bd8d192b-bea3-4660-88c1-9949488f3249, client=GridTcpNioCommunicationClient [ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=0, bytesRcvd=1840, bytesSent=6968, bytesRcvd0=853, bytesSent0=0, select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-0, igniteInstanceName=integration_tests, finished=false, hashCode=1400285321, interrupted=false, runner=grid-nio-worker-tcp-comm-0-#25%integration_tests%]]], writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], inRecovery=GridNioRecoveryDescriptor [acked=0, resendCnt=0, rcvCnt=3, sentCnt=1, reserved=true, lastAck=0, nodeLeft=false, node=TcpDiscoveryNode [id=bd8d192b-bea3-4660-88c1-9949488f3249, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=7, intOrder=5, lastExchangeTime=1540383624919, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], connected=true, connectCnt=0, queueLimit=4096, reserveCnt=1, pairedConnections=false], outRecovery=GridNioRecoveryDescriptor [acked=0, resendCnt=0, rcvCnt=3, sentCnt=1, reserved=true, lastAck=0, nodeLeft=false, node=TcpDiscoveryNode [id=bd8d192b-bea3-4660-88c1-9949488f3249, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=7, intOrder=5, lastExchangeTime=1540383624919, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], connected=true, connectCnt=0, queueLimit=4096, reserveCnt=1, pairedConnections=false], super=GridNioSessionImpl [locAddr=/127.0.0.1:47101, rmtAddr=/127.0.0.1:52997, createTime=1540383625057, closeTime=0, bytesSent=6570, bytesRcvd=1816, bytesSent0=0, bytesRcvd0=853, sndSchedTime=1540383625057, lastSndTime=1540383635135, lastRcvTime=1540383655266, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=org.apache.ignite.internal.util.nio.GridDirectParser@1fff65b2, directMode=true], GridConnectionBytesVerifyFilter], accepted=true]], super=GridAbstractCommunicationClient [lastUsed=1540383625057, closed=false, connIdx=0]]] NIO sessions statistics: >> Selector info [idx=0, keysCnt=1, bytesRcvd=1840, bytesRcvd0=853, >> bytesSent=6968, bytesSent0=0] Connection info [in=true, rmtAddr=/127.0.0.1:52997, locAddr=/127.0.0.1:47101, msgsSent=1, msgsAckedByRmt=0, descIdHash=689424991, unackedMsgs=[IgniteDiagnosticMessage], msgsRcvd=3, lastAcked=0, descIdHash=689424991, bytesRcvd=1816, bytesRcvd0=853, bytesSent=6570, bytesSent0=0, opQueueSize=0] Exchange future: GridDhtPartitionsExchangeFuture [firstDiscoEvt=null, crd=null, exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], discoEvt=null, nodeId=bd8d192b, evt=NODE_JOINED], added=false, initFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1896978384], init=false, lastVer=null, partReleaseFut=null, exchActions=null, affChangeMsg=null, initTs=0, centralizedAff=false, forceAffReassignment=false, changeGlobalStateE=null, done=false, state=null, evtLatch=1, remaining=[], super=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=155967337]] Local communication statistics: Communication SPI statistics [rmtNode=09007637-a72e-4a68-940a-10392309eb64] Communication SPI recovery descriptors: [key=ConnectionKey [nodeId=09007637-a72e-4a68-940a-10392309eb64, idx=0, connCnt=-1, dummy=false], msgsSent=3, msgsAckedByRmt=0, msgsRcvd=2, lastAcked=0, reserveCnt=1, descIdHash=472467749] Communication SPI clients: [node=09007637-a72e-4a68-940a-10392309eb64, client=GridTcpNioCommunicationClient [ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=0, bytesRcvd=13133, bytesSent=1774, bytesRcvd0=6591, bytesSent0=853, select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-0, igniteInstanceName=null, finished=false, hashCode=2080814698, interrupted=false, runner=grid-nio-worker-tcp-comm-0-#202]]], writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], inRecovery=GridNioRecoveryDescriptor [acked=0, resendCnt=0, rcvCnt=2, sentCnt=3, reserved=true, lastAck=0, nodeLeft=false, node=TcpDiscoveryNode [id=09007637-a72e-4a68-940a-10392309eb64, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[V7ABDMZP101404.prudential.com/48.159.12.168:47501, /0:0:0:0:0:0:0:1:47501, /127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1540383624962, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=false], connected=false, connectCnt=1, queueLimit=4096, reserveCnt=1, pairedConnections=false], outRecovery=GridNioRecoveryDescriptor [acked=0, resendCnt=0, rcvCnt=2, sentCnt=3, reserved=true, lastAck=0, nodeLeft=false, node=TcpDiscoveryNode [id=09007637-a72e-4a68-940a-10392309eb64, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[V7ABDMZP101404.prudential.com/48.159.12.168:47501, /0:0:0:0:0:0:0:1:47501, /127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1540383624962, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=false], connected=false, connectCnt=1, queueLimit=4096, reserveCnt=1, pairedConnections=false], super=GridNioSessionImpl [locAddr=/127.0.0.1:52997, rmtAddr=/127.0.0.1:47101, createTime=1540383625060, closeTime=0, bytesSent=1774, bytesRcvd=13133, bytesSent0=853, bytesRcvd0=6591, sndSchedTime=1540383625060, lastSndTime=1540383655266, lastRcvTime=1540383655287, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=org.apache.ignite.internal.util.nio.GridDirectParser@2ab0a278, directMode=true], GridConnectionBytesVerifyFilter], accepted=false]], super=GridAbstractCommunicationClient [lastUsed=1540383625060, closed=false, connIdx=0]]] NIO sessions statistics: >> Selector info [idx=0, keysCnt=1, bytesRcvd=13133, bytesRcvd0=6591, >> bytesSent=1774, bytesSent0=853] Connection info [in=false, rmtAddr=/127.0.0.1:47101, locAddr=/127.0.0.1:52997, msgsSent=3, msgsAckedByRmt=0, descIdHash=472467749, unackedMsgs=[GridDhtPartitionsSingleMessage, IgniteDiagnosticMessage, IgniteDiagnosticMessage], msgsRcvd=2, lastAcked=0, descIdHash=472467749, bytesRcvd=13133, bytesRcvd0=6591, bytesSent=1774, bytesSent0=853, opQueueSize=0] [08:21:05,041][WARNING][Test worker][GridCachePartitionExchangeManager] Still waiting for initial partition map exchange [fut=GridDhtPartitionsExchangeFuture [firstDiscoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=bd8d192b-bea3-4660-88c1-9949488f3249, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=7, intOrder=0, lastExchangeTime=1540383623902, loc=true, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], topVer=7, nodeId8=bd8d192b, msg=null, type=NODE_JOINED, tstamp=1540383624974], crd=TcpDiscoveryNode [id=09007637-a72e-4a68-940a-10392309eb64, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[V7ABDMZP101404.prudential.com/48.159.12.168:47501, /0:0:0:0:0:0:0:1:47501, /127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1540383624962, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=false], exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=bd8d192b-bea3-4660-88c1-9949488f3249, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=7, intOrder=0, lastExchangeTime=1540383623902, loc=true, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], topVer=7, nodeId8=bd8d192b, msg=null, type=NODE_JOINED, tstamp=1540383624974], nodeId=bd8d192b, evt=NODE_JOINED], added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=true, hash=1924369423], init=true, lastVer=null, partReleaseFut=null, exchActions=ExchangeActions [startCaches=null, stopCaches=null, startGrps=[], stopGrps=[], resetParts=null, stateChangeRequest=null], affChangeMsg=null, initTs=1540383625027, centralizedAff=false, forceAffReassignment=false, changeGlobalStateE=null, done=false, state=CLIENT, evtLatch=0, remaining=[1943a94c-caa2-4082-a04b-a06913489a60, 09007637-a72e-4a68-940a-10392309eb64], super=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=529982688]]] [08:21:15,280][WARNING][exchange-worker-#212][diagnostic] Failed to wait for partition map exchange [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], node=bd8d192b-bea3-4660-88c1-9949488f3249]. Consider changing TransactionConfiguration.txTimeoutOnPartitionMapSynchronization to non default value to avoid this message. Dumping pending objects that might be the cause: [08:21:15,282][WARNING][exchange-worker-#212][diagnostic] Ready affinity version: AffinityTopologyVersion [topVer=-1, minorTopVer=0] [08:21:15,285][WARNING][exchange-worker-#212][diagnostic] Last exchange future: GridDhtPartitionsExchangeFuture [firstDiscoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=bd8d192b-bea3-4660-88c1-9949488f3249, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=7, intOrder=0, lastExchangeTime=1540383623902, loc=true, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], topVer=7, nodeId8=bd8d192b, msg=null, type=NODE_JOINED, tstamp=1540383624974], crd=TcpDiscoveryNode [id=09007637-a72e-4a68-940a-10392309eb64, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[V7ABDMZP101404.prudential.com/48.159.12.168:47501, /0:0:0:0:0:0:0:1:47501, /127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1540383624962, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=false], exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=bd8d192b-bea3-4660-88c1-9949488f3249, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=7, intOrder=0, lastExchangeTime=1540383623902, loc=true, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], topVer=7, nodeId8=bd8d192b, msg=null, type=NODE_JOINED, tstamp=1540383624974], nodeId=bd8d192b, evt=NODE_JOINED], added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=true, hash=1924369423], init=true, lastVer=null, partReleaseFut=null, exchActions=ExchangeActions [startCaches=null, stopCaches=null, startGrps=[], stopGrps=[], resetParts=null, stateChangeRequest=null], affChangeMsg=null, initTs=1540383625027, centralizedAff=false, forceAffReassignment=false, changeGlobalStateE=null, done=false, state=CLIENT, evtLatch=0, remaining=[1943a94c-caa2-4082-a04b-a06913489a60, 09007637-a72e-4a68-940a-10392309eb64], super=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=529982688]] [08:21:15,288][WARNING][exchange-worker-#212][GridCachePartitionExchangeManager] First 10 pending exchange futures [total=0] [08:21:15,290][WARNING][exchange-worker-#212][diagnostic] Last 10 exchange futures (total: 1): [08:21:15,292][WARNING][exchange-worker-#212][diagnostic] >>> GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=bd8d192b-bea3-4660-88c1-9949488f3249, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=7, intOrder=0, lastExchangeTime=1540383623902, loc=true, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], done=false] [08:21:15,293][WARNING][exchange-worker-#212][diagnostic] Latch manager state: ExchangeLatchManager [serverLatches={}, clientLatches={}] [08:21:15,294][WARNING][exchange-worker-#212][diagnostic] Pending transactions: [08:21:15,296][WARNING][exchange-worker-#212][diagnostic] Pending explicit locks: [08:21:15,297][WARNING][exchange-worker-#212][diagnostic] Pending cache futures: [08:21:15,301][WARNING][exchange-worker-#212][diagnostic] Pending atomic cache futures: [08:21:15,303][WARNING][exchange-worker-#212][diagnostic] Pending data streamer futures: [08:21:15,304][WARNING][exchange-worker-#212][diagnostic] Pending transaction deadlock detection futures: [08:21:15,317][INFO][sys-#215][diagnostic] Exchange future waiting for coordinator response [crd=09007637-a72e-4a68-940a-10392309eb64, topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0]] Remote node information: General node info [id=09007637-a72e-4a68-940a-10392309eb64, client=false, discoTopVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], time=08:21:15.301] Partitions exchange info [readyVer=AffinityTopologyVersion [topVer=6, minorTopVer=0]] Last initialized exchange future: GridDhtPartitionsExchangeFuture [firstDiscoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=22efeb8f-176c-455a-b86a-9435f0bf0736, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=5, intOrder=4, lastExchangeTime=1540383621546, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], topVer=6, nodeId8=09007637, msg=Node left: TcpDiscoveryNode [id=22efeb8f-176c-455a-b86a-9435f0bf0736, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=5, intOrder=4, lastExchangeTime=1540383621546, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], type=NODE_LEFT, tstamp=1540383623258], crd=TcpDiscoveryNode [id=09007637-a72e-4a68-940a-10392309eb64, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[V7ABDMZP101404.prudential.com/48.159.12.168:47501, /0:0:0:0:0:0:0:1:47501, /127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1540383675011, loc=true, ver=2.5.0#20180523-sha1:86e110c7, isClient=false], exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=22efeb8f-176c-455a-b86a-9435f0bf0736, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=5, intOrder=4, lastExchangeTime=1540383621546, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], topVer=6, nodeId8=09007637, msg=Node left: TcpDiscoveryNode [id=22efeb8f-176c-455a-b86a-9435f0bf0736, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=5, intOrder=4, lastExchangeTime=1540383621546, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], type=NODE_LEFT, tstamp=1540383623258], nodeId=22efeb8f, evt=NODE_LEFT], added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=true, hash=760946415], init=false, lastVer=null, partReleaseFut=null, exchActions=null, affChangeMsg=null, initTs=1540383623279, centralizedAff=false, forceAffReassignment=false, changeGlobalStateE=null, done=true, state=CRD, evtLatch=0, remaining=[1943a94c-caa2-4082-a04b-a06913489a60], super=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=AffinityTopologyVersion [topVer=6, minorTopVer=0], hash=605301579]] Communication SPI statistics [rmtNode=bd8d192b-bea3-4660-88c1-9949488f3249] Communication SPI recovery descriptors: [key=ConnectionKey [nodeId=bd8d192b-bea3-4660-88c1-9949488f3249, idx=0, connCnt=0, dummy=false], msgsSent=2, msgsAckedByRmt=0, msgsRcvd=4, lastAcked=0, reserveCnt=1, descIdHash=689424991] Communication SPI clients: [node=bd8d192b-bea3-4660-88c1-9949488f3249, client=GridTcpNioCommunicationClient [ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=0, bytesRcvd=2693, bytesSent=13559, bytesRcvd0=853, bytesSent0=0, select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-0, igniteInstanceName=integration_tests, finished=false, hashCode=1400285321, interrupted=false, runner=grid-nio-worker-tcp-comm-0-#25%integration_tests%]]], writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], inRecovery=GridNioRecoveryDescriptor [acked=0, resendCnt=0, rcvCnt=4, sentCnt=2, reserved=true, lastAck=0, nodeLeft=false, node=TcpDiscoveryNode [id=bd8d192b-bea3-4660-88c1-9949488f3249, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=7, intOrder=5, lastExchangeTime=1540383624919, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], connected=true, connectCnt=0, queueLimit=4096, reserveCnt=1, pairedConnections=false], outRecovery=GridNioRecoveryDescriptor [acked=0, resendCnt=0, rcvCnt=4, sentCnt=2, reserved=true, lastAck=0, nodeLeft=false, node=TcpDiscoveryNode [id=bd8d192b-bea3-4660-88c1-9949488f3249, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=7, intOrder=5, lastExchangeTime=1540383624919, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], connected=true, connectCnt=0, queueLimit=4096, reserveCnt=1, pairedConnections=false], super=GridNioSessionImpl [locAddr=/127.0.0.1:47101, rmtAddr=/127.0.0.1:52997, createTime=1540383625057, closeTime=0, bytesSent=13161, bytesRcvd=2669, bytesSent0=0, bytesRcvd0=853, sndSchedTime=1540383625057, lastSndTime=1540383655287, lastRcvTime=1540383675301, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=org.apache.ignite.internal.util.nio.GridDirectParser@1fff65b2, directMode=true], GridConnectionBytesVerifyFilter], accepted=true]], super=GridAbstractCommunicationClient [lastUsed=1540383625057, closed=false, connIdx=0]]] NIO sessions statistics: >> Selector info [idx=0, keysCnt=1, bytesRcvd=2693, bytesRcvd0=853, >> bytesSent=13559, bytesSent0=0] Connection info [in=true, rmtAddr=/127.0.0.1:52997, locAddr=/127.0.0.1:47101, msgsSent=2, msgsAckedByRmt=0, descIdHash=689424991, unackedMsgs=[IgniteDiagnosticMessage, IgniteDiagnosticMessage], msgsRcvd=4, lastAcked=0, descIdHash=689424991, bytesRcvd=2669, bytesRcvd0=853, bytesSent=13161, bytesSent0=0, opQueueSize=0] Exchange future: GridDhtPartitionsExchangeFuture [firstDiscoEvt=null, crd=null, exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], discoEvt=null, nodeId=bd8d192b, evt=NODE_JOINED], added=false, initFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1896978384], init=false, lastVer=null, partReleaseFut=null, exchActions=null, affChangeMsg=null, initTs=0, centralizedAff=false, forceAffReassignment=false, changeGlobalStateE=null, done=false, state=null, evtLatch=1, remaining=[], super=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=155967337]] Local communication statistics: Communication SPI statistics [rmtNode=09007637-a72e-4a68-940a-10392309eb64] Communication SPI recovery descriptors: [key=ConnectionKey [nodeId=09007637-a72e-4a68-940a-10392309eb64, idx=0, connCnt=-1, dummy=false], msgsSent=4, msgsAckedByRmt=0, msgsRcvd=3, lastAcked=0, reserveCnt=1, descIdHash=472467749] Communication SPI clients: [node=09007637-a72e-4a68-940a-10392309eb64, client=GridTcpNioCommunicationClient [ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=0, bytesRcvd=19753, bytesSent=2627, bytesRcvd0=6620, bytesSent0=853, select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-0, igniteInstanceName=null, finished=false, hashCode=2080814698, interrupted=false, runner=grid-nio-worker-tcp-comm-0-#202]]], writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], inRecovery=GridNioRecoveryDescriptor [acked=0, resendCnt=0, rcvCnt=3, sentCnt=4, reserved=true, lastAck=0, nodeLeft=false, node=TcpDiscoveryNode [id=09007637-a72e-4a68-940a-10392309eb64, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[V7ABDMZP101404.prudential.com/48.159.12.168:47501, /0:0:0:0:0:0:0:1:47501, /127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1540383624962, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=false], connected=false, connectCnt=1, queueLimit=4096, reserveCnt=1, pairedConnections=false], outRecovery=GridNioRecoveryDescriptor [acked=0, resendCnt=0, rcvCnt=3, sentCnt=4, reserved=true, lastAck=0, nodeLeft=false, node=TcpDiscoveryNode [id=09007637-a72e-4a68-940a-10392309eb64, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[V7ABDMZP101404.prudential.com/48.159.12.168:47501, /0:0:0:0:0:0:0:1:47501, /127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1540383624962, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=false], connected=false, connectCnt=1, queueLimit=4096, reserveCnt=1, pairedConnections=false], super=GridNioSessionImpl [locAddr=/127.0.0.1:52997, rmtAddr=/127.0.0.1:47101, createTime=1540383625060, closeTime=0, bytesSent=2627, bytesRcvd=19753, bytesSent0=853, bytesRcvd0=6620, sndSchedTime=1540383625060, lastSndTime=1540383675301, lastRcvTime=1540383675312, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=org.apache.ignite.internal.util.nio.GridDirectParser@2ab0a278, directMode=true], GridConnectionBytesVerifyFilter], accepted=false]], super=GridAbstractCommunicationClient [lastUsed=1540383625060, closed=false, connIdx=0]]] NIO sessions statistics: >> Selector info [idx=0, keysCnt=1, bytesRcvd=19753, bytesRcvd0=6620, >> bytesSent=2627, bytesSent0=853] Connection info [in=false, rmtAddr=/127.0.0.1:47101, locAddr=/127.0.0.1:52997, msgsSent=4, msgsAckedByRmt=0, descIdHash=472467749, unackedMsgs=[GridDhtPartitionsSingleMessage, IgniteDiagnosticMessage, IgniteDiagnosticMessage, IgniteDiagnosticMessage], msgsRcvd=3, lastAcked=0, descIdHash=472467749, bytesRcvd=19753, bytesRcvd0=6620, bytesSent=2627, bytesSent0=853, opQueueSize=0] [08:21:17,642][INFO][ignite-update-notifier-timer][GridUpdateNotifier] Update status is not available. [08:21:25,065][WARNING][Test worker][GridCachePartitionExchangeManager] Still waiting for initial partition map exchange [fut=GridDhtPartitionsExchangeFuture [firstDiscoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=bd8d192b-bea3-4660-88c1-9949488f3249, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=7, intOrder=0, lastExchangeTime=1540383623902, loc=true, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], topVer=7, nodeId8=bd8d192b, msg=null, type=NODE_JOINED, tstamp=1540383624974], crd=TcpDiscoveryNode [id=09007637-a72e-4a68-940a-10392309eb64, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[V7ABDMZP101404.prudential.com/48.159.12.168:47501, /0:0:0:0:0:0:0:1:47501, /127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1540383624962, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=false], exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=bd8d192b-bea3-4660-88c1-9949488f3249, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=7, intOrder=0, lastExchangeTime=1540383623902, loc=true, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], topVer=7, nodeId8=bd8d192b, msg=null, type=NODE_JOINED, tstamp=1540383624974], nodeId=bd8d192b, evt=NODE_JOINED], added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=true, hash=1924369423], init=true, lastVer=null, partReleaseFut=null, exchActions=ExchangeActions [startCaches=null, stopCaches=null, startGrps=[], stopGrps=[], resetParts=null, stateChangeRequest=null], affChangeMsg=null, initTs=1540383625027, centralizedAff=false, forceAffReassignment=false, changeGlobalStateE=null, done=false, state=CLIENT, evtLatch=0, remaining=[1943a94c-caa2-4082-a04b-a06913489a60, 09007637-a72e-4a68-940a-10392309eb64], super=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=529982688]]] [08:21:32,021][INFO][ignite-update-notifier-timer][GridUpdateNotifier] Update status is not available. [08:21:35,186][INFO][ignite-update-notifier-timer][GridUpdateNotifier] Update status is not available. [08:21:45,229][WARNING][Test worker][GridCachePartitionExchangeManager] Still waiting for initial partition map exchange [fut=GridDhtPartitionsExchangeFuture [firstDiscoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=bd8d192b-bea3-4660-88c1-9949488f3249, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=7, intOrder=0, lastExchangeTime=1540383623902, loc=true, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], topVer=7, nodeId8=bd8d192b, msg=null, type=NODE_JOINED, tstamp=1540383624974], crd=TcpDiscoveryNode [id=09007637-a72e-4a68-940a-10392309eb64, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[V7ABDMZP101404.prudential.com/48.159.12.168:47501, /0:0:0:0:0:0:0:1:47501, /127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1540383624962, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=false], exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=bd8d192b-bea3-4660-88c1-9949488f3249, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=7, intOrder=0, lastExchangeTime=1540383623902, loc=true, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], topVer=7, nodeId8=bd8d192b, msg=null, type=NODE_JOINED, tstamp=1540383624974], nodeId=bd8d192b, evt=NODE_JOINED], added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=true, hash=1924369423], init=true, lastVer=null, partReleaseFut=null, exchActions=ExchangeActions [startCaches=null, stopCaches=null, startGrps=[], stopGrps=[], resetParts=null, stateChangeRequest=null], affChangeMsg=null, initTs=1540383625027, centralizedAff=false, forceAffReassignment=false, changeGlobalStateE=null, done=false, state=CLIENT, evtLatch=0, remaining=[1943a94c-caa2-4082-a04b-a06913489a60, 09007637-a72e-4a68-940a-10392309eb64], super=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=529982688]]] [08:21:55,325][WARNING][exchange-worker-#212][diagnostic] Failed to wait for partition map exchange [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], node=bd8d192b-bea3-4660-88c1-9949488f3249]. Consider changing TransactionConfiguration.txTimeoutOnPartitionMapSynchronization to non default value to avoid this message. Dumping pending objects that might be the cause: [08:21:55,332][WARNING][exchange-worker-#212][diagnostic] Ready affinity version: AffinityTopologyVersion [topVer=-1, minorTopVer=0] [08:21:55,334][WARNING][exchange-worker-#212][diagnostic] Last exchange future: GridDhtPartitionsExchangeFuture [firstDiscoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=bd8d192b-bea3-4660-88c1-9949488f3249, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=7, intOrder=0, lastExchangeTime=1540383623902, loc=true, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], topVer=7, nodeId8=bd8d192b, msg=null, type=NODE_JOINED, tstamp=1540383624974], crd=TcpDiscoveryNode [id=09007637-a72e-4a68-940a-10392309eb64, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[V7ABDMZP101404.prudential.com/48.159.12.168:47501, /0:0:0:0:0:0:0:1:47501, /127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1540383624962, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=false], exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=bd8d192b-bea3-4660-88c1-9949488f3249, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=7, intOrder=0, lastExchangeTime=1540383623902, loc=true, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], topVer=7, nodeId8=bd8d192b, msg=null, type=NODE_JOINED, tstamp=1540383624974], nodeId=bd8d192b, evt=NODE_JOINED], added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=true, hash=1924369423], init=true, lastVer=null, partReleaseFut=null, exchActions=ExchangeActions [startCaches=null, stopCaches=null, startGrps=[], stopGrps=[], resetParts=null, stateChangeRequest=null], affChangeMsg=null, initTs=1540383625027, centralizedAff=false, forceAffReassignment=false, changeGlobalStateE=null, done=false, state=CLIENT, evtLatch=0, remaining=[1943a94c-caa2-4082-a04b-a06913489a60, 09007637-a72e-4a68-940a-10392309eb64], super=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=529982688]] [08:21:55,345][WARNING][exchange-worker-#212][GridCachePartitionExchangeManager] First 10 pending exchange futures [total=0] [08:21:55,347][WARNING][exchange-worker-#212][diagnostic] Last 10 exchange futures (total: 1): [08:21:55,349][WARNING][exchange-worker-#212][diagnostic] >>> GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=bd8d192b-bea3-4660-88c1-9949488f3249, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=7, intOrder=0, lastExchangeTime=1540383623902, loc=true, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], done=false] [08:21:55,350][WARNING][exchange-worker-#212][diagnostic] Latch manager state: ExchangeLatchManager [serverLatches={}, clientLatches={}] [08:21:55,351][WARNING][exchange-worker-#212][diagnostic] Pending transactions: [08:21:55,352][WARNING][exchange-worker-#212][diagnostic] Pending explicit locks: [08:21:55,353][WARNING][exchange-worker-#212][diagnostic] Pending cache futures: [08:21:55,354][WARNING][exchange-worker-#212][diagnostic] Pending atomic cache futures: [08:21:55,355][WARNING][exchange-worker-#212][diagnostic] Pending data streamer futures: [08:21:55,356][WARNING][exchange-worker-#212][diagnostic] Pending transaction deadlock detection futures: [08:21:55,378][INFO][grid-nio-worker-tcp-comm-3-#205][diagnostic] Exchange future waiting for coordinator response [crd=09007637-a72e-4a68-940a-10392309eb64, topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0]] Remote node information: General node info [id=09007637-a72e-4a68-940a-10392309eb64, client=false, discoTopVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], time=08:21:55.356] Partitions exchange info [readyVer=AffinityTopologyVersion [topVer=6, minorTopVer=0]] Last initialized exchange future: GridDhtPartitionsExchangeFuture [firstDiscoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=22efeb8f-176c-455a-b86a-9435f0bf0736, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=5, intOrder=4, lastExchangeTime=1540383621546, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], topVer=6, nodeId8=09007637, msg=Node left: TcpDiscoveryNode [id=22efeb8f-176c-455a-b86a-9435f0bf0736, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=5, intOrder=4, lastExchangeTime=1540383621546, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], type=NODE_LEFT, tstamp=1540383623258], crd=TcpDiscoveryNode [id=09007637-a72e-4a68-940a-10392309eb64, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[V7ABDMZP101404.prudential.com/48.159.12.168:47501, /0:0:0:0:0:0:0:1:47501, /127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1540383715148, loc=true, ver=2.5.0#20180523-sha1:86e110c7, isClient=false], exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=22efeb8f-176c-455a-b86a-9435f0bf0736, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=5, intOrder=4, lastExchangeTime=1540383621546, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], topVer=6, nodeId8=09007637, msg=Node left: TcpDiscoveryNode [id=22efeb8f-176c-455a-b86a-9435f0bf0736, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=5, intOrder=4, lastExchangeTime=1540383621546, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], type=NODE_LEFT, tstamp=1540383623258], nodeId=22efeb8f, evt=NODE_LEFT], added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=true, hash=760946415], init=false, lastVer=null, partReleaseFut=null, exchActions=null, affChangeMsg=null, initTs=1540383623279, centralizedAff=false, forceAffReassignment=false, changeGlobalStateE=null, done=true, state=CRD, evtLatch=0, remaining=[1943a94c-caa2-4082-a04b-a06913489a60], super=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=AffinityTopologyVersion [topVer=6, minorTopVer=0], hash=605301579]] Communication SPI statistics [rmtNode=bd8d192b-bea3-4660-88c1-9949488f3249] Communication SPI recovery descriptors: [key=ConnectionKey [nodeId=bd8d192b-bea3-4660-88c1-9949488f3249, idx=0, connCnt=0, dummy=false], msgsSent=3, msgsAckedByRmt=0, msgsRcvd=5, lastAcked=0, reserveCnt=1, descIdHash=689424991] Communication SPI clients: [node=bd8d192b-bea3-4660-88c1-9949488f3249, client=GridTcpNioCommunicationClient [ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=0, bytesRcvd=3546, bytesSent=20179, bytesRcvd0=853, bytesSent0=0, select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-0, igniteInstanceName=integration_tests, finished=false, hashCode=1400285321, interrupted=false, runner=grid-nio-worker-tcp-comm-0-#25%integration_tests%]]], writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], inRecovery=GridNioRecoveryDescriptor [acked=0, resendCnt=0, rcvCnt=5, sentCnt=3, reserved=true, lastAck=0, nodeLeft=false, node=TcpDiscoveryNode [id=bd8d192b-bea3-4660-88c1-9949488f3249, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=7, intOrder=5, lastExchangeTime=1540383624919, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], connected=true, connectCnt=0, queueLimit=4096, reserveCnt=1, pairedConnections=false], outRecovery=GridNioRecoveryDescriptor [acked=0, resendCnt=0, rcvCnt=5, sentCnt=3, reserved=true, lastAck=0, nodeLeft=false, node=TcpDiscoveryNode [id=bd8d192b-bea3-4660-88c1-9949488f3249, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=7, intOrder=5, lastExchangeTime=1540383624919, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], connected=true, connectCnt=0, queueLimit=4096, reserveCnt=1, pairedConnections=false], super=GridNioSessionImpl [locAddr=/127.0.0.1:47101, rmtAddr=/127.0.0.1:52997, createTime=1540383625057, closeTime=0, bytesSent=19781, bytesRcvd=3522, bytesSent0=0, bytesRcvd0=853, sndSchedTime=1540383625057, lastSndTime=1540383675312, lastRcvTime=1540383715356, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=org.apache.ignite.internal.util.nio.GridDirectParser@1fff65b2, directMode=true], GridConnectionBytesVerifyFilter], accepted=true]], super=GridAbstractCommunicationClient [lastUsed=1540383625057, closed=false, connIdx=0]]] NIO sessions statistics: >> Selector info [idx=0, keysCnt=1, bytesRcvd=3546, bytesRcvd0=853, >> bytesSent=20179, bytesSent0=0] Connection info [in=true, rmtAddr=/127.0.0.1:52997, locAddr=/127.0.0.1:47101, msgsSent=3, msgsAckedByRmt=0, descIdHash=689424991, unackedMsgs=[IgniteDiagnosticMessage, IgniteDiagnosticMessage, IgniteDiagnosticMessage], msgsRcvd=5, lastAcked=0, descIdHash=689424991, bytesRcvd=3522, bytesRcvd0=853, bytesSent=19781, bytesSent0=0, opQueueSize=0] Exchange future: GridDhtPartitionsExchangeFuture [firstDiscoEvt=null, crd=null, exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], discoEvt=null, nodeId=bd8d192b, evt=NODE_JOINED], added=false, initFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1896978384], init=false, lastVer=null, partReleaseFut=null, exchActions=null, affChangeMsg=null, initTs=0, centralizedAff=false, forceAffReassignment=false, changeGlobalStateE=null, done=false, state=null, evtLatch=1, remaining=[], super=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=155967337]] Local communication statistics: Communication SPI statistics [rmtNode=09007637-a72e-4a68-940a-10392309eb64] Communication SPI recovery descriptors: [key=ConnectionKey [nodeId=09007637-a72e-4a68-940a-10392309eb64, idx=0, connCnt=-1, dummy=false], msgsSent=5, msgsAckedByRmt=0, msgsRcvd=4, lastAcked=0, reserveCnt=1, descIdHash=472467749] Communication SPI clients: [node=09007637-a72e-4a68-940a-10392309eb64, client=GridTcpNioCommunicationClient [ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=0, bytesRcvd=26398, bytesSent=3480, bytesRcvd0=6645, bytesSent0=853, select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-0, igniteInstanceName=null, finished=false, hashCode=2080814698, interrupted=false, runner=grid-nio-worker-tcp-comm-0-#202]]], writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], inRecovery=GridNioRecoveryDescriptor [acked=0, resendCnt=0, rcvCnt=4, sentCnt=5, reserved=true, lastAck=0, nodeLeft=false, node=TcpDiscoveryNode [id=09007637-a72e-4a68-940a-10392309eb64, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[V7ABDMZP101404.prudential.com/48.159.12.168:47501, /0:0:0:0:0:0:0:1:47501, /127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1540383624962, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=false], connected=false, connectCnt=1, queueLimit=4096, reserveCnt=1, pairedConnections=false], outRecovery=GridNioRecoveryDescriptor [acked=0, resendCnt=0, rcvCnt=4, sentCnt=5, reserved=true, lastAck=0, nodeLeft=false, node=TcpDiscoveryNode [id=09007637-a72e-4a68-940a-10392309eb64, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[V7ABDMZP101404.prudential.com/48.159.12.168:47501, /0:0:0:0:0:0:0:1:47501, /127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1540383624962, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=false], connected=false, connectCnt=1, queueLimit=4096, reserveCnt=1, pairedConnections=false], super=GridNioSessionImpl [locAddr=/127.0.0.1:52997, rmtAddr=/127.0.0.1:47101, createTime=1540383625060, closeTime=0, bytesSent=3480, bytesRcvd=26398, bytesSent0=853, bytesRcvd0=6645, sndSchedTime=1540383625060, lastSndTime=1540383715354, lastRcvTime=1540383715365, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=org.apache.ignite.internal.util.nio.GridDirectParser@2ab0a278, directMode=true], GridConnectionBytesVerifyFilter], accepted=false]], super=GridAbstractCommunicationClient [lastUsed=1540383625060, closed=false, connIdx=0]]] NIO sessions statistics: >> Selector info [idx=0, keysCnt=1, bytesRcvd=26398, bytesRcvd0=6645, >> bytesSent=3480, bytesSent0=853] Connection info [in=false, rmtAddr=/127.0.0.1:47101, locAddr=/127.0.0.1:52997, msgsSent=5, msgsAckedByRmt=0, descIdHash=472467749, unackedMsgs=[GridDhtPartitionsSingleMessage, IgniteDiagnosticMessage, IgniteDiagnosticMessage, IgniteDiagnosticMessage, IgniteDiagnosticMessage], msgsRcvd=4, lastAcked=0, descIdHash=472467749, bytesRcvd=26398, bytesRcvd0=6645, bytesSent=3480, bytesSent0=853, opQueueSize=0] [08:22:05,234][WARNING][Test worker][GridCachePartitionExchangeManager] Still waiting for initial partition map exchange [fut=GridDhtPartitionsExchangeFuture [firstDiscoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=bd8d192b-bea3-4660-88c1-9949488f3249, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=7, intOrder=0, lastExchangeTime=1540383623902, loc=true, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], topVer=7, nodeId8=bd8d192b, msg=null, type=NODE_JOINED, tstamp=1540383624974], crd=TcpDiscoveryNode [id=09007637-a72e-4a68-940a-10392309eb64, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[V7ABDMZP101404.prudential.com/48.159.12.168:47501, /0:0:0:0:0:0:0:1:47501, /127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1540383624962, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=false], exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=bd8d192b-bea3-4660-88c1-9949488f3249, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=7, intOrder=0, lastExchangeTime=1540383623902, loc=true, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], topVer=7, nodeId8=bd8d192b, msg=null, type=NODE_JOINED, tstamp=1540383624974], nodeId=bd8d192b, evt=NODE_JOINED], added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=true, hash=1924369423], init=true, lastVer=null, partReleaseFut=null, exchActions=ExchangeActions [startCaches=null, stopCaches=null, startGrps=[], stopGrps=[], resetParts=null, stateChangeRequest=null], affChangeMsg=null, initTs=1540383625027, centralizedAff=false, forceAffReassignment=false, changeGlobalStateE=null, done=false, state=CLIENT, evtLatch=0, remaining=[1943a94c-caa2-4082-a04b-a06913489a60, 09007637-a72e-4a68-940a-10392309eb64], super=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=529982688]]] [08:22:25,352][WARNING][Test worker][GridCachePartitionExchangeManager] Still waiting for initial partition map exchange [fut=GridDhtPartitionsExchangeFuture [firstDiscoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=bd8d192b-bea3-4660-88c1-9949488f3249, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=7, intOrder=0, lastExchangeTime=1540383623902, loc=true, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], topVer=7, nodeId8=bd8d192b, msg=null, type=NODE_JOINED, tstamp=1540383624974], crd=TcpDiscoveryNode [id=09007637-a72e-4a68-940a-10392309eb64, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[V7ABDMZP101404.prudential.com/48.159.12.168:47501, /0:0:0:0:0:0:0:1:47501, /127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1540383624962, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=false], exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=bd8d192b-bea3-4660-88c1-9949488f3249, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=7, intOrder=0, lastExchangeTime=1540383623902, loc=true, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], topVer=7, nodeId8=bd8d192b, msg=null, type=NODE_JOINED, tstamp=1540383624974], nodeId=bd8d192b, evt=NODE_JOINED], added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=true, hash=1924369423], init=true, lastVer=null, partReleaseFut=null, exchActions=ExchangeActions [startCaches=null, stopCaches=null, startGrps=[], stopGrps=[], resetParts=null, stateChangeRequest=null], affChangeMsg=null, initTs=1540383625027, centralizedAff=false, forceAffReassignment=false, changeGlobalStateE=null, done=false, state=CLIENT, evtLatch=0, remaining=[1943a94c-caa2-4082-a04b-a06913489a60, 09007637-a72e-4a68-940a-10392309eb64], super=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=529982688]]] [08:22:45,358][WARNING][Test worker][GridCachePartitionExchangeManager] Still waiting for initial partition map exchange [fut=GridDhtPartitionsExchangeFuture [firstDiscoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=bd8d192b-bea3-4660-88c1-9949488f3249, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=7, intOrder=0, lastExchangeTime=1540383623902, loc=true, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], topVer=7, nodeId8=bd8d192b, msg=null, type=NODE_JOINED, tstamp=1540383624974], crd=TcpDiscoveryNode [id=09007637-a72e-4a68-940a-10392309eb64, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[V7ABDMZP101404.prudential.com/48.159.12.168:47501, /0:0:0:0:0:0:0:1:47501, /127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1540383624962, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=false], exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=bd8d192b-bea3-4660-88c1-9949488f3249, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=7, intOrder=0, lastExchangeTime=1540383623902, loc=true, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], topVer=7, nodeId8=bd8d192b, msg=null, type=NODE_JOINED, tstamp=1540383624974], nodeId=bd8d192b, evt=NODE_JOINED], added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=true, hash=1924369423], init=true, lastVer=null, partReleaseFut=null, exchActions=ExchangeActions [startCaches=null, stopCaches=null, startGrps=[], stopGrps=[], resetParts=null, stateChangeRequest=null], affChangeMsg=null, initTs=1540383625027, centralizedAff=false, forceAffReassignment=false, changeGlobalStateE=null, done=false, state=CLIENT, evtLatch=0, remaining=[1943a94c-caa2-4082-a04b-a06913489a60, 09007637-a72e-4a68-940a-10392309eb64], super=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=529982688]]] [08:23:05,364][WARNING][Test worker][GridCachePartitionExchangeManager] Still waiting for initial partition map exchange [fut=GridDhtPartitionsExchangeFuture [firstDiscoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=bd8d192b-bea3-4660-88c1-9949488f3249, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=7, intOrder=0, lastExchangeTime=1540383623902, loc=true, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], topVer=7, nodeId8=bd8d192b, msg=null, type=NODE_JOINED, tstamp=1540383624974], crd=TcpDiscoveryNode [id=09007637-a72e-4a68-940a-10392309eb64, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[V7ABDMZP101404.prudential.com/48.159.12.168:47501, /0:0:0:0:0:0:0:1:47501, /127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1540383624962, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=false], exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=bd8d192b-bea3-4660-88c1-9949488f3249, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=7, intOrder=0, lastExchangeTime=1540383623902, loc=true, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], topVer=7, nodeId8=bd8d192b, msg=null, type=NODE_JOINED, tstamp=1540383624974], nodeId=bd8d192b, evt=NODE_JOINED], added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=true, hash=1924369423], init=true, lastVer=null, partReleaseFut=null, exchActions=ExchangeActions [startCaches=null, stopCaches=null, startGrps=[], stopGrps=[], resetParts=null, stateChangeRequest=null], affChangeMsg=null, initTs=1540383625027, centralizedAff=false, forceAffReassignment=false, changeGlobalStateE=null, done=false, state=CLIENT, evtLatch=0, remaining=[1943a94c-caa2-4082-a04b-a06913489a60, 09007637-a72e-4a68-940a-10392309eb64], super=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=529982688]]] [08:23:15,378][WARNING][exchange-worker-#212][diagnostic] Failed to wait for partition map exchange [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], node=bd8d192b-bea3-4660-88c1-9949488f3249]. Consider changing TransactionConfiguration.txTimeoutOnPartitionMapSynchronization to non default value to avoid this message. Dumping pending objects that might be the cause: [08:23:15,379][WARNING][exchange-worker-#212][diagnostic] Ready affinity version: AffinityTopologyVersion [topVer=-1, minorTopVer=0] [08:23:15,381][WARNING][exchange-worker-#212][diagnostic] Last exchange future: GridDhtPartitionsExchangeFuture [firstDiscoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=bd8d192b-bea3-4660-88c1-9949488f3249, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=7, intOrder=0, lastExchangeTime=1540383623902, loc=true, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], topVer=7, nodeId8=bd8d192b, msg=null, type=NODE_JOINED, tstamp=1540383624974], crd=TcpDiscoveryNode [id=09007637-a72e-4a68-940a-10392309eb64, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[V7ABDMZP101404.prudential.com/48.159.12.168:47501, /0:0:0:0:0:0:0:1:47501, /127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1540383624962, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=false], exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=bd8d192b-bea3-4660-88c1-9949488f3249, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=7, intOrder=0, lastExchangeTime=1540383623902, loc=true, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], topVer=7, nodeId8=bd8d192b, msg=null, type=NODE_JOINED, tstamp=1540383624974], nodeId=bd8d192b, evt=NODE_JOINED], added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=true, hash=1924369423], init=true, lastVer=null, partReleaseFut=null, exchActions=ExchangeActions [startCaches=null, stopCaches=null, startGrps=[], stopGrps=[], resetParts=null, stateChangeRequest=null], affChangeMsg=null, initTs=1540383625027, centralizedAff=false, forceAffReassignment=false, changeGlobalStateE=null, done=false, state=CLIENT, evtLatch=0, remaining=[1943a94c-caa2-4082-a04b-a06913489a60, 09007637-a72e-4a68-940a-10392309eb64], super=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=529982688]] [08:23:15,386][WARNING][exchange-worker-#212][GridCachePartitionExchangeManager] First 10 pending exchange futures [total=0] [08:23:15,387][WARNING][exchange-worker-#212][diagnostic] Last 10 exchange futures (total: 1): [08:23:15,388][WARNING][exchange-worker-#212][diagnostic] >>> GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=bd8d192b-bea3-4660-88c1-9949488f3249, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=7, intOrder=0, lastExchangeTime=1540383623902, loc=true, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], done=false] [08:23:15,388][WARNING][exchange-worker-#212][diagnostic] Latch manager state: ExchangeLatchManager [serverLatches={}, clientLatches={}] [08:23:15,389][WARNING][exchange-worker-#212][diagnostic] Pending transactions: [08:23:15,389][WARNING][exchange-worker-#212][diagnostic] Pending explicit locks: [08:23:15,390][WARNING][exchange-worker-#212][diagnostic] Pending cache futures: [08:23:15,391][WARNING][exchange-worker-#212][diagnostic] Pending atomic cache futures: [08:23:15,391][WARNING][exchange-worker-#212][diagnostic] Pending data streamer futures: [08:23:15,392][WARNING][exchange-worker-#212][diagnostic] Pending transaction deadlock detection futures: [08:23:15,422][INFO][grid-nio-worker-tcp-comm-3-#205][diagnostic] Exchange future waiting for coordinator response [crd=09007637-a72e-4a68-940a-10392309eb64, topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0]] Remote node information: General node info [id=09007637-a72e-4a68-940a-10392309eb64, client=false, discoTopVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], time=08:23:15.396] Partitions exchange info [readyVer=AffinityTopologyVersion [topVer=6, minorTopVer=0]] Last initialized exchange future: GridDhtPartitionsExchangeFuture [firstDiscoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=22efeb8f-176c-455a-b86a-9435f0bf0736, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=5, intOrder=4, lastExchangeTime=1540383621546, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], topVer=6, nodeId8=09007637, msg=Node left: TcpDiscoveryNode [id=22efeb8f-176c-455a-b86a-9435f0bf0736, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=5, intOrder=4, lastExchangeTime=1540383621546, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], type=NODE_LEFT, tstamp=1540383623258], crd=TcpDiscoveryNode [id=09007637-a72e-4a68-940a-10392309eb64, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[V7ABDMZP101404.prudential.com/48.159.12.168:47501, /0:0:0:0:0:0:0:1:47501, /127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1540383793387, loc=true, ver=2.5.0#20180523-sha1:86e110c7, isClient=false], exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=22efeb8f-176c-455a-b86a-9435f0bf0736, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=5, intOrder=4, lastExchangeTime=1540383621546, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], topVer=6, nodeId8=09007637, msg=Node left: TcpDiscoveryNode [id=22efeb8f-176c-455a-b86a-9435f0bf0736, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=5, intOrder=4, lastExchangeTime=1540383621546, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], type=NODE_LEFT, tstamp=1540383623258], nodeId=22efeb8f, evt=NODE_LEFT], added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=true, hash=760946415], init=false, lastVer=null, partReleaseFut=null, exchActions=null, affChangeMsg=null, initTs=1540383623279, centralizedAff=false, forceAffReassignment=false, changeGlobalStateE=null, done=true, state=CRD, evtLatch=0, remaining=[1943a94c-caa2-4082-a04b-a06913489a60], super=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=AffinityTopologyVersion [topVer=6, minorTopVer=0], hash=605301579]] Communication SPI statistics [rmtNode=bd8d192b-bea3-4660-88c1-9949488f3249] Communication SPI recovery descriptors: [key=ConnectionKey [nodeId=bd8d192b-bea3-4660-88c1-9949488f3249, idx=0, connCnt=0, dummy=false], msgsSent=4, msgsAckedByRmt=0, msgsRcvd=6, lastAcked=0, reserveCnt=1, descIdHash=689424991] Communication SPI clients: [node=bd8d192b-bea3-4660-88c1-9949488f3249, client=GridTcpNioCommunicationClient [ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=0, bytesRcvd=4399, bytesSent=26824, bytesRcvd0=853, bytesSent0=0, select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-0, igniteInstanceName=integration_tests, finished=false, hashCode=1400285321, interrupted=false, runner=grid-nio-worker-tcp-comm-0-#25%integration_tests%]]], writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], inRecovery=GridNioRecoveryDescriptor [acked=0, resendCnt=0, rcvCnt=6, sentCnt=4, reserved=true, lastAck=0, nodeLeft=false, node=TcpDiscoveryNode [id=bd8d192b-bea3-4660-88c1-9949488f3249, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=7, intOrder=5, lastExchangeTime=1540383624919, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], connected=true, connectCnt=0, queueLimit=4096, reserveCnt=1, pairedConnections=false], outRecovery=GridNioRecoveryDescriptor [acked=0, resendCnt=0, rcvCnt=6, sentCnt=4, reserved=true, lastAck=0, nodeLeft=false, node=TcpDiscoveryNode [id=bd8d192b-bea3-4660-88c1-9949488f3249, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=7, intOrder=5, lastExchangeTime=1540383624919, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], connected=true, connectCnt=0, queueLimit=4096, reserveCnt=1, pairedConnections=false], super=GridNioSessionImpl [locAddr=/127.0.0.1:47101, rmtAddr=/127.0.0.1:52997, createTime=1540383625057, closeTime=0, bytesSent=26426, bytesRcvd=4375, bytesSent0=0, bytesRcvd0=853, sndSchedTime=1540383625057, lastSndTime=1540383715356, lastRcvTime=1540383795385, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=org.apache.ignite.internal.util.nio.GridDirectParser@1fff65b2, directMode=true], GridConnectionBytesVerifyFilter], accepted=true]], super=GridAbstractCommunicationClient [lastUsed=1540383625057, closed=false, connIdx=0]]] NIO sessions statistics: >> Selector info [idx=0, keysCnt=1, bytesRcvd=4399, bytesRcvd0=853, >> bytesSent=26824, bytesSent0=0] Connection info [in=true, rmtAddr=/127.0.0.1:52997, locAddr=/127.0.0.1:47101, msgsSent=4, msgsAckedByRmt=0, descIdHash=689424991, unackedMsgs=[IgniteDiagnosticMessage, IgniteDiagnosticMessage, IgniteDiagnosticMessage, IgniteDiagnosticMessage], msgsRcvd=6, lastAcked=0, descIdHash=689424991, bytesRcvd=4375, bytesRcvd0=853, bytesSent=26426, bytesSent0=0, opQueueSize=0] Exchange future: GridDhtPartitionsExchangeFuture [firstDiscoEvt=null, crd=null, exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], discoEvt=null, nodeId=bd8d192b, evt=NODE_JOINED], added=false, initFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1896978384], init=false, lastVer=null, partReleaseFut=null, exchActions=null, affChangeMsg=null, initTs=0, centralizedAff=false, forceAffReassignment=false, changeGlobalStateE=null, done=false, state=null, evtLatch=1, remaining=[], super=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=155967337]] Local communication statistics: Communication SPI statistics [rmtNode=09007637-a72e-4a68-940a-10392309eb64] Communication SPI recovery descriptors: [key=ConnectionKey [nodeId=09007637-a72e-4a68-940a-10392309eb64, idx=0, connCnt=-1, dummy=false], msgsSent=6, msgsAckedByRmt=0, msgsRcvd=5, lastAcked=0, reserveCnt=1, descIdHash=472467749] Communication SPI clients: [node=09007637-a72e-4a68-940a-10392309eb64, client=GridTcpNioCommunicationClient [ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=0, bytesRcvd=33068, bytesSent=4333, bytesRcvd0=6670, bytesSent0=853, select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-0, igniteInstanceName=null, finished=false, hashCode=2080814698, interrupted=false, runner=grid-nio-worker-tcp-comm-0-#202]]], writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], inRecovery=GridNioRecoveryDescriptor [acked=0, resendCnt=0, rcvCnt=5, sentCnt=6, reserved=true, lastAck=0, nodeLeft=false, node=TcpDiscoveryNode [id=09007637-a72e-4a68-940a-10392309eb64, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[V7ABDMZP101404.prudential.com/48.159.12.168:47501, /0:0:0:0:0:0:0:1:47501, /127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1540383624962, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=false], connected=false, connectCnt=1, queueLimit=4096, reserveCnt=1, pairedConnections=false], outRecovery=GridNioRecoveryDescriptor [acked=0, resendCnt=0, rcvCnt=5, sentCnt=6, reserved=true, lastAck=0, nodeLeft=false, node=TcpDiscoveryNode [id=09007637-a72e-4a68-940a-10392309eb64, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[V7ABDMZP101404.prudential.com/48.159.12.168:47501, /0:0:0:0:0:0:0:1:47501, /127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1540383624962, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=false], connected=false, connectCnt=1, queueLimit=4096, reserveCnt=1, pairedConnections=false], super=GridNioSessionImpl [locAddr=/127.0.0.1:52997, rmtAddr=/127.0.0.1:47101, createTime=1540383625060, closeTime=0, bytesSent=4333, bytesRcvd=33068, bytesSent0=853, bytesRcvd0=6670, sndSchedTime=1540383625060, lastSndTime=1540383795386, lastRcvTime=1540383795396, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=org.apache.ignite.internal.util.nio.GridDirectParser@2ab0a278, directMode=true], GridConnectionBytesVerifyFilter], accepted=false]], super=GridAbstractCommunicationClient [lastUsed=1540383625060, closed=false, connIdx=0]]] NIO sessions statistics: >> Selector info [idx=0, keysCnt=1, bytesRcvd=33068, bytesRcvd0=6670, >> bytesSent=4333, bytesSent0=853] Connection info [in=false, rmtAddr=/127.0.0.1:47101, locAddr=/127.0.0.1:52997, msgsSent=6, msgsAckedByRmt=0, descIdHash=472467749, unackedMsgs=[GridDhtPartitionsSingleMessage, IgniteDiagnosticMessage, IgniteDiagnosticMessage, IgniteDiagnosticMessage, IgniteDiagnosticMessage], msgsRcvd=5, lastAcked=0, descIdHash=472467749, bytesRcvd=33068, bytesRcvd0=6670, bytesSent=4333, bytesSent0=853, opQueueSize=0] [08:23:25,369][WARNING][Test worker][GridCachePartitionExchangeManager] Still waiting for initial partition map exchange [fut=GridDhtPartitionsExchangeFuture [firstDiscoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=bd8d192b-bea3-4660-88c1-9949488f3249, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=7, intOrder=0, lastExchangeTime=1540383623902, loc=true, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], topVer=7, nodeId8=bd8d192b, msg=null, type=NODE_JOINED, tstamp=1540383624974], crd=TcpDiscoveryNode [id=09007637-a72e-4a68-940a-10392309eb64, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[V7ABDMZP101404.prudential.com/48.159.12.168:47501, /0:0:0:0:0:0:0:1:47501, /127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1540383624962, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=false], exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=bd8d192b-bea3-4660-88c1-9949488f3249, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=7, intOrder=0, lastExchangeTime=1540383623902, loc=true, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], topVer=7, nodeId8=bd8d192b, msg=null, type=NODE_JOINED, tstamp=1540383624974], nodeId=bd8d192b, evt=NODE_JOINED], added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=true, hash=1924369423], init=true, lastVer=null, partReleaseFut=null, exchActions=ExchangeActions [startCaches=null, stopCaches=null, startGrps=[], stopGrps=[], resetParts=null, stateChangeRequest=null], affChangeMsg=null, initTs=1540383625027, centralizedAff=false, forceAffReassignment=false, changeGlobalStateE=null, done=false, state=CLIENT, evtLatch=0, remaining=[1943a94c-caa2-4082-a04b-a06913489a60, 09007637-a72e-4a68-940a-10392309eb64], super=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=529982688]]] [08:23:45,372][WARNING][Test worker][GridCachePartitionExchangeManager] Still waiting for initial partition map exchange [fut=GridDhtPartitionsExchangeFuture [firstDiscoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=bd8d192b-bea3-4660-88c1-9949488f3249, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=7, intOrder=0, lastExchangeTime=1540383623902, loc=true, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], topVer=7, nodeId8=bd8d192b, msg=null, type=NODE_JOINED, tstamp=1540383624974], crd=TcpDiscoveryNode [id=09007637-a72e-4a68-940a-10392309eb64, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[V7ABDMZP101404.prudential.com/48.159.12.168:47501, /0:0:0:0:0:0:0:1:47501, /127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1540383624962, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=false], exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=bd8d192b-bea3-4660-88c1-9949488f3249, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=7, intOrder=0, lastExchangeTime=1540383623902, loc=true, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], topVer=7, nodeId8=bd8d192b, msg=null, type=NODE_JOINED, tstamp=1540383624974], nodeId=bd8d192b, evt=NODE_JOINED], added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=true, hash=1924369423], init=true, lastVer=null, partReleaseFut=null, exchActions=ExchangeActions [startCaches=null, stopCaches=null, startGrps=[], stopGrps=[], resetParts=null, stateChangeRequest=null], affChangeMsg=null, initTs=1540383625027, centralizedAff=false, forceAffReassignment=false, changeGlobalStateE=null, done=false, state=CLIENT, evtLatch=0, remaining=[1943a94c-caa2-4082-a04b-a06913489a60, 09007637-a72e-4a68-940a-10392309eb64], super=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=529982688]]] [08:24:05,378][WARNING][Test worker][GridCachePartitionExchangeManager] Still waiting for initial partition map exchange [fut=GridDhtPartitionsExchangeFuture [firstDiscoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=bd8d192b-bea3-4660-88c1-9949488f3249, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=7, intOrder=0, lastExchangeTime=1540383623902, loc=true, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], topVer=7, nodeId8=bd8d192b, msg=null, type=NODE_JOINED, tstamp=1540383624974], crd=TcpDiscoveryNode [id=09007637-a72e-4a68-940a-10392309eb64, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[V7ABDMZP101404.prudential.com/48.159.12.168:47501, /0:0:0:0:0:0:0:1:47501, /127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1540383624962, loc=false, ver=2.5.0#20180523-sha1:86e110c7, isClient=false], exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=bd8d192b-bea3-4660-88c1-9949488f3249, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 48.159.12.168], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, V7ABDMZP101404.prudential.com/48.159.12.168:0], discPort=0, order=7, intOrder=0, lastExchangeTime=1540383623902, loc=true, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], topVer=7, nodeId8=bd8d192b, msg=null, type=NODE_JOINED, tstamp=1540383624974], nodeId=bd8d192b, evt=NODE_JOINED], added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=true, hash=1924369423], init=true, lastVer=null, partReleaseFut=null, exchActions=ExchangeActions [startCaches=null, stopCaches=null, startGrps=[], stopGrps=[], resetParts=null, stateChangeRequest=null], affChangeMsg=null, initTs=1540383625027, centralizedAff=false, forceAffReassignment=false, changeGlobalStateE=null, done=false, state=CLIENT, evtLatch=0, remaining=[1943a94c-caa2-4082-a04b-a06913489a60, 09007637-a72e-4a68-940a-10392309eb64], super=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=529982688]]] [08:24:13,852][INFO][sys-#218][GridDhtPartitionsExchangeFuture] Received full message, will finish exchange [node=09007637-a72e-4a68-940a-10392309eb64, resVer=AffinityTopologyVersion [topVer=7, minorTopVer=0]] [08:24:13,855][INFO][sys-#218][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], err=null] [08:24:13,869][INFO][Test worker][IgniteKernal] Performance suggestions for grid (fix if possible) [08:24:13,876][INFO][Test worker][IgniteKernal] To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true [08:24:13,886][INFO][Test worker][IgniteKernal] ^-- Enable G1 Garbage Collector (add '-XX:+UseG1GC' to JVM options) [08:24:13,887][INFO][grid-nio-worker-tcp-comm-1-#203][TcpCommunicationSpi] Established outgoing communication connection [locAddr=/127.0.0.1:53101, rmtAddr=/127.0.0.1:47102] [08:24:13,891][INFO][Test worker][IgniteKernal] ^-- Set max direct memory size if getting 'OOME: Direct buffer memory' (add '-XX:MaxDirectMemorySize=<size>[g|G|m|M|k|K]' to JVM options) [08:24:13,892][INFO][Test worker][IgniteKernal] ^-- Disable processing of calls to System.gc() (add '-XX:+DisableExplicitGC' to JVM options) [08:24:13,893][INFO][Test worker][IgniteKernal] ^-- Disable assertions (remove '-ea' from JVM options) [08:24:13,894][INFO][Test worker][IgniteKernal] Refer to this page for more performance suggestions: https://apacheignite.readme.io/docs/jvm-and-system-tuning [08:24:13,900][INFO][Test worker][IgniteKernal] [08:24:13,907][INFO][Test worker][IgniteKernal] To start Console Management & Monitoring run ignitevisorcmd.{sh|bat} [08:24:13,910][INFO][Test worker][IgniteKernal] [08:24:13,913][INFO][Test worker][IgniteKernal]
