[ 
https://issues.apache.org/jira/browse/IGNITE-837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Artem Shutak updated IGNITE-837:
--------------------------------
    Description: 
IgniteProjectionStartStopRestartSelfTest.testStartFiveNodesInTwoCalls.

{code}

{code}

Default node do not get EVT_NODE_LEFT.
Log:
{noformat}
java.lang.AssertionError: null
    at 
org.apache.ignite.internal.IgniteProjectionStartStopRestartSelfTest.afterTest(IgniteProjectionStartStopRestartSelfTest.java:153)
------- Stdout: -------
[18:49:11,266][INFO ][main][root] >>> Starting test: 
testStartFiveNodesInTwoCalls <<<
[18:49:11,266][INFO ][main][root] Username: teamcity
[18:49:11,266][INFO ][main][root] Password: teamcity
[18:49:11,266][INFO ][main][root] Key path: null
[18:49:11,267][INFO ][main][XmlBeanDefinitionReader] Loading XML bean 
definitions from URL 
[file:/usr/local/BuildAgent/work/c326f6bf1927b309/modules/core/src/test/config/spring-start-nodes.xml]
[18:49:11,293][INFO ][main][GenericApplicationContext] Refreshing 
org.springframework.context.support.GenericApplicationContext@1ed7fbc4: startup 
date [Tue Apr 28 18:49:11 MSK 2015]; root of context hierarchy
[18:49:11,298][INFO ][main][IgniteKernal] 

>>>    __________  ________________  
>>>   /  _/ ___/ |/ /  _/_  __/ __/  
>>>  _/ // (7 7    // /  / / / _/    
>>> /___/\___/_/|_/___/ /_/ /___/   
>>> 
>>> ver. 1.0.3-SNAPSHOT#20150428-sha1:48c12bb7
>>> 2015 Copyright(C) Apache Software Foundation

[18:49:11,298][INFO ][main][IgniteKernal] Config URL: 
file:/usr/local/BuildAgent/work/c326f6bf1927b309/modules/core/src/test/config/spring-start-nodes.xml
[18:49:11,298][INFO ][main][IgniteKernal] Daemon mode: on
[18:49:11,298][INFO ][main][IgniteKernal] OS: Linux 3.13.0-41-generic amd64
[18:49:11,298][INFO ][main][IgniteKernal] OS user: teamcity
[18:49:11,298][INFO ][main][IgniteKernal] Language runtime: Java Platform API 
Specification ver. 1.7
[18:49:11,298][INFO ][main][IgniteKernal] VM information: Java(TM) SE Runtime 
Environment 1.7.0_67-b01 Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 
24.65-b04
[18:49:11,298][INFO ][main][IgniteKernal] VM total memory: 2.7GB
[18:49:11,298][INFO ][main][IgniteKernal] Remote Management [restart: off, 
REST: off, JMX (remote: off)]
[18:49:11,298][INFO ][main][IgniteKernal] 
IGNITE_HOME=/usr/local/BuildAgent/work/c326f6bf1927b309
[18:49:11,298][INFO ][main][IgniteKernal] VM arguments: 
[-DJAVA_HOME=/usr/lib/jvm/java-7-oracle, -DMCAST_GRP=229.23.1.2, 
-Dagent.home.dir=/usr/local/BuildAgent, -Dagent.name=ip_192.168.2.23, 
-Dagent.ownPort=9090, -Dagent.work.dir=/usr/local/BuildAgent/work, 
-Dbuild.number=2086, 
-Dbuild.vcs.number=48c12bb753d86f9fbe20d8fc23672f29b6f790bf, 
-Dbuild.vcs.number.1=48c12bb753d86f9fbe20d8fc23672f29b6f790bf, 
-Dbuild.vcs.number.IgniteApache=48c12bb753d86f9fbe20d8fc23672f29b6f790bf, 
-Dclassworlds.conf=/usr/local/BuildAgent/temp/buildTmp/teamcity.m2.conf, 
-Dcom.jetbrains.maven.watcher.report.file=/usr/local/BuildAgent/temp/buildTmp/maven-build-info.xml,
 -Djava.io.tmpdir=/usr/local/BuildAgent/temp/buildTmp, 
-Dmaven.home=/usr/local/BuildAgent/tools/maven3, 
-Dmaven.repo.local=/home/teamcity/.m2/repository, 
-Dteamcity.agent.cpuBenchmark=724, 
-Dteamcity.agent.dotnet.agent_url=http://localhost:9090/RPC2, 
-Dteamcity.agent.dotnet.build_id=437568, 
-Dteamcity.auth.password=xaeut7b99lFebuZknJOjhFvxx1qMLIIo, 
-Dteamcity.auth.userId=TeamCityBuildId=437568, 
-Dteamcity.build.changedFiles.file=/usr/local/BuildAgent/temp/buildTmp/changedFiles4219226629992489923.txt,
 -Dteamcity.build.checkoutDir=/usr/local/BuildAgent/work/c326f6bf1927b309, 
-Dteamcity.build.id=437568, 
-Dteamcity.build.properties.file=/usr/local/BuildAgent/temp/buildTmp/teamcity.build7100002810635920006.properties,
 -Dteamcity.build.tempDir=/usr/local/BuildAgent/temp/buildTmp, 
-Dteamcity.build.workingDir=/usr/local/BuildAgent/work/c326f6bf1927b309, 
-Dteamcity.buildConfName=Ignite Start Nodes, 
-Dteamcity.buildType.id=Ignite_StartNodes, 
-Dteamcity.configuration.properties.file=/usr/local/BuildAgent/temp/buildTmp/teamcity.config2404423307058673025.properties,
 
-Dteamcity.maven.watcher.home=/usr/local/BuildAgent/plugins/mavenPlugin/maven-watcher,
 -Dteamcity.projectName=Ignite, 
-Dteamcity.runner.properties.file=/usr/local/BuildAgent/temp/buildTmp/teamcity.runner2837394402491896395.properties,
 
-Dteamcity.tests.recentlyFailedTests.file=/usr/local/BuildAgent/temp/buildTmp/testsToRunFirst1902214237945091123.txt,
 -Dteamcity.version=8.1.4 (build 30168), -ea, -XX:MaxPermSize=512m, -Xms1g, 
-Xmx3g, -XX:+HeapDumpOnOutOfMemoryError, -XX:+AggressiveOpts, 
-DIGNITE_TEST_HOME=/usr/local/BuildAgent/work/c326f6bf1927b309, 
-DIGNITE_UPDATE_NOTIFIER=false, -DIGNITE_NO_DISCO_ORDER=true, 
-DIGNITE_QUIET=false]
[18:49:11,298][INFO ][main][IgniteKernal] Configured caches 
['ignite-marshaller-sys-cache', 'ignite-sys-cache', 'ignite-atomics-sys-cache']
[18:49:11,298][INFO ][main][IgniteKernal] 3-rd party licenses can be found at: 
/usr/local/BuildAgent/work/c326f6bf1927b309/libs/licenses
[18:49:11,301][INFO ][main][IgniteKernal] Non-loopback local IPs: 192.168.2.23, 
fe80:0:0:0:96de:80ff:feba:5463%2
[18:49:11,301][INFO ][main][IgniteKernal] Enabled local MACs: 94DE80BA5463
[18:49:11,301][INFO ][main][IgnitePluginProcessor] Configured plugins:
[18:49:11,301][INFO ][main][IgnitePluginProcessor]   ^-- None
[18:49:11,301][INFO ][main][IgnitePluginProcessor] 
[18:49:11,305][INFO ][main][TcpCommunicationSpi] Successfully bound to TCP port 
[port=47100, locHost=/127.0.0.1]
[18:49:11,311][INFO ][main][IgniteKernal] Security status [authentication=off]
[18:49:11,318][INFO ][main][TcpDiscoverySpi] Successfully bound to TCP port 
[port=47500, localHost=/127.0.0.1]
[18:49:11,326][INFO ][main][IgniteKernal] To start Console Management & 
Monitoring run ignitevisorcmd.{sh|bat}
[18:49:11,326][INFO ][main][IgniteKernal] 
[18:49:11,326][INFO ][main][IgniteKernal] 

>>> +-------------------------------------------------------------------------------+
>>> Ignite ver. 
>>> 1.0.3-SNAPSHOT#20150428-sha1:48c12bb753d86f9fbe20d8fc23672f29b6f790bf
>>> +-------------------------------------------------------------------------------+
>>> OS name: Linux 3.13.0-41-generic amd64
>>> CPU(s): 8
>>> Heap: 2.7GB
>>> VM name: 16539@teamcity23
>>> Grid name: null
>>> Local node [ID=F1330264-0901-41B5-A1A8-CDADD1345510, order=1]
>>> Local node addresses: [/127.0.0.1]
>>> Local ports: TCP:47100 TCP:47500 
>>> Ignite documentation: http://ignite.incubator.apache.org

[18:49:12,535][INFO ][ignite-#61%sys-null%][StartNodeCallableImpl] Starting 
remote node with SSH command: nohup 
"/usr/local/BuildAgent/work/c326f6bf1927b309/bin/ignite.sh" -v 
"modules/core/src/test/config/spring-start-nodes.xml" 
-J-DIGNITE_SSH_HOST="127.0.0.1" -J-DIGNITE_SSH_USER_NAME="teamcity" > 
ignite-startNodes/04-28-2015--18-49-11-f5e4d3af.log 2>& 1 &
[18:49:12,540][INFO ][ignite-#63%sys-null%][StartNodeCallableImpl] Starting 
remote node with SSH command: nohup 
"/usr/local/BuildAgent/work/c326f6bf1927b309/bin/ignite.sh" -v 
"modules/core/src/test/config/spring-start-nodes.xml" 
-J-DIGNITE_SSH_HOST="127.0.0.1" -J-DIGNITE_SSH_USER_NAME="teamcity" > 
ignite-startNodes/04-28-2015--18-49-11-e7946d60.log 2>& 1 &
[18:49:12,547][INFO ][ignite-#62%sys-null%][StartNodeCallableImpl] Starting 
remote node with SSH command: nohup 
"/usr/local/BuildAgent/work/c326f6bf1927b309/bin/ignite.sh" -v 
"modules/core/src/test/config/spring-start-nodes.xml" 
-J-DIGNITE_SSH_HOST="127.0.0.1" -J-DIGNITE_SSH_USER_NAME="teamcity" > 
ignite-startNodes/04-28-2015--18-49-11-6be6e056.log 2>& 1 &
[18:49:14,149][INFO ][disco-event-worker-#84%null%][root] Received event: 
NODE_JOINED: id8=1be4a0e7, ip=127.0.0.1
[18:49:14,227][INFO ][disco-event-worker-#84%null%][root] Received event: 
NODE_JOINED: id8=cfd721ac, ip=127.0.0.1
[18:49:14,288][INFO ][disco-event-worker-#84%null%][root] Received event: 
NODE_JOINED: id8=af0e7d37, ip=127.0.0.1
[18:49:15,587][INFO ][ignite-#64%sys-null%][StartNodeCallableImpl] Starting 
remote node with SSH command: nohup 
"/usr/local/BuildAgent/work/c326f6bf1927b309/bin/ignite.sh" -v 
"modules/core/src/test/config/spring-start-nodes.xml" 
-J-DIGNITE_SSH_HOST="127.0.0.1" -J-DIGNITE_SSH_USER_NAME="teamcity" > 
ignite-startNodes/04-28-2015--18-49-14-60ab0020.log 2>& 1 &
[18:49:15,589][INFO ][ignite-#65%sys-null%][StartNodeCallableImpl] Starting 
remote node with SSH command: nohup 
"/usr/local/BuildAgent/work/c326f6bf1927b309/bin/ignite.sh" -v 
"modules/core/src/test/config/spring-start-nodes.xml" 
-J-DIGNITE_SSH_HOST="127.0.0.1" -J-DIGNITE_SSH_USER_NAME="teamcity" > 
ignite-startNodes/04-28-2015--18-49-14-5cde179b.log 2>& 1 &
[18:49:16,921][INFO ][disco-event-worker-#84%null%][root] Received event: 
NODE_JOINED: id8=84118af5, ip=127.0.0.1
[18:49:17,116][INFO ][disco-event-worker-#84%null%][root] Received event: 
NODE_JOINED: id8=f10286c9, ip=127.0.0.1
[18:49:17,116][INFO ][main][root] >>> Stopping test: 
testStartFiveNodesInTwoCalls in 5790 ms <<<
[18:49:17,180][INFO ][disco-event-worker-#84%null%][root] Received event: 
NODE_LEFT: id8=af0e7d37, ip=127.0.0.1
[18:49:17,327][INFO ][disco-event-worker-#84%null%][root] Received event: 
NODE_LEFT: id8=1be4a0e7, ip=127.0.0.1
------- Stderr: -------
[18:49:17,202][WARN ][grid-nio-worker-0-#79%null%][TcpDiscoverySpi] Failed to 
ping node (status check will be initiated): af0e7d37-9127-4c97-85c8-cc0c568726d7
[18:49:17,297][WARN ][grid-nio-worker-3-#82%null%][TcpDiscoverySpi] Failed to 
ping node (status check will be initiated): cfd721ac-7be6-45b5-8adc-fd9db11dd800
[18:49:17,328][WARN ][grid-nio-worker-2-#81%null%][TcpDiscoverySpi] Failed to 
ping node (status check will be initiated): 1be4a0e7-3dc2-4443-a633-d3ee9d325e26
{noformat}

  was:
Nodes after test cannot be stopped.

Log:
{noformat}
java.lang.AssertionError: null
    at 
org.apache.ignite.internal.IgniteProjectionStartStopRestartSelfTest.afterTest(IgniteProjectionStartStopRestartSelfTest.java:153)
------- Stdout: -------
[18:49:11,266][INFO ][main][root] >>> Starting test: 
testStartFiveNodesInTwoCalls <<<
[18:49:11,266][INFO ][main][root] Username: teamcity
[18:49:11,266][INFO ][main][root] Password: teamcity
[18:49:11,266][INFO ][main][root] Key path: null
[18:49:11,267][INFO ][main][XmlBeanDefinitionReader] Loading XML bean 
definitions from URL 
[file:/usr/local/BuildAgent/work/c326f6bf1927b309/modules/core/src/test/config/spring-start-nodes.xml]
[18:49:11,293][INFO ][main][GenericApplicationContext] Refreshing 
org.springframework.context.support.GenericApplicationContext@1ed7fbc4: startup 
date [Tue Apr 28 18:49:11 MSK 2015]; root of context hierarchy
[18:49:11,298][INFO ][main][IgniteKernal] 

>>>    __________  ________________  
>>>   /  _/ ___/ |/ /  _/_  __/ __/  
>>>  _/ // (7 7    // /  / / / _/    
>>> /___/\___/_/|_/___/ /_/ /___/   
>>> 
>>> ver. 1.0.3-SNAPSHOT#20150428-sha1:48c12bb7
>>> 2015 Copyright(C) Apache Software Foundation

[18:49:11,298][INFO ][main][IgniteKernal] Config URL: 
file:/usr/local/BuildAgent/work/c326f6bf1927b309/modules/core/src/test/config/spring-start-nodes.xml
[18:49:11,298][INFO ][main][IgniteKernal] Daemon mode: on
[18:49:11,298][INFO ][main][IgniteKernal] OS: Linux 3.13.0-41-generic amd64
[18:49:11,298][INFO ][main][IgniteKernal] OS user: teamcity
[18:49:11,298][INFO ][main][IgniteKernal] Language runtime: Java Platform API 
Specification ver. 1.7
[18:49:11,298][INFO ][main][IgniteKernal] VM information: Java(TM) SE Runtime 
Environment 1.7.0_67-b01 Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 
24.65-b04
[18:49:11,298][INFO ][main][IgniteKernal] VM total memory: 2.7GB
[18:49:11,298][INFO ][main][IgniteKernal] Remote Management [restart: off, 
REST: off, JMX (remote: off)]
[18:49:11,298][INFO ][main][IgniteKernal] 
IGNITE_HOME=/usr/local/BuildAgent/work/c326f6bf1927b309
[18:49:11,298][INFO ][main][IgniteKernal] VM arguments: 
[-DJAVA_HOME=/usr/lib/jvm/java-7-oracle, -DMCAST_GRP=229.23.1.2, 
-Dagent.home.dir=/usr/local/BuildAgent, -Dagent.name=ip_192.168.2.23, 
-Dagent.ownPort=9090, -Dagent.work.dir=/usr/local/BuildAgent/work, 
-Dbuild.number=2086, 
-Dbuild.vcs.number=48c12bb753d86f9fbe20d8fc23672f29b6f790bf, 
-Dbuild.vcs.number.1=48c12bb753d86f9fbe20d8fc23672f29b6f790bf, 
-Dbuild.vcs.number.IgniteApache=48c12bb753d86f9fbe20d8fc23672f29b6f790bf, 
-Dclassworlds.conf=/usr/local/BuildAgent/temp/buildTmp/teamcity.m2.conf, 
-Dcom.jetbrains.maven.watcher.report.file=/usr/local/BuildAgent/temp/buildTmp/maven-build-info.xml,
 -Djava.io.tmpdir=/usr/local/BuildAgent/temp/buildTmp, 
-Dmaven.home=/usr/local/BuildAgent/tools/maven3, 
-Dmaven.repo.local=/home/teamcity/.m2/repository, 
-Dteamcity.agent.cpuBenchmark=724, 
-Dteamcity.agent.dotnet.agent_url=http://localhost:9090/RPC2, 
-Dteamcity.agent.dotnet.build_id=437568, 
-Dteamcity.auth.password=xaeut7b99lFebuZknJOjhFvxx1qMLIIo, 
-Dteamcity.auth.userId=TeamCityBuildId=437568, 
-Dteamcity.build.changedFiles.file=/usr/local/BuildAgent/temp/buildTmp/changedFiles4219226629992489923.txt,
 -Dteamcity.build.checkoutDir=/usr/local/BuildAgent/work/c326f6bf1927b309, 
-Dteamcity.build.id=437568, 
-Dteamcity.build.properties.file=/usr/local/BuildAgent/temp/buildTmp/teamcity.build7100002810635920006.properties,
 -Dteamcity.build.tempDir=/usr/local/BuildAgent/temp/buildTmp, 
-Dteamcity.build.workingDir=/usr/local/BuildAgent/work/c326f6bf1927b309, 
-Dteamcity.buildConfName=Ignite Start Nodes, 
-Dteamcity.buildType.id=Ignite_StartNodes, 
-Dteamcity.configuration.properties.file=/usr/local/BuildAgent/temp/buildTmp/teamcity.config2404423307058673025.properties,
 
-Dteamcity.maven.watcher.home=/usr/local/BuildAgent/plugins/mavenPlugin/maven-watcher,
 -Dteamcity.projectName=Ignite, 
-Dteamcity.runner.properties.file=/usr/local/BuildAgent/temp/buildTmp/teamcity.runner2837394402491896395.properties,
 
-Dteamcity.tests.recentlyFailedTests.file=/usr/local/BuildAgent/temp/buildTmp/testsToRunFirst1902214237945091123.txt,
 -Dteamcity.version=8.1.4 (build 30168), -ea, -XX:MaxPermSize=512m, -Xms1g, 
-Xmx3g, -XX:+HeapDumpOnOutOfMemoryError, -XX:+AggressiveOpts, 
-DIGNITE_TEST_HOME=/usr/local/BuildAgent/work/c326f6bf1927b309, 
-DIGNITE_UPDATE_NOTIFIER=false, -DIGNITE_NO_DISCO_ORDER=true, 
-DIGNITE_QUIET=false]
[18:49:11,298][INFO ][main][IgniteKernal] Configured caches 
['ignite-marshaller-sys-cache', 'ignite-sys-cache', 'ignite-atomics-sys-cache']
[18:49:11,298][INFO ][main][IgniteKernal] 3-rd party licenses can be found at: 
/usr/local/BuildAgent/work/c326f6bf1927b309/libs/licenses
[18:49:11,301][INFO ][main][IgniteKernal] Non-loopback local IPs: 192.168.2.23, 
fe80:0:0:0:96de:80ff:feba:5463%2
[18:49:11,301][INFO ][main][IgniteKernal] Enabled local MACs: 94DE80BA5463
[18:49:11,301][INFO ][main][IgnitePluginProcessor] Configured plugins:
[18:49:11,301][INFO ][main][IgnitePluginProcessor]   ^-- None
[18:49:11,301][INFO ][main][IgnitePluginProcessor] 
[18:49:11,305][INFO ][main][TcpCommunicationSpi] Successfully bound to TCP port 
[port=47100, locHost=/127.0.0.1]
[18:49:11,311][INFO ][main][IgniteKernal] Security status [authentication=off]
[18:49:11,318][INFO ][main][TcpDiscoverySpi] Successfully bound to TCP port 
[port=47500, localHost=/127.0.0.1]
[18:49:11,326][INFO ][main][IgniteKernal] To start Console Management & 
Monitoring run ignitevisorcmd.{sh|bat}
[18:49:11,326][INFO ][main][IgniteKernal] 
[18:49:11,326][INFO ][main][IgniteKernal] 

>>> +-------------------------------------------------------------------------------+
>>> Ignite ver. 
>>> 1.0.3-SNAPSHOT#20150428-sha1:48c12bb753d86f9fbe20d8fc23672f29b6f790bf
>>> +-------------------------------------------------------------------------------+
>>> OS name: Linux 3.13.0-41-generic amd64
>>> CPU(s): 8
>>> Heap: 2.7GB
>>> VM name: 16539@teamcity23
>>> Grid name: null
>>> Local node [ID=F1330264-0901-41B5-A1A8-CDADD1345510, order=1]
>>> Local node addresses: [/127.0.0.1]
>>> Local ports: TCP:47100 TCP:47500 
>>> Ignite documentation: http://ignite.incubator.apache.org

[18:49:12,535][INFO ][ignite-#61%sys-null%][StartNodeCallableImpl] Starting 
remote node with SSH command: nohup 
"/usr/local/BuildAgent/work/c326f6bf1927b309/bin/ignite.sh" -v 
"modules/core/src/test/config/spring-start-nodes.xml" 
-J-DIGNITE_SSH_HOST="127.0.0.1" -J-DIGNITE_SSH_USER_NAME="teamcity" > 
ignite-startNodes/04-28-2015--18-49-11-f5e4d3af.log 2>& 1 &
[18:49:12,540][INFO ][ignite-#63%sys-null%][StartNodeCallableImpl] Starting 
remote node with SSH command: nohup 
"/usr/local/BuildAgent/work/c326f6bf1927b309/bin/ignite.sh" -v 
"modules/core/src/test/config/spring-start-nodes.xml" 
-J-DIGNITE_SSH_HOST="127.0.0.1" -J-DIGNITE_SSH_USER_NAME="teamcity" > 
ignite-startNodes/04-28-2015--18-49-11-e7946d60.log 2>& 1 &
[18:49:12,547][INFO ][ignite-#62%sys-null%][StartNodeCallableImpl] Starting 
remote node with SSH command: nohup 
"/usr/local/BuildAgent/work/c326f6bf1927b309/bin/ignite.sh" -v 
"modules/core/src/test/config/spring-start-nodes.xml" 
-J-DIGNITE_SSH_HOST="127.0.0.1" -J-DIGNITE_SSH_USER_NAME="teamcity" > 
ignite-startNodes/04-28-2015--18-49-11-6be6e056.log 2>& 1 &
[18:49:14,149][INFO ][disco-event-worker-#84%null%][root] Received event: 
NODE_JOINED: id8=1be4a0e7, ip=127.0.0.1
[18:49:14,227][INFO ][disco-event-worker-#84%null%][root] Received event: 
NODE_JOINED: id8=cfd721ac, ip=127.0.0.1
[18:49:14,288][INFO ][disco-event-worker-#84%null%][root] Received event: 
NODE_JOINED: id8=af0e7d37, ip=127.0.0.1
[18:49:15,587][INFO ][ignite-#64%sys-null%][StartNodeCallableImpl] Starting 
remote node with SSH command: nohup 
"/usr/local/BuildAgent/work/c326f6bf1927b309/bin/ignite.sh" -v 
"modules/core/src/test/config/spring-start-nodes.xml" 
-J-DIGNITE_SSH_HOST="127.0.0.1" -J-DIGNITE_SSH_USER_NAME="teamcity" > 
ignite-startNodes/04-28-2015--18-49-14-60ab0020.log 2>& 1 &
[18:49:15,589][INFO ][ignite-#65%sys-null%][StartNodeCallableImpl] Starting 
remote node with SSH command: nohup 
"/usr/local/BuildAgent/work/c326f6bf1927b309/bin/ignite.sh" -v 
"modules/core/src/test/config/spring-start-nodes.xml" 
-J-DIGNITE_SSH_HOST="127.0.0.1" -J-DIGNITE_SSH_USER_NAME="teamcity" > 
ignite-startNodes/04-28-2015--18-49-14-5cde179b.log 2>& 1 &
[18:49:16,921][INFO ][disco-event-worker-#84%null%][root] Received event: 
NODE_JOINED: id8=84118af5, ip=127.0.0.1
[18:49:17,116][INFO ][disco-event-worker-#84%null%][root] Received event: 
NODE_JOINED: id8=f10286c9, ip=127.0.0.1
[18:49:17,116][INFO ][main][root] >>> Stopping test: 
testStartFiveNodesInTwoCalls in 5790 ms <<<
[18:49:17,180][INFO ][disco-event-worker-#84%null%][root] Received event: 
NODE_LEFT: id8=af0e7d37, ip=127.0.0.1
[18:49:17,327][INFO ][disco-event-worker-#84%null%][root] Received event: 
NODE_LEFT: id8=1be4a0e7, ip=127.0.0.1
------- Stderr: -------
[18:49:17,202][WARN ][grid-nio-worker-0-#79%null%][TcpDiscoverySpi] Failed to 
ping node (status check will be initiated): af0e7d37-9127-4c97-85c8-cc0c568726d7
[18:49:17,297][WARN ][grid-nio-worker-3-#82%null%][TcpDiscoverySpi] Failed to 
ping node (status check will be initiated): cfd721ac-7be6-45b5-8adc-fd9db11dd800
[18:49:17,328][WARN ][grid-nio-worker-2-#81%null%][TcpDiscoverySpi] Failed to 
ping node (status check will be initiated): 1be4a0e7-3dc2-4443-a633-d3ee9d325e26
{noformat}


> [Test] IgniteProjectionStartStopRestartSelfTest.testStartFiveNodesInTwoCalls.
> -----------------------------------------------------------------------------
>
>                 Key: IGNITE-837
>                 URL: https://issues.apache.org/jira/browse/IGNITE-837
>             Project: Ignite
>          Issue Type: Task
>            Reporter: Artem Shutak
>            Assignee: Artem Shutak
>             Fix For: sprint-4
>
>
> IgniteProjectionStartStopRestartSelfTest.testStartFiveNodesInTwoCalls.
> {code}
> {code}
> Default node do not get EVT_NODE_LEFT.
> Log:
> {noformat}
> java.lang.AssertionError: null
>     at 
> org.apache.ignite.internal.IgniteProjectionStartStopRestartSelfTest.afterTest(IgniteProjectionStartStopRestartSelfTest.java:153)
> ------- Stdout: -------
> [18:49:11,266][INFO ][main][root] >>> Starting test: 
> testStartFiveNodesInTwoCalls <<<
> [18:49:11,266][INFO ][main][root] Username: teamcity
> [18:49:11,266][INFO ][main][root] Password: teamcity
> [18:49:11,266][INFO ][main][root] Key path: null
> [18:49:11,267][INFO ][main][XmlBeanDefinitionReader] Loading XML bean 
> definitions from URL 
> [file:/usr/local/BuildAgent/work/c326f6bf1927b309/modules/core/src/test/config/spring-start-nodes.xml]
> [18:49:11,293][INFO ][main][GenericApplicationContext] Refreshing 
> org.springframework.context.support.GenericApplicationContext@1ed7fbc4: 
> startup date [Tue Apr 28 18:49:11 MSK 2015]; root of context hierarchy
> [18:49:11,298][INFO ][main][IgniteKernal] 
> >>>    __________  ________________  
> >>>   /  _/ ___/ |/ /  _/_  __/ __/  
> >>>  _/ // (7 7    // /  / / / _/    
> >>> /___/\___/_/|_/___/ /_/ /___/   
> >>> 
> >>> ver. 1.0.3-SNAPSHOT#20150428-sha1:48c12bb7
> >>> 2015 Copyright(C) Apache Software Foundation
> [18:49:11,298][INFO ][main][IgniteKernal] Config URL: 
> file:/usr/local/BuildAgent/work/c326f6bf1927b309/modules/core/src/test/config/spring-start-nodes.xml
> [18:49:11,298][INFO ][main][IgniteKernal] Daemon mode: on
> [18:49:11,298][INFO ][main][IgniteKernal] OS: Linux 3.13.0-41-generic amd64
> [18:49:11,298][INFO ][main][IgniteKernal] OS user: teamcity
> [18:49:11,298][INFO ][main][IgniteKernal] Language runtime: Java Platform API 
> Specification ver. 1.7
> [18:49:11,298][INFO ][main][IgniteKernal] VM information: Java(TM) SE Runtime 
> Environment 1.7.0_67-b01 Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 
> 24.65-b04
> [18:49:11,298][INFO ][main][IgniteKernal] VM total memory: 2.7GB
> [18:49:11,298][INFO ][main][IgniteKernal] Remote Management [restart: off, 
> REST: off, JMX (remote: off)]
> [18:49:11,298][INFO ][main][IgniteKernal] 
> IGNITE_HOME=/usr/local/BuildAgent/work/c326f6bf1927b309
> [18:49:11,298][INFO ][main][IgniteKernal] VM arguments: 
> [-DJAVA_HOME=/usr/lib/jvm/java-7-oracle, -DMCAST_GRP=229.23.1.2, 
> -Dagent.home.dir=/usr/local/BuildAgent, -Dagent.name=ip_192.168.2.23, 
> -Dagent.ownPort=9090, -Dagent.work.dir=/usr/local/BuildAgent/work, 
> -Dbuild.number=2086, 
> -Dbuild.vcs.number=48c12bb753d86f9fbe20d8fc23672f29b6f790bf, 
> -Dbuild.vcs.number.1=48c12bb753d86f9fbe20d8fc23672f29b6f790bf, 
> -Dbuild.vcs.number.IgniteApache=48c12bb753d86f9fbe20d8fc23672f29b6f790bf, 
> -Dclassworlds.conf=/usr/local/BuildAgent/temp/buildTmp/teamcity.m2.conf, 
> -Dcom.jetbrains.maven.watcher.report.file=/usr/local/BuildAgent/temp/buildTmp/maven-build-info.xml,
>  -Djava.io.tmpdir=/usr/local/BuildAgent/temp/buildTmp, 
> -Dmaven.home=/usr/local/BuildAgent/tools/maven3, 
> -Dmaven.repo.local=/home/teamcity/.m2/repository, 
> -Dteamcity.agent.cpuBenchmark=724, 
> -Dteamcity.agent.dotnet.agent_url=http://localhost:9090/RPC2, 
> -Dteamcity.agent.dotnet.build_id=437568, 
> -Dteamcity.auth.password=xaeut7b99lFebuZknJOjhFvxx1qMLIIo, 
> -Dteamcity.auth.userId=TeamCityBuildId=437568, 
> -Dteamcity.build.changedFiles.file=/usr/local/BuildAgent/temp/buildTmp/changedFiles4219226629992489923.txt,
>  -Dteamcity.build.checkoutDir=/usr/local/BuildAgent/work/c326f6bf1927b309, 
> -Dteamcity.build.id=437568, 
> -Dteamcity.build.properties.file=/usr/local/BuildAgent/temp/buildTmp/teamcity.build7100002810635920006.properties,
>  -Dteamcity.build.tempDir=/usr/local/BuildAgent/temp/buildTmp, 
> -Dteamcity.build.workingDir=/usr/local/BuildAgent/work/c326f6bf1927b309, 
> -Dteamcity.buildConfName=Ignite Start Nodes, 
> -Dteamcity.buildType.id=Ignite_StartNodes, 
> -Dteamcity.configuration.properties.file=/usr/local/BuildAgent/temp/buildTmp/teamcity.config2404423307058673025.properties,
>  
> -Dteamcity.maven.watcher.home=/usr/local/BuildAgent/plugins/mavenPlugin/maven-watcher,
>  -Dteamcity.projectName=Ignite, 
> -Dteamcity.runner.properties.file=/usr/local/BuildAgent/temp/buildTmp/teamcity.runner2837394402491896395.properties,
>  
> -Dteamcity.tests.recentlyFailedTests.file=/usr/local/BuildAgent/temp/buildTmp/testsToRunFirst1902214237945091123.txt,
>  -Dteamcity.version=8.1.4 (build 30168), -ea, -XX:MaxPermSize=512m, -Xms1g, 
> -Xmx3g, -XX:+HeapDumpOnOutOfMemoryError, -XX:+AggressiveOpts, 
> -DIGNITE_TEST_HOME=/usr/local/BuildAgent/work/c326f6bf1927b309, 
> -DIGNITE_UPDATE_NOTIFIER=false, -DIGNITE_NO_DISCO_ORDER=true, 
> -DIGNITE_QUIET=false]
> [18:49:11,298][INFO ][main][IgniteKernal] Configured caches 
> ['ignite-marshaller-sys-cache', 'ignite-sys-cache', 
> 'ignite-atomics-sys-cache']
> [18:49:11,298][INFO ][main][IgniteKernal] 3-rd party licenses can be found 
> at: /usr/local/BuildAgent/work/c326f6bf1927b309/libs/licenses
> [18:49:11,301][INFO ][main][IgniteKernal] Non-loopback local IPs: 
> 192.168.2.23, fe80:0:0:0:96de:80ff:feba:5463%2
> [18:49:11,301][INFO ][main][IgniteKernal] Enabled local MACs: 94DE80BA5463
> [18:49:11,301][INFO ][main][IgnitePluginProcessor] Configured plugins:
> [18:49:11,301][INFO ][main][IgnitePluginProcessor]   ^-- None
> [18:49:11,301][INFO ][main][IgnitePluginProcessor] 
> [18:49:11,305][INFO ][main][TcpCommunicationSpi] Successfully bound to TCP 
> port [port=47100, locHost=/127.0.0.1]
> [18:49:11,311][INFO ][main][IgniteKernal] Security status [authentication=off]
> [18:49:11,318][INFO ][main][TcpDiscoverySpi] Successfully bound to TCP port 
> [port=47500, localHost=/127.0.0.1]
> [18:49:11,326][INFO ][main][IgniteKernal] To start Console Management & 
> Monitoring run ignitevisorcmd.{sh|bat}
> [18:49:11,326][INFO ][main][IgniteKernal] 
> [18:49:11,326][INFO ][main][IgniteKernal] 
> >>> +-------------------------------------------------------------------------------+
> >>> Ignite ver. 
> >>> 1.0.3-SNAPSHOT#20150428-sha1:48c12bb753d86f9fbe20d8fc23672f29b6f790bf
> >>> +-------------------------------------------------------------------------------+
> >>> OS name: Linux 3.13.0-41-generic amd64
> >>> CPU(s): 8
> >>> Heap: 2.7GB
> >>> VM name: 16539@teamcity23
> >>> Grid name: null
> >>> Local node [ID=F1330264-0901-41B5-A1A8-CDADD1345510, order=1]
> >>> Local node addresses: [/127.0.0.1]
> >>> Local ports: TCP:47100 TCP:47500 
> >>> Ignite documentation: http://ignite.incubator.apache.org
> [18:49:12,535][INFO ][ignite-#61%sys-null%][StartNodeCallableImpl] Starting 
> remote node with SSH command: nohup 
> "/usr/local/BuildAgent/work/c326f6bf1927b309/bin/ignite.sh" -v 
> "modules/core/src/test/config/spring-start-nodes.xml" 
> -J-DIGNITE_SSH_HOST="127.0.0.1" -J-DIGNITE_SSH_USER_NAME="teamcity" > 
> ignite-startNodes/04-28-2015--18-49-11-f5e4d3af.log 2>& 1 &
> [18:49:12,540][INFO ][ignite-#63%sys-null%][StartNodeCallableImpl] Starting 
> remote node with SSH command: nohup 
> "/usr/local/BuildAgent/work/c326f6bf1927b309/bin/ignite.sh" -v 
> "modules/core/src/test/config/spring-start-nodes.xml" 
> -J-DIGNITE_SSH_HOST="127.0.0.1" -J-DIGNITE_SSH_USER_NAME="teamcity" > 
> ignite-startNodes/04-28-2015--18-49-11-e7946d60.log 2>& 1 &
> [18:49:12,547][INFO ][ignite-#62%sys-null%][StartNodeCallableImpl] Starting 
> remote node with SSH command: nohup 
> "/usr/local/BuildAgent/work/c326f6bf1927b309/bin/ignite.sh" -v 
> "modules/core/src/test/config/spring-start-nodes.xml" 
> -J-DIGNITE_SSH_HOST="127.0.0.1" -J-DIGNITE_SSH_USER_NAME="teamcity" > 
> ignite-startNodes/04-28-2015--18-49-11-6be6e056.log 2>& 1 &
> [18:49:14,149][INFO ][disco-event-worker-#84%null%][root] Received event: 
> NODE_JOINED: id8=1be4a0e7, ip=127.0.0.1
> [18:49:14,227][INFO ][disco-event-worker-#84%null%][root] Received event: 
> NODE_JOINED: id8=cfd721ac, ip=127.0.0.1
> [18:49:14,288][INFO ][disco-event-worker-#84%null%][root] Received event: 
> NODE_JOINED: id8=af0e7d37, ip=127.0.0.1
> [18:49:15,587][INFO ][ignite-#64%sys-null%][StartNodeCallableImpl] Starting 
> remote node with SSH command: nohup 
> "/usr/local/BuildAgent/work/c326f6bf1927b309/bin/ignite.sh" -v 
> "modules/core/src/test/config/spring-start-nodes.xml" 
> -J-DIGNITE_SSH_HOST="127.0.0.1" -J-DIGNITE_SSH_USER_NAME="teamcity" > 
> ignite-startNodes/04-28-2015--18-49-14-60ab0020.log 2>& 1 &
> [18:49:15,589][INFO ][ignite-#65%sys-null%][StartNodeCallableImpl] Starting 
> remote node with SSH command: nohup 
> "/usr/local/BuildAgent/work/c326f6bf1927b309/bin/ignite.sh" -v 
> "modules/core/src/test/config/spring-start-nodes.xml" 
> -J-DIGNITE_SSH_HOST="127.0.0.1" -J-DIGNITE_SSH_USER_NAME="teamcity" > 
> ignite-startNodes/04-28-2015--18-49-14-5cde179b.log 2>& 1 &
> [18:49:16,921][INFO ][disco-event-worker-#84%null%][root] Received event: 
> NODE_JOINED: id8=84118af5, ip=127.0.0.1
> [18:49:17,116][INFO ][disco-event-worker-#84%null%][root] Received event: 
> NODE_JOINED: id8=f10286c9, ip=127.0.0.1
> [18:49:17,116][INFO ][main][root] >>> Stopping test: 
> testStartFiveNodesInTwoCalls in 5790 ms <<<
> [18:49:17,180][INFO ][disco-event-worker-#84%null%][root] Received event: 
> NODE_LEFT: id8=af0e7d37, ip=127.0.0.1
> [18:49:17,327][INFO ][disco-event-worker-#84%null%][root] Received event: 
> NODE_LEFT: id8=1be4a0e7, ip=127.0.0.1
> ------- Stderr: -------
> [18:49:17,202][WARN ][grid-nio-worker-0-#79%null%][TcpDiscoverySpi] Failed to 
> ping node (status check will be initiated): 
> af0e7d37-9127-4c97-85c8-cc0c568726d7
> [18:49:17,297][WARN ][grid-nio-worker-3-#82%null%][TcpDiscoverySpi] Failed to 
> ping node (status check will be initiated): 
> cfd721ac-7be6-45b5-8adc-fd9db11dd800
> [18:49:17,328][WARN ][grid-nio-worker-2-#81%null%][TcpDiscoverySpi] Failed to 
> ping node (status check will be initiated): 
> 1be4a0e7-3dc2-4443-a633-d3ee9d325e26
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to