[ https://issues.apache.org/jira/browse/IGNITE-5883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477464#comment-16477464 ]
Andrew Medvedev edited comment on IGNITE-5883 at 5/16/18 2:34 PM: ------------------------------------------------------------------ To confirm: This is full build log: [https://ci.ignite.apache.org/downloadBuildLog.html?buildId=1299039&plain=true] This is artifacts: [https://ci.ignite.apache.org/viewLog.html?buildId=1299039&buildTypeId=IgniteTests24Java8_StartNodes&tab=artifacts#!on6mxhrq,-174x1k12sd58u] For passing tests file logs are present For failed test file log is absent Since file logs for failed tests are present when run locally, I really suspect problem during shell invocation. Since we use third-party library for that (jsch), it makes sense to contact authors (?) And some more debug to make sure we prepare correct syntax for ssh call: {quote}{{passed testStopNodes from [https://ci.ignite.apache.org/downloadBuildLog.html?buildId=1296520&plain=true]}} {{[04:40:37] : [Step 3/4] [2018-05-16 01:40:37,312][INFO ][sys-#75|#75][StartNodeCallableImpl] Starting remote node with SSH command: nohup "/data/teamcity/work/fe52c920c1589f47/bin/ignite.sh" v "modules/core/src/test/config/spring-start-nodes.xml" -J-DIGNITE_SSH_HOST="127.0.0.1" -J-DIGNITE_SSH_USER_NAME="teamcity" > /data/teamcity/work/fe52c920c1589f47/work/log/ignite-startNodes/05-16-2018-01-40-36-3cb6f766.log 2>& 1 &}} {{[04:40:37] : [Step 3/4] [2018-05-16 01:40:37,312][INFO ][sys-#73|#73][StartNodeCallableImpl] Starting remote node with SSH command: nohup "/data/teamcity/work/fe52c920c1589f47/bin/ignite.sh" v "modules/core/src/test/config/spring-start-nodes.xml" -J-DIGNITE_SSH_HOST="127.0.0.1" -J-DIGNITE_SSH_USER_NAME="teamcity" > /data/teamcity/work/fe52c920c1589f47/work/log/ignite-startNodes/05-16-2018-01-40-36-fd9f1e5b.log 2>& 1 &}} {{[04:40:37] : [Step 3/4] [2018-05-16 01:40:37,315][INFO ][sys-#74|#74][StartNodeCallableImpl] Starting remote node with SSH command: nohup "/data/teamcity/work/fe52c920c1589f47/bin/ignite.sh" v "modules/core/src/test/config/spring-start-nodes.xml" -J-DIGNITE_SSH_HOST="127.0.0.1" -J-DIGNITE_SSH_USER_NAME="teamcity" > /data/teamcity/work/fe52c920c1589f47/work/log/ignite-startNodes/05-16-2018-01-40-36-78201858.log 2>& 1 &}} {{failed testStopNodes from [https://ci.ignite.apache.org/downloadBuildLog.html?buildId=1299026&plain=true]}} {{[15:54:09] : [Step 3/4] [2018-05-16 12:54:09,412][INFO ][sys-#75|#75][StartNodeCallableImpl] Starting remote node with SSH command: nohup "/data/teamcity/work/fe52c920c1589f47/bin/ignite.sh" v "modules/core/src/test/config/spring-start-nodes.xml" -J-DIGNITE_SSH_HOST="127.0.0.1" -J-DIGNITE_SSH_USER_NAME="teamcity" > /data/teamcity/work/fe52c920c1589f47/work/log/ignite-startNodes/05-16-2018-12-54-08-6ee45d23.log 2>& 1 &}} {{[15:54:09] : [Step 3/4] [2018-05-16 12:54:09,412][INFO ][sys-#74|#74][StartNodeCallableImpl] Starting remote node with SSH command: nohup "/data/teamcity/work/fe52c920c1589f47/bin/ignite.sh" v "modules/core/src/test/config/spring-start-nodes.xml" -J-DIGNITE_SSH_HOST="127.0.0.1" -J-DIGNITE_SSH_USER_NAME="teamcity" > /data/teamcity/work/fe52c920c1589f47/work/log/ignite-startNodes/05-16-2018-12-54-08-befa2f0e.log 2>& 1 &}} {{[15:54:09] : [Step 3/4] [2018-05-16 12:54:09,412][INFO ][sys-#73|#73][StartNodeCallableImpl] Starting remote node with SSH command: nohup "/data/teamcity/work/fe52c920c1589f47/bin/ignite.sh" v "modules/core/src/test/config/spring-start-nodes.xml" -J-DIGNITE_SSH_HOST="127.0.0.1" -J-DIGNITE_SSH_USER_NAME="teamcity" > /data/teamcity/work/fe52c920c1589f47/work/log/ignite-startNodes/05-16-2018-12-54-08-db27de67.log 2>& 1 &}} {quote} was (Author: andmed): To confirm: This is full build log: [https://ci.ignite.apache.org/downloadBuildLog.html?buildId=1299039&plain=true] This is artifacts: [https://ci.ignite.apache.org/viewLog.html?buildId=1299039&buildTypeId=IgniteTests24Java8_StartNodes&tab=artifacts#!on6mxhrq,-174x1k12sd58u] For passing tests file logs are present For failed test file log is absent Since file logs for failed tests are present when run locally, I really suspect problem during shell invocation. Since we use third-party library for that (jsch), it makes sense to contact authors (?) And some more debug to make sure we prepare correct syntax for ssh call: {quote}{{passed testStopNodes from [https://ci.ignite.apache.org/downloadBuildLog.html?buildId=1296520&plain=true]}}{{[04:40:37] : [Step 3/4] [2018-05-16 01:40:37,312][INFO ][sys-#75|#75][StartNodeCallableImpl] Starting remote node with SSH command: nohup "/data/teamcity/work/fe52c920c1589f47/bin/ignite.sh" v "modules/core/src/test/config/spring-start-nodes.xml" -J-DIGNITE_SSH_HOST="127.0.0.1" -J-DIGNITE_SSH_USER_NAME="teamcity" > /data/teamcity/work/fe52c920c1589f47/work/log/ignite-startNodes/05-16-2018-01-40-36-3cb6f766.log 2>& 1 &}} {{[04:40:37] : [Step 3/4] [2018-05-16 01:40:37,312][INFO ][sys-#73|#73][StartNodeCallableImpl] Starting remote node with SSH command: nohup "/data/teamcity/work/fe52c920c1589f47/bin/ignite.sh" v "modules/core/src/test/config/spring-start-nodes.xml" -J-DIGNITE_SSH_HOST="127.0.0.1" -J-DIGNITE_SSH_USER_NAME="teamcity" > /data/teamcity/work/fe52c920c1589f47/work/log/ignite-startNodes/05-16-2018-01-40-36-fd9f1e5b.log 2>& 1 &}} {{[04:40:37] : [Step 3/4] [2018-05-16 01:40:37,315][INFO ][sys-#74|#74][StartNodeCallableImpl] Starting remote node with SSH command: nohup "/data/teamcity/work/fe52c920c1589f47/bin/ignite.sh" v "modules/core/src/test/config/spring-start-nodes.xml" -J-DIGNITE_SSH_HOST="127.0.0.1" -J-DIGNITE_SSH_USER_NAME="teamcity" > /data/teamcity/work/fe52c920c1589f47/work/log/ignite-startNodes/05-16-2018-01-40-36-78201858.log 2>& 1 &}} {{failed testStopNodes from [https://ci.ignite.apache.org/downloadBuildLog.html?buildId=1299026&plain=true]}}{{[15:54:09] : [Step 3/4] [2018-05-16 12:54:09,412][INFO ][sys-#75|#75][StartNodeCallableImpl] Starting remote node with SSH command: nohup "/data/teamcity/work/fe52c920c1589f47/bin/ignite.sh" v "modules/core/src/test/config/spring-start-nodes.xml" -J-DIGNITE_SSH_HOST="127.0.0.1" -J-DIGNITE_SSH_USER_NAME="teamcity" > /data/teamcity/work/fe52c920c1589f47/work/log/ignite-startNodes/05-16-2018-12-54-08-6ee45d23.log 2>& 1 &}} {{[15:54:09] : [Step 3/4] [2018-05-16 12:54:09,412][INFO ][sys-#74|#74][StartNodeCallableImpl] Starting remote node with SSH command: nohup "/data/teamcity/work/fe52c920c1589f47/bin/ignite.sh" v "modules/core/src/test/config/spring-start-nodes.xml" -J-DIGNITE_SSH_HOST="127.0.0.1" -J-DIGNITE_SSH_USER_NAME="teamcity" > /data/teamcity/work/fe52c920c1589f47/work/log/ignite-startNodes/05-16-2018-12-54-08-befa2f0e.log 2>& 1 &}} {{[15:54:09] : [Step 3/4] [2018-05-16 12:54:09,412][INFO ][sys-#73|#73][StartNodeCallableImpl] Starting remote node with SSH command: nohup "/data/teamcity/work/fe52c920c1589f47/bin/ignite.sh" v "modules/core/src/test/config/spring-start-nodes.xml" -J-DIGNITE_SSH_HOST="127.0.0.1" -J-DIGNITE_SSH_USER_NAME="teamcity" > /data/teamcity/work/fe52c920c1589f47/work/log/ignite-startNodes/05-16-2018-12-54-08-db27de67.log 2>& 1 &}} {quote} > Ignite Start Nodes test suite is flaky > -------------------------------------- > > Key: IGNITE-5883 > URL: https://issues.apache.org/jira/browse/IGNITE-5883 > Project: Ignite > Issue Type: Bug > Reporter: Ivan Rakov > Priority: Major > Labels: MakeTeamcityGreenAgain > > Ignite Start Nodes suite contains several flaky tests. > Example runs with sporadic fails: > 1) > http://ci.ignite.apache.org/viewLog.html?buildId=746188&tab=buildResultsDiv&buildTypeId=Ignite20Tests_IgniteStartNodes > 2) > http://ci.ignite.apache.org/viewLog.html?buildId=747364&tab=buildResultsDiv&buildTypeId=Ignite20Tests_IgniteStartNodes > The ticket is for investigation and making tests stable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)