Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1588#issuecomment-186835009 The expected output of the test is the following ``` Test testQueryCluster(org.apache.flink.yarn.YARNSessionFIFOITCase) is running. -------------------------------------------------------------------------------- 20:12:25,692 INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at testing-worker-linux-docker-8281db7b-3371-linux-16/172.17.6.245:8032 20:12:25,701 INFO org.apache.hadoop.yarn.webapp.WebApps - Registered webapp guice modules 20:12:25,712 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl - Registering with RM using finished containers :[] 20:12:25,712 INFO org.apache.hadoop.yarn.util.RackResolver - Resolved testing-worker-linux-docker-8281db7b-3371-linux-16.prod.travis-ci.org to /default-rack 20:12:25,712 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService - NodeManager from node testing-worker-linux-docker-8281db7b-3371-linux-16.prod.travis-ci.org(cmPort: 59877 httpPort: 39611) registered with capability: <memory:4096, vCores:666>, assigned nodeId testing-worker-linux-docker-8281db7b-3371-linux-16.prod.travis-ci.org:59877 20:12:25,712 INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl - testing-worker-linux-docker-8281db7b-3371-linux-16.prod.travis-ci.org:59877 Node Transitioned from NEW to RUNNING 20:12:25,712 INFO org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager - Rolling master-key for container-tokens, got key with id 1069868518 20:12:25,712 INFO org.apache.hadoop.yarn.server.nodemanager.security.NMTokenSecretManagerInNM - Rolling master-key for nm-tokens, got key with id :-1168902475 20:12:25,712 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl - Registered with ResourceManager as testing-worker-linux-docker-8281db7b-3371-linux-16.prod.travis-ci.org:59877 with total resource of <memory:4096, vCores:666> 20:12:25,712 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl - Notifying ContainerManager to unblock new container-requests 20:12:25,939 INFO org.apache.flink.yarn.YARNSessionFIFOITCase - Starting testQueryCluster() 20:12:25,939 INFO org.apache.flink.yarn.YarnTestBase - Running with args [-q] 20:12:25,993 INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at /0.0.0.0:8032 20:12:26,940 INFO org.apache.flink.yarn.YarnTestBase - Found expected output in redirected streams 20:12:26,940 INFO org.apache.flink.yarn.YarnTestBase - RunWithArgs: request runner to stop 20:12:26,940 WARN org.apache.flink.yarn.YarnTestBase - RunWithArgs runner stopped. 20:12:26,940 INFO org.apache.flink.yarn.YarnTestBase - Sending stdout content through logger: NodeManagers in the Cluster 2|Property |Value +---------------------------------------+ |NodeID |testing-worker-linux-docker-8281db7b-3371-linux-16.prod.travis-ci.org:59877 |Memory |4096 MB |vCores |666 |HealthReport | |Containers |0 +---------------------------------------+ |NodeID |testing-worker-linux-docker-8281db7b-3371-linux-16.prod.travis-ci.org:44161 |Memory |4096 MB |vCores |666 |HealthReport | |Containers |0 +---------------------------------------+ Summary: totalMemory 8192 totalCores 1332 20:12:26,940 INFO org.apache.flink.yarn.YarnTestBase - Sending stderr content through logger: 20:12:26,940 INFO org.apache.flink.yarn.YarnTestBase - Test was successful 20:12:26,940 INFO org.apache.flink.yarn.YARNSessionFIFOITCase - Finished testQueryCluster() 20:12:27,443 INFO org.apache.flink.yarn.YARNSessionFIFOITCase - ``` but when its failing, its outputting ``` Test testQueryCluster(org.apache.flink.yarn.YARNSessionFIFOITCase) is running. -------------------------------------------------------------------------------- 21:50:20,684 INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at testing-worker-linux-docker-1305e497-3358-linux-10/172.17.4.146:8032 21:50:22,041 INFO org.apache.flink.yarn.YARNSessionFIFOITCase - Starting testQueryCluster() 21:50:22,041 INFO org.apache.flink.yarn.YarnTestBase - Running with args [-q] 21:50:22,212 INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at /0.0.0.0:8032 21:50:22,559 INFO org.mortbay.log - Started HttpServer2$SelectChannelConnectorWithSafeStartup@testing-worker-linux-docker-1305e497-3358-linux-10:45772 21:50:22,576 INFO org.apache.hadoop.yarn.webapp.WebApps - Web app /node started at 45772 21:50:22,577 INFO org.mortbay.log - Started HttpServer2$SelectChannelConnectorWithSafeStartup@testing-worker-linux-docker-1305e497-3358-linux-10:58616 21:50:22,577 INFO org.apache.hadoop.yarn.webapp.WebApps - Web app /node started at 58616 21:50:22,664 INFO org.apache.hadoop.yarn.webapp.WebApps - Registered webapp guice modules 21:50:22,665 INFO org.apache.hadoop.yarn.webapp.WebApps - Registered webapp guice modules 21:50:22,691 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl - Sending out 0 NM container statuses: [] 21:50:22,696 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl - Registering with RM using containers :[] 21:50:22,746 INFO org.apache.hadoop.yarn.util.RackResolver - Resolved testing-worker-linux-docker-1305e497-3358-linux-10.prod.travis-ci.org to /default-rack 21:50:22,753 INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl - testing-worker-linux-docker-1305e497-3358-linux-10.prod.travis-ci.org:42997 Node Transitioned from NEW to RUNNING 21:50:22,758 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl - Sending out 0 NM container statuses: [] 21:50:22,758 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl - Registering with RM using containers :[] 21:50:22,758 INFO org.apache.hadoop.yarn.util.RackResolver - Resolved testing-worker-linux-docker-1305e497-3358-linux-10.prod.travis-ci.org to /default-rack 21:50:22,753 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService - NodeManager from node testing-worker-linux-docker-1305e497-3358-linux-10.prod.travis-ci.org(cmPort: 42997 httpPort: 58616) registered with capability: <memory:4096, vCores:666>, assigned nodeId testing-worker-linux-docker-1305e497-3358-linux-10.prod.travis-ci.org:42997 21:50:22,758 INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl - testing-worker-linux-docker-1305e497-3358-linux-10.prod.travis-ci.org:50661 Node Transitioned from NEW to RUNNING 21:50:22,759 INFO org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager - Rolling master-key for container-tokens, got key with id -1878662703 21:50:22,758 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService - NodeManager from node testing-worker-linux-docker-1305e497-3358-linux-10.prod.travis-ci.org(cmPort: 50661 httpPort: 45772) registered with capability: <memory:4096, vCores:666>, assigned nodeId testing-worker-linux-docker-1305e497-3358-linux-10.prod.travis-ci.org:50661 21:50:22,760 INFO org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager - Rolling master-key for container-tokens, got key with id -1878662703 21:50:22,760 INFO org.apache.hadoop.yarn.server.nodemanager.security.NMTokenSecretManagerInNM - Rolling master-key for nm-tokens, got key with id :-748258147 21:50:22,761 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl - Registered with ResourceManager as testing-worker-linux-docker-1305e497-3358-linux-10.prod.travis-ci.org:42997 with total resource of <memory:4096, vCores:666> 21:50:22,761 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl - Notifying ContainerManager to unblock new container-requests 21:50:22,761 INFO org.apache.hadoop.yarn.server.nodemanager.security.NMTokenSecretManagerInNM - Rolling master-key for nm-tokens, got key with id :-748258147 21:50:22,762 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl - Registered with ResourceManager as testing-worker-linux-docker-1305e497-3358-linux-10.prod.travis-ci.org:50661 with total resource of <memory:4096, vCores:666> 21:50:22,762 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl - Notifying ContainerManager to unblock new container-requests 21:50:23,077 INFO org.apache.flink.yarn.YarnTestBase - Runner stopped earlier than expected with return value = 0 21:50:23,077 INFO org.apache.flink.yarn.YarnTestBase - Sending stdout content through logger: NodeManagers in the Cluster 0|Property |Value +---------------------------------------+ Summary: totalMemory 0 totalCores 0 21:50:23,077 INFO org.apache.flink.yarn.YarnTestBase - Sending stderr content through logger: 21:50:23,580 ERROR org.apache.flink.yarn.YARNSessionFIFOITCase - -------------------------------------------------------------------------------- Test testQueryCluster(org.apache.flink.yarn.YARNSessionFIFOITCase) failed with: java.lang.AssertionError: During the timeout period of 180 seconds the expected string did not show up ``` Looking through the logs, I think the issue is the following: When the `testQueryCluster()` is executed, the NodeManagers are not yet registered with YARN. That's why the number of nodemanagers in the test is 0. The issue is occurring after you change, because the test execution order changed. Before your change, the `testQueryCluster()` test was executed after other tests, so the NM's were always registered. Since you removed many tests from the FIFOITCase, the `testQueryCluster()` is the first test to be executed. Apparently, the test setup is not waiting until all NM's are connected. I think you can solve the issue using the `waitForNodeManagersToConnect` from the MiniYARNCluster.
--- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---