[
https://issues.apache.org/jira/browse/FLINK-20568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17248911#comment-17248911
]
Huang Xingbo commented on FLINK-20568:
--------------------------------------
[~trohrmann] I re-analyzed the test code and the error log, the above exception
is expected. It is not the root cause of the test failure. Two tests failed in
the progress of `start hadoop cluster`, but they have some difference.
The first test had retried about 70 times, but still failed to set up kerberos,
which caused it to fail.
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=10780&view=logs&j=08866332-78f7-59e4-4f7e-49a56faa3179&t=3e8647c1-5a28-5917-dd93-bf78594ea994]
{code:java}
2020-12-10T22:43:17.8181321Z Waiting for hadoop cluster to come up. We have
been trying for 0 seconds, retrying ...
2020-12-10T22:43:22.9718942Z Waiting for hadoop cluster to come up. We have
been trying for 5 seconds, retrying ...
2020-12-10T22:43:28.1210588Z Waiting for hadoop cluster to come up. We have
been trying for 11 seconds, retrying ...
2020-12-10T22:43:33.2153581Z Waiting for hadoop cluster to come up. We have
been trying for 16 seconds, retrying ...
2020-12-10T22:43:38.3663723Z Waiting for hadoop cluster to come up. We have
been trying for 21 seconds, retrying ...
2020-12-10T22:43:43.5593286Z Waiting for hadoop cluster to come up. We have
been trying for 26 seconds, retrying ...
2020-12-10T22:43:48.7274576Z Waiting for hadoop cluster to come up. We have
been trying for 31 seconds, retrying ...
2020-12-10T22:43:53.8564893Z Waiting for hadoop cluster to come up. We have
been trying for 36 seconds, retrying ...
2020-12-10T22:43:58.9918800Z Waiting for hadoop cluster to come up. We have
been trying for 41 seconds, retrying ...
2020-12-10T22:44:04.1774480Z Waiting for hadoop cluster to come up. We have
been trying for 47 seconds, retrying ...
2020-12-10T22:44:09.2669098Z Waiting for hadoop cluster to come up. We have
been trying for 52 seconds, retrying ...
2020-12-10T22:44:14.4303297Z Waiting for hadoop cluster to come up. We have
been trying for 57 seconds, retrying ...
2020-12-10T22:44:19.5714381Z Waiting for hadoop cluster to come up. We have
been trying for 62 seconds, retrying ...
2020-12-10T22:44:24.7270110Z Waiting for hadoop cluster to come up. We have
been trying for 67 seconds, retrying ...
2020-12-10T22:44:29.8639018Z Waiting for hadoop cluster to come up. We have
been trying for 72 seconds, retrying ...
2020-12-10T22:44:35.0812405Z Waiting for hadoop cluster to come up. We have
been trying for 78 seconds, retrying ...
2020-12-10T22:44:40.1799383Z Waiting for hadoop cluster to come up. We have
been trying for 83 seconds, retrying ...
2020-12-10T22:44:45.4086777Z Waiting for hadoop cluster to come up. We have
been trying for 88 seconds, retrying ...
2020-12-10T22:44:50.5826135Z Waiting for hadoop cluster to come up. We have
been trying for 93 seconds, retrying ...
2020-12-10T22:44:55.6757865Z Waiting for hadoop cluster to come up. We have
been trying for 98 seconds, retrying ...
2020-12-10T22:45:00.8399405Z Waiting for hadoop cluster to come up. We have
been trying for 103 seconds, retrying ...
2020-12-10T22:45:06.0493092Z Waiting for hadoop cluster to come up. We have
been trying for 109 seconds, retrying ...
2020-12-10T22:45:11.3264321Z Waiting for hadoop cluster to come up. We have
been trying for 114 seconds, retrying ...
2020-12-10T22:45:16.4317246Z Waiting for hadoop cluster to come up. We have
been trying for 119 seconds, retrying ...
2020-12-10T22:45:21.6490142Z Command: start_hadoop_cluster failed. Retrying...
2020-12-10T22:45:21.6509181Z Command: start_hadoop_cluster failed 3 times.
2020-12-10T22:45:21.6509946Z ERROR: Could not start hadoop cluster. Aborting...
{code}
In the second test, after successfully starting two NodeManagers, the progress
of start_hadoop_cluster hangs for some unknown reasons.
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=10827&view=logs&j=91bf6583-3fb2-592f-e4d4-d79d79c3230a&t=3425d8ba-5f03-540a-c64b-51b8481bf7d6]
{code:java}
2020-12-12T23:09:01.9221574Z Dec 12 23:09:01 Waiting for hadoop cluster to come
up. We have been trying for 5 seconds, retrying ...
2020-12-12T23:09:07.0336403Z Dec 12 23:09:07 Waiting for hadoop cluster to come
up. We have been trying for 11 seconds, retrying ...
2020-12-12T23:09:12.1363406Z Dec 12 23:09:12 Waiting for hadoop cluster to come
up. We have been trying for 16 seconds, retrying ...
2020-12-12T23:09:17.2244415Z Dec 12 23:09:17 Waiting for hadoop cluster to come
up. We have been trying for 21 seconds, retrying ...
2020-12-12T23:09:22.3257323Z Dec 12 23:09:22 Waiting for hadoop cluster to come
up. We have been trying for 26 seconds, retrying ...
2020-12-12T23:09:27.3920007Z Dec 12 23:09:27 Waiting for hadoop cluster to come
up. We have been trying for 31 seconds, retrying ...
2020-12-12T23:09:32.4700252Z Dec 12 23:09:32 Waiting for hadoop cluster to come
up. We have been trying for 36 seconds, retrying ...
2020-12-12T23:09:37.5409059Z Dec 12 23:09:37 Waiting for hadoop cluster to come
up. We have been trying for 41 seconds, retrying ...
2020-12-12T23:09:42.6145465Z Dec 12 23:09:42 Waiting for hadoop cluster to come
up. We have been trying for 46 seconds, retrying ...
2020-12-12T23:09:47.8639056Z Dec 12 23:09:47 We only have 0 NodeManagers up. We
have been trying for 0 seconds, retrying ...
2020-12-12T23:09:50.3793819Z 20/12/12 23:09:50 INFO client.RMProxy: Connecting
to ResourceManager at master.docker-hadoop-cluster-network/172.21.0.3:8032
2020-12-12T23:09:50.4429131Z 20/12/12 23:09:50 INFO client.AHSProxy: Connecting
to Application History server at
master.docker-hadoop-cluster-network/172.21.0.3:10200
2020-12-12T23:09:50.9774663Z Dec 12 23:09:50 We now have 2 NodeManagers up.
2020-12-13T01:34:54.0786309Z
==========================================================================================
2020-12-13T01:34:54.0787940Z === WARNING: This E2E Run took already 80% of the
allocated time budget of 250 minutes ===
{code}
> Kerberized YARN per-job on Docker test failed with "Hadoop security with
> Kerberos is enabled but the login user does not have Kerberos credentials or
> delegation tokens!"
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-20568
> URL: https://issues.apache.org/jira/browse/FLINK-20568
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
> Affects Versions: 1.11.0, 1.12.0, 1.13.0
> Reporter: Huang Xingbo
> Priority: Major
> Labels: test-stability
>
> Instance on 1.11 branch
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=10780&view=logs&j=08866332-78f7-59e4-4f7e-49a56faa3179&t=3e8647c1-5a28-5917-dd93-bf78594ea994]
> {code:java}
> 2020-12-10T22:38:25.1087443Z The program finished with the following
> exception:
> 2020-12-10T22:38:25.1087688Z
> 2020-12-10T22:38:25.1088094Z
> org.apache.flink.client.program.ProgramInvocationException: The main method
> caused an error: Could not deploy Yarn job cluster.
> 2020-12-10T22:38:25.1088717Z at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:302)
> 2020-12-10T22:38:25.1089321Z at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:198)
> 2020-12-10T22:38:25.1090233Z at
> org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:149)
> 2020-12-10T22:38:25.1090749Z at
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:699)
> 2020-12-10T22:38:25.1091233Z at
> org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:232)
> 2020-12-10T22:38:25.1091705Z at
> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:916)
> 2020-12-10T22:38:25.1092225Z at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:992)
> 2020-12-10T22:38:25.1095464Z at
> java.security.AccessController.doPrivileged(Native Method)
> 2020-12-10T22:38:25.1095961Z at
> javax.security.auth.Subject.doAs(Subject.java:422)
> 2020-12-10T22:38:25.1096436Z at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1840)
> 2020-12-10T22:38:25.1097027Z at
> org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
> 2020-12-10T22:38:25.1097859Z at
> org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:992)
> 2020-12-10T22:38:25.1098474Z Caused by:
> org.apache.flink.client.deployment.ClusterDeploymentException: Could not
> deploy Yarn job cluster.
> 2020-12-10T22:38:25.1099065Z at
> org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:431)
> 2020-12-10T22:38:25.1099674Z at
> org.apache.flink.client.deployment.executors.AbstractJobClusterExecutor.execute(AbstractJobClusterExecutor.java:70)
> 2020-12-10T22:38:25.1100918Z at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:1818)
> 2020-12-10T22:38:25.1101607Z at
> org.apache.flink.client.program.StreamContextEnvironment.executeAsync(StreamContextEnvironment.java:128)
> 2020-12-10T22:38:25.1102202Z at
> org.apache.flink.client.program.StreamContextEnvironment.execute(StreamContextEnvironment.java:76)
> 2020-12-10T22:38:25.1102840Z at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1700)
> 2020-12-10T22:38:25.1103467Z at
> org.apache.flink.streaming.examples.wordcount.WordCount.main(WordCount.java:96)
> 2020-12-10T22:38:25.1104174Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2020-12-10T22:38:25.1104638Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2020-12-10T22:38:25.1105174Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2020-12-10T22:38:25.1105645Z at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2020-12-10T22:38:25.1106119Z at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:288)
> 2020-12-10T22:38:25.1106495Z ... 11 more
> 2020-12-10T22:38:25.1106940Z Caused by: java.lang.RuntimeException: Hadoop
> security with Kerberos is enabled but the login user does not have Kerberos
> credentials or delegation tokens!
> 2020-12-10T22:38:25.1107584Z at
> org.apache.flink.yarn.YarnClusterDescriptor.deployInternal(YarnClusterDescriptor.java:475)
> 2020-12-10T22:38:25.1108484Z at
> org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:424)
> 2020-12-10T22:38:25.1109201Z ... 22 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
