Re:flink1.10.0任务失败但是application仍然在yarn运行不会自动kill

2020-08-12 文章 魏烽 
是的，但是连接超时判定任务失败后也应该把application在yarn kill掉吧，不然会一直挂着占用资源

 原始邮件
发件人: Michael Ran
收件人: user-zh
发送时间: 2020年8月12日(周三) 15:41
主题: Re:flink1.10.0任务失败但是application仍然在yarn运行不会自动kill


Connection timed out: 
nb-bdh-hadoop.slave14/10.10.5.14:21913是不是连接超时，请求没发送过去？
在 2020-08-12 13:57:52，"魏烽" mailto:weif...@nequal.com>> 写道：
>各位好：
>
>Flink1.10.0版本发现偶发任务失败后但是web ui仍然挂着该任务并没有停止，日志如下：
>
>请问有遇到一样的情况的嘛
>
>[INFO] 2020-07-28 16:34:00.938  - [taskAppId=TASK-38-97-193]:[106] -  -> 
>2020-07-28 16:33:52,863 INFO  org.apache.flink.yarn.YarnClusterDescriptor  
> - Submitting application master application_1571540269403_52656
>
>2020-07-28 16:33:52,895 INFO  
>org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted 
>application application_1571540269403_52656
>
>2020-07-28 16:33:52,895 INFO  org.apache.flink.yarn.YarnClusterDescriptor  
> - Waiting for the cluster to be allocated
>
>2020-07-28 16:33:52,897 INFO  org.apache.flink.yarn.YarnClusterDescriptor  
> - Deploying cluster, current state ACCEPTED
>
>2020-07-28 16:34:00,938 INFO  org.apache.flink.yarn.YarnClusterDescriptor  
> - YARN application has been deployed successfully.
>
>[INFO] 2020-07-28 16:36:04.622  - [taskAppId=TASK-38-97-193]:[106] -  -> 
>2020-07-28 16:34:00,941 INFO  org.apache.flink.yarn.YarnClusterDescriptor  
> - Found Web Interface nb-bdh-hadoop.slave14:21913 of application 
>'application_1571540269403_52656'.
>
>Job has been submitted with JobID 54ac0f3db08f29022d9f3d51d797a724
>
>[INFO] 2020-07-28 16:36:04.623  - [taskAppId=TASK-38-97-193]:[106] -  -> 
>
>
>The program finished with the following exception:
>
>org.apache.flink.client.program.ProgramInvocationException: The main method 
>caused an error: 
>org.apache.flink.runtime.concurrent.FutureUtils$RetryException: Could not 
>complete the operation. Number of retries has been exhausted.
>
>at 
>org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:335)
>
>at 
>org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:205)
>
>at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:138)
>
>at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:664)
>
>at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:213)
>
>at 
>org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:895)
>
>at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:968)
>
>at java.security.AccessController.doPrivileged(Native Method)
>
>at javax.security.auth.Subject.doAs(Subject.java:422)
>
>at 
>org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
>
>at 
>org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
>
>at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:968)
>
>Caused by: java.util.concurrent.ExecutionException: 
>org.apache.flink.runtime.concurrent.FutureUtils$RetryException: Could not 
>complete the operation. Number of retries has been exhausted.
>
>at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
>
>at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
>
>at 
>org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:83)
>
>at 
>org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1620)
>
>at 
>org.apache.flink.streaming.api.scala.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.scala:678)
>
>at com.nequal.bdh.cdp.IDMappingLauncher$.main(IDMappingLauncher.scala:140)
>
>at com.nequal.bdh.cdp.IDMappingLauncher.main(IDMappingLauncher.scala)
>
>at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
>at 
>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
>at 
>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
>at java.lang.reflect.Method.invoke(Method.java:498)
>
>at 
>org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:321)
>
>... 11 more
>
>Caused by: org.apache.flink.runtime.concurrent.FutureUtils$RetryException: 
>Could not complete the operation. Number of retries has been exhausted.
>
>at 
>org.apache.flink.runtime.concurrent.FutureUtils.lambda$retryOperationWithDelay$8(FutureUtils.java:284)
>
>at 
>java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.ja

Re:flink1.10.0任务失败但是application仍然在yarn运行不会自动kill

2020-08-12 文章 Michael Ran 
Connection timed out: 
nb-bdh-hadoop.slave14/10.10.5.14:21913是不是连接超时，请求没发送过去？
在 2020-08-12 13:57:52，"魏烽"  写道：
>各位好：
>
>Flink1.10.0版本发现偶发任务失败后但是web ui仍然挂着该任务并没有停止，日志如下：
>
>请问有遇到一样的情况的嘛
>
>[INFO] 2020-07-28 16:34:00.938  - [taskAppId=TASK-38-97-193]:[106] -  -> 
>2020-07-28 16:33:52,863 INFO  org.apache.flink.yarn.YarnClusterDescriptor  
> - Submitting application master application_1571540269403_52656
>
>2020-07-28 16:33:52,895 INFO  
>org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted 
>application application_1571540269403_52656
>
>2020-07-28 16:33:52,895 INFO  org.apache.flink.yarn.YarnClusterDescriptor  
> - Waiting for the cluster to be allocated
>
>2020-07-28 16:33:52,897 INFO  org.apache.flink.yarn.YarnClusterDescriptor  
> - Deploying cluster, current state ACCEPTED
>
>2020-07-28 16:34:00,938 INFO  org.apache.flink.yarn.YarnClusterDescriptor  
> - YARN application has been deployed successfully.
>
>[INFO] 2020-07-28 16:36:04.622  - [taskAppId=TASK-38-97-193]:[106] -  -> 
>2020-07-28 16:34:00,941 INFO  org.apache.flink.yarn.YarnClusterDescriptor  
> - Found Web Interface nb-bdh-hadoop.slave14:21913 of application 
>'application_1571540269403_52656'.
>
>Job has been submitted with JobID 54ac0f3db08f29022d9f3d51d797a724
>
>[INFO] 2020-07-28 16:36:04.623  - [taskAppId=TASK-38-97-193]:[106] -  -> 
>
>
>The program finished with the following exception:
>
>org.apache.flink.client.program.ProgramInvocationException: The main method 
>caused an error: 
>org.apache.flink.runtime.concurrent.FutureUtils$RetryException: Could not 
>complete the operation. Number of retries has been exhausted.
>
>at 
>org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:335)
>
>at 
>org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:205)
>
>at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:138)
>
>at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:664)
>
>at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:213)
>
>at 
>org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:895)
>
>at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:968)
>
>at java.security.AccessController.doPrivileged(Native Method)
>
>at javax.security.auth.Subject.doAs(Subject.java:422)
>
>at 
>org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
>
>at 
>org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
>
>at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:968)
>
>Caused by: java.util.concurrent.ExecutionException: 
>org.apache.flink.runtime.concurrent.FutureUtils$RetryException: Could not 
>complete the operation. Number of retries has been exhausted.
>
>at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
>
>at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
>
>at 
>org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:83)
>
>at 
>org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1620)
>
>at 
>org.apache.flink.streaming.api.scala.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.scala:678)
>
>at com.nequal.bdh.cdp.IDMappingLauncher$.main(IDMappingLauncher.scala:140)
>
>at com.nequal.bdh.cdp.IDMappingLauncher.main(IDMappingLauncher.scala)
>
>at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
>at 
>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
>at 
>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
>at java.lang.reflect.Method.invoke(Method.java:498)
>
>at 
>org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:321)
>
>... 11 more
>
>Caused by: org.apache.flink.runtime.concurrent.FutureUtils$RetryException: 
>Could not complete the operation. Number of retries has been exhausted.
>
>at 
>org.apache.flink.runtime.concurrent.FutureUtils.lambda$retryOperationWithDelay$8(FutureUtils.java:284)
>
>at 
>java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>
>at 
>java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>
>at 
>java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
>
>at 
>java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
>
>at 
>org.apache.flink.runtime.rest.RestClient.lambda$submitRequest$1(RestClient.java:342)
>
>at 
>org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:500)
>
>at