[ https://issues.apache.org/jira/browse/FLINK-22599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17541744#comment-17541744 ]
Mason Chen commented on FLINK-22599: ------------------------------------ +1 for fixing this issue and changing the priority of the ticket if it's trivial to solve. This issue is pretty confusing and led me in the wrong direction in debugging. > Point to `client.timeout` in error messages where it is used for rpc timeout > ---------------------------------------------------------------------------- > > Key: FLINK-22599 > URL: https://issues.apache.org/jira/browse/FLINK-22599 > Project: Flink > Issue Type: Improvement > Components: Client / Job Submission > Affects Versions: 1.13.0 > Reporter: Xintong Song > Priority: Not a Priority > Labels: auto-deprioritized-major, auto-deprioritized-minor, > starter > > `client.timeout` is used, instead of `akka.ask.timeout`, as RPC timeout on > the client side. However, this is a bit implicit for users. > E.g., the following shows an error stack of a rpc timeout during job > submission in application mode. A user seeing "Caused by: > akka.pattern.AskTimeoutException" could be misled to `akka.ask.timeout`. > {code:java} > 2021-05-07 21:02:37,513 WARN > org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap > [] - Application failed unexpectedly: > java.util.concurrent.CompletionException: > org.apache.flink.client.deployment.application.ApplicationExecutionException: > Could not execute application. > at > java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) > ~[?:1.8.0_181] > at > java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) > ~[?:1.8.0_181] > at > java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:943) > ~[?:1.8.0_181] > at > java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:926) > ~[?:1.8.0_181] > at > java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) > ~[?:1.8.0_181] > at > java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) > ~[?:1.8.0_181] > at > org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.runApplicationEntryPoint(ApplicationDispatcherBootstrap.java:257) > ~[flink-dist_2.11-1.12.1.jar:1.12.1] > at > org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.lambda$runApplicationAsync$1(ApplicationDispatcherBootstrap.java:212) > ~[flink-dist_2.11-1.12.1.jar:1.12.1] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [?:1.8.0_181] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > [?:1.8.0_181] > at > org.apache.flink.runtime.concurrent.akka.ActorSystemScheduledExecutorAdapter$ScheduledFutureTask.run(ActorSystemScheduledExecutorAdapter.java:159) > [flink-dist_2.11-1.12.1.jar:1.12.1] > at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40) > [flink-dist_2.11-1.12.1.jar:1.12.1] > at > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44) > [flink-dist_2.11-1.12.1.jar:1.12.1] > at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > [flink-dist_2.11-1.12.1.jar:1.12.1] > at > akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) > [flink-dist_2.11-1.12.1.jar:1.12.1] > at > akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > [flink-dist_2.11-1.12.1.jar:1.12.1] > at > akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > [flink-dist_2.11-1.12.1.jar:1.12.1] > Caused by: > org.apache.flink.client.deployment.application.ApplicationExecutionException: > Could not execute application. > ... 11 more > Caused by: org.apache.flink.client.program.ProgramInvocationException: The > main method caused an error: Failed to execute sql > at > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:360) > ~[flink-dist_2.11-1.12.1.jar:1.12.1] > at > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:213) > ~[flink-dist_2.11-1.12.1.jar:1.12.1] > at > org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114) > ~[flink-dist_2.11-1.12.1.jar:1.12.1] > at > org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.runApplicationEntryPoint(ApplicationDispatcherBootstrap.java:242) > ~[flink-dist_2.11-1.12.1.jar:1.12.1] > ... 10 more > Caused by: org.apache.flink.table.api.TableException: Failed to execute sql > at > org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:699) > ~[flink-table-blink_2.11-1.12.1.jar:1.12.1] > at > org.apache.flink.table.api.internal.StatementSetImpl.execute(StatementSetImpl.java:98) > ~[flink-table-blink_2.11-1.12.1.jar:1.12.1] > at > com.kad.cube.dwd.retail.initial.DwdOrderRetailOrderPayInitial.sinkDwdRetailPay(DwdOrderRetailOrderPayInitial.java:91) > ~[cube-data-dwd-order-offline-flink.jar:?] > at > com.kad.cube.dwd.retail.initial.DwdOrderRetailOrderPayInitial.main(DwdOrderRetailOrderPayInitial.java:59) > ~[cube-data-dwd-order-offline-flink.jar:?] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > ~[?:1.8.0_181] > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > ~[?:1.8.0_181] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[?:1.8.0_181] > at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181] > at > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:343) > ~[flink-dist_2.11-1.12.1.jar:1.12.1] > at > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:213) > ~[flink-dist_2.11-1.12.1.jar:1.12.1] > at > org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114) > ~[flink-dist_2.11-1.12.1.jar:1.12.1] > at > org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.runApplicationEntryPoint(ApplicationDispatcherBootstrap.java:242) > ~[flink-dist_2.11-1.12.1.jar:1.12.1] > ... 10 more > Caused by: org.apache.flink.util.FlinkException: Failed to execute job > 'insert-into_kudu.default_database.impala::cube_kudu.dwd_order_retail_order_pay'. > at > org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:1918) > ~[flink-dist_2.11-1.12.1.jar:1.12.1] > at > org.apache.flink.client.program.StreamContextEnvironment.executeAsync(StreamContextEnvironment.java:135) > ~[flink-dist_2.11-1.12.1.jar:1.12.1] > at > org.apache.flink.table.planner.delegation.ExecutorBase.executeAsync(ExecutorBase.java:55) > ~[flink-table-blink_2.11-1.12.1.jar:1.12.1] > at > org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:681) > ~[flink-table-blink_2.11-1.12.1.jar:1.12.1] > at > org.apache.flink.table.api.internal.StatementSetImpl.execute(StatementSetImpl.java:98) > ~[flink-table-blink_2.11-1.12.1.jar:1.12.1] > at > com.kad.cube.dwd.retail.initial.DwdOrderRetailOrderPayInitial.sinkDwdRetailPay(DwdOrderRetailOrderPayInitial.java:91) > ~[cube-data-dwd-order-offline-flink.jar:?] > at > com.kad.cube.dwd.retail.initial.DwdOrderRetailOrderPayInitial.main(DwdOrderRetailOrderPayInitial.java:59) > ~[cube-data-dwd-order-offline-flink.jar:?] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > ~[?:1.8.0_181] > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > ~[?:1.8.0_181] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[?:1.8.0_181] > at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181] > at > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:343) > ~[flink-dist_2.11-1.12.1.jar:1.12.1] > at > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:213) > ~[flink-dist_2.11-1.12.1.jar:1.12.1] > at > org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114) > ~[flink-dist_2.11-1.12.1.jar:1.12.1] > at > org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.runApplicationEntryPoint(ApplicationDispatcherBootstrap.java:242) > ~[flink-dist_2.11-1.12.1.jar:1.12.1] > ... 10 more > Caused by: java.lang.RuntimeException: Error while waiting for job to be > initialized > at > org.apache.flink.client.ClientUtils.waitUntilJobInitializationFinished(ClientUtils.java:160) > ~[flink-dist_2.11-1.12.1.jar:1.12.1] > at > org.apache.flink.client.deployment.application.executors.EmbeddedExecutor.lambda$submitAndGetJobClientFuture$2(EmbeddedExecutor.java:140) > ~[flink-dist_2.11-1.12.1.jar:1.12.1] > at > org.apache.flink.util.function.FunctionUtils.lambda$uncheckedFunction$2(FunctionUtils.java:73) > ~[flink-dist_2.11-1.12.1.jar:1.12.1] > at > java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) > ~[?:1.8.0_181] > at > java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) > ~[?:1.8.0_181] > at > java.util.concurrent.CompletableFuture$Completion.exec(CompletableFuture.java:443) > ~[?:1.8.0_181] > at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) > ~[?:1.8.0_181] > at > java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) > ~[?:1.8.0_181] > at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) > ~[?:1.8.0_181] > at > java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157) > ~[?:1.8.0_181] > Caused by: java.util.concurrent.ExecutionException: > java.util.concurrent.TimeoutException: Invocation of public default > java.util.concurrent.CompletableFuture > org.apache.flink.runtime.webmonitor.RestfulGateway.requestJobStatus(org.apache.flink.api.common.JobID,org.apache.flink.api.common.time.Time) > timed out. > at > java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) > ~[?:1.8.0_181] > at > java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) > ~[?:1.8.0_181] > at > org.apache.flink.client.deployment.application.executors.EmbeddedExecutor.lambda$null$0(EmbeddedExecutor.java:145) > ~[flink-dist_2.11-1.12.1.jar:1.12.1] > at > org.apache.flink.client.ClientUtils.waitUntilJobInitializationFinished(ClientUtils.java:144) > ~[flink-dist_2.11-1.12.1.jar:1.12.1] > at > org.apache.flink.client.deployment.application.executors.EmbeddedExecutor.lambda$submitAndGetJobClientFuture$2(EmbeddedExecutor.java:140) > ~[flink-dist_2.11-1.12.1.jar:1.12.1] > at > org.apache.flink.util.function.FunctionUtils.lambda$uncheckedFunction$2(FunctionUtils.java:73) > ~[flink-dist_2.11-1.12.1.jar:1.12.1] > at > java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) > ~[?:1.8.0_181] > at > java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) > ~[?:1.8.0_181] > at > java.util.concurrent.CompletableFuture$Completion.exec(CompletableFuture.java:443) > ~[?:1.8.0_181] > at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) > ~[?:1.8.0_181] > at > java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) > ~[?:1.8.0_181] > at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) > ~[?:1.8.0_181] > at > java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157) > ~[?:1.8.0_181] > Caused by: java.util.concurrent.TimeoutException: Invocation of public > default java.util.concurrent.CompletableFuture > org.apache.flink.runtime.webmonitor.RestfulGateway.requestJobStatus(org.apache.flink.api.common.JobID,org.apache.flink.api.common.time.Time) > timed out. > at org.apache.flink.runtime.rpc.akka.$Proxy27.requestJobStatus(Unknown > Source) ~[?:1.12.1] > at > org.apache.flink.client.deployment.application.executors.EmbeddedExecutor.lambda$null$0(EmbeddedExecutor.java:143) > ~[flink-dist_2.11-1.12.1.jar:1.12.1] > at > org.apache.flink.client.ClientUtils.waitUntilJobInitializationFinished(ClientUtils.java:144) > ~[flink-dist_2.11-1.12.1.jar:1.12.1] > at > org.apache.flink.client.deployment.application.executors.EmbeddedExecutor.lambda$submitAndGetJobClientFuture$2(EmbeddedExecutor.java:140) > ~[flink-dist_2.11-1.12.1.jar:1.12.1] > at > org.apache.flink.util.function.FunctionUtils.lambda$uncheckedFunction$2(FunctionUtils.java:73) > ~[flink-dist_2.11-1.12.1.jar:1.12.1] > at > java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) > ~[?:1.8.0_181] > at > java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) > ~[?:1.8.0_181] > at > java.util.concurrent.CompletableFuture$Completion.exec(CompletableFuture.java:443) > ~[?:1.8.0_181] > at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) > ~[?:1.8.0_181] > at > java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) > ~[?:1.8.0_181] > at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) > ~[?:1.8.0_181] > at > java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157) > ~[?:1.8.0_181] > Caused by: akka.pattern.AskTimeoutException: Ask timed out on > [Actor[akka://flink/user/rpc/dispatcher_1#1316257960]] after [60000 ms]. > Message of type [org.apache.flink.runtime.rpc.messages.LocalFencedMessage]. A > typical reason for `AskTimeoutException` is that the recipient actor didn't > send a reply. > at akka.pattern.PromiseActorRef$$anonfun$2.apply(AskSupport.scala:635) > ~[flink-dist_2.11-1.12.1.jar:1.12.1] > at akka.pattern.PromiseActorRef$$anonfun$2.apply(AskSupport.scala:635) > ~[flink-dist_2.11-1.12.1.jar:1.12.1] > at > akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:648) > ~[flink-dist_2.11-1.12.1.jar:1.12.1] > at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:205) > ~[flink-dist_2.11-1.12.1.jar:1.12.1] > at > scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601) > ~[flink-dist_2.11-1.12.1.jar:1.12.1] > at > scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109) > ~[flink-dist_2.11-1.12.1.jar:1.12.1] > at > scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599) > ~[flink-dist_2.11-1.12.1.jar:1.12.1] > at > akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:328) > ~[flink-dist_2.11-1.12.1.jar:1.12.1] > at > akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:279) > ~[flink-dist_2.11-1.12.1.jar:1.12.1] > at > akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:283) > ~[flink-dist_2.11-1.12.1.jar:1.12.1] > at > akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:235) > ~[flink-dist_2.11-1.12.1.jar:1.12.1] > at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]{code} > It would be helpful to catch such timeout exceptions, add a message that > points to the right configuration option, and re-throw them. In addition to > the above stack, we should also check other places where `client.timeout` is > used. -- This message was sent by Atlassian Jira (v8.20.7#820007)