Re: Application Mode 模式部署失败
Hi！
从报错日志来看是找不到方法。很可能客户端的版本和集群里的版本不一致，或客户端的 Flink 文件有损坏。这两方面可以都检查一下。
周瑞 于2021年7月30日周五 上午11:30写道：
> 您好：Flink Appliaction mode 模式启动失败，启动命令和错误日志如下./flink run-application -t
> yarn-application \
> -yD yarn.application.name="MyFlinkApp" \
> -yD yarn.provided.lib.dirs="hdfs://
> 10.10.98.226:8020/user/myflink/flink-common-deps/libs/yarn-flink-1.13.0/lib/;hdfs://10.10.98.226:8020/user/myflink/flink-common-deps/libs/yarn-flink-1.13.0/plugins/"
> \
> /app/qmatrix/yarn-flink-1.13.0/qmatrix_jars/Sink/Hive/0.0.2/flink-hive-sink-3.0.0-jar-with-dependencies.jar
> \
> --config
> '{"centerName":"rui","checkPointInterval":1000,"checkPointPath":"hdfs://
> 10.10.98.226:8020/tmp/checkpoint66/Sink_rui_ruihiveha_HaHive
> ","configProperties":{"tableKeyName":"{\"rui.fruitest1\":[\"show_integer\"]}"},"jobParallelism":3,"kafkaBootstrapServers":"qmatrix-1:9092,qmatrix-2:9092,qmatrix-3:9092","kafkaConsumerProperties":{"enable.auto.commit":"false","partition.assignment.strategy":"org.apache.kafka.clients.consumer.RoundRobinAssignor","max.poll.records":"500","
> group.id":"Sink_rui_ruihiveha_HaHive_g","auto.offset.reset":"earliest","
> session.timeout.ms
> ":"15000","bootstrap.servers":"qmatrix-1:9092,qmatrix-2:9092,qmatrix-3:9092","max.partition.fetch.bytes":"1048576","
> max.poll.interval.ms":"180","heartbeat.interval.ms
> ":"3000","isolation.level":"read_committed","auto.commit.interval.ms
> ":"1000"},"managerJdbcProperties":{"url":"jdbc:mysql://qmatrix-mysql:3306/qmatrix","phyTimeoutMillis":"3","maxActive":"1","driverClassName":"com.mysql.jdbc.Driver","removeAbandoned":"true","minEvictableIdleTimeMillis":"3","username":"root","minIdle":"0","removeAbandonedTimeout":"3","timeBetweenEvictionRunsMillis":"3","password":"Cljslrl0620$","keepAlive":"false","initialSize":"0"},"passwordKey":"GOODLUCKEVERYONE","pipeName":"ruihiveha","targetConfig":{"rui.ruihiveha.rac80_forever.rac80_ruitest1.1627609625759.router.qmatrix":{"migratePartitionTime":1,"partitionField":"pt_dt","partitionInterval":1,"partitionTime":0,"partitionTimeField":"SHOW_TIMESTAMP","periodMigratePartitionTime":0,"targetSchema":"rui","targetTable":"fruitest1"}},"targetNodeName":"HaHive","targetNodeProperties":{"defaultSchema":"default","hiveConfDir":"/app/qmatrix/kerberos/HaHive","hiveUser":"root","password":"123456","url":"jdbc:hive2://
> 10.10.98.42:1","username":"root"},"targetNodeType":"Hive","topologyId":1137,"topologyName":"Sink_rui_ruihiveha_HaHive","yarnJobManagerMemory":"1024MB","yarnTaskManagerMemory":"2048MB","zookeeperUrl":"qmatrix-1:2181,qmatrix-2:2181,qmatrix-3:2181"}'
> \
> --nodeType Hive \
> --jobType Sink \
> 2021-07-30 11:20:01,402 INFO flink-akka.actor.default-dispatcher-3
> org.apache.flink.runtime.externalresource.ExternalResourceUtils [] -
> Enabled external resources: [] 2021-07-30 11:20:01,416 INFO
> flink-akka.actor.default-dispatcher-3
> org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl [] - Upper
> bound of the thread pool size is 500 2021-07-30 11:20:01,418 INFO
> flink-akka.actor.default-dispatcher-3
> org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy []
> - yarn.client.max-cached-nodemanagers-proxies : 0 2021-07-30 11:20:01,421
> INFO flink-akka.actor.default-dispatcher-3
> org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] -
> ResourceManager akka.tcp://flink@qmatrix-web:37982/user/rpc/resourcemanager_0
> was granted leadership with fencing token
> 2021-07-30 11:20:01,818 INFO flink-akka.actor.default-dispatcher-4
> com.alibaba.druid.pool.DruidDataSource [] -
> {dataSource-1} inited 2021-07-30 11:20:02,648 WARN
> flink-akka.actor.default-dispatcher-4
> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap
> [] - Application failed unexpectedly:
> java.util.concurrent.CompletionException:
> org.apache.flink.client.deployment.application.ApplicationExecutionException:
> Could not execute application. at
> java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
> ~[?:1.8.0_161]at
> java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
> ~[?:1.8.0_161] at
> java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:943)
> ~[?:1.8.0_161] at
> java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:926)
> ~[?:1.8.0_161] at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> ~[?:1.8.0_161] at
> java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
> ~[?:1.8.0_161] at
> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.runApplicationEntryPoint(ApplicationDispatcherBootstrap.java:257)
> ~[flink-dist_2.11-1.13.0.jar:1.13.0] at
>
Application Mode 模式部署失败
您好：Flink Appliaction mode 模式启动失败，启动命令和错误日志如下./flink run-application -t
yarn-application \
-yD yarn.application.name="MyFlinkApp" \
-yD
yarn.provided.lib.dirs="hdfs://10.10.98.226:8020/user/myflink/flink-common-deps/libs/yarn-flink-1.13.0/lib/;hdfs://10.10.98.226:8020/user/myflink/flink-common-deps/libs/yarn-flink-1.13.0/plugins/"
\
/app/qmatrix/yarn-flink-1.13.0/qmatrix_jars/Sink/Hive/0.0.2/flink-hive-sink-3.0.0-jar-with-dependencies.jar
\
--config
'{"centerName":"rui","checkPointInterval":1000,"checkPointPath":"hdfs://10.10.98.226:8020/tmp/checkpoint66/Sink_rui_ruihiveha_HaHive","configProperties":{"tableKeyName":"{\"rui.fruitest1\":[\"show_integer\"]}"},"jobParallelism":3,"kafkaBootstrapServers":"qmatrix-1:9092,qmatrix-2:9092,qmatrix-3:9092","kafkaConsumerProperties":{"enable.auto.commit":"false","partition.assignment.strategy":"org.apache.kafka.clients.consumer.RoundRobinAssignor","max.poll.records":"500","group.id":"Sink_rui_ruihiveha_HaHive_g","auto.offset.reset":"earliest","session.timeout.ms":"15000","bootstrap.servers":"qmatrix-1:9092,qmatrix-2:9092,qmatrix-3:9092","max.partition.fetch.bytes":"1048576","max.poll.interval.ms":"180","heartbeat.interval.ms":"3000","isolation.level":"read_committed","auto.commit.interval.ms":"1000"},"managerJdbcProperties":{"url":"jdbc:mysql://qmatrix-mysql:3306/qmatrix","phyTimeoutMillis":"3","maxActive":"1","driverClassName":"com.mysql.jdbc.Driver","removeAbandoned":"true","minEvictableIdleTimeMillis":"3","username":"root","minIdle":"0","removeAbandonedTimeout":"3","timeBetweenEvictionRunsMillis":"3","password":"Cljslrl0620$","keepAlive":"false","initialSize":"0"},"passwordKey":"GOODLUCKEVERYONE","pipeName":"ruihiveha","targetConfig":{"rui.ruihiveha.rac80_forever.rac80_ruitest1.1627609625759.router.qmatrix":{"migratePartitionTime":1,"partitionField":"pt_dt","partitionInterval":1,"partitionTime":0,"partitionTimeField":"SHOW_TIMESTAMP","periodMigratePartitionTime":0,"targetSchema":"rui","targetTable":"fruitest1"}},"targetNodeName":"HaHive","targetNodeProperties":{"defaultSchema":"default","hiveConfDir":"/app/qmatrix/kerberos/HaHive","hiveUser":"root","password":"123456","url":"jdbc:hive2://10.10.98.42:1","username":"root"},"targetNodeType":"Hive","topologyId":1137,"topologyName":"Sink_rui_ruihiveha_HaHive","yarnJobManagerMemory":"1024MB","yarnTaskManagerMemory":"2048MB","zookeeperUrl":"qmatrix-1:2181,qmatrix-2:2181,qmatrix-3:2181"}'
\
--nodeType Hive \
--jobType Sink \
2021-07-30 11:20:01,402 INFO flink-akka.actor.default-dispatcher-3
org.apache.flink.runtime.externalresource.ExternalResourceUtils [] - Enabled
external resources: [] 2021-07-30 11:20:01,416 INFO
flink-akka.actor.default-dispatcher-3
org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl [] - Upper bound
of the thread pool size is 500 2021-07-30 11:20:01,418 INFO
flink-akka.actor.default-dispatcher-3
org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] -
yarn.client.max-cached-nodemanagers-proxies : 0 2021-07-30 11:20:01,421 INFO
flink-akka.actor.default-dispatcher-3
org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] -
ResourceManager akka.tcp://flink@qmatrix-web:37982/user/rpc/resourcemanager_0
was granted leadership with fencing token
2021-07-30 11:20:01,818 INFO flink-akka.actor.default-dispatcher-4
com.alibaba.druid.pool.DruidDataSource [] -
{dataSource-1} inited 2021-07-30 11:20:02,648 WARN
flink-akka.actor.default-dispatcher-4
org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap
[] - Application failed unexpectedly:
java.util.concurrent.CompletionException:
org.apache.flink.client.deployment.application.ApplicationExecutionException:
Could not execute application. at
java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
~[?:1.8.0_161]at
java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
~[?:1.8.0_161] at
java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:943)
~[?:1.8.0_161] at
java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:926)
~[?:1.8.0_161] at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
~[?:1.8.0_161] at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
~[?:1.8.0_161] at
org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.runApplicationEntryPoint(ApplicationDispatcherBootstrap.java:257)
~[flink-dist_2.11-1.13.0.jar:1.13.0] at
org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.lambda$runApplicationAsync$1(ApplicationDispatcherBootstrap.java:212)
~[flink-dist_2.11-1.13.0.jar:1.13.0] at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[?:1.8.0_161]at
Application Mode 模式部署失败
您好： 我的
