perjob模式,运行多个任务时,如果配置了high-availability.zookeeper.path.root: /flink 和high-availability.cluster-id: /v1.1/perjob-20191130 由于每个任务一个集群,所以每个集群在zookeeper上的目录都一样,导致yarn关联的flink相同,推荐删除上面两个配置,对于perjob模式。 发件人: pengchenglin 发送时间: 2019-12-02 15:54 收件人: user-zh@flink.apache.org 主题: perjob yarn页面链接flink出错 各位大佬: 有遇到过这种问题吗,perjob模式,4台机器,同时运行了多个任务,在yarn的管理页面上,点tracking ui,跳转到flink页面,都是同一个任务的flink页面。
flink配置如下: high-availability.zookeeper.client.max-retry-attempts: 10 historyserver.web.address: 0.0.0.0 state.checkpoints.num-retained: 3 historyserver.web.port: 8082 env.java.opts: -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/xx/oomdump -XX:ReservedCodeCacheSize=2048M high-availability.cluster-id: /v1.1/perjob-20191130 jobmanager.execution.failover-strategy: region #jobmanager.rpc.address: xx state.savepoints.dir: hdfs://xx:8020/flink/v1.1/mxm1130_savepoint high-availability.zookeeper.path.root: /flink high-availability.zookeeper.client.session-timeout: 300000 taskmanager.registration.timeout: 20 min high-availability.storageDir: hdfs://xx:8020/flink/v1.1/mxm_ha/ task.cancellation.timeout: 0 taskmanager.network.numberOfBuffers: 10240 parallelism.default: 8 taskmanager.numberOfTaskSlots: 8 akka.ask.timeout: 600 s historyserver.archive.fs.dir: hdfs://xx:8020/flink/v1.1/completed-jobs-mxm1130/ jobmanager.heap.size: 2048m jobmanager.archive.fs.dir: hdfs://xx:8020/flink/v1.1/mxm1130completed-jobs/ heartbeat.timeout: 300000 restart-strategy.fixed-delay.attempts: 360 high-availability.zookeeper.client.connection-timeout: 60000 historyserver.archive.fs.refresh-interval: 10000 jobmanager.rpc.port: 6123 jobstore.expiration-time: 14400 #rest.port: 8983 high-availability.zookeeper.quorum: xx:6666,xx:6666,xx:6666 restart-strategy.fixed-delay.delay: 1 s high-availability: zookeeper state.backend: filesystem restart-strategy: fixed-delay taskmanager.heap.size: 8192m akka.client.timeout: 600 s state.checkpoints.dir: hdfs://xx:8020/flink/v1.1/mxm1130_checkpoint