Re: Re: flink on yarn任务启动报错 The assigned slot container_e10_1579661300080_0005_01_000002_0 was removed.
没有log，只有err和out，out为空 zjfpla...@hotmail.com 发件人： tison<mailto:wander4...@gmail.com> 发送时间： 2020-01-24 10:03 收件人： user-zh<mailto:user-zh@flink.apache.org> 抄送： zhisheng2018<mailto:zhisheng2...@gmail.com> 主题： Re: Re: flink on yarn任务启动报错 The assigned slot container_e10_1579661300080_0005_01_02_0 was removed. 你上面的是 taskmanager.err，需要的是 taskmanager.log Best, tison. 郑 洁锋 于2020年1月23日周四 下午10:22写道： > 之前挂过 后面启动的时候 是checkpoints的文件丢了？ 你是这个意思吗？ > > > zjfpla...@hotmail.com > > 发件人： zhisheng<mailto:zhisheng2...@gmail.com> > 发送时间： 2020-01-22 16:45 > 收件人： user-zh<mailto:user-zh@flink.apache.org> > 主题： Re: flink on yarn任务启动报错 The assigned slot > container_e10_1579661300080_0005_01_02_0 was removed. > 应该是你作业之前挂过了 > > 郑 洁锋 于2020年1月22日周三 上午11:16写道： > > > 大家好， > >flink on yarn任务启动时，发现报错了The assigned slot > > container_e10_1579661300080_0005_01_02_0 was removed. > >环境：flink1.8.1，cdh5.14.2，kafka0.10，jdk1.8.0_241 > > > > flink版本为1.8.1，yarn上的日志： > > > > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: > > > > > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: Starting > > YarnJobClusterEntrypoint (Version: , Rev:7297bac, > Date:24.06.2019 > > @ 23:04:28 CST) > > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: OS current user: > > cloudera-scm > > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: Current > > Hadoop/Kerberos user: root > > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: JVM: Java > > HotSpot(TM) 64-Bit Server VM - Oracle Corporation - 1.8/25.241-b07 > > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: Maximum heap size: > > 406 MiBytes > > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: JAVA_HOME: > > /usr/java/default > > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: Hadoop version: > 2.6.5 > > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: JVM Options: > > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: -Xms424m > > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: -Xmx424m > > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: Program Arguments: > > (none) > > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: Classpath: > > > core-1.8.0_release.jar:flink-shaded-hadoop-2-uber-2.6.5-7.0.jar:kafka10-source-1.8.0_release.jar:log4j-1.2.17.jar:mysql-all-side-1.8.0_release.jar:mysql-sink-1.8.0_release.jar:slf4j-log4j12-1.7.15.jar:sql.launcher-1.0-SNAPSHOT.jar:flink.jar:flink-conf.yaml:job.graph::/etc/hadoop/conf.cloudera.yarn:/run/cloudera-scm-agent/process/1129-yarn-NODEMANAGER:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-annotations.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-auth.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-aws.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-azure-datalake.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-common-tests.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-common.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-nfs.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-nfs-2.6.0-cdh5.14.2.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-common-2.6.0-cdh5.14.2.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-common-2.6.0-cdh5.14.2-tests.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-azure-datalake-2.6.0-cdh5.14.2.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-aws-2.6.0-cdh5.14.2.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-auth-2.6.0-cdh5.14.2.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-annotations-2.6.0-cdh5.14.2.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/parquet-format.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/parquet-format-sources.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/parquet-format-javadoc.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/parquet-tools.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/parquet-thrift.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/parquet-test-hadoop2.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/parquet-scrooge_2.10.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/parquet-scala_2.10.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/parquet-protobuf.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/parquet-pi
Re: Re: flink on yarn任务启动报错 The assigned slot container_e10_1579661300080_0005_01_000002_0 was removed.
之前挂过 后面启动的时候 是checkpoints的文件丢了？ 你是这个意思吗？ zjfpla...@hotmail.com 发件人： zhisheng<mailto:zhisheng2...@gmail.com> 发送时间： 2020-01-22 16:45 收件人： user-zh<mailto:user-zh@flink.apache.org> 主题： Re: flink on yarn任务启动报错 The assigned slot container_e10_1579661300080_0005_01_02_0 was removed. 应该是你作业之前挂过了 郑 洁锋 于2020年1月22日周三 上午11:16写道： > 大家好， >flink on yarn任务启动时，发现报错了The assigned slot > container_e10_1579661300080_0005_01_02_0 was removed. >环境：flink1.8.1，cdh5.14.2，kafka0.10，jdk1.8.0_241 > > flink版本为1.8.1，yarn上的日志： > > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: > > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: Starting > YarnJobClusterEntrypoint (Version: , Rev:7297bac, Date:24.06.2019 > @ 23:04:28 CST) > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: OS current user: > cloudera-scm > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: Current > Hadoop/Kerberos user: root > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: JVM: Java > HotSpot(TM) 64-Bit Server VM - Oracle Corporation - 1.8/25.241-b07 > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: Maximum heap size: > 406 MiBytes > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: JAVA_HOME: > /usr/java/default > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: Hadoop version: 2.6.5 > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: JVM Options: > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: -Xms424m > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: -Xmx424m > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: Program Arguments: > (none) > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: Classpath: > core-1.8.0_release.jar:flink-shaded-hadoop-2-uber-2.6.5-7.0.jar:kafka10-source-1.8.0_release.jar:log4j-1.2.17.jar:mysql-all-side-1.8.0_release.jar:mysql-sink-1.8.0_release.jar:slf4j-log4j12-1.7.15.jar:sql.launcher-1.0-SNAPSHOT.jar:flink.jar:flink-conf.yaml:job.graph::/etc/hadoop/conf.cloudera.yarn:/run/cloudera-scm-agent/process/1129-yarn-NODEMANAGER:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-annotations.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-auth.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-aws.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-azure-datalake.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-common-tests.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-common.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-nfs.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-nfs-2.6.0-cdh5.14.2.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-common-2.6.0-cdh5.14.2.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-common-2.6.0-cdh5.14.2-tests.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-azure-datalake-2.6.0-cdh5.14.2.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-aws-2.6.0-cdh5.14.2.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-auth-2.6.0-cdh5.14.2.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-annotations-2.6.0-cdh5.14.2.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/parquet-format.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/parquet-format-sources.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/parquet-format-javadoc.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/parquet-tools.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/parquet-thrift.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/parquet-test-hadoop2.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/parquet-scrooge_2.10.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/parquet-scala_2.10.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/parquet-protobuf.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/parquet-pig.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/parquet-pig-bundle.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/parquet-jackson.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/parquet-hadoop.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/parquet-hadoop-bundle.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/parquet-generator.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/parquet-encoding.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/parquet-common.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/parquet-column.jar:/opt/cloudera/parcels/CDH-5.14
Re: Re: flink on yarn任务启动报错 The assigned slot container_e10_1579661300080_0005_01_000002_0 was removed.
日志已经在前面的邮件里面了 zjfpla...@hotmail.com 发件人： tison<mailto:wander4...@gmail.com> 发送时间： 2020-01-22 12:10 收件人： user-zh<mailto:user-zh@flink.apache.org> 主题： Re: Re: flink on yarn任务启动报错 The assigned slot container_e10_1579661300080_0005_01_02_0 was removed. 那你看下 TM 那台机器上的 TM 日志，从 JM 端来看 TM 曾经成功起来过并注册了自己，你看看 TM 是怎么挂的或者别的什么情况 Best, tison. 郑 洁锋 于2020年1月22日周三 上午11:54写道： > TM没有起来，服务器本身内存cpu都是够的，还很空闲 > > > zjfpla...@hotmail.com > > 发件人： tison<mailto:wander4...@gmail.com> > 发送时间： 2020-01-22 11:25 > 收件人： user-zh<mailto:user-zh@flink.apache.org> > 主题： Re: flink on yarn任务启动报错 The assigned slot > container_e10_1579661300080_0005_01_02_0 was removed. > 20/01/22 11:08:49 INFO yarn.YarnResourceManager: Closing TaskExecutor > connection container_e10_1579661300080_0005_01_02 because: The > heartbeat of TaskManager with id container_e10_1579661300080_0005_01_02 > timed out. > > 你请求资源的时候把 slot 请求发到这台机器上了，然后它心跳超时了，你看看 TM 有没有正常起来，有没有资源不够或者挂了 > > Best, > tison. > > > 郑 洁锋 于2020年1月22日周三 上午11:16写道： > > > 大家好， > >flink on yarn任务启动时，发现报错了The assigned slot > > container_e10_1579661300080_0005_01_02_0 was removed. > >环境：flink1.8.1，cdh5.14.2，kafka0.10，jdk1.8.0_241 > > > > flink版本为1.8.1，yarn上的日志： > > > > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: > > > > > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: Starting > > YarnJobClusterEntrypoint (Version: , Rev:7297bac, > Date:24.06.2019 > > @ 23:04:28 CST) > > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: OS current user: > > cloudera-scm > > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: Current > > Hadoop/Kerberos user: root > > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: JVM: Java > > HotSpot(TM) 64-Bit Server VM - Oracle Corporation - 1.8/25.241-b07 > > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: Maximum heap size: > > 406 MiBytes > > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: JAVA_HOME: > > /usr/java/default > > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: Hadoop version: > 2.6.5 > > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: JVM Options: > > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: -Xms424m > > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: -Xmx424m > > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: Program Arguments: > > (none) > > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: Classpath: > > > core-1.8.0_release.jar:flink-shaded-hadoop-2-uber-2.6.5-7.0.jar:kafka10-source-1.8.0_release.jar:log4j-1.2.17.jar:mysql-all-side-1.8.0_release.jar:mysql-sink-1.8.0_release.jar:slf4j-log4j12-1.7.15.jar:sql.launcher-1.0-SNAPSHOT.jar:flink.jar:flink-conf.yaml:job.graph::/etc/hadoop/conf.cloudera.yarn:/run/cloudera-scm-agent/process/1129-yarn-NODEMANAGER:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-annotations.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-auth.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-aws.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-azure-datalake.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-common-tests.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-common.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-nfs.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-nfs-2.6.0-cdh5.14.2.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-common-2.6.0-cdh5.14.2.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-common-2.6.0-cdh5.14.2-tests.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-azure-datalake-2.6.0-cdh5.14.2.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-aws-2.6.0-cdh5.14.2.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-auth-2.6.0-cdh5.14.2.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-annotations-2.6.0-cdh5.14.2.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/parquet-format.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/parquet-format-sources.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/parquet-format-javadoc.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/parquet-tools.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/parquet-thrift.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/parquet-test-hadoop2.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.
Re: Re: flink on yarn任务启动报错 The assigned slot container_e10_1579661300080_0005_01_000002_0 was removed.
TM没有起来，服务器本身内存cpu都是够的，还很空闲 zjfpla...@hotmail.com 发件人： tison<mailto:wander4...@gmail.com> 发送时间： 2020-01-22 11:25 收件人： user-zh<mailto:user-zh@flink.apache.org> 主题： Re: flink on yarn任务启动报错 The assigned slot container_e10_1579661300080_0005_01_02_0 was removed. 20/01/22 11:08:49 INFO yarn.YarnResourceManager: Closing TaskExecutor connection container_e10_1579661300080_0005_01_02 because: The heartbeat of TaskManager with id container_e10_1579661300080_0005_01_02 timed out. 你请求资源的时候把 slot 请求发到这台机器上了，然后它心跳超时了，你看看 TM 有没有正常起来，有没有资源不够或者挂了 Best, tison. 郑 洁锋 于2020年1月22日周三 上午11:16写道： > 大家好， >flink on yarn任务启动时，发现报错了The assigned slot > container_e10_1579661300080_0005_01_02_0 was removed. >环境：flink1.8.1，cdh5.14.2，kafka0.10，jdk1.8.0_241 > > flink版本为1.8.1，yarn上的日志： > > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: > > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: Starting > YarnJobClusterEntrypoint (Version: , Rev:7297bac, Date:24.06.2019 > @ 23:04:28 CST) > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: OS current user: > cloudera-scm > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: Current > Hadoop/Kerberos user: root > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: JVM: Java > HotSpot(TM) 64-Bit Server VM - Oracle Corporation - 1.8/25.241-b07 > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: Maximum heap size: > 406 MiBytes > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: JAVA_HOME: > /usr/java/default > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: Hadoop version: 2.6.5 > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: JVM Options: > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: -Xms424m > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: -Xmx424m > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: Program Arguments: > (none) > 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: Classpath: > core-1.8.0_release.jar:flink-shaded-hadoop-2-uber-2.6.5-7.0.jar:kafka10-source-1.8.0_release.jar:log4j-1.2.17.jar:mysql-all-side-1.8.0_release.jar:mysql-sink-1.8.0_release.jar:slf4j-log4j12-1.7.15.jar:sql.launcher-1.0-SNAPSHOT.jar:flink.jar:flink-conf.yaml:job.graph::/etc/hadoop/conf.cloudera.yarn:/run/cloudera-scm-agent/process/1129-yarn-NODEMANAGER:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-annotations.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-auth.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-aws.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-azure-datalake.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-common-tests.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-common.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-nfs.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-nfs-2.6.0-cdh5.14.2.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-common-2.6.0-cdh5.14.2.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-common-2.6.0-cdh5.14.2-tests.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-azure-datalake-2.6.0-cdh5.14.2.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-aws-2.6.0-cdh5.14.2.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-auth-2.6.0-cdh5.14.2.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/hadoop-annotations-2.6.0-cdh5.14.2.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/parquet-format.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/parquet-format-sources.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/parquet-format-javadoc.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/parquet-tools.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/parquet-thrift.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/parquet-test-hadoop2.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/parquet-scrooge_2.10.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/parquet-scala_2.10.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/parquet-protobuf.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/parquet-pig.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/parquet-pig-bundle.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/parquet-jackson.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/parquet-hadoop.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/parquet-hadoop-bundle.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/parquet-generator.jar:/
flink on yarn任务启动报错 The assigned slot container_e10_1579661300080_0005_01_000002_0 was removed.
大家好， flink on yarn任务启动时，发现报错了The assigned slot container_e10_1579661300080_0005_01_02_0 was removed. 环境：flink1.8.1，cdh5.14.2，kafka0.10，jdk1.8.0_241 flink版本为1.8.1，yarn上的日志： 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: Starting YarnJobClusterEntrypoint (Version: , Rev:7297bac, Date:24.06.2019 @ 23:04:28 CST) 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: OS current user: cloudera-scm 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: Current Hadoop/Kerberos user: root 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: JVM: Java HotSpot(TM) 64-Bit Server VM - Oracle Corporation - 1.8/25.241-b07 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: Maximum heap size: 406 MiBytes 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: JAVA_HOME: /usr/java/default 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: Hadoop version: 2.6.5 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: JVM Options: 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: -Xms424m 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: -Xmx424m 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: Program Arguments: (none) 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: Classpath:
Re: Re: MiniCluster问题
我是不是可以理解为是local的cluster，本身会去启动一个flink cluster（当前服务器模拟分布式环境），无需单独部署一个flink集群
zjfpla...@hotmail.com
发件人： tison<mailto:wander4...@gmail.com>
发送时间： 2020-01-16 14:29
收件人： user-zh<mailto:user-zh@flink.apache.org>
主题： Re: Re: MiniCluster问题
你这完全是把几个概念混在一起了，MiniCluster 就是一个集群，是一个内部的部署模式，跟 standalone
是平行的概念。我看不懂你要干什么，但是就你的问题来说，我上面说了，就是你 MiniCluster new 出来之后没
start。但我不确定这个是不是你要的效果，MiniCluster 和 standalone 是平行的两种东西。
Best,
tison.
郑 洁锋 于2020年1月16日周四 下午2:27写道：
> 因为我这边看他是standalone，然后看到flink官方文档里面部署模式也有standalone模式，所以按照那个模式部署后来尝试
>
>
> zjfpla...@hotmail.com
>
> 发件人： 郑 洁锋<mailto:zjfpla...@hotmail.com>
> 发送时间： 2020-01-16 14:24
> 收件人： user-zh<mailto:user-zh@flink.apache.org>
> 主题： Re: Re: MiniCluster问题
> 这是完整的到启动的代码
>
> public class ClusterClientFactory {
>
> public static ClusterClient createClusterClient(Options
> launcherOptions) throws Exception {
> String mode = launcherOptions.getMode();
> if(mode.equals(ClusterMode.standalone.name())) {
> return createStandaloneClient(launcherOptions);
> } else if(mode.equals(ClusterMode.yarn.name())) {
> return createYarnClient(launcherOptions,mode);
> }
> throw new IllegalArgumentException("Unsupported cluster client
> type: ");
> }
>
> public static ClusterClient createStandaloneClient(Options
> launcherOptions) throws Exception {
> String flinkConfDir = launcherOptions.getFlinkconf();
> Configuration config =
> GlobalConfiguration.loadConfiguration(flinkConfDir);
> MiniClusterConfiguration.Builder configBuilder = new
> MiniClusterConfiguration.Builder();
> configBuilder.setConfiguration(config);
> MiniCluster miniCluster = new MiniCluster(configBuilder.build());
> MiniClusterClient clusterClient = new MiniClusterClient(config,
> miniCluster);
> LeaderConnectionInfo connectionInfo =
> clusterClient.getClusterConnectionInfo();
> InetSocketAddress address =
> AkkaUtils.getInetSocketAddressFromAkkaURL(connectionInfo.getAddress());
> config.setString(JobManagerOptions.ADDRESS,
> address.getAddress().getHostName());
> config.setInteger(JobManagerOptions.PORT, address.getPort());
> clusterClient.setDetached(true);
> return clusterClient;
> }
>
>
> 启动类中：
>
> ClusterClient clusterClient =
> ClusterClientFactory.createClusterClient(launcherOptions);
> clusterClient.run(program, 1);
> clusterClient.shutdown();
>
>
> zjfpla...@hotmail.com
>
> 发件人： tison<mailto:wander4...@gmail.com>
> 发送时间： 2020-01-16 13:31
> 收件人： user-zh<mailto:user-zh@flink.apache.org>
> 主题： Re: Re: MiniCluster问题
> MiniCluster miniCluster = new MiniCluster(configBuilder.build());
>
> miniCluster.start();
>
>
> MiniClusterClient clusterClient = new MiniClusterClient(config,
> miniCluster)
> ;
>
> Best,
> tison.
>
>
> tison 于2020年1月16日周四 下午1:30写道：
>
> > 跟集群无关
> > Best,
> > tison.
> >
> >
> > tison 于2020年1月16日周四 下午1:30写道：
> >
> >> 1. 这是个内部实现类，文档是一个问题，但最好不要直接依赖
> >>
> >> 2. ？你截图的代码片段在哪？我看是你自己代码创建使用的啊
> >>
> >> Best,
> >> tison.
> >>
> >>
> >> 郑 洁锋 于2020年1月16日周四 下午1:18写道：
> >>
> >>> MiniCluster有没有相关文档，官方文档里面看到部署模式里面没有相关的内容？
> >>> 我是通过bin/start-cluster.sh启动的flink standalone集群
> >>>
> >>>
> >>>
> >>> zjfpla...@hotmail.com
> >>>
> >>> 发件人： tison<mailto:wander4...@gmail.com>
> >>> 发送时间： 2020-01-16 12:39
> >>> 收件人： user-zh<mailto:user-zh@flink.apache.org>
> >>> 主题： Re: MiniCluster问题
> >>> 你 MiniCluster 要 start 啊（x
> >>>
> >>> Best,
> >>> tison.
> >>>
> >>>
> >>> 郑 洁锋 于2020年1月16日周四 上午11:38写道：
> >>>
> >>> > MiniCluster代码执行过程中报错：
> >>> >
> >>> > SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> >>> > SLF4J: Defaulting to no-operation (NOP) logger implementation
> >>> > SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for
> >>> further details.
> >>> > Exception in thread "main" java.lang.IllegalStateException:
> >>> MiniCluster is not yet running.
> >>> > at
> >>> org.apache.flink.util.Preconditions.checkState(Preconditions.java:195)
> >>> > at
> >>&g
Re: Re: MiniCluster问题
因为我这边看他是standalone，然后看到flink官方文档里面部署模式也有standalone模式，所以按照那个模式部署后来尝试
zjfpla...@hotmail.com
发件人： 郑 洁锋<mailto:zjfpla...@hotmail.com>
发送时间： 2020-01-16 14:24
收件人： user-zh<mailto:user-zh@flink.apache.org>
主题： Re: Re: MiniCluster问题
这是完整的到启动的代码
public class ClusterClientFactory {
public static ClusterClient createClusterClient(Options launcherOptions)
throws Exception {
String mode = launcherOptions.getMode();
if(mode.equals(ClusterMode.standalone.name())) {
return createStandaloneClient(launcherOptions);
} else if(mode.equals(ClusterMode.yarn.name())) {
return createYarnClient(launcherOptions,mode);
}
throw new IllegalArgumentException("Unsupported cluster client type: ");
}
public static ClusterClient createStandaloneClient(Options launcherOptions)
throws Exception {
String flinkConfDir = launcherOptions.getFlinkconf();
Configuration config =
GlobalConfiguration.loadConfiguration(flinkConfDir);
MiniClusterConfiguration.Builder configBuilder = new
MiniClusterConfiguration.Builder();
configBuilder.setConfiguration(config);
MiniCluster miniCluster = new MiniCluster(configBuilder.build());
MiniClusterClient clusterClient = new MiniClusterClient(config,
miniCluster);
LeaderConnectionInfo connectionInfo =
clusterClient.getClusterConnectionInfo();
InetSocketAddress address =
AkkaUtils.getInetSocketAddressFromAkkaURL(connectionInfo.getAddress());
config.setString(JobManagerOptions.ADDRESS,
address.getAddress().getHostName());
config.setInteger(JobManagerOptions.PORT, address.getPort());
clusterClient.setDetached(true);
return clusterClient;
}
启动类中：
ClusterClient clusterClient =
ClusterClientFactory.createClusterClient(launcherOptions);
clusterClient.run(program, 1);
clusterClient.shutdown();
zjfpla...@hotmail.com
发件人： tison<mailto:wander4...@gmail.com>
发送时间： 2020-01-16 13:31
收件人： user-zh<mailto:user-zh@flink.apache.org>
主题： Re: Re: MiniCluster问题
MiniCluster miniCluster = new MiniCluster(configBuilder.build());
miniCluster.start();
MiniClusterClient clusterClient = new MiniClusterClient(config, miniCluster)
;
Best,
tison.
tison 于2020年1月16日周四 下午1:30写道：
> 跟集群无关
> Best,
> tison.
>
>
> tison 于2020年1月16日周四 下午1:30写道：
>
>> 1. 这是个内部实现类，文档是一个问题，但最好不要直接依赖
>>
>> 2. ？你截图的代码片段在哪？我看是你自己代码创建使用的啊
>>
>> Best,
>> tison.
>>
>>
>> 郑 洁锋 于2020年1月16日周四 下午1:18写道：
>>
>>> MiniCluster有没有相关文档，官方文档里面看到部署模式里面没有相关的内容？
>>> 我是通过bin/start-cluster.sh启动的flink standalone集群
>>>
>>>
>>>
>>> zjfpla...@hotmail.com
>>>
>>> 发件人： tison<mailto:wander4...@gmail.com>
>>> 发送时间： 2020-01-16 12:39
>>> 收件人： user-zh<mailto:user-zh@flink.apache.org>
>>> 主题： Re: MiniCluster问题
>>> 你 MiniCluster 要 start 啊（x
>>>
>>> Best,
>>> tison.
>>>
>>>
>>> 郑 洁锋 于2020年1月16日周四 上午11:38写道：
>>>
>>> > MiniCluster代码执行过程中报错：
>>> >
>>> > SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
>>> > SLF4J: Defaulting to no-operation (NOP) logger implementation
>>> > SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for
>>> further details.
>>> > Exception in thread "main" java.lang.IllegalStateException:
>>> MiniCluster is not yet running.
>>> > at
>>> org.apache.flink.util.Preconditions.checkState(Preconditions.java:195)
>>> > at
>>> org.apache.flink.runtime.minicluster.MiniCluster.getHighAvailabilityServices(MiniCluster.java:223)
>>> > at
>>> org.apache.flink.client.program.MiniClusterClient.(MiniClusterClient.java:61)
>>> > at
>>> com.dtstack.flink.sql.launcher.ClusterClientFactory.createStandaloneClient(ClusterClientFactory.java:82)
>>> > at
>>> com.dtstack.flink.sql.launcher.ClusterClientFactory.createClusterClient(ClusterClientFactory.java:69)
>>> > at
>>> com.dtstack.flink.sql.launcher.LauncherMain.main(LauncherMain.java:99)
>>> >
>>> > 报错段代码如下：
>>> >
>>> > Configuration config =
>>> GlobalConfiguration.loadConfiguration(flinkConfDir);
>>> > MiniClusterConfiguration.Builder configBuilder = new
>>> MiniClusterConfiguration.Builder();
>>> > configBuilder.setConfiguration(config);
>>> > MiniCluster miniCluster = new MiniCluster(configBuilder.build());
>>> > MiniClusterClient clusterClient = new MiniClusterClient(config,
>>> miniCluster);
>>> >
>>> > 其中flinkConfDir为/opt/flink/conf
>>> >
>>> >
>>> > flink standalone HA集群信息如下：
>>> > --
>>> > zjfpla...@hotmail.com
>>> >
>>> >
>>> >
>>>
>>
Re: Re: MiniCluster问题
这是完整的到启动的代码
public class ClusterClientFactory {
public static ClusterClient createClusterClient(Options launcherOptions)
throws Exception {
String mode = launcherOptions.getMode();
if(mode.equals(ClusterMode.standalone.name())) {
return createStandaloneClient(launcherOptions);
} else if(mode.equals(ClusterMode.yarn.name())) {
return createYarnClient(launcherOptions,mode);
}
throw new IllegalArgumentException("Unsupported cluster client type: ");
}
public static ClusterClient createStandaloneClient(Options launcherOptions)
throws Exception {
String flinkConfDir = launcherOptions.getFlinkconf();
Configuration config =
GlobalConfiguration.loadConfiguration(flinkConfDir);
MiniClusterConfiguration.Builder configBuilder = new
MiniClusterConfiguration.Builder();
configBuilder.setConfiguration(config);
MiniCluster miniCluster = new MiniCluster(configBuilder.build());
MiniClusterClient clusterClient = new MiniClusterClient(config,
miniCluster);
LeaderConnectionInfo connectionInfo =
clusterClient.getClusterConnectionInfo();
InetSocketAddress address =
AkkaUtils.getInetSocketAddressFromAkkaURL(connectionInfo.getAddress());
config.setString(JobManagerOptions.ADDRESS,
address.getAddress().getHostName());
config.setInteger(JobManagerOptions.PORT, address.getPort());
clusterClient.setDetached(true);
return clusterClient;
}
启动类中：
ClusterClient clusterClient =
ClusterClientFactory.createClusterClient(launcherOptions);
clusterClient.run(program, 1);
clusterClient.shutdown();
zjfpla...@hotmail.com
发件人： tison<mailto:wander4...@gmail.com>
发送时间： 2020-01-16 13:31
收件人： user-zh<mailto:user-zh@flink.apache.org>
主题： Re: Re: MiniCluster问题
MiniCluster miniCluster = new MiniCluster(configBuilder.build());
miniCluster.start();
MiniClusterClient clusterClient = new MiniClusterClient(config, miniCluster)
;
Best,
tison.
tison 于2020年1月16日周四 下午1:30写道：
> 跟集群无关
> Best,
> tison.
>
>
> tison 于2020年1月16日周四 下午1:30写道：
>
>> 1. 这是个内部实现类，文档是一个问题，但最好不要直接依赖
>>
>> 2. ？你截图的代码片段在哪？我看是你自己代码创建使用的啊
>>
>> Best,
>> tison.
>>
>>
>> 郑 洁锋 于2020年1月16日周四 下午1:18写道：
>>
>>> MiniCluster有没有相关文档，官方文档里面看到部署模式里面没有相关的内容？
>>> 我是通过bin/start-cluster.sh启动的flink standalone集群
>>>
>>>
>>>
>>> zjfpla...@hotmail.com
>>>
>>> 发件人： tison<mailto:wander4...@gmail.com>
>>> 发送时间： 2020-01-16 12:39
>>> 收件人： user-zh<mailto:user-zh@flink.apache.org>
>>> 主题： Re: MiniCluster问题
>>> 你 MiniCluster 要 start 啊（x
>>>
>>> Best,
>>> tison.
>>>
>>>
>>> 郑 洁锋 于2020年1月16日周四 上午11:38写道：
>>>
>>> > MiniCluster代码执行过程中报错：
>>> >
>>> > SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
>>> > SLF4J: Defaulting to no-operation (NOP) logger implementation
>>> > SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for
>>> further details.
>>> > Exception in thread "main" java.lang.IllegalStateException:
>>> MiniCluster is not yet running.
>>> > at
>>> org.apache.flink.util.Preconditions.checkState(Preconditions.java:195)
>>> > at
>>> org.apache.flink.runtime.minicluster.MiniCluster.getHighAvailabilityServices(MiniCluster.java:223)
>>> > at
>>> org.apache.flink.client.program.MiniClusterClient.(MiniClusterClient.java:61)
>>> > at
>>> com.dtstack.flink.sql.launcher.ClusterClientFactory.createStandaloneClient(ClusterClientFactory.java:82)
>>> > at
>>> com.dtstack.flink.sql.launcher.ClusterClientFactory.createClusterClient(ClusterClientFactory.java:69)
>>> > at
>>> com.dtstack.flink.sql.launcher.LauncherMain.main(LauncherMain.java:99)
>>> >
>>> > 报错段代码如下：
>>> >
>>> > Configuration config =
>>> GlobalConfiguration.loadConfiguration(flinkConfDir);
>>> > MiniClusterConfiguration.Builder configBuilder = new
>>> MiniClusterConfiguration.Builder();
>>> > configBuilder.setConfiguration(config);
>>> > MiniCluster miniCluster = new MiniCluster(configBuilder.build());
>>> > MiniClusterClient clusterClient = new MiniClusterClient(config,
>>> miniCluster);
>>> >
>>> > 其中flinkConfDir为/opt/flink/conf
>>> >
>>> >
>>> > flink standalone HA集群信息如下：
>>> > --
>>> > zjfpla...@hotmail.com
>>> >
>>> >
>>> >
>>>
>>
Re: Re: MiniCluster问题
MiniCluster有没有相关文档，官方文档里面看到部署模式里面没有相关的内容？ 我是通过bin/start-cluster.sh启动的flink standalone集群 zjfpla...@hotmail.com 发件人： tison<mailto:wander4...@gmail.com> 发送时间： 2020-01-16 12:39 收件人： user-zh<mailto:user-zh@flink.apache.org> 主题： Re: MiniCluster问题 你 MiniCluster 要 start 啊（x Best, tison. 郑 洁锋 于2020年1月16日周四 上午11:38写道： > MiniCluster代码执行过程中报错： > > SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". > SLF4J: Defaulting to no-operation (NOP) logger implementation > SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further > details. > Exception in thread "main" java.lang.IllegalStateException: MiniCluster is > not yet running. > at > org.apache.flink.util.Preconditions.checkState(Preconditions.java:195) > at > org.apache.flink.runtime.minicluster.MiniCluster.getHighAvailabilityServices(MiniCluster.java:223) > at > org.apache.flink.client.program.MiniClusterClient.(MiniClusterClient.java:61) > at > com.dtstack.flink.sql.launcher.ClusterClientFactory.createStandaloneClient(ClusterClientFactory.java:82) > at > com.dtstack.flink.sql.launcher.ClusterClientFactory.createClusterClient(ClusterClientFactory.java:69) > at > com.dtstack.flink.sql.launcher.LauncherMain.main(LauncherMain.java:99) > > 报错段代码如下： > > Configuration config = GlobalConfiguration.loadConfiguration(flinkConfDir); > MiniClusterConfiguration.Builder configBuilder = new > MiniClusterConfiguration.Builder(); > configBuilder.setConfiguration(config); > MiniCluster miniCluster = new MiniCluster(configBuilder.build()); > MiniClusterClient clusterClient = new MiniClusterClient(config, miniCluster); > > 其中flinkConfDir为/opt/flink/conf > > > flink standalone HA集群信息如下： > -- > zjfpla...@hotmail.com > > >
MiniCluster问题
MiniCluster代码执行过程中报错： SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. Exception in thread "main" java.lang.IllegalStateException: MiniCluster is not yet running. at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195) at org.apache.flink.runtime.minicluster.MiniCluster.getHighAvailabilityServices(MiniCluster.java:223) at org.apache.flink.client.program.MiniClusterClient.(MiniClusterClient.java:61) at com.dtstack.flink.sql.launcher.ClusterClientFactory.createStandaloneClient(ClusterClientFactory.java:82) at com.dtstack.flink.sql.launcher.ClusterClientFactory.createClusterClient(ClusterClientFactory.java:69) at com.dtstack.flink.sql.launcher.LauncherMain.main(LauncherMain.java:99) 报错段代码如下： Configuration config = GlobalConfiguration.loadConfiguration(flinkConfDir); MiniClusterConfiguration.Builder configBuilder = new MiniClusterConfiguration.Builder(); configBuilder.setConfiguration(config); MiniCluster miniCluster = new MiniCluster(configBuilder.build()); MiniClusterClient clusterClient = new MiniClusterClient(config, miniCluster); 其中flinkConfDir为/opt/flink/conf flink standalone HA集群信息如下： [cid:_Foxmail.1@1b958fff-6637-1c63-42a4-5dff5b86583b] zjfpla...@hotmail.com
Re: Re: flink on yarn jdk版本问题
) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:561) at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:929) at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.flink.runtime.rest.util.RestClientException: [Internal server error., ] at org.apache.flink.runtime.rest.RestClient.parseResponse(RestClient.java:389) at org.apache.flink.runtime.rest.RestClient.lambda$submitRequest$3(RestClient.java:373) at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:952) at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:926) ... 4 more 有尝试过添加配置： akka.ask.timeout: 20s web.timeout: 2 还是一样的报错（超时时间生效了，日志中变更了）。 而且dashboard也出现了问题： [cid:_Foxmail.1@8d04627a-7e3d-784d-7c66-eae463c0d643] 这个有人遇到过吗，我搜索还有说是flink版本的问题，但是我已经是最新的1.9.1版本了,这个是还未修复的bug还是其他的问题？ zjfpla...@hotmail.com 发件人： Benchao Li<mailto:libenc...@gmail.com> 发送时间： 2020-01-15 12:56 收件人： user-zh<mailto:user-zh@flink.apache.org>; zjfplayer<mailto:zjfpla...@hotmail.com> 主题： Re: flink on yarn jdk版本问题 Hi , Flink也支持传递环境变量的，也可以尝试一下：https://ci.apache.org/projects/flink/flink-docs-master/ops/config.html#configuration-runtime-environment-variables 郑 洁锋 mailto:zjfpla...@hotmail.com>> 于2020年1月15日周三 上午11:34写道： Hi， 我们使用了TDH中的yarn（hadoop2.7，jdk1.7），原先使用spark的时候，也遇到过jdk版本最低1.8的情况，我们是通过如下方式解决的（考虑到现场公用TDH集群不能更改）： spark-submit中可以通过添加参数 --conf "spark.yarn.appMasterEnv.JAVA_HOME=/usr/java/jdk1.8.0_25/" --conf "spark.executorEnv.JAVA_HOME=/usr/java/jdk1.8.0_25/"来指定当前spark任务特定在yarn中执行时的jdk目录 请问下flink启动&提交任务时（bin/yarn-session.sh run）有没有参数可以指定flink特定使用yarn时的jdk目录，flink中是否有这样的功能？（不影响其他yarn上应用正常运行） zjfpla...@hotmail.com<mailto:zjfpla...@hotmail.com> -- Benchao Li School of Electronics Engineering and Computer Science, Peking University Tel:+86-15650713730 Email: libenc...@gmail.com<mailto:libenc...@gmail.com>; libenc...@pku.edu.cn<mailto:libenc...@pku.edu.cn>
flink on yarn jdk版本问题
Hi， 我们使用了TDH中的yarn（hadoop2.7，jdk1.7），原先使用spark的时候，也遇到过jdk版本最低1.8的情况，我们是通过如下方式解决的（考虑到现场公用TDH集群不能更改）： spark-submit中可以通过添加参数 --conf "spark.yarn.appMasterEnv.JAVA_HOME=/usr/java/jdk1.8.0_25/" --conf "spark.executorEnv.JAVA_HOME=/usr/java/jdk1.8.0_25/"来指定当前spark任务特定在yarn中执行时的jdk目录 请问下flink启动&提交任务时（bin/yarn-session.sh run）有没有参数可以指定flink特定使用yarn时的jdk目录，flink中是否有这样的功能？（不影响其他yarn上应用正常运行） zjfpla...@hotmail.com
Re: Re: Using async io in cep
Hi， Our business is to dynamically query (such as left join/right join) the mysql / oracle database during the cep process, or is there any other way to achieve this function? zjfpla...@hotmail.com From: Dawid Wysakowicz<mailto:dwysakow...@apache.org> Date: 2020-01-06 18:09 To: 郑 洁锋<mailto:zjfpla...@hotmail.com>; user<mailto:user@flink.apache.org> Subject: Re: Using async io in cep Hi, You cannot use the Async IO as described here[1] in the CEP library, if that's what you are asking for. It is also not that straightforward to say what would an async processing in that case mean. Primary use case for Async IO is to execute parallel computations of independent data. In case of CEP it does not stand for processing of records within a single key, as those assume strict ordering and in general case depend on results of processing previous records. One could think of asynchronous processing of records from different keys in a single parallel instance, but that would require a careful key processing. If I am not mistaken Async IO also does not support a stateful processing on a keyed stream. Best, Dawid [1] https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/stream/operators/asyncio.html On 06/01/2020 09:59, 郑 洁锋 wrote: Hi， Is there a way to use asynchronous io to query the database in the process of cep? zjfpla...@hotmail.com<mailto:zjfpla...@hotmail.com>
Using async io in cep
Hi， Is there a way to use asynchronous io to query the database in the process of cep? zjfpla...@hotmail.com
Using async io in cep
Hi， Is there a way to use asynchronous io to query the database in the process of cep? zjfpla...@hotmail.com
