flink1.13.1在sql文件和flink-conf.yaml文件中不生效

2021-06-10 文章 Fei Han
大家好:
flink1.13.1在sql文件和flink-conf.yaml文件中不生效。我的sql文件内容如下:

尝试多次,数据无法写入iceberg。hive版本2.3.6


flink sql cli 模式下,flink-conf.yaml 配置checkpoint无法生效

2021-06-02 文章 guozhi mang
各位好,我在flink1.13版本
的flink配置文件里配置checkpoint和savepoint参数时,相关配置项并没有生效,现将我的配置文件信息和日志放在下文。

*配置文件*
#==
# Fault tolerance and checkpointing
#==

# The backend that will be used to store operator state checkpoints if
# checkpointing is enabled.
#
# Supported backends are 'jobmanager', 'filesystem', 'rocksdb', or the
# .
#

# Directory for checkpoints filesystem, when using any of the default
bundled
# state backends.
#
# state.checkpoints.dir: hdfs://namenode-host:port/flink-checkpoints
execution.checkpointing.interval: 1
state.backend: filesystem
state.checkpoints.dir: file:///opt/xxx/flink-1.13.0/savepoint/checkpoints
state.savepoints.dir: file:///opt/xxx/flink-1.13.0/savepoint/savepoints
# Default target directory for savepoints, optional.
#
# state.savepoints.dir: hdfs://namenode-host:port/flink-savepoints

# Flag to enable/disable incremental checkpoints for backends that
# support incremental checkpoints (like the RocksDB state backend).
#
state.backend.incremental: false

# The failover strategy, i.e., how the job computation recovers from task
failures.
# Only restart tasks that may have been affected by the task failure, which
typically includes
# downstream tasks and potentially upstream tasks if their produced data is
no longer available for consumption.

jobmanager.execution.failover-strategy: region

#==

*服务器log日志:*

flink-root-sql-client-xxx03.log


2021-06-02 11:29:57,566 INFO
 org.apache.flink.configuration.GlobalConfiguration   [] - Loading
configuration property: execution.checkpointing.interval, 1
2021-06-02 11:29:57,566 INFO
 org.apache.flink.configuration.GlobalConfiguration   [] - Loading
configuration property: state.backend, filesystem
2021-06-02 11:29:57,566 INFO
 org.apache.flink.configuration.GlobalConfiguration   [] - Loading
configuration property: state.checkpoints.dir,
file:///opt/xxx/flink-1.13.0/savepoint/checkpoints
2021-06-02 11:29:57,566 INFO
 org.apache.flink.configuration.GlobalConfiguration   [] - Loading
configuration property: state.savepoints.dir,
file:///opt/xxx/flink-1.13.0/savepoint/savepoints
2021-06-02 11:29:57,566 INFO
 org.apache.flink.configuration.GlobalConfiguration   [] - Loading
configuration property: state.backend.incremental, false
2021-06-02 11:29:57,566 INFO
 org.apache.flink.configuration.GlobalConfiguration   [] - Loading
configuration property: jobmanager.execution.failover-strategy, region
2021-06-02 11:29:57,567 INFO
 org.apache.flink.configuration.GlobalConfiguration   [] - Loading
configuration property: rest.bind-port, 8086
2021-06-02 11:29:57,596 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli
   [] - Found Yarn properties file under
/tmp/.yarn-properties-root.
2021-06-02 11:29:57,998 INFO
 org.apache.flink.table.client.config.entries.ExecutionEntry  [] - Property
'execution.restart-strategy.type' not specified. Using default value:
fallback
2021-06-02 11:29:58,028 INFO
 org.apache.flink.table.client.gateway.context.DefaultContext [] - Executor
config: {execution.savepoint.ignore-unclaimed-state=false,
execution.attached=true, yarn.application.id=xxx,
execution.shutdown-on-attached-exit=false,
pipeline.jars=[file:/opt/xxx/flink-1.13.0/opt/flink-sql-client_2.11-1.13.0.jar],
high-availability.cluster-id=application_1620482572059_3697,
pipeline.classpaths=[], execution.target=yarn-session,
$internal.deployment.config-dir=/opt/xxx/flink-1.13.0/conf}
2021-06-02 11:30:02,947 WARN
 org.apache.flink.yarn.configuration.YarnLogConfigUtil[] - The
configuration directory ('/opt/xxx/flink-1.13.0/conf') already contains a
LOG4J config file.If you want to use logback, then please delete or rename
the log configuration file.
2021-06-02 11:30:02,986 WARN  org.apache.flink.runtime.util.HadoopUtils
   [] - Could not find Hadoop configuration via any of the
supported methods (Flink configuration, environment variables).
2021-06-02 11:30:03,391 INFO  org.apache.flink.yarn.YarnClusterDescriptor
   [] - No path for the flink jar passed. Using the location of
class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
2021-06-02 11:30:03,393 WARN  org.apache.flink.yarn.YarnClusterDescriptor
   [] - Neither the HADOOP_CONF_DIR nor the YARN_CONF_DIR
environment variable is set.The Flink YARN Client needs one of these to be
set to properly load the Hadoop configuration for accessing YARN.
2021-06-02 11:30:03,445 INFO
 org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider [] -
Failing over to rm236
2021-06-02 11:30:03,502 INFO  org.apache.flink.yarn.YarnClusterDescriptor
   [] - Found Web Interface xxx:54194 of application 'xxx


*通过web ui观察到的日志:*

2021-05-28 15:32:30,673 INFO 

Re: flink -conf.yaml修改

2021-04-12 文章 JasonLee
hi

如果是 session 模式需要重启集群,如果是 per-job 模式直接提交任务即可.



-
Best Wishes
JasonLee
--
Sent from: http://apache-flink.147419.n8.nabble.com/


??????flink -conf.yaml????

2021-04-12 文章 Long


回复: 回复:flink -conf.yaml修改

2021-04-12 文章 明启 孙
多谢

smq

发件人: 熊云昆
发送时间: 2021年4月12日 18:48
收件人: smq
抄送: user-zh
主题: 回复:flink -conf.yaml修改

重启job就可以了,不需要重启集群


| |
熊云昆
|
|
邮箱:xiongyun...@163.com
|

签名由 网易邮箱大师 定制

在2021年04月12日 14:34,smq 写道:
大家好,这个配置文件修改之后需要什么操作才生效吗,比如重启集群,还是说直接启动作业自动就应用了呢



回复:flink -conf.yaml修改

2021-04-12 文章 熊云昆
重启job就可以了,不需要重启集群


| |
熊云昆
|
|
邮箱:xiongyun...@163.com
|

签名由 网易邮箱大师 定制

在2021年04月12日 14:34,smq 写道:
大家好,这个配置文件修改之后需要什么操作才生效吗,比如重启集群,还是说直接启动作业自动就应用了呢

flink -conf.yaml修改

2021-04-12 文章 smq
大家好,这个配置文件修改之后需要什么操作才生效吗,比如重启集群,还是说直接启动作业自动就应用了呢