qian wang created FLINK-15097:
---------------------------------
Summary: flink can not use user specified hdfs conf
Key: FLINK-15097
URL: https://issues.apache.org/jira/browse/FLINK-15097
Project: Flink
Issue Type: Bug
Components: Client / Job Submission
Affects Versions: 1.9.1
Reporter: qian wang
Attachments: 0001-adjust-read-hdfs-conf-order.patch
now if cluster node had set env HADOOP_CONF_DIR，flink would force use the
hdfs-site.xml in the corresponding dir, then user who submitted app in the
client node couldn't use custom specified hdfs-site.xml/hdfs-default through
setting fs.hdfs.hdfssite or fs.hdfs.hdfsdefault so as to set custom blocksize
or replication num. For example Using yarnship to upload my hdfs conf dir and
set fs.hdfs.hdfssite direct to \{conf dir}/hdfs-site.xml is useless
Deep in code it is due to the order of choosing conf in HadoopUtils.java，the
conf in HADOOP_CONF_DIR will override user's uploaded conf, i think the way is
not sensible, so i reverse the order which flink read hdfs conf in order to let
user custom conf uploaded override HADOOP_CONF_DIR
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
