Github user djh4230 commented on the issue:
https://github.com/apache/flink/pull/4926
The hadoop config files are in the flink lauched classpath
---
Github user tillrohrmann commented on the issue:
https://github.com/apache/flink/pull/4926
@djh4230 could you check whether the Hadoop configuration directory is in
the classpath of the launched Flink components?
---
Github user djh4230 commented on the issue:
https://github.com/apache/flink/pull/4926
Not work! still can't read hdfs-site.xml and can't read hdfs namespace
Github user tillrohrmann commented on the issue:
https://github.com/apache/flink/pull/4926
Hi @djh4230, try to make sure that your Hadoop configuration file is on the
classpath or that the `HADOOP_CONF_DIR` is properly set.
---
Github user djh4230 commented on the issue:
https://github.com/apache/flink/pull/4926
Does that mean it's not a issue? But the issue still happened in
flink-1.4.How can i fix the issue?
---
Github user tillrohrmann commented on the issue:
https://github.com/apache/flink/pull/4926
Alright, thanks for the infos @steveloughran.
After an offline discussion with @StephanEwen, we decided that we should
probably not try to explicitly load a Hadoop configuration file
Github user steveloughran commented on the issue:
https://github.com/apache/flink/pull/4926
creating YarnConfiguration & HdfsConfiguration through some dynamic
classloading is enough to force in these files & configs underneath your own
Configurations. You shouldn't be reading in all
Github user tillrohrmann commented on the issue:
https://github.com/apache/flink/pull/4926
Thanks for the review @steveloughran, @StephanEwen and @aljoscha.
If I understood it correctly, then we shouldn't upload the configuration
files from the client and instead try to find
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/4926
+1 to merge this
---
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/4926
Thanks @steveloughran!
I think this approach here should work for then, for now.
---
Github user steveloughran commented on the issue:
https://github.com/apache/flink/pull/4926
ah. everything lives in Configuration, and is has some historical structure
based on the history of how things all got split up from one big hadoop-default;
1. You can register a new
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/4926
@steveloughran if it is okay, maybe we could pick could brain here quickly?
What is the Yarn-ideomatic way to handle configurations and what are the
assumptions?
Specifically:
- Do
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/4926
The changes look good but let's see what you find regarding
`yarn-site.xml`. ð
---
Github user tillrohrmann commented on the issue:
https://github.com/apache/flink/pull/4926
A user reported that we maybe have to explicitly load the `yarn-site.xml`
in the `HadoopUtils.getHadoopConfiguration` method as we do it for the
`core-site.xml` and `hdfs-site.xml`. Will have
14 matches
Mail list logo
