[ https://issues.apache.org/jira/browse/DRILL-6268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17612020#comment-17612020 ]
ASF GitHub Bot commented on DRILL-6268: --------------------------------------- cgivre closed pull request #2139: DRILL-6268: Drill-on-YARN client obtains HDFS URL incorrectly URL: https://github.com/apache/drill/pull/2139 > Drill-on-YARN client obtains HDFS URL incorrectly > ------------------------------------------------- > > Key: DRILL-6268 > URL: https://issues.apache.org/jira/browse/DRILL-6268 > Project: Apache Drill > Issue Type: Bug > Affects Versions: 1.13.0 > Reporter: Paul Rogers > Assignee: Charles Givre > Priority: Major > > The Drill-on-YARN client must upload files to HDFS so that YARN can localize > them. The code that does so is in {{DfsFacade}}. This code obtains the URL > twice. The first time is correct: > {code} > private void loadYarnConfig() { > ... > URI fsUri = FileSystem.getDefaultUri( yarnConf ); > if(fsUri.toString().startsWith("file:/")) { > System.err.println("Warning: Default DFS URI is for a local file > system: " + fsUri); > } > } > } > {code} > The {{fsUri}} returned is {{hdfs://localhost:9000}}, which is the correct > value for an out-of-the-box Hadoop 2.9.0 install after following [these > instructions|https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html]. > The instructions have the reader explicitly set the port number to 9000: > {code} > <configuration> > <property> > <name>fs.defaultFS</name> > <value>hdfs://localhost:9000</value> > </property> > </configuration> > {code} > The other place that gets the URL, this time or real, is > {{DfsFacade.connect()}}: > {code} > String dfsConnection = config.getString(DrillOnYarnConfig.DFS_CONNECTION); > {code} > This value comes back as {{hdfs://localhost/}}, which causes HDFS to try to > connect on port 8020 (the Hadoop default), resulting in the following error: > {noformat} > Connecting to DFS... Connected. > Uploading /Users/paulrogers/bin/apache-drill-1.13.0.tar.gz to > /users/drill/apache-drill-1.13.0.tar.gz ... Failed. > Failed to upload Drill archive > Caused by: Failed to create DFS directory: /users/drill > Caused by: Call From Pauls-MBP/192.168.1.243 to localhost:8020 failed on > connection exception: java.net.ConnectException: Connection refused; > For more details see: http://wiki.apache.org/hadoop/ConnectionRefused > {noformat} > (Shout out here to [~arjun-kr] for suggesting we include the extra exception > details; very helpful here.) > The workaround is to manually change the port to 8020 in the config setting > shown above. > The full fix is to change the code to use the following line in {{connect()}}: > {code} > String dfsConnection = FileSystem.getDefaultUri(yarnConf); > {code} > This bug is serious because it constrains the ability of users to select > non-default HDFS ports. -- This message was sent by Atlassian Jira (v8.20.10#820010)