[ https://issues.apache.org/jira/browse/ARROW-4820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16789304#comment-16789304 ]
Tiger068 edited comment on ARROW-4820 at 3/11/19 8:54 AM: ---------------------------------------------------------- use hadoop commd to get classpath like below, the $HADOOP_CONF_DIR is in the classpath {code:java} (base) tiger:~ tiger$ $HADOOP_HOME/bin/hadoop classpath --glob /Users/tiger/devtools/hadoop/hadoop-2.8.2/etc/hadoop:/Volumes/LaCie/devtools/hadoop/hadoop-2.8.2/share/hadoop/common/lib/activation-1.1.jar:/Volumes/LaCie/devtools/hadoop/hadoop-2.8.2/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/Volumes/LaCie/devtools/hadoop/hadoop-2.8.2/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/Volumes/LaCie/devtools/hadoop/hadoop-2.8.2/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/Volumes/LaCie/devtools/hadoop/hadoop-2.8.2/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/Volumes/LaCie/devtools/hadoop/hadoop-2.8.2/share/hadoop/common/lib/asm-3.2.jar:/Volumes/LaCie/devtools/hadoop/hadoop-2.8.2/share/hadoop/common/lib/avro-1.7.4.jar:/Volumes/LaCie/devtools/hadoop/hadoop-2.8.2/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/Volumes/LaCie/devtools/hadoop/hadoop-2.8.2/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/Volumes/LaCie/devtools/hadoop/hadoop-2.8.2/share/hadoop/common/lib/commons {code} the metho derive_hadoop_classpath only add .jar files into hadoop classpath {code:java} def _derive_hadoop_classpath(): import subprocess find_args = ('find', os.environ['HADOOP_HOME'], '-name', '*.jar') find = subprocess.Popen(find_args, stdout=subprocess.PIPE) xargs_echo = subprocess.Popen(('xargs', 'echo'), stdin=find.stdout, stdout=subprocess.PIPE) return subprocess.check_output(('tr', "' '", "':'"), stdin=xargs_echo.stdout){code} was (Author: tiger068): use hadoop commd to get classpath like below, the $HADOOP_CONF_DIR is in the classpath {code:java} (base) tiger:~ tiger$ $HADOOP_HOME/bin/hadoop classpath --glob /Users/tiger/devtools/hadoop/hadoop-2.8.2/etc/hadoop:/Volumes/LaCie/devtools/hadoop/hadoop-2.8.2/share/hadoop/common/lib/activation-1.1.jar:/Volumes/LaCie/devtools/hadoop/hadoop-2.8.2/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/Volumes/LaCie/devtools/hadoop/hadoop-2.8.2/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/Volumes/LaCie/devtools/hadoop/hadoop-2.8.2/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/Volumes/LaCie/devtools/hadoop/hadoop-2.8.2/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/Volumes/LaCie/devtools/hadoop/hadoop-2.8.2/share/hadoop/common/lib/asm-3.2.jar:/Volumes/LaCie/devtools/hadoop/hadoop-2.8.2/share/hadoop/common/lib/avro-1.7.4.jar:/Volumes/LaCie/devtools/hadoop/hadoop-2.8.2/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/Volumes/LaCie/devtools/hadoop/hadoop-2.8.2/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/Volumes/LaCie/devtools/hadoop/hadoop-2.8.2/share/hadoop/common/lib/commons {code} the metho derive_hadoop_classpath only add .jar files into hadoop classpath {code:java} def _derive_hadoop_classpath(): import subprocess find_args = ('find', os.environ['HADOOP_HOME'], '-name', '*.jar') find = subprocess.Popen(find_args, stdout=subprocess.PIPE) xargs_echo = subprocess.Popen(('xargs', 'echo'), stdin=find.stdout, stdout=subprocess.PIPE) return subprocess.check_output(('tr', "' '", "':'"), stdin=xargs_echo.stdout){code} > [Python] hadoop class path derived not correct > ---------------------------------------------- > > Key: ARROW-4820 > URL: https://issues.apache.org/jira/browse/ARROW-4820 > Project: Apache Arrow > Issue Type: Bug > Components: Python > Affects Versions: 0.12.0, 0.12.1 > Reporter: Tiger068 > Priority: Major > Fix For: 0.13.0 > > Attachments: image-2019-03-11-16-39-25-195.png > > > in hdfs.py, the method _derive_hadoop_classpath add jar files under > $HADOOP_HOME into hadoop classpath,but the hadoop config directory is not > contained in classpath. > > when hadoop HA mode enabled,the hdfs uri like this: hdfs://ns > when the HADOOP_CONF_DIR directory is not in the hadoop classpath,the libhdfs > can not locate the right hdfs-site.xml, in the HA mode, hdfs service name > was passed as host ,it is not correct > !image-2019-03-11-16-39-25-195.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)