[
https://issues.apache.org/jira/browse/ARROW-4820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16789304#comment-16789304
]
Tiger068 edited comment on ARROW-4820 at 3/11/19 9:01 AM:
----------------------------------------------------------
use hadoop commd to get classpath like below, the $HADOOP_CONF_DIR is in the
classpath
{code:java}
(base) tiger:~ tiger$ $HADOOP_HOME/bin/hadoop classpath --glob
/Users/tiger/devtools/hadoop/hadoop-2.8.2/etc/hadoop:/Volumes/LaCie/devtools/hadoop/hadoop-2.8.2/share/hadoop/common/lib/activation-1.1.jar:/Volumes/LaCie/devtools/hadoop/hadoop-2.8.2/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/Volumes/LaCie/devtools/hadoop/hadoop-2.8.2/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/Volumes/LaCie/devtools/hadoop/hadoop-2.8.2/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/Volumes/LaCie/devtools/hadoop/hadoop-2.8.2/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/Volumes/LaCie/devtools/hadoop/hadoop-2.8.2/share/hadoop/common/lib/asm-3.2.jar:/Volumes/LaCie/devtools/hadoop/hadoop-2.8.2/share/hadoop/common/lib/avro-1.7.4.jar:/Volumes/LaCie/devtools/hadoop/hadoop-2.8.2/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/Volumes/LaCie/devtools/hadoop/hadoop-2.8.2/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/Volumes/LaCie/devtools/hadoop/hadoop-2.8.2/share/hadoop/common/lib/commons
{code}
the method derive_hadoop_classpath only add .jar files into hadoop classpath
{code:java}
def _derive_hadoop_classpath():
import subprocess
find_args = ('find', os.environ['HADOOP_HOME'], '-name', '*.jar')
find = subprocess.Popen(find_args, stdout=subprocess.PIPE)
xargs_echo = subprocess.Popen(('xargs', 'echo'),
stdin=find.stdout,
stdout=subprocess.PIPE)
return subprocess.check_output(('tr', "' '", "':'"),
stdin=xargs_echo.stdout){code}
was (Author: tiger068):
use hadoop commd to get classpath like below, the $HADOOP_CONF_DIR is in the
classpath
{code:java}
(base) tiger:~ tiger$ $HADOOP_HOME/bin/hadoop classpath --glob
/Users/tiger/devtools/hadoop/hadoop-2.8.2/etc/hadoop:/Volumes/LaCie/devtools/hadoop/hadoop-2.8.2/share/hadoop/common/lib/activation-1.1.jar:/Volumes/LaCie/devtools/hadoop/hadoop-2.8.2/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/Volumes/LaCie/devtools/hadoop/hadoop-2.8.2/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/Volumes/LaCie/devtools/hadoop/hadoop-2.8.2/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/Volumes/LaCie/devtools/hadoop/hadoop-2.8.2/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/Volumes/LaCie/devtools/hadoop/hadoop-2.8.2/share/hadoop/common/lib/asm-3.2.jar:/Volumes/LaCie/devtools/hadoop/hadoop-2.8.2/share/hadoop/common/lib/avro-1.7.4.jar:/Volumes/LaCie/devtools/hadoop/hadoop-2.8.2/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/Volumes/LaCie/devtools/hadoop/hadoop-2.8.2/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/Volumes/LaCie/devtools/hadoop/hadoop-2.8.2/share/hadoop/common/lib/commons
{code}
the metho derive_hadoop_classpath only add .jar files into hadoop classpath
{code:java}
def _derive_hadoop_classpath():
import subprocess
find_args = ('find', os.environ['HADOOP_HOME'], '-name', '*.jar')
find = subprocess.Popen(find_args, stdout=subprocess.PIPE)
xargs_echo = subprocess.Popen(('xargs', 'echo'),
stdin=find.stdout,
stdout=subprocess.PIPE)
return subprocess.check_output(('tr', "' '", "':'"),
stdin=xargs_echo.stdout){code}
> [Python] hadoop class path derived not correct
> ----------------------------------------------
>
> Key: ARROW-4820
> URL: https://issues.apache.org/jira/browse/ARROW-4820
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Affects Versions: 0.12.0, 0.12.1
> Reporter: Tiger068
> Priority: Major
> Fix For: 0.13.0
>
> Attachments: image-2019-03-11-16-39-25-195.png
>
>
> in hdfs.py, the method _derive_hadoop_classpath add jar files under
> $HADOOP_HOME into hadoop classpath,but the hadoop config directory is not
> contained in classpath.
>
> when hadoop HA mode enabled,the hdfs uri like this: hdfs://ns
> when the HADOOP_CONF_DIR directory is not in the hadoop classpath,the libhdfs
> can not locate the right hdfs-site.xml, in the HA mode, hdfs service name
> was passed as host name ,it is not correct
> !image-2019-03-11-16-39-25-195.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
