[
https://issues.apache.org/jira/browse/HADOOP-14498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16039692#comment-16039692
]
Allen Wittenauer edited comment on HADOOP-14498 at 6/6/17 9:36 PM:
-------------------------------------------------------------------
OK, using 'hadoop --debug classpath' we can see exactly what is happening with
the specific construction of "hadoop-azure,hadoop-aws,hadoop-azure-datalake"
{code}
$ bin/hadoop --debug classpath 2>&1 | grep PROFILES
DEBUG: HADOOP_SHELL_PROFILES accepted hadoop-aws
DEBUG: HADOOP_SHELL_PROFILES accepted hadoop-azure-datalake
DEBUG: HADOOP_SHELL_PROFILES declined hadoop-azure
DEBUG: HADOOP_SHELL_PROFILES accepted hdfs
DEBUG: HADOOP_SHELL_PROFILES accepted mapred
DEBUG: HADOOP_SHELL_PROFILES accepted yarn
{code}
hadoop-azure is getting rejected by the shell profiles code because it is
getting up in the dedupe pattern match code. Converting this to use the new
array code added will probably fix this. Flipping the order will also make it
pass.
was (Author: aw):
OK, using 'hadoop --debug classpath' we can see exactly what is happening with
the specific construction of "hadoop-azure,hadoop-aws,hadoop-azure-datalake"
{code}
$ bin/hadoop --debug classpath 2>&1 | grep PROFILES
DEBUG: HADOOP_SHELL_PROFILES accepted hadoop-aws
DEBUG: HADOOP_SHELL_PROFILES accepted hadoop-azure-datalake
DEBUG: HADOOP_SHELL_PROFILES declined hadoop-azure
DEBUG: HADOOP_SHELL_PROFILES accepted hdfs
DEBUG: HADOOP_SHELL_PROFILES accepted mapred
DEBUG: HADOOP_SHELL_PROFILES accepted yarn
{code}
hadoop-azure is getting rejected by the shell profiles code because it is
getting up in the dedupe pattern match code. Converting this to use the new
array code added will probably fix this.
> HADOOP_OPTIONAL_TOOLS not parsed correctly
> ------------------------------------------
>
> Key: HADOOP-14498
> URL: https://issues.apache.org/jira/browse/HADOOP-14498
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
> Affects Versions: 3.0.0-alpha1
> Reporter: Mingliang Liu
> Priority: Critical
>
> # This will make hadoop-azure not show up in the hadoop classpath, though
> both hadoop-aws and hadoop-azure-datalake are in the
> classpath.{code:title=hadoop-env.sh}
> export HADOOP_OPTIONAL_TOOLS="hadoop-azure,hadoop-aws,hadoop-azure-datalake"
> {code}
> # And if we put only hadoop-azure and hadoop-aws, both of them are shown in
> the classpath.
> {code:title=hadoop-env.sh}
> export HADOOP_OPTIONAL_TOOLS="hadoop-azure,hadoop-aws"
> {code}
> This makes me guess that, while parsing the {{HADOOP_OPTIONAL_TOOLS}}, we
> make some assumptions that hadoop tool modules have a single "-" in names,
> and the _hadoop-azure-datalake_ overrides the _hadoop-azure_. Or any other
> assumptions about the {{${project.artifactId\}}}?
> Ping [~aw].
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
