[jira] [Commented] (HADOOP-11485) Pluggable shell integration
[ https://issues.apache.org/jira/browse/HADOOP-11485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14311393#comment-14311393 ] Hudson commented on HADOOP-11485: - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #98 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/98/]) Pluggable shell integration --- Key: HADOOP-11485 URL: https://issues.apache.org/jira/browse/HADOOP-11485 Project: Hadoop Common Issue Type: New Feature Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: scripts, shell Fix For: 3.0.0 Attachments: HADOOP-11485-00.patch, HADOOP-11485-01.patch, HADOOP-11485-02.patch, HADOOP-11485-03.patch, HADOOP-11485-04.patch It would be useful to provide a way for core and non-core Hadoop components to plug into the shell infrastructure. This would allow us to pull the HDFS, MapReduce, and YARN shell functions out of hadoop-functions.sh. Additionally, it should let 3rd parties such as HBase influence things like classpaths at runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11485) Pluggable shell integration
[ https://issues.apache.org/jira/browse/HADOOP-11485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14311363#comment-14311363 ] Hudson commented on HADOOP-11485: - FAILURE: Integrated in Hadoop-Yarn-trunk #832 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/832/]) Pluggable shell integration --- Key: HADOOP-11485 URL: https://issues.apache.org/jira/browse/HADOOP-11485 Project: Hadoop Common Issue Type: New Feature Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: scripts, shell Fix For: 3.0.0 Attachments: HADOOP-11485-00.patch, HADOOP-11485-01.patch, HADOOP-11485-02.patch, HADOOP-11485-03.patch, HADOOP-11485-04.patch It would be useful to provide a way for core and non-core Hadoop components to plug into the shell infrastructure. This would allow us to pull the HDFS, MapReduce, and YARN shell functions out of hadoop-functions.sh. Additionally, it should let 3rd parties such as HBase influence things like classpaths at runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11485) Pluggable shell integration
[ https://issues.apache.org/jira/browse/HADOOP-11485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14311568#comment-14311568 ] Hudson commented on HADOOP-11485: - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #95 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/95/]) Pluggable shell integration --- Key: HADOOP-11485 URL: https://issues.apache.org/jira/browse/HADOOP-11485 Project: Hadoop Common Issue Type: New Feature Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: scripts, shell Fix For: 3.0.0 Attachments: HADOOP-11485-00.patch, HADOOP-11485-01.patch, HADOOP-11485-02.patch, HADOOP-11485-03.patch, HADOOP-11485-04.patch It would be useful to provide a way for core and non-core Hadoop components to plug into the shell infrastructure. This would allow us to pull the HDFS, MapReduce, and YARN shell functions out of hadoop-functions.sh. Additionally, it should let 3rd parties such as HBase influence things like classpaths at runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11485) Pluggable shell integration
[ https://issues.apache.org/jira/browse/HADOOP-11485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14311536#comment-14311536 ] Hudson commented on HADOOP-11485: - SUCCESS: Integrated in Hadoop-Hdfs-trunk #2030 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2030/]) Pluggable shell integration --- Key: HADOOP-11485 URL: https://issues.apache.org/jira/browse/HADOOP-11485 Project: Hadoop Common Issue Type: New Feature Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: scripts, shell Fix For: 3.0.0 Attachments: HADOOP-11485-00.patch, HADOOP-11485-01.patch, HADOOP-11485-02.patch, HADOOP-11485-03.patch, HADOOP-11485-04.patch It would be useful to provide a way for core and non-core Hadoop components to plug into the shell infrastructure. This would allow us to pull the HDFS, MapReduce, and YARN shell functions out of hadoop-functions.sh. Additionally, it should let 3rd parties such as HBase influence things like classpaths at runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11485) Pluggable shell integration
[ https://issues.apache.org/jira/browse/HADOOP-11485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14311649#comment-14311649 ] Hudson commented on HADOOP-11485: - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #99 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/99/]) Pluggable shell integration --- Key: HADOOP-11485 URL: https://issues.apache.org/jira/browse/HADOOP-11485 Project: Hadoop Common Issue Type: New Feature Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: scripts, shell Fix For: 3.0.0 Attachments: HADOOP-11485-00.patch, HADOOP-11485-01.patch, HADOOP-11485-02.patch, HADOOP-11485-03.patch, HADOOP-11485-04.patch It would be useful to provide a way for core and non-core Hadoop components to plug into the shell infrastructure. This would allow us to pull the HDFS, MapReduce, and YARN shell functions out of hadoop-functions.sh. Additionally, it should let 3rd parties such as HBase influence things like classpaths at runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11485) Pluggable shell integration
[ https://issues.apache.org/jira/browse/HADOOP-11485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14311635#comment-14311635 ] Hudson commented on HADOOP-11485: - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2049 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2049/]) Pluggable shell integration --- Key: HADOOP-11485 URL: https://issues.apache.org/jira/browse/HADOOP-11485 Project: Hadoop Common Issue Type: New Feature Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: scripts, shell Fix For: 3.0.0 Attachments: HADOOP-11485-00.patch, HADOOP-11485-01.patch, HADOOP-11485-02.patch, HADOOP-11485-03.patch, HADOOP-11485-04.patch It would be useful to provide a way for core and non-core Hadoop components to plug into the shell infrastructure. This would allow us to pull the HDFS, MapReduce, and YARN shell functions out of hadoop-functions.sh. Additionally, it should let 3rd parties such as HBase influence things like classpaths at runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11485) Pluggable shell integration
[ https://issues.apache.org/jira/browse/HADOOP-11485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310668#comment-14310668 ] Hudson commented on HADOOP-11485: - FAILURE: Integrated in Hadoop-Yarn-trunk #831 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/831/]) HADOOP-11485. Pluggable shell integration (aw) (aw: rev 5c79439568ff0c73062cf09d87f1e739703c7dc0) * hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh * hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml * hadoop-common-project/hadoop-common/src/main/conf/shellprofile.d/example * hadoop-yarn-project/hadoop-yarn/shellprofile.d/yarn * hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/shellprofile.d/hdfs * hadoop-yarn-project/hadoop-yarn/bin/yarn * hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml * hadoop-mapreduce-project/shellprofile.d/mapreduce * hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml Pluggable shell integration --- Key: HADOOP-11485 URL: https://issues.apache.org/jira/browse/HADOOP-11485 Project: Hadoop Common Issue Type: New Feature Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: scripts, shell Fix For: 3.0.0 Attachments: HADOOP-11485-00.patch, HADOOP-11485-01.patch, HADOOP-11485-02.patch, HADOOP-11485-03.patch, HADOOP-11485-04.patch It would be useful to provide a way for core and non-core Hadoop components to plug into the shell infrastructure. This would allow us to pull the HDFS, MapReduce, and YARN shell functions out of hadoop-functions.sh. Additionally, it should let 3rd parties such as HBase influence things like classpaths at runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11485) Pluggable shell integration
[ https://issues.apache.org/jira/browse/HADOOP-11485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310661#comment-14310661 ] Hudson commented on HADOOP-11485: - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #97 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/97/]) HADOOP-11485. Pluggable shell integration (aw) (aw: rev 5c79439568ff0c73062cf09d87f1e739703c7dc0) * hadoop-common-project/hadoop-common/src/main/conf/shellprofile.d/example * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml * hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh * hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml * hadoop-yarn-project/hadoop-yarn/bin/yarn * hadoop-mapreduce-project/shellprofile.d/mapreduce * hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh * hadoop-yarn-project/hadoop-yarn/shellprofile.d/yarn * hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml * hadoop-hdfs-project/hadoop-hdfs/src/main/shellprofile.d/hdfs Pluggable shell integration --- Key: HADOOP-11485 URL: https://issues.apache.org/jira/browse/HADOOP-11485 Project: Hadoop Common Issue Type: New Feature Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: scripts, shell Fix For: 3.0.0 Attachments: HADOOP-11485-00.patch, HADOOP-11485-01.patch, HADOOP-11485-02.patch, HADOOP-11485-03.patch, HADOOP-11485-04.patch It would be useful to provide a way for core and non-core Hadoop components to plug into the shell infrastructure. This would allow us to pull the HDFS, MapReduce, and YARN shell functions out of hadoop-functions.sh. Additionally, it should let 3rd parties such as HBase influence things like classpaths at runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11485) Pluggable shell integration
[ https://issues.apache.org/jira/browse/HADOOP-11485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310732#comment-14310732 ] Hudson commented on HADOOP-11485: - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #94 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/94/]) HADOOP-11485. Pluggable shell integration (aw) (aw: rev 5c79439568ff0c73062cf09d87f1e739703c7dc0) * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml * hadoop-mapreduce-project/shellprofile.d/mapreduce * hadoop-yarn-project/hadoop-yarn/bin/yarn * hadoop-common-project/hadoop-common/src/main/conf/shellprofile.d/example * hadoop-yarn-project/hadoop-yarn/shellprofile.d/yarn * hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh * hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml * hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml * hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh * hadoop-hdfs-project/hadoop-hdfs/src/main/shellprofile.d/hdfs Pluggable shell integration --- Key: HADOOP-11485 URL: https://issues.apache.org/jira/browse/HADOOP-11485 Project: Hadoop Common Issue Type: New Feature Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: scripts, shell Fix For: 3.0.0 Attachments: HADOOP-11485-00.patch, HADOOP-11485-01.patch, HADOOP-11485-02.patch, HADOOP-11485-03.patch, HADOOP-11485-04.patch It would be useful to provide a way for core and non-core Hadoop components to plug into the shell infrastructure. This would allow us to pull the HDFS, MapReduce, and YARN shell functions out of hadoop-functions.sh. Additionally, it should let 3rd parties such as HBase influence things like classpaths at runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11485) Pluggable shell integration
[ https://issues.apache.org/jira/browse/HADOOP-11485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310724#comment-14310724 ] Hudson commented on HADOOP-11485: - FAILURE: Integrated in Hadoop-Hdfs-trunk #2029 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2029/]) HADOOP-11485. Pluggable shell integration (aw) (aw: rev 5c79439568ff0c73062cf09d87f1e739703c7dc0) * hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh * hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml * hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml * hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml * hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh * hadoop-yarn-project/hadoop-yarn/shellprofile.d/yarn * hadoop-yarn-project/hadoop-yarn/bin/yarn * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-mapreduce-project/shellprofile.d/mapreduce * hadoop-hdfs-project/hadoop-hdfs/src/main/shellprofile.d/hdfs * hadoop-common-project/hadoop-common/src/main/conf/shellprofile.d/example Pluggable shell integration --- Key: HADOOP-11485 URL: https://issues.apache.org/jira/browse/HADOOP-11485 Project: Hadoop Common Issue Type: New Feature Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: scripts, shell Fix For: 3.0.0 Attachments: HADOOP-11485-00.patch, HADOOP-11485-01.patch, HADOOP-11485-02.patch, HADOOP-11485-03.patch, HADOOP-11485-04.patch It would be useful to provide a way for core and non-core Hadoop components to plug into the shell infrastructure. This would allow us to pull the HDFS, MapReduce, and YARN shell functions out of hadoop-functions.sh. Additionally, it should let 3rd parties such as HBase influence things like classpaths at runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11485) Pluggable shell integration
[ https://issues.apache.org/jira/browse/HADOOP-11485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310756#comment-14310756 ] Hudson commented on HADOOP-11485: - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #98 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/98/]) HADOOP-11485. Pluggable shell integration (aw) (aw: rev 5c79439568ff0c73062cf09d87f1e739703c7dc0) * hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml * hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml * hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh * hadoop-yarn-project/hadoop-yarn/shellprofile.d/yarn * hadoop-common-project/hadoop-common/src/main/conf/shellprofile.d/example * hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml * hadoop-yarn-project/hadoop-yarn/bin/yarn * hadoop-mapreduce-project/shellprofile.d/mapreduce * hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh * hadoop-hdfs-project/hadoop-hdfs/src/main/shellprofile.d/hdfs * hadoop-common-project/hadoop-common/CHANGES.txt Pluggable shell integration --- Key: HADOOP-11485 URL: https://issues.apache.org/jira/browse/HADOOP-11485 Project: Hadoop Common Issue Type: New Feature Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: scripts, shell Fix For: 3.0.0 Attachments: HADOOP-11485-00.patch, HADOOP-11485-01.patch, HADOOP-11485-02.patch, HADOOP-11485-03.patch, HADOOP-11485-04.patch It would be useful to provide a way for core and non-core Hadoop components to plug into the shell infrastructure. This would allow us to pull the HDFS, MapReduce, and YARN shell functions out of hadoop-functions.sh. Additionally, it should let 3rd parties such as HBase influence things like classpaths at runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11485) Pluggable shell integration
[ https://issues.apache.org/jira/browse/HADOOP-11485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310770#comment-14310770 ] Hudson commented on HADOOP-11485: - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2048 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2048/]) HADOOP-11485. Pluggable shell integration (aw) (aw: rev 5c79439568ff0c73062cf09d87f1e739703c7dc0) * hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh * hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml * hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh * hadoop-yarn-project/hadoop-yarn/shellprofile.d/yarn * hadoop-common-project/hadoop-common/src/main/conf/shellprofile.d/example * hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml * hadoop-yarn-project/hadoop-yarn/bin/yarn * hadoop-mapreduce-project/shellprofile.d/mapreduce * hadoop-hdfs-project/hadoop-hdfs/src/main/shellprofile.d/hdfs Pluggable shell integration --- Key: HADOOP-11485 URL: https://issues.apache.org/jira/browse/HADOOP-11485 Project: Hadoop Common Issue Type: New Feature Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: scripts, shell Fix For: 3.0.0 Attachments: HADOOP-11485-00.patch, HADOOP-11485-01.patch, HADOOP-11485-02.patch, HADOOP-11485-03.patch, HADOOP-11485-04.patch It would be useful to provide a way for core and non-core Hadoop components to plug into the shell infrastructure. This would allow us to pull the HDFS, MapReduce, and YARN shell functions out of hadoop-functions.sh. Additionally, it should let 3rd parties such as HBase influence things like classpaths at runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11485) Pluggable shell integration
[
https://issues.apache.org/jira/browse/HADOOP-11485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309785#comment-14309785
]
Hadoop QA commented on HADOOP-11485:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12697103/HADOOP-11485-04.patch
against trunk revision c1957fe.
{color:red}-1 @author{color}. The patch appears to contain @author tags
which the Hadoop community has agreed to not allow in code contributions.
{color:green}+1 tests included{color}. The patch appears to include new
or modified test files.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5616//console
This message is automatically generated.
Pluggable shell integration
---
Key: HADOOP-11485
URL: https://issues.apache.org/jira/browse/HADOOP-11485
Project: Hadoop Common
Issue Type: New Feature
Components: scripts
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer
Labels: scripts, shell
Attachments: HADOOP-11485-00.patch, HADOOP-11485-01.patch,
HADOOP-11485-02.patch, HADOOP-11485-03.patch, HADOOP-11485-04.patch
It would be useful to provide a way for core and non-core Hadoop components
to plug into the shell infrastructure. This would allow us to pull the HDFS,
MapReduce, and YARN shell functions out of hadoop-functions.sh.
Additionally, it should let 3rd parties such as HBase influence things like
classpaths at runtime.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11485) Pluggable shell integration
[ https://issues.apache.org/jira/browse/HADOOP-11485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309930#comment-14309930 ] Hudson commented on HADOOP-11485: - FAILURE: Integrated in Hadoop-trunk-Commit #7044 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7044/]) HADOOP-11485. Pluggable shell integration (aw) (aw: rev 5c79439568ff0c73062cf09d87f1e739703c7dc0) * hadoop-hdfs-project/hadoop-hdfs/src/main/shellprofile.d/hdfs * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-mapreduce-project/shellprofile.d/mapreduce * hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml * hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh * hadoop-yarn-project/hadoop-yarn/bin/yarn * hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml * hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml * hadoop-common-project/hadoop-common/src/main/conf/shellprofile.d/example * hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh * hadoop-yarn-project/hadoop-yarn/shellprofile.d/yarn Pluggable shell integration --- Key: HADOOP-11485 URL: https://issues.apache.org/jira/browse/HADOOP-11485 Project: Hadoop Common Issue Type: New Feature Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: scripts, shell Fix For: 3.0.0 Attachments: HADOOP-11485-00.patch, HADOOP-11485-01.patch, HADOOP-11485-02.patch, HADOOP-11485-03.patch, HADOOP-11485-04.patch It would be useful to provide a way for core and non-core Hadoop components to plug into the shell infrastructure. This would allow us to pull the HDFS, MapReduce, and YARN shell functions out of hadoop-functions.sh. Additionally, it should let 3rd parties such as HBase influence things like classpaths at runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11485) Pluggable shell integration
[
https://issues.apache.org/jira/browse/HADOOP-11485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308773#comment-14308773
]
Hadoop QA commented on HADOOP-11485:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12696956/HADOOP-11485-03.patch
against trunk revision af3aadf.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-assemblies hadoop-common-project/hadoop-common
hadoop-hdfs-project/hadoop-hdfs.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5612//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5612//console
This message is automatically generated.
Pluggable shell integration
---
Key: HADOOP-11485
URL: https://issues.apache.org/jira/browse/HADOOP-11485
Project: Hadoop Common
Issue Type: New Feature
Components: scripts
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer
Labels: scripts, shell
Attachments: HADOOP-11485-00.patch, HADOOP-11485-01.patch,
HADOOP-11485-02.patch, HADOOP-11485-03.patch
It would be useful to provide a way for core and non-core Hadoop components
to plug into the shell infrastructure. This would allow us to pull the HDFS,
MapReduce, and YARN shell functions out of hadoop-functions.sh.
Additionally, it should let 3rd parties such as HBase influence things like
classpaths at runtime.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11485) Pluggable shell integration
[
https://issues.apache.org/jira/browse/HADOOP-11485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308655#comment-14308655
]
Chris Nauroth commented on HADOOP-11485:
I think v03 has some issues with usage of {{hadoop_translate_cygwin_path}} in
shellprofile.d/yarn:
{code}
hadoop_translate_cygwin_path ${yld}
{code}
Inside {{hadoop_translate_cygwin_path}}, the argument is part of an {{eval}}
for setting (possibly changing) the variable value, so instead of expanding the
variable's value at the call site, we just need to pass through the variable
name:
{code}
hadoop_translate_cygwin_path yld
{code}
Also, I think I see a typo, defining variable {{yhd}}, but then trying to use
{{ygd}}:
{code}
local yhd=$HADOOP_YARN_HOME
hadoop_translate_cygwin_path ${ygd}
{code}
Pluggable shell integration
---
Key: HADOOP-11485
URL: https://issues.apache.org/jira/browse/HADOOP-11485
Project: Hadoop Common
Issue Type: New Feature
Components: scripts
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer
Labels: scripts, shell
Attachments: HADOOP-11485-00.patch, HADOOP-11485-01.patch,
HADOOP-11485-02.patch, HADOOP-11485-03.patch
It would be useful to provide a way for core and non-core Hadoop components
to plug into the shell infrastructure. This would allow us to pull the HDFS,
MapReduce, and YARN shell functions out of hadoop-functions.sh.
Additionally, it should let 3rd parties such as HBase influence things like
classpaths at runtime.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11485) Pluggable shell integration
[
https://issues.apache.org/jira/browse/HADOOP-11485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308135#comment-14308135
]
Chris Nauroth commented on HADOOP-11485:
Hi, [~aw]. Thank you for doing this. This looks great.
I took it for a test run with a concrete use case that I mentioned on another
jira: adding hadoop-azure to the classpath at distro time. Everything worked
as advertised in the example. I read through that and then dropped the
following file at shellprofile.d/hadoop_azure:
{code}
hadoop_add_profile hadoop_azure
function _hadoop_azure_hadoop_classpath
{
hadoop_add_classpath
${HADOOP_PREFIX}/share/hadoop/tools/lib/hadoop-azure-3.0.0-SNAPSHOT.jar after
hadoop_add_classpath
${HADOOP_PREFIX}/share/hadoop/tools/lib/microsoft-windowsazure-storage-sdk-0.6.0.jar
after
}
{code}
With that in place, I can run fs shell commands that access paths in Azure
Storage. This is easier than the existing options for trying to achieve this
kind of customization.
I have only 2 meta-comments, which don't necessarily need to be addressed in
the scope of this jira:
# With the introduction of shell profiles, we essentially have a new public API
contract. We'll need to maintain compatibility so that we don't break existing
shell profiles during routine code maintenance. (i.e. {{hadoop_add_classpath}}
can't be renamed within a major version). Maybe we need a commenting
convention, similar to the interface visibility and stability annotations that
we use in Java, to help guide developers.
# I think we'll also eventually need user-facing documentation on
hadoop.apache.org to describe the new functionality in the shell rewrite. For
shell profile implementors, something like API docs would be helpful to let
people know about useful things like {{hadoop_debug}}.
I think the current patch needs a small rebase. After that, +1.
Pluggable shell integration
---
Key: HADOOP-11485
URL: https://issues.apache.org/jira/browse/HADOOP-11485
Project: Hadoop Common
Issue Type: New Feature
Components: scripts
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer
Labels: scripts, shell
Attachments: HADOOP-11485-00.patch, HADOOP-11485-01.patch,
HADOOP-11485-02.patch
It would be useful to provide a way for core and non-core Hadoop components
to plug into the shell infrastructure. This would allow us to pull the HDFS,
MapReduce, and YARN shell functions out of hadoop-functions.sh.
Additionally, it should let 3rd parties such as HBase influence things like
classpaths at runtime.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11485) Pluggable shell integration
[ https://issues.apache.org/jira/browse/HADOOP-11485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308514#comment-14308514 ] Allen Wittenauer commented on HADOOP-11485: --- I did see a couple minor (non-interface) things in the example file, which I'll fix. Thanks for the review. I'll fix those minor issues and rebase the patch. On your two points, yeah, with this we're starting to approach a point where need to probably formalize the shell code now. I've opened HADOOP-11553 to start working on that. Pluggable shell integration --- Key: HADOOP-11485 URL: https://issues.apache.org/jira/browse/HADOOP-11485 Project: Hadoop Common Issue Type: New Feature Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: scripts, shell Attachments: HADOOP-11485-00.patch, HADOOP-11485-01.patch, HADOOP-11485-02.patch It would be useful to provide a way for core and non-core Hadoop components to plug into the shell infrastructure. This would allow us to pull the HDFS, MapReduce, and YARN shell functions out of hadoop-functions.sh. Additionally, it should let 3rd parties such as HBase influence things like classpaths at runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11485) Pluggable shell integration
[ https://issues.apache.org/jira/browse/HADOOP-11485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14306380#comment-14306380 ] Allen Wittenauer commented on HADOOP-11485: --- Given no further comments, I'll commit this next week. Pluggable shell integration --- Key: HADOOP-11485 URL: https://issues.apache.org/jira/browse/HADOOP-11485 Project: Hadoop Common Issue Type: New Feature Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: scripts, shell Attachments: HADOOP-11485-00.patch, HADOOP-11485-01.patch, HADOOP-11485-02.patch It would be useful to provide a way for core and non-core Hadoop components to plug into the shell infrastructure. This would allow us to pull the HDFS, MapReduce, and YARN shell functions out of hadoop-functions.sh. Additionally, it should let 3rd parties such as HBase influence things like classpaths at runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11485) Pluggable shell integration
[ https://issues.apache.org/jira/browse/HADOOP-11485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293884#comment-14293884 ] John Smith commented on HADOOP-11485: - +1 (non-binding) Pluggable shell integration --- Key: HADOOP-11485 URL: https://issues.apache.org/jira/browse/HADOOP-11485 Project: Hadoop Common Issue Type: New Feature Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: scripts, shell Attachments: HADOOP-11485-00.patch, HADOOP-11485-01.patch, HADOOP-11485-02.patch It would be useful to provide a way for core and non-core Hadoop components to plug into the shell infrastructure. This would allow us to pull the HDFS, MapReduce, and YARN shell functions out of hadoop-functions.sh. Additionally, it should let 3rd parties such as HBase influence things like classpaths at runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11485) Pluggable shell integration
[ https://issues.apache.org/jira/browse/HADOOP-11485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14284026#comment-14284026 ] Allen Wittenauer commented on HADOOP-11485: --- BTW, I'm debating merging this patch and HADOOP-11460 into one patch. Pluggable shell integration --- Key: HADOOP-11485 URL: https://issues.apache.org/jira/browse/HADOOP-11485 Project: Hadoop Common Issue Type: New Feature Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: scripts, shell Attachments: HADOOP-11485-00.patch, HADOOP-11485-01.patch, HADOOP-11485-02.patch It would be useful to provide a way for core and non-core Hadoop components to plug into the shell infrastructure. This would allow us to pull the HDFS, MapReduce, and YARN shell functions out of hadoop-functions.sh. Additionally, it should let 3rd parties such as HBase influence things like classpaths at runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11485) Pluggable shell integration
[ https://issues.apache.org/jira/browse/HADOOP-11485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14284395#comment-14284395 ] Allen Wittenauer commented on HADOOP-11485: --- Let's say I'm an enterprise user that wants to use three other commercial products with my commercial Hadoop distribution: an input format, an encryption library, and an auth handler. Today, doing this requires the Hadoop admin to modify HADOOP_USER_CLASSPATH or shove the contents into the Hadoop binary distribution directories in order to get the daemons to always have these jars available. HADOOP_USER_CLASSPATH is extra bad because this means that the end users of the system will also need to take care that they don't drop these extra classpaths if they also need custom jars for their job. By making this pluggable, 3rd parties can inject their bits in a MUCH safer way plus the end user controls are left open for them to do whatever with. Pluggable shell integration --- Key: HADOOP-11485 URL: https://issues.apache.org/jira/browse/HADOOP-11485 Project: Hadoop Common Issue Type: New Feature Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: scripts, shell Attachments: HADOOP-11485-00.patch, HADOOP-11485-01.patch, HADOOP-11485-02.patch It would be useful to provide a way for core and non-core Hadoop components to plug into the shell infrastructure. This would allow us to pull the HDFS, MapReduce, and YARN shell functions out of hadoop-functions.sh. Additionally, it should let 3rd parties such as HBase influence things like classpaths at runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11485) Pluggable shell integration
[
https://issues.apache.org/jira/browse/HADOOP-11485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14284358#comment-14284358
]
Colin Patrick McCabe commented on HADOOP-11485:
---
Sorry, this is probably a dumb question, but what third parties do we want to
be plugging shell commands in to {{./bin/hadoop}} ? Can't these guys just use
yarn classpath and / or hadoop classpath to get the Hadoop classpath, and
run their own jars? I'm not sure I see a benefit to the increased
complexity... maybe I'm missing something.
Pluggable shell integration
---
Key: HADOOP-11485
URL: https://issues.apache.org/jira/browse/HADOOP-11485
Project: Hadoop Common
Issue Type: New Feature
Components: scripts
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer
Labels: scripts, shell
Attachments: HADOOP-11485-00.patch, HADOOP-11485-01.patch,
HADOOP-11485-02.patch
It would be useful to provide a way for core and non-core Hadoop components
to plug into the shell infrastructure. This would allow us to pull the HDFS,
MapReduce, and YARN shell functions out of hadoop-functions.sh.
Additionally, it should let 3rd parties such as HBase influence things like
classpaths at runtime.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11485) Pluggable shell integration
[
https://issues.apache.org/jira/browse/HADOOP-11485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14284389#comment-14284389
]
Hadoop QA commented on HADOOP-11485:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12693327/HADOOP-11485-02.patch
against trunk revision c94c0d2.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-assemblies hadoop-common-project/hadoop-common
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.ha.TestZKFailoverController
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5435//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5435//console
This message is automatically generated.
Pluggable shell integration
---
Key: HADOOP-11485
URL: https://issues.apache.org/jira/browse/HADOOP-11485
Project: Hadoop Common
Issue Type: New Feature
Components: scripts
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer
Labels: scripts, shell
Attachments: HADOOP-11485-00.patch, HADOOP-11485-01.patch,
HADOOP-11485-02.patch
It would be useful to provide a way for core and non-core Hadoop components
to plug into the shell infrastructure. This would allow us to pull the HDFS,
MapReduce, and YARN shell functions out of hadoop-functions.sh.
Additionally, it should let 3rd parties such as HBase influence things like
classpaths at runtime.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11485) Pluggable shell integration
[
https://issues.apache.org/jira/browse/HADOOP-11485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14283371#comment-14283371
]
Hadoop QA commented on HADOOP-11485:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12693179/HADOOP-11485-01.patch
against trunk revision cb0a15d.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-assemblies hadoop-common-project/hadoop-common
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.ha.TestZKFailoverController
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5432//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5432//artifact/patchprocess/patchReleaseAuditProblems.txt
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5432//console
This message is automatically generated.
Pluggable shell integration
---
Key: HADOOP-11485
URL: https://issues.apache.org/jira/browse/HADOOP-11485
Project: Hadoop Common
Issue Type: New Feature
Components: scripts
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer
Labels: scripts, shell
Attachments: HADOOP-11485-00.patch, HADOOP-11485-01.patch
It would be useful to provide a way for core and non-core Hadoop components
to plug into the shell infrastructure. This would allow us to pull the HDFS,
MapReduce, and YARN shell functions out of hadoop-functions.sh.
Additionally, it should let 3rd parties such as HBase influence things like
classpaths at runtime.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11485) Pluggable shell integration
[ https://issues.apache.org/jira/browse/HADOOP-11485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14279946#comment-14279946 ] Allen Wittenauer commented on HADOOP-11485: --- I'm making HADOOP-11460 required, as it is nearly impossible to give 3rd parties guidance on what they can/cannot change without removing some of the cruft generated by previous shell work. Pluggable shell integration --- Key: HADOOP-11485 URL: https://issues.apache.org/jira/browse/HADOOP-11485 Project: Hadoop Common Issue Type: New Feature Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: scripts, shell Attachments: HADOOP-11485-00.patch It would be useful to provide a way for core and non-core Hadoop components to plug into the shell infrastructure. This would allow us to pull the HDFS, MapReduce, and YARN shell functions out of hadoop-functions.sh. Additionally, it should let 3rd parties such as HBase influence things like classpaths at runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11485) Pluggable shell integration
[
https://issues.apache.org/jira/browse/HADOOP-11485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14278931#comment-14278931
]
Allen Wittenauer commented on HADOOP-11485:
---
I'm specifically thinking of doing something like this:
* directory off of HADOOP_CONF_DIR or HADOOP_LIBEXEC_DIR that contains shell
fragments
* initializer that uses hadoop_add_colonpath to add a prefix to a var
(HADOOP_SHELLFRAG or something)
* each shell frag could define the following functions:
{code}
_(frag)_hadoop_classpath
_(frag)_hadoop_init
_(frag)_hadoop_finalizer
{code}
i.e., the current hadoop_add_to_classpath_yarn would get moved out of
hadoop-functions.sh into this fragment file and renamed to
_yarn_hadoop_classpath.
A few other notes:
* HADOOP_CONF_DIR would need to get moved to first in to last in+prepend. It
must *always* be first in the classpath. We don't want 3rd parties coming in
front.
* We could provide no guarantees, really, as to when a jar appears in the
classpath using this method. So this wouldn't be a way to override classes.
* Currently, the only way to manage $\@ via this fragment is going to be on
source'ing it. So sourcing will happen after we do our normal shell param
processing. So hadoop-common options will need to come first, 3rd party after,
followed by the appropriate shell subcommand. e.g., yarn --conf foo
--hbaseconf bar jar myhbase.jar
Pluggable shell integration
---
Key: HADOOP-11485
URL: https://issues.apache.org/jira/browse/HADOOP-11485
Project: Hadoop Common
Issue Type: New Feature
Reporter: Allen Wittenauer
It would be useful to provide a way for core and non-core Hadoop components
to plug into the shell infrastructure. This would allow us to pull the HDFS,
MapReduce, and YARN components out of common. Additionally, it should let
3rd parties such as HBase influence things like classpaths at runtime.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
