[jira] [Updated] (HADOOP-8270) hadoop-daemon.sh stop action is not LSB compliant
[
https://issues.apache.org/jira/browse/HADOOP-8270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roman Shaposhnik updated HADOOP-8270:
-
Status: Patch Available (was: Open)
> hadoop-daemon.sh stop action is not LSB compliant
> -
>
> Key: HADOOP-8270
> URL: https://issues.apache.org/jira/browse/HADOOP-8270
> Project: Hadoop Common
> Issue Type: Bug
> Components: scripts
>Affects Versions: 0.23.1
>Reporter: Roman Shaposhnik
>Assignee: Roman Shaposhnik
> Fix For: 2.0.0
>
> Attachments: HADOOP-8270.patch.txt
>
>
> The following bit of code from hadoop-daemon.sh is not LSB compliant, since
> according to
> http://refspecs.linuxbase.org/LSB_3.1.0/LSB-Core-generic/LSB-Core-generic/iniscrptact.html
> a stop action on an already stopped service should return 0.
> {noformat}
> (stop)
> if [ -f $pid ]; then
> if kill -0 `cat $pid` > /dev/null 2>&1; then
> echo stopping $command
> kill `cat $pid`
> else
> echo no $command to stop
> exit 1
> fi
> else
> echo no $command to stop
> exit 1
> fi
> ;;
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8270) hadoop-daemon.sh stop action is not LSB compliant
[
https://issues.apache.org/jira/browse/HADOOP-8270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roman Shaposhnik updated HADOOP-8270:
-
Attachment: HADOOP-8270.patch.txt
> hadoop-daemon.sh stop action is not LSB compliant
> -
>
> Key: HADOOP-8270
> URL: https://issues.apache.org/jira/browse/HADOOP-8270
> Project: Hadoop Common
> Issue Type: Bug
> Components: scripts
>Affects Versions: 0.23.1
>Reporter: Roman Shaposhnik
>Assignee: Roman Shaposhnik
> Fix For: 2.0.0
>
> Attachments: HADOOP-8270.patch.txt
>
>
> The following bit of code from hadoop-daemon.sh is not LSB compliant, since
> according to
> http://refspecs.linuxbase.org/LSB_3.1.0/LSB-Core-generic/LSB-Core-generic/iniscrptact.html
> a stop action on an already stopped service should return 0.
> {noformat}
> (stop)
> if [ -f $pid ]; then
> if kill -0 `cat $pid` > /dev/null 2>&1; then
> echo stopping $command
> kill `cat $pid`
> else
> echo no $command to stop
> exit 1
> fi
> else
> echo no $command to stop
> exit 1
> fi
> ;;
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8226) a few native libraries are missing from a full build and binary tarball
[
https://issues.apache.org/jira/browse/HADOOP-8226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roman Shaposhnik updated HADOOP-8226:
-
Status: Patch Available (was: Open)
> a few native libraries are missing from a full build and binary tarball
> ---
>
> Key: HADOOP-8226
> URL: https://issues.apache.org/jira/browse/HADOOP-8226
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 0.23.1
>Reporter: Roman Shaposhnik
>Assignee: Roman Shaposhnik
> Attachments: HADOOP-8226.patch.txt
>
>
> Hadoop 1.0.X ships the following native libraries in its tarball:
> {noformat}
> hadoop-1.0.1/c++/Linux-amd64-64/lib/libhadooppipes.a
> hadoop-1.0.1/c++/Linux-amd64-64/lib/libhadooputils.a
> hadoop-1.0.1/c++/Linux-i386-32/lib/libhadooppipes.a
> hadoop-1.0.1/c++/Linux-i386-32/lib/libhadooputils.a
> hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.a
> hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.a
> hadoop-1.0.1/c++/Linux-amd64-64/lib/libhdfs.so
> hadoop-1.0.1/c++/Linux-i386-32/lib/libhdfs.so
> hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so
> hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so
> {noformat}
> While Hadoop 0.23.1 ships a subset:
> {noformat}
> ./lib/native/libhadoop.a
> ./lib/native/libhdfs.a
> ./lib/native/libhadoop.so
> ./lib/native/libhdfs.so
> {noformat}
> The question is: do we have any reason not to support/publish the following
> libraries:
> libhadooppipes.a, libhadooputils.a ?
> And we also used to have librecordio that now seems to be missing from both.
> If this is not intentional I can provide a patch to re-enable these bits.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8226) a few native libraries are missing from a full build and binary tarball
[
https://issues.apache.org/jira/browse/HADOOP-8226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roman Shaposhnik updated HADOOP-8226:
-
Attachment: HADOOP-8226.patch.txt
Here's the first cut at the patch. It makes hadooppipes and utils bits appear
under include and lib subdirectories and it also makes C++ examples appear
under examples.
Please let me know if you'd rather have me integrate the native parts of the
build with make-maven-plugin.
> a few native libraries are missing from a full build and binary tarball
> ---
>
> Key: HADOOP-8226
> URL: https://issues.apache.org/jira/browse/HADOOP-8226
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 0.23.1
>Reporter: Roman Shaposhnik
>Assignee: Roman Shaposhnik
> Attachments: HADOOP-8226.patch.txt
>
>
> Hadoop 1.0.X ships the following native libraries in its tarball:
> {noformat}
> hadoop-1.0.1/c++/Linux-amd64-64/lib/libhadooppipes.a
> hadoop-1.0.1/c++/Linux-amd64-64/lib/libhadooputils.a
> hadoop-1.0.1/c++/Linux-i386-32/lib/libhadooppipes.a
> hadoop-1.0.1/c++/Linux-i386-32/lib/libhadooputils.a
> hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.a
> hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.a
> hadoop-1.0.1/c++/Linux-amd64-64/lib/libhdfs.so
> hadoop-1.0.1/c++/Linux-i386-32/lib/libhdfs.so
> hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so
> hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so
> {noformat}
> While Hadoop 0.23.1 ships a subset:
> {noformat}
> ./lib/native/libhadoop.a
> ./lib/native/libhdfs.a
> ./lib/native/libhadoop.so
> ./lib/native/libhdfs.so
> {noformat}
> The question is: do we have any reason not to support/publish the following
> libraries:
> libhadooppipes.a, libhadooputils.a ?
> And we also used to have librecordio that now seems to be missing from both.
> If this is not intentional I can provide a patch to re-enable these bits.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8125) make hadoop-client set of curated jars available in a distribution tarball
[ https://issues.apache.org/jira/browse/HADOOP-8125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Shaposhnik updated HADOOP-8125: - Attachment: HADOOP-8125-4.patch.txt A colleague of mine suggested even simpler solution then all of the above. Attaching. > make hadoop-client set of curated jars available in a distribution tarball > -- > > Key: HADOOP-8125 > URL: https://issues.apache.org/jira/browse/HADOOP-8125 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 0.23.1 >Reporter: Roman Shaposhnik >Assignee: Roman Shaposhnik > Attachments: HADOOP-8125-2.patch.txt, HADOOP-8125-3.patch.txt, > HADOOP-8125-4.patch.txt, HADOOP-8125.patch.txt > > > One thing that the original patch for HADOOP-8082 didn't address is the need > for those curated jars to be visible in the final tarball. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8125) make hadoop-client set of curated jars available in a distribution tarball
[ https://issues.apache.org/jira/browse/HADOOP-8125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Shaposhnik updated HADOOP-8125: - Attachment: HADOOP-8125-3.patch.txt Attaching a minimalistic version of the patch that has an assembly creation part but doesn't push the bits into the final tarball. For the Bigtop purposes either one of the patches would suffice. Feel free to pick one that works better for the Hadoop community. > make hadoop-client set of curated jars available in a distribution tarball > -- > > Key: HADOOP-8125 > URL: https://issues.apache.org/jira/browse/HADOOP-8125 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 0.23.1 >Reporter: Roman Shaposhnik >Assignee: Roman Shaposhnik > Attachments: HADOOP-8125-2.patch.txt, HADOOP-8125-3.patch.txt, > HADOOP-8125.patch.txt > > > One thing that the original patch for HADOOP-8082 didn't address is the need > for those curated jars to be visible in the final tarball. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8125) make hadoop-client set of curated jars available in a distribution tarball
[
https://issues.apache.org/jira/browse/HADOOP-8125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roman Shaposhnik updated HADOOP-8125:
-
Attachment: HADOOP-8125-2.patch.txt
Attaching a second version of the patch that takes care of all the raised
concerns. Tested with:
{noformat}
$ mvn clean package -DskipTests -Pdist
$ ls -l hadoop-dist/target/hadoop*/share/hadoop/client/
{noformat}
on Ubuntu Linux
> make hadoop-client set of curated jars available in a distribution tarball
> --
>
> Key: HADOOP-8125
> URL: https://issues.apache.org/jira/browse/HADOOP-8125
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
>Affects Versions: 0.23.1
>Reporter: Roman Shaposhnik
>Assignee: Roman Shaposhnik
> Attachments: HADOOP-8125-2.patch.txt, HADOOP-8125.patch.txt
>
>
> One thing that the original patch for HADOOP-8082 didn't address is the need
> for those curated jars to be visible in the final tarball.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8125) make hadoop-client set of curated jars available in a distribution tarball
[ https://issues.apache.org/jira/browse/HADOOP-8125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Shaposhnik updated HADOOP-8125: - Status: Patch Available (was: Open) > make hadoop-client set of curated jars available in a distribution tarball > -- > > Key: HADOOP-8125 > URL: https://issues.apache.org/jira/browse/HADOOP-8125 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 0.23.1 >Reporter: Roman Shaposhnik >Assignee: Roman Shaposhnik > Attachments: HADOOP-8125.patch.txt > > > One thing that the original patch for HADOOP-8082 didn't address is the need > for those curated jars to be visible in the final tarball. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8125) make hadoop-client set of curated jars available in a distribution tarball
[ https://issues.apache.org/jira/browse/HADOOP-8125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Shaposhnik updated HADOOP-8125: - Attachment: HADOOP-8125.patch.txt Attaching a patch that creates share/hadoop/client in the final tarball and populate it with symlinks to the actual jar files. It is now safe for downstream components to simply add share/hadoop/client and share/hadoop/client/lib to their classpath without getting extra dependencies that can be dangerous in a client type environment (like getting jettty/tomcat stuff) > make hadoop-client set of curated jars available in a distribution tarball > -- > > Key: HADOOP-8125 > URL: https://issues.apache.org/jira/browse/HADOOP-8125 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 0.23.1 >Reporter: Roman Shaposhnik >Assignee: Roman Shaposhnik > Attachments: HADOOP-8125.patch.txt > > > One thing that the original patch for HADOOP-8082 didn't address is the need > for those curated jars to be visible in the final tarball. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8012) hadoop-daemon.sh and yarn-daemon.sh are trying to mkdir and chow log/pid dirs which can fail
[
https://issues.apache.org/jira/browse/HADOOP-8012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roman Shaposhnik updated HADOOP-8012:
-
Status: Patch Available (was: Open)
> hadoop-daemon.sh and yarn-daemon.sh are trying to mkdir and chow log/pid dirs
> which can fail
>
>
> Key: HADOOP-8012
> URL: https://issues.apache.org/jira/browse/HADOOP-8012
> Project: Hadoop Common
> Issue Type: Bug
> Components: scripts
>Affects Versions: 0.23.0
>Reporter: Roman Shaposhnik
>Assignee: Roman Shaposhnik
>Priority: Minor
> Fix For: 0.23.1
>
> Attachments: HADOOP-8012.patch.txt
>
>
> Here's what I see when using Hadoop in Bigtop:
> {noformat}
> $ sudo /sbin/service hadoop-hdfs-namenode start
> Starting Hadoop namenode daemon (hadoop-namenode): chown: changing ownership
> of `/var/log/hadoop': Operation not permitted
> starting namenode, logging to /var/log/hadoop/hadoop-hdfs-namenode-centos5.out
> {noformat}
> This is a cosmetic issue, but it would be nice to fix it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8012) hadoop-daemon.sh and yarn-daemon.sh are trying to mkdir and chow log/pid dirs which can fail
[
https://issues.apache.org/jira/browse/HADOOP-8012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roman Shaposhnik updated HADOOP-8012:
-
Attachment: HADOOP-8012.patch.txt
> hadoop-daemon.sh and yarn-daemon.sh are trying to mkdir and chow log/pid dirs
> which can fail
>
>
> Key: HADOOP-8012
> URL: https://issues.apache.org/jira/browse/HADOOP-8012
> Project: Hadoop Common
> Issue Type: Bug
> Components: scripts
>Affects Versions: 0.23.0
>Reporter: Roman Shaposhnik
>Assignee: Roman Shaposhnik
>Priority: Minor
> Fix For: 0.23.1
>
> Attachments: HADOOP-8012.patch.txt
>
>
> Here's what I see when using Hadoop in Bigtop:
> {noformat}
> $ sudo /sbin/service hadoop-hdfs-namenode start
> Starting Hadoop namenode daemon (hadoop-namenode): chown: changing ownership
> of `/var/log/hadoop': Operation not permitted
> starting namenode, logging to /var/log/hadoop/hadoop-hdfs-namenode-centos5.out
> {noformat}
> This is a cosmetic issue, but it would be nice to fix it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8010) hadoop-config.sh spews error message when HADOOP_HOME_WARN_SUPPRESS is set to true and HADOOP_HOME is present
[
https://issues.apache.org/jira/browse/HADOOP-8010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roman Shaposhnik updated HADOOP-8010:
-
Status: Patch Available (was: Open)
> hadoop-config.sh spews error message when HADOOP_HOME_WARN_SUPPRESS is set to
> true and HADOOP_HOME is present
> -
>
> Key: HADOOP-8010
> URL: https://issues.apache.org/jira/browse/HADOOP-8010
> Project: Hadoop Common
> Issue Type: Bug
> Components: scripts
>Affects Versions: 1.0.0
>Reporter: Roman Shaposhnik
>Assignee: Roman Shaposhnik
>Priority: Minor
> Fix For: 1.0.1
>
> Attachments: HADOOP-8010.patch.txt
>
>
> Running hadoop daemon commands when HADOOP_HOME_WARN_SUPPRESS is set to true
> and HADOOP_HOME is present produces:
> {noformat}
> [: 76: true: unexpected operator
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8010) hadoop-config.sh spews error message when HADOOP_HOME_WARN_SUPPRESS is set to true and HADOOP_HOME is present
[
https://issues.apache.org/jira/browse/HADOOP-8010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roman Shaposhnik updated HADOOP-8010:
-
Attachment: HADOOP-8010.patch.txt
> hadoop-config.sh spews error message when HADOOP_HOME_WARN_SUPPRESS is set to
> true and HADOOP_HOME is present
> -
>
> Key: HADOOP-8010
> URL: https://issues.apache.org/jira/browse/HADOOP-8010
> Project: Hadoop Common
> Issue Type: Bug
> Components: scripts
>Affects Versions: 1.0.0
>Reporter: Roman Shaposhnik
>Assignee: Roman Shaposhnik
>Priority: Minor
> Fix For: 1.0.1
>
> Attachments: HADOOP-8010.patch.txt
>
>
> Running hadoop daemon commands when HADOOP_HOME_WARN_SUPPRESS is set to true
> and HADOOP_HOME is present produces:
> {noformat}
> [: 76: true: unexpected operator
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7939) Improve Hadoop subcomponent integration in Hadoop 0.23
[ https://issues.apache.org/jira/browse/HADOOP-7939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Shaposhnik updated HADOOP-7939: - Attachment: HADOOP-7939-simplified-3.patch.txt Patch updated > Improve Hadoop subcomponent integration in Hadoop 0.23 > -- > > Key: HADOOP-7939 > URL: https://issues.apache.org/jira/browse/HADOOP-7939 > Project: Hadoop Common > Issue Type: Improvement > Components: build, conf, documentation, scripts >Affects Versions: 0.23.0 >Reporter: Roman Shaposhnik >Assignee: Roman Shaposhnik > Fix For: 0.23.1 > > Attachments: HADOOP-7939-simplified-2.patch.txt, > HADOOP-7939-simplified-3.patch.txt, HADOOP-7939-simplified.patch.txt, > HADOOP-7939.patch.txt, hadoop-layout.sh > > > h1. Introduction > For the rest of this proposal it is assumed that the current set > of Hadoop subcomponents is: > * hadoop-common > * hadoop-hdfs > * hadoop-yarn > * hadoop-mapreduce > It must be noted that this is an open ended list, though. For example, > implementations of additional frameworks on top of yarn (e.g. MPI) would > also be considered a subcomponent. > h1. Problem statement > Currently there's an unfortunate coupling and hard-coding present at the > level of launcher scripts, configuration scripts and Java implementation > code that prevents us from treating all subcomponents of Hadoop independently > of each other. In a lot of places it is assumed that bits and pieces > from individual subcomponents *must* be located at predefined places > and they can not be dynamically registered/discovered during the runtime. > This prevents a truly flexible deployment of Hadoop 0.23. > h1. Proposal > NOTE: this is NOT a proposal for redefining the layout from HADOOP-6255. > The goal here is to keep as much of that layout in place as possible, > while permitting different deployment layouts. > The aim of this proposal is to introduce the needed level of indirection and > flexibility in order to accommodate the current assumed layout of Hadoop > tarball > deployments and all the other styles of deployments as well. To this end the > following set of environment variables needs to be uniformly used in all of > the subcomponent's launcher scripts, configuration scripts and Java code > ( stands for a literal name of a subcomponent). These variables are > expected to be defined by -env.sh scripts and sourcing those files is > expected to have the desired effect of setting the environment up correctly. > # HADOOP__HOME >## root of the subtree in a filesystem where a subcomponent is expected to > be installed >## default value: $0/.. > # HADOOP__JARS >## a subdirectory with all of the jar files comprising subcomponent's > implementation >## default value: $(HADOOP__HOME)/share/hadoop/$() > # HADOOP__EXT_JARS >## a subdirectory with all of the jar files needed for extended > functionality of the subcomponent (nonessential for correct work of the basic > functionality) >## default value: $(HADOOP__HOME)/share/hadoop/$()/ext > # HADOOP__NATIVE_LIBS >## a subdirectory with all the native libraries that component requires >## default value: $(HADOOP__HOME)/share/hadoop/$()/native > # HADOOP__BIN >## a subdirectory with all of the launcher scripts specific to the client > side of the component >## default value: $(HADOOP__HOME)/bin > # HADOOP__SBIN >## a subdirectory with all of the launcher scripts specific to the > server/system side of the component >## default value: $(HADOOP__HOME)/sbin > # HADOOP__LIBEXEC >## a subdirectory with all of the launcher scripts that are internal to > the implementation and should *not* be invoked directly >## default value: $(HADOOP__HOME)/libexec > # HADOOP__CONF >## a subdirectory containing configuration files for a subcomponent >## default value: $(HADOOP__HOME)/conf > # HADOOP__DATA >## a subtree in the local filesystem for storing component's persistent > state >## default value: $(HADOOP__HOME)/data > # HADOOP__LOG >## a subdirectory for subcomponents's log files to be stored >## default value: $(HADOOP__HOME)/log > # HADOOP__RUN >## a subdirectory with runtime system specific information >## default value: $(HADOOP__HOME)/run > # HADOOP__TMP >## a subdirectory with temprorary files >## default value: $(HADOOP__HOME)/tmp -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7939) Improve Hadoop subcomponent integration in Hadoop 0.23
[ https://issues.apache.org/jira/browse/HADOOP-7939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Shaposhnik updated HADOOP-7939: - Attachment: HADOOP-7939-simplified-2.patch.txt > Improve Hadoop subcomponent integration in Hadoop 0.23 > -- > > Key: HADOOP-7939 > URL: https://issues.apache.org/jira/browse/HADOOP-7939 > Project: Hadoop Common > Issue Type: Improvement > Components: build, conf, documentation, scripts >Affects Versions: 0.23.0 >Reporter: Roman Shaposhnik >Assignee: Roman Shaposhnik > Fix For: 0.23.1 > > Attachments: HADOOP-7939-simplified-2.patch.txt, > HADOOP-7939-simplified.patch.txt, HADOOP-7939.patch.txt, hadoop-layout.sh > > > h1. Introduction > For the rest of this proposal it is assumed that the current set > of Hadoop subcomponents is: > * hadoop-common > * hadoop-hdfs > * hadoop-yarn > * hadoop-mapreduce > It must be noted that this is an open ended list, though. For example, > implementations of additional frameworks on top of yarn (e.g. MPI) would > also be considered a subcomponent. > h1. Problem statement > Currently there's an unfortunate coupling and hard-coding present at the > level of launcher scripts, configuration scripts and Java implementation > code that prevents us from treating all subcomponents of Hadoop independently > of each other. In a lot of places it is assumed that bits and pieces > from individual subcomponents *must* be located at predefined places > and they can not be dynamically registered/discovered during the runtime. > This prevents a truly flexible deployment of Hadoop 0.23. > h1. Proposal > NOTE: this is NOT a proposal for redefining the layout from HADOOP-6255. > The goal here is to keep as much of that layout in place as possible, > while permitting different deployment layouts. > The aim of this proposal is to introduce the needed level of indirection and > flexibility in order to accommodate the current assumed layout of Hadoop > tarball > deployments and all the other styles of deployments as well. To this end the > following set of environment variables needs to be uniformly used in all of > the subcomponent's launcher scripts, configuration scripts and Java code > ( stands for a literal name of a subcomponent). These variables are > expected to be defined by -env.sh scripts and sourcing those files is > expected to have the desired effect of setting the environment up correctly. > # HADOOP__HOME >## root of the subtree in a filesystem where a subcomponent is expected to > be installed >## default value: $0/.. > # HADOOP__JARS >## a subdirectory with all of the jar files comprising subcomponent's > implementation >## default value: $(HADOOP__HOME)/share/hadoop/$() > # HADOOP__EXT_JARS >## a subdirectory with all of the jar files needed for extended > functionality of the subcomponent (nonessential for correct work of the basic > functionality) >## default value: $(HADOOP__HOME)/share/hadoop/$()/ext > # HADOOP__NATIVE_LIBS >## a subdirectory with all the native libraries that component requires >## default value: $(HADOOP__HOME)/share/hadoop/$()/native > # HADOOP__BIN >## a subdirectory with all of the launcher scripts specific to the client > side of the component >## default value: $(HADOOP__HOME)/bin > # HADOOP__SBIN >## a subdirectory with all of the launcher scripts specific to the > server/system side of the component >## default value: $(HADOOP__HOME)/sbin > # HADOOP__LIBEXEC >## a subdirectory with all of the launcher scripts that are internal to > the implementation and should *not* be invoked directly >## default value: $(HADOOP__HOME)/libexec > # HADOOP__CONF >## a subdirectory containing configuration files for a subcomponent >## default value: $(HADOOP__HOME)/conf > # HADOOP__DATA >## a subtree in the local filesystem for storing component's persistent > state >## default value: $(HADOOP__HOME)/data > # HADOOP__LOG >## a subdirectory for subcomponents's log files to be stored >## default value: $(HADOOP__HOME)/log > # HADOOP__RUN >## a subdirectory with runtime system specific information >## default value: $(HADOOP__HOME)/run > # HADOOP__TMP >## a subdirectory with temprorary files >## default value: $(HADOOP__HOME)/tmp -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7939) Improve Hadoop subcomponent integration in Hadoop 0.23
[ https://issues.apache.org/jira/browse/HADOOP-7939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Shaposhnik updated HADOOP-7939: - Status: Patch Available (was: Open) > Improve Hadoop subcomponent integration in Hadoop 0.23 > -- > > Key: HADOOP-7939 > URL: https://issues.apache.org/jira/browse/HADOOP-7939 > Project: Hadoop Common > Issue Type: Improvement > Components: build, conf, documentation, scripts >Affects Versions: 0.23.0 >Reporter: Roman Shaposhnik >Assignee: Roman Shaposhnik > Fix For: 0.23.1 > > Attachments: HADOOP-7939-simplified.patch.txt, HADOOP-7939.patch.txt, > hadoop-layout.sh > > > h1. Introduction > For the rest of this proposal it is assumed that the current set > of Hadoop subcomponents is: > * hadoop-common > * hadoop-hdfs > * hadoop-yarn > * hadoop-mapreduce > It must be noted that this is an open ended list, though. For example, > implementations of additional frameworks on top of yarn (e.g. MPI) would > also be considered a subcomponent. > h1. Problem statement > Currently there's an unfortunate coupling and hard-coding present at the > level of launcher scripts, configuration scripts and Java implementation > code that prevents us from treating all subcomponents of Hadoop independently > of each other. In a lot of places it is assumed that bits and pieces > from individual subcomponents *must* be located at predefined places > and they can not be dynamically registered/discovered during the runtime. > This prevents a truly flexible deployment of Hadoop 0.23. > h1. Proposal > NOTE: this is NOT a proposal for redefining the layout from HADOOP-6255. > The goal here is to keep as much of that layout in place as possible, > while permitting different deployment layouts. > The aim of this proposal is to introduce the needed level of indirection and > flexibility in order to accommodate the current assumed layout of Hadoop > tarball > deployments and all the other styles of deployments as well. To this end the > following set of environment variables needs to be uniformly used in all of > the subcomponent's launcher scripts, configuration scripts and Java code > ( stands for a literal name of a subcomponent). These variables are > expected to be defined by -env.sh scripts and sourcing those files is > expected to have the desired effect of setting the environment up correctly. > # HADOOP__HOME >## root of the subtree in a filesystem where a subcomponent is expected to > be installed >## default value: $0/.. > # HADOOP__JARS >## a subdirectory with all of the jar files comprising subcomponent's > implementation >## default value: $(HADOOP__HOME)/share/hadoop/$() > # HADOOP__EXT_JARS >## a subdirectory with all of the jar files needed for extended > functionality of the subcomponent (nonessential for correct work of the basic > functionality) >## default value: $(HADOOP__HOME)/share/hadoop/$()/ext > # HADOOP__NATIVE_LIBS >## a subdirectory with all the native libraries that component requires >## default value: $(HADOOP__HOME)/share/hadoop/$()/native > # HADOOP__BIN >## a subdirectory with all of the launcher scripts specific to the client > side of the component >## default value: $(HADOOP__HOME)/bin > # HADOOP__SBIN >## a subdirectory with all of the launcher scripts specific to the > server/system side of the component >## default value: $(HADOOP__HOME)/sbin > # HADOOP__LIBEXEC >## a subdirectory with all of the launcher scripts that are internal to > the implementation and should *not* be invoked directly >## default value: $(HADOOP__HOME)/libexec > # HADOOP__CONF >## a subdirectory containing configuration files for a subcomponent >## default value: $(HADOOP__HOME)/conf > # HADOOP__DATA >## a subtree in the local filesystem for storing component's persistent > state >## default value: $(HADOOP__HOME)/data > # HADOOP__LOG >## a subdirectory for subcomponents's log files to be stored >## default value: $(HADOOP__HOME)/log > # HADOOP__RUN >## a subdirectory with runtime system specific information >## default value: $(HADOOP__HOME)/run > # HADOOP__TMP >## a subdirectory with temprorary files >## default value: $(HADOOP__HOME)/tmp -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7939) Improve Hadoop subcomponent integration in Hadoop 0.23
[ https://issues.apache.org/jira/browse/HADOOP-7939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Shaposhnik updated HADOOP-7939: - Attachment: HADOOP-7939-simplified.patch.txt Thanks for all the feedback. Based on it I decided to go ahead with a very simplified form of an original proposal. A patch that is now attached doesn't try to enforce any kind of naming convention, etc. It simply replaces all the hardcoded string constants with variables. That's it. Hopefully this will be small enough change to be accepted without much debate. > Improve Hadoop subcomponent integration in Hadoop 0.23 > -- > > Key: HADOOP-7939 > URL: https://issues.apache.org/jira/browse/HADOOP-7939 > Project: Hadoop Common > Issue Type: Improvement > Components: build, conf, documentation, scripts >Affects Versions: 0.23.0 >Reporter: Roman Shaposhnik >Assignee: Roman Shaposhnik > Fix For: 0.23.1 > > Attachments: HADOOP-7939-simplified.patch.txt, HADOOP-7939.patch.txt, > hadoop-layout.sh > > > h1. Introduction > For the rest of this proposal it is assumed that the current set > of Hadoop subcomponents is: > * hadoop-common > * hadoop-hdfs > * hadoop-yarn > * hadoop-mapreduce > It must be noted that this is an open ended list, though. For example, > implementations of additional frameworks on top of yarn (e.g. MPI) would > also be considered a subcomponent. > h1. Problem statement > Currently there's an unfortunate coupling and hard-coding present at the > level of launcher scripts, configuration scripts and Java implementation > code that prevents us from treating all subcomponents of Hadoop independently > of each other. In a lot of places it is assumed that bits and pieces > from individual subcomponents *must* be located at predefined places > and they can not be dynamically registered/discovered during the runtime. > This prevents a truly flexible deployment of Hadoop 0.23. > h1. Proposal > NOTE: this is NOT a proposal for redefining the layout from HADOOP-6255. > The goal here is to keep as much of that layout in place as possible, > while permitting different deployment layouts. > The aim of this proposal is to introduce the needed level of indirection and > flexibility in order to accommodate the current assumed layout of Hadoop > tarball > deployments and all the other styles of deployments as well. To this end the > following set of environment variables needs to be uniformly used in all of > the subcomponent's launcher scripts, configuration scripts and Java code > ( stands for a literal name of a subcomponent). These variables are > expected to be defined by -env.sh scripts and sourcing those files is > expected to have the desired effect of setting the environment up correctly. > # HADOOP__HOME >## root of the subtree in a filesystem where a subcomponent is expected to > be installed >## default value: $0/.. > # HADOOP__JARS >## a subdirectory with all of the jar files comprising subcomponent's > implementation >## default value: $(HADOOP__HOME)/share/hadoop/$() > # HADOOP__EXT_JARS >## a subdirectory with all of the jar files needed for extended > functionality of the subcomponent (nonessential for correct work of the basic > functionality) >## default value: $(HADOOP__HOME)/share/hadoop/$()/ext > # HADOOP__NATIVE_LIBS >## a subdirectory with all the native libraries that component requires >## default value: $(HADOOP__HOME)/share/hadoop/$()/native > # HADOOP__BIN >## a subdirectory with all of the launcher scripts specific to the client > side of the component >## default value: $(HADOOP__HOME)/bin > # HADOOP__SBIN >## a subdirectory with all of the launcher scripts specific to the > server/system side of the component >## default value: $(HADOOP__HOME)/sbin > # HADOOP__LIBEXEC >## a subdirectory with all of the launcher scripts that are internal to > the implementation and should *not* be invoked directly >## default value: $(HADOOP__HOME)/libexec > # HADOOP__CONF >## a subdirectory containing configuration files for a subcomponent >## default value: $(HADOOP__HOME)/conf > # HADOOP__DATA >## a subtree in the local filesystem for storing component's persistent > state >## default value: $(HADOOP__HOME)/data > # HADOOP__LOG >## a subdirectory for subcomponents's log files to be stored >## default value: $(HADOOP__HOME)/log > # HADOOP__RUN >## a subdirectory with runtime system specific information >## default value: $(HADOOP__HOME)/run > # HADOOP__TMP >## a subdirectory with temprorary files >## default value: $(HADOOP__HOME)/tmp -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please
[jira] [Updated] (HADOOP-7939) Improve Hadoop subcomponent integration in Hadoop 0.23
[ https://issues.apache.org/jira/browse/HADOOP-7939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Shaposhnik updated HADOOP-7939: - Attachment: hadoop-layout.sh Attaching a newly introduced file as-is for ease of review > Improve Hadoop subcomponent integration in Hadoop 0.23 > -- > > Key: HADOOP-7939 > URL: https://issues.apache.org/jira/browse/HADOOP-7939 > Project: Hadoop Common > Issue Type: Improvement > Components: build, conf, documentation, scripts >Affects Versions: 0.23.0 >Reporter: Roman Shaposhnik >Assignee: Roman Shaposhnik > Fix For: 0.23.1 > > Attachments: HADOOP-7939.patch.txt, hadoop-layout.sh > > > h1. Introduction > For the rest of this proposal it is assumed that the current set > of Hadoop subcomponents is: > * hadoop-common > * hadoop-hdfs > * hadoop-yarn > * hadoop-mapreduce > It must be noted that this is an open ended list, though. For example, > implementations of additional frameworks on top of yarn (e.g. MPI) would > also be considered a subcomponent. > h1. Problem statement > Currently there's an unfortunate coupling and hard-coding present at the > level of launcher scripts, configuration scripts and Java implementation > code that prevents us from treating all subcomponents of Hadoop independently > of each other. In a lot of places it is assumed that bits and pieces > from individual subcomponents *must* be located at predefined places > and they can not be dynamically registered/discovered during the runtime. > This prevents a truly flexible deployment of Hadoop 0.23. > h1. Proposal > NOTE: this is NOT a proposal for redefining the layout from HADOOP-6255. > The goal here is to keep as much of that layout in place as possible, > while permitting different deployment layouts. > The aim of this proposal is to introduce the needed level of indirection and > flexibility in order to accommodate the current assumed layout of Hadoop > tarball > deployments and all the other styles of deployments as well. To this end the > following set of environment variables needs to be uniformly used in all of > the subcomponent's launcher scripts, configuration scripts and Java code > ( stands for a literal name of a subcomponent). These variables are > expected to be defined by -env.sh scripts and sourcing those files is > expected to have the desired effect of setting the environment up correctly. > # HADOOP__HOME >## root of the subtree in a filesystem where a subcomponent is expected to > be installed >## default value: $0/.. > # HADOOP__JARS >## a subdirectory with all of the jar files comprising subcomponent's > implementation >## default value: $(HADOOP__HOME)/share/hadoop/$() > # HADOOP__EXT_JARS >## a subdirectory with all of the jar files needed for extended > functionality of the subcomponent (nonessential for correct work of the basic > functionality) >## default value: $(HADOOP__HOME)/share/hadoop/$()/ext > # HADOOP__NATIVE_LIBS >## a subdirectory with all the native libraries that component requires >## default value: $(HADOOP__HOME)/share/hadoop/$()/native > # HADOOP__BIN >## a subdirectory with all of the launcher scripts specific to the client > side of the component >## default value: $(HADOOP__HOME)/bin > # HADOOP__SBIN >## a subdirectory with all of the launcher scripts specific to the > server/system side of the component >## default value: $(HADOOP__HOME)/sbin > # HADOOP__LIBEXEC >## a subdirectory with all of the launcher scripts that are internal to > the implementation and should *not* be invoked directly >## default value: $(HADOOP__HOME)/libexec > # HADOOP__CONF >## a subdirectory containing configuration files for a subcomponent >## default value: $(HADOOP__HOME)/conf > # HADOOP__DATA >## a subtree in the local filesystem for storing component's persistent > state >## default value: $(HADOOP__HOME)/data > # HADOOP__LOG >## a subdirectory for subcomponents's log files to be stored >## default value: $(HADOOP__HOME)/log > # HADOOP__RUN >## a subdirectory with runtime system specific information >## default value: $(HADOOP__HOME)/run > # HADOOP__TMP >## a subdirectory with temprorary files >## default value: $(HADOOP__HOME)/tmp -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7939) Improve Hadoop subcomponent integration in Hadoop 0.23
[ https://issues.apache.org/jira/browse/HADOOP-7939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Shaposhnik updated HADOOP-7939: - Attachment: HADOOP-7939.patch.txt First of all, I would like thank everybody for their feedback. And especially Alejandro for summing up the approach I'm taking. I'm attaching a patch for common that, I hope, takes care of some of the concerns raised on this JIRA. I'm also going to open up corresponding JIRAs for HDFS and MAPREDUCE so that I can provide patches for review over there. Please note, that this patch has been somewhat tested on Linux, but hasn't seen any exposure to Cygwin nor MacOS/X nor Solaris. I'm going to do that over the weekend. Once again, thanks to everybody who provided practical feedback and thanks for your time (in advance! ;-)) for reviewing the patch. > Improve Hadoop subcomponent integration in Hadoop 0.23 > -- > > Key: HADOOP-7939 > URL: https://issues.apache.org/jira/browse/HADOOP-7939 > Project: Hadoop Common > Issue Type: Improvement > Components: build, conf, documentation, scripts >Affects Versions: 0.23.0 >Reporter: Roman Shaposhnik >Assignee: Roman Shaposhnik > Fix For: 0.23.1 > > Attachments: HADOOP-7939.patch.txt > > > h1. Introduction > For the rest of this proposal it is assumed that the current set > of Hadoop subcomponents is: > * hadoop-common > * hadoop-hdfs > * hadoop-yarn > * hadoop-mapreduce > It must be noted that this is an open ended list, though. For example, > implementations of additional frameworks on top of yarn (e.g. MPI) would > also be considered a subcomponent. > h1. Problem statement > Currently there's an unfortunate coupling and hard-coding present at the > level of launcher scripts, configuration scripts and Java implementation > code that prevents us from treating all subcomponents of Hadoop independently > of each other. In a lot of places it is assumed that bits and pieces > from individual subcomponents *must* be located at predefined places > and they can not be dynamically registered/discovered during the runtime. > This prevents a truly flexible deployment of Hadoop 0.23. > h1. Proposal > NOTE: this is NOT a proposal for redefining the layout from HADOOP-6255. > The goal here is to keep as much of that layout in place as possible, > while permitting different deployment layouts. > The aim of this proposal is to introduce the needed level of indirection and > flexibility in order to accommodate the current assumed layout of Hadoop > tarball > deployments and all the other styles of deployments as well. To this end the > following set of environment variables needs to be uniformly used in all of > the subcomponent's launcher scripts, configuration scripts and Java code > ( stands for a literal name of a subcomponent). These variables are > expected to be defined by -env.sh scripts and sourcing those files is > expected to have the desired effect of setting the environment up correctly. > # HADOOP__HOME >## root of the subtree in a filesystem where a subcomponent is expected to > be installed >## default value: $0/.. > # HADOOP__JARS >## a subdirectory with all of the jar files comprising subcomponent's > implementation >## default value: $(HADOOP__HOME)/share/hadoop/$() > # HADOOP__EXT_JARS >## a subdirectory with all of the jar files needed for extended > functionality of the subcomponent (nonessential for correct work of the basic > functionality) >## default value: $(HADOOP__HOME)/share/hadoop/$()/ext > # HADOOP__NATIVE_LIBS >## a subdirectory with all the native libraries that component requires >## default value: $(HADOOP__HOME)/share/hadoop/$()/native > # HADOOP__BIN >## a subdirectory with all of the launcher scripts specific to the client > side of the component >## default value: $(HADOOP__HOME)/bin > # HADOOP__SBIN >## a subdirectory with all of the launcher scripts specific to the > server/system side of the component >## default value: $(HADOOP__HOME)/sbin > # HADOOP__LIBEXEC >## a subdirectory with all of the launcher scripts that are internal to > the implementation and should *not* be invoked directly >## default value: $(HADOOP__HOME)/libexec > # HADOOP__CONF >## a subdirectory containing configuration files for a subcomponent >## default value: $(HADOOP__HOME)/conf > # HADOOP__DATA >## a subtree in the local filesystem for storing component's persistent > state >## default value: $(HADOOP__HOME)/data > # HADOOP__LOG >## a subdirectory for subcomponents's log files to be stored >## default value: $(HADOOP__HOME)/log > # HADOOP__RUN >## a subdirectory with runtime system specific information >## default valu
[jira] [Updated] (HADOOP-7730) Allow TestCLI to be run against a cluster
[ https://issues.apache.org/jira/browse/HADOOP-7730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Shaposhnik updated HADOOP-7730: - Affects Version/s: 0.20.205.0 Can we, please, port this to 0.20.205 ? > Allow TestCLI to be run against a cluster > - > > Key: HADOOP-7730 > URL: https://issues.apache.org/jira/browse/HADOOP-7730 > Project: Hadoop Common > Issue Type: Test > Components: test >Affects Versions: 0.20.205.0, 0.22.0 >Reporter: Konstantin Boudnik >Assignee: Konstantin Boudnik > Fix For: 0.22.0 > > Attachments: HADOOP-7730.patch, HADOOP-7730.trunk.patch, > HADOOP-7730.trunk.patch > > > Use the same CLI test to test cluster bits (see HDFS-1762 for more info) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
