[ https://issues.apache.org/jira/browse/HADOOP-7939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13177008#comment-13177008 ]
Roman Shaposhnik commented on HADOOP-7939: ------------------------------------------ @Eli, I strongly disagree with your take on complexity. In fact, while implementing parts of this proposal I was astonished at all of the inconsistencies I've discovered so far in use of *already* present variables. It is next to impossible to reason about scripts in their current form. To me this is the kind of complexity worth eliminating (some of the _HOME variables start with HADOOP_, some don't, some arguments setting vars start with subcomponents name, some don't, etc.). That said, I'm tired of fighting what seems to be a vocal minority on this JIRA. It is fine if this JIRA gets closed as will-not-fix. As far as Bigtop is concerned we always have an option along the lines that Allen has mentioned -- forking and patching the shell scripts to the point of them making sense. In fact, in Bigtop it'll be somewhat easier. As to your counter-proposal of changing the default layout to be a simple flat {noformat} ${HADOOP_XX_HOME}/[bin|lib|...] {noformat} I do not feel comfortable tackling that at the moment. The way I see it, implemeting changes to the layout will require changes in the following areas: # assembly portion of all of the Hadoop subcomponent's Maven builds # RPM/DEB packaging code # config file generation code I don't have familiarity with that portion of the Hadoop project (it is all new code) and it will be a significant investment of time for me to get up to speed. Feel free to tackle it, though. If it happens -- Bigtop's life is going to be a little bit easier. > Improve Hadoop subcomponent integration in Hadoop 0.23 > ------------------------------------------------------ > > Key: HADOOP-7939 > URL: https://issues.apache.org/jira/browse/HADOOP-7939 > Project: Hadoop Common > Issue Type: Improvement > Components: build, conf, documentation, scripts > Affects Versions: 0.23.0 > Reporter: Roman Shaposhnik > Assignee: Roman Shaposhnik > Fix For: 0.23.1 > > > h1. Introduction > For the rest of this proposal it is assumed that the current set > of Hadoop subcomponents is: > * hadoop-common > * hadoop-hdfs > * hadoop-yarn > * hadoop-mapreduce > It must be noted that this is an open ended list, though. For example, > implementations of additional frameworks on top of yarn (e.g. MPI) would > also be considered a subcomponent. > h1. Problem statement > Currently there's an unfortunate coupling and hard-coding present at the > level of launcher scripts, configuration scripts and Java implementation > code that prevents us from treating all subcomponents of Hadoop independently > of each other. In a lot of places it is assumed that bits and pieces > from individual subcomponents *must* be located at predefined places > and they can not be dynamically registered/discovered during the runtime. > This prevents a truly flexible deployment of Hadoop 0.23. > h1. Proposal > NOTE: this is NOT a proposal for redefining the layout from HADOOP-6255. > The goal here is to keep as much of that layout in place as possible, > while permitting different deployment layouts. > The aim of this proposal is to introduce the needed level of indirection and > flexibility in order to accommodate the current assumed layout of Hadoop > tarball > deployments and all the other styles of deployments as well. To this end the > following set of environment variables needs to be uniformly used in all of > the subcomponent's launcher scripts, configuration scripts and Java code > (<SC> stands for a literal name of a subcomponent). These variables are > expected to be defined by <SC>-env.sh scripts and sourcing those files is > expected to have the desired effect of setting the environment up correctly. > # HADOOP_<SC>_HOME > ## root of the subtree in a filesystem where a subcomponent is expected to > be installed > ## default value: $0/.. > # HADOOP_<SC>_JARS > ## a subdirectory with all of the jar files comprising subcomponent's > implementation > ## default value: $(HADOOP_<SC>_HOME)/share/hadoop/$(<SC>) > # HADOOP_<SC>_EXT_JARS > ## a subdirectory with all of the jar files needed for extended > functionality of the subcomponent (nonessential for correct work of the basic > functionality) > ## default value: $(HADOOP_<SC>_HOME)/share/hadoop/$(<SC>)/ext > # HADOOP_<SC>_NATIVE_LIBS > ## a subdirectory with all the native libraries that component requires > ## default value: $(HADOOP_<SC>_HOME)/share/hadoop/$(<SC>)/native > # HADOOP_<SC>_BIN > ## a subdirectory with all of the launcher scripts specific to the client > side of the component > ## default value: $(HADOOP_<SC>_HOME)/bin > # HADOOP_<SC>_SBIN > ## a subdirectory with all of the launcher scripts specific to the > server/system side of the component > ## default value: $(HADOOP_<SC>_HOME)/sbin > # HADOOP_<SC>_LIBEXEC > ## a subdirectory with all of the launcher scripts that are internal to > the implementation and should *not* be invoked directly > ## default value: $(HADOOP_<SC>_HOME)/libexec > # HADOOP_<SC>_CONF > ## a subdirectory containing configuration files for a subcomponent > ## default value: $(HADOOP_<SC>_HOME)/conf > # HADOOP_<SC>_DATA > ## a subtree in the local filesystem for storing component's persistent > state > ## default value: $(HADOOP_<SC>_HOME)/data > # HADOOP_<SC>_LOG > ## a subdirectory for subcomponents's log files to be stored > ## default value: $(HADOOP_<SC>_HOME)/log > # HADOOP_<SC>_RUN > ## a subdirectory with runtime system specific information > ## default value: $(HADOOP_<SC>_HOME)/run > # HADOOP_<SC>_TMP > ## a subdirectory with temprorary files > ## default value: $(HADOOP_<SC>_HOME)/tmp -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira