[
https://issues.apache.org/jira/browse/HADOOP-785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12521222
]
Arun C Murthy commented on HADOOP-785:
--------------------------------------
Here is my final (final final) say on the topic, basically re-stating my
preference:
If *I* was maintaining more than one large cluster, I'd prefer to have a
separate hadoop-final.xml where I'd put not only the varying paths, but also
some of the hardware specific stuff (e.g. {{mapred.tasktracker.tasks.maximum}}
would depend on how beefy a node is, in a given cluster).
Why?
Personally I'd like the convenience of being able to put a small set of
hard-limits in different hadoop-final.xml files
(hadoop-final-cluster{1,2,3}.xml), stick them up in subversion/cvs and being
able to quickly glean insights into the various limits.
IMHO it's harder to do such stuff with a single, large hadoop-site.xml,
especially minus the ability to quickly diff (harder to diff on xml files via
cmd-line diff) etc.
Having said that, I'd like to know what *real* admins think of course! *smile*
> Divide the server and client configurations
> -------------------------------------------
>
> Key: HADOOP-785
> URL: https://issues.apache.org/jira/browse/HADOOP-785
> Project: Hadoop
> Issue Type: Improvement
> Components: conf
> Affects Versions: 0.9.0
> Reporter: Owen O'Malley
> Assignee: Arun C Murthy
> Fix For: 0.15.0
>
>
> The configuration system is easy to misconfigure and I think we need to
> strongly divide the server from client configs.
> An example of the problem was a configuration where the task tracker has a
> hadoop-site.xml that set mapred.reduce.tasks to 1. Therefore, the job tracker
> had the right number of reduces, but the map task thought there was a single
> reduce. This lead to a hard to find diagnose failure.
> Therefore, I propose separating out the configuration types as:
> class Configuration;
> // reads site-default.xml, hadoop-default.xml
> class ServerConf extends Configuration;
> // reads hadoop-server.xml, $super
> class DfsServerConf extends ServerConf;
> // reads dfs-server.xml, $super
> class MapRedServerConf extends ServerConf;
> // reads mapred-server.xml, $super
> class ClientConf extends Configuration;
> // reads hadoop-client.xml, $super
> class JobConf extends ClientConf;
> // reads job.xml, $super
> Note in particular, that nothing corresponds to hadoop-site.xml, which
> overrides both client and server configs. Furthermore, the properties from
> the *-default.xml files should never be saved into the job.xml.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
