[
https://issues.apache.org/jira/browse/HADOOP-785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12520679
]
Doug Cutting commented on HADOOP-785:
-------------------------------------
Owen is concerned that having different files with different semantics (initial
versus final) is confusing. That we should rather just have a list of files
(e.g., hadoop-default.xml, hadoop-site.xml, job.xml) that are all treated
identically. That has merit. It is simpler.
But how do we specify that some parameters may not be overridden by files later
in the list? Instead of having separate files for that, perhaps we can
annotate the parameters themselves, adding a <final/> tag or somesuch to their
definitions. The first 'final' value found for a parameter when processing the
files would determine the value: no values in subsequent files would modify the
value of that parameter. Thus, in a tasktracker's hadoop-site.xml, the
dfs.client.buffer.dir would be set final, and a job would not be able to
override it, while the job could override the non-final dfs.block.size set
there.
Owen, does this address your concern?
> Divide the server and client configurations
> -------------------------------------------
>
> Key: HADOOP-785
> URL: https://issues.apache.org/jira/browse/HADOOP-785
> Project: Hadoop
> Issue Type: Improvement
> Components: conf
> Affects Versions: 0.9.0
> Reporter: Owen O'Malley
> Assignee: Arun C Murthy
> Fix For: 0.15.0
>
>
> The configuration system is easy to misconfigure and I think we need to
> strongly divide the server from client configs.
> An example of the problem was a configuration where the task tracker has a
> hadoop-site.xml that set mapred.reduce.tasks to 1. Therefore, the job tracker
> had the right number of reduces, but the map task thought there was a single
> reduce. This lead to a hard to find diagnose failure.
> Therefore, I propose separating out the configuration types as:
> class Configuration;
> // reads site-default.xml, hadoop-default.xml
> class ServerConf extends Configuration;
> // reads hadoop-server.xml, $super
> class DfsServerConf extends ServerConf;
> // reads dfs-server.xml, $super
> class MapRedServerConf extends ServerConf;
> // reads mapred-server.xml, $super
> class ClientConf extends Configuration;
> // reads hadoop-client.xml, $super
> class JobConf extends ClientConf;
> // reads job.xml, $super
> Note in particular, that nothing corresponds to hadoop-site.xml, which
> overrides both client and server configs. Furthermore, the properties from
> the *-default.xml files should never be saved into the job.xml.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
