[
https://issues.apache.org/jira/browse/MAPREDUCE-861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743165#action_12743165
]
Hemanth Yamijala commented on MAPREDUCE-861:
--------------------------------------------
Current state:
- Queue configuration lives in mapred-queues.xml. It contains the list of
queues, their ACLs and state. The pulling of these properties into a separate
file happened in HADOOP-5396 and HADOOP-5913 in Hadoop 0.21. Scheduler specific
queue properties live in capacity-scheduler.xml. They have the same format of
property names and values as Hadoop configuration (like mapred-site.xml).
- For backwards compatibility, the queue configuration is also supported in
mapred-site.xml, but is deprecated.
The considerations are:
- To change the format of the file to be more friendly towards hierarchical
queues. The file format can be based on a well defined XML schema that we can
discuss and agree upon, that makes sense for nested queues. The other option
for the file format was JSON. However, in internal discussions, it was decided
that it is best not to introduce a new format for managing this information
now. XML -> XML migration is less intrusive. Also, it is similar in format to
other configuration in Hadoop.
- We have an option to merge the contents of mapred-queues.xml and
capacity-scheduler.xml. It seems good to be able to do this because then the
nested hierarchy of queues will be defined in exactly one place and hence
chances of users making mistakes in configuring the hierarchy is lesser.
Otherwise, the need for validation tools greatly increases.
- If we do agree to merge the contents, we may want to make scheduler specific
properties be treated as a black-box for the framework. An option could be to
treat it as string contents. And QueueManager could provide an API like
getSchedulerProperties(queue), which returns a string.
- For backwards compatibility, queue properties can still be defined in
mapred-site.xml in the old format, but they cannot define hierarchies of queues
there.
Does this make sense overall ?
> Modify queue configuration format and parsing to support a hierarchy of
> queues.
> -------------------------------------------------------------------------------
>
> Key: MAPREDUCE-861
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-861
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Reporter: Hemanth Yamijala
> Assignee: rahul k singh
>
> MAPREDUCE-853 proposes to introduce a hierarchy of queues into the Map/Reduce
> framework. This JIRA is for defining changes to the configuration related to
> queues.
> The current format for defining a queue and its properties is as follows:
> mapred.queue.<queue-name>.<property-name>. For e.g.
> mapred.queue.<queue-name>.acl-submit-job. The reason for using this verbose
> format was to be able to reuse the Configuration parser in Hadoop. However,
> administrators currently using the queue configuration have already indicated
> a very strong desire for a more manageable format. Since, this becomes more
> unwieldy with hierarchical queues, the time may be good to introduce a new
> format for representing queue configuration.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
