[
https://issues.apache.org/jira/browse/MAPREDUCE-861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744452#action_12744452
]
rahul k singh commented on MAPREDUCE-861:
-----------------------------------------
As mentioned above , we had an internal agreement that we would be going ahead
with
xml based configuration for hierarchial queues.
In terms of how configuration would be structured for hierarchial queues , we
had 2 options in mind.
Option 1:
----------
mapred-queues.xml would consist of the hierarchial queue hierarchy.
Typical hierarchial queue configuration would look like:
{code:xml}
<queue>
<name>q1</name>
<queue>
<name>q1q1</name>
<administrators>
u1,u2,u3
</administrators>
<submitters>
u1,u2
</submitters>
<state>stop/running</state>
<schedulingContext>
<capacity></capacity>
<maxCapacity></maxCapacity>
</schedulingContext>
</queue>
</queue>
{code}
The configuration above defines a queue "q1" and a single child "q1q1"
<schedulingContext> tag would act as an black box kind of section for the
mapred based parsers.
The xsd definition of <schedulingContext> would be
{code:xml}
<xs:element name="schedulingContext" minOccurs="0" maxOccurs="unbounded">
<xs:complexType>
<xs:any/>
</xs:complexType>
</xs:element>
{code}
By defining <schedulingContext> as <xs:any> we can extend this section of
configuration to add any kind of
tags to the <schedulingContext>.
Advantage:
1.This approach allows to have a single configuration file
2. It is generic enough as in it allows users to declare scheduler properties
the way they want.
Disadvantage:
1. This would result in having the parsing logic at different places , for
framework level changes in framework and scheduler
specific parsing would be done in scheduler.
2. More cumbersome to implement .
Option 2:
---------
Same as option 1 except the definition of <schedulingContext> would change .
It would have
child tags <key> and <value> which would define the key value mappings of the
various properties required
by schedulers.
For example:
{code:xml}
<queue>
<name>q1</name>
<queue>
<name>q1q1</name>
<administrators>
u1,u2,u3
</administrators>
<submitters>
u1,u2
</submitters>
<state>stop/running</state>
<schedulingContext>
<key>capacity</key>
<value></value>
<key>maxCapacity</key>
<value></value>
</schedulingContext>
</queue>
</queue>
{code}
the new xsd for <schedulingContext> would look like
{code:xml}
<xs:element name="schedulingContext" minOccurs="0" maxOccurs="unbounded">
<xs:complexType>
<xs:sequence>
<xs:element name="key" minOccurs="0" maxOccurs="unbounded">
</xs:element>
</xs:element>
<xs:element name="value" minOccurs="0" maxOccurs="unbounded">
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
{code}
Advantage:
1. Allows to have a single configuration file.
2. Provides constant way to specify scheduling properties.
3. Easier to implement and parsing logic now resides at one common place.
Disadvantage:
1. Doesn't allows the nested settings for scheduler properties.
2. Assumes that scheduler properties would always be in key value format.
> Modify queue configuration format and parsing to support a hierarchy of
> queues.
> -------------------------------------------------------------------------------
>
> Key: MAPREDUCE-861
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-861
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Reporter: Hemanth Yamijala
> Assignee: rahul k singh
>
> MAPREDUCE-853 proposes to introduce a hierarchy of queues into the Map/Reduce
> framework. This JIRA is for defining changes to the configuration related to
> queues.
> The current format for defining a queue and its properties is as follows:
> mapred.queue.<queue-name>.<property-name>. For e.g.
> mapred.queue.<queue-name>.acl-submit-job. The reason for using this verbose
> format was to be able to reuse the Configuration parser in Hadoop. However,
> administrators currently using the queue configuration have already indicated
> a very strong desire for a more manageable format. Since, this becomes more
> unwieldy with hierarchical queues, the time may be good to introduce a new
> format for representing queue configuration.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
