[
https://issues.apache.org/jira/browse/OOZIE-3007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Sasvari updated OOZIE-3007:
----------------------------------
Description:
According to https://oozie.apache.org/docs/4.2.0/AG_ActionConfiguration.html ,
it is possible to specify for default configuration parameters for Oozie
actions by the means of action configuration files.
However, this mechanism does not work for Map Reduce actions.
For example, assume we created {{/var/lib/oozie/action-conf/map-reduce.xml}}
(readable by Oozie) with the following content:
{code}
cat map-reduce.xml
<configuration>
<property>
<name>oozie.launcher.mapred.fairscheduler.pool</name>
<value>root.support.coe</value>
</property>
<property>
<name>oozie.launcher.mapred.job.queue.name</name>
<value>oozie_queue</value>
</property>
<property>
<name>mapreduce.job.queuename</name>
<value>oozie_queue</value>
</property>
<property>
<name>mapred.queue.name</name>
<value>oozie_queue</value>
</property>
</configuration>
{code}
and added the following to {{oozie-site.xml}}
{code}
<property>
<name>oozie.service.HadoopAccessorService.action.configurations</name>
<value>*=/var/lib/oozie/action-conf</value>
</property>
{code}
the mapper task started by Oozie will be put into the {{default}} queue instead
of {{oozie-launcher}}.
h5. Possible reason:
HadoopAccessorService's {{loadHadoopConfigs()}} loads default hadoop
configuration settings from hadoop config files.
{{JavaActionExecutor}} loads default action configuration from given files with
[createActionDefaultConf|https://github.com/apache/oozie/blob/84eb493642d35b8bb1e7527ef51b6ecfa40dd592/core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java#L458]
via HadoopAccessorService in
[setupActionConf|https://github.com/apache/oozie/blob/84eb493642d35b8bb1e7527ef51b6ecfa40dd592/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L389].
{code}
XConfiguration actionDefaults =
has.createActionDefaultConf(conf.get(HADOOP_YARN_RM), getType());
XConfiguration.injectDefaults(actionDefaults, actionConf);
{code}
However, {{actionConf}} already contains default value for
"{{mapreduce.job.queuename}}" so
[injectDefaults|https://github.com/apache/oozie/blob/84eb493642d35b8bb1e7527ef51b6ecfa40dd592/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L389]
wont add user specified "{{mapreduce.job.queuename}}" default queue (i.e.
{{oozie_queue}}).
I tested this behaviour with a *4.2.0* based Oozie, but the existing code did
not change here by OYA.
was:
According to https://oozie.apache.org/docs/4.2.0/AG_ActionConfiguration.html ,
it is possible to specify for default configuration parameters for Oozie
actions by the means of action configuration files.
However, this mechanism does not work for Map Reduce actions.
For example, assume we created {{/var/lib/oozie/action-conf/map-reduce.xml}}
(readable by Oozie) with the following content:
{code}
cat map-reduce.xml
<configuration>
<property>
<name>oozie.launcher.mapred.fairscheduler.pool</name>
<value>root.support.coe</value>
</property>
<property>
<name>oozie.launcher.mapred.job.queue.name</name>
<value>oozie_queue</value>
</property>
<property>
<name>mapreduce.job.queuename</name>
<value>oozie_queue</value>
</property>
<property>
<name>mapred.queue.name</name>
<value>oozie_queue</value>
</property>
</configuration>
{code}
and added the following to {{oozie-site.xml}}
{code}
<property>
<name>oozie.service.HadoopAccessorService.action.configurations</name>
<value>*=/var/lib/oozie/action-conf</value>
</property>
{code}
the mapper task started by Oozie will be put into the {{default}} queue instead
of {{oozie-launcher}}.
h5. Possible reason:
HadoopAccessorService's {{loadHadoopConfigs()}} loads default hadoop
configuration settings from hadoop config files.
{{JavaActionExecutor}} loads default action configuration from given files with
[createActionDefaultConf|https://github.com/apache/oozie/blob/84eb493642d35b8bb1e7527ef51b6ecfa40dd592/core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java#L458]
via HadoopAccessorService in
[setupActionConf|https://github.com/apache/oozie/blob/84eb493642d35b8bb1e7527ef51b6ecfa40dd592/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L389].
{code}
XConfiguration actionDefaults =
has.createActionDefaultConf(conf.get(HADOOP_YARN_RM), getType());
XConfiguration.injectDefaults(actionDefaults, actionConf);
{code}
However, {{actionConf}} already contains default value for
"{{mapreduce.job.queuename}}" so
[injectDefaults|https://github.com/apache/oozie/blob/84eb493642d35b8bb1e7527ef51b6ecfa40dd592/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L389]
wont add user specified "{{mapreduce.job.queuename}}" default queue (i.e.
{{oozie_queue}}).
I tested this behaviour with a 4.2.0 based Oozie, but the existing code did not
change here by OYA.
> Ability to override default Map Reduce properties via default action config
> file
> --------------------------------------------------------------------------------
>
> Key: OOZIE-3007
> URL: https://issues.apache.org/jira/browse/OOZIE-3007
> Project: Oozie
> Issue Type: Bug
> Reporter: Attila Sasvari
>
> According to https://oozie.apache.org/docs/4.2.0/AG_ActionConfiguration.html
> , it is possible to specify for default configuration parameters for Oozie
> actions by the means of action configuration files.
> However, this mechanism does not work for Map Reduce actions.
> For example, assume we created {{/var/lib/oozie/action-conf/map-reduce.xml}}
> (readable by Oozie) with the following content:
> {code}
> cat map-reduce.xml
> <configuration>
> <property>
> <name>oozie.launcher.mapred.fairscheduler.pool</name>
> <value>root.support.coe</value>
> </property>
> <property>
> <name>oozie.launcher.mapred.job.queue.name</name>
> <value>oozie_queue</value>
> </property>
> <property>
> <name>mapreduce.job.queuename</name>
> <value>oozie_queue</value>
> </property>
> <property>
> <name>mapred.queue.name</name>
> <value>oozie_queue</value>
> </property>
> </configuration>
> {code}
> and added the following to {{oozie-site.xml}}
> {code}
> <property>
> <name>oozie.service.HadoopAccessorService.action.configurations</name>
> <value>*=/var/lib/oozie/action-conf</value>
> </property>
> {code}
> the mapper task started by Oozie will be put into the {{default}} queue
> instead of {{oozie-launcher}}.
> h5. Possible reason:
> HadoopAccessorService's {{loadHadoopConfigs()}} loads default hadoop
> configuration settings from hadoop config files.
> {{JavaActionExecutor}} loads default action configuration from given files
> with
> [createActionDefaultConf|https://github.com/apache/oozie/blob/84eb493642d35b8bb1e7527ef51b6ecfa40dd592/core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java#L458]
> via HadoopAccessorService in
> [setupActionConf|https://github.com/apache/oozie/blob/84eb493642d35b8bb1e7527ef51b6ecfa40dd592/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L389].
>
> {code}
> XConfiguration actionDefaults =
> has.createActionDefaultConf(conf.get(HADOOP_YARN_RM), getType());
> XConfiguration.injectDefaults(actionDefaults, actionConf);
> {code}
> However, {{actionConf}} already contains default value for
> "{{mapreduce.job.queuename}}" so
> [injectDefaults|https://github.com/apache/oozie/blob/84eb493642d35b8bb1e7527ef51b6ecfa40dd592/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L389]
> wont add user specified "{{mapreduce.job.queuename}}" default queue (i.e.
> {{oozie_queue}}).
> I tested this behaviour with a *4.2.0* based Oozie, but the existing code did
> not change here by OYA.
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
