[
https://issues.apache.org/jira/browse/MAPREDUCE-6565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15668458#comment-15668458
]
Jason Lowe commented on MAPREDUCE-6565:
---------------------------------------
bq. I think MR over distributed cache shouldn't load configurations that
included in MR tar ball.
We explicitly have MapReduce load configurations that are in the tarball. We
used to run without that and had the jobs pick up the configurations from the
node, but we often ran into cases where rolling upgrades broke jobs on the
cluster (e.g.: config changes to reference a plugin class that only exists in
the new release, or old config setting breaks job running new release, etc.).
Things got a lot simpler for us when we put the configs in the MR tarball and
made sure those configs reflect the entire cluster. Added bonus: we don't need
to push new confs to every node when the conf changes are just client-side
(i.e.: not for the datanode/nodemanager). We just update our tarball in HDFS
and every new job picks up the change.
Back to the original reported issue: one reason not all security settings in
job.xml aren't honored is because, unlike the *-site.xml files, job.xml is not
added as a default Configuration resource. That means anything that simply
instantiates a Configuration instance (e.g.: SecurityUtil, etc.) will not "see"
job.xml settings. I'm not sure why job.xml wasn't added as a default resource.
That may have been an intentional omission to avoid some problem, but part of
me wonders if it should have been setup so that all Configuration objects start
using job.xml settings if job.xml applies (i.e: we're using JobConf).
> Configuration to use host name in delegation token service is not read from
> job.xml during MapReduce job execution.
> -------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-6565
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6565
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Chris Nauroth
> Assignee: Li Lu
>
> By default, the service field of a delegation token is populated based on
> server IP address. Setting {{hadoop.security.token.service.use_ip}} to
> {{false}} changes this behavior to use host name instead of IP address.
> However, this configuration property is not read from job.xml. Instead, it's
> read from a separate {{Configuration}} instance created during static
> initialization of {{SecurityUtil}}. This does not work correctly with
> MapReduce jobs if the framework is distributed by setting
> {{mapreduce.application.framework.path}} and the
> {{mapreduce.application.classpath}} is isolated to avoid reading
> core-site.xml from the cluster nodes. MapReduce tasks will fail to
> authenticate to HDFS, because they'll try to find a delegation token based on
> the NameNode IP address, even though at job submission time the tokens were
> generated using the host name.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
