[
https://issues.apache.org/jira/browse/YARN-7677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16377034#comment-16377034
]
Jim Brennan commented on YARN-7677:
-----------------------------------
Now that YARN-5714 has been resolved, we have two options for resolving this
Jira. Environment variables now go through a dependency sort before being
written to the launch_container.sh script. So we need to decide whether we want
to retain the ordering of the 3 categories of environment variables from the
last patch for this Jira, or just do the minimal changes to address the
original issue- allowing (docker) container images to override the whitelisted
variables.
So the two options are:
Minimal changes:
* Remove the explicit setting of HADOOP_CONF_DIR, and treat it like other
whitelisted variables.
* Whitelisted variables not overridden by the user are written first, in the
order they are listed in the NM_ENV_WHITELIST, using the {{var:-default}}
variable expansion syntax
* All other variables are then written in dependency sorted order.
Category sorted order:
* Remove the explicit setting of HADOOP_CONF_DIR, and treat it like other
whitelisted variables.
* Whitelisted variables not overridden by the user are written first, in the
order they are listed in the NM_ENV_WHITELIST, using the {{var:-default}}
variable expansion syntax.
* Then write variables set explicitly by the NM, in the order they are written
in code.
* Finally, write user-defined variables in dependency sorted order.
The main difference is that the second approach ensures all of the explicitly
set NM variables are always written before all user variables, in a consistent
order. I do like seeing that consistency when looking at the scripts, but I'm
not sure it's required, just nice.
[~jlowe], [~ebadger], [~shaneku...@gmail.com], [~billie.rinaldi], please let me
know if you have a preference.
> Docker image cannot set HADOOP_CONF_DIR
> ---------------------------------------
>
> Key: YARN-7677
> URL: https://issues.apache.org/jira/browse/YARN-7677
> Project: Hadoop YARN
> Issue Type: Bug
> Affects Versions: 3.0.0
> Reporter: Eric Badger
> Assignee: Jim Brennan
> Priority: Major
> Attachments: YARN-7677.001.patch, YARN-7677.002.patch,
> YARN-7677.003.patch, YARN-7677.004.patch, YARN-7677.005.patch
>
>
> Currently, {{HADOOP_CONF_DIR}} is being put into the task environment whether
> it's set by the user or not. It completely bypasses the whitelist and so
> there is no way for a task to not have {{HADOOP_CONF_DIR}} set. This causes
> problems in the Docker use case where Docker containers will set up their own
> environment and have their own {{HADOOP_CONF_DIR}} preset in the image
> itself.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
