[
https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16179698#comment-16179698
]
Sidharta Seethana commented on YARN-7226:
-----------------------------------------
{quote}
But how does this work today for Docker containers? The launch script is using
the var:-value syntax, which means if the image specified the value then it
will not take the value the user desires. In other words, it looks like for
Docker containers the semantics of the whitelist is the list of variables that
cannot be overridden by the container.
I hope we can all agree that if the container explicitly sets an environment
variable then that variable should be set to the value the user specified. I
think the only issue then is what to do about variables that are getting
implicitly set via the NM whitelist that end up overriding the Docker image
variables unintentionally.
{quote}
[~jlowe], you are right on both counts - short of moving more of the common
functionality into the executors/container runtimes, there wasn't a better way
of handling this at the time. The goal was to ensure that non-docker
containers' behavior is not affected (however, based on this JIRA it looks like
there was a scenario that wasn't considered).
The approach of moving the implementation into the executors/runtimes makes
sense to me as well - though keeping the semantics in sync (to the extent
possible) is something we'll need to be careful about in the future. Maybe, at
some point in the future, the docker implementation could use a different
approach altogether (e.g specifying these env vars as a part of the docker run
command).
> Whitelisted variables do not support delayed variable expansion
> ---------------------------------------------------------------
>
> Key: YARN-7226
> URL: https://issues.apache.org/jira/browse/YARN-7226
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 2.9.0, 2.8.1, 3.0.0-alpha4
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Attachments: YARN-7226.001.patch, YARN-7226.002.patch
>
>
> The nodemanager supports a configurable list of environment variables, via
> yarn.nodemanager.env-whitelist, that will be propagated to the container's
> environment unless those variables were specified in the container launch
> context. Unfortunately the handling of these whitelisted variables prevents
> using delayed variable expansion. For example, if a user shipped their own
> version of hadoop with their job via the distributed cache and specified:
> {noformat}
> HADOOP_COMMON_HOME={{PWD}}/my-private-hadoop/
> {noformat}
> as part of their job, the variable will be set as the *literal* string:
> {noformat}
> $PWD/my-private-hadoop/
> {noformat}
> rather than having $PWD expand to the container's current directory as it
> does for any other, non-whitelisted variable being set to the same value.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
