[
https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16179115#comment-16179115
]
Jason Lowe commented on YARN-7226:
----------------------------------
bq. If the user doesn't override the env vars, it is expected that what is
specified in the image takes precedence.
But how does this work today for Docker containers? The launch script is using
the {{var:-value}} syntax, which means if the image specified the value then it
will _not_ take the value the user desires. In other words, it looks like for
Docker containers the semantics of the whitelist is the list of variables that
_cannot_ be overridden by the container.
I hope we can all agree that if the container explicitly sets an environment
variable then that variable should be set to the value the user specified. I
think the only issue then is what to do about variables that are getting
implicitly set via the NM whitelist that end up overriding the Docker image
variables unintentionally. Here's my proposal to fix this:
The whitelist processing moves to the container runtime.
* For non-Docker containers the behavior works semantically as in my patch (and
before YARN-3853) where NM environment variables in the NM whitelist are added
to the container's environment if and only if the user did not specify them.
* For Docker containers the NM whitelist is *ignored*. Either the user set a
value directly or it will implicitly pick it up from the Docker image. We
won't try to port environment variables from the NM's env to the container env
because we expect those variables to be specifically setup by the Docker image.
Will that be acceptable? It will solve the case where JAVA_HOME is implicitly
getting picked up from the NM whitelist and accidentally overriding the one in
the Docker image. However if the user _explicitly_ sets JAVA_HOME then that
user's value will be honored even if the Docker image specified it.
> Whitelisted variables do not support delayed variable expansion
> ---------------------------------------------------------------
>
> Key: YARN-7226
> URL: https://issues.apache.org/jira/browse/YARN-7226
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 2.9.0, 2.8.1, 3.0.0-alpha4
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Attachments: YARN-7226.001.patch, YARN-7226.002.patch
>
>
> The nodemanager supports a configurable list of environment variables, via
> yarn.nodemanager.env-whitelist, that will be propagated to the container's
> environment unless those variables were specified in the container launch
> context. Unfortunately the handling of these whitelisted variables prevents
> using delayed variable expansion. For example, if a user shipped their own
> version of hadoop with their job via the distributed cache and specified:
> {noformat}
> HADOOP_COMMON_HOME={{PWD}}/my-private-hadoop/
> {noformat}
> as part of their job, the variable will be set as the *literal* string:
> {noformat}
> $PWD/my-private-hadoop/
> {noformat}
> rather than having $PWD expand to the container's current directory as it
> does for any other, non-whitelisted variable being set to the same value.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
