[ https://issues.apache.org/jira/browse/YARN-5877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15727893#comment-15727893 ]
Sunil G commented on YARN-5877: ------------------------------- We have checked various cases for this issue in detail and had some offline discussions with [~vvasudev] and [~bibinchundatt] also. Now coming to the issue mentioned. MAPREDUCE-6704 has come in to a common consensus that HADOOP_MAPRED_HOME could be published with nm-white-list so that even if this variable is not present in env, we can take the configured value. This will solve the container launch problem mentioned there. As of today, if we publish HADOOP_MAPRED_HOME through nm-whitelist-env, it will look like below in launch_container.sh. {{export HADOOP_MAPRED_HOME="/home/hadoopbuild"}} As mentioned above, this will be fine in case of normal container launch. But in docker scenario, the path value given may not be valid. In such case, we will see the failure as mentioned in this jira. Ideally if this environment variable were published as whitelist env, then it could try to take from env itself first. {code} export HADOOP_MAPRED_HOME=${HADOOP_MAPRED_HOME:-"/home/hadoopbuild"} {code} Here container will try to get the value from *HADOOP_MAPRED_HOME* if its available in env first. If not, it will take value which is given as "/home/hadoopbuild". In case of docker, this value was set in its system env itself (i think Bibin used sequence iq docker image here which had this value). So in conclusion, all variables mentioned in {{yarn.nodemanager.env-whitelist}} could be whitelisted. This can solve the problem. If some docker images doesn't have this env variable set in its env, and if the default path to HADOOP_MAPRED_HOME is also invalid, then user has to set correct mapred home path in {{yarn.app.mapreduce.am.env}}. I think YARN-5877.0002.patch is good one to go. [~bibinchundatt] could you rebase the patch and give a latest version number to compile against trunk. If possible, could you please share test results with this latest patch too in docker. > Allow all nm-whitelist-env to get overridden during launch > ---------------------------------------------------------- > > Key: YARN-5877 > URL: https://issues.apache.org/jira/browse/YARN-5877 > Project: Hadoop YARN > Issue Type: Bug > Reporter: Bibin A Chundatt > Assignee: Bibin A Chundatt > Attachments: Dockerfile, YARN-5877.0001.patch, YARN-5877.0002.patch, > YARN-5877.0003.patch, bootstrap.sh, yarn-site.xml > > > As per the {{yarn.nodemanager.env-whitelist}} for the configured values > should containers may override rather than use NodeManager's default. > {code} > <property> > <description>Environment variables that containers may override rather > than use NodeManager's default.</description> > <name>yarn.nodemanager.env-whitelist</name> > > <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME</value> > </property> > {code} > But only the following containers can override > {code} > whitelist.add(ApplicationConstants.Environment.HADOOP_YARN_HOME.name()); > whitelist.add(ApplicationConstants.Environment.HADOOP_COMMON_HOME.name()); > whitelist.add(ApplicationConstants.Environment.HADOOP_HDFS_HOME.name()); > whitelist.add(ApplicationConstants.Environment.HADOOP_CONF_DIR.name()); > whitelist.add(ApplicationConstants.Environment.JAVA_HOME.name()); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org