[ https://issues.apache.org/jira/browse/YARN-10607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17279223#comment-17279223 ]
Eric Badger commented on YARN-10607: ------------------------------------ Thanks for the review, [~Jim_Brennan]. I've attached a new patch that adds the {{!Shell.WINDOWS}} check > User environment is unable to prepend PATH when mapreduce.admin.user.env also > sets PATH > --------------------------------------------------------------------------------------- > > Key: YARN-10607 > URL: https://issues.apache.org/jira/browse/YARN-10607 > Project: Hadoop YARN > Issue Type: Bug > Reporter: Eric Badger > Assignee: Eric Badger > Priority: Major > Attachments: YARN-10607.001.patch, YARN-10607.002.patch, > YARN-10607.003.patch, YARN-10607.004.patch, YARN-10607.004.patch > > > When using the tarball approach to ship relevant Hadoop jars to containers, > it is helpful to set {{mapreduce.admin.user.env}} to something like > {{PATH=./hadoop-tarball:\{\{PATH\}\}}} to make sure that all of the Hadoop > binaries are on the PATH. This way you can call {{hadoop}} instead of > {{./hadoop-tarball/hadoop}}. The intention here is to force prepend > {{./hadoop-tarball}} and then append the set {{PATH}} afterwards. But if a > user would like to override the appended portion of {{PATH}} in their > environment, they are unable to do so. This is because {{PATH}} ends up > getting parsed twice. Initially it is set via {{mapreduce.admin.user.env}} to > {{PATH=./hadoop-tarball:$SYS_PATH}}}. In this case {{SYS_PATH}} is what I'll > refer to as the normal system path. E.g. {{/usr/local/bin:/usr/bin}}, etc. > After this, the user env parsing happens. For example, let's say the user > sets their {{PATH}} to {{PATH=.:$PATH}}. We have already parsed {{PATH}} from > the admin.user.env. Then we go to parse the user environment and find the > user also specified {{PATH}}. So {{$PATH}} ends up getting getting expanded > to {{./hadoop-tarball:$SYS_PATH}}, which leads to the user's {{PATH}} being > {{PATH=.:./hadoop-tarball:$SYS_PATH}}. We then append this to {{PATH}}, which > has already been set in the environment map via the admin.user.env. So we > finally end up with > {{PATH=./hadoop-tarball:$SYS_PATH:.:./hadoop-tarball:$SYS_PATH}}. > This normally isn't a huge deal, but if you want to ship a version of > python/perl/etc. that clashes with the one that is already there in > {{SYS_PATH}}, you will need to refer to it by its full path. Since in the > above example, {{.}} doesn't appear until after {{$SYS_PATH}}. This is a pain > and it should be possible to prepend its {{PATH}} to override the > system/container {{SYS_PATH}}, even when also forcefully prepending to > {{PATH}} with you hadoop tarball. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org