[
https://issues.apache.org/jira/browse/MAPREDUCE-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401800#comment-13401800
]
John Gordon commented on MAPREDUCE-4368:
----------------------------------------
for 1) We scoped it to Windows because in Windows " is not legal in a file name
at all, but it is legal in filenames in Posix. While "file name" is similarly
used in linux and windows, \"file\"\ name is also a legal construct in linux
for a file that actually contains quotes in the path. In that case, stripping
internal quotes would potentially break execution and the shell would have
handled everything just fine. I did initially experiment with doing this for
all arguments, in hopes of improving upstream quality for things like
pig.additional.jars where we may find quotes in the path specified there.
However, we kept it scoped down to java.library.path here because it is the
only one explicitly interpreted as a path here. One can imagine a scenario
where an upstream component has an argument like pig.query=^"a=load ...^" --
which may need embedded quotes and may also be escaped properly. In such a
case, blanket removal of embedded quotes without handling escaping would be
potentially dangerous.
3) is particularly interesting and maybe needs examples to explain.
Here is a javaopts string that the space split makes sense for:
-Dpig.additional.jars=/tmp/foo.jar
-Djava.library.path=/usr/lib/java/;/usr/lib64/java -esa
This parses into this javaoptssplit:
-Dpig.additional.jars=/tmp/foo.jar
-Djava.library.path=/usr/lib/java;/usr/lib64/java
-esa
Which goes to the command line as:
java "-Dpig.additional.jars=/tmp/foo.jar"
"-Djava.library.path=/usr/lib/java;/usr/lib64/java" "-esa" job.jar
Now, here is an example of what this patch fixes and why the combination of
space splitting and quote escaping is broken:
-Dpig.additional.jars=/tmp/foo.jar
-Djava.library.path=/usr/lib/java;/usr/lib64/java;"/opt/hadoop/somewhat
memorable/path";/lib/java -esa
This parses into this javaoptssplit:
-Dpig.additional.jars=/tmp/foo.jar
-Djava.library.path=/usr/lib/java;/usr/lib64/java;"/opt/hadoop/somewhat
memorable/path";/lib/java
-esa
Which goes to the command line as:
java "-Dpig.additional.jars=/tmp/foo.jar"
"-Djava.library.path=/usr/lib/java;/usr/lib64/java;"/opt/hadoop/somewhat"
"memorable/path";/lib/java" "-esa" job.jar
This is bad in a number of ways, but here are the two most obvious to me:
1.) The semicolon is no longer quoted, so the shell read end of line after
<path"> and will try to execute /lib/java as a separate shell command after
java completes.
2.) java's command-line parsing rules will cause it to interpret
"memorable/path" as the name of the jar it should execute and will yield (most
likely) a ClassNoDefFoundException for "memorable/path" or "memorable.path".
Hope that explains what the comment was trying to read.
Thanks, Bikas, for the feedback -- really appreciate it. Great catch on the
break!
> TaskRunner fails to start jars when the java.library.path contains a quoted
> path with embedded spaces
> -----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4368
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4368
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 1-win
> Environment: on Windows:
> set PATH=%PATH%;"C:\this memorable place".
> Reporter: John Gordon
> Labels: newbie, patch
> Attachments: TaskRunner.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> TaskRunner splits arguments by space before it adds them back to the vargs
> list, so it loses all context of quote escaped strings with embedded spaces.
> This gets fixed up later by wrapping all arguments with " -- so you get
> something like java "-D<opt>=<value>". This is problematic for paths with
> embedded spaces, where we end up creating "-D<opt>=<first part" "last part>".
> To java, the jar being run is last part. So with the environment above, you
> will see "ClassNoDefFoundError: memorable" and the jar will fail to start.
> In this particular case, we know that java.libarary.path contains paths and
> the tests often use %PATH% to seed this, so the fix is to remove embedded
> quotes in listed path elements because we know the aggregate will be quoted
> when the JVM is started.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
