[ https://issues.apache.org/jira/browse/MAPREDUCE-4076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj K updated MAPREDUCE-4076: --------------------------------- Status: Patch Available (was: Open) When we use 'yarn jar' command, RunJar.java tries to create temp directory if doesn't exist using configuration property "hadoop.tmp.dir". When it gets from the conf object, it will get the value as ${hadoop.home.dir}/hadoop-${user.name}. Here these vars are not replaced with system properties because of unavailability of 'hadoop.home.dir' system property. It will create the temp dir with the same name(i.e ${hadoop.home.dir}/hadoop-${user.name}) in the current dir. StreamJob unjars and keeps classes in the directory current-dir/${hadoop.home.dir}/hadoop-${user.name}, and then it tries to find "org/apache/hadoop/streaming/StreamJob.class" in the classpath and it gets the path as curent-dir/$%7Bhadoop.home.dir%7D/hadoop-$%7Buser.name%7D/hadoop-unjar8421477351848586067/ due to special chars in the directory name. And finally fails to merge from this path to the job jar file. If we do the same with 'hadoop jar', it will get the prop as $HADOOP_HOME/hadoop-username which is replaced with 'hadoop.home.dir' and 'user.name' properties , it will create the temp dir properly uses the same for other things to do and works fine. I have attached the patch to address the above problem by adding the hadoop.home.dir system property in yarn file. > Stream job fails with ZipException when use yarn jar command > ------------------------------------------------------------ > > Key: MAPREDUCE-4076 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4076 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 > Affects Versions: 0.23.1 > Reporter: Devaraj K > Assignee: Devaraj K > Priority: Critical > Attachments: MAPREDUCE-4076.patch > > > Stream job fails with ZipException when use yarn jar command and executes > successfully with hadoop jar command. > {code:xml} > linux-f330:/home/devaraj/hadoop/trunk/hadoop-0.24.0-SNAPSHOT/bin # ./yarn jar > ../share/hadoop/tools/lib/hadoop-streaming-0.24.0-SNAPSHOT.jar -input /hadoop > -output /test/output/1 -mapper cat -reducer wc > packageJobJar: [] > [/home/devaraj/hadoop/trunk/hadoop-0.24.0-SNAPSHOT/bin/$%7Bhadoop.home.dir%7D/hadoop-$%7Buser.name%7D/hadoop-unjar4241129353499211360/] > /tmp/streamjob7683981905208294893.jar tmpDir=null > Exception in thread "main" java.io.IOException: java.util.zip.ZipException: > ZIP file must have at least one entry > at org.apache.hadoop.streaming.JarBuilder.merge(JarBuilder.java:82) > at > org.apache.hadoop.streaming.StreamJob.packageJobJar(StreamJob.java:707) > at > org.apache.hadoop.streaming.StreamJob.setJobConf(StreamJob.java:948) > at org.apache.hadoop.streaming.StreamJob.run(StreamJob.java:127) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > at > org.apache.hadoop.streaming.HadoopStreaming.main(HadoopStreaming.java:50) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.util.RunJar.main(RunJar.java:200) > {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira