Hi All
I am facing a hard problem. I am running a map reduce job using streaming
but it fails and it gives the following error.
Caught: java.lang.OutOfMemoryError: Java heap space
at Nodemapper5.parseXML(Nodemapper5.groovy:25)
java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess
failed with code 1
at
org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:362)
at
org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:572)
at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:136)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
I have increased the heap size in hadoop-env.sh and make it 2000M. Also I
tell the job manually by following line.
-D mapred.child.java.opts=-Xmx2000M \
but it still gives the error. The same job runs fine if i run on shell using
1024M heap size like
cat file.xml | /root/Nodemapper5.groovy
Any clue?????????
Thanks in advance.
--
Regards
Shuja-ur-Rehman Baig
_________________________________
MS CS - School of Science and Engineering
Lahore University of Management Sciences (LUMS)
Sector U, DHA, Lahore, 54792, Pakistan
Cell: +92 3214207445
