Of course, a 32 bits OS can handle only 4 GB of memory (2^32), for that reason, It's getting that error.
I personally recommend: - Decreasy the value of HADOOP_HEAPSIZE - Use a 64 bits Unix/Linux OS Regards 2011/11/28 Wayne Wan <bestirw...@gmail.com>: > oh! seems your setting is out of the size that OS(32-bit) can manage! > > 2011/11/13 Hoot Thompson <h...@ptpnow.com> > >> I can't seem to get past this heapsize error. Relevant settings are as >> follows: >> >> # The maximum amount of heap to use, in MB. Default is 1000. >> export HADOOP_HEAPSIZE=10000 >> >> <name>mapred.child.java.opts</name> >> <value>-Xmx1048576m</value> >> </property> >> >> >> >> 11/11/12 17:38:27 INFO hpc.Driver: Jar >> Name: /home/hadoop/querry/HadoopTest.jar >> 0 [main] INFO nccs.hpc.Driver - Jar >> Name: /home/hadoop/querry/HadoopTest.jar >> 0 [main] INFO nccs.hpc.Driver - Jar >> Name: /home/hadoop/querry/HadoopTest.jar >> 11/11/12 17:38:27 WARN mapred.JobClient: Use GenericOptionsParser for >> parsing the arguments. Applications should implement Tool for the same. >> 146 [main] WARN org.apache.hadoop.mapred.JobClient - Use >> GenericOptionsParser for parsing the arguments. Applications should >> implement Tool for the same. >> 146 [main] WARN org.apache.hadoop.mapred.JobClient - Use >> GenericOptionsParser for parsing the arguments. Applications should >> implement Tool for the same. >> 11/11/12 17:38:30 INFO input.FileInputFormat: Total input paths to >> process : 2329 >> 3330 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat >> - Total input paths to process : 2329 >> 3330 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat >> - Total input paths to process : 2329 >> 11/11/12 17:38:34 INFO mapred.JobClient: Running job: >> job_201111121210_0001 >> 7919 [main] INFO org.apache.hadoop.mapred.JobClient - Running job: >> job_201111121210_0001 >> 7919 [main] INFO org.apache.hadoop.mapred.JobClient - Running job: >> job_201111121210_0001 >> 11/11/12 17:38:35 INFO mapred.JobClient: map 0% reduce 0% >> 8922 [main] INFO org.apache.hadoop.mapred.JobClient - map 0% reduce 0% >> 8922 [main] INFO org.apache.hadoop.mapred.JobClient - map 0% reduce 0% >> 11/11/12 17:40:37 INFO mapred.JobClient: map 1% reduce 0% >> 130058 [main] INFO org.apache.hadoop.mapred.JobClient - map 1% reduce >> 0% >> 130058 [main] INFO org.apache.hadoop.mapred.JobClient - map 1% reduce >> 0% >> 11/11/12 17:42:17 INFO mapred.JobClient: map 2% reduce 0% >> 230162 [main] INFO org.apache.hadoop.mapred.JobClient - map 2% reduce >> 0% >> 230162 [main] INFO org.apache.hadoop.mapred.JobClient - map 2% reduce >> 0% >> 11/11/12 17:43:47 INFO mapred.JobClient: map 3% reduce 0% >> 320251 [main] INFO org.apache.hadoop.mapred.JobClient - map 3% reduce >> 0% >> 320251 [main] INFO org.apache.hadoop.mapred.JobClient - map 3% reduce >> 0% >> 11/11/12 17:45:26 INFO mapred.JobClient: map 4% reduce 0% >> 419346 [main] INFO org.apache.hadoop.mapred.JobClient - map 4% reduce >> 0% >> 419346 [main] INFO org.apache.hadoop.mapred.JobClient - map 4% reduce >> 0% >> 11/11/12 17:46:59 INFO mapred.JobClient: map 5% reduce 0% >> 512435 [main] INFO org.apache.hadoop.mapred.JobClient - map 5% reduce >> 0% >> 512435 [main] INFO org.apache.hadoop.mapred.JobClient - map 5% reduce >> 0% >> 11/11/12 17:48:26 INFO mapred.JobClient: Task Id : >> attempt_201111121210_0001_r_000000_0, Status : FAILED >> 599520 [main] INFO org.apache.hadoop.mapred.JobClient - Task Id : >> attempt_201111121210_0001_r_000000_0, Status : FAILED >> 599520 [main] INFO org.apache.hadoop.mapred.JobClient - Task Id : >> attempt_201111121210_0001_r_000000_0, Status : FAILED >> Error: java.lang.OutOfMemoryError: Java heap space >> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier >> $MapOutputCopier.shuffleInMemory(ReduceTask.java:1685) >> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier >> $MapOutputCopier.getMapOutput(ReduceTask.java:1545) >> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier >> $MapOutputCopier.copyOutput(ReduceTask.java:1394) >> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier >> $MapOutputCopier.run(ReduceTask.java:1326) >> >> >> > -- Marcos Luis Ortíz Valmaseda Linux Infrastructure Engineer Linux User # 418229 http://marcosluis2186.posterous.com http://www.linkedin.com/in/marcosluis2186 Twitter: @marcosluis2186
