Hi All,
When I ran crawl script [1] (not nutch's crawl), I got hava OOM heap space:
2013-10-29 12:56:25,407 WARN mapred.LocalJobRunner -
job_local1484958909_0001
java.lang.Exception: java.lang.OutOfMemoryError: Java heap space
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:354)
Caused by: java.lang.OutOfMemoryError: Java heap space
at
org.apache.hadoop.mapred.IFile$Reader.readNextBlock(IFile.java:344)
at org.apache.hadoop.mapred.IFile$Reader.next(IFile.java:406)
at org.apache.hadoop.mapred.Merger$Segment.next(Merger.java:238)
at
org.apache.hadoop.mapred.Merger$MergeQueue.adjustPriorityQueue(Merger.java:348)
at org.apache.hadoop.mapred.Merger$MergeQueue.next(Merger.java:368)
at org.apache.hadoop.mapred.Merger.writeFile(Merger.java:156)
at org.apache.hadoop.mapred.Merger$MergeQueue.merge(Merger.java:517)
at org.apache.hadoop.mapred.Merger$MergeQueue.merge(Merger.java:399)
at org.apache.hadoop.mapred.Merger.merge(Merger.java:77)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1698)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1328)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:431)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366)
at
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:223)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2013-10-29 12:56:25,787 ERROR fetcher.Fetcher - Fetcher:
java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1357)
at org.apache.nutch.fetcher.Fetcher.fetch(Fetcher.java:1340)
at org.apache.nutch.fetcher.Fetcher.run(Fetcher.java:1376)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.nutch.fetcher.Fetcher.main(Fetcher.java:1349)
I use nutch 1.7 and JDK 1.7.0.45.
How to put java max heap size on crawl script? (-Xmx option)?
Thanks in advance.-
[1] http://wiki.apache.org/nutch/NutchTutorial#A3.3._Using_the_crawl_script
--
wassalam,
[bayu]
