Hello all,
I'm attempting to index a large internal website with 6.7 m urls and I'm
running into a map failure after fetching (for 5+ days):
2009-07-20 07:09:23,316 INFO fetcher.Fetcher - -activeThreads=0
2009-07-20 07:09:23,806 WARN mapred.LocalJobRunner - job_local_0005
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
any valid local directory for
taskTracker/jobcache/job_local_0005/attempt_local_0005_m_000000_0/output/file.out
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:335)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
at
org.apache.hadoop.mapred.MapOutputFile.getOutputFileForWrite(MapOutputFile.java:61)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1209)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:867)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:138)
hadoop-site.xml:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<!--
We need LOTS of memory... And we need to disable the gc overhead limit,
per this page:
http://java.sun.com/javase/technologies/hotspot/gc/gc_tuning_6.html#par_gc.oom
-->
<property>
<name>mapred.child.java.opts</name>
<value>-Xmx4096m -XX:-UseGCOverheadLimit</value>
</property>
</configuration>
nutch-site.xml (excluding http.agent directives for brevity):
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<!-- http.agent properties excluded -->
<property>
<name>http.timeout</name>
<value>20000</value>
<description>The default network timeout, in milliseconds.</description>
</property>
<property>
<name>fetcher.threads.fetch</name>
<value>20</value>
<description>The number of FetcherThreads the fetcher should use.
This is also determines the maximum number of requests that are
made at once (each FetcherThread handles one connection).</description>
</property>
<property>
<name>fetcher.threads.per.host</name>
<value>20</value>
<description>This number is the maximum number of threads that
should be allowed to access a host at one time.</description>
</property>
<property>
<name>fetcher.server.delay</name>
<value>0.1</value>
<description>The number of seconds the fetcher will delay between
successive requests to the same server.</description>
</property>
</configuration>
Relevant environment variables:
NUTCH_JAVA_HOME=/usr/lib/jvm/jre-1.7.0-icedtea.x86_64
NUTCH_HEAPSIZE=3072
JAVA_HOME=/usr/lib/jvm/jre-1.7.0-icedtea.x86_64
I ran nutch with the following command/cwd:
[/home/fred/nutch-1.0]$ bin/nutch crawl urls_wiki_mirror -dir
crawl_wiki_mirror -threads 3 -depth 1
The seed file in urls_wiki_mirror contains 6739469 urls... Those are the
only urls I wish to crawl -- hence depth 1. The configuration I have set
up allows me to crawl this local server with 3 fetchers at the same time
at a rate that it doesn't overwhelm the server.
I'm using defaults for temp directories. Thus, /tmp/hadoop-fred/ is the
temp file location. The error message notes the following partial path:
taskTracker/jobcache/job_local_0005/attempt_local_0005_m_000000_0/output/file.out
I figure that equates to this full path:
/tmp/hadoop-fred/mapred/local/taskTracker/jobcache/job_local_0005/attempt_local_0005_m_000000_0/output/
The contents of this directory is spill[0-906].out... Nothing else. No
file.out. There is 68G of data in this folder (ie. it looks to have
downloaded everything i need)... There is 9+ GB of free space on the
filesystem -- is it possible this is insufficient?
So, what happened? Is there a way I can recover without re-crawling?
I am running on a Fedora Core 8 virtual machine with two cores, 4 GB memory.
Let me know if any more information is needed...
Thanks,
/FjK
