I can confirm this error as I just tried running the last revision of
Nutch, rev-620818 on Debian as well as Cygwin on Windows.
It works fine on Debian but fails on Cygwin with this error:-
2008-02-14 19:49:47,756 WARN regex.RegexURLNormalizer - can\'t find
rules for scope \'inject\', using default
2008-02-14 19:49:48,381 WARN mapred.LocalJobRunner - job_local_1
java.io.IOException: Target
file:/D:/tmp/hadoop-guest/mapred/temp/inject-temp-322737506/_reduce_bjm6rw/part-00000
already exists
at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:246)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:125)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:116)
at
org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:196)
at
org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:394)
at org.apache.hadoop.mapred.Task.moveTaskOutputs(Task.java:452)
at org.apache.hadoop.mapred.Task.moveTaskOutputs(Task.java:469)
at org.apache.hadoop.mapred.Task.saveTaskOutput(Task.java:426)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:165)
2008-02-14 19:49:49,225 FATAL crawl.Injector - Injector:
java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:831)
at org.apache.nutch.crawl.Injector.inject(Injector.java:162)
at org.apache.nutch.crawl.Injector.run(Injector.java:192)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:54)
at org.apache.nutch.crawl.Injector.main(Injector.java:182)
Indeed the \'inject-temp-322737506\' is present in the specified
folder of D drive and doesn\'t get deleted.
Is this because multiple map/reduce is running and one of them is
finding the directory to be present and therefore fails?
So, I also tried setting this in \'conf/hadoop-site.xml\':-
<property>
<name>mapred.speculative.execution</name>
<value>false</value>
<description></description>
</property>
I wonder why the same issue doesn\'t occur in Linux. I am not well
acquainted with the Hadoop code yet. Could someone throw light on what
might be going wrong?
Regards,
Susam Pal
On 2/7/08, DS jha <[EMAIL PROTECTED]> wrote:
Hi -
>
> Looks like latest trunk version of nutch is failing with the following
> exception when trying to perform inject operation:
>
> java.io.IOException: Target
> file:/tmp/hadoop-user/mapred/temp/inject-temp-1280136828/_reduce_dv90x0/part-00000
> already exists
> at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:246)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:125)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:116)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:196)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:394)
> at org.apache.hadoop.mapred.Task.moveTaskOutputs(Task.java:452)
> at org.apache.hadoop.mapred.Task.moveTaskOutputs(Task.java:469)
> at org.apache.hadoop.mapred.Task.saveTaskOutput(Task.java:426)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:165)
>
> Any thoughts?
>
> Thanks
> Jha
>
