Hi there,
I just noticed after checking out the latest SVN of Nutch that I am
currently failing the TestSegmentMergeTool Junit test when I type "ant test"
for Nutch. Is anyone experiencing the same problem? Here is the relevant
information which I captured out of the
$NUTCH_HOME/build/test/TEST-org.apache.nutch.tools.TestSegmentMergeTool.txt
file:
Testsuite: org.apache.nutch.tools.TestSegmentMergeTool
Tests run: 3, Failures: 1, Errors: 0, Time elapsed: 46.256 sec
------------- Standard Error -----------------
050926 215316 parsing
file:/C:/Program%20Files/eclipse/workspace/nutch/conf/nutch-default.xml
050926 215316 parsing
file:/C:/Program%20Files/eclipse/workspace/nutch/build/test/classes/nutch-si
te.xml
050926 215316 Using URL normalizer: org.apache.nutch.net.BasicUrlNormalizer
050926 215321 No FS indicated, using default:local
050926 215321 * Opening 10 segments:
050926 215321 - segment seg0: 500 records.
050926 215321 - segment seg1: 500 records.
050926 215321 - segment seg2: 500 records.
050926 215321 - segment seg3: 500 records.
050926 215321 - segment seg4: 500 records.
050926 215321 - segment seg5: 500 records.
050926 215321 - segment seg6: 500 records.
050926 215321 - segment seg7: 500 records.
050926 215321 - segment seg8: 500 records.
050926 215321 - segment seg9: 500 records.
050926 215321 * TOTAL 5000 input records in 10 segments.
050926 215321 * Creating master index...
050926 215328 * Creating index took 6356 ms
050926 215328 * Optimizing index took 0 ms
050926 215328 * Removing duplicate entries...
050926 215328 * Deduplicating took 652 ms
050926 215328 * Merging all segments into output
050926 215333 * Merging took 4381 ms
050926 215333 * Deleting old segments...
050926 215333 Finished SegmentMergeTool: INPUT: 5000 -> OUTPUT: 5000 entries
in 12.15 s (416.66666 entries/sec).
050926 215339 No FS indicated, using default:local
050926 215339 * Opening 10 segments:
050926 215339 - segment seg0: 500 records.
050926 215339 - segment seg1: 500 records.
050926 215339 - segment seg2: 500 records.
050926 215339 - segment seg3: 500 records.
050926 215339 - segment seg4: 500 records.
050926 215339 - segment seg5: 500 records.
050926 215339 - segment seg6: 500 records.
050926 215339 - segment seg7: 500 records.
050926 215339 - segment seg8: 500 records.
050926 215339 - segment seg9: 500 records.
050926 215339 * TOTAL 5000 input records in 10 segments.
050926 215339 * Creating master index...
050926 215344 * Creating index took 5083 ms
050926 215344 * Optimizing index took 0 ms
050926 215344 * Removing duplicate entries...
050926 215344 * Deduplicating took 150 ms
050926 215344 * Merging all segments into output
050926 215345 * Merging took 662 ms
050926 215345 * Deleting old segments...
050926 215345 Finished SegmentMergeTool: INPUT: 5000 -> OUTPUT: 500 entries
in 6.316 s (833.3333 entries/sec).
java.lang.Exception: Missing or invalid 'fetcher' or 'fetcher_output'
directory in
c:\DOCUME~1\mattmann\LOCALS~1\Temp\.smttest63088\output\.fastmerge_index
at
org.apache.nutch.segment.SegmentReader.isParsedSegment(SegmentReader.java:16
8)
at
org.apache.nutch.segment.SegmentReader.<init>(SegmentReader.java:143)
at
org.apache.nutch.segment.SegmentReader.<init>(SegmentReader.java:82)
at
org.apache.nutch.tools.TestSegmentMergeTool.testSameMerge(TestSegmentMergeTo
ol.java:185)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at junit.framework.TestCase.runTest(TestCase.java:154)
at junit.framework.TestCase.runBare(TestCase.java:127)
at junit.framework.TestResult$1.protect(TestResult.java:106)
at junit.framework.TestResult.runProtected(TestResult.java:124)
at junit.framework.TestResult.run(TestResult.java:109)
at junit.framework.TestCase.run(TestCase.java:118)
at junit.framework.TestSuite.runTest(TestSuite.java:208)
at junit.framework.TestSuite.run(TestSuite.java:203)
at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRu
nner.java:289)
at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestR
unner.java:523)
junit.framework.AssertionFailedError: Missing or invalid 'fetcher' or
'fetcher_output' directory in
c:\DOCUME~1\mattmann\LOCALS~1\Temp\.smttest63088\output\.fastmerge_index
at junit.framework.Assert.fail(Assert.java:47)
at
org.apache.nutch.tools.TestSegmentMergeTool.testSameMerge(TestSegmentMergeTo
ol.java:190)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at junit.framework.TestCase.runTest(TestCase.java:154)
at junit.framework.TestCase.runBare(TestCase.java:127)
at junit.framework.TestResult$1.protect(TestResult.java:106)
at junit.framework.TestResult.runProtected(TestResult.java:124)
at junit.framework.TestResult.run(TestResult.java:109)
at junit.framework.TestCase.run(TestCase.java:118)
at junit.framework.TestSuite.runTest(TestSuite.java:208)
at junit.framework.TestSuite.run(TestSuite.java:203)
at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRu
nner.java:289)
at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestR
unner.java:523)
050926 215351 No FS indicated, using default:local
050926 215351 * Opening 10 segments:
050926 215351 - segment seg0: 0 records.
050926 215351 - segment seg1: 500 records.
050926 215351 Unable to fix segment seg2: Missing data file in
c:\DOCUME~1\mattmann\LOCALS~1\Temp\.smttest63089\segments\seg2\content,
impossible to fix this.
050926 215351 - fixed fetcher
050926 215351 - fixed content
050926 215351 - fixed parse_data
050926 215351 - fixed parse_text
050926 215351 Finished fixing seg3
050926 215351 - segment seg3: 500 records.
050926 215351 - segment seg4: 500 records.
050926 215351 - segment seg5: 500 records.
050926 215351 - segment seg6: 500 records.
050926 215351 - segment seg7: 500 records.
050926 215351 - segment seg8: 500 records.
050926 215351 - segment seg9: 500 records.
050926 215351 * TOTAL 4000 input records in 9 segments.
050926 215351 * Creating master index...
050926 215352 - segment seg1 truncated to 376 records
050926 215355 * Creating index took 4020 ms
050926 215355 * Optimizing index took 0 ms
050926 215355 * Removing duplicate entries...
050926 215356 * Deduplicating took 311 ms
050926 215356 * Merging all segments into output
050926 215402 * Merging took 6356 ms
050926 215402 * Deleting old segments...
050926 215402 Finished SegmentMergeTool: INPUT: 4000 -> OUTPUT: 3875 entries
in 11.218 s (363.63635 entries/sec).
------------- ---------------- ---------------
Testcase: testUniqueMerge took 16.982 sec
Testcase: testSameMerge took 12.181 sec
FAILED
Missing or invalid 'fetcher' or 'fetcher_output' directory in
c:\DOCUME~1\mattmann\LOCALS~1\Temp\.smttest63088\output\.fastmerge_index
junit.framework.AssertionFailedError: Missing or invalid 'fetcher' or
'fetcher_output' directory in
c:\DOCUME~1\mattmann\LOCALS~1\Temp\.smttest63088\output\.fastmerge_index
at
org.apache.nutch.tools.TestSegmentMergeTool.testSameMerge(TestSegmentMergeTo
ol.java:194)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)
Testcase: testCorruptSegmentMerge took 17.093 sec
Any ideas why this is happening?
Thanks,
Chris
______________________________________________
Chris A. Mattmann
[EMAIL PROTECTED]
Staff Member
Modeling and Data Management Systems Section (387)
Data Management Systems and Technologies Group
_________________________________________________
Jet Propulsion Laboratory Pasadena, CA
Office: 171-266B Mailstop: 171-246
_______________________________________________________
Disclaimer: The opinions presented within are my own and do not reflect
those of either NASA, JPL, or the California Institute of Technology.
