Hi Stefan,
I get the same errors with analyze (link-analysis) on a box with 8GB ram and Java set to
-Xmx1500m and Xms1500m.
Massimo
Stefan Groschupf wrote:
I had tried to track down the problem.
Thats is very difficult..
May be i found something:
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:359)
at net.nutch.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:42)
at net.nutch.io.DataOutputBuffer.write(DataOutputBuffer.java:76)
at net.nutch.io.SequenceFile$Reader.next(SequenceFile.java:226)
at net.nutch.io.SequenceFile$Sorter$SortPass.run(SequenceFile.java:485)
at net.nutch.io.SequenceFile$Sorter.sortPass(SequenceFile.java:452)
at net.nutch.io.SequenceFile$Sorter.sort(SequenceFile.java:440)
I run a linux box with 512 MB ram and the segment i try to update to the db is around 1 GB.
That should be ok, isn't?
Any ideas where i can start to search for the problem. I allready tried since 5 hours to find a starting point to fix my problem.
Has anyone else updated the db with segment > 1GB and <512 MB RAM? Thanks for hints! Stefan
Am 15.06.2004 um 00:24 schrieb Stefan Groschupf:
Hi,
I notice something strange, I had a system crash until fetching (our admin made a mistake).
Then i had deleted the latest segment.
I try to continue fetching but until segment generation i got a OutOfMemoryError.
I tried different setups with -Xms and Xmx but nothing changed.
My db is > 1 GB.
here my the std.out:
[EMAIL PROTECTED]:~/nutch-0.5-dev$ bin/nutch generate db segments -topN 1000
040615 001657 FetchListTool started
040615 001657 topN:1000
040615 001658 loading file:/home/nutch/nutch-0.5-dev/conf/nutch-default.xml
040615 001658 loading file:/home/nutch/nutch-0.5-dev/conf/nutch-site.xml
040615 001659 Processing page 50000...
040615 001659 Processing page 100000...
040615 001659 Processing page 150000...
040615 001659 Processing page 200000...
040615 001700 Processing page 250000...
040615 001700 Processing page 300000...
040615 001700 Processing page 350000...
040615 001700 Processing page 400000...
040615 001701 Processing page 450000...
040615 001701 Processing page 500000...
040615 001701 Processing page 550000...
040615 001701 Overall processing: Sorted 0 entries in 0.0 seconds.
040615 001701 Overall processing: Sorted NaN entries/second
Exception in thread "main" java.lang.OutOfMemoryError
I wondering why it crashs until sorting and not until parsing. Is there may be a other error in the db? Any idea how to solve the problem? Or should i refetch all ? ;(
Thanks for any hints. Stefan
--------------------------------------------------------------- enterprise information technology consulting open technology: http://www.media-style.com open source: http://www.weta-group.net open discussion: http://www.text-mining.org
------------------------------------------------------- This SF.Net email is sponsored by The 2004 JavaOne(SM) Conference Learn from the experts at JavaOne(SM), Sun's Worldwide Java Developer Conference, June 28 - July 1 at the Moscone Center in San Francisco, CA REGISTER AND SAVE! http://java.sun.com/javaone/sf Priority Code NWMGYKND _______________________________________________ Nutch-developers mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/nutch-developers
--------------------------------------------------------------- enterprise information technology consulting open technology: http://www.media-style.com open source: http://www.weta-group.net open discussion: http://www.text-mining.org
------------------------------------------------------- This SF.Net email is sponsored by The 2004 JavaOne(SM) Conference Learn from the experts at JavaOne(SM), Sun's Worldwide Java Developer Conference, June 28 - July 1 at the Moscone Center in San Francisco, CA REGISTER AND SAVE! http://java.sun.com/javaone/sf Priority Code NWMGYKND _______________________________________________ Nutch-developers mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/nutch-developers
------------------------------------------------------- This SF.Net email is sponsored by The 2004 JavaOne(SM) Conference Learn from the experts at JavaOne(SM), Sun's Worldwide Java Developer Conference, June 28 - July 1 at the Moscone Center in San Francisco, CA REGISTER AND SAVE! http://java.sun.com/javaone/sf Priority Code NWMGYKND _______________________________________________ Nutch-developers mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/nutch-developers
