Thanks Julien I looked into the nutch-default.xml and I can't find a directive that can control the number of incoming links to be taken into account for scoring a document. I can find db.max.inlinks, but it look like controling the invertlinks process only. Could you tell me how to do it?
Regards Edwin On Fri, Mar 20, 2009 at 6:45 AM, Julien Nioche < lists.digitalpeb...@gmail.com> wrote: > Hi Edwin, > > I had a similar issue which I solved by capping the number of incoming > links > to be taken into account for scoring a document. Another option is to use > the patch I submitted (NUTCH-702) on JIRA and does the lazy instanciation > of > metadata; that should save a lot of RAM (and CPU). > > HTH > > Julien > > > -- > DigitalPebble Ltd > http://www.digitalpebble.com > > 2009/3/19 Edwin Chu <edwinche...@gmail.com> > > > Hi, > > I am using the trunk version of Nutch in a cluster of 5 EC2 nodes to > crawl > > the Internet. Each nodes has 7GB of memory and I have > > configured mapred.child.java.opts to -Xmx3000m in hadoop-site.xml. When I > > tried to update the crawldb of about 20M of urls with a crawl segment > with > > 5M of fetched content, I got the following error: > > > > java.lang.OutOfMemoryError: Java heap space > > > > at java.util.concurrent.locks.ReentrantLock.(Unknown Source) > > > > at java.util.concurrent.ConcurrentHashMap$Segment.(Unknown Source) > > > > at java.util.concurrent.ConcurrentHashMap.(Unknown Source) > > > > at java.util.concurrent.ConcurrentHashMap.(Unknown Source) > > > > at org.apache.hadoop.io.AbstractMapWritable.(AbstractMapWritable.java:46) > > > > at org.apache.hadoop.io.MapWritable.(MapWritable.java:42) > > > > at org.apache.hadoop.io.MapWritable.(MapWritable.java:52) > > > > at org.apache.nutch.crawl.CrawlDatum.set(CrawlDatum.java:311) > > > > at org.apache.nutch.crawl.CrawlDbReducer.reduce(CrawlDbReducer.java:96) > > > > at org.apache.nutch.crawl.CrawlDbReducer.reduce(CrawlDbReducer.java:1) > > > > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:430) > > > > at org.apache.hadoop.mapred.Child.main(Child.java:155) > > > > > > java.lang.OutOfMemoryError: GC overhead limit exceeded > > > > at java.util.concurrent.locks.ReentrantLock.(Unknown Source) > > > > at java.util.concurrent.ConcurrentHashMap$Segment.(Unknown Source) > > > > at java.util.concurrent.ConcurrentHashMap.(Unknown Source) > > > > at java.util.concurrent.ConcurrentHashMap.(Unknown Source) > > > > at org.apache.hadoop.io.AbstractMapWritable.(AbstractMapWritable.java:46) > > > > at org.apache.hadoop.io.MapWritable.(MapWritable.java:42) > > > > at org.apache.nutch.crawl.CrawlDatum.(CrawlDatum.java:135) > > > > at org.apache.nutch.crawl.CrawlDbReducer.reduce(CrawlDbReducer.java:95) > > > > at org.apache.nutch.crawl.CrawlDbReducer.reduce(CrawlDbReducer.java:1) > > > > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:430) > > > > at org.apache.hadoop.mapred.Child.main(Child.java:155) > > > > > > Anyone has an idea on this problem? I supposed that output of reduce > > function is written to filesystem immediately instead of being hold in > > memory longer than necessary, otherwise the system would not be able to > > scale. I think 3GB limit is maximum because there is no swap space in EC2 > > and each node can run maximum of 2 map/reduce tasks. > > > > Thank you very much. > > > > Regards > > > > Edwin > > >
