Hi,
> It just keeps increasing after each crawling.
What does this precisely mean?
(a) Are you running one crawl process with many cycles (depth)
by launching "bin/nutch crawl" (org.apache.nutch.crawl.Crawl)
(b) or in separate steps (inject, generate, fetch, parse, updatedb, ...)?
For (a) see NUTCH-356 and NUTCH-1087 which may cause the problem.
Sebastian
On 04/09/2013 01:49 AM, Deals Collect wrote:
> Hi all,
>
> I have a site crawling the deals every days. I have a problem with permgen
> size. It just keeps increasing after each crawling. That's why I got the
> "Out of memory" issue. As I know, Permgen heap just stores the class
> definition, metadata... I'm not sure why it keeps increasing like this...
> I'm using Nutch 1.4 and Solr 3.1. Here is the link that you can see the
> permgen size: http://www.dealscollect.com.au/permgen It is the last one:
>
> Name: PS Perm Gen
> Type: Non-heap memory
> Max Usage: 268435456
> Used: 75770136
>
> Many thanks,
> Vu
>
