[ https://issues.apache.org/jira/browse/NUTCH-925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma updated NUTCH-925: -------------------------------- Fix Version/s: 1.4 This has been fixed for 2.0 in NUTCH-844 but not in 1.x. > plugins stored in weakhashmap lead memory leak > ---------------------------------------------- > > Key: NUTCH-925 > URL: https://issues.apache.org/jira/browse/NUTCH-925 > Project: Nutch > Issue Type: Bug > Affects Versions: 1.2 > Reporter: congliu > Fix For: 1.4 > > > I suffer serious memory leak using Nutch 1.2 though a very deep crawl. I get > the error like this: > Exception in thread "Thread-113544" java.lang.OutOfMemoryError: PermGen space > at java.lang.Throwable.getStackTraceElement(Native Method) > at java.lang.Throwable.getOurStackTrace(Throwable.java:591) > at java.lang.Throwable.printStackTrace(Throwable.java:510) > at > org.apache.log4j.spi.ThrowableInformation.getThrowableStrRep(ThrowableInformation.java:76) > at > org.apache.log4j.spi.LoggingEvent.getThrowableStrRep(LoggingEvent.java:407) > at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:305) > at > org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:359) > at org.apache.log4j.WriterAppender.append(WriterAppender.java:160) > at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251) > at > org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66) > at org.apache.log4j.Category.callAppenders(Category.java:206) > at org.apache.log4j.Category.forcedLog(Category.java:391) > at org.apache.log4j.Category.log(Category.java:856) > at org.slf4j.impl.Log4jLoggerAdapter.log(Log4jLoggerAdapter.java:509) > at > org.apache.commons.logging.impl.SLF4JLocationAwareLog.warn(SLF4JLocationAwareLog.java:173) > at > org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:256) > Exception in thread "main" java.io.IOException: Job failed! > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252) > at org.apache.nutch.fetcher.Fetcher.fetch(Fetcher.java:1107) > at org.apache.nutch.crawl.Crawl.main(Crawl.java:133) > I guess Plugin repository cache lead to memory leak. > As u know plugins is stored in weakhashmap <conf, plugins>, and new class > classload > create when u need plugins. > Usually,WeakHashMap object can been gc, but class and classload is stored in > Perm NOT stack and gc can't perform in Perm, SO (java.lang.OutOfMemoryError: > PermGen space) occured..., is any nutch-issues have concerned this promble? > or there is any solution? > nutch-356 may help? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira