What is your index size? if you have enough RAM , try out a bigger perm gen size. If you shard, you will not be able to use the same data dir. The data has to be split among shards. Which mean reindexing. --Noble
On Fri, Oct 2, 2009 at 7:20 AM, Artem Russakovskii (JIRA) <j...@apache.org> wrote: > > [ > https://issues.apache.org/jira/browse/SOLR-1482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12761449#action_12761449 > ] > > Artem Russakovskii commented on SOLR-1482: > ------------------------------------------ > > I'm getting an error even just trying to access a single shard's admin > interface, even after adjusting -XX:MaxPermSize=512m > > {quote} > ==> catalina.out <== > Oct 1, 2009 6:47:06 PM org.apache.coyote.http11.Http11Processor process > SEVERE: Error processing request > java.lang.OutOfMemoryError: PermGen space > at java.lang.Throwable.getStackTraceElement(Native Method) > at java.lang.Throwable.getOurStackTrace(Throwable.java:591) > at java.lang.Throwable.printStackTrace(Throwable.java:510) > at java.util.logging.SimpleFormatter.format(SimpleFormatter.java:72) > at org.apache.juli.FileHandler.publish(FileHandler.java:129) > at java.util.logging.Logger.log(Logger.java:458) > at java.util.logging.Logger.doLog(Logger.java:480) > at java.util.logging.Logger.logp(Logger.java:680) > at org.apache.juli.logging.DirectJDKLog.log(DirectJDKLog.java:167) > at org.apache.juli.logging.DirectJDKLog.error(DirectJDKLog.java:135) > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:324) > at > org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:849) > at > org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583) > at > org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:454) > at java.lang.Thread.run(Thread.java:619) > {quote} > > :-/ > >> Solr master and slave freeze after query >> ---------------------------------------- >> >> Key: SOLR-1482 >> URL: https://issues.apache.org/jira/browse/SOLR-1482 >> Project: Solr >> Issue Type: Bug >> Affects Versions: 1.4 >> Environment: Nightly 9/28/09. >> 14 individual instances per server, using JNDI. >> replicateAfter commit, 5 min interval polling. >> All caches are currently commented out, on both slave and master. >> Lots of ongoing commits - large chunks of data, each accompanied by a >> commit. This is to guarantee that anything we think is now in Solr remains >> there in case the server crashes. >> Reporter: Artem Russakovskii >> Priority: Critical >> >> We're having issues with the deployment of 2 master-slave setups. >> One of the master-slave setups is OK (so far) but on the other both the >> master and the slave keep freezing, but only after I send a query to them. >> And by freezing I mean indefinite hanging, with almost no output to log, no >> errors, nothing. It's as if there's some sort of a deadlock. The hanging >> servers need to be killed with -9, otherwise they keep hanging. >> The query I send queries all instances at the same time using the ?shards= >> syntax. >> On the slave, the logs just stop - nothing shows up anymore after the query >> is issued. On the master, they're a bit more descriptive. This information >> seeps through very-very slowly, as you can see from the timestamps: >> {quote} >> SEVERE: java.lang.OutOfMemoryError: PermGen space >> Oct 1, 2009 2:16:00 PM org.apache.solr.common.SolrException log >> SEVERE: java.lang.OutOfMemoryError: PermGen space >> Oct 1, 2009 2:19:37 PM org.apache.catalina.connector.CoyoteAdapter service >> SEVERE: An exception or error occurred in the container during the request >> processing >> java.lang.OutOfMemoryError: PermGen space >> Oct 1, 2009 2:19:37 PM org.apache.coyote.http11.Http11Processor process >> SEVERE: Error processing request >> java.lang.OutOfMemoryError: PermGen space >> Oct 1, 2009 2:19:39 PM org.apache.catalina.connector.CoyoteAdapter service >> SEVERE: An exception or error occurred in the container during the request >> processing >> java.lang.OutOfMemoryError: PermGen space >> Exception in thread "ContainerBackException in thread "pool-29-threadOct 1, >> 2009 2:21:47 PM org.apache.catalina.connector.CoyoteAdapter service >> SEVERE: An exception or error occurred in the container during the request >> processing >> java.lang.OutOfMemoryError: PermGen space >> Oct 1, 2009 2:21:47 PM >> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler process >> SEVERE: Error reading request, ignored >> java.lang.OutOfMemoryError: PermGen space >> Oct 1, 2009 2:21:47 PM >> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler process >> SEVERE: Error reading request, ignored >> java.lang.OutOfMemoryError: PermGen space >> -22" java.lang.OutOfMemoryError: PermGen space >> Oct 1, 2009 2:21:47 PM >> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler process >> SEVERE: Error reading request, ignored >> java.lang.OutOfMemoryError: PermGen space >> Exception in thread "http-8080-42" Oct 1, 2009 2:21:47 PM >> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler process >> SEVERE: Error reading request, ignored >> java.lang.OutOfMemoryError: PermGen space >> Oct 1, 2009 2:21:47 PM >> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler process >> SEVERE: Error reading request, ignored >> java.lang.OutOfMemoryError: PermGen space >> Oct 1, 2009 2:21:47 PM >> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler process >> SEVERE: Error reading request, ignored >> java.lang.OutOfMemoryError: PermGen space >> Exception in thread "http-8080-26" Exception in thread "http-8080-32" >> Exception in thread "http-8080-25" Exception in thread "http-8080-22" >> Exception in thread "http-8080-15" Exception in thread "http-8080-45" >> Exception in thread "http-8080-13" Exception in thread "http-8080-48" >> Exception in thread "http-8080-7" Exception in thread "http-8080-38" >> Exception in thread "http-8080-39" Exception in thread "http-8080-28" >> Exception in thread "http-8080-1" Exception in thread "http-8080-2" >> Exception in thread "http-8080-12" Exception in thread "http-8080-44" >> Exception in thread "http-8080-47" Exception in thread "http-8080-29" >> Exception in thread "http-8080-33" Exception in thread "http-8080-27" >> Exception in thread "http-8080-36" Exception in thread "http-8080-113" >> Exception in thread "http-8080-112" Exception in thread "http-8080-37" >> Exception in thread "http-8080-18" java.lang.OutOfMemoryError: PermGen space >> java.lang.OutOfMemoryError: PermGen space >> java.lang.OutOfMemoryError: PermGen space >> java.lang.OutOfMemoryError: PermGen space >> java.lang.OutOfMemoryError: PermGen space >> Exception in thread "http-8080-34" java.lang.OutOfMemoryError: PermGen space >> java.lang.OutOfMemoryError: PermGen space >> Exception in thread "http-8080-103" >> {quote} >> So the problem seems to be related to PermGen space. I found >> http://www.nabble.com/Number-of-webapps-td22198080.html and tried >> -XX:MaxPermSize=256m, but it didn't fix the problem. The current >> CATALINA_OPTS looks like this: >> {quote} >> export CATALINA_OPTS="-XX:MaxPermSize=256m -Xmx6500m -XX:+UseConcMarkSweepGC" >> {quote} >> Is the only solution at this point going multicore, as Noble suggested (is >> Noble your first name? I always assumed it was Paul and Noble was part of >> the nickname)? Will multicore get rid of the problem, before we spend time >> looking at it? For multicore, will the existing data dirs be compatible or >> would a complete reindex be needed? >> I'm willing to provide any information to you guys, just not sure what at >> the moment. I'm also open to communicating outside of JIRA, at artem [_aT_] >> plaxo {dot} com. >> Thanks. > > -- > This message is automatically generated by JIRA. > - > You can reply to this email to add a comment to the issue online. > > -- ----------------------------------------------------- Noble Paul | Principal Engineer| AOL | http://aol.com
