Hello Simon, thank you for the answer and the hint! Leon On Sat, Jul 7, 2012 at 1:29 PM, Simon Willnauer <simon.willna...@gmail.com> wrote: > On Fri, Jul 6, 2012 at 9:28 PM, Leon Rosenberg <rosenberg.l...@gmail.com> > wrote: >> Hello, >> >> we have a small internet shop which uses lucene for product search. >> With increasing traffic we have continuos problem with literaly >> hundreds of threads being BLOCKED in lucene code: >> >> here is an example taken with jstack on prod system: >> >> "ajp-0.0.0.0-8009-1504" daemon prio=10 tid=0x00007f81a518f800 >> nid=0x6554 waiting for monitor entry [0x00007f8142c29000] >> java.lang.Thread.State: BLOCKED (on object monitor) >> at org.apache.lucene.store.RAMFile.numBuffers(RAMFile.java:79) >> - waiting to lock <0x000000054b1daa98> (a >> org.apache.lucene.store.RAMFile) >> at >> org.apache.lucene.store.RAMInputStream.switchCurrentBuffer(RAMInputStream.java:87) >> at >> org.apache.lucene.store.RAMInputStream.readBytes(RAMInputStream.java:73) >> at org.apache.lucene.index.TermBuffer.read(TermBuffer.java:82) >> at >> org.apache.lucene.index.SegmentTermEnum.next(SegmentTermEnum.java:131) >> at >> org.apache.lucene.search.FilteredTermEnum.next(FilteredTermEnum.java:77) >> at >> org.apache.lucene.search.MultiTermQueryWrapperFilter.getDocIdSet(MultiTermQueryWrapperFilter.java:131) >> at >> org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:139) >> at >> org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:298) >> at >> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:524) >> at >> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:391) >> at >> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:298) >> >> we have 1172 threads in same state. >> >> there are two threads which already lock the object: >> >> "ajp-0.0.0.0-8009-548" daemon prio=10 tid=0x00007f81a510b000 >> nid=0x1c64 waiting for monitor entry [0x00007f81897d5000] >> java.lang.Thread.State: BLOCKED (on object monitor) >> at org.apache.lucene.store.RAMFile.getBuffer(RAMFile.java:75) >> - locked <0x000000054b1daa98> (a org.apache.lucene.store.RAMFile) >> at >> org.apache.lucene.store.RAMInputStream.switchCurrentBuffer(RAMInputStream.java:97) >> at >> org.apache.lucene.store.RAMInputStream.readBytes(RAMInputStream.java:73) >> at org.apache.lucene.index.TermBuffer.read(TermBuffer.java:82) >> at >> org.apache.lucene.index.SegmentTermEnum.next(SegmentTermEnum.java:131) >> at >> org.apache.lucene.search.FilteredTermEnum.next(FilteredTermEnum.java:77) >> at >> org.apache.lucene.search.TermCollectingRewrite.collectTerms(TermCollectingRewrite.java:40) >> at >> org.apache.lucene.search.TopTermsRewrite.rewrite(TopTermsRewrite.java:58) >> at >> org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:296) >> at >> org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:378) >> at >> org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:378) >> at >> org.apache.lucene.search.IndexSearcher.rewrite(IndexSearcher.java:589) >> at >> org.apache.lucene.search.Searcher.createNormalizedWeight(Searcher.java:167) >> at >> org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:661) >> at >> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:298) >> >> >> "ajp-0.0.0.0-8009-663" daemon prio=10 tid=0x00007f81c503d800 >> nid=0x225b waiting for monitor entry [0x00007f8182462000] >> java.lang.Thread.State: BLOCKED (on object monitor) >> at org.apache.lucene.store.RAMFile.numBuffers(RAMFile.java:79) >> - locked <0x000000054b1daa98> (a org.apache.lucene.store.RAMFile) >> at >> org.apache.lucene.store.RAMInputStream.switchCurrentBuffer(RAMInputStream.java:87) >> at >> org.apache.lucene.store.RAMInputStream.readByte(RAMInputStream.java:63) >> at org.apache.lucene.store.DataInput.readVInt(DataInput.java:105) >> at org.apache.lucene.index.TermBuffer.read(TermBuffer.java:64) >> at >> org.apache.lucene.index.SegmentTermEnum.next(SegmentTermEnum.java:131) >> at >> org.apache.lucene.search.FilteredTermEnum.next(FilteredTermEnum.java:77) >> at >> org.apache.lucene.search.FilteredTermEnum.setEnum(FilteredTermEnum.java:56) >> at >> org.apache.lucene.search.WildcardTermEnum.<init>(WildcardTermEnum.java:65) >> at >> org.apache.lucene.search.WildcardQuery.getEnum(WildcardQuery.java:59) >> at >> org.apache.lucene.search.MultiTermQueryWrapperFilter.getDocIdSet(MultiTermQueryWrapperFilter.java:103) >> at >> org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:139) >> at >> org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:298) >> at >> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:524) >> at >> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:391) >> at >> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:298) >> >> after some time the BLOCKED threads are waiting to lock another >> object. Over time the amount of locked threads increases. >> >> due to this fact the machine is permanently context-switching and has >> 100% cpu load (vmstat, top). >> >> The responsiveness of search is extremely low, the search needs minutes... >> At the moment of the stacktrace we had approx. 1000 sessions (but the >> problems occured with 400 too) and 1500 threads (we start with 80). >> >> We are reindexing the index every 10 minutes, due to changes in the >> backoffice system. >> >> any ideas what is happening here or what we are doing wrong? >> >> we replaced call to lucene with String.indexOf() to check if the >> problem is in our code, it didn't show the problematic behavior. >> >> Is there a non-blocking search alternative in lucene? > > why don't you just use FSDirectory instead of RAMDirectory? Speed is > usually better too and it doesn't have the problems that RAMDirectory > has. > > simon >> thanx in advance >> Leon >> >> P.S. >> some data about machine, java, etc >> >> >> >> machine: quadcore, ubuntu linux, 2.6.32, 24Gb RAM >> java version "1.6.0_30" >> Java(TM) SE Runtime Environment (build 1.6.0_30-b12) >> Java HotSpot(TM) 64-Bit Server VM (build 20.5-b03, mixed mode) >> >> memory config: >> jmap -heap 29596 >> Attaching to process ID 29596, please wait... >> Debugger attached successfully. >> Server compiler detected. >> JVM version is 20.5-b03 >> >> using thread-local object allocation. >> Parallel GC with 8 thread(s) >> >> Heap Configuration: >> MinHeapFreeRatio = 40 >> MaxHeapFreeRatio = 70 >> MaxHeapSize = 17179869184 (16384.0MB) >> NewSize = 1310720 (1.25MB) >> MaxNewSize = 17592186044415 MB >> OldSize = 5439488 (5.1875MB) >> NewRatio = 2 >> SurvivorRatio = 8 >> PermSize = 21757952 (20.75MB) >> MaxPermSize = 4294967296 (4096.0MB) >> >> >> The machine runs with parallelOldGc and has a full gc every 15-20 minutes. >> The app runs in jboss 6 with jsf. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >> For additional commands, e-mail: java-user-h...@lucene.apache.org >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org >
--------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
