[jira] Commented: (LUCENE-806) Synchronization bottleneck in FieldSortedHitQueue with many concurrent readers
[ https://issues.apache.org/jira/browse/LUCENE-806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12486252 ] Paul Cowan commented on LUCENE-806: --- Otis, you're probably right -- it may not be wise to tackle two birds with one stone. My concern is that if I do this a quick and dirty way, it may involve exposing an API to enable/disable this behaviour which a subsequent refactor would then remove, and I'd obviously rather keep the API stable. I'm about to attach 3 patches, with varying levels of effect on the code. I'd be interested to see what people think is the best approach given the possible refactor. Hoss, I've had a look at your patch, and rather like it. That's kind of tackling a slightly different problem; that's cleaning up the FieldCache (which is a great idea) whereas cleaning up FSHQ is only incidentally related to FieldCache. It uses it (and if it was broken up, each comparator source would be using your much cleaner API) but I think the two coexist quite happily. I'd like to see both, in other words! > Synchronization bottleneck in FieldSortedHitQueue with many concurrent readers > -- > > Key: LUCENE-806 > URL: https://issues.apache.org/jira/browse/LUCENE-806 > Project: Lucene - Java > Issue Type: Improvement > Components: Search >Affects Versions: 2.0.0 >Reporter: Paul Cowan >Priority: Minor > Attachments: lucene-806-proposed-direction.patch, lucene-806.patch > > > The below is from a post by (my colleague) Paul Smith to the java-users list: > --- > Hi ho peoples. > We have an application that is internationalized, and stores data from many > languages (each project has it's own index, mostly aligned with a single > language, maybe 2). > Anyway, I've noticed during some thread dumps diagnosing some performance > issues, that there appears to be a _potential_ synchronization bottleneck > using Locale-based sorting of Strings. I don't think this problem is the > root cause of our performance problem, but I thought I'd mention it here. > Here's the stack dump of a thread waiting: > "http-1001-Processor245" daemon prio=1 tid=0x31434da0 nid=0x3744 waiting for > monitor entry [0x2cd44000..0x2cd45f30] > at java.text.RuleBasedCollator.compare(RuleBasedCollator.java) > - waiting to lock <0x6b1e8c68> (a java.text.RuleBasedCollator) > at > org.apache.lucene.search.FieldSortedHitQueue$4.compare(FieldSortedHitQueue.java:320) > at > org.apache.lucene.search.FieldSortedHitQueue.lessThan(FieldSortedHitQueue.java:114) > at org.apache.lucene.util.PriorityQueue.upHeap(PriorityQueue.java:120) > at org.apache.lucene.util.PriorityQueue.put(PriorityQueue.java:47) > at org.apache.lucene.util.PriorityQueue.insert(PriorityQueue.java:58) > at > org.apache.lucene.search.FieldSortedHitQueue.insert(FieldSortedHitQueue.java:90) > at > org.apache.lucene.search.FieldSortedHitQueue.insert(FieldSortedHitQueue.java:97) > at > org.apache.lucene.search.TopFieldDocCollector.collect(TopFieldDocCollector.java:47) > at > org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:291) > at > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:132) > at > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:110) > at > com.aconex.index.search.FastLocaleSortIndexSearcher.search(FastLocaleSortIndexSearcher.java:90) > . > In our case we had 12 threads waiting like this, while one thread had the > lock on the RuleBasedCollator. Turns out RuleBasedCollator's.compare(...) > method is synchronized. I wonder if a ThreadLocal based collator would be > better here... ? There doesn't appear to be a reason for other threads > searching the same index to wait on this sort. Be just as easy to use their > own. (Is RuleBasedCollator a "heavy" object memory wise? Wouldn't have > thought so, per thread) > Thoughts? > --- > I've investigated this somewhat, and agree that this is a potential problem > with a series of possible workarounds. Further discussion (including > proof-of-concept patch) to follow. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-806) Synchronization bottleneck in FieldSortedHitQueue with many concurrent readers
[
https://issues.apache.org/jira/browse/LUCENE-806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12483409
]
Hoss Man commented on LUCENE-806:
-
Paul: I have not had a chance to look at your patch (or most patches i've
wanted to look at the last few weeks) but skimming the mail this sentence
jumped out at me...
> This lets me work towards the next step, replacing the logic in the
> FieldCacheImpl.Cache
> { createValue() } with another SCS, using some sort of 'pluggable' mechanism
> rather
> than just creating new SCSes on the fly
I'm curious as to what you think of LUCENE-831 ... does that patch go in the
direction you're thinking?
> Synchronization bottleneck in FieldSortedHitQueue with many concurrent readers
> --
>
> Key: LUCENE-806
> URL: https://issues.apache.org/jira/browse/LUCENE-806
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
>Affects Versions: 2.0.0
>Reporter: Paul Cowan
>Priority: Minor
> Attachments: lucene-806-proposed-direction.patch, lucene-806.patch
>
>
> The below is from a post by (my colleague) Paul Smith to the java-users list:
> ---
> Hi ho peoples.
> We have an application that is internationalized, and stores data from many
> languages (each project has it's own index, mostly aligned with a single
> language, maybe 2).
> Anyway, I've noticed during some thread dumps diagnosing some performance
> issues, that there appears to be a _potential_ synchronization bottleneck
> using Locale-based sorting of Strings. I don't think this problem is the
> root cause of our performance problem, but I thought I'd mention it here.
> Here's the stack dump of a thread waiting:
> "http-1001-Processor245" daemon prio=1 tid=0x31434da0 nid=0x3744 waiting for
> monitor entry [0x2cd44000..0x2cd45f30]
> at java.text.RuleBasedCollator.compare(RuleBasedCollator.java)
> - waiting to lock <0x6b1e8c68> (a java.text.RuleBasedCollator)
> at
> org.apache.lucene.search.FieldSortedHitQueue$4.compare(FieldSortedHitQueue.java:320)
> at
> org.apache.lucene.search.FieldSortedHitQueue.lessThan(FieldSortedHitQueue.java:114)
> at org.apache.lucene.util.PriorityQueue.upHeap(PriorityQueue.java:120)
> at org.apache.lucene.util.PriorityQueue.put(PriorityQueue.java:47)
> at org.apache.lucene.util.PriorityQueue.insert(PriorityQueue.java:58)
> at
> org.apache.lucene.search.FieldSortedHitQueue.insert(FieldSortedHitQueue.java:90)
> at
> org.apache.lucene.search.FieldSortedHitQueue.insert(FieldSortedHitQueue.java:97)
> at
> org.apache.lucene.search.TopFieldDocCollector.collect(TopFieldDocCollector.java:47)
> at
> org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:291)
> at
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:132)
> at
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:110)
> at
> com.aconex.index.search.FastLocaleSortIndexSearcher.search(FastLocaleSortIndexSearcher.java:90)
> .
> In our case we had 12 threads waiting like this, while one thread had the
> lock on the RuleBasedCollator. Turns out RuleBasedCollator's.compare(...)
> method is synchronized. I wonder if a ThreadLocal based collator would be
> better here... ? There doesn't appear to be a reason for other threads
> searching the same index to wait on this sort. Be just as easy to use their
> own. (Is RuleBasedCollator a "heavy" object memory wise? Wouldn't have
> thought so, per thread)
> Thoughts?
> ---
> I've investigated this somewhat, and agree that this is a potential problem
> with a series of possible workarounds. Further discussion (including
> proof-of-concept patch) to follow.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-806) Synchronization bottleneck in FieldSortedHitQueue with many concurrent readers
[ https://issues.apache.org/jira/browse/LUCENE-806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12483322 ] Otis Gospodnetic commented on LUCENE-806: - Paul: Haven't looked at the patch, but liked that 2x-4x performance increase (even though I personally don't use Sort with Locale). I didn't follow everything you said, but it sounds like a lot of internal shuffling. Perhaps, to address just the performance, you can just go with adding ThreadLocal RuleBasedCollators in FSHQ and do the other changes in a separate issue and patch, if people think this would be an improvement. > Synchronization bottleneck in FieldSortedHitQueue with many concurrent readers > -- > > Key: LUCENE-806 > URL: https://issues.apache.org/jira/browse/LUCENE-806 > Project: Lucene - Java > Issue Type: Improvement > Components: Search >Affects Versions: 2.0.0 >Reporter: Paul Cowan >Priority: Minor > Attachments: lucene-806-proposed-direction.patch, lucene-806.patch > > > The below is from a post by (my colleague) Paul Smith to the java-users list: > --- > Hi ho peoples. > We have an application that is internationalized, and stores data from many > languages (each project has it's own index, mostly aligned with a single > language, maybe 2). > Anyway, I've noticed during some thread dumps diagnosing some performance > issues, that there appears to be a _potential_ synchronization bottleneck > using Locale-based sorting of Strings. I don't think this problem is the > root cause of our performance problem, but I thought I'd mention it here. > Here's the stack dump of a thread waiting: > "http-1001-Processor245" daemon prio=1 tid=0x31434da0 nid=0x3744 waiting for > monitor entry [0x2cd44000..0x2cd45f30] > at java.text.RuleBasedCollator.compare(RuleBasedCollator.java) > - waiting to lock <0x6b1e8c68> (a java.text.RuleBasedCollator) > at > org.apache.lucene.search.FieldSortedHitQueue$4.compare(FieldSortedHitQueue.java:320) > at > org.apache.lucene.search.FieldSortedHitQueue.lessThan(FieldSortedHitQueue.java:114) > at org.apache.lucene.util.PriorityQueue.upHeap(PriorityQueue.java:120) > at org.apache.lucene.util.PriorityQueue.put(PriorityQueue.java:47) > at org.apache.lucene.util.PriorityQueue.insert(PriorityQueue.java:58) > at > org.apache.lucene.search.FieldSortedHitQueue.insert(FieldSortedHitQueue.java:90) > at > org.apache.lucene.search.FieldSortedHitQueue.insert(FieldSortedHitQueue.java:97) > at > org.apache.lucene.search.TopFieldDocCollector.collect(TopFieldDocCollector.java:47) > at > org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:291) > at > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:132) > at > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:110) > at > com.aconex.index.search.FastLocaleSortIndexSearcher.search(FastLocaleSortIndexSearcher.java:90) > . > In our case we had 12 threads waiting like this, while one thread had the > lock on the RuleBasedCollator. Turns out RuleBasedCollator's.compare(...) > method is synchronized. I wonder if a ThreadLocal based collator would be > better here... ? There doesn't appear to be a reason for other threads > searching the same index to wait on this sort. Be just as easy to use their > own. (Is RuleBasedCollator a "heavy" object memory wise? Wouldn't have > thought so, per thread) > Thoughts? > --- > I've investigated this somewhat, and agree that this is a potential problem > with a series of possible workarounds. Further discussion (including > proof-of-concept patch) to follow. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
