[ 
https://issues.apache.org/jira/browse/LUCENE-7882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16074536#comment-16074536
 ] 

Michael McCandless commented on LUCENE-7882:
--------------------------------------------

Sorry, I did not experience a full hang/deadlock: I only experienced that the 
QPS capacity of the searcher went down drastically for periods of time.  The 
pattern is odd ... at first the QPS is high, then it gradually slows down, then 
it enters periods where it's extremely slow, like 10X slower than "normal" for 
maybe ~20-30 seconds, then it gets somewhat faster again, then another super 
slow period.  But it never outright hangs.

It is odd to me that the code cache was allowed to grow to nearly it's maximum 
size; seems like these methods should very quickly become dead since after that 
one query executes, it should no longer be referenced.

bq. Can you repeat and add -XX:+UseCodeCacheFlushing switch, see if this helps?

OK I'll try that, may be a while before I can though.  Though from its 
description it seems like it shouldn't be necessary here, i.e. the JVM should 
be able to tell, quickly, that these methods are not referenced anymore.  But I 
know very little about this part of the JVM!

> Maybe expression compiler should cache recently compiled expressions?
> ---------------------------------------------------------------------
>
>                 Key: LUCENE-7882
>                 URL: https://issues.apache.org/jira/browse/LUCENE-7882
>             Project: Lucene - Core
>          Issue Type: Improvement
>          Components: modules/expressions
>            Reporter: Michael McCandless
>
> I've been running search performance tests using a simple expression 
> ({{_score + ln(1000+unit_sales)}}) for sorting and hit this odd bottleneck:
> {noformat}
> "pool-1-thread-30" #70 prio=5 os_prio=0 tid=0x00007eea7000a000 nid=0x1ea8a 
> waiting for monitor entry [0x00007eea867dd000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
>       at 
> org.apache.lucene.expressions.js.JavascriptCompiler$CompiledExpression.evaluate(_score
>  + ln(1000+unit_sales))
>       at 
> org.apache.lucene.expressions.ExpressionFunctionValues.doubleValue(ExpressionFunctionValues.java:49)
>       at 
> com.amazon.lucene.OrderedVELeafCollector.collectInternal(OrderedVELeafCollector.java:123)
>       at 
> com.amazon.lucene.OrderedVELeafCollector.collect(OrderedVELeafCollector.java:108)
>       at 
> org.apache.lucene.search.MultiCollectorManager$Collectors$LeafCollectors.collect(MultiCollectorManager.java:102)
>       at 
> org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:241)
>       at 
> org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:184)
>       at org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
>       at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:658)
>       at org.apache.lucene.search.IndexSearcher$5.call(IndexSearcher.java:600)
>       at org.apache.lucene.search.IndexSearcher$5.call(IndexSearcher.java:597)
>       at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>       at java.lang.Thread.run(Thread.java:745)
> {noformat}
> I couldn't see any {{synchronized}} in the sources here, so I'm not sure 
> which object monitor it's blocked on.
> I was accidentally compiling a new expression for every query, and that 
> bottleneck would cause overall QPS to slow down drastically (~4X slower after 
> ~1 hour of redline tests), as if the JVM is getting slower and slower to 
> evaluate each expression the more expressions I had compiled.
> I tested JDK 9-ea and it also kept slowing down over time as the performance 
> test ran.
> Maybe we should put a small cache in front of the expressions compiler to 
> make it less trappy?  Or maybe we can get to the root cause of why the JVM 
> slows down more and more, the more expressions you compile?
> I won't have time to work on this in the near future so if anyone else feels 
> the itch, please scratch it!



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Reply via email to