[ https://issues.apache.org/jira/browse/LUCENE-7882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16599076#comment-16599076 ]
Tony Xu commented on LUCENE-7882: --------------------------------- Oh I forgot to paste the test output. 2 Loading class <org.apache.lucene.expressions.Expression> , using Loaderorg.apache.lucene.expressions.js.JavascriptCompiler$Loader@3973f90b Loading class <java.lang.String> , using Loaderorg.apache.lucene.expressions.js.JavascriptCompiler$Loader@3973f90b Loading class <java.lang.Math> , using Loaderorg.apache.lucene.expressions.js.JavascriptCompiler$Loader@3973f90b 3.0 Loading class <org.apache.lucene.expressions.Expression> , using Loaderorg.apache.lucene.expressions.js.JavascriptCompiler$Loader@2a44f939 Loading class <java.lang.String> , using Loaderorg.apache.lucene.expressions.js.JavascriptCompiler$Loader@2a44f939 Loading class <java.lang.Math> , using Loaderorg.apache.lucene.expressions.js.JavascriptCompiler$Loader@2a44f939 3.0 > Maybe expression compiler should cache recently compiled expressions? > --------------------------------------------------------------------- > > Key: LUCENE-7882 > URL: https://issues.apache.org/jira/browse/LUCENE-7882 > Project: Lucene - Core > Issue Type: Improvement > Components: modules/expressions > Reporter: Michael McCandless > Priority: Major > Attachments: demo.patch > > > I've been running search performance tests using a simple expression > ({{_score + ln(1000+unit_sales)}}) for sorting and hit this odd bottleneck: > {noformat} > "pool-1-thread-30" #70 prio=5 os_prio=0 tid=0x00007eea7000a000 nid=0x1ea8a > waiting for monitor entry [0x00007eea867dd000] > java.lang.Thread.State: BLOCKED (on object monitor) > at > org.apache.lucene.expressions.js.JavascriptCompiler$CompiledExpression.evaluate(_score > + ln(1000+unit_sales)) > at > org.apache.lucene.expressions.ExpressionFunctionValues.doubleValue(ExpressionFunctionValues.java:49) > at > com.amazon.lucene.OrderedVELeafCollector.collectInternal(OrderedVELeafCollector.java:123) > at > com.amazon.lucene.OrderedVELeafCollector.collect(OrderedVELeafCollector.java:108) > at > org.apache.lucene.search.MultiCollectorManager$Collectors$LeafCollectors.collect(MultiCollectorManager.java:102) > at > org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:241) > at > org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:184) > at org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39) > at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:658) > at org.apache.lucene.search.IndexSearcher$5.call(IndexSearcher.java:600) > at org.apache.lucene.search.IndexSearcher$5.call(IndexSearcher.java:597) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > {noformat} > I couldn't see any {{synchronized}} in the sources here, so I'm not sure > which object monitor it's blocked on. > I was accidentally compiling a new expression for every query, and that > bottleneck would cause overall QPS to slow down drastically (~4X slower after > ~1 hour of redline tests), as if the JVM is getting slower and slower to > evaluate each expression the more expressions I had compiled. > I tested JDK 9-ea and it also kept slowing down over time as the performance > test ran. > Maybe we should put a small cache in front of the expressions compiler to > make it less trappy? Or maybe we can get to the root cause of why the JVM > slows down more and more, the more expressions you compile? > I won't have time to work on this in the near future so if anyone else feels > the itch, please scratch it! -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org