[
https://issues.apache.org/jira/browse/LUCENE-9185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17025233#comment-17025233
]
Dawid Weiss commented on LUCENE-9185:
-------------------------------------
It looks great, thanks Robert.
I'd love to have some kind of task to display all these build options at some
point. Currently this is done just for randomization options (try gradlew
testOpts -p lucene/core) but I'm sure it could be pulled from other parts of
the build and displayed consistently. For now it can stay as it is.
> add "tests.profile" to gradle build to aid fixing slow tests
> ------------------------------------------------------------
>
> Key: LUCENE-9185
> URL: https://issues.apache.org/jira/browse/LUCENE-9185
> Project: Lucene - Core
> Issue Type: Task
> Reporter: Robert Muir
> Priority: Major
> Attachments: LUCENE-9185.patch, LUCENE-9185.patch, LUCENE-9185.patch
>
>
> It is kind of a hassle to profile slow tests to fix the bottlenecks
> The idea here is to make it dead easy to profile (just) the tests, capturing
> samples at a very low granularity, reducing noise as much as possible (e.g.
> not profiling entire gradle build or anything) and print a simple report for
> quick iterating.
> Here's a prototype of what I hacked together:
> All of lucene core: {{./gradlew -p lucene/core test -Dtests.profile=true}}
> {noformat}
> ...
> PROFILE SUMMARY from 122464 samples
> tests.profile.count=10
> tests.profile.stacksize=1
> tests.profile.linenumbers=false
> PERCENT SAMPLES STACK
> 2.59% 3170
> org.apache.lucene.util.compress.LZ4$HighCompressionHashTable#assertReset()
> 2.26% 2762 java.util.Arrays#fill()
> 1.59% 1953 com.carrotsearch.randomizedtesting.RandomizedContext#context()
> 1.24% 1523 java.util.Random#nextInt()
> 1.19% 1456 java.lang.StringUTF16#compress()
> 1.08% 1319 java.lang.StringLatin1#inflate()
> 1.00% 1228 java.lang.Integer#getChars()
> 0.99% 1214 java.util.Arrays#compareUnsigned()
> 0.96% 1179 java.util.zip.Inflater#inflateBytesBytes()
> 0.91% 1114 java.util.concurrent.atomic.AtomicLong#compareAndSet()
> BUILD SUCCESSFUL in 3m 59s
> {noformat}
> If you look at this LZ4 assertReset method, you can see its indeed way too
> expensive, checking 64K items every time.
> To dig deeper into potential problems you can pass additional parameters (all
> of them used here for demonstration):
> {{./gradlew -p solr/core test --tests TestLRUStatsCache -Dtests.profile=true
> -Dtests.profile.count=8 -Dtests.profile.stacksize=20
> -Dtests.profile.linenumbers=true}}
> This clearly finds SOLR-14223 (expensive RSA key generation in CryptoKeys) ...
> {noformat}
> ...
> PROFILE SUMMARY from 21355 samples
> tests.profile.count=8
> tests.profile.stacksize=20
> tests.profile.linenumbers=true
> PERCENT SAMPLES STACK
> 26.30% 5617 sun.nio.ch.EPoll#wait():(Native code)
> at sun.nio.ch.EPollSelectorImpl#doSelect():120
> at sun.nio.ch.SelectorImpl#lockAndDoSelect():124
> at sun.nio.ch.SelectorImpl#select():141
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer#select():472
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer#produce():409
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill#produceTask():360
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill#doProduce():184
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill#tryProduce():171
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill#produce():135
> at
> org.eclipse.jetty.io.ManagedSelector$$Lambda$235.1914126144#run():(Interpreted
> code)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool#runJob():806
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner#run():938
> at java.lang.Thread#run():830
> 16.19% 3458 sun.nio.ch.EPoll#wait():(Native code)
> at sun.nio.ch.EPollSelectorImpl#doSelect():120
> at sun.nio.ch.SelectorImpl#lockAndDoSelect():124
> at sun.nio.ch.SelectorImpl#select():141
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer#select():472
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer#produce():409
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill#produceTask():360
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill#doProduce():184
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill#tryProduce():171
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill#produce():135
> at
> org.eclipse.jetty.io.ManagedSelector$$Lambda$235.1914126144#run():(Interpreted
> code)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor#lambda$execute$0():210
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$270.1779693615#run():(Interpreted
> code)
> at
> java.util.concurrent.ThreadPoolExecutor#runWorker():1128
> at
> java.util.concurrent.ThreadPoolExecutor$Worker#run():628
> at java.lang.Thread#run():830
> 13.15% 2808 sun.nio.ch.Net#accept():(Native code)
> at sun.nio.ch.ServerSocketChannelImpl#accept():276
> at
> org.eclipse.jetty.server.ServerConnector#accept():385
> at
> org.eclipse.jetty.server.AbstractConnector$Acceptor#run():701
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool#runJob():806
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner#run():938
> at java.lang.Thread#run():830
> 2.95% 630 java.math.BigInteger#implMulAdd():3191
> at java.math.BigInteger#mulAdd():3161
> at java.math.BigInteger#montReduce():3110
> at java.math.BigInteger#implMontgomerySquare():2857
> at java.math.BigInteger#montgomerySquare():2813
> at java.math.BigInteger#oddModPow():3083
> at java.math.BigInteger#modPow():2744
> at java.math.BigInteger#passesMillerRabin():1103
> at java.math.BigInteger#primeToCertainty():959
> at java.math.BitSieve#retrieve():203
> at java.math.BigInteger#largePrime():828
> at java.math.BigInteger#probablePrime():766
> at
> sun.security.rsa.RSAKeyPairGenerator#generateKeyPair():140
> at
> java.security.KeyPairGenerator$Delegate#generateKeyPair():728
> at java.security.KeyPairGenerator#genKeyPair():495
> at
> org.apache.solr.util.CryptoKeys$RSAKeyPair#<init>():342
> at
> org.apache.solr.security.PublicKeyHandler#<init>():27
> at org.apache.solr.core.CoreContainer#<init>():336
> at org.apache.solr.core.CoreContainer#<init>():326
> at
> org.apache.solr.servlet.SolrDispatchFilter#createCoreContainer():261
> 2.74% 586 java.math.BigInteger#implMulAdd():3194
> at java.math.BigInteger#mulAdd():3161
> at java.math.BigInteger#montReduce():3110
> at java.math.BigInteger#implMontgomerySquare():2857
> at java.math.BigInteger#montgomerySquare():2813
> at java.math.BigInteger#oddModPow():3083
> at java.math.BigInteger#modPow():2744
> at java.math.BigInteger#passesMillerRabin():1103
> at java.math.BigInteger#primeToCertainty():959
> at java.math.BitSieve#retrieve():203
> at java.math.BigInteger#largePrime():828
> at java.math.BigInteger#probablePrime():766
> at
> sun.security.rsa.RSAKeyPairGenerator#generateKeyPair():140
> at
> java.security.KeyPairGenerator$Delegate#generateKeyPair():728
> at java.security.KeyPairGenerator#genKeyPair():495
> at
> org.apache.solr.util.CryptoKeys$RSAKeyPair#<init>():342
> at
> org.apache.solr.security.PublicKeyHandler#<init>():27
> at org.apache.solr.core.CoreContainer#<init>():336
> at org.apache.solr.core.CoreContainer#<init>():326
> at
> org.apache.solr.servlet.SolrDispatchFilter#createCoreContainer():261
> 2.21% 472 java.math.BigInteger#implMulAdd():3191
> at java.math.BigInteger#mulAdd():3161
> at java.math.BigInteger#montReduce():3110
> at java.math.BigInteger#implMontgomerySquare():2857
> at java.math.BigInteger#montgomerySquare():2813
> at java.math.BigInteger#oddModPow():3083
> at java.math.BigInteger#modPow():2744
> at java.math.BigInteger#passesMillerRabin():1103
> at java.math.BigInteger#primeToCertainty():959
> at java.math.BitSieve#retrieve():203
> at java.math.BigInteger#largePrime():828
> at java.math.BigInteger#probablePrime():766
> at
> sun.security.rsa.RSAKeyPairGenerator#generateKeyPair():137
> at
> java.security.KeyPairGenerator$Delegate#generateKeyPair():728
> at java.security.KeyPairGenerator#genKeyPair():495
> at
> org.apache.solr.util.CryptoKeys$RSAKeyPair#<init>():342
> at
> org.apache.solr.security.PublicKeyHandler#<init>():27
> at org.apache.solr.core.CoreContainer#<init>():336
> at org.apache.solr.core.CoreContainer#<init>():326
> at
> org.apache.solr.servlet.SolrDispatchFilter#createCoreContainer():261
> 1.93% 412 java.math.BigInteger#implMulAdd():3194
> at java.math.BigInteger#mulAdd():3161
> at java.math.BigInteger#montReduce():3110
> at java.math.BigInteger#implMontgomerySquare():2857
> at java.math.BigInteger#montgomerySquare():2813
> at java.math.BigInteger#oddModPow():3083
> at java.math.BigInteger#modPow():2744
> at java.math.BigInteger#passesMillerRabin():1103
> at java.math.BigInteger#primeToCertainty():959
> at java.math.BitSieve#retrieve():203
> at java.math.BigInteger#largePrime():828
> at java.math.BigInteger#probablePrime():766
> at
> sun.security.rsa.RSAKeyPairGenerator#generateKeyPair():137
> at
> java.security.KeyPairGenerator$Delegate#generateKeyPair():728
> at java.security.KeyPairGenerator#genKeyPair():495
> at
> org.apache.solr.util.CryptoKeys$RSAKeyPair#<init>():342
> at
> org.apache.solr.security.PublicKeyHandler#<init>():27
> at org.apache.solr.core.CoreContainer#<init>():336
> at org.apache.solr.core.CoreContainer#<init>():326
> at
> org.apache.solr.servlet.SolrDispatchFilter#createCoreContainer():261
> 1.40% 299 java.math.BigInteger#implMulAdd():3194
> at java.math.BigInteger#mulAdd():3161
> at java.math.BigInteger#implSquareToLen():2185
> at java.math.BigInteger#squareToLen():2105
> at java.math.BigInteger#implMontgomerySquare():2856
> at java.math.BigInteger#montgomerySquare():2813
> at java.math.BigInteger#oddModPow():3083
> at java.math.BigInteger#modPow():2744
> at java.math.BigInteger#passesMillerRabin():1103
> at java.math.BigInteger#primeToCertainty():959
> at java.math.BitSieve#retrieve():203
> at java.math.BigInteger#largePrime():828
> at java.math.BigInteger#probablePrime():766
> at
> sun.security.rsa.RSAKeyPairGenerator#generateKeyPair():140
> at
> java.security.KeyPairGenerator$Delegate#generateKeyPair():728
> at java.security.KeyPairGenerator#genKeyPair():495
> at
> org.apache.solr.util.CryptoKeys$RSAKeyPair#<init>():342
> at
> org.apache.solr.security.PublicKeyHandler#<init>():27
> at org.apache.solr.core.CoreContainer#<init>():336
> at org.apache.solr.core.CoreContainer#<init>():326
> BUILD SUCCESSFUL in 39s
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
