[
https://issues.apache.org/jira/browse/LUCENE-9189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17025303#comment-17025303
]
Robert Muir commented on LUCENE-9189:
-------------------------------------
OK, I see the issue. it also "tracks" (by track we mean, recomputes by calling
listAll and then summing fileLength of every file... on every writeByte etc)
the disk usage if you setMaxSizeInBytes.
So it only impacts these disk full tests. The tracking should get more
efficient, but the scope is limited and I don't want to wrestle with this logic
right now. Going with Nightly until we fix the efficiency of this thing.
> TestIndexWriterDelete.testDeletesOnDiskFull can run for minutes
> ---------------------------------------------------------------
>
> Key: LUCENE-9189
> URL: https://issues.apache.org/jira/browse/LUCENE-9189
> Project: Lucene - Core
> Issue Type: Task
> Reporter: Robert Muir
> Priority: Major
>
> I thought it was just the testUpdatesOnDiskFull, but looks like this one
> needs to be nightly too.
> Should look more into the test, but I know something causes it to make such
> an insane amount of files, that sorting them becomes a bottleneck.
> I guess also related is that it would be great if MockDirectoryWrapper's disk
> full check didn't trigger a sort of the files (via listAll): it does this
> check on like every i/o, would be nice for it to be less absurd. Maybe
> instead the test could check for disk full on not every i/o but some random
> sample of them?
> Temporarily lets make it nightly...
> {noformat}
> PROFILE SUMMARY from 182501 samples
> tests.profile.count=10
> tests.profile.stacksize=1
> tests.profile.linenumbers=false
> PERCENT SAMPLES STACK
> 15.89% 28995 java.lang.StringLatin1#compareTo()
> 6.61% 12069 java.util.TimSort#mergeHi()
> 5.96% 10878 java.util.TimSort#binarySort()
> 3.41% 6231 java.util.concurrent.ConcurrentHashMap#tabAt()
> 2.98% 5433 java.util.Comparators$NaturalOrderComparator#compare()
> 2.12% 3876 org.apache.lucene.store.DataOutput#copyBytes()
> 2.03% 3712 java.lang.String#compareTo()
> 1.84% 3350 java.util.concurrent.ConcurrentHashMap#get()
> 1.83% 3337 java.util.TimSort#mergeLo()
> 1.67% 3047 java.util.ArrayList#add()
> {noformat}
> All the file sorting is called from stacks like this, so its literally
> happening every writeByte() and so on
> {noformat}
> 0.73% 1329 java.util.TimSort#binarySort()
> at java.util.TimSort#sort()
> at java.util.Arrays#sort()
> at java.util.ArrayList#sort()
> at java.util.stream.SortedOps$RefSortingSink#end()
> at java.util.stream.AbstractPipeline#copyInto()
> at java.util.stream.AbstractPipeline#wrapAndCopyInto()
> at java.util.stream.AbstractPipeline#evaluate()
> at
> java.util.stream.AbstractPipeline#evaluateToArrayNode()
> at java.util.stream.ReferencePipeline#toArray()
> at
> org.apache.lucene.store.ByteBuffersDirectory#listAll()
> at
> org.apache.lucene.store.MockDirectoryWrapper#sizeInBytes()
> at
> org.apache.lucene.store.MockIndexOutputWrapper#checkDiskFull()
> at
> org.apache.lucene.store.MockIndexOutputWrapper#writeBytes()
> at
> org.apache.lucene.store.MockIndexOutputWrapper#writeByte()
> at org.apache.lucene.store.DataOutput#writeInt()
> at org.apache.lucene.codecs.CodecUtil#writeFooter()
> at
> org.apache.lucene.codecs.lucene50.Lucene50LiveDocsFormat#writeLiveDocs()
> at
> org.apache.lucene.codecs.asserting.AssertingLiveDocsFormat#writeLiveDocs()
> at
> org.apache.lucene.index.PendingDeletes#writeLiveDocs()
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
