Toke Eskildsen created LUCENE-8628: -------------------------------------- Summary: ByteBuffersDirectory merge exception Key: LUCENE-8628 URL: https://issues.apache.org/jira/browse/LUCENE-8628 Project: Lucene - Core Issue Type: Bug Components: core/store Affects Versions: 7.5 Reporter: Toke Eskildsen
An exception during merge for {{ByteBufferDirectory}} was [reported on the lucene-dev mailinglist|https://mail-archives.apache.org/mod_mbox/lucene-dev/201812.mbox/raw/%3CCAL%3D08JvGioNvfy8xKbb7%2B999YnNnU4iQv6F6ktRkudrkr-6jGg%40mail.gmail.com%3E]: {{Exception in thread "Lucene Merge Thread #879" org.apache.lucene.index.MergePolicy$MergeException: org.apache.lucene.store.AlreadyClosedException: refusing to delete any files: this IndexWriter hit an unrecoverable exception}}{{ at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:705)}}{{ at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:685)}}{{Caused by: org.apache.lucene.store.AlreadyClosedException: refusing to delete any files: this IndexWriter hit an unrecoverable exception}}{{ at org.apache.lucene.index.IndexFileDeleter.ensureOpen(IndexFileDeleter.java:349)}}{{ at org.apache.lucene.index.IndexFileDeleter.deleteFiles(IndexFileDeleter.java:669)}}{{ at org.apache.lucene.index.IndexFileDeleter.deleteNewFiles(IndexFileDeleter.java:664)}}{{ at org.apache.lucene.index.IndexWriter.deleteNewFiles(IndexWriter.java:5024)}}{{ at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4539)}}{{ at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4075)}}{{ at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:626)}}{{ at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:663)}}{{Caused by: java.lang.IllegalArgumentException: cannot write negative vLong (got: -4294878395)}}{{ at org.apache.lucene.store.DataOutput.writeVLong(DataOutput.java:225)}}{{ at org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.writeSkipData(Lucene50SkipWriter.java:180)}}{{ at org.apache.lucene.codecs.MultiLevelSkipListWriter.bufferSkip(MultiLevelSkipListWriter.java:143)}}{{ at org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.bufferSkip(Lucene50SkipWriter.java:162)}}{{ at org.apache.lucene.codecs.lucene50.Lucene50PostingsWriter.startDoc(Lucene50PostingsWriter.java:228)}}{{ at org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:148)}}{{ at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:865)}}{{ at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:344)}}{{ at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)}}{{ at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:169)}}{{ at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:244)}}{{ at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:139)}}{{ at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4453)}}{{ ... 3 more}} Following the trace shows no apparent problems with the {{Lucene50SkipWripter}}. However, it uses {{getFilePointer}} to resolve the skip lengths, which for {{ByteBuffersDirectory}} ends in {{ByteBuffersDataOutput.size()}} with the code {{ public long size() {}} {{ long size = 0;}} {{ int blockCount = blocks.size();}} {{ if (blockCount >= 1) {}} {{ int fullBlockSize = (blockCount - 1) * blockSize();}} {{ int lastBlockSize = blocks.getLast().position();}} {{ size = fullBlockSize + lastBlockSize;}} {{ }}} {{ return size;}} {{ }}} One problem here is that {{int fullBlockSize = (blockCount - 1) * blockSize();}} can overflow. This should be changed to something like {{long fullBlockSize = (blockCount - 1) * (long)blockSize();}} -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org