The data set I'm experimenting with is ~100G. If I break it up into four 25G data sets and index is individually, it works just fine. But if I try to index the entire 100G it breaks. I'm not sure if there is a size limitation or sth (FYI, I have a really large heap of size 256G).
Thanks, On Thu, Dec 27, 2018 at 12:27 AM Dawid Weiss <[email protected]> wrote: > This looks like a bug, the underlying cause is: > > Caused by: java.lang.IllegalArgumentException: cannot write negative > vLong (got: -4294878395) > at > org.apache.lucene.store.DataOutput.writeVLong(DataOutput.java:225) > at > org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.writeSkipData(Lucene50SkipWriter.java:180) > > Can you reproduce it on a smaller example and file a Jira issue? > > Dawid > > On Thu, Dec 27, 2018 at 7:18 AM John Wilson <[email protected]> > wrote: > > > > Hi, > > > > I'm getting the below error message while running a simple > multi-threaded indexing using ByteBuffersDirectory. Any suggestions? > > > > Exception in thread "Lucene Merge Thread #879" > org.apache.lucene.index.MergePolicy$MergeException: > org.apache.lucene.store.AlreadyClosedException: refusing to delete any > files: this IndexWriter hit an unrecoverable exception > > at > org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:705) > > at > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:685) > > Caused by: org.apache.lucene.store.AlreadyClosedException: refusing to > delete any files: this IndexWriter hit an unrecoverable exception > > at > org.apache.lucene.index.IndexFileDeleter.ensureOpen(IndexFileDeleter.java:349) > > at > org.apache.lucene.index.IndexFileDeleter.deleteFiles(IndexFileDeleter.java:669) > > at > org.apache.lucene.index.IndexFileDeleter.deleteNewFiles(IndexFileDeleter.java:664) > > at > org.apache.lucene.index.IndexWriter.deleteNewFiles(IndexWriter.java:5024) > > at > org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4539) > > at > org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4075) > > at > org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:626) > > at > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:663) > > Caused by: java.lang.IllegalArgumentException: cannot write negative > vLong (got: -4294878395) > > at > org.apache.lucene.store.DataOutput.writeVLong(DataOutput.java:225) > > at > org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.writeSkipData(Lucene50SkipWriter.java:180) > > at > org.apache.lucene.codecs.MultiLevelSkipListWriter.bufferSkip(MultiLevelSkipListWriter.java:143) > > at > org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.bufferSkip(Lucene50SkipWriter.java:162) > > at > org.apache.lucene.codecs.lucene50.Lucene50PostingsWriter.startDoc(Lucene50PostingsWriter.java:228) > > at > org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:148) > > at > org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:865) > > at > org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:344) > > at > org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105) > > at > org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:169) > > at > org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:244) > > at > org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:139) > > at > org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4453) > > ... 3 more > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
