Andras Salamon created LUCENE-9927: -------------------------------------- Summary: Configurable BKDWriter maximum heap size Key: LUCENE-9927 URL: https://issues.apache.org/jira/browse/LUCENE-9927 Project: Lucene - Core Issue Type: Improvement Reporter: Andras Salamon
With Lucene's default codec, when writing dimensional points, we only give {{BKDWriter}} 16 MB heap to use for sorting. If we reach this limit the codec will change to [OfflinePointWriter|https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/util/bkd/OfflinePointWriter.java]. This is specially bad for Solr versions not containing SOLR-14242, because we get the following error: {noformat} Caused by: java.lang.IllegalStateException: this writer hit an unrecoverable error; cannot complete forceMerge at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1998) at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1940) at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:662) at org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:102) at org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:68) at org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:68) at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:1079) at org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1066) at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:160) at org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:69) at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:72) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211) ... 15 more Caused by: java.lang.UnsupportedOperationException at org.apache.solr.store.hdfs.HdfsDirectory.createTempOutput(HdfsDirectory.java:119) at org.apache.lucene.store.FilterDirectory.createTempOutput(FilterDirectory.java:79) at org.apache.lucene.store.FilterDirectory.createTempOutput(FilterDirectory.java:79) at org.apache.lucene.store.TrackingDirectoryWrapper.createTempOutput(TrackingDirectoryWrapper.java:51) at org.apache.lucene.store.TrackingDirectoryWrapper.createTempOutput(TrackingDirectoryWrapper.java:51) at org.apache.lucene.util.bkd.OfflinePointWriter.<init>(OfflinePointWriter.java:45) at org.apache.lucene.util.bkd.BKDWriter.initPointWriter(BKDWriter.java:223) at org.apache.lucene.util.bkd.BKDWriter.add(BKDWriter.java:241) at org.apache.lucene.codecs.lucene60.Lucene60PointsWriter$1.visit(Lucene60PointsWriter.java:123) at org.apache.lucene.codecs.PointsWriter$1$1$1.visit(PointsWriter.java:118) at org.apache.lucene.util.bkd.BKDReader.visitCompressedDocValues(BKDReader.java:735) at org.apache.lucene.util.bkd.BKDReader.visitDocValuesWithCardinality(BKDReader.java:684) at org.apache.lucene.util.bkd.BKDReader.visitDocValues(BKDReader.java:590) at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:790) at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:813) at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:813) at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:813) at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:813) at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:813) at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:813) at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:813) at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:813) at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:506) at org.apache.lucene.codecs.PointsWriter$1$1.intersect(PointsWriter.java:106) at org.apache.lucene.codecs.lucene60.Lucene60PointsWriter.writeField(Lucene60PointsWriter.java:116) at org.apache.lucene.codecs.PointsWriter.mergeOneField(PointsWriter.java:63) at org.apache.lucene.codecs.lucene60.Lucene60PointsWriter.merge(Lucene60PointsWriter.java:227) at org.apache.lucene.index.SegmentMerger.mergePoints(SegmentMerger.java:206) at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:164) at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4467) at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4059) at org.apache.solr.update.SolrIndexWriter.merge(SolrIndexWriter.java:201) at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625) at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662){noformat} It would be great to make this 16MB value configurable. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org