itschrispeck opened a new issue, #11402: URL: https://github.com/apache/pinot/issues/11402
#### Symptoms: After a heavy query, no more values can be inserted into the mutable Lucene index for a segment. #### Reproduction: Lucene index a large amount of data, then execute a `text_match` filter that will time out or a query with a number of text_matches such that at least one will time out. Following this `failed to index value with text_index` will be logged for all values until the segment is sealed. #### Initial investigation: It appears that this a byproduct of a limitation of FSDirectories that support concurrency ([NIOFSDirectory](https://lucene.apache.org/core/8_2_0/core/org/apache/lucene/store/NIOFSDirectory.html)/[MMapDirectory](https://lucene.apache.org/core/8_2_0/core/org/apache/lucene/store/MMapDirectory.html)): >NOTE: Accessing this class either directly or indirectly from a thread while it's interrupted can close the underlying file descriptor immediately if at the same time the thread is blocked on IO. The file descriptor will remain closed and subsequent access to [NIOFSDirectory](https://lucene.apache.org/core/8_2_0/core/org/apache/lucene/store/NIOFSDirectory.html) will throw a [ClosedChannelException](https://docs.oracle.com/javase/8/docs/api/java/nio/channels/ClosedChannelException.html?is-external=true). If your application uses either [Thread.interrupt()](https://docs.oracle.com/javase/8/docs/api/java/lang/Thread.html?is-external=true#interrupt--) or [Future.cancel(boolean)](https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/Future.html?is-external=true#cancel-boolean-) you should use the legacy RAFDirectory from the Lucene misc module in favor of [NIOFSDirectory](https://lucene.apache.org/core/8_2_0/core/org/apache/lucene/store/NIOFSDirectory.html). Presumably a slow query gets canceled/timed out while thread is blocked on IO. The channel is closed preventing future events from occurring normally. I think this should affect queries against the consuming segment as well, but have not yet verified it. ClosedChannelException: ``` j.n.c.ClosedChannelException: null at sun.nio.ch.FileLockImpl.release(FileLockImpl.java:58) at java.nio.channels.FileLock.close(FileLock.java:317) at o.a.l.s.NativeFSLockFactory$NativeFSLock.close(NativeFSLockFactory.java:206) at org.apache.lucene.util.IOUtils.close(IOUtils.java:89) at org.apache.lucene.util.IOUtils.close(IOUtils.java:77) at o.a.lucene.index.IndexWriter.rollbackInternalNoCommit(IndexWriter.java:2337) at o.a.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2268) at o.a.lucene.index.IndexWriter.maybeCloseOnTragicEvent(IndexWriter.java:4910) at o.a.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1610) at o.a.lucene.index.IndexWriter.addDocument(IndexWriter.java:1213) at o.a.p.s.l.s.c.i.t.LuceneTextIndexCreator.add(LuceneTextIndexCreator.java:136) <...> ``` Insert failure: ``` java.lang.RuntimeException: Caught exception while adding a new document to the Lucene index for column: col_name at org.apache.pinot.segment.local.segment.creator.impl.text.LuceneTextIndexCreator.add(LuceneTextIndexCreator.java:138) at org.apache.pinot.segment.local.realtime.impl.invertedindex.RealtimeLuceneTextIndex.add(RealtimeLuceneTextIndex.java:93) at org.apache.pinot.segment.spi.index.mutable.MutableTextIndex.add(MutableTextIndex.java:29) at org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl.addNewRow(MutableSegmentImpl.java:694) at org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl.index(MutableSegmentImpl.java:502) <...> Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:681) at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:695) at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1591) at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1213) at org.apache.pinot.segment.local.segment.creator.impl.text.LuceneTextIndexCreator.add(LuceneTextIndexCreator.java:136) ... 8 common frames omitted ``` Refresh failure: ``` Caused by: org.apache.lucene.store.AlreadyClosedException: FileLock invalidated by an external force: NativeFSLock(path=/<...>/dataDir/table_name_REALTIME/consumers/table_name__11__15__20230811T1226Z/col_name.lucene.index/write.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive invalid],creationTime=2023-08-11T12:26:13.731708Z) at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:178) at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:43) at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43) at org.apache.lucene.codecs.lucene80.Lucene80NormsConsumer.<init>(Lucene80NormsConsumer.java:45) at org.apache.lucene.codecs.lucene80.Lucene80NormsFormat.normsConsumer(Lucene80NormsFormat.java:85) at org.apache.lucene.index.DefaultIndexingChain.writeNorms(DefaultIndexingChain.java:304) at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:131) at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:468) at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:555) at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:722) at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:494) at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:297) at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:272) at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:262) at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:165) at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:156) at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58) at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176) at org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:225) at org.apache.pinot.segment.local.realtime.impl.invertedindex.RealtimeLuceneIndexReaderRefreshThread.run(RealtimeLuceneIndexReaderRefreshThread.java:126) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
