itschrispeck opened a new issue, #11402:
URL: https://github.com/apache/pinot/issues/11402

   #### Symptoms:
   After a heavy query, no more values can be inserted into the mutable Lucene 
index for a segment. 
   
   #### Reproduction:
   Lucene index a large amount of data, then execute a `text_match` filter that 
will time out or a query with a number of text_matches such that at least one 
will time out. Following this `failed to index value with text_index` will be 
logged for all values until the segment is sealed. 
   
   #### Initial investigation:
   It appears that this a byproduct of a limitation of FSDirectories that 
support concurrency 
([NIOFSDirectory](https://lucene.apache.org/core/8_2_0/core/org/apache/lucene/store/NIOFSDirectory.html)/[MMapDirectory](https://lucene.apache.org/core/8_2_0/core/org/apache/lucene/store/MMapDirectory.html)):
   >NOTE: Accessing this class either directly or indirectly from a thread 
while it's interrupted can close the underlying file descriptor immediately if 
at the same time the thread is blocked on IO. The file descriptor will remain 
closed and subsequent access to 
[NIOFSDirectory](https://lucene.apache.org/core/8_2_0/core/org/apache/lucene/store/NIOFSDirectory.html)
 will throw a 
[ClosedChannelException](https://docs.oracle.com/javase/8/docs/api/java/nio/channels/ClosedChannelException.html?is-external=true).
 If your application uses either 
[Thread.interrupt()](https://docs.oracle.com/javase/8/docs/api/java/lang/Thread.html?is-external=true#interrupt--)
 or 
[Future.cancel(boolean)](https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/Future.html?is-external=true#cancel-boolean-)
 you should use the legacy RAFDirectory from the Lucene misc module in favor of 
[NIOFSDirectory](https://lucene.apache.org/core/8_2_0/core/org/apache/lucene/store/NIOFSDirectory.html).
   
   Presumably a slow query gets canceled/timed out while thread is blocked on 
IO.  The channel is closed preventing future events from occurring normally. 
   
   I think this should affect queries against the consuming segment as well, 
but have not yet verified it.
   
   ClosedChannelException:
   ```
   j.n.c.ClosedChannelException: null at 
sun.nio.ch.FileLockImpl.release(FileLockImpl.java:58) 
           at java.nio.channels.FileLock.close(FileLock.java:317) 
           at 
o.a.l.s.NativeFSLockFactory$NativeFSLock.close(NativeFSLockFactory.java:206) 
           at org.apache.lucene.util.IOUtils.close(IOUtils.java:89) at 
org.apache.lucene.util.IOUtils.close(IOUtils.java:77) 
           at 
o.a.lucene.index.IndexWriter.rollbackInternalNoCommit(IndexWriter.java:2337) 
           at 
o.a.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2268) 
           at 
o.a.lucene.index.IndexWriter.maybeCloseOnTragicEvent(IndexWriter.java:4910) 
           at 
o.a.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1610) 
           at o.a.lucene.index.IndexWriter.addDocument(IndexWriter.java:1213) 
           at 
o.a.p.s.l.s.c.i.t.LuceneTextIndexCreator.add(LuceneTextIndexCreator.java:136) 
<...> 
   ```
   
   Insert failure:
   ```
   java.lang.RuntimeException: Caught exception while adding a new document to 
the Lucene index for column: col_name
           at 
org.apache.pinot.segment.local.segment.creator.impl.text.LuceneTextIndexCreator.add(LuceneTextIndexCreator.java:138)
           at 
org.apache.pinot.segment.local.realtime.impl.invertedindex.RealtimeLuceneTextIndex.add(RealtimeLuceneTextIndex.java:93)
           at 
org.apache.pinot.segment.spi.index.mutable.MutableTextIndex.add(MutableTextIndex.java:29)
           at 
org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl.addNewRow(MutableSegmentImpl.java:694)
           at 
org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl.index(MutableSegmentImpl.java:502)
   <...>
   Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter 
is closed
           at 
org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:681)
           at 
org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:695)
           at 
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1591)
           at 
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1213)
           at 
org.apache.pinot.segment.local.segment.creator.impl.text.LuceneTextIndexCreator.add(LuceneTextIndexCreator.java:136)
           ... 8 common frames omitted
   ```
   
   Refresh failure:
   ```
   Caused by: org.apache.lucene.store.AlreadyClosedException: FileLock 
invalidated by an external force: 
NativeFSLock(path=/<...>/dataDir/table_name_REALTIME/consumers/table_name__11__15__20230811T1226Z/col_name.lucene.index/write.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807
 exclusive invalid],creationTime=2023-08-11T12:26:13.731708Z)
           at 
org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:178)
           at 
org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:43)
           at 
org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)
           at 
org.apache.lucene.codecs.lucene80.Lucene80NormsConsumer.<init>(Lucene80NormsConsumer.java:45)
           at 
org.apache.lucene.codecs.lucene80.Lucene80NormsFormat.normsConsumer(Lucene80NormsFormat.java:85)
           at 
org.apache.lucene.index.DefaultIndexingChain.writeNorms(DefaultIndexingChain.java:304)
           at 
org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:131)
           at 
org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:468)
           at 
org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:555)
           at 
org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:722)
           at 
org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:494)
           at 
org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:297)
           at 
org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:272)
           at 
org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:262)
           at 
org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:165)
           at 
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:156)
           at 
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58)
           at 
org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176)
           at 
org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:225)
           at 
org.apache.pinot.segment.local.realtime.impl.invertedindex.RealtimeLuceneIndexReaderRefreshThread.run(RealtimeLuceneIndexReaderRefreshThread.java:126)
   ```
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to