[
https://issues.apache.org/jira/browse/LUCENE-9508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231109#comment-17231109
]
Zach Chen commented on LUCENE-9508:
-----------------------------------
Hi [~simonw], just to clarify I wasn't the one who originally reported the
issue. I just poked around in code and run some tests to see if I can help here.
> DocumentsWriter doesn't check for BlockedFlushes in stall mode``
> ----------------------------------------------------------------
>
> Key: LUCENE-9508
> URL: https://issues.apache.org/jira/browse/LUCENE-9508
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/index
> Affects Versions: 8.5.1
> Reporter: Sorabh Hamirwasia
> Priority: Major
> Labels: IndexWriter
>
> Hi,
> I was investigating an issue where the memory usage by a single Lucene
> IndexWriter went up to ~23GB. Lucene has a concept of stalling in case the
> memory used by each index breaches the 2 X ramBuffer limit (10% of JVM heap,
> this case ~3GB). So ideally memory usage should not go above that limit. I
> looked into the heap dump and found that the fullFlush thread when enters
> *markForFullFlush* method, it tries to take lock on the ThreadStates of all
> the DWPT thread sequentially. If lock on one of the ThreadState is blocked
> then it will block indefinitely. This is what happened in my case, where one
> of the DWPT thread was stuck in indexing process. Due to this fullFlush
> thread was unable to populate the flush queue even though the stall mode was
> detected. This caused the new indexing request which came on indexing thread
> to continue after sleeping for a second, and continue with indexing. In
> **preUpdate()** method it looks for the stalled case and see if there is any
> pending flushes (based on flush queue), if not then sleep and continue.
> Question:
> 1) Should **preUpdate** look into the blocked flushes information as well
> instead of just flush queue ?
> 2) Should the fullFlush thread wait indefinitely for the lock on ThreadStates
> ? Since single blocking writing thread can block the full flush here.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
