onestardao commented on issue #15054:
URL: https://github.com/apache/lucene/issues/15054#issuecomment-3182112262
Thanks for the clear report — tracking this as #15054.
From your description/logs, the exhaustion appears only after switching the
open calls to IOContext.DEFAULT for remote store. That strongly suggests a
lifecycle / caching leak tied to thread-local or map-based input caches:
In several Lucene paths, DEFAULT enables buffered/random access and may
hit helpers that cache IndexInput/buffers per thread (e.g., via
CloseableThreadLocal or a Map keyed by thread/file).
Because your open and close happen on different threads in remote store,
those per-thread entries may never get removed, leading to an unbounded map
growth and the “exhaustion of maps” you’re seeing.
Quick triage you can try:
Verify same-thread close: ensure every IndexInput opened under DEFAULT
is closed on the same thread (wrap in try-with-resources and log
Thread.currentThread() on open/close).
Heap histo to find the owner: capture a short heap dump when growth
starts; look for big ConcurrentHashMap / CloseableThreadLocal instances under
*IndexInputProvider* / store layer to confirm what’s retaining entries.
Use READONCE where sequential is sufficient: for bulk/streamed reads
(e.g., copy/restore paths) prefer READONCE, reserving DEFAULT only for random
access code paths.
Temporary guard: if you do need DEFAULT, add an explicit cleanup hook in
the remote store layer to remove per-thread cache entries on close (same
thread), or avoid thread-keyed caches on that path.
Repro test: a small test that opens via thread A and closes via thread B
under load should reproduce the leak if that’s the cause.
If you can share a minimal repro (or heap histo dominant types / the map
owner class), it’ll help pinpoint which cache needs a fix in Lucene vs. in the
remote store integration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
