neoremind commented on code in PR #16499:
URL: https://github.com/apache/lucene/pull/16499#discussion_r3764879491
##########
lucene/core/src/java/org/apache/lucene/util/BytesRefHash.java:
##########
@@ -428,7 +418,7 @@ public int addByPoolOffset(int offset) {
}
/**
- * Called when hash is too small ({@code > 50%} occupied) or too large
({@code < 20%} occupied).
+ * Called when hash is too small ({@code > 50%} occupied) or too large
({@code < 25%} occupied).
Review Comment:
Yes, `shrink()` is only called inside `clear()`. And `clear()` is only
triggered when `storeTermVectors` is enabled on a field. My read is that the
term vectors
[path](https://github.com/apache/lucene/blob/d0ba760f5f08b087f5e83ccefd0f5c771ed0487f/lucene/core/src/java/org/apache/lucene/index/TermVectorsConsumer.java#L119)
has a per-document lifecycle, a dedicated `BytesRefHash` grows and shrinks as
each document is indexed. This is unlike regular inverted index path for doc
id, freq, and prox where `clear()` is never called, when segment flushes, we
recreate the indexing needed data structures from scratch for the next segment,
so the previous `BytesRefHash` just gets GCed away.
I debugged this locally. For example, if the previous doc's
term-vector-enabled field contains 40 unique terms, the `ids[]` grows to 128.
Then `clear()` is called, no size change since 40 is between 25%-50% of 128.
But if the next doc's term-vector-enabled field only has 14 terms, the `ids[]`
shrinks to 32 on that doc's `clear()` call to keep `BytesRefHash` sizing
best-fit to the current doc's term count, always between 25%-50%. I think such
always-on-resizing does create many short-lived `int[]` that may press GC,
maybe a separate performance optimization thread is needed.
<img width="1329" height="618" alt="Screenshot 2026-08-12 at 11 33 52 AM"
src="https://github.com/user-attachments/assets/836baec4-b944-4e0b-8e54-803ea7415dd7";
/>
Revised the comment. To be precise, `rehash()` is called when count reaches
50% of hashSize, never goes over.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
