neoremind commented on issue #11608: URL: https://github.com/apache/lucene/issues/11608#issuecomment-5268478766
I agree with the mental model that **`BytesRefHash` should be generic enough to balance multiple workloads, and being adaptive as much as possible**. I do think there are still chances worth exploring like I mentioned above, 1) splitting the term bytes pool, one pool for short terms, one for long, 2) separating the posting chain from the term bytes pool. Both address the memory-wall issue we see in the flamegraphs and benchmarks. Besides, some other improvement directions in my mind: 3) better hash algorithm, exploring alternatives to `murmurhash3_x86_32`, current avg 3.3 steps if probing per wikipedia benchmark, is it best-in-class? see below screenshot for example, ideally they should be pretty spread-out; and performance optimization of the current `murmurhash3_x86_32`, 4) Robin Hood hashing that reduces worst-case probe lengths, 5) borrow good parts from Swiss hash table. <img width="336" height="648" alt="Image" src="https://github.com/user-attachments/assets/2e659cad-41b6-4d71-a729-0f7e6f1d9bf6"; /> As for the optimization attempts I've already tried recently: 6) 3-byte inline into `bytesStart[]` offset, 7) load factor of 0.75. I am holding those back as they are not mature enough to align the mental model. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
