costin commented on PR #16356: URL: https://github.com/apache/lucene/pull/16356#issuecomment-5120208949
Good point. CharArrayMap doesn't support removal so proper LRU isn't possible without a custom data structure. Instead I changed the cache to clear-and-refill when full so it adapts to the current distribution rather than being stuck with early entries. Kept the 128 default. Also made allocation lazy so the cache is only created when the first token is actually stemmed, so analyzers used for query parsing pay zero memory. Updated benchmark results: | language | cache | stop=true | × | stop=false | × | |----------|------:|----------------:|-----:|----------------:|-----:| | Romanian | 0 | 323 ± 2 | — | 266 ± 2 | — | | Romanian | 128 | 466 ± 3 | 1.4× | 352 ± 3 | 1.3× | | Romanian | 256 | 546 ± 4 | 1.7× | 442 ± 5 | 1.7× | | Turkish | 0 | 186 ± 1 | — | 115 ± 0 | — | | Turkish | 128 | 269 ± 1 | 1.4× | 184 ± 1 | 1.6× | | Turkish | 256 | 276 ± 3 | 1.5× | 230 ± 1 | 2.0× | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
