rmuir commented on PR #16371:
URL: https://github.com/apache/lucene/pull/16371#issuecomment-5053592144
> Uses a flat char[65536] lookup table (128 KB) lazily initialized on first
use (once per JVM), built from ~1200 {codepoint, folded} pairs (which get
discarded after initialization) generated from ICU4J via ./gradlew
:lucene:core:generateFoldTable
I think we've gone thru this before with the ASCIIFoldingFilter, and the
most efficient way is to just tableize basic ascii (first 128) and do a switch
statement for the remainder. You dont need to build explicit table, just return
toLowerCase() for value < 128. its already using two-stage tables behind the
scenes.
Otherwise you blow away too much cache with the big lookup table. It might
look good in a microbenchmark but not in an overall indexer run. Plus the data
is pretty sparse (most of the codepoints dont have case), using a big lookup
table is overkill.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
