costin commented on PR #16356: URL: https://github.com/apache/lucene/pull/16356#issuecomment-5109181166
You're right, English and German were the wrong languages to benchmark. I've changed this to Romanian and Turkish (which has more complicated stemming rules). From what I can tell, 18 default language analyzers use SnowballFilter without a dedicated, lighter alternative. I reran the benchmark with the two languages and used different cache sizes for better tuning: (AMD EPYC, JDK 25, vocab=5000) | language | cache | stop=true | × | stop=false | × | |----------|------:|----------------:|-----:|----------------:|-----:| | Romanian | 0 | 328 ± 2 | — | 255 ± 2 | — | | Romanian | 128 | 563 ± 4 | 1.7× | 467 ± 3 | 1.8× | | Romanian | 256 | 618 ± 4 | 1.9× | 546 ± 5 | 2.1× | | Turkish | 0 | 178 ± 1 | — | 115 ± 0 | — | | Turkish | 128 | 356 ± 1 | 2.0× | 262 ± 1 | 2.3× | | Turkish | 256 | 430 ± 3 | 2.4× | 324 ± 1 | 2.8× | the sweet spot looks to be between 128-256. 128 entries take around ~9KB (capped) for ~2x better stemming so made that the default. > if they can load the stoplist and protect it from being stemmed, they will see a perf > improvement, but with much less memory usage Agreed this works and it's a good complementary technique. The cache gives ~2x regardless of whether stop filter is in the pipeline, so it helps even for analyzers that already remove stop words. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
