rmuir commented on PR #16356: URL: https://github.com/apache/lucene/pull/16356#issuecomment-5107210594
> Regarding the generic wrapping approach, I opted for improving the performance of the existing filters so that users can benefit from it after the upgrade without having to change their setup/mappings. I think this is the concern though. 75KB or N strings is too big by default. analyzer is cached in threadlocal and not just used at indexing time, but also at query time. For users with high query volumes, this cache would just cause problems I think. users have already complained before about the fact we reuse the termbuffer (termattribute), which is much less memory than this. Yes, the snowball stemmers are slower but most of our Analyzers don't default to them either... instead they default to lighter and faster approaches. For example, with English even if you dont want a light/minimal approach, you can use the PorterStemmer.java which is much faster than the snowball variant. That's the default used by EnglishAnalyzer. https://github.com/apache/lucene/blob/main/lucene/analysis/common/src/java/org/apache/lucene/analysis/en/EnglishAnalyzer.java#L108 For German, the default is also not snowball, but Savoy's: https://github.com/apache/lucene/blob/main/lucene/analysis/common/src/java/org/apache/lucene/analysis/de/GermanAnalyzer.java#L133 So I'm not sure who the users are that are using English/German snowball filters? If the user doesn't want to use stopfilter, again, if they can load the stoplist and protect it from being stemmed, they will see a perf improvement, but with much less memory usage and the set doesn't need to be duplicated everywhere but loaded a single time. Anyway, these are just my thoughts. For both these languages I think we have faster solutions already, and the faster solutions are already the default? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
