: Should we keep the HOLDER.DEFAULT pattern to not create the default stop : set if not needed (when there is a custom building)?
I did not mean to imply that i think we eliminate the HOLDER pattern/optimization -- i just didn't include it in my "(simplified)" example to try and focus on the my core question: wether we should invert the current "public get() returns DEFAULT" relationship to a "DEFAULT initialized using public build()" relationship. : > : There's also one other problem with those sets: Unfortunately they are : > : modifiable, because they are not real "Set<String>" but CharArraySets. : > There : > : is no 100% unmodifiable view of them. This was the main reason why we : > did not : > : make them public for newer variants of analyzers. I think we should add : > : unmodifable "views" of CharArraySet, but this is also not 100% possible, : > as : > : the underlying char[] cannot be protected. : > : > Interesting. : > : > That makes it seem like Lucene should probably deprecate (and remove in : > 10) the 40+ existing "public static CharArraySet getDefaultSOMETHING()" : > methods that are documented like this... : > : > /** : > * Returns an unmodifiable instance of the default stop words set. : > * : > * @return default stop words set. : > */ : > public static CharArraySet getDefaultStopSet() { : > : > ...but return direct access to a private static (not so) "unmodifiable" : > CharArraySet, and replace them with methods like... : > : > /** : > * Creates an instance of the default stop words set. : > * : > * @return default stop words set. : > */ : > public static CharArraySet buildDefaultStopSet() { : > : > ...and invert the current (simplified) DEFAULT/getter logic from... : > : > private static final CharArraySet DEFAULT; : > static { : > DEFAULT = WordlistLoader.getSnowballWordSet(...) : > } : > public static CharArraySet getDefaultStopSet() { : > return HOLDER.DEFAULT : > } : > : > ...to something like... : > : > private static final CharArraySet DEFAULT = buildDefaultStopSet(); : > public static CharArraySet buildDefaultStopSet() { : > return WordlistLoader.getSnowballWordSet(...); : > } : > : > ... the net result for the Stock Analyzers would be the same, but that : > would eliminate the risk that anyone calling buildDefaultStopSet() from : > their custom Analyzer would break the stock Analyzer. right? : > : > : > : > I am fine with this. But on the other hand: Why do you want to : > replicate the : > : > files into Solr's config folder? A Solr configuration should better be : > able : > : > to load the stopwords file from resources, too. I was always wondering : > why : > : > 1) We include copies of these files in the _default configset for the same : > reason we copy the _default configset when you create a collection: To : > help ensure backcompatiblity of behavior even if the _default configset : > (or resource files) change in a future version of Solr (or Lucene). As : > long as you continue to use your "old" copy of your configset, you should : > (hopefully) get the consistent analysis behavior from your fieldtypes. : > The "new" _default configset may have a "better" set of stopwords, or : > stemming dictionary (or completley new list of token filters) for the : > "new" field type definition with the same names as the field type you are : > using, but it's your choice if/when to modify your schema to get that new : > behavior. : > : > : > 2) It's been a while since i personally tested it, but AFAIK the : > SolrResourceLoader will still happily look for a resource file in the : > classpath *after* it checks the configset -- if the configset did not : > contain a file with that path. So it should certainly be : > possible for someone to put something like this in their schema... : > : > <filter name="stop" ignoreCase="true" : > words="org/apache/lucene/analysis/snowball/irish_stop.txt"/> : > : > ...but ignoring the backcompat reasosn mentioned above, telling a Solr : > user they can point a <filter/> directly at a file path from a LUcene jar : > doesn't help most of the situations I was asking about (plus a : > few more), because in these cases the stock Lucene : > Analyzers don't build the CharArraySet/CharArrayMap from a resource : > file in the classpath. They are build from hardcoded java ararys in the : > sourcecode. : > : > If examples like the ones below were moved to resource files (and the FQN : > of the "default" resource file was available via a "public static final : > String") that would also certainly be an improvement in terms of : > reusability in : > custom subclasses (and Solr's ability to copy them on upgrade)... : > : > : > // IrishAnalyzer : > private static final CharArraySet DEFAULT_ARTICLES = : > CharArraySet.unmodifiableSet(new CharArraySet(Arrays.asList("d", "m", : > "b"), true)); : > private static final CharArraySet HYPHENATIONS = : > CharArraySet.unmodifiableSet(new CharArraySet(Arrays.asList("h", "n", : > "t"), true)); : > : > // DutchAnalyzer : > DEFAULT_STEM_DICT = new CharArrayMap<>(4, false); : > DEFAULT_STEM_DICT.put("fiets", "fiets"); // otherwise fiet : > DEFAULT_STEM_DICT.put("bromfiets", "bromfiets"); // otherwise bromfiet : > DEFAULT_STEM_DICT.put("ei", "eier"); : > DEFAULT_STEM_DICT.put("kind", "kinder"); : > : > // EnglishAnalyzer : > final List<String> stopWords = : > Arrays.asList( : > "a", "an", "and", "are", "as", "at", "be", "but", "by", "for", "if", : > "in", "into", "is", : > "it", "no", "not", "of", "on", "or", "such", "that", "the", "their", : > "then", "there", : > "these", "they", "this", "to", "was", "will", "with"); : > final CharArraySet stopSet = new CharArraySet(stopWords, false); : > : > // FrenchAnalyzer, ItalianAnalyzer, CatalanAnalyzer, etc... : > : > : > : > -Hoss : > http://www.lucidworks.com/ : > : > --------------------------------------------------------------------- : > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org : > For additional commands, e-mail: dev-h...@lucene.apache.org : > : > : -Hoss http://www.lucidworks.com/ --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org