: There's also one other problem with those sets: Unfortunately they are
: modifiable, because they are not real "Set<String>" but CharArraySets. There
: is no 100% unmodifiable view of them. This was the main reason why we did not
: make them public for newer variants of analyzers. I think we should add
: unmodifable "views" of CharArraySet, but this is also not 100% possible, as
: the underlying char[] cannot be protected.
Interesting.
That makes it seem like Lucene should probably deprecate (and remove in
10) the 40+ existing "public static CharArraySet getDefaultSOMETHING()"
methods that are documented like this...
/**
* Returns an unmodifiable instance of the default stop words set.
*
* @return default stop words set.
*/
public static CharArraySet getDefaultStopSet() {
...but return direct access to a private static (not so) "unmodifiable"
CharArraySet, and replace them with methods like...
/**
* Creates an instance of the default stop words set.
*
* @return default stop words set.
*/
public static CharArraySet buildDefaultStopSet() {
...and invert the current (simplified) DEFAULT/getter logic from...
private static final CharArraySet DEFAULT;
static {
DEFAULT = WordlistLoader.getSnowballWordSet(...)
}
public static CharArraySet getDefaultStopSet() {
return HOLDER.DEFAULT
}
...to something like...
private static final CharArraySet DEFAULT = buildDefaultStopSet();
public static CharArraySet buildDefaultStopSet() {
return WordlistLoader.getSnowballWordSet(...);
}
... the net result for the Stock Analyzers would be the same, but that
would eliminate the risk that anyone calling buildDefaultStopSet() from
their custom Analyzer would break the stock Analyzer. right?
: > I am fine with this. But on the other hand: Why do you want to replicate the
: > files into Solr's config folder? A Solr configuration should better be able
: > to load the stopwords file from resources, too. I was always wondering why
1) We include copies of these files in the _default configset for the same
reason we copy the _default configset when you create a collection: To
help ensure backcompatiblity of behavior even if the _default configset
(or resource files) change in a future version of Solr (or Lucene). As
long as you continue to use your "old" copy of your configset, you should
(hopefully) get the consistent analysis behavior from your fieldtypes.
The "new" _default configset may have a "better" set of stopwords, or
stemming dictionary (or completley new list of token filters) for the
"new" field type definition with the same names as the field type you are
using, but it's your choice if/when to modify your schema to get that new
behavior.
2) It's been a while since i personally tested it, but AFAIK the
SolrResourceLoader will still happily look for a resource file in the
classpath *after* it checks the configset -- if the configset did not
contain a file with that path. So it should certainly be
possible for someone to put something like this in their schema...
<filter name="stop" ignoreCase="true"
words="org/apache/lucene/analysis/snowball/irish_stop.txt"/>
...but ignoring the backcompat reasosn mentioned above, telling a Solr
user they can point a <filter/> directly at a file path from a LUcene jar
doesn't help most of the situations I was asking about (plus a
few more), because in these cases the stock Lucene
Analyzers don't build the CharArraySet/CharArrayMap from a resource
file in the classpath. They are build from hardcoded java ararys in the
sourcecode.
If examples like the ones below were moved to resource files (and the FQN
of the "default" resource file was available via a "public static final
String") that would also certainly be an improvement in terms of reusability in
custom subclasses (and Solr's ability to copy them on upgrade)...
// IrishAnalyzer
private static final CharArraySet DEFAULT_ARTICLES =
CharArraySet.unmodifiableSet(new CharArraySet(Arrays.asList("d", "m", "b"),
true));
private static final CharArraySet HYPHENATIONS =
CharArraySet.unmodifiableSet(new CharArraySet(Arrays.asList("h", "n", "t"),
true));
// DutchAnalyzer
DEFAULT_STEM_DICT = new CharArrayMap<>(4, false);
DEFAULT_STEM_DICT.put("fiets", "fiets"); // otherwise fiet
DEFAULT_STEM_DICT.put("bromfiets", "bromfiets"); // otherwise bromfiet
DEFAULT_STEM_DICT.put("ei", "eier");
DEFAULT_STEM_DICT.put("kind", "kinder");
// EnglishAnalyzer
final List<String> stopWords =
Arrays.asList(
"a", "an", "and", "are", "as", "at", "be", "but", "by", "for", "if", "in",
"into", "is",
"it", "no", "not", "of", "on", "or", "such", "that", "the", "their",
"then", "there",
"these", "they", "this", "to", "was", "will", "with");
final CharArraySet stopSet = new CharArraySet(stopWords, false);
// FrenchAnalyzer, ItalianAnalyzer, CatalanAnalyzer, etc...
-Hoss
http://www.lucidworks.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
