Arjen,
An approach requiring less list maintenance could be more advanced
linguistic processing to distinguish the stop word from the content word,
such as lemmatization rather than stemming.
A commercial offering, Rosette Search Essentials from Basis
<http://www.basistech.com/search-essentials/> (full disclosure: my
employer), which is free for development use and can be downloaded via that
link, uses textual context to disambiguate lemmas as in the screenshot
below -- compare the lemma for token #13 (van) v. token #25 (vans). (I
don't read/write Dutch; I took these snippets from the web.) The work
integrating OpenNLP <https://issues.apache.org/jira/browse/LUCENE-2899>
might also prove helpful.
Best,
David Murgatroyd
ww.linkedin.com/in/dmurga/ <http://www.linkedin.com/in/dmurga/>
[image: Inline image 1]
On Mon, Jul 7, 2014 at 5:53 PM, Sujit Pal wrote:
> Hi Arjen,
>
> You could also mark a token as "keyword" so the stemmer passes it through
> unchanged. For example, per the Javadocs for PorterStemFilter:
>
> http://lucene.apache.org/core/4_6_0/analyzers-common/org/apache/lucene/analysis/en/PorterStemFilter.html
>
> Note: This filter is aware of the KeywordAttribute
> <
> http://lucene.apache.org/core/4_6_0/core/org/apache/lucene/analysis/tokenattributes/KeywordAttribute.html?is-external=true
> >.
> To prevent certain terms from being passed to the stemmer
> KeywordAttribute.isKeyword()
> <
> http://lucene.apache.org/core/4_6_0/core/org/apache/lucene/analysis/tokenattributes/KeywordAttribute.html?is-external=true#isKeyword()
> >
> should
> be set to true in a previousTokenStream
> <
> http://lucene.apache.org/core/4_6_0/core/org/apache/lucene/analysis/TokenStream.html?is-external=true
> >.
> Note: For including the original term as well as the stemmed version, see
> KeywordRepeatFilterFactory
> <
> http://lucene.apache.org/core/4_6_0/analyzers-common/org/apache/lucene/analysis/miscellaneous/KeywordRepeatFilterFactory.html
> >
>
> Assuming your stemmer is also keyword attribute aware, you could build a
> filter that reads a list of words (such as "vans") that should be protected
> from stemming and marks them with the KeywordAttribute before sending to
> the Porter stemmer and put it into your analysis chain.
>
> -sujit
>
>
> On Mon, Jul 7, 2014 at 2:06 PM, Tri Cao wrote:
>
> > I think emitting two tokens for "vans" is the right (potentially only)
> way
> > to do it. You could
> > also control the dictionary of terms that require this special treatment.
> >
> > Any reason makes you not happy with this approach?
> >
> > On Jul 06, 2014, at 11:48 AM, Arjen van der Meijden <
> > acmmail...@tweakers.net> wrote:
> >
> > Hello list,
> >
> > We have a fairly large Lucene database for a 30+ million post forum.
> > Users post and search for all kinds of things. To make sure users don't
> > have to type exact matches, we combine a WordDelimiterFilter with a
> > (Dutch) SnowballFilter.
> >
> > Unfortunately users sometimes find examples of words that get stemmed to
> > a word that's basically a stop word. Or reversely, where a very common
> > word is stemmed so that it becomes the same as a rare word.
> >
> > We do index stop words, so theoretically they could still find their
> > result. But when a rare word is stemmed in such a way it yields a
> > million hits, that makes it very unusable...
> >
> > One example is the Dutch word 'van' which is the equivalent of 'of' in
> > English. A user tried to search for the shoe brand 'vans', which gets
> > stemmed to 'van' and obviously gives useless results.
> >
> > I already noticed the 'KeywordRepeatFilter' to index/search both 'vans'
> > and 'van' and the StemmerOverrideFilter to try and prevent these cases.
> > Are there any other solutions for these kinds of problems?
> >
> > Best regards,
> >
> > Arjen van der Meijden
> >
> > -
> > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
> > For additional commands, e-mail: java-user-h...@lucene.apache.org
> >
> >
>
