Could you expand on your example and show the output you want? FWIW, you could
simply write a token filter that does the same thing as the WhitespaceTokenizer.
-Grant
On Dec 3, 2010, at 1:14 PM, Matthew Hall wrote:
> Hey folks, I'm working with a fairly specific set of requirements for our
> corpus that needs a somewhat tricky text type for both indexing and searching.
>
> The chain currently looks like this:
>
> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> <filter class="solr.PatternReplaceFilterFactory"
> pattern="(.*?)(\p{Punct}*)$"
> replacement="$1"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.StopFilterFactory"
> ignoreCase="true"
> words="stopwords.txt"
> enablePositionIncrements="true"
> />
> <filter class="solr.SnowballPorterFilterFactory" language="English"
> protected="protwords.txt"/>
> <filter class="solr.PatternReplaceFilterFactory"
> pattern="\p{Punct}"
> replacement=" "/>
> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>
> Now you will notice that I'm trying to add in a second tokenizer to this
> chain at the very end, this is due to the final replacement of punctuation to
> whitespace. At that point I'd like to further break up these tokens to
> smaller tokens.
>
> The reason for this is that we have a mixed normal english word and
> scientific corpus. For example you could expect string like "The symposium
> of Tg<The>(RX3fg+and) gene studies" being added to the index, and parts of
> those phrases being searched on.
>
> We want to be able to remove the stopwords in the mostly english parts of
> these types of statements, which the whitespace tokenizer, followed by
> removing trailing punctuation, followed by the stopfilter takes care of. We
> do not want to remove references to genetic information contained in allele
> symbols and the like.
>
> Sadly as far as I can tell, you cannot chain tokenizers in the schema.xml, so
> does anyone have some suggestions on how this could be accomplished?
>
> Oh, and let me add that the WordDelimiterFilter comes really close to what I
> want, but since we are unwilling to promote our solr version to the trunk (we
> are on the 1.4x) version atm, the inability to turn off the automatic phrase
> queries makes it a no go. We need to be able to make searches on
> "left/right" match "right/left."
>
> My searches through the old material on this subject isn't really showing me
> much except some advice on using the copyField attribute. But my
> understanding is that this will simply take your original input to the field,
> and then analyze it in two different ways depending on the field definitions.
> It would be very nice if it were copying the already analyzed version of the
> text... but that's not what its doing, right?
>
> Thanks for any advice on this matter.
>
> Matt
>
>
--------------------------
Grant Ingersoll
http://www.lucidimagination.com
