[ https://issues.apache.org/jira/browse/SOLR-211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12491852 ]
Ken Krugler commented on SOLR-211: ---------------------------------- I think we must be working on similar types of projects :) I did something similar to the above, but in two different ways: # I extended WhitespaceTokenizerFactory to take optional pattern & replacement parameters. If these exist, then I apply them before the tokenizer gets called. This lets me do something like strip out all XML fields other than the content of the one that I want to index from a bunch of XML going into a Solr field. # I added a CSVTokenizerFactory, which takes an optional split character and an optional remapping file. This lets me get a field like "Java,Python,C#" and turn it into "java python csharp", which are the index tokens I need, while leaving the display text as-is. I don't know if your new PatternTokenizerFactory could replace either of these, though. For the first case, I still want the white space tokenization after I've stripped off all the junk I don't want. And for the second, I need to be able to do the remapping. > regex split() Tokenizer > ----------------------- > > Key: SOLR-211 > URL: https://issues.apache.org/jira/browse/SOLR-211 > Project: Solr > Issue Type: New Feature > Components: search > Reporter: Ryan McKinley > Assigned To: Ryan McKinley > Attachments: SOLR-211-RegexSplitTokenizer.patch, > SOLR-211-RegexSplitTokenizer.patch, SOLR-211-RegexSplitTokenizer.patch > > > A TokenizerFactory that makes tokens from: > string.split( regex ); -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.