[ https://issues.apache.org/jira/browse/SOLR-822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Koji Sekiguchi reopened SOLR-822: --------------------------------- Reopening because currentPosition() method in CharFilter class is not for token position, but for token offset. It should be renamed before releasing Solr 1.4. > CharFilter - normalize characters before tokenizer > -------------------------------------------------- > > Key: SOLR-822 > URL: https://issues.apache.org/jira/browse/SOLR-822 > Project: Solr > Issue Type: New Feature > Components: Analysis > Affects Versions: 1.3 > Reporter: Koji Sekiguchi > Assignee: Koji Sekiguchi > Priority: Minor > Fix For: 1.4 > > Attachments: character-normalization.JPG, sample_mapping_ja.txt, > sample_mapping_ja.txt, SOLR-822-for-1.3.patch, SOLR-822.patch, > SOLR-822.patch, SOLR-822.patch, SOLR-822.patch, SOLR-822.patch > > > A new plugin which can be placed in front of <tokenizer/>. > {code:xml} > <fieldType name="textCharNorm" class="solr.TextField" > positionIncrementGap="100" > > <analyzer> > <charFilter class="solr.MappingCharFilterFactory" > mapping="mapping_ja.txt" /> > <tokenizer class="solr.MappingCJKTokenizerFactory"/> > <filter class="solr.StopFilterFactory" ignoreCase="true" > words="stopwords.txt"/> > <filter class="solr.LowerCaseFilterFactory"/> > </analyzer> > </fieldType> > {code} > <charFilter/> can be multiple (chained). I'll post a JPEG file to show > character normalization sample soon. > MOTIVATION: > In Japan, there are two types of tokenizers -- N-gram (CJKTokenizer) and > Morphological Analyzer. > When we use morphological analyzer, because the analyzer uses Japanese > dictionary to detect terms, > we need to normalize characters before tokenization. > I'll post a patch soon, too. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.