[ https://issues.apache.org/jira/browse/SOLR-2400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002491#comment-13002491 ]
Uwe Schindler edited comment on SOLR-2400 at 3/4/11 8:44 AM: ------------------------------------------------------------- Stefan, this is a general issue of TokenStreams adding Tokens. TokenStreams that remove Tokens *should* automatically preserve position, but not even all of those do that correctly (we were fixing some of them lately). The way of how the Lucene analysis works makes it impossible to guarantee any corresponence of the position numbers. Because for the indexer it's only important what comes out at the end, the steps inbetween are not interesting. AnalysisReqHandler on the other hand does some bad "hacks" to look "inside" the analysis (by using temporary TokenStreams that buffer tokens), which are not the general use-case of TokenStreams. I wonder a little bit about your xml file, it only contains text and position, but it should also contain rawTerm, startOffset, endOffset. When I call analysis i get all of those attributes not only two of them. Is this a hand-made file or what is the problem? Which Solr version? One possibility to handle the thing might be the char offset in the original text, because that the req handler may use the character offset of begin and end of the token in the original stream instead of the token position, but this is likely to break for lots of TokenFilters (WordDelimiterFilter would work as long as you don't do stemming before...). The problem is incorrect handling of offset calculation (also leading to bugs in highlighting) when the inserted terms are longer than their originals. Alltogether: Its unlikely that you can implement that and it will work for all combinations of TokenStream components. was (Author: thetaphi): Stefan, this is an egeneral issue of TokenStreams adding Tokens. TokenStreams that remove Tokens *should* automatically preserve position, but not even all of those do that correctly (we were fixing some of them lately). The way of how the Lucene analysis works makes it impossible to guarantee any corresponence of the position numbers. Because for the indexer its only important what comes out at the end, the steps inbetween are impossible. AnalysisReqHandler on the other hand does some bad "hacks" to look "inside" the analysis (by using temporary TokenStreams that buffer tokens), which are not the general use-case of TokenStreams. I wonder a little bit about your xml file, it only contains text and position, but it should also contain rawTerm, startOffset, endOffset. When I call analysis i get all of those attributes not only two of them. Is this a hand-made file or what is the problem? Which Solr version? One possibility to handle the thing might be the char offset in the original text, because that one should point to the character offset of begin and end of the token in the original stream instead of the token position, but this is likely to break for lots of TokenFilters (WordDelimiterFilter would work as long as you don't do stemming before...). The problem is incorrect handling of offset calculation (also leading to bugs in highlighting) when the inserted terms are longer than their originals. Alltogether: Its unlikely that you can implement that and it will work for all combinations of TokenStream components. > FieldAnalysisRequestHandler; add information about token-relation > ----------------------------------------------------------------- > > Key: SOLR-2400 > URL: https://issues.apache.org/jira/browse/SOLR-2400 > Project: Solr > Issue Type: Improvement > Components: Schema and Analysis > Reporter: Stefan Matheis (steffkes) > Priority: Minor > Attachments: 110303_FieldAnalysisRequestHandler_output.xml, > 110303_FieldAnalysisRequestHandler_view.png > > > The XML-Output (simplified example attached) is missing one small information > .. which could be very useful to build an nice Analysis-Output, and that's > "Token-Relation" (if there is special/correct word for this, please correct > me). > Meaning, that is actually not possible to "follow" the Analysis-Process > (completly) while the Tokenizers/Filters will drop out Tokens (f.e. StopWord) > or split it into multiple Tokens (f.e. WordDelimiter). > Would it be possible to include this Information? If so, it would be possible > to create an improved Analysis-Page for the new Solr Admin (SOLR-2399) - > short scribble attached -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org