[ https://issues.apache.org/jira/browse/SOLR-1826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12848271#action_12848271 ]
Sanjoy Ghosh commented on SOLR-1826: ------------------------------------ Hi, I investigated this some more. The problem seems to be in: org.apache.lucene.search.highlight.TokenSource.java public static TokenStream getTokenStream(TermPositionVector tpv, boolean tokenPositionsGuaranteedContiguous) { has at the end the following code to sort the tokens into original document order. Arrays.sort(tokensInOriginalOrder, new Comparator(){ public int compare(Object o1, Object o2) { Token t1=(Token) o1; Token t2=(Token) o2; if(t1.startOffset()>t2.endOffset()) return 1; if(t1.startOffset()<t2.startOffset()) return -1; return 0; }}); This is not sorting the tokens into the right original order. The order should be lorem, power, powershotcom, shot, com, ipsum for this to work correctly. Instead we are getting lorem, power, com, powershotcom, shot, ipsum which confuses TokenGroup.isDistinct(). I would be happy to fix this bug. Should we fix this as a Lucene bug or fix it in Solr by creating a new TokenStream that handles overlapping tokens correctly. > highlighting breaks when using WordDelimiterFilter and setting > termOffsets=true > ------------------------------------------------------------------------------- > > Key: SOLR-1826 > URL: https://issues.apache.org/jira/browse/SOLR-1826 > Project: Solr > Issue Type: Bug > Components: highlighter > Affects Versions: 1.4 > Reporter: Stefan Oestreicher > Attachments: SOLR-1826.txt, SOLR-1826.txt, SOLR-1826.txt > > > When using the WordDelimiterFilter and setting termOffsets to true the > highlighting breaks in certain cases. This did not happen in the 1.3 release. > For example, if I index the term "PowerShot.com" and search for {{pow*}} the > highlighting snippet contains {{<em>Power</em><em>PowerShot.com</em>}}. > I will attach a patch which adds tests to the highlighter unittest to > demonstrate the issue. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.