[ https://issues.apache.org/jira/browse/LUCENE-8776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16825202#comment-16825202 ]
Michael Gibney commented on LUCENE-8776: ---------------------------------------- Ram, it's good that this solution worked for you, but taking a step back, your solution seems like a workaround for LUCENE-7398 and LUCENE-4312. Workarounds aren't inherently _bad_ of course, but when they depend on ambiguity of (or lack of enforcement of) contracts, backward compatibility can't be guaranteed (to paraphrase what I take Robert and Adrien to be saying). Of course, one person's "patch" is another person's "workaround", but I'd be curious to know whether any of the ["LUCENE-7398/*" branches|https://github.com/magibney/lucene-solr/tree/LUCENE-7398/branch_7_6] might help for your use case. (There's a high-level description in this comment on the LUCENE-7398 issue). Particularly relevant to this discussion: the patch supports recording token positionLength in the index, and enforces index ordering by startPosition and endPosition (compatible with ordering specified for the Spans API). > Start offset going backwards has a legitimate purpose > ----------------------------------------------------- > > Key: LUCENE-8776 > URL: https://issues.apache.org/jira/browse/LUCENE-8776 > Project: Lucene - Core > Issue Type: Bug > Components: core/search > Affects Versions: 7.6 > Reporter: Ram Venkat > Priority: Major > > Here is the use case where startOffset can go backwards: > Say there is a line "Organic light-emitting-diode glows", and I want to run > span queries and highlight them properly. > During index time, light-emitting-diode is split into three words, which > allows me to search for 'light', 'emitting' and 'diode' individually. The > three words occupy adjacent positions in the index, as 'light' adjacent to > 'emitting' and 'light' at a distance of two words from 'diode' need to match > this word. So, the order of words after splitting are: Organic, light, > emitting, diode, glows. > But, I also want to search for 'organic' being adjacent to > 'light-emitting-diode' or 'light-emitting-diode' being adjacent to 'glows'. > The way I solved this was to also generate 'light-emitting-diode' at two > positions: (a) In the same position as 'light' and (b) in the same position > as 'glows', like below: > ||organic||light||emitting||diode||glows|| > | |light-emitting-diode| |light-emitting-diode| | > |0|1|2|3|4| > The positions of the two 'light-emitting-diode' are 1 and 3, but the offsets > are obviously the same. This works beautifully in Lucene 5.x in both > searching and highlighting with span queries. > But when I try this in Lucene 7.6, it hits the condition "Offsets must not go > backwards" at DefaultIndexingChain:818. This IllegalArgumentException is > being thrown without any comments on why this check is needed. As I explained > above, startOffset going backwards is perfectly valid, to deal with word > splitting and span operations on these specialized use cases. On the other > hand, it is not clear what value is added by this check and which highlighter > code is affected by offsets going backwards. This same check is done at > BaseTokenStreamTestCase:245. > I see others talk about how this check found bugs in WordDelimiter etc. but > it also prevents legitimate use cases. Can this check be removed? -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org