[jira] Commented: (LUCENE-1448) add getFinalOffset() to TokenStream
[ https://issues.apache.org/jira/browse/LUCENE-1448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12660074#action_12660074 ] Michael McCandless commented on LUCENE-1448: See also LUCENE-579 which looks like a dup of this one. Michael what's the game plan on this issue? I think your EOSA approach makes sense... > add getFinalOffset() to TokenStream > --- > > Key: LUCENE-1448 > URL: https://issues.apache.org/jira/browse/LUCENE-1448 > Project: Lucene - Java > Issue Type: Bug > Components: Analysis >Reporter: Michael McCandless >Assignee: Michael Busch >Priority: Minor > Fix For: 2.9 > > Attachments: LUCENE-1448.patch, LUCENE-1448.patch, LUCENE-1448.patch, > LUCENE-1448.patch > > > If you add multiple Fieldable instances for the same field name to a > document, and you then index those fields with TermVectors storing offsets, > it's very likely the offsets for all but the first field instance will be > wrong. > This is because IndexWriter under the hood adds a cumulative base to the > offsets of each field instance, where that base is 1 + the endOffset of the > last token it saw when analyzing that field. > But this logic is overly simplistic. For example, if the WhitespaceAnalyzer > is being used, and the text being analyzed ended in 3 whitespace characters, > then that information is lost and then next field's offsets are then all 3 > too small. Similarly, if a StopFilter appears in the chain, and the last N > tokens were stop words, then the base will be 1 + the endOffset of the last > non-stopword token. > To fix this, I'd like to add a new getFinalOffset() to TokenStream. I'm > thinking by default it returns -1, which means "I don't know so you figure it > out", meaning we fallback to the faulty logic we have today. > This has come up several times on the user's list. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-1448) add getFinalOffset() to TokenStream
[ https://issues.apache.org/jira/browse/LUCENE-1448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12660128#action_12660128 ] Michael Busch commented on LUCENE-1448: --- I'm currently on vacation visiting my family in Germany till the 11th. I'm planning to work on this as soon as I'm back to get all the TokenStream changes (also LUCENE-1460) ready before 2.9. > add getFinalOffset() to TokenStream > --- > > Key: LUCENE-1448 > URL: https://issues.apache.org/jira/browse/LUCENE-1448 > Project: Lucene - Java > Issue Type: Bug > Components: Analysis >Reporter: Michael McCandless >Assignee: Michael Busch >Priority: Minor > Fix For: 2.9 > > Attachments: LUCENE-1448.patch, LUCENE-1448.patch, LUCENE-1448.patch, > LUCENE-1448.patch > > > If you add multiple Fieldable instances for the same field name to a > document, and you then index those fields with TermVectors storing offsets, > it's very likely the offsets for all but the first field instance will be > wrong. > This is because IndexWriter under the hood adds a cumulative base to the > offsets of each field instance, where that base is 1 + the endOffset of the > last token it saw when analyzing that field. > But this logic is overly simplistic. For example, if the WhitespaceAnalyzer > is being used, and the text being analyzed ended in 3 whitespace characters, > then that information is lost and then next field's offsets are then all 3 > too small. Similarly, if a StopFilter appears in the chain, and the last N > tokens were stop words, then the base will be 1 + the endOffset of the last > non-stopword token. > To fix this, I'd like to add a new getFinalOffset() to TokenStream. I'm > thinking by default it returns -1, which means "I don't know so you figure it > out", meaning we fallback to the faulty logic we have today. > This has come up several times on the user's list. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-1448) add getFinalOffset() to TokenStream
[ https://issues.apache.org/jira/browse/LUCENE-1448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12718176#action_12718176 ] Michael McCandless commented on LUCENE-1448: Michael are you going to get to this soonish? Else let's push until after 3.0? > add getFinalOffset() to TokenStream > --- > > Key: LUCENE-1448 > URL: https://issues.apache.org/jira/browse/LUCENE-1448 > Project: Lucene - Java > Issue Type: Bug > Components: Analysis >Reporter: Michael McCandless >Assignee: Michael Busch >Priority: Minor > Fix For: 2.9 > > Attachments: LUCENE-1448.patch, LUCENE-1448.patch, LUCENE-1448.patch, > LUCENE-1448.patch > > > If you add multiple Fieldable instances for the same field name to a > document, and you then index those fields with TermVectors storing offsets, > it's very likely the offsets for all but the first field instance will be > wrong. > This is because IndexWriter under the hood adds a cumulative base to the > offsets of each field instance, where that base is 1 + the endOffset of the > last token it saw when analyzing that field. > But this logic is overly simplistic. For example, if the WhitespaceAnalyzer > is being used, and the text being analyzed ended in 3 whitespace characters, > then that information is lost and then next field's offsets are then all 3 > too small. Similarly, if a StopFilter appears in the chain, and the last N > tokens were stop words, then the base will be 1 + the endOffset of the last > non-stopword token. > To fix this, I'd like to add a new getFinalOffset() to TokenStream. I'm > thinking by default it returns -1, which means "I don't know so you figure it > out", meaning we fallback to the faulty logic we have today. > This has come up several times on the user's list. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-1448) add getFinalOffset() to TokenStream
[ https://issues.apache.org/jira/browse/LUCENE-1448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12723591#action_12723591 ] Mark Miller commented on LUCENE-1448: - Will you have time for this Michael? It would be great to have this bug fixed for 2.9, but if we have to push to 3.0 its not the end of the word. Wasnt it done till that darn new Token API came along? ;) > add getFinalOffset() to TokenStream > --- > > Key: LUCENE-1448 > URL: https://issues.apache.org/jira/browse/LUCENE-1448 > Project: Lucene - Java > Issue Type: Bug > Components: Analysis >Reporter: Michael McCandless >Assignee: Michael Busch >Priority: Minor > Fix For: 2.9 > > Attachments: LUCENE-1448.patch, LUCENE-1448.patch, LUCENE-1448.patch, > LUCENE-1448.patch > > > If you add multiple Fieldable instances for the same field name to a > document, and you then index those fields with TermVectors storing offsets, > it's very likely the offsets for all but the first field instance will be > wrong. > This is because IndexWriter under the hood adds a cumulative base to the > offsets of each field instance, where that base is 1 + the endOffset of the > last token it saw when analyzing that field. > But this logic is overly simplistic. For example, if the WhitespaceAnalyzer > is being used, and the text being analyzed ended in 3 whitespace characters, > then that information is lost and then next field's offsets are then all 3 > too small. Similarly, if a StopFilter appears in the chain, and the last N > tokens were stop words, then the base will be 1 + the endOffset of the last > non-stopword token. > To fix this, I'd like to add a new getFinalOffset() to TokenStream. I'm > thinking by default it returns -1, which means "I don't know so you figure it > out", meaning we fallback to the faulty logic we have today. > This has come up several times on the user's list. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-1448) add getFinalOffset() to TokenStream
[ https://issues.apache.org/jira/browse/LUCENE-1448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12723764#action_12723764 ] Michael Busch commented on LUCENE-1448: --- Oh man, what did I do suggesting you as the RM?!? Now there's another guy chasing me! ;) Currently I have to sacrifice some of my already very limited sleep for everything I do on Lucene. After next week I'll have more time. When everything else for 2.9 is done, then I don't think this should block it. Otherwise, I'll try to do it for 2.9. > add getFinalOffset() to TokenStream > --- > > Key: LUCENE-1448 > URL: https://issues.apache.org/jira/browse/LUCENE-1448 > Project: Lucene - Java > Issue Type: Bug > Components: Analysis >Reporter: Michael McCandless >Assignee: Michael Busch >Priority: Minor > Fix For: 2.9 > > Attachments: LUCENE-1448.patch, LUCENE-1448.patch, LUCENE-1448.patch, > LUCENE-1448.patch > > > If you add multiple Fieldable instances for the same field name to a > document, and you then index those fields with TermVectors storing offsets, > it's very likely the offsets for all but the first field instance will be > wrong. > This is because IndexWriter under the hood adds a cumulative base to the > offsets of each field instance, where that base is 1 + the endOffset of the > last token it saw when analyzing that field. > But this logic is overly simplistic. For example, if the WhitespaceAnalyzer > is being used, and the text being analyzed ended in 3 whitespace characters, > then that information is lost and then next field's offsets are then all 3 > too small. Similarly, if a StopFilter appears in the chain, and the last N > tokens were stop words, then the base will be 1 + the endOffset of the last > non-stopword token. > To fix this, I'd like to add a new getFinalOffset() to TokenStream. I'm > thinking by default it returns -1, which means "I don't know so you figure it > out", meaning we fallback to the faulty logic we have today. > This has come up several times on the user's list. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-1448) add getFinalOffset() to TokenStream
[ https://issues.apache.org/jira/browse/LUCENE-1448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12646563#action_12646563 ] Mark Miller commented on LUCENE-1448: - bq. Second: I can't figure out how to ask StandardTokenizerImpl for the number of chars it pulled from the Reader. Can someone (who understands JFlex well) help out here? Whats wrong with? public int getFinalOffset() { return scanner.yychar() + scanner.yylength(); } > add getFinalOffset() to TokenStream > --- > > Key: LUCENE-1448 > URL: https://issues.apache.org/jira/browse/LUCENE-1448 > Project: Lucene - Java > Issue Type: Bug > Components: Analysis >Reporter: Michael McCandless >Assignee: Michael McCandless >Priority: Minor > Fix For: 2.9 > > Attachments: LUCENE-1448.patch > > > If you add multiple Fieldable instances for the same field name to a > document, and you then index those fields with TermVectors storing offsets, > it's very likely the offsets for all but the first field instance will be > wrong. > This is because IndexWriter under the hood adds a cumulative base to the > offsets of each field instance, where that base is 1 + the endOffset of the > last token it saw when analyzing that field. > But this logic is overly simplistic. For example, if the WhitespaceAnalyzer > is being used, and the text being analyzed ended in 3 whitespace characters, > then that information is lost and then next field's offsets are then all 3 > too small. Similarly, if a StopFilter appears in the chain, and the last N > tokens were stop words, then the base will be 1 + the endOffset of the last > non-stopword token. > To fix this, I'd like to add a new getFinalOffset() to TokenStream. I'm > thinking by default it returns -1, which means "I don't know so you figure it > out", meaning we fallback to the faulty logic we have today. > This has come up several times on the user's list. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-1448) add getFinalOffset() to TokenStream
[ https://issues.apache.org/jira/browse/LUCENE-1448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12646549#action_12646549 ] Mark Miller commented on LUCENE-1448: - You need that +1 or you will have the subsequent token starting on the tail of the 'stopword'. What I can't figure it out is how exactly these offsets are supposed to match up...abcd has offsets of s:0 e:4, which seems to imply it thinks abcd is 5 chars or the end is one greater than the end index (like with spans). In either case, it seems even if you put back the +1, the endoffsets are off somehow, because some will have an end of +1 the end index, while secondary multi-fields will have an end equal to the end index. Would be cool to have fixed as this also stymies highlighting with multi-fields. > add getFinalOffset() to TokenStream > --- > > Key: LUCENE-1448 > URL: https://issues.apache.org/jira/browse/LUCENE-1448 > Project: Lucene - Java > Issue Type: Bug > Components: Analysis >Reporter: Michael McCandless >Assignee: Michael McCandless >Priority: Minor > Fix For: 2.9 > > Attachments: LUCENE-1448.patch > > > If you add multiple Fieldable instances for the same field name to a > document, and you then index those fields with TermVectors storing offsets, > it's very likely the offsets for all but the first field instance will be > wrong. > This is because IndexWriter under the hood adds a cumulative base to the > offsets of each field instance, where that base is 1 + the endOffset of the > last token it saw when analyzing that field. > But this logic is overly simplistic. For example, if the WhitespaceAnalyzer > is being used, and the text being analyzed ended in 3 whitespace characters, > then that information is lost and then next field's offsets are then all 3 > too small. Similarly, if a StopFilter appears in the chain, and the last N > tokens were stop words, then the base will be 1 + the endOffset of the last > non-stopword token. > To fix this, I'd like to add a new getFinalOffset() to TokenStream. I'm > thinking by default it returns -1, which means "I don't know so you figure it > out", meaning we fallback to the faulty logic we have today. > This has come up several times on the user's list. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-1448) add getFinalOffset() to TokenStream
[ https://issues.apache.org/jira/browse/LUCENE-1448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12648080#action_12648080 ] Michael Busch commented on LUCENE-1448: --- {quote} First: this patch only addresses the final offset, but shouldn't we also address the final position? Eg if StopFilter removes the last few tokens, shouldn't we make it possible to report those skipped positonIncrements? {quote} Hmm now that we have getPositionIncrementGap() and getOffsetGap(), I think it would make sense to also add getFinalPositionIncrement()? {quote} To fix this, I'd like to add a new getFinalOffset() to TokenStream. {quote} Could we add this as Attributes using the new API? FinalOffsetAttribute and FinalPositionIncrementAttribute? > add getFinalOffset() to TokenStream > --- > > Key: LUCENE-1448 > URL: https://issues.apache.org/jira/browse/LUCENE-1448 > Project: Lucene - Java > Issue Type: Bug > Components: Analysis >Reporter: Michael McCandless >Assignee: Michael McCandless >Priority: Minor > Fix For: 2.9 > > Attachments: LUCENE-1448.patch, LUCENE-1448.patch, LUCENE-1448.patch, > LUCENE-1448.patch > > > If you add multiple Fieldable instances for the same field name to a > document, and you then index those fields with TermVectors storing offsets, > it's very likely the offsets for all but the first field instance will be > wrong. > This is because IndexWriter under the hood adds a cumulative base to the > offsets of each field instance, where that base is 1 + the endOffset of the > last token it saw when analyzing that field. > But this logic is overly simplistic. For example, if the WhitespaceAnalyzer > is being used, and the text being analyzed ended in 3 whitespace characters, > then that information is lost and then next field's offsets are then all 3 > too small. Similarly, if a StopFilter appears in the chain, and the last N > tokens were stop words, then the base will be 1 + the endOffset of the last > non-stopword token. > To fix this, I'd like to add a new getFinalOffset() to TokenStream. I'm > thinking by default it returns -1, which means "I don't know so you figure it > out", meaning we fallback to the faulty logic we have today. > This has come up several times on the user's list. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-1448) add getFinalOffset() to TokenStream
[ https://issues.apache.org/jira/browse/LUCENE-1448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12648122#action_12648122 ] Michael McCandless commented on LUCENE-1448: bq. Hmm now that we have getPositionIncrementGap() and getOffsetGap(), I think it would make sense to also add getFinalPositionIncrement()? We could do that. But how would you implement it? EG StopFilter skips tokens, and (if enabled) already tracks the skippedPositions, so it could return that PLUS whatever its input reports as its getFinalPositionIncrement, I guess? bq. Could we add this as Attributes using the new API? FinalOffsetAttribute and FinalPositionIncrementAttribute? Hmm we could do that... but it seems awkward to add new attributes that apply only to ending state of the tokenizer. I wonder if instead, w/ the new API, we could simply allow querying of certain attributes (offset, posincr) after incrementToken returns "false"? Why don't you commit the new TokenStream API first, and we can iterate on this issue & commit 2nd? > add getFinalOffset() to TokenStream > --- > > Key: LUCENE-1448 > URL: https://issues.apache.org/jira/browse/LUCENE-1448 > Project: Lucene - Java > Issue Type: Bug > Components: Analysis >Reporter: Michael McCandless >Assignee: Michael McCandless >Priority: Minor > Fix For: 2.9 > > Attachments: LUCENE-1448.patch, LUCENE-1448.patch, LUCENE-1448.patch, > LUCENE-1448.patch > > > If you add multiple Fieldable instances for the same field name to a > document, and you then index those fields with TermVectors storing offsets, > it's very likely the offsets for all but the first field instance will be > wrong. > This is because IndexWriter under the hood adds a cumulative base to the > offsets of each field instance, where that base is 1 + the endOffset of the > last token it saw when analyzing that field. > But this logic is overly simplistic. For example, if the WhitespaceAnalyzer > is being used, and the text being analyzed ended in 3 whitespace characters, > then that information is lost and then next field's offsets are then all 3 > too small. Similarly, if a StopFilter appears in the chain, and the last N > tokens were stop words, then the base will be 1 + the endOffset of the last > non-stopword token. > To fix this, I'd like to add a new getFinalOffset() to TokenStream. I'm > thinking by default it returns -1, which means "I don't know so you figure it > out", meaning we fallback to the faulty logic we have today. > This has come up several times on the user's list. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-1448) add getFinalOffset() to TokenStream
[ https://issues.apache.org/jira/browse/LUCENE-1448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12648290#action_12648290 ] Michael Busch commented on LUCENE-1448: --- {quote} Hmm we could do that... but it seems awkward to add new attributes that apply only to ending state of the tokenizer. {quote} Yeah. Also you wouldn't want to pay overhead in TokenFilters that can buffer tokens to serialize or clone those attributes for every token. {quote} I wonder if instead, w/ the new API, we could simply allow querying of certain attributes (offset, posincr) after incrementToken returns "false"? {quote} Yeah, maybe we can make the AttributeSource more sophisticated, so that it can distinguish between per-field (instance) and per-token attributes. But as a separate patch, not as part of LUCENE-1422. {quote} Why don't you commit the new TokenStream API first, and we can iterate on this issue & commit 2nd? {quote} OK, will do. I think 1422 is ready now. > add getFinalOffset() to TokenStream > --- > > Key: LUCENE-1448 > URL: https://issues.apache.org/jira/browse/LUCENE-1448 > Project: Lucene - Java > Issue Type: Bug > Components: Analysis >Reporter: Michael McCandless >Assignee: Michael McCandless >Priority: Minor > Fix For: 2.9 > > Attachments: LUCENE-1448.patch, LUCENE-1448.patch, LUCENE-1448.patch, > LUCENE-1448.patch > > > If you add multiple Fieldable instances for the same field name to a > document, and you then index those fields with TermVectors storing offsets, > it's very likely the offsets for all but the first field instance will be > wrong. > This is because IndexWriter under the hood adds a cumulative base to the > offsets of each field instance, where that base is 1 + the endOffset of the > last token it saw when analyzing that field. > But this logic is overly simplistic. For example, if the WhitespaceAnalyzer > is being used, and the text being analyzed ended in 3 whitespace characters, > then that information is lost and then next field's offsets are then all 3 > too small. Similarly, if a StopFilter appears in the chain, and the last N > tokens were stop words, then the base will be 1 + the endOffset of the last > non-stopword token. > To fix this, I'd like to add a new getFinalOffset() to TokenStream. I'm > thinking by default it returns -1, which means "I don't know so you figure it > out", meaning we fallback to the faulty logic we have today. > This has come up several times on the user's list. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-1448) add getFinalOffset() to TokenStream
[ https://issues.apache.org/jira/browse/LUCENE-1448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12651999#action_12651999 ] Michael McCandless commented on LUCENE-1448: I'm torn on how to add getFinalOffset()/getFinalPositionIncrement(). One option is to add a set/getFinalOffset to OffsetAttribute. The downside here is it's another int added to that class, that gets copied for caching streams yet is only used at the very end. Another option is to "define" the API such that when incrementToken() returns false, then it has actually advanced to an "end-of-stream token". OffsetAttribute.getEndOffset() should return the final offset. Since we have not released the new API, we could simply make this change (and fix all instances in the core/contrib that use the new API accordingly). I think I like this option best. Yet another option is to open up "per stream" attrs rather than "per token attrs". This seems like alot of added complexity. Are there other things, besides these two, that would be an example of a "per stream" attr? > add getFinalOffset() to TokenStream > --- > > Key: LUCENE-1448 > URL: https://issues.apache.org/jira/browse/LUCENE-1448 > Project: Lucene - Java > Issue Type: Bug > Components: Analysis >Reporter: Michael McCandless >Assignee: Michael McCandless >Priority: Minor > Fix For: 2.9 > > Attachments: LUCENE-1448.patch, LUCENE-1448.patch, LUCENE-1448.patch, > LUCENE-1448.patch > > > If you add multiple Fieldable instances for the same field name to a > document, and you then index those fields with TermVectors storing offsets, > it's very likely the offsets for all but the first field instance will be > wrong. > This is because IndexWriter under the hood adds a cumulative base to the > offsets of each field instance, where that base is 1 + the endOffset of the > last token it saw when analyzing that field. > But this logic is overly simplistic. For example, if the WhitespaceAnalyzer > is being used, and the text being analyzed ended in 3 whitespace characters, > then that information is lost and then next field's offsets are then all 3 > too small. Similarly, if a StopFilter appears in the chain, and the last N > tokens were stop words, then the base will be 1 + the endOffset of the last > non-stopword token. > To fix this, I'd like to add a new getFinalOffset() to TokenStream. I'm > thinking by default it returns -1, which means "I don't know so you figure it > out", meaning we fallback to the faulty logic we have today. > This has come up several times on the user's list. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-1448) add getFinalOffset() to TokenStream
[ https://issues.apache.org/jira/browse/LUCENE-1448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653065#action_12653065 ] Mark Miller commented on LUCENE-1448: - bq. Another option is to "define" the API such that when incrementToken() returns false, then it has actually advanced to an "end-of-stream token". OffsetAttribute.getEndOffset() should return the final offset. Since we have not released the new API, we could simply make this change (and fix all instances in the core/contrib that use the new API accordingly). I think I like this option best. +1. I like this. > add getFinalOffset() to TokenStream > --- > > Key: LUCENE-1448 > URL: https://issues.apache.org/jira/browse/LUCENE-1448 > Project: Lucene - Java > Issue Type: Bug > Components: Analysis >Reporter: Michael McCandless >Assignee: Michael McCandless >Priority: Minor > Fix For: 2.9 > > Attachments: LUCENE-1448.patch, LUCENE-1448.patch, LUCENE-1448.patch, > LUCENE-1448.patch > > > If you add multiple Fieldable instances for the same field name to a > document, and you then index those fields with TermVectors storing offsets, > it's very likely the offsets for all but the first field instance will be > wrong. > This is because IndexWriter under the hood adds a cumulative base to the > offsets of each field instance, where that base is 1 + the endOffset of the > last token it saw when analyzing that field. > But this logic is overly simplistic. For example, if the WhitespaceAnalyzer > is being used, and the text being analyzed ended in 3 whitespace characters, > then that information is lost and then next field's offsets are then all 3 > too small. Similarly, if a StopFilter appears in the chain, and the last N > tokens were stop words, then the base will be 1 + the endOffset of the last > non-stopword token. > To fix this, I'd like to add a new getFinalOffset() to TokenStream. I'm > thinking by default it returns -1, which means "I don't know so you figure it > out", meaning we fallback to the faulty logic we have today. > This has come up several times on the user's list. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-1448) add getFinalOffset() to TokenStream
[ https://issues.apache.org/jira/browse/LUCENE-1448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653079#action_12653079 ] Michael McCandless commented on LUCENE-1448: OK, me too. I'll move forward with that approach. > add getFinalOffset() to TokenStream > --- > > Key: LUCENE-1448 > URL: https://issues.apache.org/jira/browse/LUCENE-1448 > Project: Lucene - Java > Issue Type: Bug > Components: Analysis >Reporter: Michael McCandless >Assignee: Michael McCandless >Priority: Minor > Fix For: 2.9 > > Attachments: LUCENE-1448.patch, LUCENE-1448.patch, LUCENE-1448.patch, > LUCENE-1448.patch > > > If you add multiple Fieldable instances for the same field name to a > document, and you then index those fields with TermVectors storing offsets, > it's very likely the offsets for all but the first field instance will be > wrong. > This is because IndexWriter under the hood adds a cumulative base to the > offsets of each field instance, where that base is 1 + the endOffset of the > last token it saw when analyzing that field. > But this logic is overly simplistic. For example, if the WhitespaceAnalyzer > is being used, and the text being analyzed ended in 3 whitespace characters, > then that information is lost and then next field's offsets are then all 3 > too small. Similarly, if a StopFilter appears in the chain, and the last N > tokens were stop words, then the base will be 1 + the endOffset of the last > non-stopword token. > To fix this, I'd like to add a new getFinalOffset() to TokenStream. I'm > thinking by default it returns -1, which means "I don't know so you figure it > out", meaning we fallback to the faulty logic we have today. > This has come up several times on the user's list. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-1448) add getFinalOffset() to TokenStream
[ https://issues.apache.org/jira/browse/LUCENE-1448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653513#action_12653513 ] Michael Busch commented on LUCENE-1448: --- {quote} Another option is to "define" the API such that when incrementToken() returns false, then it has actually advanced to an "end-of-stream token". OffsetAttribute.getEndOffset() should return the final offset. Since we have not released the new API, we could simply make this change (and fix all instances in the core/contrib that use the new API accordingly). I think I like this option best. {quote} This adds some "cleaning up" responsibilities to all existing TokenFilters out there. So far it is very straightforward to change an existing TokenFilter to use the new API. You simply have to: - add attributes the filter needs in its constructor - change next() to incrementToken() and change return calls that return null to false, others to true (or what input returns) - don't access a token but the appropriate attributes to set the data But maybe there's a custom filter in the end of the chain that returns more tokens even after its input returned the last one. For example a SynonymExpansionFilter might return a synonym for the last word it received from its input before it returns false. In this case it might overwrite endOffset that another filter/stream already set to the final endOffset. It needs to cache that value and set it when it returns false. ALso all filters that currently use an offset need to know now to clean up before returning false. I'm not saying this is necessarily bad. I also find this approach tempting, because it's simple. But it might be a common pitfall for bugs? What I'd like to work on soon is an efficient way to buffer attributes (maybe add methods to attribute that write into a bytebuffer). Then attributes can implement what variables need to be serialized and which ones don't. In that case we could add a finalOffset to OffsetAttribute that does not get serialiezd/deserialized. And possibly it might be worthwhile to have explicit states defined in a TokenStream that we can enforce with three methods: start(), increment(), end(). Then people would now if they have to do something at the end of a stream they have to do it in end(). > add getFinalOffset() to TokenStream > --- > > Key: LUCENE-1448 > URL: https://issues.apache.org/jira/browse/LUCENE-1448 > Project: Lucene - Java > Issue Type: Bug > Components: Analysis >Reporter: Michael McCandless >Assignee: Michael McCandless >Priority: Minor > Fix For: 2.9 > > Attachments: LUCENE-1448.patch, LUCENE-1448.patch, LUCENE-1448.patch, > LUCENE-1448.patch > > > If you add multiple Fieldable instances for the same field name to a > document, and you then index those fields with TermVectors storing offsets, > it's very likely the offsets for all but the first field instance will be > wrong. > This is because IndexWriter under the hood adds a cumulative base to the > offsets of each field instance, where that base is 1 + the endOffset of the > last token it saw when analyzing that field. > But this logic is overly simplistic. For example, if the WhitespaceAnalyzer > is being used, and the text being analyzed ended in 3 whitespace characters, > then that information is lost and then next field's offsets are then all 3 > too small. Similarly, if a StopFilter appears in the chain, and the last N > tokens were stop words, then the base will be 1 + the endOffset of the last > non-stopword token. > To fix this, I'd like to add a new getFinalOffset() to TokenStream. I'm > thinking by default it returns -1, which means "I don't know so you figure it > out", meaning we fallback to the faulty logic we have today. > This has come up several times on the user's list. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-1448) add getFinalOffset() to TokenStream
[ https://issues.apache.org/jira/browse/LUCENE-1448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653893#action_12653893 ] Michael McCandless commented on LUCENE-1448: {quote} What I'd like to work on soon is an efficient way to buffer attributes (maybe add methods to attribute that write into a bytebuffer). Then attributes can implement what variables need to be serialized and which ones don't. In that case we could add a finalOffset to OffsetAttribute that does not get serialiezd/deserialized. {quote} I like that (it'd make streams like CachingTokenFilter much more efficient). It'd also presumably lead to more efficiently serialized token streams. But: you'd still need a way in this model to serialize finalOffset, once, at the end? {quote} And possibly it might be worthwhile to have explicit states defined in a TokenStream that we can enforce with three methods: start(), increment(), end(). Then people would now if they have to do something at the end of a stream they have to do it in end(). {quote} This also seems good. So end() would be the obvious place to set the OffsetAttribute.finalOffset, PositionIncrementAttribute.positionIncrementGap, etc. OK I'm gonna assign this one to you, Michael ;) > add getFinalOffset() to TokenStream > --- > > Key: LUCENE-1448 > URL: https://issues.apache.org/jira/browse/LUCENE-1448 > Project: Lucene - Java > Issue Type: Bug > Components: Analysis >Reporter: Michael McCandless >Assignee: Michael McCandless >Priority: Minor > Fix For: 2.9 > > Attachments: LUCENE-1448.patch, LUCENE-1448.patch, LUCENE-1448.patch, > LUCENE-1448.patch > > > If you add multiple Fieldable instances for the same field name to a > document, and you then index those fields with TermVectors storing offsets, > it's very likely the offsets for all but the first field instance will be > wrong. > This is because IndexWriter under the hood adds a cumulative base to the > offsets of each field instance, where that base is 1 + the endOffset of the > last token it saw when analyzing that field. > But this logic is overly simplistic. For example, if the WhitespaceAnalyzer > is being used, and the text being analyzed ended in 3 whitespace characters, > then that information is lost and then next field's offsets are then all 3 > too small. Similarly, if a StopFilter appears in the chain, and the last N > tokens were stop words, then the base will be 1 + the endOffset of the last > non-stopword token. > To fix this, I'd like to add a new getFinalOffset() to TokenStream. I'm > thinking by default it returns -1, which means "I don't know so you figure it > out", meaning we fallback to the faulty logic we have today. > This has come up several times on the user's list. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-1448) add getFinalOffset() to TokenStream
[ https://issues.apache.org/jira/browse/LUCENE-1448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12654044#action_12654044 ] Michael Busch commented on LUCENE-1448: --- {quote} But: you'd still need a way in this model to serialize finalOffset, once, at the end? {quote} Maybe we can introduce an abstract EndOfStreamAttribute and FinalOffsetAttribute and FinalPosIncrAttribute that extend EOSA. Then in a stream like CachingTokenFilter a AttributeAcceptor can be used that doesn't accept attributes of type EOSA in increment(). In end() it would use an AttributeAcceptor that accepts EOSA atts and cache those. {quote} OK I'm gonna assign this one to you, Michael ;) {quote} Bummer! Why did I say anything? ;) j/k > add getFinalOffset() to TokenStream > --- > > Key: LUCENE-1448 > URL: https://issues.apache.org/jira/browse/LUCENE-1448 > Project: Lucene - Java > Issue Type: Bug > Components: Analysis >Reporter: Michael McCandless >Assignee: Michael Busch >Priority: Minor > Fix For: 2.9 > > Attachments: LUCENE-1448.patch, LUCENE-1448.patch, LUCENE-1448.patch, > LUCENE-1448.patch > > > If you add multiple Fieldable instances for the same field name to a > document, and you then index those fields with TermVectors storing offsets, > it's very likely the offsets for all but the first field instance will be > wrong. > This is because IndexWriter under the hood adds a cumulative base to the > offsets of each field instance, where that base is 1 + the endOffset of the > last token it saw when analyzing that field. > But this logic is overly simplistic. For example, if the WhitespaceAnalyzer > is being used, and the text being analyzed ended in 3 whitespace characters, > then that information is lost and then next field's offsets are then all 3 > too small. Similarly, if a StopFilter appears in the chain, and the last N > tokens were stop words, then the base will be 1 + the endOffset of the last > non-stopword token. > To fix this, I'd like to add a new getFinalOffset() to TokenStream. I'm > thinking by default it returns -1, which means "I don't know so you figure it > out", meaning we fallback to the faulty logic we have today. > This has come up several times on the user's list. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-1448) add getFinalOffset() to TokenStream
[ https://issues.apache.org/jira/browse/LUCENE-1448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734022#action_12734022 ] Michael Busch commented on LUCENE-1448: --- OK I think I have this basically working with old and new API (including 1693 changes). The approach I took is fairly simple, it doesn't require adding a new Attribute. I added the following method to TokenSteam: {code:java} /** * This method is called by the consumer after the last token has been consumed, * i.e. after {...@link #incrementToken()} returned false (using the new TokenStream API) * or after {...@link #next(Token)} or {...@link #next()} returned null (old TokenStream API). * * This method can be used to perform any end-of-stream operations, such as setting the final * offset of a stream. The final offset of a stream might differ from the offset of the last token * e.g. in case one or more whitespaces followed after the last token, but a {...@link WhitespaceTokenizer} * was used. * * * @throws IOException */ public void end() throws IOException { // do nothing by default } {code} Then I took Mike's patch and implemented end() in all classes where his patch added getFinalOffset(). E.g. in CharTokenizer the implementations looks like this: {code:java} public void end() { // set final offset int finalOffset = input.correctOffset(offset); offsetAtt.setOffset(finalOffset, finalOffset); } {code} I changed DocInverterPerField to call end() after the stream is fully consumed and use what offsetAttribute.endOffset() returns as final offset. I also added all new tests from Mike's latest patch. All unit tests, including the new ones, pass. Also test-tag. I'm not posting a patch yet, because this depends on 1693. Mike, Uwe, others: could you please review if this approach makes sense? > add getFinalOffset() to TokenStream > --- > > Key: LUCENE-1448 > URL: https://issues.apache.org/jira/browse/LUCENE-1448 > Project: Lucene - Java > Issue Type: Bug > Components: Analysis >Reporter: Michael McCandless >Assignee: Michael Busch >Priority: Minor > Fix For: 2.9 > > Attachments: LUCENE-1448.patch, LUCENE-1448.patch, LUCENE-1448.patch, > LUCENE-1448.patch > > > If you add multiple Fieldable instances for the same field name to a > document, and you then index those fields with TermVectors storing offsets, > it's very likely the offsets for all but the first field instance will be > wrong. > This is because IndexWriter under the hood adds a cumulative base to the > offsets of each field instance, where that base is 1 + the endOffset of the > last token it saw when analyzing that field. > But this logic is overly simplistic. For example, if the WhitespaceAnalyzer > is being used, and the text being analyzed ended in 3 whitespace characters, > then that information is lost and then next field's offsets are then all 3 > too small. Similarly, if a StopFilter appears in the chain, and the last N > tokens were stop words, then the base will be 1 + the endOffset of the last > non-stopword token. > To fix this, I'd like to add a new getFinalOffset() to TokenStream. I'm > thinking by default it returns -1, which means "I don't know so you figure it > out", meaning we fallback to the faulty logic we have today. > This has come up several times on the user's list. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-1448) add getFinalOffset() to TokenStream
[ https://issues.apache.org/jira/browse/LUCENE-1448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734023#action_12734023 ] Michael Busch commented on LUCENE-1448: --- Hmm one thing I haven't done yet is changing Tee/Sink and CachingTokenFilter. But it should be simple: CachingTokenFilter.end() should call input.end() when it is called for the first time and store the captured state locally as finalState. Then whenever CachingTokenFilter.end() is called again, it just restores the finalState. For Tee/Sink it should work similarly: The tee just puts a finalState into the sink(s) the first time end() is called. And when end() of a sink is called it restores the finalState. This should work? > add getFinalOffset() to TokenStream > --- > > Key: LUCENE-1448 > URL: https://issues.apache.org/jira/browse/LUCENE-1448 > Project: Lucene - Java > Issue Type: Bug > Components: Analysis >Reporter: Michael McCandless >Assignee: Michael Busch >Priority: Minor > Fix For: 2.9 > > Attachments: LUCENE-1448.patch, LUCENE-1448.patch, LUCENE-1448.patch, > LUCENE-1448.patch > > > If you add multiple Fieldable instances for the same field name to a > document, and you then index those fields with TermVectors storing offsets, > it's very likely the offsets for all but the first field instance will be > wrong. > This is because IndexWriter under the hood adds a cumulative base to the > offsets of each field instance, where that base is 1 + the endOffset of the > last token it saw when analyzing that field. > But this logic is overly simplistic. For example, if the WhitespaceAnalyzer > is being used, and the text being analyzed ended in 3 whitespace characters, > then that information is lost and then next field's offsets are then all 3 > too small. Similarly, if a StopFilter appears in the chain, and the last N > tokens were stop words, then the base will be 1 + the endOffset of the last > non-stopword token. > To fix this, I'd like to add a new getFinalOffset() to TokenStream. I'm > thinking by default it returns -1, which means "I don't know so you figure it > out", meaning we fallback to the faulty logic we have today. > This has come up several times on the user's list. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-1448) add getFinalOffset() to TokenStream
[ https://issues.apache.org/jira/browse/LUCENE-1448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734025#action_12734025 ] Michael Busch commented on LUCENE-1448: --- Hmm another reason why I don't like two Tees feeding one Sink: What is the finalOffset and finalState then? > add getFinalOffset() to TokenStream > --- > > Key: LUCENE-1448 > URL: https://issues.apache.org/jira/browse/LUCENE-1448 > Project: Lucene - Java > Issue Type: Bug > Components: Analysis >Reporter: Michael McCandless >Assignee: Michael Busch >Priority: Minor > Fix For: 2.9 > > Attachments: LUCENE-1448.patch, LUCENE-1448.patch, LUCENE-1448.patch, > LUCENE-1448.patch > > > If you add multiple Fieldable instances for the same field name to a > document, and you then index those fields with TermVectors storing offsets, > it's very likely the offsets for all but the first field instance will be > wrong. > This is because IndexWriter under the hood adds a cumulative base to the > offsets of each field instance, where that base is 1 + the endOffset of the > last token it saw when analyzing that field. > But this logic is overly simplistic. For example, if the WhitespaceAnalyzer > is being used, and the text being analyzed ended in 3 whitespace characters, > then that information is lost and then next field's offsets are then all 3 > too small. Similarly, if a StopFilter appears in the chain, and the last N > tokens were stop words, then the base will be 1 + the endOffset of the last > non-stopword token. > To fix this, I'd like to add a new getFinalOffset() to TokenStream. I'm > thinking by default it returns -1, which means "I don't know so you figure it > out", meaning we fallback to the faulty logic we have today. > This has come up several times on the user's list. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-1448) add getFinalOffset() to TokenStream
[ https://issues.apache.org/jira/browse/LUCENE-1448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734063#action_12734063 ] Uwe Schindler commented on LUCENE-1448: --- This is not the only problem with multiple Tees: The offsets are also completely mixed together, especially if the two tees feed into the sink at the same time (not after each other). In my opinion, the last call to end should be cached by the sink as end state (so if two tees add a end state to the tee, the second one overwrites the first one). > add getFinalOffset() to TokenStream > --- > > Key: LUCENE-1448 > URL: https://issues.apache.org/jira/browse/LUCENE-1448 > Project: Lucene - Java > Issue Type: Bug > Components: Analysis >Reporter: Michael McCandless >Assignee: Michael Busch >Priority: Minor > Fix For: 2.9 > > Attachments: LUCENE-1448.patch, LUCENE-1448.patch, LUCENE-1448.patch, > LUCENE-1448.patch > > > If you add multiple Fieldable instances for the same field name to a > document, and you then index those fields with TermVectors storing offsets, > it's very likely the offsets for all but the first field instance will be > wrong. > This is because IndexWriter under the hood adds a cumulative base to the > offsets of each field instance, where that base is 1 + the endOffset of the > last token it saw when analyzing that field. > But this logic is overly simplistic. For example, if the WhitespaceAnalyzer > is being used, and the text being analyzed ended in 3 whitespace characters, > then that information is lost and then next field's offsets are then all 3 > too small. Similarly, if a StopFilter appears in the chain, and the last N > tokens were stop words, then the base will be 1 + the endOffset of the last > non-stopword token. > To fix this, I'd like to add a new getFinalOffset() to TokenStream. I'm > thinking by default it returns -1, which means "I don't know so you figure it > out", meaning we fallback to the faulty logic we have today. > This has come up several times on the user's list. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-1448) add getFinalOffset() to TokenStream
[ https://issues.apache.org/jira/browse/LUCENE-1448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734280#action_12734280 ] Michael McCandless commented on LUCENE-1448: This approach (adding end()) sounds good! > add getFinalOffset() to TokenStream > --- > > Key: LUCENE-1448 > URL: https://issues.apache.org/jira/browse/LUCENE-1448 > Project: Lucene - Java > Issue Type: Bug > Components: Analysis >Reporter: Michael McCandless >Assignee: Michael Busch >Priority: Minor > Fix For: 2.9 > > Attachments: LUCENE-1448.patch, LUCENE-1448.patch, LUCENE-1448.patch, > LUCENE-1448.patch > > > If you add multiple Fieldable instances for the same field name to a > document, and you then index those fields with TermVectors storing offsets, > it's very likely the offsets for all but the first field instance will be > wrong. > This is because IndexWriter under the hood adds a cumulative base to the > offsets of each field instance, where that base is 1 + the endOffset of the > last token it saw when analyzing that field. > But this logic is overly simplistic. For example, if the WhitespaceAnalyzer > is being used, and the text being analyzed ended in 3 whitespace characters, > then that information is lost and then next field's offsets are then all 3 > too small. Similarly, if a StopFilter appears in the chain, and the last N > tokens were stop words, then the base will be 1 + the endOffset of the last > non-stopword token. > To fix this, I'd like to add a new getFinalOffset() to TokenStream. I'm > thinking by default it returns -1, which means "I don't know so you figure it > out", meaning we fallback to the faulty logic we have today. > This has come up several times on the user's list. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-1448) add getFinalOffset() to TokenStream
[ https://issues.apache.org/jira/browse/LUCENE-1448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734292#action_12734292 ] Michael Busch commented on LUCENE-1448: --- Cool, I will take this approach and submit a patch as soon as LUCENE-1693 is committed. > add getFinalOffset() to TokenStream > --- > > Key: LUCENE-1448 > URL: https://issues.apache.org/jira/browse/LUCENE-1448 > Project: Lucene - Java > Issue Type: Bug > Components: Analysis >Reporter: Michael McCandless >Assignee: Michael Busch >Priority: Minor > Fix For: 2.9 > > Attachments: LUCENE-1448.patch, LUCENE-1448.patch, LUCENE-1448.patch, > LUCENE-1448.patch > > > If you add multiple Fieldable instances for the same field name to a > document, and you then index those fields with TermVectors storing offsets, > it's very likely the offsets for all but the first field instance will be > wrong. > This is because IndexWriter under the hood adds a cumulative base to the > offsets of each field instance, where that base is 1 + the endOffset of the > last token it saw when analyzing that field. > But this logic is overly simplistic. For example, if the WhitespaceAnalyzer > is being used, and the text being analyzed ended in 3 whitespace characters, > then that information is lost and then next field's offsets are then all 3 > too small. Similarly, if a StopFilter appears in the chain, and the last N > tokens were stop words, then the base will be 1 + the endOffset of the last > non-stopword token. > To fix this, I'd like to add a new getFinalOffset() to TokenStream. I'm > thinking by default it returns -1, which means "I don't know so you figure it > out", meaning we fallback to the faulty logic we have today. > This has come up several times on the user's list. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-1448) add getFinalOffset() to TokenStream
[ https://issues.apache.org/jira/browse/LUCENE-1448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735213#action_12735213 ] Michael Busch commented on LUCENE-1448: --- Note that my latest patch only contains fixes for the core TokenStreams. I'll open a separate issue to implement end() for the contrib TokenStreams, which we can commit after LUCENE-1460 is resolved. > add getFinalOffset() to TokenStream > --- > > Key: LUCENE-1448 > URL: https://issues.apache.org/jira/browse/LUCENE-1448 > Project: Lucene - Java > Issue Type: Bug > Components: Analysis >Reporter: Michael McCandless >Assignee: Michael Busch >Priority: Minor > Fix For: 2.9 > > Attachments: lucene-1448.patch, LUCENE-1448.patch, LUCENE-1448.patch, > LUCENE-1448.patch, LUCENE-1448.patch > > > If you add multiple Fieldable instances for the same field name to a > document, and you then index those fields with TermVectors storing offsets, > it's very likely the offsets for all but the first field instance will be > wrong. > This is because IndexWriter under the hood adds a cumulative base to the > offsets of each field instance, where that base is 1 + the endOffset of the > last token it saw when analyzing that field. > But this logic is overly simplistic. For example, if the WhitespaceAnalyzer > is being used, and the text being analyzed ended in 3 whitespace characters, > then that information is lost and then next field's offsets are then all 3 > too small. Similarly, if a StopFilter appears in the chain, and the last N > tokens were stop words, then the base will be 1 + the endOffset of the last > non-stopword token. > To fix this, I'd like to add a new getFinalOffset() to TokenStream. I'm > thinking by default it returns -1, which means "I don't know so you figure it > out", meaning we fallback to the faulty logic we have today. > This has come up several times on the user's list. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org