[jira] [Commented] (SOLR-5330) PerSegmentSingleValuedFaceting overwrites facet values
[ https://issues.apache.org/jira/browse/SOLR-5330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13797244#comment-13797244 ] Yonik Seeley commented on SOLR-5330: I never was able to reproduce... so I just fixed the issue anyway, making a copy of the bytes only instead of cloning the complete BytesRef. > PerSegmentSingleValuedFaceting overwrites facet values > -- > > Key: SOLR-5330 > URL: https://issues.apache.org/jira/browse/SOLR-5330 > Project: Solr > Issue Type: Bug >Affects Versions: 4.2.1 >Reporter: Michael Froh >Assignee: Yonik Seeley > Fix For: 4.5.1, 4.6, 5.0 > > Attachments: solr-5330.patch > > > I recently tried enabling facet.method=fcs for one of my indexes and found a > significant performance improvement (with a large index, many facet values, > and near-realtime updates). Unfortunately, the results were also wrong. > Specifically, some facet values were being partially overwritten by other > facet values. (That is, if I expected facet values like "abcdef" and "123", I > would get a value like "123def".) > Debugging through the code, it looks like the problem was in > PerSegmentSingleValuedFaceting, specifically in the getFacetCounts method, > when BytesRef val is shallow-copied from the temporary per-segment BytesRef. > The byte array assigned to val is shared with the byte array for seg.tempBR, > and is overwritten a few lines down by the call to seg.tenum.next(). > I managed to fix it locally by replacing the shallow copy with a deep copy. > While I encountered this problem on Solr 4.2.1, I see that the code is > identical in 4.5. Unless the behavior of TermsEnum.next() has changed, I > believe this bug still exists. -- This message was sent by Atlassian JIRA (v6.1#6144) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5330) PerSegmentSingleValuedFaceting overwrites facet values
[ https://issues.apache.org/jira/browse/SOLR-5330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13797240#comment-13797240 ] ASF subversion and git services commented on SOLR-5330: --- Commit 1532905 from [~yo...@apache.org] in branch 'dev/branches/lucene_solr_4_5' [ https://svn.apache.org/r1532905 ] SOLR-5330: make copy of term bytes before calling next > PerSegmentSingleValuedFaceting overwrites facet values > -- > > Key: SOLR-5330 > URL: https://issues.apache.org/jira/browse/SOLR-5330 > Project: Solr > Issue Type: Bug >Affects Versions: 4.2.1 >Reporter: Michael Froh >Assignee: Yonik Seeley > Attachments: solr-5330.patch > > > I recently tried enabling facet.method=fcs for one of my indexes and found a > significant performance improvement (with a large index, many facet values, > and near-realtime updates). Unfortunately, the results were also wrong. > Specifically, some facet values were being partially overwritten by other > facet values. (That is, if I expected facet values like "abcdef" and "123", I > would get a value like "123def".) > Debugging through the code, it looks like the problem was in > PerSegmentSingleValuedFaceting, specifically in the getFacetCounts method, > when BytesRef val is shallow-copied from the temporary per-segment BytesRef. > The byte array assigned to val is shared with the byte array for seg.tempBR, > and is overwritten a few lines down by the call to seg.tenum.next(). > I managed to fix it locally by replacing the shallow copy with a deep copy. > While I encountered this problem on Solr 4.2.1, I see that the code is > identical in 4.5. Unless the behavior of TermsEnum.next() has changed, I > believe this bug still exists. -- This message was sent by Atlassian JIRA (v6.1#6144) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5330) PerSegmentSingleValuedFaceting overwrites facet values
[ https://issues.apache.org/jira/browse/SOLR-5330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13797237#comment-13797237 ] ASF subversion and git services commented on SOLR-5330: --- Commit 1532903 from [~yo...@apache.org] in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1532903 ] SOLR-5330: make copy of term bytes before calling next > PerSegmentSingleValuedFaceting overwrites facet values > -- > > Key: SOLR-5330 > URL: https://issues.apache.org/jira/browse/SOLR-5330 > Project: Solr > Issue Type: Bug >Affects Versions: 4.2.1 >Reporter: Michael Froh >Assignee: Yonik Seeley > Attachments: solr-5330.patch > > > I recently tried enabling facet.method=fcs for one of my indexes and found a > significant performance improvement (with a large index, many facet values, > and near-realtime updates). Unfortunately, the results were also wrong. > Specifically, some facet values were being partially overwritten by other > facet values. (That is, if I expected facet values like "abcdef" and "123", I > would get a value like "123def".) > Debugging through the code, it looks like the problem was in > PerSegmentSingleValuedFaceting, specifically in the getFacetCounts method, > when BytesRef val is shallow-copied from the temporary per-segment BytesRef. > The byte array assigned to val is shared with the byte array for seg.tempBR, > and is overwritten a few lines down by the call to seg.tenum.next(). > I managed to fix it locally by replacing the shallow copy with a deep copy. > While I encountered this problem on Solr 4.2.1, I see that the code is > identical in 4.5. Unless the behavior of TermsEnum.next() has changed, I > believe this bug still exists. -- This message was sent by Atlassian JIRA (v6.1#6144) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5330) PerSegmentSingleValuedFaceting overwrites facet values
[ https://issues.apache.org/jira/browse/SOLR-5330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13797235#comment-13797235 ] ASF subversion and git services commented on SOLR-5330: --- Commit 1532900 from [~yo...@apache.org] in branch 'dev/trunk' [ https://svn.apache.org/r1532900 ] SOLR-5330: make copy of term bytes before calling next > PerSegmentSingleValuedFaceting overwrites facet values > -- > > Key: SOLR-5330 > URL: https://issues.apache.org/jira/browse/SOLR-5330 > Project: Solr > Issue Type: Bug >Affects Versions: 4.2.1 >Reporter: Michael Froh >Assignee: Yonik Seeley > Attachments: solr-5330.patch > > > I recently tried enabling facet.method=fcs for one of my indexes and found a > significant performance improvement (with a large index, many facet values, > and near-realtime updates). Unfortunately, the results were also wrong. > Specifically, some facet values were being partially overwritten by other > facet values. (That is, if I expected facet values like "abcdef" and "123", I > would get a value like "123def".) > Debugging through the code, it looks like the problem was in > PerSegmentSingleValuedFaceting, specifically in the getFacetCounts method, > when BytesRef val is shallow-copied from the temporary per-segment BytesRef. > The byte array assigned to val is shared with the byte array for seg.tempBR, > and is overwritten a few lines down by the call to seg.tenum.next(). > I managed to fix it locally by replacing the shallow copy with a deep copy. > While I encountered this problem on Solr 4.2.1, I see that the code is > identical in 4.5. Unless the behavior of TermsEnum.next() has changed, I > believe this bug still exists. -- This message was sent by Atlassian JIRA (v6.1#6144) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5330) PerSegmentSingleValuedFaceting overwrites facet values
[
https://issues.apache.org/jira/browse/SOLR-5330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13793213#comment-13793213
]
Yonik Seeley commented on SOLR-5330:
So I instrumented the faceting code like so:
{code}
seg.tempBR = seg.tenum.next();
if (seg.tempBR.bytes == val.bytes) {
System.err.println("##SHARING DETECTED: val.offset="+val.offset + "
val.length="+val.length + " new.offset="+seg.tempBR.offset + "
new.length="+seg.tempBR.length);
if (val.offset == seg.tempBR.offset) {
System.err.println("!!SHARING USING SAME OFFSET");
}
{code}
And it detects tons of sharing (the returned bytesref still pointing to the
same byte[]) of course... but the thing is, it never generates an invalid
result. calling next() on the term enum never changes the bytes that were
previously pointed to... it simply points to a different part of the same byte
array. I can never detect a case where the original bytes are changed, thus
invalidating the shallow copy.
> PerSegmentSingleValuedFaceting overwrites facet values
> --
>
> Key: SOLR-5330
> URL: https://issues.apache.org/jira/browse/SOLR-5330
> Project: Solr
> Issue Type: Bug
>Affects Versions: 4.2.1
>Reporter: Michael Froh
>Assignee: Yonik Seeley
> Attachments: solr-5330.patch
>
>
> I recently tried enabling facet.method=fcs for one of my indexes and found a
> significant performance improvement (with a large index, many facet values,
> and near-realtime updates). Unfortunately, the results were also wrong.
> Specifically, some facet values were being partially overwritten by other
> facet values. (That is, if I expected facet values like "abcdef" and "123", I
> would get a value like "123def".)
> Debugging through the code, it looks like the problem was in
> PerSegmentSingleValuedFaceting, specifically in the getFacetCounts method,
> when BytesRef val is shallow-copied from the temporary per-segment BytesRef.
> The byte array assigned to val is shared with the byte array for seg.tempBR,
> and is overwritten a few lines down by the call to seg.tenum.next().
> I managed to fix it locally by replacing the shallow copy with a deep copy.
> While I encountered this problem on Solr 4.2.1, I see that the code is
> identical in 4.5. Unless the behavior of TermsEnum.next() has changed, I
> believe this bug still exists.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5330) PerSegmentSingleValuedFaceting overwrites facet values
[ https://issues.apache.org/jira/browse/SOLR-5330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13791896#comment-13791896 ] Yonik Seeley commented on SOLR-5330: It's unfortunate that even the random faceting testing didn't catch this... I'm trying to fix that now. > PerSegmentSingleValuedFaceting overwrites facet values > -- > > Key: SOLR-5330 > URL: https://issues.apache.org/jira/browse/SOLR-5330 > Project: Solr > Issue Type: Bug >Affects Versions: 4.2.1 >Reporter: Michael Froh >Assignee: Yonik Seeley > Attachments: solr-5330.patch > > > I recently tried enabling facet.method=fcs for one of my indexes and found a > significant performance improvement (with a large index, many facet values, > and near-realtime updates). Unfortunately, the results were also wrong. > Specifically, some facet values were being partially overwritten by other > facet values. (That is, if I expected facet values like "abcdef" and "123", I > would get a value like "123def".) > Debugging through the code, it looks like the problem was in > PerSegmentSingleValuedFaceting, specifically in the getFacetCounts method, > when BytesRef val is shallow-copied from the temporary per-segment BytesRef. > The byte array assigned to val is shared with the byte array for seg.tempBR, > and is overwritten a few lines down by the call to seg.tenum.next(). > I managed to fix it locally by replacing the shallow copy with a deep copy. > While I encountered this problem on Solr 4.2.1, I see that the code is > identical in 4.5. Unless the behavior of TermsEnum.next() has changed, I > believe this bug still exists. -- This message was sent by Atlassian JIRA (v6.1#6144) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5330) PerSegmentSingleValuedFaceting overwrites facet values
[ https://issues.apache.org/jira/browse/SOLR-5330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13791612#comment-13791612 ] Yonik Seeley commented on SOLR-5330: Yikes, looks like this bug has been around a while. I'll take it. > PerSegmentSingleValuedFaceting overwrites facet values > -- > > Key: SOLR-5330 > URL: https://issues.apache.org/jira/browse/SOLR-5330 > Project: Solr > Issue Type: Bug >Affects Versions: 4.2.1 >Reporter: Michael Froh > Attachments: solr-5330.patch > > > I recently tried enabling facet.method=fcs for one of my indexes and found a > significant performance improvement (with a large index, many facet values, > and near-realtime updates). Unfortunately, the results were also wrong. > Specifically, some facet values were being partially overwritten by other > facet values. (That is, if I expected facet values like "abcdef" and "123", I > would get a value like "123def".) > Debugging through the code, it looks like the problem was in > PerSegmentSingleValuedFaceting, specifically in the getFacetCounts method, > when BytesRef val is shallow-copied from the temporary per-segment BytesRef. > The byte array assigned to val is shared with the byte array for seg.tempBR, > and is overwritten a few lines down by the call to seg.tenum.next(). > I managed to fix it locally by replacing the shallow copy with a deep copy. > While I encountered this problem on Solr 4.2.1, I see that the code is > identical in 4.5. Unless the behavior of TermsEnum.next() has changed, I > believe this bug still exists. -- This message was sent by Atlassian JIRA (v6.1#6144) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5330) PerSegmentSingleValuedFaceting overwrites facet values
[ https://issues.apache.org/jira/browse/SOLR-5330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13791394#comment-13791394 ] Erick Erickson commented on SOLR-5330: -- Michael: Thanks for the detailed explanation! Could I trouble you to go ahead and attach a patch? Don't worry about "polish", having your code change as a place to at least start (if not use entire) makes things easier for whoever picks this up... > PerSegmentSingleValuedFaceting overwrites facet values > -- > > Key: SOLR-5330 > URL: https://issues.apache.org/jira/browse/SOLR-5330 > Project: Solr > Issue Type: Bug >Affects Versions: 4.2.1 >Reporter: Michael Froh > > I recently tried enabling facet.method=fcs for one of my indexes and found a > significant performance improvement (with a large index, many facet values, > and near-realtime updates). Unfortunately, the results were also wrong. > Specifically, some facet values were being partially overwritten by other > facet values. (That is, if I expected facet values like "abcdef" and "123", I > would get a value like "123def".) > Debugging through the code, it looks like the problem was in > PerSegmentSingleValuedFaceting, specifically in the getFacetCounts method, > when BytesRef val is shallow-copied from the temporary per-segment BytesRef. > The byte array assigned to val is shared with the byte array for seg.tempBR, > and is overwritten a few lines down by the call to seg.tenum.next(). > I managed to fix it locally by replacing the shallow copy with a deep copy. > While I encountered this problem on Solr 4.2.1, I see that the code is > identical in 4.5. Unless the behavior of TermsEnum.next() has changed, I > believe this bug still exists. -- This message was sent by Atlassian JIRA (v6.1#6144) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
