[jira] [Commented] (SOLR-3875) Document boost does not work correctly when using multi-valued fields
[
https://issues.apache.org/jira/browse/SOLR-3875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13610769#comment-13610769
]
Commit Tag Bot commented on SOLR-3875:
--
[branch_4x commit] Chris M. Hostetter
http://svn.apache.org/viewvc?view=revision&revision=1389636
SOLR-3875: Fixed index boosts on multi-valued fields when docBoost is used
(merge r1389628)
> Document boost does not work correctly when using multi-valued fields
> -
>
> Key: SOLR-3875
> URL: https://issues.apache.org/jira/browse/SOLR-3875
> Project: Solr
> Issue Type: Bug
> Components: Schema and Analysis, update
>Affects Versions: 4.0-BETA
>Reporter: Toke Eskildsen
>Assignee: Hoss Man
>Priority: Critical
> Fix For: 4.0, 4.1, 5.0
>
> Attachments: SOLR-3875.patch
>
>
> In Solr 4 BETA & trunk, document boosts skews the ranking for documents with
> multi value fields tremendously. A document boost of 5 combined with 15
> values in a multi value field results in scores above 1,000,000,000, while a
> boost of 0,5 results in scores below 0,001. The error is not present in Solr
> 3.6.
> Thomas Egense and I have tracked it down to a change in Solr DocumentBuilder
> committed 20110827 (@1162347) by Mike McCandless, as part of work done on
> LUCENE-2308. The problem is that Lucene multiplies the boosts of multiple
> instances of the same field when updating the index.
> The old DocumentBuilder, used in Lucene 3.6, handled this by calculating the
> score for the field (docBoost*fieldBoost) and assigning it to the first
> instance of the field, then setting the boost to 1.0f and assigning that to
> subsequent instances of the field. This effectively assigned
> docBoost*fieldBoost to the field, regardless of the number of instances.
> The updated DocumentBuilder (see
> https://svn.apache.org/viewvc/lucene/dev/branches/lucene_solr_4_0/solr/core/src/java/org/apache/solr/update/DocumentBuilder.java?revision=1388778&view=markup),
> used in Lucene 4 BETA & trunk, also assigns docBoost*fieldBoost to the first
> instance of the field. Then it sets fieldBoost = docBoost and continues to
> assign docBoost*fieldBoost to subsequent instances. Using the example
> mentioned above, the generated IndexableFields will get assigned boosts of 5,
> 5*5, 5*5... 5*5. As Lucene multiplies all the values, 15 instances of the
> same field will have a collective boost of 5*25^14.
> This can be demonstrated with the Solr tutorial example by indexing the
> sample documents and adding the document
> {code:xml}
>
>
> Insane score Example. Score = 10E9
> Document boost broken for multivalued fields
> Thomas Egense and Toke Eskildsen
> Test
> bug
> insane_boost
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
>
>
> {code}
> The _manu_ & _features_-fields gets copied to _text_ and a search for
> _thomas_ matches the _text_-field with query explanation
> {code:xml}
>
> 2.44373361E10 = (MATCH) weight(text:thomas in 0) [DefaultSimilarity], result
> of:
> 2.44373361E10 = fieldWeight in 0, product of:
> 1.0 = tf(freq=1.0), with freq of:
> 1.0 = termFreq=1.0
> 3.2512918 = idf(docFreq=3, maxDocs=38)
> 7.5161928E9 = fieldNorm(doc=0)
>
> {code}
> Thomas and I are too pressed for time to attempt a proper patch at the
> moment, but we guess that a reversion to the old algorithm of assigning the
> combined boost to the first instance and 1.0f to all subsequent instances
> would work?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3875) Document boost does not work correctly when using multi-valued fields
[
https://issues.apache.org/jira/browse/SOLR-3875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13482746#comment-13482746
]
Hoss Man commented on SOLR-3875:
Toke: thanks for following up - too bad we didn't catch this other problem
before 4.0.
I've spun off SOLR-3981 to work on this since SOLR-3875 is already resolved and
listed as fixed in 4.0 (we can't (sanely) re-open issues that were recorded in
CHANGES.txt for official releases since it would leave users confused as to
what parts of those issues were resolved in each version)
> Document boost does not work correctly when using multi-valued fields
> -
>
> Key: SOLR-3875
> URL: https://issues.apache.org/jira/browse/SOLR-3875
> Project: Solr
> Issue Type: Bug
> Components: Schema and Analysis, update
>Affects Versions: 4.0-BETA
>Reporter: Toke Eskildsen
>Assignee: Hoss Man
>Priority: Critical
> Fix For: 4.0, 4.1, 5.0
>
> Attachments: SOLR-3875.patch
>
>
> In Solr 4 BETA & trunk, document boosts skews the ranking for documents with
> multi value fields tremendously. A document boost of 5 combined with 15
> values in a multi value field results in scores above 1,000,000,000, while a
> boost of 0,5 results in scores below 0,001. The error is not present in Solr
> 3.6.
> Thomas Egense and I have tracked it down to a change in Solr DocumentBuilder
> committed 20110827 (@1162347) by Mike McCandless, as part of work done on
> LUCENE-2308. The problem is that Lucene multiplies the boosts of multiple
> instances of the same field when updating the index.
> The old DocumentBuilder, used in Lucene 3.6, handled this by calculating the
> score for the field (docBoost*fieldBoost) and assigning it to the first
> instance of the field, then setting the boost to 1.0f and assigning that to
> subsequent instances of the field. This effectively assigned
> docBoost*fieldBoost to the field, regardless of the number of instances.
> The updated DocumentBuilder (see
> https://svn.apache.org/viewvc/lucene/dev/branches/lucene_solr_4_0/solr/core/src/java/org/apache/solr/update/DocumentBuilder.java?revision=1388778&view=markup),
> used in Lucene 4 BETA & trunk, also assigns docBoost*fieldBoost to the first
> instance of the field. Then it sets fieldBoost = docBoost and continues to
> assign docBoost*fieldBoost to subsequent instances. Using the example
> mentioned above, the generated IndexableFields will get assigned boosts of 5,
> 5*5, 5*5... 5*5. As Lucene multiplies all the values, 15 instances of the
> same field will have a collective boost of 5*25^14.
> This can be demonstrated with the Solr tutorial example by indexing the
> sample documents and adding the document
> {code:xml}
>
>
> Insane score Example. Score = 10E9
> Document boost broken for multivalued fields
> Thomas Egense and Toke Eskildsen
> Test
> bug
> insane_boost
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
>
>
> {code}
> The _manu_ & _features_-fields gets copied to _text_ and a search for
> _thomas_ matches the _text_-field with query explanation
> {code:xml}
>
> 2.44373361E10 = (MATCH) weight(text:thomas in 0) [DefaultSimilarity], result
> of:
> 2.44373361E10 = fieldWeight in 0, product of:
> 1.0 = tf(freq=1.0), with freq of:
> 1.0 = termFreq=1.0
> 3.2512918 = idf(docFreq=3, maxDocs=38)
> 7.5161928E9 = fieldNorm(doc=0)
>
> {code}
> Thomas and I are too pressed for time to attempt a proper patch at the
> moment, but we guess that a reversion to the old algorithm of assigning the
> combined boost to the first instance and 1.0f to all subsequent instances
> would work?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3875) Document boost does not work correctly when using multi-valued fields
[
https://issues.apache.org/jira/browse/SOLR-3875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13482233#comment-13482233
]
Toke Eskildsen commented on SOLR-3875:
--
Unfortunately, the bug is only partly solved. Thomas and I encountered strange
scores again. While boosting of multi-value fields is handled correctly in Solr
4.0.0, boosting for copyFields are not. A sample document:
{code}
Insane score Example. Score = 10E9
Document boost broken for copyFields
video ThomasEgense and Toke Eskildsen
Test
bug
something else
bug
bug
{code}
The fields _name_, _manu_, _cat_, _features_, keywords and _content_ gets
copied to text and a search for thomasegense matches the text-field with query
explanation
{code}
70384.67 = (MATCH) weight(text:thomasegense in 0) [DefaultSimilarity], result
of:
70384.67 = fieldWeight in 0, product of:
1.0 = tf(freq=1.0), with freq of:
1.0 = termFreq=1.0
0.30685282 = idf(docFreq=1, maxDocs=1)
229376.0 = fieldNorm(doc=0)
{code}
If the two last fields _keywords_ and _content_ are removed from the sample
document, the score is reduced by a factor 100 (docBoost^2).
The current DocumentBuilder
https://svn.apache.org/viewvc/lucene/dev/branches/lucene_solr_4_0/solr/core/src/java/org/apache/solr/update/DocumentBuilder.java?revision=1389648&view=markup
works roughly like this:
{code}
foreach (field) {
boost = docBoost*fieldBoost
foreach (value) {
assignField(field, value, boost)
foreach (copyField) {
assignField(copyField, value, boost)
}
boost = 1f
}
}
{code}
When all fields share the same copyField (_text_ in this example), the
copyField will have the full boost assigned for each directly specified field
which uses that copyField. That's 5 times with the provided sample, so the
total boost for the field _text_ will be 10^5.
One solution would be to keep track of used fields (directly specified as well
as copyFields) and only assign the full boost once per document. If the number
of unique fields/document is low, a simple list would probably be the fastest
and with low GC impact. For a higher number of unique fields, a Set might be
better. An optimization would be to only create the tracking structure once a
boost != 1.0f is encountered and only store the fields with boost != 1.0f, so
that an update without boosts would not get a performance penalty.
> Document boost does not work correctly when using multi-valued fields
> -
>
> Key: SOLR-3875
> URL: https://issues.apache.org/jira/browse/SOLR-3875
> Project: Solr
> Issue Type: Bug
> Components: Schema and Analysis, update
>Affects Versions: 4.0-BETA
>Reporter: Toke Eskildsen
>Assignee: Hoss Man
>Priority: Critical
> Fix For: 4.0, 4.1, 5.0
>
> Attachments: SOLR-3875.patch
>
>
> In Solr 4 BETA & trunk, document boosts skews the ranking for documents with
> multi value fields tremendously. A document boost of 5 combined with 15
> values in a multi value field results in scores above 1,000,000,000, while a
> boost of 0,5 results in scores below 0,001. The error is not present in Solr
> 3.6.
> Thomas Egense and I have tracked it down to a change in Solr DocumentBuilder
> committed 20110827 (@1162347) by Mike McCandless, as part of work done on
> LUCENE-2308. The problem is that Lucene multiplies the boosts of multiple
> instances of the same field when updating the index.
> The old DocumentBuilder, used in Lucene 3.6, handled this by calculating the
> score for the field (docBoost*fieldBoost) and assigning it to the first
> instance of the field, then setting the boost to 1.0f and assigning that to
> subsequent instances of the field. This effectively assigned
> docBoost*fieldBoost to the field, regardless of the number of instances.
> The updated DocumentBuilder (see
> https://svn.apache.org/viewvc/lucene/dev/branches/lucene_solr_4_0/solr/core/src/java/org/apache/solr/update/DocumentBuilder.java?revision=1388778&view=markup),
> used in Lucene 4 BETA & trunk, also assigns docBoost*fieldBoost to the first
> instance of the field. Then it sets fieldBoost = docBoost and continues to
> assign docBoost*fieldBoost to subsequent instances. Using the example
> mentioned above, the generated IndexableFields will get assigned boosts of 5,
> 5*5, 5*5... 5*5. As Lucene multiplies all the values, 15 instances of the
> same field will have a collective boost of 5*25^14.
> This can be demonstrated with the Solr tutorial example by indexing the
> sample documents and adding the document
> {code:xml}
>
>
> Insane score Example. Score = 10E9
> Document boost broken for multivalued fields
> Thomas Egense and Toke Eskildsen
> Test
> bug
> insane_boos
[jira] [Commented] (SOLR-3875) Document boost does not work correctly when using multi-valued fields
[
https://issues.apache.org/jira/browse/SOLR-3875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13462661#comment-13462661
]
Michael McCandless commented on SOLR-3875:
--
Thanks Hoss.
> Document boost does not work correctly when using multi-valued fields
> -
>
> Key: SOLR-3875
> URL: https://issues.apache.org/jira/browse/SOLR-3875
> Project: Solr
> Issue Type: Bug
> Components: Schema and Analysis, update
>Affects Versions: 4.0-BETA
>Reporter: Toke Eskildsen
>Assignee: Hoss Man
>Priority: Critical
> Fix For: 4.0, 4.1, 5.0
>
> Attachments: SOLR-3875.patch
>
>
> In Solr 4 BETA & trunk, document boosts skews the ranking for documents with
> multi value fields tremendously. A document boost of 5 combined with 15
> values in a multi value field results in scores above 1,000,000,000, while a
> boost of 0,5 results in scores below 0,001. The error is not present in Solr
> 3.6.
> Thomas Egense and I have tracked it down to a change in Solr DocumentBuilder
> committed 20110827 (@1162347) by Mike McCandless, as part of work done on
> LUCENE-2308. The problem is that Lucene multiplies the boosts of multiple
> instances of the same field when updating the index.
> The old DocumentBuilder, used in Lucene 3.6, handled this by calculating the
> score for the field (docBoost*fieldBoost) and assigning it to the first
> instance of the field, then setting the boost to 1.0f and assigning that to
> subsequent instances of the field. This effectively assigned
> docBoost*fieldBoost to the field, regardless of the number of instances.
> The updated DocumentBuilder (see
> https://svn.apache.org/viewvc/lucene/dev/branches/lucene_solr_4_0/solr/core/src/java/org/apache/solr/update/DocumentBuilder.java?revision=1388778&view=markup),
> used in Lucene 4 BETA & trunk, also assigns docBoost*fieldBoost to the first
> instance of the field. Then it sets fieldBoost = docBoost and continues to
> assign docBoost*fieldBoost to subsequent instances. Using the example
> mentioned above, the generated IndexableFields will get assigned boosts of 5,
> 5*5, 5*5... 5*5. As Lucene multiplies all the values, 15 instances of the
> same field will have a collective boost of 5*25^14.
> This can be demonstrated with the Solr tutorial example by indexing the
> sample documents and adding the document
> {code:xml}
>
>
> Insane score Example. Score = 10E9
> Document boost broken for multivalued fields
> Thomas Egense and Toke Eskildsen
> Test
> bug
> insane_boost
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
>
>
> {code}
> The _manu_ & _features_-fields gets copied to _text_ and a search for
> _thomas_ matches the _text_-field with query explanation
> {code:xml}
>
> 2.44373361E10 = (MATCH) weight(text:thomas in 0) [DefaultSimilarity], result
> of:
> 2.44373361E10 = fieldWeight in 0, product of:
> 1.0 = tf(freq=1.0), with freq of:
> 1.0 = termFreq=1.0
> 3.2512918 = idf(docFreq=3, maxDocs=38)
> 7.5161928E9 = fieldNorm(doc=0)
>
> {code}
> Thomas and I are too pressed for time to attempt a proper patch at the
> moment, but we guess that a reversion to the old algorithm of assigning the
> combined boost to the first instance and 1.0f to all subsequent instances
> would work?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3875) Document boost does not work correctly when using multi-valued fields
[
https://issues.apache.org/jira/browse/SOLR-3875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13462568#comment-13462568
]
Thomas Egense commented on SOLR-3875:
-
Thank you for the fast response and fix! Great job.
> Document boost does not work correctly when using multi-valued fields
> -
>
> Key: SOLR-3875
> URL: https://issues.apache.org/jira/browse/SOLR-3875
> Project: Solr
> Issue Type: Bug
> Components: Schema and Analysis, update
>Affects Versions: 4.0-BETA
>Reporter: Toke Eskildsen
>Assignee: Hoss Man
>Priority: Critical
> Fix For: 4.0, 4.1, 5.0
>
> Attachments: SOLR-3875.patch
>
>
> In Solr 4 BETA & trunk, document boosts skews the ranking for documents with
> multi value fields tremendously. A document boost of 5 combined with 15
> values in a multi value field results in scores above 1,000,000,000, while a
> boost of 0,5 results in scores below 0,001. The error is not present in Solr
> 3.6.
> Thomas Egense and I have tracked it down to a change in Solr DocumentBuilder
> committed 20110827 (@1162347) by Mike McCandless, as part of work done on
> LUCENE-2308. The problem is that Lucene multiplies the boosts of multiple
> instances of the same field when updating the index.
> The old DocumentBuilder, used in Lucene 3.6, handled this by calculating the
> score for the field (docBoost*fieldBoost) and assigning it to the first
> instance of the field, then setting the boost to 1.0f and assigning that to
> subsequent instances of the field. This effectively assigned
> docBoost*fieldBoost to the field, regardless of the number of instances.
> The updated DocumentBuilder (see
> https://svn.apache.org/viewvc/lucene/dev/branches/lucene_solr_4_0/solr/core/src/java/org/apache/solr/update/DocumentBuilder.java?revision=1388778&view=markup),
> used in Lucene 4 BETA & trunk, also assigns docBoost*fieldBoost to the first
> instance of the field. Then it sets fieldBoost = docBoost and continues to
> assign docBoost*fieldBoost to subsequent instances. Using the example
> mentioned above, the generated IndexableFields will get assigned boosts of 5,
> 5*5, 5*5... 5*5. As Lucene multiplies all the values, 15 instances of the
> same field will have a collective boost of 5*25^14.
> This can be demonstrated with the Solr tutorial example by indexing the
> sample documents and adding the document
> {code:xml}
>
>
> Insane score Example. Score = 10E9
> Document boost broken for multivalued fields
> Thomas Egense and Toke Eskildsen
> Test
> bug
> insane_boost
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
>
>
> {code}
> The _manu_ & _features_-fields gets copied to _text_ and a search for
> _thomas_ matches the _text_-field with query explanation
> {code:xml}
>
> 2.44373361E10 = (MATCH) weight(text:thomas in 0) [DefaultSimilarity], result
> of:
> 2.44373361E10 = fieldWeight in 0, product of:
> 1.0 = tf(freq=1.0), with freq of:
> 1.0 = termFreq=1.0
> 3.2512918 = idf(docFreq=3, maxDocs=38)
> 7.5161928E9 = fieldNorm(doc=0)
>
> {code}
> Thomas and I are too pressed for time to attempt a proper patch at the
> moment, but we guess that a reversion to the old algorithm of assigning the
> combined boost to the first instance and 1.0f to all subsequent instances
> would work?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3875) Document boost does not work correctly when using multi-valued fields
[
https://issues.apache.org/jira/browse/SOLR-3875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13462243#comment-13462243
]
Hoss Man commented on SOLR-3875:
Committed revision 1389648. - 4.0
> Document boost does not work correctly when using multi-valued fields
> -
>
> Key: SOLR-3875
> URL: https://issues.apache.org/jira/browse/SOLR-3875
> Project: Solr
> Issue Type: Bug
> Components: Schema and Analysis, update
>Affects Versions: 4.0-BETA
>Reporter: Toke Eskildsen
>Assignee: Hoss Man
>Priority: Critical
> Fix For: 4.0, 4.1, 5.0
>
> Attachments: SOLR-3875.patch
>
>
> In Solr 4 BETA & trunk, document boosts skews the ranking for documents with
> multi value fields tremendously. A document boost of 5 combined with 15
> values in a multi value field results in scores above 1,000,000,000, while a
> boost of 0,5 results in scores below 0,001. The error is not present in Solr
> 3.6.
> Thomas Egense and I have tracked it down to a change in Solr DocumentBuilder
> committed 20110827 (@1162347) by Mike McCandless, as part of work done on
> LUCENE-2308. The problem is that Lucene multiplies the boosts of multiple
> instances of the same field when updating the index.
> The old DocumentBuilder, used in Lucene 3.6, handled this by calculating the
> score for the field (docBoost*fieldBoost) and assigning it to the first
> instance of the field, then setting the boost to 1.0f and assigning that to
> subsequent instances of the field. This effectively assigned
> docBoost*fieldBoost to the field, regardless of the number of instances.
> The updated DocumentBuilder (see
> https://svn.apache.org/viewvc/lucene/dev/branches/lucene_solr_4_0/solr/core/src/java/org/apache/solr/update/DocumentBuilder.java?revision=1388778&view=markup),
> used in Lucene 4 BETA & trunk, also assigns docBoost*fieldBoost to the first
> instance of the field. Then it sets fieldBoost = docBoost and continues to
> assign docBoost*fieldBoost to subsequent instances. Using the example
> mentioned above, the generated IndexableFields will get assigned boosts of 5,
> 5*5, 5*5... 5*5. As Lucene multiplies all the values, 15 instances of the
> same field will have a collective boost of 5*25^14.
> This can be demonstrated with the Solr tutorial example by indexing the
> sample documents and adding the document
> {code:xml}
>
>
> Insane score Example. Score = 10E9
> Document boost broken for multivalued fields
> Thomas Egense and Toke Eskildsen
> Test
> bug
> insane_boost
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
>
>
> {code}
> The _manu_ & _features_-fields gets copied to _text_ and a search for
> _thomas_ matches the _text_-field with query explanation
> {code:xml}
>
> 2.44373361E10 = (MATCH) weight(text:thomas in 0) [DefaultSimilarity], result
> of:
> 2.44373361E10 = fieldWeight in 0, product of:
> 1.0 = tf(freq=1.0), with freq of:
> 1.0 = termFreq=1.0
> 3.2512918 = idf(docFreq=3, maxDocs=38)
> 7.5161928E9 = fieldNorm(doc=0)
>
> {code}
> Thomas and I are too pressed for time to attempt a proper patch at the
> moment, but we guess that a reversion to the old algorithm of assigning the
> combined boost to the first instance and 1.0f to all subsequent instances
> would work?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3875) Document boost does not work correctly when using multi-valued fields
[
https://issues.apache.org/jira/browse/SOLR-3875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13462224#comment-13462224
]
Robert Muir commented on SOLR-3875:
---
Is it possible we can improve tests somehow in this area?
Both Mike and I have been ensnared by this file recently and introduced sneaky
bugs.
Additional tests showing we didnt introduce any regressions (especially with
the crazy field types) would make me very happy.
Additional tests that just make this less fragile would be very good for the
future.
> Document boost does not work correctly when using multi-valued fields
> -
>
> Key: SOLR-3875
> URL: https://issues.apache.org/jira/browse/SOLR-3875
> Project: Solr
> Issue Type: Bug
> Components: Schema and Analysis, update
>Affects Versions: 4.0-BETA
>Reporter: Toke Eskildsen
>Assignee: Hoss Man
>Priority: Critical
> Fix For: 4.0, 4.1, 5.0
>
> Attachments: SOLR-3875.patch
>
>
> In Solr 4 BETA & trunk, document boosts skews the ranking for documents with
> multi value fields tremendously. A document boost of 5 combined with 15
> values in a multi value field results in scores above 1,000,000,000, while a
> boost of 0,5 results in scores below 0,001. The error is not present in Solr
> 3.6.
> Thomas Egense and I have tracked it down to a change in Solr DocumentBuilder
> committed 20110827 (@1162347) by Mike McCandless, as part of work done on
> LUCENE-2308. The problem is that Lucene multiplies the boosts of multiple
> instances of the same field when updating the index.
> The old DocumentBuilder, used in Lucene 3.6, handled this by calculating the
> score for the field (docBoost*fieldBoost) and assigning it to the first
> instance of the field, then setting the boost to 1.0f and assigning that to
> subsequent instances of the field. This effectively assigned
> docBoost*fieldBoost to the field, regardless of the number of instances.
> The updated DocumentBuilder (see
> https://svn.apache.org/viewvc/lucene/dev/branches/lucene_solr_4_0/solr/core/src/java/org/apache/solr/update/DocumentBuilder.java?revision=1388778&view=markup),
> used in Lucene 4 BETA & trunk, also assigns docBoost*fieldBoost to the first
> instance of the field. Then it sets fieldBoost = docBoost and continues to
> assign docBoost*fieldBoost to subsequent instances. Using the example
> mentioned above, the generated IndexableFields will get assigned boosts of 5,
> 5*5, 5*5... 5*5. As Lucene multiplies all the values, 15 instances of the
> same field will have a collective boost of 5*25^14.
> This can be demonstrated with the Solr tutorial example by indexing the
> sample documents and adding the document
> {code:xml}
>
>
> Insane score Example. Score = 10E9
> Document boost broken for multivalued fields
> Thomas Egense and Toke Eskildsen
> Test
> bug
> insane_boost
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
>
>
> {code}
> The _manu_ & _features_-fields gets copied to _text_ and a search for
> _thomas_ matches the _text_-field with query explanation
> {code:xml}
>
> 2.44373361E10 = (MATCH) weight(text:thomas in 0) [DefaultSimilarity], result
> of:
> 2.44373361E10 = fieldWeight in 0, product of:
> 1.0 = tf(freq=1.0), with freq of:
> 1.0 = termFreq=1.0
> 3.2512918 = idf(docFreq=3, maxDocs=38)
> 7.5161928E9 = fieldNorm(doc=0)
>
> {code}
> Thomas and I are too pressed for time to attempt a proper patch at the
> moment, but we guess that a reversion to the old algorithm of assigning the
> combined boost to the first instance and 1.0f to all subsequent instances
> would work?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3875) Document boost does not work correctly when using multi-valued fields
[
https://issues.apache.org/jira/browse/SOLR-3875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13462210#comment-13462210
]
Hoss Man commented on SOLR-3875:
Committed revision 1389636. -4x
> Document boost does not work correctly when using multi-valued fields
> -
>
> Key: SOLR-3875
> URL: https://issues.apache.org/jira/browse/SOLR-3875
> Project: Solr
> Issue Type: Bug
> Components: Schema and Analysis, update
>Affects Versions: 4.0-BETA
>Reporter: Toke Eskildsen
>Assignee: Hoss Man
>Priority: Critical
> Fix For: 4.0, 4.1, 5.0
>
> Attachments: SOLR-3875.patch
>
>
> In Solr 4 BETA & trunk, document boosts skews the ranking for documents with
> multi value fields tremendously. A document boost of 5 combined with 15
> values in a multi value field results in scores above 1,000,000,000, while a
> boost of 0,5 results in scores below 0,001. The error is not present in Solr
> 3.6.
> Thomas Egense and I have tracked it down to a change in Solr DocumentBuilder
> committed 20110827 (@1162347) by Mike McCandless, as part of work done on
> LUCENE-2308. The problem is that Lucene multiplies the boosts of multiple
> instances of the same field when updating the index.
> The old DocumentBuilder, used in Lucene 3.6, handled this by calculating the
> score for the field (docBoost*fieldBoost) and assigning it to the first
> instance of the field, then setting the boost to 1.0f and assigning that to
> subsequent instances of the field. This effectively assigned
> docBoost*fieldBoost to the field, regardless of the number of instances.
> The updated DocumentBuilder (see
> https://svn.apache.org/viewvc/lucene/dev/branches/lucene_solr_4_0/solr/core/src/java/org/apache/solr/update/DocumentBuilder.java?revision=1388778&view=markup),
> used in Lucene 4 BETA & trunk, also assigns docBoost*fieldBoost to the first
> instance of the field. Then it sets fieldBoost = docBoost and continues to
> assign docBoost*fieldBoost to subsequent instances. Using the example
> mentioned above, the generated IndexableFields will get assigned boosts of 5,
> 5*5, 5*5... 5*5. As Lucene multiplies all the values, 15 instances of the
> same field will have a collective boost of 5*25^14.
> This can be demonstrated with the Solr tutorial example by indexing the
> sample documents and adding the document
> {code:xml}
>
>
> Insane score Example. Score = 10E9
> Document boost broken for multivalued fields
> Thomas Egense and Toke Eskildsen
> Test
> bug
> insane_boost
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
>
>
> {code}
> The _manu_ & _features_-fields gets copied to _text_ and a search for
> _thomas_ matches the _text_-field with query explanation
> {code:xml}
>
> 2.44373361E10 = (MATCH) weight(text:thomas in 0) [DefaultSimilarity], result
> of:
> 2.44373361E10 = fieldWeight in 0, product of:
> 1.0 = tf(freq=1.0), with freq of:
> 1.0 = termFreq=1.0
> 3.2512918 = idf(docFreq=3, maxDocs=38)
> 7.5161928E9 = fieldNorm(doc=0)
>
> {code}
> Thomas and I are too pressed for time to attempt a proper patch at the
> moment, but we guess that a reversion to the old algorithm of assigning the
> combined boost to the first instance and 1.0f to all subsequent instances
> would work?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3875) Document boost does not work correctly when using multi-valued fields
[
https://issues.apache.org/jira/browse/SOLR-3875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13462199#comment-13462199
]
Hoss Man commented on SOLR-3875:
Committed revision 1389628. - trunk
> Document boost does not work correctly when using multi-valued fields
> -
>
> Key: SOLR-3875
> URL: https://issues.apache.org/jira/browse/SOLR-3875
> Project: Solr
> Issue Type: Bug
> Components: Schema and Analysis, update
>Affects Versions: 4.0-BETA
>Reporter: Toke Eskildsen
>Assignee: Hoss Man
>Priority: Critical
> Fix For: 4.0, 4.1, 5.0
>
> Attachments: SOLR-3875.patch
>
>
> In Solr 4 BETA & trunk, document boosts skews the ranking for documents with
> multi value fields tremendously. A document boost of 5 combined with 15
> values in a multi value field results in scores above 1,000,000,000, while a
> boost of 0,5 results in scores below 0,001. The error is not present in Solr
> 3.6.
> Thomas Egense and I have tracked it down to a change in Solr DocumentBuilder
> committed 20110827 (@1162347) by Mike McCandless, as part of work done on
> LUCENE-2308. The problem is that Lucene multiplies the boosts of multiple
> instances of the same field when updating the index.
> The old DocumentBuilder, used in Lucene 3.6, handled this by calculating the
> score for the field (docBoost*fieldBoost) and assigning it to the first
> instance of the field, then setting the boost to 1.0f and assigning that to
> subsequent instances of the field. This effectively assigned
> docBoost*fieldBoost to the field, regardless of the number of instances.
> The updated DocumentBuilder (see
> https://svn.apache.org/viewvc/lucene/dev/branches/lucene_solr_4_0/solr/core/src/java/org/apache/solr/update/DocumentBuilder.java?revision=1388778&view=markup),
> used in Lucene 4 BETA & trunk, also assigns docBoost*fieldBoost to the first
> instance of the field. Then it sets fieldBoost = docBoost and continues to
> assign docBoost*fieldBoost to subsequent instances. Using the example
> mentioned above, the generated IndexableFields will get assigned boosts of 5,
> 5*5, 5*5... 5*5. As Lucene multiplies all the values, 15 instances of the
> same field will have a collective boost of 5*25^14.
> This can be demonstrated with the Solr tutorial example by indexing the
> sample documents and adding the document
> {code:xml}
>
>
> Insane score Example. Score = 10E9
> Document boost broken for multivalued fields
> Thomas Egense and Toke Eskildsen
> Test
> bug
> insane_boost
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
>
>
> {code}
> The _manu_ & _features_-fields gets copied to _text_ and a search for
> _thomas_ matches the _text_-field with query explanation
> {code:xml}
>
> 2.44373361E10 = (MATCH) weight(text:thomas in 0) [DefaultSimilarity], result
> of:
> 2.44373361E10 = fieldWeight in 0, product of:
> 1.0 = tf(freq=1.0), with freq of:
> 1.0 = termFreq=1.0
> 3.2512918 = idf(docFreq=3, maxDocs=38)
> 7.5161928E9 = fieldNorm(doc=0)
>
> {code}
> Thomas and I are too pressed for time to attempt a proper patch at the
> moment, but we guess that a reversion to the old algorithm of assigning the
> combined boost to the first instance and 1.0f to all subsequent instances
> would work?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3875) Document boost does not work correctly when using multi-valued fields
[
https://issues.apache.org/jira/browse/SOLR-3875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13462115#comment-13462115
]
Michael McCandless commented on SOLR-3875:
--
Thanks Hoss, patch looks great!
> Document boost does not work correctly when using multi-valued fields
> -
>
> Key: SOLR-3875
> URL: https://issues.apache.org/jira/browse/SOLR-3875
> Project: Solr
> Issue Type: Bug
> Components: Schema and Analysis, update
>Affects Versions: 4.0-BETA
>Reporter: Toke Eskildsen
>Priority: Critical
> Fix For: 4.0, 4.1, 5.0
>
> Attachments: SOLR-3875.patch
>
>
> In Solr 4 BETA & trunk, document boosts skews the ranking for documents with
> multi value fields tremendously. A document boost of 5 combined with 15
> values in a multi value field results in scores above 1,000,000,000, while a
> boost of 0,5 results in scores below 0,001. The error is not present in Solr
> 3.6.
> Thomas Egense and I have tracked it down to a change in Solr DocumentBuilder
> committed 20110827 (@1162347) by Mike McCandless, as part of work done on
> LUCENE-2308. The problem is that Lucene multiplies the boosts of multiple
> instances of the same field when updating the index.
> The old DocumentBuilder, used in Lucene 3.6, handled this by calculating the
> score for the field (docBoost*fieldBoost) and assigning it to the first
> instance of the field, then setting the boost to 1.0f and assigning that to
> subsequent instances of the field. This effectively assigned
> docBoost*fieldBoost to the field, regardless of the number of instances.
> The updated DocumentBuilder (see
> https://svn.apache.org/viewvc/lucene/dev/branches/lucene_solr_4_0/solr/core/src/java/org/apache/solr/update/DocumentBuilder.java?revision=1388778&view=markup),
> used in Lucene 4 BETA & trunk, also assigns docBoost*fieldBoost to the first
> instance of the field. Then it sets fieldBoost = docBoost and continues to
> assign docBoost*fieldBoost to subsequent instances. Using the example
> mentioned above, the generated IndexableFields will get assigned boosts of 5,
> 5*5, 5*5... 5*5. As Lucene multiplies all the values, 15 instances of the
> same field will have a collective boost of 5*25^14.
> This can be demonstrated with the Solr tutorial example by indexing the
> sample documents and adding the document
> {code:xml}
>
>
> Insane score Example. Score = 10E9
> Document boost broken for multivalued fields
> Thomas Egense and Toke Eskildsen
> Test
> bug
> insane_boost
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
>
>
> {code}
> The _manu_ & _features_-fields gets copied to _text_ and a search for
> _thomas_ matches the _text_-field with query explanation
> {code:xml}
>
> 2.44373361E10 = (MATCH) weight(text:thomas in 0) [DefaultSimilarity], result
> of:
> 2.44373361E10 = fieldWeight in 0, product of:
> 1.0 = tf(freq=1.0), with freq of:
> 1.0 = termFreq=1.0
> 3.2512918 = idf(docFreq=3, maxDocs=38)
> 7.5161928E9 = fieldNorm(doc=0)
>
> {code}
> Thomas and I are too pressed for time to attempt a proper patch at the
> moment, but we guess that a reversion to the old algorithm of assigning the
> combined boost to the first instance and 1.0f to all subsequent instances
> would work?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3875) Document boost does not work correctly when using multi-valued fields
[
https://issues.apache.org/jira/browse/SOLR-3875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13462105#comment-13462105
]
Mark Miller commented on SOLR-3875:
---
bq. patch with proposed test & fix
+1
I applied the patch, inspected the fix, inspected the test. It looks right to
me.
I also ran all tests, and verified the new test fails as expected without the
fix.
> Document boost does not work correctly when using multi-valued fields
> -
>
> Key: SOLR-3875
> URL: https://issues.apache.org/jira/browse/SOLR-3875
> Project: Solr
> Issue Type: Bug
> Components: Schema and Analysis, update
>Affects Versions: 4.0-BETA
>Reporter: Toke Eskildsen
>Priority: Critical
> Fix For: 4.0, 4.1, 5.0
>
> Attachments: SOLR-3875.patch
>
>
> In Solr 4 BETA & trunk, document boosts skews the ranking for documents with
> multi value fields tremendously. A document boost of 5 combined with 15
> values in a multi value field results in scores above 1,000,000,000, while a
> boost of 0,5 results in scores below 0,001. The error is not present in Solr
> 3.6.
> Thomas Egense and I have tracked it down to a change in Solr DocumentBuilder
> committed 20110827 (@1162347) by Mike McCandless, as part of work done on
> LUCENE-2308. The problem is that Lucene multiplies the boosts of multiple
> instances of the same field when updating the index.
> The old DocumentBuilder, used in Lucene 3.6, handled this by calculating the
> score for the field (docBoost*fieldBoost) and assigning it to the first
> instance of the field, then setting the boost to 1.0f and assigning that to
> subsequent instances of the field. This effectively assigned
> docBoost*fieldBoost to the field, regardless of the number of instances.
> The updated DocumentBuilder (see
> https://svn.apache.org/viewvc/lucene/dev/branches/lucene_solr_4_0/solr/core/src/java/org/apache/solr/update/DocumentBuilder.java?revision=1388778&view=markup),
> used in Lucene 4 BETA & trunk, also assigns docBoost*fieldBoost to the first
> instance of the field. Then it sets fieldBoost = docBoost and continues to
> assign docBoost*fieldBoost to subsequent instances. Using the example
> mentioned above, the generated IndexableFields will get assigned boosts of 5,
> 5*5, 5*5... 5*5. As Lucene multiplies all the values, 15 instances of the
> same field will have a collective boost of 5*25^14.
> This can be demonstrated with the Solr tutorial example by indexing the
> sample documents and adding the document
> {code:xml}
>
>
> Insane score Example. Score = 10E9
> Document boost broken for multivalued fields
> Thomas Egense and Toke Eskildsen
> Test
> bug
> insane_boost
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
>
>
> {code}
> The _manu_ & _features_-fields gets copied to _text_ and a search for
> _thomas_ matches the _text_-field with query explanation
> {code:xml}
>
> 2.44373361E10 = (MATCH) weight(text:thomas in 0) [DefaultSimilarity], result
> of:
> 2.44373361E10 = fieldWeight in 0, product of:
> 1.0 = tf(freq=1.0), with freq of:
> 1.0 = termFreq=1.0
> 3.2512918 = idf(docFreq=3, maxDocs=38)
> 7.5161928E9 = fieldNorm(doc=0)
>
> {code}
> Thomas and I are too pressed for time to attempt a proper patch at the
> moment, but we guess that a reversion to the old algorithm of assigning the
> combined boost to the first instance and 1.0f to all subsequent instances
> would work?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3875) Document boost does not work correctly when using multi-valued fields
[
https://issues.apache.org/jira/browse/SOLR-3875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13462096#comment-13462096
]
Yonik Seeley commented on SOLR-3875:
+1, fix looks good!
> Document boost does not work correctly when using multi-valued fields
> -
>
> Key: SOLR-3875
> URL: https://issues.apache.org/jira/browse/SOLR-3875
> Project: Solr
> Issue Type: Bug
> Components: Schema and Analysis, update
>Affects Versions: 4.0-BETA
>Reporter: Toke Eskildsen
>Priority: Critical
> Fix For: 4.0, 4.1, 5.0
>
> Attachments: SOLR-3875.patch
>
>
> In Solr 4 BETA & trunk, document boosts skews the ranking for documents with
> multi value fields tremendously. A document boost of 5 combined with 15
> values in a multi value field results in scores above 1,000,000,000, while a
> boost of 0,5 results in scores below 0,001. The error is not present in Solr
> 3.6.
> Thomas Egense and I have tracked it down to a change in Solr DocumentBuilder
> committed 20110827 (@1162347) by Mike McCandless, as part of work done on
> LUCENE-2308. The problem is that Lucene multiplies the boosts of multiple
> instances of the same field when updating the index.
> The old DocumentBuilder, used in Lucene 3.6, handled this by calculating the
> score for the field (docBoost*fieldBoost) and assigning it to the first
> instance of the field, then setting the boost to 1.0f and assigning that to
> subsequent instances of the field. This effectively assigned
> docBoost*fieldBoost to the field, regardless of the number of instances.
> The updated DocumentBuilder (see
> https://svn.apache.org/viewvc/lucene/dev/branches/lucene_solr_4_0/solr/core/src/java/org/apache/solr/update/DocumentBuilder.java?revision=1388778&view=markup),
> used in Lucene 4 BETA & trunk, also assigns docBoost*fieldBoost to the first
> instance of the field. Then it sets fieldBoost = docBoost and continues to
> assign docBoost*fieldBoost to subsequent instances. Using the example
> mentioned above, the generated IndexableFields will get assigned boosts of 5,
> 5*5, 5*5... 5*5. As Lucene multiplies all the values, 15 instances of the
> same field will have a collective boost of 5*25^14.
> This can be demonstrated with the Solr tutorial example by indexing the
> sample documents and adding the document
> {code:xml}
>
>
> Insane score Example. Score = 10E9
> Document boost broken for multivalued fields
> Thomas Egense and Toke Eskildsen
> Test
> bug
> insane_boost
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
> something else
>
>
> {code}
> The _manu_ & _features_-fields gets copied to _text_ and a search for
> _thomas_ matches the _text_-field with query explanation
> {code:xml}
>
> 2.44373361E10 = (MATCH) weight(text:thomas in 0) [DefaultSimilarity], result
> of:
> 2.44373361E10 = fieldWeight in 0, product of:
> 1.0 = tf(freq=1.0), with freq of:
> 1.0 = termFreq=1.0
> 3.2512918 = idf(docFreq=3, maxDocs=38)
> 7.5161928E9 = fieldNorm(doc=0)
>
> {code}
> Thomas and I are too pressed for time to attempt a proper patch at the
> moment, but we guess that a reversion to the old algorithm of assigning the
> combined boost to the first instance and 1.0f to all subsequent instances
> would work?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
