[ https://issues.apache.org/jira/browse/SOLR-2403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13007497#comment-13007497 ]
Toke Eskildsen commented on SOLR-2403: -------------------------------------- Dividing by shard count is fairly risky. An example could be the shards {code} Shard 1: A(9) B(6) C(10) D(8) Shard 2: A(4) B(5) C(4) D(3) {code} where the request of the top-3 elements with mincount=5 from each shard would give the merged result {code} B(11) C(10) {code} where the correct result would be {code} A(13) B(11) C(14) D(11) {code} The problem with using mincount=1 for each shard-call is of course that the single shard result sets needs to be humongous in order to ensure that the correct values are returned, when the field contains many value with low count and few values with high count. With shards like {code} Shard 1: A(1) B(1) C(1) D(1) E(1) F(9) G(1) H(1) Shard 2: A(1) B(1) C(1) D(1) E(1) F(1) G(1) H(10) {code} and a request for mincount=10, all terms must be returned from both shards in order to get the result {code} F(10) H(11) {code} As you, Yonik, point out, a variant of the problem exists when sorting on count. However, for count it is mitigated by the fact that the results from the individual shards are sorted by the selecting key (count). This means that the chance of missing or miscounting tags is low and can be lowered further by relatively little over-requesting. With lexical sorting, the selecting key (count again) is independent of the sorting key. Over-requesting helps, but only linear to the fraction of the full result-set from each shard that is requested. Furthermore, the need for over-requesting grows with the number of shards as the overlapping hills can be smaller while still summing up to mincount. I do not have any real solution for the problem. One minor improvement would be a collector that kept collecting terms with a mincount=y until limit=n or the number of collected terms with mincount=x was equal to m, where x is the original mincount and y is dependent on the number of shards. This would at least stop the collection process when the result set was guaranteed to contain enough values above the given threshold. It would work well with spikes but poorly with hills just below mincount x and it would still not guarantee correctness of the sums of the counts, only correctness of the terms. > Problem with facet.sort=lex, shards, and facet.mincount > ------------------------------------------------------- > > Key: SOLR-2403 > URL: https://issues.apache.org/jira/browse/SOLR-2403 > Project: Solr > Issue Type: Bug > Components: search > Affects Versions: 4.0 > Environment: RHEL5, Ubuntu 10.04 > Reporter: Peter Cline > > I tested this on a recent trunk snapshot (2/25), haven't verified with 3.1 or > 1.4.1. I can if necessary and update. > Solr is not returning the proper number of facet values when sorting > alphabetically, using distributed search, and using a facet.mincount that > excludes some of the values in the first facet.limit values. > Easiest explained by example. Sorting alphabetically, the first 20 values > for my "subject_facet" field have few documents. 19 facet values have only 1 > document associated, and 1 has 2 documents. There are plenty after that have > more than 2. > {code} > http://localhost:8082/solr/select?q=*:*&facet=true&facet.field=subject_facet&facet.limit=20&facet.sort=lex&facet.mincount=2 > {code} > comes back with the expected 20 facet values with >= 2 documents associated. > If I add a shards parameter that points back to itself, the result is > different. > {code} > http://localhost:8082/solr/select?q=*:*&facet=true&facet.field=subject_facet&facet.limit=20&facet.sort=lex&facet.mincount=2&shards=localhost:8082/solr > {code} > comes back with only 1 facet value: the single value in the first 20 that had > more than 1 document. > It appears to me that mincount is ignored when doing the original query to > the shards, then applied afterwards. > Let me know if you need any more info. > Thanks, > Peter -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org