[
https://issues.apache.org/jira/browse/LUCENE-5476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13917414#comment-13917414
]
Rob Audenaerde edited comment on LUCENE-5476 at 3/2/14 2:01 PM:
----------------------------------------------------------------
Thanks all for the insight!
My implementation is not correct in the fact that I indeed run over all the
bits in the returning FixedBitSet, and will work better when only looking at
the set bits (the real results). The sampling I have implemented will sample
the total set of documents which is correct in the MatchAllDocsQuery, but will
behave worse and worse the fewer hits are returned; as the chance to sample a
hit decreases fast.
The reason I choose to do random sampling in bins instead of just using modulo
is to prevent accidental periodicity-effects. For example, if a document is
created each day and you sample with modulo 7, you only get the 'monday'
results. This can cause weird biased results.
So my next tasks will be:
* implement {{xorshift}}, as it seems a good improvement over the
{{Math.random()}} (although I switched to hte faster {{Random.nextInt( n )}}).
* only sample the hits
* remove the package
was (Author: robau):
Thanks all for the insight!
My implementation is not correct in the fact that I indeed run over all the
bits in the returning FixedBitSet, and will work better when only looking at
the set bits (the real results). The sampling I have implemented will sample
the total set of documents which is correct in the MatchAllDocsQuery, but will
behave worse and worse the fewer hits are returned; as the chance to sample a
hit decreases fast.
The reason I choose to do random sampling in bins instead of just using modulo
is to prevent accidental periodicity-effects. For example, if a document is
created each day and you sample with modulo 7, you only get the 'monday'
results. This can cause weird biased results.
So my next tasks will be:
* implement {{xorshift}}, as it seems a good improvement over the
{{Math.random()}} (although I switched to hte faster {{Random.nextInt(n)}}).
* only sample the hits
* remove the package
> Facet sampling
> --------------
>
> Key: LUCENE-5476
> URL: https://issues.apache.org/jira/browse/LUCENE-5476
> Project: Lucene - Core
> Issue Type: Improvement
> Reporter: Rob Audenaerde
> Attachments: LUCENE-5476.patch, SamplingFacetsCollector.java
>
>
> With LUCENE-5339 facet sampling disappeared.
> When trying to display facet counts on large datasets (>10M documents)
> counting facets is rather expensive, as all the hits are collected and
> processed.
> Sampling greatly reduced this and thus provided a nice speedup. Could it be
> brought back?
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
