[
https://issues.apache.org/jira/browse/SOLR-11725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16288086#comment-16288086
]
ASF subversion and git services commented on SOLR-11725:
--------------------------------------------------------
Commit 53f2d4aa3aa171d5f37284eba9ca56d987729796 in lucene-solr's branch
refs/heads/branch_7x from Chris Hostetter
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=53f2d4a ]
Beef up testing of json.facet 'refine:simple' when dealing with 'Long Tail'
terms
In an attempt to get more familiar with json.facet refinement, I set out to try
and refactor/generalize/clone
some of the existing facet.pivot refinement tests to assert that json.facet
could produce the same results.
This test is a baby step towards doing that: Cloning
DistributedFacetPivotLongTailTest into
DistributedFacetSimpleRefinementLongTailTest (with shared index building code).
Along the way, I learned that the core logic of 'refine:simple' is actually
quite different then how facet.field
& facet.pivot work (see discussion in SOLR-11733), so they do *NOT* produce the
same results in many "Long Tail"
Sitautions. As a result, many of the logic/assertions
inDistributedFacetSimpleRefinementLongTailTest are very
differnet then their counter parts in DistributedFacetPivotLongTailTest, with
detailed explanations in comments.
Hopefully this test will prove useful down the road to anyone who might want to
compare/contrast facet.pivot
with json.facet, and to prevent regressions in 'refine:simple' if/when we add
more complex refinement
approaches in the future.
There are also a few TODOs in the test related to some other small
discrepencies between json.facet and
stats.field that I opened along the way, indicating where the tests should be
modified once those issues are
addressed in json.facet...
- SOLR-11706: support for multivalued numeric fields in stats
- SOLR-11695: support for 'missing()' & 'num_vals()' (aka: 'count' from
stats.field) numeric stats
- SOLR-11725: switch from 'uncorrected stddev' to 'corrected stddev'
(cherry picked from commit 2990c88a927213177483b61fe8e6971df04fc3ed)
> json.facet's stddev() function should be changed to use the "Corrected sample
> stddev" formula
> ---------------------------------------------------------------------------------------------
>
> Key: SOLR-11725
> URL: https://issues.apache.org/jira/browse/SOLR-11725
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Reporter: Hoss Man
> Attachments: SOLR-11725.patch
>
>
> While working on some equivalence tests/demonstrations for
> {{facet.pivot+stats.field}} vs {{json.facet}} I noticed that the {{stddev}}
> calculations done between the two code paths can be measurably different, and
> realized this is due to them using very different code...
> * {{json.facet=foo:stddev(foo)}}
> ** {{StddevAgg.java}}
> ** {{Math.sqrt((sumSq/count)-Math.pow(sum/count, 2))}}
> * {{stats.field=\{!stddev=true\}foo}}
> ** {{StatsValuesFactory.java}}
> ** {{Math.sqrt(((count * sumOfSquares) - (sum * sum)) / (count * (count -
> 1.0D)))}}
> Since I"m not really a math guy, I consulting with a bunch of smart math/stat
> nerds I know online to help me sanity check if these equations (some how)
> reduced to eachother (In which case the discrepancies I was seeing in my
> results might have just been due to the order of intermediate operation
> execution & floating point rounding differences).
> They confirmed that the two bits of code are _not_ equivalent to each other,
> and explained that the code JSON Faceting is using is equivalent to the
> "Uncorrected sample stddev" formula, while StatsComponent's code is
> equivalent to the the "Corrected sample stddev" formula...
> https://en.wikipedia.org/wiki/Standard_deviation#Uncorrected_sample_standard_deviation
> When I told them that stuff like this is why no one likes mathematicians and
> pressed them to explain which one was the "most canonical" (or "most
> generally applicable" or "best") definition of stddev, I was told that:
> # This is something statisticians frequently disagree on
> # Practically speaking the diff between the calculations doesn't tend to
> differ significantly when count is "very large"
> # _"Corrected sample stddev" is more appropriate when comparing two
> distributions_
> Given that:
> * the primary usage of computing the stddev of a field/function against a
> Solr result set (or against a sub-set of results defined by a facet
> constraint) is probably to compare that distribution to a different Solr
> result set (or to compare N sub-sets of results defined by N facet
> constraints)
> * the size of the sets of documents (values) can be relatively small when
> computing stats over facet constraint sub-sets
> ...it seems like {{StddevAgg.java}} should be updated to use the "Corrected
> sample stddev" equation.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
