[
https://issues.apache.org/jira/browse/LUCENE-8010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16304092#comment-16304092
]
Robert Muir commented on LUCENE-8010:
-------------------------------------
Sorry for the long delay, here is a really rough test (short queries only, no
stopwords). I think we should go for it. This is similar to changes you see
with e.g. modifying BM25's idf formula to prevent negatives. It will definitely
change results but its not clear it hurts for at least the common case. The SPL
changes don't do anything.
{noformat}
0 new AxiomaticF1EXP()
map 0.4375 -> 0.3919
bpref 0.5056 -> 0.4915
1 new AxiomaticF1LOG()
map 0.4495 -> 0.4076
bpref 0.5089 -> 0.4955
2 new AxiomaticF2EXP()
identical search results
3 new AxiomaticF2LOG()
identical search results
4 new AxiomaticF3EXP(0.25f, 1)
map 0.4305 -> 0.4000
bpref 0.5016 -> 0.5007
5 new AxiomaticF3LOG(0.25f, 1)
map 0.4470 -> 0.4153
bpref 0.5069 -> 0.5060
6 new IBSimilarity(new DistributionSPL(), new LambdaDF(), new
NormalizationH2())
map 0.4319 -> 0.4319
bpref 0.4943 -> 0.4943
(exact score magnitude gets changed e.g. 24.1899 vs 24.1850 but ranking seems
to be the same?)
7 new IBSimilarity(new DistributionSPL(), new LambdaTTF(), new
NormalizationH2())
identical search results
{noformat}
> fix or sandbox similarities in core with problems
> -------------------------------------------------
>
> Key: LUCENE-8010
> URL: https://issues.apache.org/jira/browse/LUCENE-8010
> Project: Lucene - Core
> Issue Type: Bug
> Reporter: Robert Muir
> Attachments: LUCENE-8010.patch
>
>
> We want to support scoring optimizations such as LUCENE-4100 and LUCENE-7993,
> which put very minimal requirements on the similarity impl. Today
> similarities of various quality are in core and tests.
> The ones with problems currently have warnings in the javadocs about their
> bugs, and if the problems are severe enough, then they are also disabled in
> randomized testing too.
> IMO lucene core should only have practical functions that won't return
> {{NaN}} scores at times or cause relevance to go backwards if the user's
> stopfilter isn't configured perfectly. Also it is important for unit tests to
> not deal with broken or semi-broken sims, and the ones in core should pass
> all unit tests.
> I propose we move the buggy ones to sandbox and deprecate them. If they can
> be fixed we can put them back in core, otherwise bye-bye.
> FWIW tests developed in LUCENE-7997 document the following requirements:
> * scores are non-negative and finite.
> * score matches the explanation exactly.
> * internal explanations calculations are sane (e.g. sum of: and so on
> actually compute sums)
> * scores don't decrease as term frequencies increase: e.g. score(freq=N +
> 1) >= score(freq=N)
> * scores don't decrease as documents get shorter, e.g. score(len=M) >=
> score(len=M+1)
> * scores don't decrease as terms get rarer, e.g. score(term=N) >=
> score(term=N+1)
> * scoring works for floating point frequencies (e.g. sloppy phrase and
> span queries will work)
> * scoring works for reasonably large 64-bit statistic values (e.g.
> distributed search will work)
> * scoring works for reasonably large boost values (0 .. Integer.MAX_VALUE,
> e.g. query boosts will work)
> * scoring works for parameters randomized within valid ranges
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
