Hi all,
First of all, I’m not a Java developer, and a SolR newbie. I have worked with
Elasticsearch for some years (not contributing, just as a user), so I think I
have the basics of text search engines covered. I am always learning new things
though!
I created an index in SolR and used more-like-this on it, by passing a
document_id. My data has a special feature, which is that one of the fields is
called “description” but is only populated about 10% of the time. Most of the
time it is empty. I am using that field to query similar documents.
So I query the /mlt endpoint using these parameters (for example):
{q=id:"0c7c4d74-0f37-44ea-8933-cd2ee7964457”,
mlt=true,
mlt.fl=description,
mlt.mindf=1,
mlt.mintf=1,
mlt.maxqt=5,
wt=json,
mlt.interestingTerms=details}
The issue I have is that when retrieving the key scored terms
(interestingTerms), the code uses the total number of documents in the index,
not the total number of documents with populated “description” field. This is
where it’s done in the code:
https://github.com/apache/lucene-solr/blob/master/lucene/queries/src/java/org/apache/lucene/queries/mlt/MoreLikeThis.java#L651
The effect of this choice is that the “idf” does not vary much, given that
numDocs >> number of documents with “description”, so the key terms end up
being just the terms with the highest term frequencies.
It is inconsistent because the MLT-search then uses these extracted key terms
and scores all documents using an idf which is computed only on the subset of
documents with “description”. So one part of the MLT uses a different numDocs
than another part. This sounds like an odd choice, and not expected at all, and
I wonder if I’m missing something.
Best,
Maria
