hello, I would like to have your opinions on the impact on relevance scoring in the scenario where multiple languages are indexed in a single index.
>> Besides, IMO, scoring / ordering documents in different >> languages is a bit like comparing apples and oranges. > Not too sure about that. If for instance you were to search for "firewall" > in a German/English index, the word may appear in documents in both > languages. Lucene's ranking algorithm is based on the number of tokens and > the number of occurrences in fields, and while of course the ratio may > vary (e.g. German tends to collate several words into one single entity, > resulting in one single token, while English rather uses phrases, > resulting in several tokens), I still think, assuming the user can > understand both, that it kind of makes sense to rank a short German > document where "firewall" occurs 10 times higher than an English document > where the same word occurs only 5 times. > In our case, it is "known" in which language the user is searching (because he tells us, and if he doesn't, we use the current GUI language). Results are returned so that results in the requested language are ordered on top, and within that, ordered by relevance. Results in other languages are also returned, and presented after the requested-language results, ordered by relevance. If the results in the requested language contain say one which has term A and one which has term B, their positions in the relevance ranking (within the requested-language results on top) can be influenced by occurrences of terms A and B in the other languages, if a single search is used. I agree to the apples/oranges remark: if a term occurs in more than one language, likely its IDF frequency is different for each language, so to have the best relevance ranking there should be separate indexes for each language. And searches should be really separate searches (no MultiSearcher which would produce combined relevance scores). So the results should also be presented as several, separate result sets. Does anyone have experience with this ? Opinions ? Is the improved relevance per language worth the "hassle" of having separate indexes, doing separate searches and presenting results per language ? We do already take care of using appropriate stopwords/differnt analyzers when indexing and searching a particular language, but that's a different issue obviously. thanks in advance, Heikki Doeleman -- View this message in context: http://lucene.472066.n3.nabble.com/Designing-a-multilingual-index-tp688766p3628920.html Sent from the Lucene - Java Users mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
