You're right Yonik, that's what I meant. As for an example, let us place ourselves in two situations: - Situation A: one field ("text") where the texts of all documents are indexed; - Situation B: one field per language ("text_en", "text_fr", etc.) where only the texts of documents in that language are indexed.
Now, take a query with the French word "voiture" (means car). There are of course much more French documents than English ones containing this word. In situation A, the scores of documents will only depends on the tf of "voiture". French and English documents will be treated equally. In situation B, the idf of "voiture" in the field "text_en" will be very important due to the rarity of this word in English. English documents will then always be ranked higher. Then, take a query with the English word "car". This word is very common in French (means "because" but can also be used for other meanings). In situation A, French documents will be ranked higher because of their high tf. Situation B is better in this case as the frequency of "car" in French is not taken in account in the ranking. The score of English documents will be calculated normally while French documents will be ranked lower due to the very low idf (or even not appear if stop words are used). So far, situation B with a boost for documents in the same language as the query seems the more promising one (at least when dealing with the relevance problem). Another interesting solution is to use language dependant stop words in situation A to avoid altering idf for other language. But this incurs other problems (stop words are not always desirable). I hope this example clarifies the identified problem of cross-lingual retrieval in Lucene. Sorry if it is not as clear as I would like (English is not my mother tongue). Nicolas -----Message d'origine----- De : [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] De la part de Yonik Seeley Envoyé : dimanche 2 mars 2008 03:45 À : solr-user@lucene.apache.org Objet : Re: Proposition of a new feature: Dynamic Field Types On Sat, Mar 1, 2008 at 9:38 PM, Otis Gospodnetic <[EMAIL PROTECTED]> wrote: > I don't quite follow everything here (examples?), but I believe IDF of a term is not a per-field value, but "index-wide". I think Nicolas meant that idfs are field specific, and that is the case (index-wide, per field). -Yonik