From: Erik Hatcher <[EMAIL PROTECTED]> Date: February 12, 2005 3:09:15 PM MST To: "Lucene Users List" <lucene-user@jakarta.apache.org> Subject: Re: Multiple Keywords/Keyphrases fields
The real question to answer is what types of queries you're planning on making. Rather than look at it from indexing forward, consider it from searching backwards.
How will users query using those keyword phrases?
Hi Erik. Good point.
There are two uses we are making of the keyphrases:
- Graphical Navigation: A Flash graphical browser will allow users to fly around in a space of documents, choosing what to be viewing: Authors, Keyphrases and Textual terms. In any of these cases, the "closeness" of any of the fields will govern how close they will appear graphically. In the case of authors, we will weight collaboration .. how often the authors work together. In the case of Keyphrases, we will want to use something like distance vectors like you show in the book using the cosine measure. Thus the keyphrases need to be separate entities within the document .. it would be a bug for us if the terms leaked across the separate kephrases within the document.
- Textual Search: In this case, we will have two ways to search the keyphrases. The first would be like the graphical navigation above where searching for "complex system" should require the terms to be in a single keyphrase. The second way will be looser, where we may simply pool the keyphrases with titles and abstract, and allow them all to be searched together within the document.
Does this make sense? So the question from the search standpoint is: do multiple instances of a field act like there are barriers across the instances, or are they somehow treated as a single instance somehow. In terms of the closeness calculation, for example, can we get separate term vectors for each instance of the keyphrase field, or will we get a single vector combining all the keyphrase terms within a single document?
I hope this is clear! Kinda hard to articulate.
Owen
Erik
On Feb 12, 2005, at 3:08 PM, Owen Densmore wrote:
I'm getting a bit more serious about the final form of our lucene index. Each document has DocNumber, Authors, Title, Abstract, and Keywords. By Keywords, I mean a comma separated list, each entry having possibly many terms in a phrase like:
temporal infomax, finite state automata, Markov chains,
conditional entropy, neural information processing
I presume I should be using a field "Keywords" which have many "entries" or "instances" per document (one per comma separated phrase). But I'm not sure the right way to handle all this. My assumption is that I should analyze them individually, just as we do for free text (the Abstract, for example), thus in the example above having 5 entries of the nature
doc.add(Field.Text("Keywords", "finite state automata"));
etc, analyzing them because these are author-supplied strings with no canonical form.
For guidance, I looked in the archive and found the attached email, but I didn't see the answer. (I'm not concerned about the dups, I presume that is equivalent to a boos of some sort) Does this seem right?
Thanks once again.
Owen
From: [EMAIL PROTECTED] <[EMAIL PROTECTED]> Subject: Multiple equal Fields? Date: Tue, 17 Feb 2004 12:47:58 +0100
Hi! What happens if I do this:
doc.add(Field.Text("foo", "bar")); doc.add(Field.Text("foo", "blah"));
Is there a field "foo" with value "blah" or are there two "foo"s (actually not
possible) or is there one "foo" with the values "bar" and "blah"?
And what does happen in this case:
doc.add(Field.Text("foo", "bar")); doc.add(Field.Text("foo", "bar")); doc.add(Field.Text("foo", "bar"));
Does lucene store this only once?
Timo
--------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
