Grant Ingersoll a écrit :
On Jun 27, 2007, at 8:51 AM, Samuel LEMOINE wrote:
Hi everyone !
I'm working on bibliographical researches on Lucene as an intern in
Lingway (which uses Lucene in its main product), and I'm currently
studying Lucene's file system.
There are several things I don't catch in Lucene's file system, and I
thought here was the right place to ask about those questions (I hope
it's the case actually).
The main resource I used is this document:
http://lucene.apache.org/java/2_1_0/fileformats.html
-in the .tvf file (Term Vector file) in Lucene 2.2.0, position &
offsets can be possibly given in the term vector... I don't
understand how it works, since there's only one .tvf per segment
(according to what I've understood), and in the architecture
described, there is no information given about the documents in which
appears each term stored in the TermVector (the informations
document-related are in the .tvd file I assume). The position/offset
informations seems to be simply a list of addresses, but how can be
known the document it refers to? Or is there one .tvf file per document?
Yes, offsets and positions can be associated with a term vector. When
you ask the IndexReader for a term vector, you give it the document
number and, optionally, a field, which it uses to go look up in the
tvd file the document location in the tvd file. The tvd file then
looks up the specific information in the tvf file. Have a look at the
TermVectorsReader for details on implementation.
-in the .prx file (prositions file), payloads are mentionned and
allow to attach meta-data... what's the purpose of such data? is
there a precise use, or is it only data for the sole user's use?
Payloads have a variety of uses. Search the java-dev archive for the
word Payload and you will find lots of discussion. I also have a few
slides on it in my ApacheCon Europe presentation at
http://cnlp.org/presentations/slides/AdvancedLuceneEU.pdf See also
http://wiki.apache.org/jakarta-lucene/Payload_Planning
Essentially, it can be used to store information on a term by term
level, things like font weight, or XML enclosing tag, or Part of
Speech. The sky really is the limit (that and your disk space) on
what can be stored in a payload.
-many adresses in many files are given under Delta shapes... Doesn't
it slacken the search among the index ? I mean, when a keyword is
looked for, in order to find its position in the right file, Lucene
must find the adress of the previous term and add the "delta"
address... but the previous term adress is also given by a delta
address, and so on, so that as far as I understand it, the whole file
must be climbed back, recursively finding the address of each term...
I assume I've misunderstood something, but don't know what.
Not quite sure what you are asking, but I will take a stab at it.
Have a look at the section on the Term Dictionary, specifically the
relationship between the tis file and the tii file. The storage
mechanism makes it very easy to find where the keyword is in the file
so that the rest of the information can be easily looked up.
HTH,
Grant
--------------------------
Grant Ingersoll
Center for Natural Language Processing
http://www.cnlp.org/tech/lucene.asp
Read the Lucene Java FAQ at http://wiki.apache.org/lucene-java/LuceneFAQ
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Thanks for the resources about payloads, I'll have a look over it.
About the positions/offsets in .tvf, please tell me if I've well
understood:
The .tvd provides the needed informations concerning the occurrences of
each term in documents, and thanks to these informations, Lucene is able
to determinate how many documents contain the term "foo".
Thus the position/offset data contained in .tvf can just consist in a
list of positions in the different documents containing "foo"
concatenated ? I mean, if foo appears in positions 1,30,65 in doc 0, and
positions 27 & 52 in doc 2, the .tvf will give "1 30 65 27 52" and
Lucene rests on .tvd to determine which positions belongs to which
document? (or rather "1 29 35 27 25" as it is delta-positions)
Hoping my interrogations will help other people ^^
Thanks,
Samuel
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
