Got it - now it makes more sense. (In that case, I question the use of
the term "index", though; I would view this instead as an "indexed
collection implementation" that's an alternative to basic file storage.
But that's just words; doesn't affect the implementation work or
performance work. :-))
On 9/17/13 2:35 PM, Steven Jacobs wrote:
The index actually has all of the data, so it is capable of recreating the
xml structure. We use a parent-child relationship to maintain an id for
each piece of data. For example, assume the following xml code:
<letter>
<title maxlength="10"> My letter</title>
<body>Here is the body </body>
<footer>
<greeting>Good Bye!</greeting>
</footer>
</letter>
>From this snippet we would get the following tuples (id, value, type):
0 "letter" element
0.0 "title" element
0.0.0 "maxlength" attribute
0.0.0.0 "10" text
0.0.1 "My Letter" text
0.1 "body" element
0.1.0 "Here is the body" text
0.2 "footer" element
0.2.0 "greeting" element
0.2.0.0 "Good Bye!" text
Then we store these tuples in a lucene index. This means that the tests I
ran on 100% of the data were actually producing the entire result from the
index. Intuitively, the index should really stand out when we are only
returning a small subset of the results, as collection will parse
everything and index will only look at the relevant data, and this does
seem to be the case (Finished in 1/4 of the time on my test).
At this point I am getting a memory overflow when trying to retrieve ALL
data from the index, so I am looking into Lucene's capability of
incremental results. I will post results when I get things working.
Steven
On Tue, Sep 17, 2013 at 2:04 PM, Vinayak Borkar <[email protected]> wrote:
Steven,
It might be useful to send a brief note describing how you use Lucene to
store XML documents.
Thanks,
Vinayak
On 9/17/13 12:56 PM, Michael Carey wrote:
PS (to my last mail) - I would have thought the index results would
still have to now be used to go get the full XML documents back? (Or is
this less of an index and more of an alternative complete storage
solution?)
On 9/17/13 11:33 AM, Steven Jacobs wrote:
My current theory is that collection faces the overhead of creating file
handlers 40,000 times while index creates a handler only once to read the
index results. I don't know if this is enough to produce the large
difference though.
Steven
On Tue, Sep 17, 2013 at 11:28 AM, Michael Carey <[email protected]>
wrote:
Interesting! I haven't followed enough yet, but now you have my
interest;. :-)
do you have an explanation for why your index wins even in the case of
100%?
(Not intuitive - maybe I am missing some details that would fill my
intuition gap.)
On 9/17/13 10:59 AM, Steven Jacobs wrote:
I ran a test on one of Preston's real-world data sets (Weather
collection) that had around 40,000 files. I am attaching the
results. There
are three graphs.
The first shows the time for returning the entire XML for all 40000
files. My index algorithm has huge gains over collection, no matter how
much of the data is returned.
The second shows how the two algorithms perform as the number of files
increases. Both linearly increase, but collection has a much higher
slope.
The last is just a one-point comparison for returning paths that only
exist in only 100 out of the 40000 files. Once again, index has a huge
advantage.
Steven
