thomasg wrote:
Hi, we are currently intending to implement a document storage / search tool
using Jackrabbit and Lucene. We have been approached by a commercial search
and indexing organisation called ISYS who are suggesting the following
problems with using Lucene. We do have a requirement to store and search
large documents and the total document store will be large too. Any comments
on the following would be greatly appreciated.
I would tend to be sceptical of claims from someone who is trying to
sell you something: they are inherently biased. The best way to answer
performance questions is to benchmark. That way you'll also get
experience with each alternative.
Ease of development, availability of support, ease of debugging, cost,
and other aspects besides raw performance may also prove more important
long-term.
1) By default, Lucene only indexes the first 10,000 words from each
document. When increasing this default out-of-memory errors can occur. This
implies that documents, or large sections thereof, are loaded into memory.
ISYS has a very small memory footprint which is not affected by document
size nor number of documents.
Lucene does store a few bytes per word indexed while indexing. But I've
never seen folks complain about this as a significant consumer of
memory, even when they crank up the limit. How large are your large
documents? (Is it really useful to get hits on such large documents
anyway, or might it be more useful to get hits on sections of these?)
2) Lucene appears to be slow at indexing, at least by ISYS' standards.
Published performance benchmarks seem to vary between almost acceptable,
down to very poor. ISYS' file readers are already optimized for the fastest
text extraction possible.
How fast do you need to index? How fast does your collection change?
In most systems I've worked on, Lucene's indexing speed has not been a
bottleneck. For example, when crawling with Nutch, downloading pages
and html parsing are typically much slower than indexing. Lucene does
not include text extractors, so it's hard to know what ISIS is comparing
to there. Also, when benchmarking indexing, one should look at the rate
when the index is very large, not just when the index is small.
3) The Lucene documentation suggests it can be slow at searching and can get
slower and slower the larger your indexes get. The tipping point is where
the index size exceeds the amount of free memory in your machine. This also
implies that whole indexes, or large portions of them, are loaded into
memory.
Lucene does not normally store entire indexes in memory. Indexes much
larger than memory can be searched quickly. For very high traffic (tens
to hundreds of searches per second) indexes smaller than memory are
required, since otherwise the disk becomes a bottleneck. My rule of
thumb is that a 10M document index of 10k documents (100G of text)
easily gives sub-second average response time for typical queries.
The bigger the index, the more powerful the machine required. ISYS'
search speed is always proportional to the size of the result set.
That sound suspicious to me, unless by "size of result set" they mean
the total number of documents that include any query term, in which case
that is also true of Lucene. But if you're using an "AND" query, the
size of the result set is usually much smaller than that total.
Do they claim they can search billions of documents with a boolean query
that only matches a few documents in time proportional to the size of
those few matching documents? And all that on a 486 with 1MB of RAM?
That sounds like magic!
Index
size does not materially affect search speed and the index is never loaded
into memory.
Index size does not affect Lucene's search speed per-se: what matters is
the frequency of the search terms. And terms tend to have larger
frequencies in larger indexes.
It also appears that Lucene requires hands-on tuning to keep
its search speed acceptable.
I don't recommend tuning Lucene, but rather just using the out-of-the
box defaults. Folks tend to cause problems for themselves trying to
make things 5% faster by setting parameters to extreme values without
understanding the consequences of those parameters. This results in
lots of tuning-related messages on the mailing lists, but I am not
convinced many of these are required.
ISYS' indexes are self-managing and do not
require any maintenance to keep them searchable at full speed.
Lucene could make the same claim.
I suspect you can achieve adequate performance with either product. The
more salient difference will be that with Lucene you'd have the full
source code and mailing lists like this to help you. With ISYS you'd
have a black box, a detailed manual, and a phone number. So it depends
on your style of development: do you want to be able to peek inside the
search-engine, or do you want to be able to pick up the phone?
Cheers,
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
