Michaels got some great points (he the lucene master), especially
possibly turning off norms if you can, but for an index like that i'd
reccomwnd solr. Solr sharding can be scaled to billions (min a billion
or two anyway) with few limitations (of course there are a few). Plus
it has further caching options, indexreader refresh managment, etc etc
etc
- Mark
On Oct 29, 2008, at 10:30 PM, "Todd Benge" <[EMAIL PROTECTED]> wrote:
Thanks Mark. I appreciate the help.
I thought our memory may be low but wanted to verify there if there is
any way to control memory usage. I think we'll likely upgrade the
memory on the machines but that may just delay the inevitable.
Wondering if anyone else has encountered similar issues with indices
if a similar size. I've been thinking we will need to move to a
clustered solution and have been reading on hadoop, nutch, solr &
terracotta for possibilities such as index sharding.
Has anyone implemented a solution using hadoop or terracotta for a
large scale system? Just wondering the pro's / con's of the various
approaches.
Thanks,
Todd
On Wed, Oct 29, 2008 at 6:07 PM, Mark Miller <[EMAIL PROTECTED]>
wrote:
The term, terminfo, indexreader internals stuff is prob on the low
end
compared to the size of your field caches (needed for sorting). If
you are
sorting by String I think the space needed is 32 bits x number of
docs + an
array to hold all of the unique terms. So checking 300 million docs
(I know
you are actually breaking it up smaller than that, but for example)
and
ignoring things like String chars being variable byte lengths and
storing
the length, etc and randomly picking 50000 unique terms at 6 chars
per:
32 bits x 300000000 + 50000 x 6 x 16 bits to MB = 1 144.98138
megabytes
Thats per field your sorting on. If you are sorting on an int field
it
should be closer to 32 bits x num docs - shorts, 32 bits x num
docs, etc.
So you have those field caches, plus the IndexReader terminfo, term
stuff,
plus whatever RAM your app needs beyond Lucene. 4 gig might just
not *quite*
cut it is my guess.
Todd Benge wrote:
There's usually only a couple sort fields and a bunch of terms in
the
various indices. The terms are user entered on various media so the
number of terms is very large.
Thanks for the help.
Todd
On 10/29/08, Todd Benge <[EMAIL PROTECTED]> wrote:
Hi,
I'm the lead engineer for search on a large website using lucene
for
search.
We're indexing about 300M documents in ~ 100 indices. The
indices add
up to ~ 60G.
The indices are sorted into 4 different Multisearcher with the
largest
handling ~50G.
The code is basically like the following:
private static MultiSearcher searcher;
public void init(File files) {
IndexSearcer [] searchers = new IndexSearcher[files.length] ();
int i = 0;
for ( File file: files ) {
searchers[i++] = new
IndexSearcher(FSDirectory.getDirectory(file);
}
searcher = new MultiSearcher(searchers);
}
public Searcher getSearcher() {
return searcher;
}
We're seeing a high cache rate with Term & TermInfo in Lucene 2.4.
Performance is good but servers are consistently hanging with
OutOfMemory errors.
We're allocating 4G in the heap to each server.
Is there any way to control the amount of memory Lucene consume for
caching? Any other suggestions on fixing the memory errors?
Thanks,
Todd
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
