RE: Search scalability
Thanks a lot. I'll use RAMDirectory and post my results. -Original Message- From: Otis Gospodnetic [mailto:[EMAIL PROTECTED] Sent: Thursday, November 11, 2004 9:09 AM To: Lucene Users List Subject: Re: Search scalability If you load it explicitly, then all 800 MB will make it into RAM. It's easy to try, the API for this is super simple. Otis --- [EMAIL PROTECTED] wrote: > Does it take 800MB of RAM to load that index into a RAMDirectory? Or > are only some of the files loaded into RAM? > > --- Otis Gospodnetic <[EMAIL PROTECTED]> wrote: > > > Hello, > > > > 100 parallel searches going against a single index on a single disk > > means a lot of disk seeks all happening at once. One simple way of > > working around this is to load your FSDirectory into RAMDirectory. > > This should be faster (could you report your > > observations/comparisons?). You can also try using ramfs if you are > > using Linux. > > > > Otis > > > > --- Ravi <[EMAIL PROTECTED]> wrote: > > > > > We have one large index for a document repository of > > 800,000 > > > documents. > > > The size of the index is 800MB. When we do searches against > > the > > > index, > > > it takes 300-500ms for a single search. We wanted to test > > the > > > scalability and tried 100 parallel searches against the > > index with > > > the > > > same query and the average response time was 13 seconds. We > > used a > > > simple IndexSearcher. Same searcher object was shared by all > > the > > > searches. I'm sure people have success in configuring lucene > > for > > > better > > > scalability. Can somebody share their approach? > > > > > > Thanks > > > Ravi. > > > > > > > > > - > > > To unsubscribe, e-mail: > > [EMAIL PROTECTED] > > > For additional commands, e-mail: > > [EMAIL PROTECTED] > > > > > > > > > > > > > - > > To unsubscribe, e-mail: > > [EMAIL PROTECTED] > > For additional commands, e-mail: > > [EMAIL PROTECTED] > > > > > > > - > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Search scalability
If you load it explicitly, then all 800 MB will make it into RAM. It's easy to try, the API for this is super simple. Otis --- [EMAIL PROTECTED] wrote: > Does it take 800MB of RAM to load that index into a > RAMDirectory? Or are only some of the files loaded into RAM? > > --- Otis Gospodnetic <[EMAIL PROTECTED]> wrote: > > > Hello, > > > > 100 parallel searches going against a single index on a single > > disk > > means a lot of disk seeks all happening at once. One simple > > way of > > working around this is to load your FSDirectory into > > RAMDirectory. > > This should be faster (could you report your > > observations/comparisons?). You can also try using ramfs if > > you are > > using Linux. > > > > Otis > > > > --- Ravi <[EMAIL PROTECTED]> wrote: > > > > > We have one large index for a document repository of > > 800,000 > > > documents. > > > The size of the index is 800MB. When we do searches against > > the > > > index, > > > it takes 300-500ms for a single search. We wanted to test > > the > > > scalability and tried 100 parallel searches against the > > index with > > > the > > > same query and the average response time was 13 seconds. We > > used a > > > simple IndexSearcher. Same searcher object was shared by all > > the > > > searches. I'm sure people have success in configuring lucene > > for > > > better > > > scalability. Can somebody share their approach? > > > > > > Thanks > > > Ravi. > > > > > > > > > - > > > To unsubscribe, e-mail: > > [EMAIL PROTECTED] > > > For additional commands, e-mail: > > [EMAIL PROTECTED] > > > > > > > > > > > > > - > > To unsubscribe, e-mail: > > [EMAIL PROTECTED] > > For additional commands, e-mail: > > [EMAIL PROTECTED] > > > > > > > - > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Search scalability
Does it take 800MB of RAM to load that index into a RAMDirectory? Or are only some of the files loaded into RAM? --- Otis Gospodnetic <[EMAIL PROTECTED]> wrote: > Hello, > > 100 parallel searches going against a single index on a single > disk > means a lot of disk seeks all happening at once. One simple > way of > working around this is to load your FSDirectory into > RAMDirectory. > This should be faster (could you report your > observations/comparisons?). You can also try using ramfs if > you are > using Linux. > > Otis > > --- Ravi <[EMAIL PROTECTED]> wrote: > > > We have one large index for a document repository of > 800,000 > > documents. > > The size of the index is 800MB. When we do searches against > the > > index, > > it takes 300-500ms for a single search. We wanted to test > the > > scalability and tried 100 parallel searches against the > index with > > the > > same query and the average response time was 13 seconds. We > used a > > simple IndexSearcher. Same searcher object was shared by all > the > > searches. I'm sure people have success in configuring lucene > for > > better > > scalability. Can somebody share their approach? > > > > Thanks > > Ravi. > > > > > - > > To unsubscribe, e-mail: > [EMAIL PROTECTED] > > For additional commands, e-mail: > [EMAIL PROTECTED] > > > > > > > - > To unsubscribe, e-mail: > [EMAIL PROTECTED] > For additional commands, e-mail: > [EMAIL PROTECTED] > > - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Search scalability
Hello, 100 parallel searches going against a single index on a single disk means a lot of disk seeks all happening at once. One simple way of working around this is to load your FSDirectory into RAMDirectory. This should be faster (could you report your observations/comparisons?). You can also try using ramfs if you are using Linux. Otis --- Ravi <[EMAIL PROTECTED]> wrote: > We have one large index for a document repository of 800,000 > documents. > The size of the index is 800MB. When we do searches against the > index, > it takes 300-500ms for a single search. We wanted to test the > scalability and tried 100 parallel searches against the index with > the > same query and the average response time was 13 seconds. We used a > simple IndexSearcher. Same searcher object was shared by all the > searches. I'm sure people have success in configuring lucene for > better > scalability. Can somebody share their approach? > > Thanks > Ravi. > > - > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Search scalability
We have one large index for a document repository of 800,000 documents. The size of the index is 800MB. When we do searches against the index, it takes 300-500ms for a single search. We wanted to test the scalability and tried 100 parallel searches against the index with the same query and the average response time was 13 seconds. We used a simple IndexSearcher. Same searcher object was shared by all the searches. I'm sure people have success in configuring lucene for better scalability. Can somebody share their approach? Thanks Ravi. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
