Re: high memory usage by indexreader

Hi Ian, Thanks once again. I think the problem is mainly due to index being on NFS. Currently, there are other process running on the server with heavy disk IO on NFS. This has resulted in sharing of disk IO. Thanks, Ashwin On Fri, Mar 22, 2013 at 5:45 AM, Ian Lea wrote: > I did ask if there wa

Re: high memory usage by indexreader

I did ask if there was anything else relevant you'd forgotten to mention ... How fast are general file operations on the NFS files? Your times are still extremely long and my guess is that your network/NFS setup are to blame. Can you run your code on the server that is exporting the index, if on

Re: high memory usage by indexreader

Hi Ian, Thanks for your reply. The index is on NFS and there is no storage local/near to machine. Operating system is CentOS 6.3 with linux 2.6. It has 16 Gigs of memory. By initializing the Indexreader, I mean opening the IndexReader. I timed my operations using System.currentTimeMillis and exe

Re: high memory usage by indexreader

That number of docs is far more than I've ever worked with but I'm still surprised it takes 4 minutes to initialize an index reader. What exactly do you mean by initialization? Show us the code that takes 4 minutes. What version of lucene? What OS? What disks? -- Ian. On Wed, Mar 20, 2013

Re: high memory usage by indexreader

Thanks Ian. Number of documents in index is 381,153,828. The data set size is 1.9TB. The index size of this dataset is 290G. It is single index. The following are the fields indexed for each of the document. 1. Document id : It is StoredField and is generally around 128 chars or more. 2. Text fie

Re: high memory usage by indexreader

Searching doesn't usually use that much memory, even on large indexes. What version of lucene are you on? How many docs in the index? What does a slow query look like (q.toString()) and what search method are you calling? Anything else relevant you forgot to tell us? Or google "lucene shardin

high memory usage by indexreader

Hi Everybody, I have created a single compound index which is of size 250 Gigs. I open a single index reader to search simple boolean queries. The process is consuming lot of memory search painfully slow. It seems that I will have to create multiple indexes and have multiple index readers. Can an