Strategy for large index files

Hi, my servlet application is running a large index of 20G. I don't think it can be loaded to RAM at one time. What are the general strategies to improve the search and write performance? Thanks

Re: Help running out of files

Ok I think I've fixed my original problem by converting everything to use commit() and never call close() except when the server shuts down. This means I'm not closing my IndexWriter or IndexSearcher after opening them. I periodically call commit() on the IndexWriter after indexing my documents.

Re: question about SearcherManager in version 3.5.0

These blog posts may also help describe SearcherManager and NRTManager: http://blog.mikemccandless.com/2011/09/lucenes-searchermanager-simplifies.html http://blog.mikemccandless.com/2011/11/near-real-time-readers-with-lucenes.html Mike McCandless http://blog.mikemccandless.com On Sat,

Re: question about SearcherManager in version 3.5.0

You only need one SearcherManager instance for every IndexWriter you have open (basically, one for your application). The SearcherManager class is thread-safe- that's the point- so all threads can retrieve their IndexSearchers from it without any additional synchronization in your code. So you ca

Re: Why read past EOF

Is the index accessed over NFS? Mike McCandless http://blog.mikemccandless.com On Fri, Jan 6, 2012 at 9:28 PM, superruiye wrote: > Hi, >   I use lucene 3.4.0 in a search project,but encounter a problem and i > don't know how to resolve. > I index and it run well,but one week or two(it appear tw

Why read past EOF

Hi, I use lucene 3.4.0 in a search project,but encounter a problem and i don't know how to resolve. I index and it run well,but one week or two(it appear two times,first run one week,second two),it can't be searched,and the trace at below. Thank you! java.io.IOException: read past EOF a

Re: Why read past EOF

and my index create is: IndexWriterConfig indexWriterConfig = new IndexWriterConfig(Version.LUCENE_34, getAnalyzer()); indexWriterConfig.setOpenMode(IndexWriterConfig.OpenMode.CREATE_OR_APPEND); writer = new IndexWriter(directory, indexWriterConfig); writer.commit(); -- View this message in conte

Re: Why read past EOF

and my IndexWriter create is: IndexWriterConfig indexWriterConfig = new IndexWriterConfig(Version.LUCENE_34, getAnalyzer()); indexWriterConfig.setOpenMode(IndexWriterConfig.OpenMode.CREATE_OR_APPEND); writer = new IndexWriter(directory, indexWriterConfig); writer.commit(); -- View this message i

question about SearcherManager in version 3.5.0

hi, i'm writing a normal web-search application with lucene 3.5.0. in version 3.5.0 lucene provides SearcherManager to manage multithreaded searching. but i don't know how to use this class. should i use it as singleton, or should i initialize an instance for each searching? expect for your rep

Re: Comparing Indexing Speed of Lucene 3.5 and 4.0

Hi Uwe, > Die, Maven, die :-) Well, I for myself have a love-hate-relationship to maven: its simple and works nice for deps management. also others can set it up quickly and IDE support is nice. But sometimes it does a bit too much (unexpected ;)) or is too complicated to customize. > (I assum

RE: Comparing Indexing Speed of Lucene 3.5 and 4.0

Hi, > > I mean my benchmarks show up > > to 300% improvement with 4.x versus older versions so something is > > weird ie. non-realistic here or there is a bug so lets figure this > > out. Can you profile you app and see if you find something suspicious? > > I'll try now and report back. > > It s

Re: Comparing Indexing Speed of Lucene 3.5 and 4.0

> I mean my benchmarks show up > to 300% improvement with 4.x versus older versions so something is > weird ie. non-realistic here or there is a bug so lets figure this > out. Can you profile you app and see if you find something suspicious? > I'll try now and report back. It seems to be largely