Agree. On Mon, Feb 6, 2012 at 11:53 PM, Uwe Schindler <u...@thetaphi.de> wrote:
> Hi Cheng, > > all pros and cons are explained in those articles written by Mike! As soon > as there are harddisks in the game, there is a slowdown, what do you > expect? > If you need it faster, buy SSDs! :-) > > Uwe > > ----- > Uwe Schindler > H.-H.-Meier-Allee 63, D-28213 Bremen > http://www.thetaphi.de > eMail: u...@thetaphi.de > > > > -----Original Message----- > > From: Cheng [mailto:zhoucheng2...@gmail.com] > > Sent: Monday, February 06, 2012 4:45 PM > > To: java-user@lucene.apache.org > > Subject: Re: Configure writer to write to FSDirectory? > > > > Uwe, when I meant speed is slow, I didn't refer to instant visibility of > changes, > > but that the changes may be synchronized with FSDirectory when I use > > writer.commit(). > > > > When I use RAMDirectory, the writer.commit() seems much faster than using > > NRTManager built upon FSDirectory. So, I am guessing the difference is > the > > index synchronization. > > > > > > > > On Mon, Feb 6, 2012 at 11:40 PM, Uwe Schindler <u...@thetaphi.de> wrote: > > > > > Please review the following articles about NRT, absolutely instant > > > updates that are visible as they are done are almost impossible (even > > > with > > > RAMDirectory): > > > > > > http://goo.gl/mzAHt > > > http://goo.gl/5RoPx > > > http://goo.gl/vSJ7x > > > > > > Uwe > > > > > > ----- > > > Uwe Schindler > > > H.-H.-Meier-Allee 63, D-28213 Bremen > > > http://www.thetaphi.de > > > eMail: u...@thetaphi.de > > > > > > > -----Original Message----- > > > > From: Cheng [mailto:zhoucheng2...@gmail.com] > > > > Sent: Monday, February 06, 2012 4:27 PM > > > > To: java-user@lucene.apache.org > > > > Subject: Re: Configure writer to write to FSDirectory? > > > > > > > > Ian, > > > > > > > > I encountered an issue that I need to frequently update the index. > > > > The NRTManager seems not very helpful on this front as the speed is > > > > slower > > > than > > > > RAMDirectory is used. > > > > > > > > Any improvement advice? > > > > > > > > > > > > > > > > On Mon, Feb 6, 2012 at 10:24 PM, Cheng <zhoucheng2...@gmail.com> > > wrote: > > > > > > > > > That really helps! I will try it out. > > > > > > > > > > Thanks. > > > > > > > > > > > > > > > On Mon, Feb 6, 2012 at 10:12 PM, Ian Lea <ian....@gmail.com> > wrote: > > > > > > > > > >> You would use NRTManagerReopenThread as a standalone thread, not > > > > >> plugged into your Executor stuff. It is a utility class which > > > > >> you don't have to use. See the javadocs. > > > > >> > > > > >> But in your case I'd use it, to start with anyway. Fire it up > > > > >> with suitable settings and forget about it, except to call > > > > >> close() eventually. Once you've got things up and running you can > > > > >> tweak things as much as you want but you appear to be having > > > > >> trouble getting up and running. > > > > >> > > > > >> So ... somewhere in the initialisation code of your app, create > > > > >> an IndexWriter, NRTManager + ReopenThread and SearcherManager as > > > > >> outlined before. Then pass the NRTManager to any/all write > > > > >> methods or threads and the SearcherManager instance to any/all > > > > >> search methods or threads and you're done. If you want to use > > > > >> threads that are part of your ExecutorService, fine. Just wrap > > > > >> it all together in whatever combination of Thread or Runnable > instances > > you want. > > > > >> > > > > >> > > > > >> Does that help? > > > > >> > > > > >> > > > > >> -- > > > > >> Ian. > > > > >> > > > > >> > > > > >> > I don't understand this following portion: > > > > >> > > > > > >> > IndexWriter iw = new IndexWriter(whatever - some standard disk > > > > >> > index); NRTManager nrtm = new NRTManager(iw, null); > > > > >> > NRTManagerReopenThread ropt = new > > NRTManagerReopenThread(nrtm, > > > > >> > ...); ropt.setXxx(...); .... > > > > >> > ropt.start(); > > > > >> > > > > > >> > I have a java ExecutorServices instance running which take care > > > > >> > of my > > > > >> own > > > > >> > applications. I don't know how this NRTManagerReopenThread > > > > >> > works with my own ExecutorService instance. > > > > >> > > > > > >> > Can both work together? How can the NRTManagerReopenThread > > > > instance > > > > >> ropt be > > > > >> > plugged into my own multithreading framework? > > > > >> > > > > > >> > On Mon, Feb 6, 2012 at 8:17 PM, Ian Lea <ian....@gmail.com> > wrote: > > > > >> > > > > > >> >> If you can use NRTManager and SearcherManager things should be > > > > >> >> easy and blazingly fast rather than unbearably slow. The > > > > >> >> latter phrase is not one often associated with lucene. > > > > >> >> > > > > >> >> IndexWriter iw = new IndexWriter(whatever - some standard disk > > > > >> >> index); NRTManager nrtm = new NRTManager(iw, null); > > > > >> >> NRTManagerReopenThread ropt = new > > > > NRTManagerReopenThread(nrtm, > > > > >> >> ...); ropt.setXxx(...); ... > > > > >> >> ropt.start(); > > > > >> >> > > > > >> >> SearcherManager srchm = nrtm.getSearcherManager(b); > > > > >> >> > > > > >> >> Then add docs to your index via nrtm.addDocument(d), update > > > > >> >> with nrtm.updateDocument(...), and to search use > > > > >> >> > > > > >> >> IndexSearcher searcher = srchm.acquire(); try { search ... > > > > >> >> } finally { > > > > >> >> srchm.release(searcher); > > > > >> >> } > > > > >> >> > > > > >> >> All thread safe so you don't have to worry about any > > > > >> >> complications there. And I bet it'll be blindingly fast. > > > > >> >> > > > > >> >> Don't forget to close() things down at the end. > > > > >> >> > > > > >> >> > > > > >> >> -- > > > > >> >> Ian. > > > > >> >> > > > > >> >> > > > > >> >> > > > > >> >> On Mon, Feb 6, 2012 at 12:15 AM, Cheng > > > > >> >> <zhoucheng2...@gmail.com> > > > > >> wrote: > > > > >> >> > I was trying to, but don't know how to even I read some of > > > > >> >> > your > > > > >> blogs. > > > > >> >> > > > > > >> >> > On Sun, Feb 5, 2012 at 10:22 PM, Michael McCandless < > > > > >> >> > luc...@mikemccandless.com> wrote: > > > > >> >> > > > > > >> >> >> Are you using near-real-time readers? > > > > >> >> >> > > > > >> >> >> (IndexReader.open(IndexWriter)) > > > > >> >> >> > > > > >> >> >> Mike McCandless > > > > >> >> >> > > > > >> >> >> http://blog.mikemccandless.com > > > > >> >> >> > > > > >> >> >> On Sun, Feb 5, 2012 at 9:03 AM, Cheng > > > > >> >> >> <zhoucheng2...@gmail.com> > > > > >> wrote: > > > > >> >> >> > Hi Uwe, > > > > >> >> >> > > > > > >> >> >> > My challenge is that I need to update/modify the indexes > > > > >> frequently > > > > >> >> while > > > > >> >> >> > providing the search capability. I was trying to use > > > > >> >> >> > FSDirectory, > > > > >> but > > > > >> >> >> found > > > > >> >> >> > out that the reading and writing from/to FSDirectory is > > > > >> >> >> > unbearably > > > > >> >> slow. > > > > >> >> >> So > > > > >> >> >> > I now am trying the RAMDirectory, which is fast. > > > > >> >> >> > > > > > >> >> >> > I don't know of MMapDirectory, and wonder if it is as > > > > >> >> >> > fast as > > > > >> >> >> RAMDirectory. > > > > >> >> >> > > > > > >> >> >> > > > > > >> >> >> > On Sun, Feb 5, 2012 at 4:14 PM, Uwe Schindler > > > > >> >> >> > <u...@thetaphi.de> > > > > >> >> wrote: > > > > >> >> >> > > > > > >> >> >> >> Hi Cheng, > > > > >> >> >> >> > > > > >> >> >> >> It seems that you use a RAMDirectory for *caching*, > > > > >> >> >> >> otherwise it > > > > >> >> makes > > > > >> >> >> no > > > > >> >> >> >> sense to write changes back. In recent Lucene versions, > > > > >> >> >> >> this is > > > > >> not a > > > > >> >> >> good > > > > >> >> >> >> idea, especially for large indexes (RAMDirectory eats > > > > >> >> >> >> your heap > > > > >> >> space, > > > > >> >> >> >> allocates millions of small byte[] arrays,...). If you > > > > >> >> >> >> need > > > > >> something > > > > >> >> >> like > > > > >> >> >> >> a > > > > >> >> >> >> caching Directory and you are working on a 64bit > > > > >> >> >> >> platform, you > > > > >> can > > > > >> >> use > > > > >> >> >> >> MMapDirectory (where the operating system kernel manages > > > > >> >> >> >> the > > > > >> >> read/write > > > > >> >> >> >> between disk an memory). MMapDirectory is returned by > > > > >> >> >> >> default for > > > > >> >> >> >> FSDirectory.open() on most 64 bit platforms. The good > thing: > > > > >> >> >> >> the > > > > >> >> >> "caching" > > > > >> >> >> >> space is outside your JVM heap, so does not slowdown the > > > > >> >> >> >> garbage > > > > >> >> >> collector. > > > > >> >> >> >> So be sure to *not* allocate too much heap space (-Xmx) > > > > >> >> >> >> to your > > > > >> >> search > > > > >> >> >> app, > > > > >> >> >> >> only the minimum needed to execute it and leave the rest > > > > >> >> >> >> of your > > > > >> RAM > > > > >> >> >> >> available for the OS kernel to manage FS cache. > > > > >> >> >> >> > > > > >> >> >> >> Uwe > > > > >> >> >> >> > > > > >> >> >> >> ----- > > > > >> >> >> >> Uwe Schindler > > > > >> >> >> >> H.-H.-Meier-Allee 63, D-28213 Bremen > > > > >> >> >> >> http://www.thetaphi.de > > > > >> >> >> >> eMail: u...@thetaphi.de > > > > >> >> >> >> > > > > >> >> >> >> > > > > >> >> >> >> > -----Original Message----- > > > > >> >> >> >> > From: Cheng [mailto:zhoucheng2...@gmail.com] > > > > >> >> >> >> > Sent: Sunday, February 05, 2012 7:56 AM > > > > >> >> >> >> > To: java-user@lucene.apache.org > > > > >> >> >> >> > Subject: Configure writer to write to FSDirectory? > > > > >> >> >> >> > > > > > >> >> >> >> > Hi, > > > > >> >> >> >> > > > > > >> >> >> >> > I build an RAMDirectory on a FSDirectory, and would > > > > >> >> >> >> > like > > > the > > > > >> writer > > > > >> >> >> >> associated > > > > >> >> >> >> > with the RAMDirectory to periodically write to hard > drive. > > > > >> >> >> >> > > > > > >> >> >> >> > Is this achievable? > > > > >> >> >> >> > > > > > >> >> >> >> > Thanks. > > > > >> >> >> >> > > > > >> >> >> >> > > > > >> >> >> >> > > > > >> ----------------------------------------------------------------- > > > > >> ---- > > > > >> >> >> >> To unsubscribe, e-mail: > > > java-user-unsubscr...@lucene.apache.org > > > > >> >> >> >> For additional commands, e-mail: > > > > >> java-user-h...@lucene.apache.org > > > > >> >> >> >> > > > > >> >> >> >> > > > > >> >> >> > > > > >> >> >> > > > > >> ----------------------------------------------------------------- > > > > >> ---- > > > > >> >> >> To unsubscribe, e-mail: > > > > >> >> >> java-user-unsubscr...@lucene.apache.org > > > > >> >> >> For additional commands, e-mail: > > > java-user-h...@lucene.apache.org > > > > >> >> >> > > > > >> >> >> > > > > >> >> > > > > >> >> > > > --------------------------------------------------------------------- > > > > >> >> To unsubscribe, e-mail: > > > > >> >> java-user-unsubscr...@lucene.apache.org > > > > >> >> For additional commands, e-mail: > > > > >> >> java-user-h...@lucene.apache.org > > > > >> >> > > > > >> >> > > > > >> > > > > >> ----------------------------------------------------------------- > > > > >> ---- To unsubscribe, e-mail: > > > > >> java-user-unsubscr...@lucene.apache.org > > > > >> For additional commands, e-mail: java-user-h...@lucene.apache.org > > > > >> > > > > >> > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > > > For additional commands, e-mail: java-user-h...@lucene.apache.org > > > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > >
