Thanks Mark, that really did the job! The speed loss in update time is more than compensated at optimizing time!
Now I am trying to do another test... but not sure if Lucene have this option, I am using Lucene 2.9-dev. As I am working with 3G index and always have to optimize (as I said before, I tried not to optimize to send my index via rsync faster but the speed loss to serve request in the slaves was huge). I wander if it's possible to do "block optimizing" (I have just invented the word). The example would be... I have a 3G index optimized. I start executing updates to the index.. would be possible to keep doing optimizes just on the new created segments?... so I would still have the 3G index and would be building another big index from the segments created from the updates This way I would have to send via rsync to the slaves just the new "blog" (suposing the slaves already had the 3G index because I would have sended it before). Is there any way to do something similar to that? This has come to my mind cause I have to serve the index to the slaves as many times as possible... and optimizing the index in just one "block" makes rsync job to take a long time. Thanks in advance markrmiller wrote: > > Marc Sturlese wrote: >> Hey there, >> I am creating an index of 3G... it's fast indexing but optimization takes >> about 10 min. I need to optimize it every time I update as if I don't do >> that, search requests will be much slower. >> Wich parameter configuration would be the best to make optimize as fast >> as >> possible (I don't mind to use a lot of memory, at least for testing, if I >> can speed up the process). >> Actually I am using for the IndexWriter: >> >> <ramBufferSizeMB>1024</ramBufferSizeMB> >> <maxMergeDocs>2147483647</maxMergeDocs> >> <maxFieldLength>10000</maxFieldLength> >> <writeLockTimeout>1000</writeLockTimeout> >> <commitLockTimeout>10000</commitLockTimeout> >> <mergeFactor>10</mergeFactor> >> Am I missing any important parameter to do that job? >> Thanks in advance >> >> > How about using a merge factor of 2? This way you are pretty much always > optimized (old large segment, new small segment at most) - you pay a bit > in update speed, but I've found it to be very reasonable for many > applications. > > -- > - Mark > > http://www.lucidimagination.com > > > > > -- View this message in context: http://www.nabble.com/optimize-an-index-as-fast-as-possible-tp22543267p22565158.html Sent from the Solr - User mailing list archive at Nabble.com.
