We need to work out why your performance is bad without optimise. What version of Solr are you using? Can you confirm that your config is using the TieredMergePolicy?
Upayavira Oe, Jun 30, 2015, at 04:48 AM, Summer Shire wrote: > Hi Upayavira and Erick, > > There are two things we are talking about here. > > First: Why am I optimizing? If I don’t our SEARCH (NOT INDEXING) > performance is 100% worst. > The problem lies in the number of total segments. We have to have max > segments 1 or 2. > I have done intensive performance related tests around number of > segments, merge factor or changing the Merge policy. > > Second: Solr does not perform better for me without an optimize. So now > that I have to optimize the second issue > is updating concurrently during an optimize. If I update when an optimize > is happening the optimize takes 5 times as long as > the normal optimize. > > So is there any way other than creating a postOptimize hook and writing > the status in a file and somehow making it available to the indexer. > All of this just sounds traumatic :) > > Thanks > Summer > > > > On Jun 29, 2015, at 5:40 AM, Erick Erickson <erickerick...@gmail.com> wrote: > > > > Steven: > > > > Yes, but.... > > > > First, here's Mike McCandles' excellent blog on segment merging: > > http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html > > > > I think the third animation is the TieredMergePolicy. In short, yes an > > optimize will reclaim disk space. But as you update, this is done for > > you anyway. About the only time optimizing is at all beneficial is > > when you have a relatively static index. If you're continually > > updating documents, and by that I mean replacing some existing > > documents, then you'll immediately start generating "holes" in your > > index. > > > > And if you _do_ optimize, you wind up with a huge segment. And since > > the default policy tries to merge segments of roughly the same size, > > it accumulates deletes for quite a while before they merged away. > > > > And if you don't update existing docs or delete docs, then there's no > > wasted space anyway. > > > > Summer: > > > > First off, why do you care about not updating during optimizing? > > There's no good reason you have to worry about that, you can freely > > update while optimizing. > > > > But frankly I have to agree with Upayavira that on the face of it > > you're doing a lot of extra work. See above, but you optimize while > > indexing, so immediately you're rather defeating the purpose. > > Personally I'd only optimize relatively static indexes and, by > > definition, you're index isn't static since the second process is just > > waiting to modify it. > > > > Best, > > Erick > > > > On Mon, Jun 29, 2015 at 8:15 AM, Steven White <swhite4...@gmail.com> wrote: > >> Hi Upayavira, > >> > >> This is news to me that we should not optimize and index. > >> > >> What about disk space saving, isn't optimization to reclaim disk space or > >> is Solr somehow does that? Where can I read more about this? > >> > >> I'm on Solr 5.1.0 (may switch to 5.2.1) > >> > >> Thanks > >> > >> Steve > >> > >> On Mon, Jun 29, 2015 at 4:16 AM, Upayavira <u...@odoko.co.uk> wrote: > >> > >>> I'm afraid I don't understand. You're saying that optimising is causing > >>> performance issues? > >>> > >>> Simple solution: DO NOT OPTIMIZE! > >>> > >>> Optimisation is very badly named. What it does is squashes all segments > >>> in your index into one segment, removing all deleted documents. It is > >>> good to get rid of deletes - in that sense the index is "optimized". > >>> However, future merges become very expensive. The best way to handle > >>> this topic is to leave it to Lucene/Solr to do it for you. Pretend the > >>> "optimize" option never existed. > >>> > >>> This is, of course, assuming you are using something like Solr 3.5+. > >>> > >>> Upayavira > >>> > >>> On Mon, Jun 29, 2015, at 08:08 AM, Summer Shire wrote: > >>>> > >>>> Have to cause of performance issues. > >>>> Just want to know if there is a way to tap into the status. > >>>> > >>>>> On Jun 28, 2015, at 11:37 PM, Upayavira <u...@odoko.co.uk> wrote: > >>>>> > >>>>> Bigger question, why are you optimizing? Since 3.6 or so, it generally > >>>>> hasn't been requires, even, is a bad thing. > >>>>> > >>>>> Upayavira > >>>>> > >>>>>> On Sun, Jun 28, 2015, at 09:37 PM, Summer Shire wrote: > >>>>>> Hi All, > >>>>>> > >>>>>> I have two indexers (Independent processes ) writing to a common solr > >>>>>> core. > >>>>>> If One indexer process issued an optimize on the core > >>>>>> I want the second indexer to wait adding docs until the optimize has > >>>>>> finished. > >>>>>> > >>>>>> Are there ways I can do this programmatically? > >>>>>> pinging the core when the optimize is happening is returning OK > >>> because > >>>>>> technically > >>>>>> solr allows you to update when an optimize is happening. > >>>>>> > >>>>>> any suggestions ? > >>>>>> > >>>>>> thanks, > >>>>>> Summer > >>> >
