All strange of course. What do your Solr logs show when this happens? And how reproducible is this?
Best, Erick On Wed, May 27, 2015 at 4:00 AM, Upayavira <u...@odoko.co.uk> wrote: > In this case, optimising makes sense, once the index is generated, you > are not updating It. > > Upayavira > > On Wed, May 27, 2015, at 06:14 AM, Modassar Ather wrote: >> Our index has almost 100M documents running on SolrCloud of 5 shards and >> each shard has an index size of about 170+GB (for the record, we are not >> using stored fields - our documents are pretty large). We perform a full >> indexing every weekend and during the week there are no updates made to >> the >> index. Most of the queries that we run are pretty complex with hundreds >> of >> terms using PhraseQuery, BooleanQuery, SpanQuery, Wildcards, boosts etc. >> and take many minutes to execute. A difference of 10-20% is also a big >> advantage for us. >> >> We have been optimizing the index after indexing for years and it has >> worked well for us. Every once in a while, we upgrade Solr to the latest >> version and try without optimizing so that we can save the many hours it >> take to optimize such a huge index, but find optimized index work well >> for >> us. >> >> Erick I was indexing today the documents and saw the optimize happening >> in >> background. >> >> On Tue, May 26, 2015 at 9:12 PM, Erick Erickson <erickerick...@gmail.com> >> wrote: >> >> > No results yet. I finished the test harness last night (not really a >> > unit test, a stand-alone program that endlessly adds stuff and tests >> > that every commit returns the correct number of docs). >> > >> > 8,000 cycles later there aren't any problems reported. >> > >> > Siiigggggh. >> > >> > >> > On Tue, May 26, 2015 at 1:51 AM, Modassar Ather <modather1...@gmail.com> >> > wrote: >> > > Hi, >> > > >> > > Erick you mentioned about a unit test to test the optimize running in >> > > background. Kindly share your findings if any. >> > > >> > > Thanks, >> > > Modassar >> > > >> > > On Mon, May 25, 2015 at 11:47 AM, Modassar Ather <modather1...@gmail.com >> > > >> > > wrote: >> > > >> > >> Thanks everybody for your replies. >> > >> >> > >> I have noticed the optimization running in background every time I >> > >> indexed. This is 5 node cluster with solr-5.1.0 and uses the >> > >> CloudSolrClient. Kindly share your findings on this issue. >> > >> >> > >> Our index has almost 100M documents running on SolrCloud. We have been >> > >> optimizing the index after indexing for years and it has worked well for >> > >> us. >> > >> >> > >> Thanks, >> > >> Modassar >> > >> >> > >> On Fri, May 22, 2015 at 11:55 PM, Erick Erickson < >> > erickerick...@gmail.com> >> > >> wrote: >> > >> >> > >>> Actually, I've recently seen very similar behavior in Solr 4.10.3, but >> > >>> involving hard commits openSearcher=true, see: >> > >>> https://issues.apache.org/jira/browse/SOLR-7572. Of course I can't >> > >>> reproduce this at will, siigggghhhh. >> > >>> >> > >>> A unit test should be very simple to write though, maybe I can get to >> > it >> > >>> today. >> > >>> >> > >>> Erick >> > >>> >> > >>> >> > >>> >> > >>> On Fri, May 22, 2015 at 8:27 AM, Upayavira <u...@odoko.co.uk> wrote: >> > >>> > >> > >>> > >> > >>> > On Fri, May 22, 2015, at 03:55 PM, Shawn Heisey wrote: >> > >>> >> On 5/21/2015 6:21 AM, Modassar Ather wrote: >> > >>> >> > I am using Solr-5.1.0. I have an indexer class which invokes >> > >>> >> > cloudSolrClient.optimize(true, true, 1). My indexer exits after >> > the >> > >>> >> > invocation of optimize and the optimization keeps on running in >> > the >> > >>> >> > background. >> > >>> >> > Kindly let me know if it is per design and how can I make my >> > indexer >> > >>> to >> > >>> >> > wait until the optimization is over. Is there a >> > >>> configuration/parameter I >> > >>> >> > need to set for the same. >> > >>> >> > >> > >>> >> > Please note that the same indexer with >> > >>> cloudSolrServer.optimize(true, true, >> > >>> >> > 1) on Solr-4.10 used to wait till the optimize was over before >> > >>> exiting. >> > >>> >> >> > >>> >> This is very odd, because I could not get HttpSolrServer to >> > optimize in >> > >>> >> the background, even when that was what I wanted. >> > >>> >> >> > >>> >> I wondered if maybe the Cloud object behaves differently with >> > regard to >> > >>> >> blocking until an optimize is finished ... except that there is no >> > code >> > >>> >> for optimizing in CloudSolrClient at all ... so I don't know where >> > the >> > >>> >> different behavior would actually be happening. >> > >>> > >> > >>> > A more important question is, why are you optimising? Generally it >> > isn't >> > >>> > recommended anymore as it reduces the natural distribution of >> > documents >> > >>> > amongst segments and makes future merges more costly. >> > >>> > >> > >>> > Upayavira >> > >>> >> > >> >> > >> >> >
