I have made some small patch to the application to make this problem less visible, and I'm trying to perform the optimize once per hour, yesterday it took 5 minutes to perform it, this morning 15 minutes. Today I will collect some statistics but the publication process sends documents every 5 minutes, and I think the optimize is taking too much time.
I have no default mergeFactor configured for this collection, do you think that setting it to a small value could improve the situation? If I have understood well having to merge segments will keep similar stats on all nodes. It's ok to have the indexing process a little bit slower. 2014-10-21 18:44 GMT+02:00 Erick Erickson <erickerick...@gmail.com>: > Giovanni: > > To see how this happens, consider a shard with a leader and two > followers. Assume your autocommit interval is 60 seconds on each. > > This interval can expire at slightly different "wall clock" times. > Even if the servers started perfectly in synch, they can get slightly > out of sync. So, you index a bunch of docs and these replicas close > the current segment and re-open a new segment with slightly different > contents. > > Now docs come in that replace older docs. The tf/idf statistics > _include_ deleted document data (which is purged on optimize). Given > that doc X an be in different segments (or, more accurately, segments > that get merged at different times on different machines), replica 1 > may have slightly different stats than replica 2, thus computing > slightly different scores. > > Optimizing purges all data related to deleted documents, so it all > regularizes itself on optimize. > > Best, > Erick > > On Tue, Oct 21, 2014 at 11:08 AM, Giovanni Bricconi > <giovanni.bricc...@banzai.it> wrote: > > I noticed again the problem, now I was able to collect some data. in my > > paste http://pastebin.com/nVwf327c you can see the result of the same > query > > issued twice, the 2nd and 3rd group are swapped. > > > > I pasted also the clusterstate and the core state for each core. > > > > The logs did'n show any problem related to indexing, only some malformed > > query. > > > > After doing an optimize the problem disappeared. > > > > So, is the problem related to documents that where deleted from the > index? > > > > The optimization took 5 minutes to complete > > > > 2014-10-21 11:41 GMT+02:00 Giovanni Bricconi < > giovanni.bricc...@banzai.it>: > > > >> Nice! > >> I will monitor the index and try this if the problem comes back. > >> Actually the problem was due to small differences in score, so I think > the > >> problem has the same origin > >> > >> 2014-10-21 8:10 GMT+02:00 lboutros <boutr...@gmail.com>: > >> > >>> Hi Giovanni, > >>> > >>> we had this problem as well. > >>> The cause was that the different nodes have slightly different idf > values. > >>> > >>> We solved this problem by doing an optimize operation which really > remove > >>> suppressed data. > >>> > >>> Ludovic. > >>> > >>> > >>> > >>> ----- > >>> Jouve > >>> France. > >>> -- > >>> View this message in context: > >>> > http://lucene.472066.n3.nabble.com/unstable-results-on-refresh-tp4164913p4165086.html > >>> Sent from the Solr - User mailing list archive at Nabble.com. > >>> > >> > >> >
