Eric,Shawn, Apologies for the late update on this thread and thank you for your inputs. My assumption about the number of segments increasing was out of incomplete understanding of the TieredMergePolicy, but I get it now. Another concern was slowing indexing rate due to constant merges. This is from reading the documentation: "Choosing the best merge factors is generally a trade-off of indexing speed vs. searching speed. Having fewer segments in the index generally accelerates searches, because there are fewer places to look. It also can also result in fewer physical files on disk. But to keep the number of segments low, merges will occur more often, which can add load to the system and slow down updates to the index"
Taking your suggestions, we have reduced hard commit interval (openSearcher=false) from 10 mins to 1 min to begin with. Also, our servers are on Windows so that could also be a cause of the service getting killed before being able to gracefully shutdown. The cascading effect is stale results while tlogs are being played on startup. I understand that although not foolproof, reducing the autoCommit interval should help mitigate the problem and we'll continue to monitor this for now. Thanks, Rahul On Fri, Mar 8, 2019 at 2:14 PM Erick Erickson <erickerick...@gmail.com> wrote: > (1) no, and Shawn’s comments are well taken. > > (2) bq. is the number of segments would drastically increase > > Not true. First of all, TieredMergePolicy will take care of merging “like > sized” segments for you. You’ll have the same number (or close) no matter > how short the autocommit interval. Second, new segments are created > whenever the internal indexing buffer is filled up, default 100M anyway so > just because you have a long autocommit interval doesn’t say much about the > number of segments that are created. > > This is really not something you should be concerned about, certainly not > something you should accept other problems because. Solr runs quite well > with 15 second autocommit and very high indexing rates, why do you think > your situation is different? Do you have any evidence that would be a > problem at all? > > Best, > Erick > > > > On Mar 8, 2019, at 11:05 AM, Shawn Heisey <apa...@elyograg.org> wrote: > > > > On 3/8/2019 10:44 AM, Rahul Goswami wrote: > >> 1) Is there currently a configuration setting in Solr that will trigger > the > >> first option you mentioned ? Which is to not serve any searches until > tlogs > >> are played. If not, since instances shutting down abruptly is not very > >> uncommon, would a JIRA to implement this configuration be warranted? > > > > In what setup is an abrupt shutdown *expected*? If that's really > common, then your setup is, in my opinion, very broken. It is our intent > that abrupt death of the Solr process should be quite rare. We do still > have a problem on Windows where the wait for clean shutdown is only five > seconds -- nowhere near enough. The Windows script still needs a lot of > work, but most of us are not adept at Windows scripting. > > > > There is an issue for the timeout interval in bin\solr.cmd on Windows: > > > > https://issues.apache.org/jira/browse/SOLR-9698 > > > >> 2) We have a setup with moderate indexing rate and moderate search rate. > >> Currently the auto commit interval is 10 mins. What should be a > recommended > >> hard commit interval for such a setup? Our concern with going too low on > >> that autoCommit interval (with openSearcher=false) is the number of > >> segments that would drastically increase, eventually causing > merges,slower > >> searches etc. > > > > Solr has shipped with a 15 second autoCommit, where openSearcher is set > to false, for a while now. This is a setting that works quite well. As > long as you're not opening a new searcher, commits are quite fast. I > personally would use 60 seconds, but 15 seconds does work well. It is > usually autoSoftCommit where you need to be concerned about short > intervals, because a soft commit opens a searcher. > > > > Thanks, > > Shawn > >
