Inline quoting ahead, sorry:
Colin:
Stop. Back up. The automatic soft commits will make updates available to
your users every second. Those documents _include_ anything from your "hard
commit" jobs. What could be faster? Parenthetically I'll add that 1 second
soft commits are rarely an actual requirement, but that's your decision.
The one-second commits not my decision, per se; it's the default value
in solrconfig.xml and is also suggested as a "common configuration" in
the "Near Real Time Searching" section of the reference guide.
(Our users at Experts Exchange used to have to wait up to five minutes
before the search index updated with the latest content. While switching
to Solr, we saw that the recommended configuration would refresh the
index in seconds, rather than minutes, and rejoiced. We'd rather not
increase the latency too far to solve this problem.)
For the hard commits. Fine. Do them if you insist. Just set
openSearcher=false. The documents will be searchable the next time the soft
commit happens, within one second. The key is openSearcher=false. That
prevents starting a brand new searcher.
Are you saying that the automatic soft commit will trigger, no matter
what, even after our code has explicitly requested a hard commit? If so,
that is, if the automatic soft commit triggers, even if no additional
update requests have come in since the hard commit, then great! We'll do
that!
BTW, your commits are not failing. It's just that _after_ the commit
happens, the warming searcher limit is exceeded.
My commits may indeed be succeeding, but the server is returning a HTTP
503 response, which leads to SolrJ throwing a SolrServerException with
the message "No live SolrServers available to handle this request." Our
code, understandably, interprets that as a failed request. This causes
our job to abort and try again the next time it runs.
You can even wait until the segments are flushed to disk. All without
opening a searcher.
We will go with this if the automatic soft commit does indeed trigger
after the explicit hard commit, thanks.
Shawn is spot on in his recommendations to not fixate on the commits. Solr
handles that. Here's a long blog about all the details of durability .vs.
visibility.
http://searchhub.org/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
You're over-thinking the problem here, trying to control commits with a
sledgehammer when you don't need to, just use the built-in capabilities.
I get what you both are saying. If the problem is that I'm doing
explicit hard commits, the solution is that I should stop doing explicit
hard commits.
That's not really a solution, though.
What if, for whatever reason, I absolutely *had to* perform explicit
hard commits? (I know you're saying I *don't* have to, but please
indulge me for a moment.) Fortunately, the SolrJ client provides a way I
can do this. But now my Solr server logs are full of "Overlapping
onDeckSearchers" performance warnings. Fine, I'll turn
maxWarmingSearchers down to 1. Now the server returns HTTP 503 responses
every now and then and SolrJ throws an exception.
I think that's a problem that the servers can solve: just queue up the
request until the number of warming searchers is under the limit. So I
filed that RFE. Even when all the above suggestions work perfectly and
fix our issues, it's still a valid RFE.
