Isn't that what this code does?
onDeckSearchers++;
if (onDeckSearchers < 1) {
// should never happen... just a sanity check
log.error(logid+"ERROR!!! onDeckSearchers is " + onDeckSearchers);
onDeckSearchers=1; // reset
} else if (onDeckSearchers > maxWarmingSearchers) {
onDeckSearchers--;
String msg="Error opening new searcher. exceeded limit of
maxWarmingSearchers="+maxWarmingSearchers + ", try again later.";
log.warn(logid+""+ msg);
// HTTP 503==service unavailable, or 409==Conflict
throw new
SolrException(SolrException.ErrorCode.SERVICE_UNAVAILABLE,msg,true);
} else if (onDeckSearchers > 1) {
log.info(logid+"PERFORMANCE WARNING: Overlapping
onDeckSearchers=" + onDeckSearchers);
}
On Tue, Nov 2, 2010 at 10:02 AM, Jonathan Rochkind <rochk...@jhu.edu> wrote:
> It's definitely a known 'issue' that you can't replicate (or do any other
> kind of index change, including a commit) at a faster frequency than your
> warming queries take to complete, or you'll wind up with something like
> you've seen.
>
> It's in some documentation somewhere I saw, for sure.
>
> The advice to 'just query against the master' is kind of odd, because,
> then... why have a slave at all, if you aren't going to query against it? I
> guess just for backup purposes.
>
> But even with just one solr, or querying master, if you commit at rate such
> that commits come before the warming queries can complete, you're going to
> have the same issue.
>
> The only answer I know of is "Don't commit (or replicate) at a faster rate
> than it takes your warming to complete." You can reduce your warming
> queries/operations, or reduce your commit/replicate frequency.
>
> Would be interesting/useful if Solr noticed this going on, and gave you some
> kind of error in the log (or even an exception when started with a certain
> parameter for testing) "Overlapping warming queries, you're committing too
> fast" or something. Because it's easy to make this happen without realizing
> it, and then your Solr does what Simon says, runs out of RAM and/or uses a
> whole lot of CPU and disk io.
>
> Lance Norskog wrote:
>>
>> You should query against the indexer. I'm impressed that you got 5s
>> replication to work reliably.
>>
>> On Mon, Nov 1, 2010 at 4:27 PM, Simon Wistow <si...@thegestalt.org> wrote:
>>
>>>
>>> We've been trying to get a setup in which a slave replicates from a
>>> master every few seconds (ideally every second but currently we have it
>>> set at every 5s).
>>>
>>> Everything seems to work fine until, periodically, the slave just stops
>>> responding from what looks like it running out of memory:
>>>
>>> org.apache.catalina.core.StandardWrapperValve invoke
>>> SEVERE: Servlet.service() for servlet jsp threw exception
>>> java.lang.OutOfMemoryError: Java heap space
>>>
>>>
>>> (our monitoring seems to confirm this).
>>>
>>> Looking around my suspicion is that it takes new Readers longer to warm
>>> than the gap between replication and thus they just build up until all
>>> memory is consumed (which, I suppose isn't really memory 'leaking' per
>>> se, more just resource consumption)
>>>
>>> That said, we've tried turning off caching on the slave and that didn't
>>> help either so it's possible I'm wrong.
>>>
>>> Is there anything we can do about this? I'm reluctant to increase the
>>> heap space since I suspect that will mean that there's just a longer
>>> period between failures. Might Zoie help here? Or should we just query
>>> against the Master?
>>>
>>>
>>> Thanks,
>>>
>>> Simon
>>>
>>>
>>
>>
>>
>>
>
--
Lance Norskog
goks...@gmail.com
