Re: Circuit breakers and update requests

[Jan Høydahl]

There was no feedback on the topic of global Circuit Breaker vs per-core.
So I opened https://issues.apache.org/jira/browse/SOLR-16974 to propose 
supporting Global CB. This is still on design level, please weigh in with your 
thoughts.

Jan

> 30. aug. 2023 kl. 22:02 skrev Jan Høydahl :
> 
> Hi
> 
> For those interested in this topic, here is the JIRA, which has a Pull 
> Request for review: https://issues.apache.org/jira/browse/SOLR-16954
> 
> Jan Høydahl
> 
>> 3. jul. 2023 kl. 22:28 skrev Pierre Salagnac :
>> 
>> Hi Jan,
>> 
>> As far as I know, Solr only supports circuit breaks for queries at the
>> moment.
>> 
>> We have a custom integration of circuit breakers for indexing (in Solr 8,
>> so that's not fully aligned with what in solr 9) with a custom
>> UpdateRequestProcessor. Basically, a new instance of every update processor
>> is created by calling the corresponding factory. We then check system stats
>> (CPU...) at the first handled doc.
>> Maybe there are other ways of integrating circuit breakers, but at least
>> doing so in an update processor was not intrusive in the code.
>> 
>> Probably it is worth it to have something standard in Solr. A "symetric"
>> class of SearchHandler to add CB integration could be UpdateRequestHandler.
>> 
>> Pierre
>> 
>> 
>> Le mar. 27 juin 2023 à 13:32, Jan Høydahl  a écrit :
>> 
>>> Hi,
>>> 
>>> Solr has CPU and Memory circuit breakers that will terminate Search
>>> Requests only.
>>> See
>>> https://solr.apache.org/guide/solr/latest/deployment-guide/circuit-breakers.html
>>> for docs.
>>> 
>>> Example:
>>> 
>>>  
>>>true
>>>75
>>>true
>>>75
>>>  
>>> 
>>> 
>>> A Solr node typically gets overloaded by the combined update and query
>>> traffic, and
>>> I'm looking into enabling ciricuit breakers for update requests. For many
>>> workloads, pausing
>>> update traffic would resolve the situation, with the benefit of users not
>>> being affected by aborted
>>> queries.
>>> 
>>> So ideally I'd want to be able to choose to enable/disable CB on
>>> update/query individually.
>>> Or better, to kill update requests on e.g. 80% threshold and search
>>> requests on 90% threshold.
>>> 
>>> The current breaker impl for search <
>>> https://github.com/apache/solr/blob/branch_9_2/solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java#L372:380>
>>> [1] is hardcoded into SearchHandler in such a way that if ANY
>>> of the configured breakers trips, search requests are aborted. Also the
>>> breakers seem to be generic
>>> in nature, named CPUCircuitBreaker <
>>> https://github.com/apache/solr/blob/branch_9_2/solr/core/src/java/org/apache/solr/util/circuitbreaker/CPUCircuitBreaker.java>
>>> and MemoryCircuitBreaker <
>>> https://github.com/apache/solr/blob/branch_9_2/solr/core/src/java/org/apache/solr/util/circuitbreaker/MemoryCircuitBreaker.java>,
>>> so adding new UpdateCPUCircuitBreaker
>>> does not seem to be the intention here.
>>> 
>>> I'm also unclear on the usefulness of having CB on the core level and not
>>> the node level. If you have
>>> 10 cores from 10 collections on a node, and only some have CBs while
>>> others do not, the node will
>>> still be overloaded unless the admin convinces every collection owner to
>>> implement the same CBs?
>>> 
>>> So my question becomes - how do we enable CB for update requests into this
>>> mix in a clean way?
>>> 
>>> Jan


-
To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org
For additional commands, e-mail: dev-h...@solr.apache.org

Re: Circuit breakers and update requests

[Jan Høydahl]

Hi

For those interested in this topic, here is the JIRA, which has a Pull Request 
for review: https://issues.apache.org/jira/browse/SOLR-16954

Jan Høydahl

> 3. jul. 2023 kl. 22:28 skrev Pierre Salagnac :
> 
> Hi Jan,
> 
> As far as I know, Solr only supports circuit breaks for queries at the
> moment.
> 
> We have a custom integration of circuit breakers for indexing (in Solr 8,
> so that's not fully aligned with what in solr 9) with a custom
> UpdateRequestProcessor. Basically, a new instance of every update processor
> is created by calling the corresponding factory. We then check system stats
> (CPU...) at the first handled doc.
> Maybe there are other ways of integrating circuit breakers, but at least
> doing so in an update processor was not intrusive in the code.
> 
> Probably it is worth it to have something standard in Solr. A "symetric"
> class of SearchHandler to add CB integration could be UpdateRequestHandler.
> 
> Pierre
> 
> 
> Le mar. 27 juin 2023 à 13:32, Jan Høydahl  a écrit :
> 
>> Hi,
>> 
>> Solr has CPU and Memory circuit breakers that will terminate Search
>> Requests only.
>> See
>> https://solr.apache.org/guide/solr/latest/deployment-guide/circuit-breakers.html
>> for docs.
>> 
>> Example:
>> 
>>  
>>true
>>75
>>true
>>75
>>  
>> 
>> 
>> A Solr node typically gets overloaded by the combined update and query
>> traffic, and
>> I'm looking into enabling ciricuit breakers for update requests. For many
>> workloads, pausing
>> update traffic would resolve the situation, with the benefit of users not
>> being affected by aborted
>> queries.
>> 
>> So ideally I'd want to be able to choose to enable/disable CB on
>> update/query individually.
>> Or better, to kill update requests on e.g. 80% threshold and search
>> requests on 90% threshold.
>> 
>> The current breaker impl for search <
>> https://github.com/apache/solr/blob/branch_9_2/solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java#L372:380>
>> [1] is hardcoded into SearchHandler in such a way that if ANY
>> of the configured breakers trips, search requests are aborted. Also the
>> breakers seem to be generic
>> in nature, named CPUCircuitBreaker <
>> https://github.com/apache/solr/blob/branch_9_2/solr/core/src/java/org/apache/solr/util/circuitbreaker/CPUCircuitBreaker.java>
>> and MemoryCircuitBreaker <
>> https://github.com/apache/solr/blob/branch_9_2/solr/core/src/java/org/apache/solr/util/circuitbreaker/MemoryCircuitBreaker.java>,
>> so adding new UpdateCPUCircuitBreaker
>> does not seem to be the intention here.
>> 
>> I'm also unclear on the usefulness of having CB on the core level and not
>> the node level. If you have
>> 10 cores from 10 collections on a node, and only some have CBs while
>> others do not, the node will
>> still be overloaded unless the admin convinces every collection owner to
>> implement the same CBs?
>> 
>> So my question becomes - how do we enable CB for update requests into this
>> mix in a clean way?
>> 
>> Jan

Re: Circuit breakers and update requests

[Pierre Salagnac]

Hi Jan,

As far as I know, Solr only supports circuit breaks for queries at the
moment.

We have a custom integration of circuit breakers for indexing (in Solr 8,
so that's not fully aligned with what in solr 9) with a custom
UpdateRequestProcessor. Basically, a new instance of every update processor
is created by calling the corresponding factory. We then check system stats
(CPU...) at the first handled doc.
Maybe there are other ways of integrating circuit breakers, but at least
doing so in an update processor was not intrusive in the code.

Probably it is worth it to have something standard in Solr. A "symetric"
class of SearchHandler to add CB integration could be UpdateRequestHandler.

Pierre


Le mar. 27 juin 2023 à 13:32, Jan Høydahl  a écrit :

> Hi,
>
> Solr has CPU and Memory circuit breakers that will terminate Search
> Requests only.
> See
> https://solr.apache.org/guide/solr/latest/deployment-guide/circuit-breakers.html
> for docs.
>
> Example:
>
>   
> true
> 75
> true
> 75
>   
>
>
> A Solr node typically gets overloaded by the combined update and query
> traffic, and
> I'm looking into enabling ciricuit breakers for update requests. For many
> workloads, pausing
> update traffic would resolve the situation, with the benefit of users not
> being affected by aborted
> queries.
>
> So ideally I'd want to be able to choose to enable/disable CB on
> update/query individually.
> Or better, to kill update requests on e.g. 80% threshold and search
> requests on 90% threshold.
>
> The current breaker impl for search <
> https://github.com/apache/solr/blob/branch_9_2/solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java#L372:380>
> [1] is hardcoded into SearchHandler in such a way that if ANY
> of the configured breakers trips, search requests are aborted. Also the
> breakers seem to be generic
> in nature, named CPUCircuitBreaker <
> https://github.com/apache/solr/blob/branch_9_2/solr/core/src/java/org/apache/solr/util/circuitbreaker/CPUCircuitBreaker.java>
> and MemoryCircuitBreaker <
> https://github.com/apache/solr/blob/branch_9_2/solr/core/src/java/org/apache/solr/util/circuitbreaker/MemoryCircuitBreaker.java>,
> so adding new UpdateCPUCircuitBreaker
> does not seem to be the intention here.
>
> I'm also unclear on the usefulness of having CB on the core level and not
> the node level. If you have
> 10 cores from 10 collections on a node, and only some have CBs while
> others do not, the node will
> still be overloaded unless the admin convinces every collection owner to
> implement the same CBs?
>
> So my question becomes - how do we enable CB for update requests into this
> mix in a clean way?
>
> Jan

Circuit breakers and update requests

[Jan Høydahl]

Hi,

Solr has CPU and Memory circuit breakers that will terminate Search Requests 
only.
See 
https://solr.apache.org/guide/solr/latest/deployment-guide/circuit-breakers.html
 for docs.

Example:

  
true
75
true
75
  


A Solr node typically gets overloaded by the combined update and query traffic, 
and
I'm looking into enabling ciricuit breakers for update requests. For many 
workloads, pausing
update traffic would resolve the situation, with the benefit of users not being 
affected by aborted
queries. 

So ideally I'd want to be able to choose to enable/disable CB on update/query 
individually.
Or better, to kill update requests on e.g. 80% threshold and search requests on 
90% threshold.

The current breaker impl for search 

 [1] is hardcoded into SearchHandler in such a way that if ANY 
of the configured breakers trips, search requests are aborted. Also the 
breakers seem to be generic
in nature, named CPUCircuitBreaker 

 and MemoryCircuitBreaker 
,
 so adding new UpdateCPUCircuitBreaker
does not seem to be the intention here.

I'm also unclear on the usefulness of having CB on the core level and not the 
node level. If you have
10 cores from 10 collections on a node, and only some have CBs while others do 
not, the node will
still be overloaded unless the admin convinces every collection owner to 
implement the same CBs?

So my question becomes - how do we enable CB for update requests into this mix 
in a clean way?

Jan