cpoerschke commented on code in PR #1905:
URL: https://github.com/apache/solr/pull/1905#discussion_r1319856395
##########
solr/core/src/java/org/apache/solr/util/circuitbreaker/MemoryCircuitBreaker.java:
##########
@@ -43,8 +46,34 @@ public class MemoryCircuitBreaker extends CircuitBreaker {
private static final ThreadLocal<Long> seenMemory =
ThreadLocal.withInitial(() -> 0L);
private static final ThreadLocal<Long> allowedMemory =
ThreadLocal.withInitial(() -> 0L);
+ /** Creates an instance which averages over 6 samples during last 30
seconds. */
public MemoryCircuitBreaker() {
+ this(6, 5);
+ }
+
+ /**
+ * Constructor that allows override of sample interval for which the memory
usage is fetched. This
+ * is provided for testing, not intended for general use.
+ *
+ * @param numSamples number of samples to calculate average for
+ * @param sampleInterval interval between each sample
+ */
+ protected MemoryCircuitBreaker(int numSamples, int sampleInterval) {
super();
+ if (averagingMetricProvider == null ||
averagingMetricProvider.getRefcount() == 0) {
+ averagingMetricProvider =
Review Comment:
Wondering how this would work if two threads concurrently instantiated a
circuit breaker e.g. two shards in the same JVM. Might an alternative be to
```
private static RefCounted<AveragingMetricProvider> averagingMetricProvider =
...
```
initialise always with the test constructor doing a replacement?
##########
solr/core/src/test/org/apache/solr/util/BaseTestCircuitBreaker.java:
##########
@@ -59,11 +62,17 @@ protected static void indexDocs() {
@Override
public void tearDown() throws Exception {
super.tearDown();
+ dummyMemBreaker.close();
+ dummyCBManager.close();
}
@After
public void after() {
- h.getCore().getCircuitBreakerRegistry().deregisterAll();
+ try {
+ h.getCore().getCircuitBreakerRegistry().deregisterAll();
+ } catch (IOException e) {
+ fail("Failed to unload circuit breakers");
+ }
Review Comment:
```suggestion
removeAllExistingCircuitBreakers();
```
##########
solr/core/src/java/org/apache/solr/util/circuitbreaker/CircuitBreakerRegistry.java:
##########
@@ -97,4 +112,34 @@ public static String toErrorMessage(List<CircuitBreaker>
circuitBreakerList) {
public boolean isEnabled(SolrRequestType requestType) {
return circuitBreakerMap.containsKey(requestType);
}
+
+ @Override
+ public void close() throws IOException {
+ synchronized (circuitBreakerMap) {
+ final AtomicBoolean closeFailed = new AtomicBoolean(false);
+ circuitBreakerMap
+ .values()
+ .forEach(
+ list ->
+ list.forEach(
+ it -> {
+ try {
+ if (log.isDebugEnabled()) {
+ log.debug(
+ "Closed circuit breaker {} for request type(s)
{}",
+ it.getClass().getSimpleName(),
+ it.getRequestTypes());
+ }
+ it.close();
+ } catch (IOException e) {
+ log.error("Failed to close circuit breaker", e);
Review Comment:
```suggestion
log.error("Failed to close circuit breaker " +
it.getClass().getSimpleName(), e);
```
or
```suggestion
log.error("Failed to close circuit breaker " + it,
e);
```
##########
solr/core/src/java/org/apache/solr/util/circuitbreaker/AveragingMetricProvider.java:
##########
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.util.circuitbreaker;
+
+import com.google.common.util.concurrent.AtomicDouble;
+import java.io.Closeable;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import org.apache.solr.common.util.SolrNamedThreadFactory;
+import org.apache.solr.logging.CircularList;
+
+/** Averages the metric value over a period of time */
+public class AveragingMetricProvider implements Closeable {
+ private final CircularList<Double> samplesRingBuffer;
+ private ScheduledExecutorService executor;
+ private final AtomicDouble currentAverageValue = new AtomicDouble(-1);
+
+ /**
+ * Creates an instance with an executor that runs every sampleInterval
seconds and averages over
+ * numSamples samples.
+ *
+ * @param metricProvider metric provider that will provide a value
+ * @param numSamples number of samples to calculate average for
+ * @param sampleInterval interval between each sample
+ */
+ public AveragingMetricProvider(
+ MetricProvider metricProvider, int numSamples, long sampleInterval) {
+ this.samplesRingBuffer = new CircularList<>(numSamples);
+ executor =
+ Executors.newSingleThreadScheduledExecutor(
+ new SolrNamedThreadFactory(
+ "AveragingMetricProvider-" +
metricProvider.getClass().getSimpleName()));
+ executor.scheduleWithFixedDelay(
+ () -> {
+ samplesRingBuffer.add(metricProvider.getMetricValue());
+ currentAverageValue.set(
+ samplesRingBuffer.toList().stream()
+ .mapToDouble(Double::doubleValue)
+ .average()
+ .orElse(-1));
+ },
+ 0,
+ sampleInterval,
+ TimeUnit.SECONDS);
+ }
+
+ /**
+ * Return current average. This is a cached value, so calling this method
will not incur any
+ * calculations
+ */
+ public double getMetricValue() {
+ return currentAverageValue.get();
+ }
+
+ @Override
+ public void close() {
+ try {
+ executor.shutdown();
+ if (!executor.awaitTermination(10, TimeUnit.SECONDS)) {
+ executor.shutdownNow();
+ }
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ }
Review Comment:
Vaguely recall seeing some "shutdown util" or similar wrappers which do some
things as a one function call, might not match exactly the logic here, haven't
looked into.
##########
solr/core/src/java/org/apache/solr/util/circuitbreaker/MemoryCircuitBreaker.java:
##########
@@ -43,8 +46,34 @@ public class MemoryCircuitBreaker extends CircuitBreaker {
private static final ThreadLocal<Long> seenMemory =
ThreadLocal.withInitial(() -> 0L);
private static final ThreadLocal<Long> allowedMemory =
ThreadLocal.withInitial(() -> 0L);
+ /** Creates an instance which averages over 6 samples during last 30
seconds. */
public MemoryCircuitBreaker() {
+ this(6, 5);
+ }
+
+ /**
+ * Constructor that allows override of sample interval for which the memory
usage is fetched. This
+ * is provided for testing, not intended for general use.
Review Comment:
```suggestion
* is provided for testing, not intended for general use because the
average metric provider implementation is the same for all instances of the
class.
```
##########
solr/core/src/java/org/apache/solr/util/circuitbreaker/CircuitBreakerRegistry.java:
##########
@@ -97,4 +112,34 @@ public static String toErrorMessage(List<CircuitBreaker>
circuitBreakerList) {
public boolean isEnabled(SolrRequestType requestType) {
return circuitBreakerMap.containsKey(requestType);
}
+
+ @Override
+ public void close() throws IOException {
+ synchronized (circuitBreakerMap) {
+ final AtomicBoolean closeFailed = new AtomicBoolean(false);
+ circuitBreakerMap
+ .values()
+ .forEach(
+ list ->
+ list.forEach(
+ it -> {
+ try {
+ if (log.isDebugEnabled()) {
+ log.debug(
+ "Closed circuit breaker {} for request type(s)
{}",
+ it.getClass().getSimpleName(),
+ it.getRequestTypes());
+ }
+ it.close();
+ } catch (IOException e) {
+ log.error("Failed to close circuit breaker", e);
+ closeFailed.set(true);
+ }
+ }));
+ circuitBreakerMap.clear();
+ if (closeFailed.get()) {
+ throw new IOException("Failed to close one or more circuit breakers");
Review Comment:
minor: swapping out of the atomic boolean for integer could allow specific
wording.
```suggestion
throw new IOException("Failed to close " + closeFailedCounter.get()
+ " circuit breakers");
```
##########
solr/core/src/java/org/apache/solr/util/circuitbreaker/CircuitBreakerManager.java:
##########
@@ -91,6 +92,16 @@ public void init(NamedList<?> args) {
}
}
+ @Override
+ public void close() throws IOException {
+ if (memEnabled) {
+ memCB.close();
+ }
+ if (cpuEnabled) {
+ cpuCB.close();
+ }
Review Comment:
edge case: `memCB.close()` throwing an exception and `cpuCB.close()` not
getting called as a result. might not be worth extra complexity of handling
since class is deprecated?
##########
solr/core/src/java/org/apache/solr/core/SolrCore.java:
##########
@@ -1764,6 +1764,16 @@ private void doClose() {
ExecutorUtil.shutdownAndAwaitTermination(coreAsyncTaskExecutor);
+ // Close circuit breakers that may have background threads
Review Comment:
```suggestion
// Close circuit breakers that may have background threads, before
metrics because some circuit breakers use metrics
```
##########
solr/core/src/java/org/apache/solr/core/SolrCore.java:
##########
@@ -1115,6 +1112,9 @@ private SolrCore(
// initialize core metrics
initializeMetrics(solrMetricsContext, null);
+ // init pluggable circuit breakers
Review Comment:
```suggestion
// init pluggable circuit breakers, after metrics because some circuit
breakers use metrics
```
##########
solr/core/src/java/org/apache/solr/util/circuitbreaker/AveragingMetricProvider.java:
##########
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.util.circuitbreaker;
+
+import com.google.common.util.concurrent.AtomicDouble;
+import java.io.Closeable;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import org.apache.solr.common.util.SolrNamedThreadFactory;
+import org.apache.solr.logging.CircularList;
+
+/** Averages the metric value over a period of time */
+public class AveragingMetricProvider implements Closeable {
Review Comment:
Am surprised such a class doesn't yet exist.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
