[jira] [Created] (SOLR-17303) CVE-2023-39410: Upgrade to apache-avro version 1.11.3
Sujeet-A created SOLR-17303: --- Summary: CVE-2023-39410: Upgrade to apache-avro version 1.11.3 Key: SOLR-17303 URL: https://issues.apache.org/jira/browse/SOLR-17303 Project: Solr Issue Type: Bug Security Level: Public (Default Security Level. Issues are Public) Components: security Affects Versions: 9.6 Reporter: Sujeet-A CVE-2023-39410: Upgrade Apache-Avro version to 1.11.3 When deserializing untrusted or corrupted data, it is possible for a reader to consume memory beyond the allowed constraints and thus lead to out of memory on the system. This issue affects Java applications using Apache Avro Java SDK up to and including 1.11.2. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17302: Convert /cluster filestore APIs to JAX-RS [solr]
epugh commented on PR #2470: URL: https://github.com/apache/solr/pull/2470#issuecomment-2121526630 If `DistribFileStore` is the lone implementation of `FileStore`, then do we need `FileStore`?I never liked the whole "Let me have an interface for everything" cause it doubles the files to navigate through. In the future, if we need a `FileStore` implementation because some wrote `NonDistribFileStore` then we can make it then... We aren't C code, we don't need a seperate `.h` file for every `.cpp` file! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17302: Convert /cluster filestore APIs to JAX-RS [solr]
gerlowskija commented on PR #2470: URL: https://github.com/apache/solr/pull/2470#issuecomment-2121513823 Thanks for the quick review Eric! > I tagged some places where it would be nice ot have more descriptive variable names, however I suspect you are trying to minimize change Your suspicion is correct. This PR aims to convert the API(s) to JAX-RS without delving too deeply into cleaning up the underlying implementation. I think some of the stuff you commented on is innocuous enough that I can be tempted into making a few cleanups though, particularly some of the naming and javadoc/comment issues. > Do we still need DistribFileStore in a world with ClusterFileStore? Yes, we do. There's some naming-related confusion going on here that I've definitely caused: 1. `FileStore` is an interface that defines the operations that Solr's file-store has to support. 2. `DistribFileStore` is the lone implementation of the `FileStore` interface. 3. `ClusterFileStore` is the entrypoint for the relevant v2 APIs. `ClusterFileStore` is, on its surface, a pretty bad naming choice given the pre-existing `FileStore` interface. It 100% appears to be an implementation of that interface, but it's not. I do have an excuse, though it may be a bad one I chose `ClusterFileStore` for the v2 API class name to be consistent with a naming convention we've had since creating the "api" gradle module earlier this year. (i.e. interfaces in "api" module are `FooApi`, with classes in solr-core being `Foo`). The filestore (for reasons I still don't really understand) makes APIs available under two paths: `/api/cluster/files` and `/api/node/files`. So I chose `ClusterFileStoreApi`/`ClusterFileStore` partially in an attempt to distinguish between these two sets of filestore APIs, and partially to maintain consistency with the existing naming convention. The end result is clearly confusing; very open if anyone has other suggestions that'd avoid the ambiguity and confusion!? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17302: Convert /cluster filestore APIs to JAX-RS [solr]
epugh commented on code in PR #2470:
URL: https://github.com/apache/solr/pull/2470#discussion_r1607425946
##
solr/core/src/java/org/apache/solr/filestore/FileStoreAPI.java:
##
Review Comment:
I sort of thought maybe `FileStoreAPI.java` would be totally removed in
favour of the new classes?
##
solr/core/src/java/org/apache/solr/filestore/DistribFileStore.java:
##
@@ -217,7 +217,7 @@ private boolean fetchFileFromNodeAndPersist(String
fromNode) {
}
boolean fetchFromAnyNode() {
- ArrayList l = coreContainer.getFileStoreAPI().shuffledNodes();
+ ArrayList l = FileStoreUtils.shuffledNodes(coreContainer);
Review Comment:
i know you are worried about scope, but maybe not `l`? `liveNodes`?
##
solr/core/src/java/org/apache/solr/pkg/PackageAPI.java:
##
@@ -254,7 +250,7 @@ public void refresh(PayloadObj payload) {
}
// first refresh my own
packageLoader.notifyListeners(p);
- for (String s : coreContainer.getFileStoreAPI().shuffledNodes()) {
+ for (String s : FileStoreUtils.shuffledNodes(coreContainer)) {
Review Comment:
better name on what is returned...?
##
solr/core/src/java/org/apache/solr/filestore/FileStoreUtils.java:
##
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.filestore;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Consumer;
+import java.util.function.Supplier;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.core.BlobRepository;
+import org.apache.solr.core.CoreContainer;
+import org.apache.solr.util.CryptoKeys;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** Common utilities used by filestore-related code. */
+public class FileStoreUtils {
+ private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ /** get a list of nodes randomly shuffled * @lucene.internal */
+ public static ArrayList shuffledNodes(CoreContainer coreContainer) {
+Set liveNodes =
+
coreContainer.getZkController().getZkStateReader().getClusterState().getLiveNodes();
+ArrayList l = new ArrayList<>(liveNodes);
+l.remove(coreContainer.getZkController().getNodeName());
+Collections.shuffle(l, BlobRepository.RANDOM);
+return l;
+ }
+
+ public static void validateFiles(
+ FileStore fileStore, List files, boolean validateSignatures,
Consumer errs) {
+for (String path : files) {
+ try {
+FileStore.FileType type = fileStore.getType(path, true);
+if (type != FileStore.FileType.FILE) {
+ errs.accept("No such file: " + path);
+ continue;
+}
+
+fileStore.get(
+path,
+entry -> {
+ if (entry.getMetaData().signatures == null
+ || entry.getMetaData().signatures.isEmpty()) {
+errs.accept(path + " has no signature");
+return;
+ }
+ if (validateSignatures) {
+try {
+ fileStore.refresh(ClusterFileStore.KEYS_DIR);
+ validate(fileStore, entry.meta.signatures, entry, false);
+} catch (Exception e) {
+ log.error("Error validating package artifact", e);
+ errs.accept(e.getMessage());
+}
+ }
+},
+false);
+ } catch (Exception e) {
+log.error("Error reading file ", e);
+errs.accept("Error reading file " + path + " " + e.getMessage());
+ }
+}
+ }
+
+ /**
+ * Validate a file for signature
+ *
+ * @param sigs the signatures. atleast one should succeed
Review Comment:
`@param signatures The Signatures, at least one should succeed` ?
##
solr/core/src/java/org/apache/solr/filestore/DistribFileStore.java:
##
@@ -634,7 +635,7 @@ public Map getKeys() throws IOException {
// reads local keys file
private static Map
Re: [PR] User Behavior Insights implementation for Apache Solr [solr]
epugh commented on PR #2452: URL: https://github.com/apache/solr/pull/2452#issuecomment-2121447550 This is definitely draft mode code... I opened it as a PR just to be able to track the work, and once it gets a bit furthur, I plan on opening a proper discussion about it. Module? Solr Sandbox? A Component? A full blown package? So many fun options... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-17297) Classloading issue with plugin and modules
[
https://issues.apache.org/jira/browse/SOLR-17297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848000#comment-17848000
]
David Smiley commented on SOLR-17297:
-
I believe it's only one classloader across modules & shardLib. On the other
hand, "packages" are isolated.
If one day modules get classloader isolation, then I think we would have to
invoke the Lucene SPI stuff for each separate classloader.
> Classloading issue with plugin and modules
> --
>
> Key: SOLR-17297
> URL: https://issues.apache.org/jira/browse/SOLR-17297
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Affects Versions: 9.3
>Reporter: Patson Luk
>Priority: Critical
>
> h2. Summary
> Using plugin jar and enabling any modules could trigger
> {{java.lang.ClassNotFoundException}}
> h2. Description
> 1. An implementation of {{org.apache.lucene.codecs.PostingsFormat}} with the
> jar within the /lib
> 2. Enable modules in solr.xml for example name="modules">opentelemetry
> 3. Now on startup. As a part of {{NodeConfig#setupSharedLib}}, it would load
> all the SPIs, it locates our jar and loads the class with a
> {{FactoryURLClassLoader}} with the classpaths point at the jar of the lib,
> which is correct
> 4. After {{NodeConfig#setupSharedLib}}, {{NodeConfig#initModules}} is
> invoked, which eventually calls {{SolrResourceLoader#addURLsToClassLoader}}
> that closes the previous class loader, which is the one used in 3.
> 5. Now a core is loaded with that codec, it runs the code which
> references other classes within our plugin jar, but unfortunately it would
> use the Classloader that loads our class in step 3., and such loader is
> marked as "closed" hence no longer load the correct resource/class. This
> triggers ClassNotFoundException.
> I have tried several things, the only thing that seems to work so far is
> commenting out {{IOUtils.closeWhileHandlingException(oldLoader);}} in
> {{SolrResourceLoader#addURLsToClassLoader}}, which is likely not the right
> workaround as the {{closeWhileHandlingException}} should be there for a
> reason ;)
> Switching {{setupSharedLib}} and {{initModules}} might work too (haven't
> tested), but I don't want to try any weird changes since I don't really know
> the ordering significance.
> Would appreciate some helps from the Solr experts! :)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-10654) Expose Metrics in Prometheus format DIRECTLY from Solr
[
https://issues.apache.org/jira/browse/SOLR-10654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17847998#comment-17847998
]
David Smiley commented on SOLR-10654:
-
Overall I wonder what others think about maintaining two parallel yet
consistent metrics mappings -- one in the "Prometheus Exporter" configured
using lots of "jq" and that which is very flexible (intended for users to
configure/hack as needed), the second what this PR does, basically as
hard-coded as can be. For example if we add a new metric, we then probably
need to update the exporter's config, and also edit source code being added
here. This could be helped by having the Prometheus Exporter fetch certain
metrics pass-through on-demand. But based on the design of the Prometheus
Exporter, I think that could be tricky/awkward.
> Expose Metrics in Prometheus format DIRECTLY from Solr
> --
>
> Key: SOLR-10654
> URL: https://issues.apache.org/jira/browse/SOLR-10654
> Project: Solr
> Issue Type: Improvement
> Components: metrics
>Reporter: Keith Laban
>Priority: Major
> Attachments: prometheus_metrics.txt
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Expose metrics via a `wt=prometheus` response type.
> Example scape_config in prometheus.yml:
> {code}
> scrape_configs:
> - job_name: 'solr'
> metrics_path: '/solr/admin/metrics'
> params:
> wt: ["prometheus"]
> static_configs:
> - targets: ['localhost:8983']
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-10654: Introduce output of Prometheus metrics for Solr Core registry [solr]
dsmiley commented on code in PR #2405:
URL: https://github.com/apache/solr/pull/2405#discussion_r1607338383
##
solr/core/src/java/org/apache/solr/core/SolrCore.java:
##
@@ -3032,6 +3034,7 @@ public PluginBag
getResponseWriters() {
m.put("csv", new CSVResponseWriter());
m.put("schema.xml", new SchemaXmlResponseWriter());
m.put("smile", new SmileResponseWriter());
+m.put(PROMETHEUS_METRICS_WT, new PrometheusResponseWriter());
Review Comment:
This is so special-purpose that I think it doesn't belong here. Can we get
away with not registering it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-10654: Introduce output of Prometheus metrics for Solr Core registry [solr]
dsmiley commented on code in PR #2405:
URL: https://github.com/apache/solr/pull/2405#discussion_r1607336727
##
solr/core/src/java/org/apache/solr/response/PrometheusResponseWriter.java:
##
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.response;
+
+import io.prometheus.metrics.expositionformats.PrometheusTextFormatWriter;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.Map;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.metrics.prometheus.SolrPrometheusCoreExporter;
+import org.apache.solr.request.SolrQueryRequest;
+
+@SuppressWarnings(value = "unchecked")
+public class PrometheusResponseWriter extends RawResponseWriter {
Review Comment:
Needs javadocs. Like say that this is *ONLY* used by the MetricsHandler
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-10654: Introduce output of Prometheus metrics for Solr Core registry [solr]
dsmiley commented on code in PR #2405:
URL: https://github.com/apache/solr/pull/2405#discussion_r1607332586
##
solr/core/src/java/org/apache/solr/metrics/prometheus/SolrPrometheusCoreExporter.java:
##
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.metrics.prometheus;
+
+import com.codahale.metrics.Meter;
+import com.codahale.metrics.Metric;
+import org.apache.solr.metrics.prometheus.core.SolrCoreCacheMetric;
+import org.apache.solr.metrics.prometheus.core.SolrCoreHandlerMetric;
+import org.apache.solr.metrics.prometheus.core.SolrCoreHighlighterMetric;
+import org.apache.solr.metrics.prometheus.core.SolrCoreIndexMetric;
+import org.apache.solr.metrics.prometheus.core.SolrCoreMetric;
+import org.apache.solr.metrics.prometheus.core.SolrCoreNoOpMetric;
+import org.apache.solr.metrics.prometheus.core.SolrCoreSearcherMetric;
+import org.apache.solr.metrics.prometheus.core.SolrCoreTlogMetric;
+
+/**
+ * This class maintains a {@link
io.prometheus.metrics.model.snapshots.MetricSnapshot}s exported
+ * from solr.core {@link com.codahale.metrics.MetricRegistry}
+ */
+public class SolrPrometheusCoreExporter extends SolrPrometheusExporter {
+ public final String coreName;
+ public final boolean cloudMode;
+ public static final String ADMIN = "ADMIN";
+ public static final String QUERY = "QUERY";
+ public static final String UPDATE = "UPDATE";
+ public static final String REPLICATION = "REPLICATION";
+ public static final String TLOG = "TLOG";
+ public static final String CACHE = "CACHE";
+ public static final String SEARCHER = "SEARCHER";
+ public static final String HIGHLIGHTER = "HIGHLIGHTER";
+ public static final String INDEX = "INDEX";
+ public static final String CORE = "CORE";
+
+ public SolrPrometheusCoreExporter(String coreName, boolean cloudMode) {
+super();
+this.coreName = coreName;
+this.cloudMode = cloudMode;
+ }
+
+ /**
+ * Export {@link Metric} to {@link
io.prometheus.metrics.model.snapshots.MetricSnapshot} and
+ * registers the Snapshot
+ *
+ * @param dropwizardMetric the {@link Meter} to be exported
+ * @param metricName Dropwizard metric name
+ */
+ @Override
+ public void exportDropwizardMetric(Metric dropwizardMetric, String
metricName) {
+SolrCoreMetric solrCoreMetric = categorizeCoreMetric(dropwizardMetric,
metricName);
+solrCoreMetric.parseLabels().toPrometheus(this);
+ }
+
+ private SolrCoreMetric categorizeCoreMetric(Metric dropwizardMetric, String
metricName) {
+String metricCategory = metricName.split("\\.")[0];
Review Comment:
I'd rather we don't split merely to get the first component.
An alternative approach here is to have many startsWith checks so that we're
not even parsing/extracting. I leave it to you to pick a path; either are fine
I think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-10654: Introduce output of Prometheus metrics for Solr Core registry [solr]
dsmiley commented on code in PR #2405:
URL: https://github.com/apache/solr/pull/2405#discussion_r1607222435
##
solr/core/src/java/org/apache/solr/util/stats/MetricUtils.java:
##
@@ -165,6 +169,71 @@ public static void toSolrInputDocuments(
});
}
+ /**
+ * Provides a representation of the given Dropwizard metric registry as
{@link
+ * SolrPrometheusCoreExporter}-s. Only those metrics are converted which
match at least one of the
+ * given MetricFilter instances.
+ *
+ * @param registry the {@link MetricRegistry} to be converted
+ * @param shouldMatchFilters a list of {@link MetricFilter} instances. A
metric must match any
+ * one of the filters from this list to be included in the output
+ * @param mustMatchFilter a {@link MetricFilter}. A metric must
match this filter to be
+ * included in the output.
+ * @param propertyFilter limit what properties of a metric are returned
+ * @param skipHistograms discard any {@link Histogram}-s and histogram parts
of {@link Timer}-s.
+ * @param skipAggregateValues discard internal values of {@link
AggregateMetric}-s.
+ * @param compact use compact representation for counters and gauges.
+ * @param consumer consumer that accepts produced {@link
SolrPrometheusCoreExporter}-s
+ */
+ public static void toPrometheusRegistry(
+ MetricRegistry registry,
+ String registryName,
+ List shouldMatchFilters,
+ MetricFilter mustMatchFilter,
+ Predicate propertyFilter,
+ boolean skipHistograms,
+ boolean skipAggregateValues,
+ boolean compact,
+ Consumer consumer) {
+String coreName;
+boolean cloudMode = false;
+Map dropwizardMetrics = registry.getMetrics();
+String[] rawParsedRegistry = registryName.split("\\.");
+List parsedRegistry = new
ArrayList<>(Arrays.asList(rawParsedRegistry));
+
+if (parsedRegistry.size() == 3) {
+ coreName = parsedRegistry.get(2);
+} else if (parsedRegistry.size() == 5) {
+ coreName =
parsedRegistry.stream().skip(1).collect(Collectors.joining("_"));
+ cloudMode = true;
+} else {
+ coreName = registryName;
+}
+
+SolrPrometheusCoreExporter solrPrometheusCoreExporter =
Review Comment:
minor: Good use for "var" here
##
solr/core/src/java/org/apache/solr/metrics/prometheus/core/SolrCoreMetric.java:
##
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.metrics.prometheus.core;
+
+import com.codahale.metrics.Metric;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.solr.metrics.prometheus.SolrPrometheusCoreExporter;
+
+/**
+ * Base class is a wrapper to categorize and export {@link
com.codahale.metrics.Metric} to {@link
+ * io.prometheus.metrics.model.snapshots.DataPointSnapshot} and register to a
{@link
+ * SolrPrometheusCoreExporter}. {@link com.codahale.metrics.MetricRegistry}
does not support tags
+ * unlike prometheus. Metrics registered to the registry need to be parsed out
from the metric name
+ * to be exported to {@link
io.prometheus.metrics.model.snapshots.DataPointSnapshot}
+ */
+public abstract class SolrCoreMetric {
+ public Metric dropwizardMetric;
+ public String coreName;
+ public String metricName;
+ public Map labels = new HashMap<>();
+
+ public SolrCoreMetric(
+ Metric dropwizardMetric, String coreName, String metricName, boolean
cloudMode) {
+this.dropwizardMetric = dropwizardMetric;
+this.coreName = coreName;
+this.metricName = metricName;
+labels.put("core", coreName);
+if (cloudMode) {
+ String[] coreNameParsed = coreName.split("_");
+ labels.put("collection", coreNameParsed[1]);
Review Comment:
I could see adding some resiliency to this such that underscores in the
collection name still work
##
solr/core/src/java/org/apache/solr/util/stats/MetricUtils.java:
##
@@ -165,6 +169,71 @@ public static void toSolrInputDocuments(
});
}
+ /**
+ * Provides a representation of the given Dropwizard metric registry as
{@link
+ * SolrPrometheusCoreExporter}-s. Only those metrics are converted which
match at least one of the
+ * given
Re: [PR] User Behavior Insights implementation for Apache Solr [solr]
chatman commented on PR #2452: URL: https://github.com/apache/solr/pull/2452#issuecomment-2121233115 Usually, features like these are discussed in the dev@ list, or in JIRA or a SIP. Most important question I have in mind is whether this needs to be in the core search engine? If not, can this not be a plugin/package, shipped outside of solr-core? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[PR] SOLR-17302: Convert /cluster filestore APIs to JAX-RS [solr]
gerlowskija opened a new pull request, #2470: URL: https://github.com/apache/solr/pull/2470 https://issues.apache.org/jira/browse/SOLR-17302 # Description Solr has slowly been modifying its v2 APIs to align around a more REST-ful model, implemented using the JAX-RS framework. Many APIs have already been through this re-alignment, but many more remain including Solr's filestore APIs (used internally by package store). # Solution This commit migrates several of the filestore APIs, namely those under the `/cluster/files` path to use the JAX-RS framework. The APIs themselves are unchanged from an end-user perspective. Other endpoints, particularly the "fetch filestore entry" (`GET /api/node/files/somepath.txt`) and "delete local filestore entry" (`DELETE /api/node/files/somepath.txt`) are not attempted in this PR, but left for another. # Tests Existing filestore and package-store tests continue to pass. # Checklist Please review the following and check all that apply: - [x] I have reviewed the guidelines for [How to Contribute](https://github.com/apache/solr/blob/main/CONTRIBUTING.md) and my code conforms to the standards described there to the best of my ability. - [x] I have created a Jira issue and added the issue ID to my pull request title. - [x] I have given Solr maintainers [access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork) to contribute to my PR branch. (optional but recommended) - [x] I have developed this patch against the `main` branch. - [ ] I have run `./gradlew check`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-13350) Explore collector managers for multi-threaded search
[ https://issues.apache.org/jira/browse/SOLR-13350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17847967#comment-17847967 ] Ishan Chattopadhyaya commented on SOLR-13350: - bq. Are there other risks/trade-offs for enabling it besides timeAllowed being unsupported? Basically, when would a user not want this? I'll add some guidance for users to the reference guide on this, before closing this issue. > Explore collector managers for multi-threaded search > > > Key: SOLR-13350 > URL: https://issues.apache.org/jira/browse/SOLR-13350 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Ishan Chattopadhyaya >Priority: Major > Attachments: SOLR-13350.patch, SOLR-13350.patch, SOLR-13350.patch > > Time Spent: 11.5h > Remaining Estimate: 0h > > AFAICT, SolrIndexSearcher can be used only to search all the segments of an > index in series. However, using CollectorManagers, segments can be searched > concurrently and result in reduced latency. Opening this issue to explore the > effectiveness of using CollectorManagers in SolrIndexSearcher from latency > and throughput perspective. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-13350) Explore collector managers for multi-threaded search
[ https://issues.apache.org/jira/browse/SOLR-13350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17847965#comment-17847965 ] Ishan Chattopadhyaya commented on SOLR-13350: - bq. This is caused by breaking the end-to-end tracking of request context in SolrRequestInfo, which uses a thread-local deque to provide the same context for both the main and all sub-requests. This tracking is needed to setup the correct query timeout instance on the searcher ( QueryLimits ) for time-limited searches in the SolrIndexSearcher:727 . However, now that this method is executed in a separate "searcherCollector" thread the SolrRequestInfo instance it obtains is empty because it doesn't match the original thread that set it. QueryLimits has two parts: timeAllowed and cpuThreadLimits. # For timeAllowed, I can see that this value is passed to the searcher https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java#L726-L735. Hence, I think timeAllowed will be honoured by Lucene properly. # For cpuThreadLimits, the limits are set in SolrRequestInfo (https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/request/SolrRequestInfo.java#L86-L87). Seems like they are not getting carried over to the sub-threads. Noble Paul pointed me to the InheritableThreadLocalProvider for this. As of now, the timeAllowed requests are anyway executed without multithreading: https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/search/MultiThreadedSearcher.java#L125 I'm considering adding the cpuThreadLimits based requests also to this exception list. > Explore collector managers for multi-threaded search > > > Key: SOLR-13350 > URL: https://issues.apache.org/jira/browse/SOLR-13350 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Ishan Chattopadhyaya >Priority: Major > Attachments: SOLR-13350.patch, SOLR-13350.patch, SOLR-13350.patch > > Time Spent: 11.5h > Remaining Estimate: 0h > > AFAICT, SolrIndexSearcher can be used only to search all the segments of an > index in series. However, using CollectorManagers, segments can be searched > concurrently and result in reduced latency. Opening this issue to explore the > effectiveness of using CollectorManagers in SolrIndexSearcher from latency > and throughput perspective. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Updated] (SOLR-17301) Support all types of requests in rate limiting
[
https://issues.apache.org/jira/browse/SOLR-17301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Noble Paul updated SOLR-17301:
--
Description:
Rate limiting is now limited to QUERY , it should be possible to support all
types
{code:json}
{
"rate-limiters":
[
{
"type": "QUERY",
"allowedRequests": 5,
"slotAcquisitionTimeoutInMS": 100
},
{
"type": "UPDATE",
"allowedRequests": 25,
"slotAcquisitionTimeoutInMS": 100
}
]
{code}
was:
Rate limiting is now limited to QUERY , it should be possible to support all
types
{code:json}
{
"rate-limiters": {
[
{
"type": "QUERY",
"allowedRequests": 5,
"slotAcquisitionTimeoutInMS": 100
},
{
"type": "UPDATE",
"allowedRequests": 25,
"slotAcquisitionTimeoutInMS": 100
}
]
}
}
{code}
> Support all types of requests in rate limiting
> --
>
> Key: SOLR-17301
> URL: https://issues.apache.org/jira/browse/SOLR-17301
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Noble Paul
>Assignee: Noble Paul
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Rate limiting is now limited to QUERY , it should be possible to support all
> types
> {code:json}
> {
> "rate-limiters":
> [
> {
> "type": "QUERY",
>
> "allowedRequests": 5,
> "slotAcquisitionTimeoutInMS": 100
> },
> {
> "type": "UPDATE",
> "allowedRequests": 25,
> "slotAcquisitionTimeoutInMS": 100
> }
> ]
>
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17300: Copy existing listeners on re-creation of Http2SolrClient [solr]
dsmiley commented on PR #2467: URL: https://github.com/apache/solr/pull/2467#issuecomment-2121028103 Hmmm [TestInterleavingTransformer](https://ge.apache.org/scans/tests?search.relativeStartTime=P90D=solr-root=America%2FNew_York=org.apache.solr.ltr.response.transform.TestInterleavingTransformer) failed and it isn't a flapper: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Created] (SOLR-17302) Migrate filestore APIs to JAX-RS
Jason Gerlowski created SOLR-17302: -- Summary: Migrate filestore APIs to JAX-RS Key: SOLR-17302 URL: https://issues.apache.org/jira/browse/SOLR-17302 Project: Solr Issue Type: Sub-task Components: v2 API Reporter: Jason Gerlowski As mentioned on SOLR-15781, the v2 API currently has an experimental designation, and the community has expressed an interest in using this period to update our v2 endpoints to be more REST-ful and consistent. The current plan is to follow the specific changes laid out in [this spreadsheet|https://docs.google.com/spreadsheets/d/1HAoBBFPpSiT8mJmgNZKkZAPwfCfPvlc08m5jz3fQBpA/edit?usp=sharing], though of course nothing there is set in stone and there are still warts to be worked out. While we're touching the code for these endpoints, we should also convert them to JAX-RS framework definitions. (This was initially tracked as a separate effort - see SOLR-16370 - but the edit that were required ended up overlapping so significantly with the "cosmetic" improvements here that in practice it almost always makes sense to do the two together.) This ticket plans to tackle making the changes required for Solr's "filestore" APIs. Most of these APIs are fine cosmetically as-is, though this might be a good opportunity to iron out a few warts (particularly in how the "/api/node" and "/api/cluster" paths are used inconsistently, unless there's a rationale there that I'm missing...) *Cosmetic Changes and JAX-RS Conversion* ||API Name||Original Form||Desired Form||Status||Volunteer|| |Upload File|PUT /api/cluster/files/foo.txt|PUT /api/cluster/files/foo.txt|Open|N/A| |Delete File|DELETE /api/cluster/files/foo.txt|DELETE /api/cluster/files/foo.txt|Open|N/A| |Delete File (Local)|DELETE /api/note/files/foo.txt|???|Open|N/A| |Fetch File|GET /api/note/files/foo.txt|GET /api/node/files/foo.txt|Open|N/A| |Fetch File Metadata|GET /api/note/files/foo.txt?meta=true|GET /api/node/files/foo.txt?meta=true|Open|N/A| Some helpful links related to these changes these changes. Should help get any interested newcomers started! * For detailed information on Solr's current and desired v2 APIs see the spreadsheet [here|https://docs.google.com/spreadsheets/d/1HAoBBFPpSiT8mJmgNZKkZAPwfCfPvlc08m5jz3fQBpA/edit?usp=sharing] * [Step by Step guide to creating APIs|https://github.com/apache/solr/blob/main/dev-docs/apis.adoc#v2-apis] * [Example PR for a similar change|https://github.com/apache/solr/pull/1679] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17300: Copy existing listeners on re-creation of Http2SolrClient [solr]
dsmiley commented on code in PR #2467:
URL: https://github.com/apache/solr/pull/2467#discussion_r1607134511
##
solr/solrj/src/java/org/apache/solr/client/solrj/impl/Http2SolrClient.java:
##
@@ -149,10 +149,6 @@ public void addListenerFactory(HttpListenerFactory
factory) {
this.listenerFactory.add(factory);
}
- public List getListenerFactory() {
Review Comment:
Ah, very good then
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17300: Copy existing listeners on re-creation of Http2SolrClient [solr]
epugh commented on code in PR #2467:
URL: https://github.com/apache/solr/pull/2467#discussion_r1607025808
##
solr/core/src/test/org/apache/solr/cloud/RecoveryZkTestWithAuth.java:
##
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.cloud;
+
+import static
org.apache.solr.client.solrj.response.RequestStatusState.COMPLETED;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import org.apache.solr.client.solrj.SolrClient;
+import org.apache.solr.client.solrj.SolrQuery;
+import org.apache.solr.client.solrj.SolrRequest;
+import org.apache.solr.client.solrj.SolrResponse;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.impl.CloudLegacySolrClient;
+import org.apache.solr.client.solrj.impl.HttpSolrClient;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
+import org.apache.solr.client.solrj.request.QueryRequest;
+import org.apache.solr.client.solrj.request.UpdateRequest;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.client.solrj.response.RequestStatusState;
+import org.apache.solr.common.cloud.DocCollection;
+import org.apache.solr.common.cloud.Replica;
+import org.apache.solr.common.cloud.Slice;
+import org.apache.solr.util.SecurityJson;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class RecoveryZkTestWithAuth extends SolrCloudTestCase {
+ @BeforeClass
+ public static void setupCluster() throws Exception {
+cluster =
+configureCluster(2)
Review Comment:
Someday in dev-docs, we ought to have "The list of things @dsmiley looks for
when writing tests" ;-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17300: Copy existing listeners on re-creation of Http2SolrClient [solr]
iamsanjay commented on code in PR #2467:
URL: https://github.com/apache/solr/pull/2467#discussion_r1606972242
##
solr/core/src/test/org/apache/solr/cloud/RecoveryZkTestWithAuth.java:
##
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.cloud;
+
+import static
org.apache.solr.client.solrj.response.RequestStatusState.COMPLETED;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import org.apache.solr.client.solrj.SolrClient;
+import org.apache.solr.client.solrj.SolrQuery;
+import org.apache.solr.client.solrj.SolrRequest;
+import org.apache.solr.client.solrj.SolrResponse;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.impl.CloudLegacySolrClient;
+import org.apache.solr.client.solrj.impl.HttpSolrClient;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
+import org.apache.solr.client.solrj.request.QueryRequest;
+import org.apache.solr.client.solrj.request.UpdateRequest;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.client.solrj.response.RequestStatusState;
+import org.apache.solr.common.cloud.DocCollection;
+import org.apache.solr.common.cloud.Replica;
+import org.apache.solr.common.cloud.Slice;
+import org.apache.solr.util.SecurityJson;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class RecoveryZkTestWithAuth extends SolrCloudTestCase {
+ @BeforeClass
+ public static void setupCluster() throws Exception {
+cluster =
+configureCluster(2)
Review Comment:
Yes one node will do. +1 to reduce resources!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17300: Copy existing listeners on re-creation of Http2SolrClient [solr]
dsmiley commented on code in PR #2467:
URL: https://github.com/apache/solr/pull/2467#discussion_r1606946759
##
solr/core/src/test/org/apache/solr/cloud/RecoveryZkTestWithAuth.java:
##
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.cloud;
+
+import static
org.apache.solr.client.solrj.response.RequestStatusState.COMPLETED;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import org.apache.solr.client.solrj.SolrClient;
+import org.apache.solr.client.solrj.SolrQuery;
+import org.apache.solr.client.solrj.SolrRequest;
+import org.apache.solr.client.solrj.SolrResponse;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.impl.CloudLegacySolrClient;
+import org.apache.solr.client.solrj.impl.HttpSolrClient;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
+import org.apache.solr.client.solrj.request.QueryRequest;
+import org.apache.solr.client.solrj.request.UpdateRequest;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.client.solrj.response.RequestStatusState;
+import org.apache.solr.common.cloud.DocCollection;
+import org.apache.solr.common.cloud.Replica;
+import org.apache.solr.common.cloud.Slice;
+import org.apache.solr.util.SecurityJson;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class RecoveryZkTestWithAuth extends SolrCloudTestCase {
+ @BeforeClass
+ public static void setupCluster() throws Exception {
+cluster =
+configureCluster(2)
Review Comment:
will the test do its job with one node? If so, reduce resources to help
make our tests faster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17300: Copy existing listeners on re-creation of Http2SolrClient [solr]
dsmiley commented on code in PR #2467:
URL: https://github.com/apache/solr/pull/2467#discussion_r1606945391
##
solr/solrj/src/java/org/apache/solr/client/solrj/impl/Http2SolrClient.java:
##
@@ -149,10 +149,6 @@ public void addListenerFactory(HttpListenerFactory
factory) {
this.listenerFactory.add(factory);
}
- public List getListenerFactory() {
Review Comment:
Uh, you are removing a public method of an important SolrJ class? That
should not be done without backwards compatibility considerations. Leave it
please. Mark it deprecated if you prefer. The "add" method should be
deprecated (for immutability).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17300: Copy existing listeners on re-creation of Http2SolrClient [solr]
dsmiley commented on code in PR #2467:
URL: https://github.com/apache/solr/pull/2467#discussion_r1606862270
##
solr/core/src/test/org/apache/solr/cloud/RecoveryZkTestWithAuth.java:
##
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.cloud;
+
+import static
org.apache.solr.client.solrj.response.RequestStatusState.COMPLETED;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import org.apache.solr.client.solrj.SolrClient;
+import org.apache.solr.client.solrj.SolrQuery;
+import org.apache.solr.client.solrj.SolrRequest;
+import org.apache.solr.client.solrj.SolrResponse;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.impl.CloudLegacySolrClient;
+import org.apache.solr.client.solrj.impl.HttpSolrClient;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
+import org.apache.solr.client.solrj.request.QueryRequest;
+import org.apache.solr.client.solrj.request.UpdateRequest;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.client.solrj.response.RequestStatusState;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.cloud.DocCollection;
+import org.apache.solr.common.cloud.Replica;
+import org.apache.solr.common.cloud.Slice;
+import org.apache.solr.util.SecurityJson;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class RecoveryZkTestWithAuth extends SolrCloudTestCase {
+ @BeforeClass
+ public static void setupCluster() throws Exception {
+cluster =
+configureCluster(2)
+.addConfig("conf", configset("cloud-minimal"))
+.withSecurityJson(SecurityJson.SIMPLE)
+.configure();
+ }
+
+ private > T withBasicAuth(T
req) {
+req.setBasicAuthCredentials(SecurityJson.USER, SecurityJson.PASS);
+return req;
+ }
+
+ private QueryResponse queryWithBasicAuth(SolrClient client, SolrQuery q)
+ throws IOException, SolrServerException {
+return withBasicAuth(new QueryRequest(q)).process(client);
+ }
+
+ @Test
+ public void testRecoveryWithAuthEnabled() throws Exception {
+final String collection = "recoverytestwithauth";
+withBasicAuth(CollectionAdminRequest.createCollection(collection, "conf",
1, 1))
+.process(cluster.getSolrClient());
+waitForState(
+"Expected a collection with one shard and one replicas", collection,
clusterShape(1, 1));
+try (SolrClient solrClient =
+
cluster.basicSolrClientBuilder().withDefaultCollection(collection).build()) {
+ UpdateRequest commitReq = new UpdateRequest();
+ withBasicAuth(commitReq);
+ for (int i = 0; i < 500; i++) {
+SolrInputDocument doc = new SolrInputDocument();
+String[] fields = {"id", i + "", "name", "name = " + i};
+for (int j = 0; j < fields.length; j += 2) {
+ doc.addField(fields[j], fields[j + 1]);
+}
Review Comment:
This looks clumsy/verbose. why not `sdoc("id", i, "name", "name = " + i)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Resolved] (SOLR-17134) Bucketed rate limiting
[
https://issues.apache.org/jira/browse/SOLR-17134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Noble Paul resolved SOLR-17134.
---
Resolution: Duplicate
> Bucketed rate limiting
> --
>
> Key: SOLR-17134
> URL: https://issues.apache.org/jira/browse/SOLR-17134
> Project: Solr
> Issue Type: Improvement
>Reporter: Noble Paul
>Assignee: Noble Paul
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The current rate limiting functionality is limiting . We either get a rate
> limit or we don't. There are cases where we would like to rate limit only a
> certain type of requests.
> The usecase is as follows.
>
> We may have normal requests that go through without any rate limiting. When a
> certain header or param is passed , the requests are throttled. we should
> be able to throttle to
> * 'n1' requests in parallel for requests with a header/param value "x"
> * "n2" requests in parallel for requests with a header/param value "y"
> and so on
> The configuration would look as following
>
> {code:json}
> {
> "rate-limiters": {
> "enabled": true,
> "readBuckets": [
> {
> "name": "x-type",
> "header": {"solr-bucket-name": "x-type"},
> "allowedRequests": 5,
> "slotAcquisitionTimeoutInMS": 100
> },
> {
> "name": "y-type",
> "header": {"solr-bucket-name": "y-type"},
> "allowedRequests": 25,
> "slotAcquisitionTimeoutInMS": 100
> }
> ]
> }
> }
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Created] (SOLR-17301) Support all types of requests in rate limiting
Noble Paul created SOLR-17301:
-
Summary: Support all types of requests in rate limiting
Key: SOLR-17301
URL: https://issues.apache.org/jira/browse/SOLR-17301
Project: Solr
Issue Type: Improvement
Security Level: Public (Default Security Level. Issues are Public)
Reporter: Noble Paul
Rate limiting is now limited to QUERY , it should be possible to support all
types
{code:json}
{
"rate-limiters": {
[
{
"type": "QUERY",
"allowedRequests": 5,
"slotAcquisitionTimeoutInMS": 100
},
{
"type": "UPDATE",
"allowedRequests": 25,
"slotAcquisitionTimeoutInMS": 100
}
]
}
}
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Assigned] (SOLR-17301) Support all types of requests in rate limiting
[
https://issues.apache.org/jira/browse/SOLR-17301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Noble Paul reassigned SOLR-17301:
-
Assignee: Noble Paul
> Support all types of requests in rate limiting
> --
>
> Key: SOLR-17301
> URL: https://issues.apache.org/jira/browse/SOLR-17301
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Noble Paul
>Assignee: Noble Paul
>Priority: Major
>
> Rate limiting is now limited to QUERY , it should be possible to support all
> types
> {code:json}
> {
> "rate-limiters": {
> [
> {
> "type": "QUERY",
>
> "allowedRequests": 5,
> "slotAcquisitionTimeoutInMS": 100
> },
> {
> "type": "UPDATE",
> "allowedRequests": 25,
> "slotAcquisitionTimeoutInMS": 100
> }
> ]
> }
> }
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
[PR] SOLR-17301: Support all types of requests in rate limiting [solr]
noblepaul opened a new pull request, #2469: URL: https://github.com/apache/solr/pull/2469 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Updated] (SOLR-17300) Copy existing listeners on re-creation of Http2SolrClient
[
https://issues.apache.org/jira/browse/SOLR-17300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sanjay Dutt updated SOLR-17300:
---
Description:
For custom settings, such as timeouts, usually a Http2SolrClient is created
using the existing HTTP client using below code.
{code:java}
Http2SolrClient.Builder(leaderBaseUrl)
.withHttpClient(existingHttp2SolrClient)
.withIdleTimeout(soTimeout, TimeUnit.MILLISECONDS)
.withConnectionTimeout(connTimeout, TimeUnit.MILLISECONDS)
.build();
{code}
If not specified, withHttpClient method would automatically copy over some of
the older configuration automatically to the new Http2SolrClient
{code:java}
if (this.basicAuthAuthorizationStr == null) {
this.basicAuthAuthorizationStr =
http2SolrClient.basicAuthAuthorizationStr;
}
if (this.followRedirects == null) {
this.followRedirects = http2SolrClient.httpClient.isFollowRedirects();
}
if (this.idleTimeoutMillis == null) {
this.idleTimeoutMillis = http2SolrClient.idleTimeoutMillis;
}
if (this.requestWriter == null) {
this.requestWriter = http2SolrClient.requestWriter;
}
if (this.requestTimeoutMillis == null) {
this.requestTimeoutMillis = http2SolrClient.requestTimeoutMillis;
}
if (this.responseParser == null) {
this.responseParser = http2SolrClient.parser;
}
if (this.urlParamNames == null) {
this.urlParamNames = http2SolrClient.urlParamNames;
}
{code}
Nonetheless there is one field that did not pass over yet -- List of
HttpListenerFactory. This list also includes the interceptor for Auth due to
which re-created client were missing auth credentials and requests were failing.
*Proposed Solution* :-
Along with other properties, List of Listener Factory should also be copied
over from old to new client using withHttpClient method.
{code:java}
if (this.listenerFactory == null) {
this.listenerFactory = new ArrayList();
http2SolrClient.listenerFactory.forEach(this.listenerFactory::add);
}
{code}
was:
For custom settings, such as timeouts, usually a Http2SolrClient is created,
however using the existing HTTP client using below code.
{code:java}
Http2SolrClient.Builder(leaderBaseUrl)
.withHttpClient(existingHttp2SolrClient)
.withIdleTimeout(soTimeout, TimeUnit.MILLISECONDS)
.withConnectionTimeout(connTimeout, TimeUnit.MILLISECONDS)
.build();
{code}
If not specified, withHttpClient method would automatically copy over some of
the older configuration automatically.
{code:java}
if (this.basicAuthAuthorizationStr == null) {
this.basicAuthAuthorizationStr =
http2SolrClient.basicAuthAuthorizationStr;
}
if (this.followRedirects == null) {
this.followRedirects = http2SolrClient.httpClient.isFollowRedirects();
}
if (this.idleTimeoutMillis == null) {
this.idleTimeoutMillis = http2SolrClient.idleTimeoutMillis;
}
if (this.requestWriter == null) {
this.requestWriter = http2SolrClient.requestWriter;
}
if (this.requestTimeoutMillis == null) {
this.requestTimeoutMillis = http2SolrClient.requestTimeoutMillis;
}
if (this.responseParser == null) {
this.responseParser = http2SolrClient.parser;
}
if (this.urlParamNames == null) {
this.urlParamNames = http2SolrClient.urlParamNames;
}
{code}
Nonetheless there is one field that did not pass over yet -- List of
HttpListenerFactory. This list also includes the interceptor for Auth due to
which re-created client were missing auth credentials and requests were failing.
*Proposed Solution* :-
Along with other properties, List of Listener Factory should also be copied
over from old to new client using withHttpClient method.
{code:java}
if (this.listenerFactory == null) {
this.listenerFactory = new ArrayList();
http2SolrClient.listenerFactory.forEach(this.listenerFactory::add);
}
{code}
> Copy existing listeners on re-creation of Http2SolrClient
> -
>
> Key: SOLR-17300
> URL: https://issues.apache.org/jira/browse/SOLR-17300
> Project: Solr
> Issue Type: Sub-task
>Reporter: Sanjay Dutt
>Priority: Major
>
> For custom settings, such as timeouts, usually a Http2SolrClient is created
> using the existing HTTP client using below code.
> {code:java}
> Http2SolrClient.Builder(leaderBaseUrl)
> .withHttpClient(existingHttp2SolrClient)
> .withIdleTimeout(soTimeout, TimeUnit.MILLISECONDS)
> .withConnectionTimeout(connTimeout, TimeUnit.MILLISECONDS)
> .build();
> {code}
> If not specified, withHttpClient method would automatically copy
Re: [PR] SOLR-16505: Add auth listener to RecoveryStrategy SolrClient [solr]
iamsanjay closed pull request #2462: SOLR-16505: Add auth listener to RecoveryStrategy SolrClient URL: https://github.com/apache/solr/pull/2462 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Created] (SOLR-17300) Copy existing listeners on re-creation of Http2SolrClient
Sanjay Dutt created SOLR-17300:
--
Summary: Copy existing listeners on re-creation of Http2SolrClient
Key: SOLR-17300
URL: https://issues.apache.org/jira/browse/SOLR-17300
Project: Solr
Issue Type: Sub-task
Reporter: Sanjay Dutt
For custom settings, such as timeouts, usually a Http2SolrClient is created,
however using the existing HTTP client using below code.
{code:java}
Http2SolrClient.Builder(leaderBaseUrl)
.withHttpClient(existingHttp2SolrClient)
.withIdleTimeout(soTimeout, TimeUnit.MILLISECONDS)
.withConnectionTimeout(connTimeout, TimeUnit.MILLISECONDS)
.build();
{code}
If not specified, withHttpClient method would automatically copy over some of
the older configuration automatically.
{code:java}
if (this.basicAuthAuthorizationStr == null) {
this.basicAuthAuthorizationStr =
http2SolrClient.basicAuthAuthorizationStr;
}
if (this.followRedirects == null) {
this.followRedirects = http2SolrClient.httpClient.isFollowRedirects();
}
if (this.idleTimeoutMillis == null) {
this.idleTimeoutMillis = http2SolrClient.idleTimeoutMillis;
}
if (this.requestWriter == null) {
this.requestWriter = http2SolrClient.requestWriter;
}
if (this.requestTimeoutMillis == null) {
this.requestTimeoutMillis = http2SolrClient.requestTimeoutMillis;
}
if (this.responseParser == null) {
this.responseParser = http2SolrClient.parser;
}
if (this.urlParamNames == null) {
this.urlParamNames = http2SolrClient.urlParamNames;
}
{code}
Nonetheless there is one field that did not pass over yet -- List of
HttpListenerFactory. This list also includes the interceptor for Auth due to
which re-created client were missing auth credentials and requests were failing.
*Proposed Solution* :-
Along with other properties, List of Listener Factory should also be copied
over from old to new client using withHttpClient method.
{code:java}
if (this.listenerFactory == null) {
this.listenerFactory = new ArrayList();
http2SolrClient.listenerFactory.forEach(this.listenerFactory::add);
}
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] Alpine as base image [solr-docker]
manideepgoudparuchuri commented on PR #22: URL: https://github.com/apache/solr-docker/pull/22#issuecomment-2119832785 This is not the duplication of other codes. I have changed **FROM to use alpine as base** image and **changed commands which installs packages ** while it starts building the image. In other codes we used ubuntu image which uses apt package manager and now in alpine we use apk package manager for installing packages inside image. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
