Re: [PR] SOLR-13350: Multithreaded search [solr]
chatman commented on PR #2248: URL: https://github.com/apache/solr/pull/2248#issuecomment-2097393260 > Merging to main was unexpected to me because of the healthy code review happening here didn't conclude. I didn't want this to languish for a long time, and there were large PRs in the waiting that could affect this PR. Example: https://github.com/apache/solr/pull/2382 > I would have given this another look! ... > And I was looking forward to checking out the thread pool aspect again. Please feel free to look at it, would be happy to address all loose ends, if any. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] EncryptionBackupRepository to back up encrypted files. [solr-sandbox]
dsmiley commented on code in PR #106:
URL: https://github.com/apache/solr-sandbox/pull/106#discussion_r1591733443
##
encryption/src/main/java/org/apache/solr/encryption/EncryptionBackupRepository.java:
##
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.encryption;
+
+import org.apache.lucene.codecs.CodecUtil;
+import org.apache.lucene.store.ChecksumIndexInput;
+import org.apache.lucene.store.Directory;
+import org.apache.lucene.store.FilterDirectory;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.core.DirectoryFactory;
+import org.apache.solr.core.backup.repository.DelegatingBackupRepository;
+
+import java.io.IOException;
+import java.net.URI;
+
+import static
org.apache.solr.core.backup.repository.AbstractBackupRepository.PARAM_VERIFY_CHECKSUM;
+
+/**
+ * Encryption {@link org.apache.solr.core.backup.repository.BackupRepository}
that delegates
+ * to another {@link org.apache.solr.core.backup.repository.BackupRepository}
and intercepts
+ * index files copy to check their checksum in the decrypted from, but to copy
them in their
Review Comment:
The wording seems off; maybe a grammar mistake / accidental extra "copy"
word.
##
encryption/src/main/java/org/apache/solr/encryption/EncryptionBackupRepository.java:
##
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.encryption;
+
+import org.apache.lucene.codecs.CodecUtil;
+import org.apache.lucene.store.ChecksumIndexInput;
+import org.apache.lucene.store.Directory;
+import org.apache.lucene.store.FilterDirectory;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.core.DirectoryFactory;
+import org.apache.solr.core.backup.repository.DelegatingBackupRepository;
+
+import java.io.IOException;
+import java.net.URI;
+
+import static
org.apache.solr.core.backup.repository.AbstractBackupRepository.PARAM_VERIFY_CHECKSUM;
+
+/**
+ * Encryption {@link org.apache.solr.core.backup.repository.BackupRepository}
that delegates
+ * to another {@link org.apache.solr.core.backup.repository.BackupRepository}
and intercepts
+ * index files copy to check their checksum in the decrypted from, but to copy
them in their
+ * encrypted form.
+ */
+public class EncryptionBackupRepository extends DelegatingBackupRepository {
+
+ @Override
+ public void init(NamedList args) {
+if (delegate != null) {
+ delegate.init(args);
+}
+ }
+
+ @Override
+ protected NamedList getDelegateInitArgs(NamedList initArgs) {
+NamedList newInitArgs = new NamedList<>(initArgs.size() + 1);
+// Ensure the delegate does not verify the checksum, otherwise it would
fail since we back up encrypted files.
+newInitArgs.add(PARAM_VERIFY_CHECKSUM, Boolean.FALSE.toString());
+newInitArgs.addAll(initArgs);
+return newInitArgs;
+ }
+
+ @Override
+ public void copyIndexFileFrom(
+ Directory sourceDir, String sourceFileName, Directory destDir,
String destFileName)
+ throws IOException {
+// Read and verify the checksum with the potentially decrypting directory.
+verifyChecksum(sourceDir, sourceFileName);
+// Copy the index file with the unwrapped (delegate) directory to keep
encryption.
+super.copyIndexFileFrom(FilterDirectory.unwrap(sourceDir), sourceFileName,
destDir, destFileName);
+ }
+
+ @Override
+ public void copyIndexFileFrom(
+ Directory
[jira] [Commented] (SOLR-16924) Restore: Have RESTORECORE set the UpdateLog state
[
https://issues.apache.org/jira/browse/SOLR-16924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17844085#comment-17844085
]
David Smiley commented on SOLR-16924:
-
[~gerlowskija] I could use your insights on something here, please. I am
looking back at this months later and got very confused for a while until I
realized we have two RestoreCore classes, one in
{{org.apache.solr.handler.admin.api}} and the other in
{{{}org.apache.solr.handler{}}}. Wow; ok! The first implements the V2 API and
calls the second, the V1 API. The change in this PR was placed at the end of
the V2 API without consideration of the ambiguity. Uh oh! Thus it would seem
the change will not take effect for the V1 API {*}but{*}, I see in SOLR-16490
that RestoreCoreOp (yet another layer below the CoreAdminHandler but above the
real impl) calls the V2 RestoreCore. Wow again; I didn't expect that! I
wonder then, is V1 RestoreCore invoked in any other code path? I see two –
{{ReplicationHandler.restore()}} and {{{}InstallCoreData.installCoreData(){}}}.
Are there any problems or bugs here? Like, _should_ the change in this PR be
placed at the end of V1 instead? Isn't it wrong for CoreAdminHandler to be
calling v2 stuff? On second thought, I could rationalize that as we transition
the migration. Should the RestoreCore classes be merged?
> Restore: Have RESTORECORE set the UpdateLog state
> --
>
> Key: SOLR-16924
> URL: https://issues.apache.org/jira/browse/SOLR-16924
> Project: Solr
> Issue Type: Improvement
>Reporter: David Smiley
>Priority: Minor
> Fix For: 9.5
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> This is a refactoring improvement designed to simplify & clarify a step in
> collection restores. One of the final phases of RestoreCmd (collection
> restore) is to call REQUESTAPPLYUPDATES on each newly restored replica in
> order to transition the state of the UpdateLog to ACTIVE (not actually to
> apply updates). The underlying call on the UpdateLog could instead be done
> inside RESTORECORE at the end with explanatory comments as to the intent. I
> think this makes more sense that RESTORECORE finish with its UpdateLog ready.
> And it's strange/curious to see requests in the cluster to apply updates
> from an updateLog when there is none to do! Adding clarifying comments is
> important.
> See my comment:
> https://issues.apache.org/jira/browse/SOLR-12065?focusedCommentId=17751792=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17751792
> I think there isn't any back-compat concern.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17049: Fix Replica Down on startup logic [solr]
murblanc commented on code in PR #2432:
URL: https://github.com/apache/solr/pull/2432#discussion_r1591646646
##
solr/core/src/java/org/apache/solr/cloud/ZkController.java:
##
@@ -2849,9 +2834,14 @@ public boolean
checkIfCoreNodeNameAlreadyExists(CoreDescriptor dcore) {
* Best effort to set DOWN state for all replicas on node.
*
* @param nodeName to operate on
+ * @return the names of the collections that have replicas on the given node
*/
- public void publishNodeAsDown(String nodeName) {
+ public Collection publishNodeAsDown(String nodeName) {
log.info("Publish node={} as DOWN", nodeName);
+
+ClusterState clusterState = cc.getZkController().getClusterState();
+Map> replicasPerCollectionOnNode =
+clusterState.getReplicaNamesPerCollectionOnNode(nodeName);
Review Comment:
We would still need the loop to bring the replica states back to ACTIVE,
right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17192: Add "field-limiting" URP to catch ill-designed schemas [solr]
dsmiley commented on PR #2395: URL: https://github.com/apache/solr/pull/2395#issuecomment-2097012109 See https://github.com/gerlowskija/solr/pull/5 for a commit that makes a large simplification: * No NumFieldsMonitor * No named URP; anonymous class is fine (and less code) * URP need not be inserted in the happy path! The single line I refer to earlier is as follows in my PR to your PR: `final int currentNumFields = req.getSearcher().getFieldInfos().size();` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[PR] EncryptionBackupRepository to back up encrypted files. [solr-sandbox]
bruno-roustant opened a new pull request, #106: URL: https://github.com/apache/solr-sandbox/pull/106 A new EncryptionBackupRepository is required to backup correctly files encrypted with the EncryptionDirectory, otherwise index files are decrypted before the backup copy. EncryptionBackupRepository ensures the encrypted files are copied encrypted, but still verifies their checksum before the copy. It also ensures the files are copied encrypted when restored (not double encrypted). EncryptionBackupRepository extends DelegatingBackupRepository, so it requires solr 9.6.0. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[PR] EncryptionBackupRepository to back up encrypted files. [solr-sandbox]
bruno-roustant opened a new pull request, #105: URL: https://github.com/apache/solr-sandbox/pull/105 A new EncryptionBackupRepository is required to backup correctly files encrypted with the EncryptionDirectory, otherwise index files are decrypted before the backup copy. EncryptionBackupRepository ensures the encrypted files are copied encrypted, but still verifies their checksum before the copy. It also ensures the files are copied encrypted when restored (not double encrypted). EncryptionBackupRepository extends DelegatingBackupRepository, so it requires solr 9.6.0. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17192: Add "field-limiting" URP to catch ill-designed schemas [solr]
dsmiley commented on code in PR #2395:
URL: https://github.com/apache/solr/pull/2395#discussion_r1591513535
##
solr/core/src/java/org/apache/solr/update/processor/NumFieldLimitingUpdateRequestProcessorFactory.java:
##
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.update.processor;
+
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.util.plugin.SolrCoreAware;
+
+/**
+ * This factory generates an UpdateRequestProcessor which fails update
requests once a core has
+ * exceeded a configurable maximum number of fields. Meant as a safeguard to
help users notice
+ * potentially-dangerous schema design before performance and stability
problems start to occur.
+ *
+ * The URP uses the core's {@link SolrIndexSearcher} to judge the current
number of fields.
+ * Accordingly, it undercounts the number of fields in the core - missing all
fields added since the
+ * previous searcher was opened. As such, the URP's request-blocking is "best
effort" - it cannot be
+ * relied on as a precise limit on the number of fields.
+ *
+ * Additionally, the field-counting includes all documents present in the
index, including any
+ * deleted docs that haven't yet been purged via segment merging. Note that
this can differ
+ * significantly from the number of fields defined in managed-schema.xml -
especially when dynamic
+ * fields are enabled. The only way to reduce this field count is to delete
documents and wait until
+ * the deleted documents have been removed by segment merges. Users may of
course speed up this
+ * process by tweaking Solr's segment-merging, triggering an "optimize"
operation, etc.
+ *
+ * {@link NumFieldLimitingUpdateRequestProcessorFactory} accepts two
configuration parameters:
+ *
+ *
+ * maxFields - (required) The maximum number of fields
before update requests
+ * should be aborted. Once this limit has been exceeded, additional
update requests will fail
+ * until fields have been removed or the "maxFields" is increased.
+ * warnOnly - (optional) If true then the URP
logs verbose warnings
+ * about the limit being exceeded but doesn't abort update requests.
Defaults to false
+ *if not specified
+ *
+ *
+ * @since 9.6.0
+ */
+public class NumFieldLimitingUpdateRequestProcessorFactory extends
UpdateRequestProcessorFactory
+implements SolrCoreAware {
+
+ private static final String MAXIMUM_FIELDS_PARAM = "maxFields";
+ private static final String WARN_ONLY_PARAM = "warnOnly";
+
+ private NumFieldsMonitor numFieldsMonitor;
+ private int maximumFields;
+ private boolean warnOnly;
+
+ @Override
+ public void inform(final SolrCore core) {
+// register a commit callback for monitoring the number of fields in the
schema
+numFieldsMonitor = new NumFieldsMonitor(core);
+core.getUpdateHandler().registerCommitCallback(numFieldsMonitor);
+core.registerNewSearcherListener(numFieldsMonitor);
+ }
+
+ @Override
+ public void init(NamedList args) {
+warnOnly = args.indexOf(WARN_ONLY_PARAM, 0) > 0 ?
args.getBooleanArg(WARN_ONLY_PARAM) : false;
+
+if (args.indexOf(MAXIMUM_FIELDS_PARAM, 0) < 0) {
+ throw new IllegalArgumentException(
+ "The "
+ + MAXIMUM_FIELDS_PARAM
+ + " parameter is required for "
+ + getClass().getName()
+ + ", but no value was provided.");
+}
+final Object rawMaxFields = args.get(MAXIMUM_FIELDS_PARAM);
+if (rawMaxFields == null || !(rawMaxFields instanceof Integer)) {
+ throw new IllegalArgumentException(
+ MAXIMUM_FIELDS_PARAM + " must be configured as a non-null ");
+}
+maximumFields = (Integer) rawMaxFields;
+if (maximumFields <= 0) {
+ throw new IllegalArgumentException(MAXIMUM_FIELDS_PARAM + " must be a
positive integer");
+}
+ }
+
+ @Override
+ public UpdateRequestProcessor getInstance(
+ SolrQueryRequest req, SolrQueryResponse rsp, UpdateRequestProcessor
next) {
+
+
Re: [PR] SOLR-13350: Multithreaded search [solr]
dsmiley commented on PR #2248: URL: https://github.com/apache/solr/pull/2248#issuecomment-2096822608 > This pull request is closed, but the jira/solr-13350 branch has unmerged commits. Not sure what didn't make it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-13350: Multithreaded search [solr]
dsmiley commented on PR #2248: URL: https://github.com/apache/solr/pull/2248#issuecomment-2096821401 Merging to main was unexpected to me because of the healthy code review happening here didn't conclude. Next time please announce an intention to do so in a couple days. I would have given this another look! For example I *thought* the DocSet merging aspect was still tentative... I was awaiting another comment from Christine. And I was looking forward to checking out the thread pool aspect again. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-17278) Timezone with DST=true and DST=0 fails test
[
https://issues.apache.org/jira/browse/SOLR-17278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17843860#comment-17843860
]
ASF subversion and git services commented on SOLR-17278:
Commit cc095cc176c838694f150d0e13dbf650924e955a in solr's branch
refs/heads/branch_9x from Ishan Chattopadhyaya
[ https://gitbox.apache.org/repos/asf?p=solr.git;h=cc095cc176c ]
SOLR-17278: Skip broken timezones in the test
> Timezone with DST=true and DST=0 fails test
> ---
>
> Key: SOLR-17278
> URL: https://issues.apache.org/jira/browse/SOLR-17278
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Ishan Chattopadhyaya
>Priority: Major
> Attachments: SOLR-17278-1.patch, SOLR-17278.patch
>
>
> The TimeZoneUtilsTest#testValidIds() fails with some timezones that have
> useDaylightTime() as true but getDSTSavings() returns 0. Such timezones can't
> be converted to String using String.valueOf().
> Here's my JVM:
> {code}
> [ishan@deskmini solr] $ java -version
> openjdk version "11.0.22" 2024-01-16
> OpenJDK Runtime Environment (Red_Hat-11.0.22.0.7-1) (build 11.0.22+7)
> OpenJDK 64-Bit Server VM (Red_Hat-11.0.22.0.7-1) (build 11.0.22+7, mixed
> mode, sharing)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Resolved] (SOLR-17278) Timezone with DST=true and DST=0 fails test
[
https://issues.apache.org/jira/browse/SOLR-17278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ishan Chattopadhyaya resolved SOLR-17278.
-
Fix Version/s: 9.7
Resolution: Fixed
> Timezone with DST=true and DST=0 fails test
> ---
>
> Key: SOLR-17278
> URL: https://issues.apache.org/jira/browse/SOLR-17278
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Ishan Chattopadhyaya
>Priority: Major
> Fix For: 9.7
>
> Attachments: SOLR-17278-1.patch, SOLR-17278.patch
>
>
> The TimeZoneUtilsTest#testValidIds() fails with some timezones that have
> useDaylightTime() as true but getDSTSavings() returns 0. Such timezones can't
> be converted to String using String.valueOf().
> Here's my JVM:
> {code}
> [ishan@deskmini solr] $ java -version
> openjdk version "11.0.22" 2024-01-16
> OpenJDK Runtime Environment (Red_Hat-11.0.22.0.7-1) (build 11.0.22+7)
> OpenJDK 64-Bit Server VM (Red_Hat-11.0.22.0.7-1) (build 11.0.22+7, mixed
> mode, sharing)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-17278) Timezone with DST=true and DST=0 fails test
[
https://issues.apache.org/jira/browse/SOLR-17278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17843859#comment-17843859
]
ASF subversion and git services commented on SOLR-17278:
Commit d5b82412d8729a6bfa8056e7000a5a7be0b9b86e in solr's branch
refs/heads/main from Ishan Chattopadhyaya
[ https://gitbox.apache.org/repos/asf?p=solr.git;h=d5b82412d87 ]
SOLR-17278: Skip broken timezones in the test
> Timezone with DST=true and DST=0 fails test
> ---
>
> Key: SOLR-17278
> URL: https://issues.apache.org/jira/browse/SOLR-17278
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Ishan Chattopadhyaya
>Priority: Major
> Attachments: SOLR-17278-1.patch, SOLR-17278.patch
>
>
> The TimeZoneUtilsTest#testValidIds() fails with some timezones that have
> useDaylightTime() as true but getDSTSavings() returns 0. Such timezones can't
> be converted to String using String.valueOf().
> Here's my JVM:
> {code}
> [ishan@deskmini solr] $ java -version
> openjdk version "11.0.22" 2024-01-16
> OpenJDK Runtime Environment (Red_Hat-11.0.22.0.7-1) (build 11.0.22+7)
> OpenJDK 64-Bit Server VM (Red_Hat-11.0.22.0.7-1) (build 11.0.22+7, mixed
> mode, sharing)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-17278) Timezone with DST=true and DST=0 fails test
[
https://issues.apache.org/jira/browse/SOLR-17278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17843858#comment-17843858
]
Ishan Chattopadhyaya commented on SOLR-17278:
-
FYI, the tzdb.dat that's on my system came from Fedora Project 39.
/usr/lib/jvm/java-11-openjdk-11.0.22.0.7-1.fc39.x86_64/lib/tzdb.dat
> Timezone with DST=true and DST=0 fails test
> ---
>
> Key: SOLR-17278
> URL: https://issues.apache.org/jira/browse/SOLR-17278
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Ishan Chattopadhyaya
>Priority: Major
> Attachments: SOLR-17278-1.patch, SOLR-17278.patch
>
>
> The TimeZoneUtilsTest#testValidIds() fails with some timezones that have
> useDaylightTime() as true but getDSTSavings() returns 0. Such timezones can't
> be converted to String using String.valueOf().
> Here's my JVM:
> {code}
> [ishan@deskmini solr] $ java -version
> openjdk version "11.0.22" 2024-01-16
> OpenJDK Runtime Environment (Red_Hat-11.0.22.0.7-1) (build 11.0.22+7)
> OpenJDK 64-Bit Server VM (Red_Hat-11.0.22.0.7-1) (build 11.0.22+7, mixed
> mode, sharing)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17263: HttpJdkSolrClient doesn't encode curly braces etc [solr]
jdyer1 commented on code in PR #2433:
URL: https://github.com/apache/solr/pull/2433#discussion_r1591485125
##
solr/solrj/src/java/org/apache/solr/client/solrj/impl/HttpJdkSolrClient.java:
##
@@ -299,10 +299,23 @@ private PreparedRequest preparePutOrPost(
InputStream is = streams.iterator().next().getStream();
bodyPublisher = HttpRequest.BodyPublishers.ofInputStream(() -> is);
} else if (queryParams != null && urlParamNames != null) {
+ // rP is a new name for the incoming qP
ModifiableSolrParams requestParams = queryParams;
+ // qP becomes a new object, with selected params moved from original set
- when would urlParamNames be specified?
queryParams = calculateQueryParams(urlParamNames, requestParams);
+ // qP gains further (?) params moved from original set if solrRequest
has params, but only if urlParamNames was set too
queryParams.add(calculateQueryParams(solrRequest.getQueryParams(),
requestParams));
- bodyPublisher =
HttpRequest.BodyPublishers.ofString(requestParams.toString());
+ // bP receives any remaining params from original set
+ // with this version the params are not fully encoded - we get raw
Unicode chars, curly braces etc
+ // and the body content length changes, presumably due to re-encoding
+ // String bodyQueryString = requestParams.toString();
+ // with this version the params are fully encoded - note the
toQueryString() method adds an unwanted leading question mark
+ // but there's no longer a length change as the query is fully encoded
before being given to HttpRequest
+ String bodyQueryString = requestParams.toQueryString().substring(1);
+ bodyPublisher = HttpRequest.BodyPublishers.ofString(bodyQueryString);
+ // this isn't intended to be merged - but it shows the content length
change noted above
+ if (bodyQueryString.length() != bodyPublisher.contentLength()) throw new
URISyntaxException("inconsistent content length", bodyQueryString + " - " +
bodyQueryString.length() + " -> " + bodyPublisher.contentLength());
Review Comment:
I agree that we should remove this and not throw an exception if the content
length is off.
Do you think instead it would be possible to add a content length check to
one of the unit tests (that would fail without your code modification)?
##
solr/solrj/src/test/org/apache/solr/client/solrj/impl/HttpSolrClientTestBase.java:
##
@@ -337,24 +347,27 @@ protected void testCollectionParameters(
protected void setReqParamsOf(UpdateRequest req, String... keys) {
if (keys != null) {
for (String k : keys) {
-req.setParam(k, k + "Value");
+// note inclusion of non-ASCII character, and curly quotes which
should be URI encoded
+req.setParam(k, k + "Value\u1234{}");
}
}
}
- protected void verifyServletState(HttpSolrClientBase client, SolrRequest
request) {
+ protected void verifyServletState(HttpSolrClientBase client, SolrRequest
request) throws Exception {
// check query String
Iterator paramNames =
request.getParams().getParameterNamesIterator();
while (paramNames.hasNext()) {
String name = paramNames.next();
String[] values = request.getParams().getParams(name);
if (values != null) {
+// this can be enabled to see the raw query string (it's not expected
to be an empty string!)
+// assertEquals("", DebugServlet.queryString);
Review Comment:
I would prefer to get rid of the commented-out code here before merging. If
you think it is helpful you could add a debug-level log message. Better yet,
you could include the raw query string in the failure message in the subsequent
assert.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-17278) Timezone with DST=true and DST=0 fails test
[
https://issues.apache.org/jira/browse/SOLR-17278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17843854#comment-17843854
]
Ishan Chattopadhyaya commented on SOLR-17278:
-
{code}
1168 WARN (TEST-TimeZoneUtilsTest.testValidIds-seed#[FC7EEBC60D14381D]) [n: c:
s: r: x: t:] o.a.s.u.TimeZoneUtilsTest Not expecting DST to be 0 for Eastern
European Standard Time (actual: Eastern European Standard Time)
1169 WARN (TEST-TimeZoneUtilsTest.testValidIds-seed#[FC7EEBC60D14381D]) [n: c:
s: r: x: t:] o.a.s.u.TimeZoneUtilsTest Not expecting DST to be 0 for Eastern
European Standard Time (actual: Eastern European Standard Time)
{code}
> Timezone with DST=true and DST=0 fails test
> ---
>
> Key: SOLR-17278
> URL: https://issues.apache.org/jira/browse/SOLR-17278
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Ishan Chattopadhyaya
>Priority: Major
> Attachments: SOLR-17278-1.patch, SOLR-17278.patch
>
>
> The TimeZoneUtilsTest#testValidIds() fails with some timezones that have
> useDaylightTime() as true but getDSTSavings() returns 0. Such timezones can't
> be converted to String using String.valueOf().
> Here's my JVM:
> {code}
> [ishan@deskmini solr] $ java -version
> openjdk version "11.0.22" 2024-01-16
> OpenJDK Runtime Environment (Red_Hat-11.0.22.0.7-1) (build 11.0.22+7)
> OpenJDK 64-Bit Server VM (Red_Hat-11.0.22.0.7-1) (build 11.0.22+7, mixed
> mode, sharing)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Updated] (SOLR-17278) Timezone with DST=true and DST=0 fails test
[
https://issues.apache.org/jira/browse/SOLR-17278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ishan Chattopadhyaya updated SOLR-17278:
Attachment: SOLR-17278-1.patch
Status: Open (was: Open)
> Timezone with DST=true and DST=0 fails test
> ---
>
> Key: SOLR-17278
> URL: https://issues.apache.org/jira/browse/SOLR-17278
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Ishan Chattopadhyaya
>Priority: Major
> Attachments: SOLR-17278-1.patch, SOLR-17278.patch
>
>
> The TimeZoneUtilsTest#testValidIds() fails with some timezones that have
> useDaylightTime() as true but getDSTSavings() returns 0. Such timezones can't
> be converted to String using String.valueOf().
> Here's my JVM:
> {code}
> [ishan@deskmini solr] $ java -version
> openjdk version "11.0.22" 2024-01-16
> OpenJDK Runtime Environment (Red_Hat-11.0.22.0.7-1) (build 11.0.22+7)
> OpenJDK 64-Bit Server VM (Red_Hat-11.0.22.0.7-1) (build 11.0.22+7, mixed
> mode, sharing)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Updated] (SOLR-17278) Timezone with DST=true and DST=0 fails test
[
https://issues.apache.org/jira/browse/SOLR-17278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ishan Chattopadhyaya updated SOLR-17278:
Attachment: SOLR-17278.patch
Status: Open (was: Open)
Here's a patch to skip such timezones.
> Timezone with DST=true and DST=0 fails test
> ---
>
> Key: SOLR-17278
> URL: https://issues.apache.org/jira/browse/SOLR-17278
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Ishan Chattopadhyaya
>Priority: Major
> Attachments: SOLR-17278.patch
>
>
> The TimeZoneUtilsTest#testValidIds() fails with some timezones that have
> useDaylightTime() as true but getDSTSavings() returns 0. Such timezones can't
> be converted to String using String.valueOf().
> Here's my JVM:
> {code}
> [ishan@deskmini solr] $ java -version
> openjdk version "11.0.22" 2024-01-16
> OpenJDK Runtime Environment (Red_Hat-11.0.22.0.7-1) (build 11.0.22+7)
> OpenJDK 64-Bit Server VM (Red_Hat-11.0.22.0.7-1) (build 11.0.22+7, mixed
> mode, sharing)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Created] (SOLR-17278) Timezone with DST=true and DST=0 fails test
Ishan Chattopadhyaya created SOLR-17278:
---
Summary: Timezone with DST=true and DST=0 fails test
Key: SOLR-17278
URL: https://issues.apache.org/jira/browse/SOLR-17278
Project: Solr
Issue Type: Bug
Security Level: Public (Default Security Level. Issues are Public)
Reporter: Ishan Chattopadhyaya
The TimeZoneUtilsTest#testValidIds() fails with some timezones that have
useDaylightTime() as true but getDSTSavings() returns 0. Such timezones can't
be converted to String using String.valueOf().
Here's my JVM:
{code}
[ishan@deskmini solr] $ java -version
openjdk version "11.0.22" 2024-01-16
OpenJDK Runtime Environment (Red_Hat-11.0.22.0.7-1) (build 11.0.22+7)
OpenJDK 64-Bit Server VM (Red_Hat-11.0.22.0.7-1) (build 11.0.22+7, mixed mode,
sharing)
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-13350) Explore collector managers for multi-threaded search
[ https://issues.apache.org/jira/browse/SOLR-13350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17843845#comment-17843845 ] ASF subversion and git services commented on SOLR-13350: Commit f823e9b8695b05566a428cd7f7055714b6c325bd in solr's branch refs/heads/main from Ishan Chattopadhyaya [ https://gitbox.apache.org/repos/asf?p=solr.git;h=f823e9b8695 ] SOLR-13350: Fixing logging to be less verbose > Explore collector managers for multi-threaded search > > > Key: SOLR-13350 > URL: https://issues.apache.org/jira/browse/SOLR-13350 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Ishan Chattopadhyaya >Priority: Major > Attachments: SOLR-13350.patch, SOLR-13350.patch, SOLR-13350.patch > > Time Spent: 10h 40m > Remaining Estimate: 0h > > AFAICT, SolrIndexSearcher can be used only to search all the segments of an > index in series. However, using CollectorManagers, segments can be searched > concurrently and result in reduced latency. Opening this issue to explore the > effectiveness of using CollectorManagers in SolrIndexSearcher from latency > and throughput perspective. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-13350) Explore collector managers for multi-threaded search
[ https://issues.apache.org/jira/browse/SOLR-13350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17843842#comment-17843842 ] ASF subversion and git services commented on SOLR-13350: Commit a4979e0b80b93bbf7d76e7101de4fa34332672d7 in solr's branch refs/heads/main from Ishan Chattopadhyaya [ https://gitbox.apache.org/repos/asf?p=solr.git;h=a4979e0b80b ] SOLR-13350: Fixing tidy > Explore collector managers for multi-threaded search > > > Key: SOLR-13350 > URL: https://issues.apache.org/jira/browse/SOLR-13350 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Ishan Chattopadhyaya >Priority: Major > Attachments: SOLR-13350.patch, SOLR-13350.patch, SOLR-13350.patch > > Time Spent: 10h 40m > Remaining Estimate: 0h > > AFAICT, SolrIndexSearcher can be used only to search all the segments of an > index in series. However, using CollectorManagers, segments can be searched > concurrently and result in reduced latency. Opening this issue to explore the > effectiveness of using CollectorManagers in SolrIndexSearcher from latency > and throughput perspective. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-13350) Explore collector managers for multi-threaded search
[ https://issues.apache.org/jira/browse/SOLR-13350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17843841#comment-17843841 ] Ishan Chattopadhyaya commented on SOLR-13350: - bq. Do we have a limited queue for the pool, leading to the "RejectedExecution"? I think we'd like a caller-runs policy so we don't wait & starve Changed it to a larger queue than then number of threads, and removed the rejected execution handler as well. bq. Commit ff6607d25f023a59f866a66820037bb215342ca8 in solr's branch refs/heads/main from Ishan Chattopadhyaya Will backport to branch_9x after a few days of baking on main. > Explore collector managers for multi-threaded search > > > Key: SOLR-13350 > URL: https://issues.apache.org/jira/browse/SOLR-13350 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Ishan Chattopadhyaya >Priority: Major > Attachments: SOLR-13350.patch, SOLR-13350.patch, SOLR-13350.patch > > Time Spent: 10h 40m > Remaining Estimate: 0h > > AFAICT, SolrIndexSearcher can be used only to search all the segments of an > index in series. However, using CollectorManagers, segments can be searched > concurrently and result in reduced latency. Opening this issue to explore the > effectiveness of using CollectorManagers in SolrIndexSearcher from latency > and throughput perspective. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-13350: Multithreaded search [solr]
chatman commented on PR #2248: URL: https://github.com/apache/solr/pull/2248#issuecomment-2096693338 Thanks @cpoerschke and @dsmiley. I've merged to main, where this can bake for some days before merging to branch_9x. If there are any major outstanding issues, or any changes needed to documentation or default behaviour, we can take it up in another PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-13350) Explore collector managers for multi-threaded search
[ https://issues.apache.org/jira/browse/SOLR-13350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17843838#comment-17843838 ] ASF subversion and git services commented on SOLR-13350: Commit ff6607d25f023a59f866a66820037bb215342ca8 in solr's branch refs/heads/main from Ishan Chattopadhyaya [ https://gitbox.apache.org/repos/asf?p=solr.git;h=ff6607d25f0 ] SOLR-13350: Multithreaded search (closes #2248) > Explore collector managers for multi-threaded search > > > Key: SOLR-13350 > URL: https://issues.apache.org/jira/browse/SOLR-13350 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Ishan Chattopadhyaya >Priority: Major > Attachments: SOLR-13350.patch, SOLR-13350.patch, SOLR-13350.patch > > Time Spent: 10h 20m > Remaining Estimate: 0h > > AFAICT, SolrIndexSearcher can be used only to search all the segments of an > index in series. However, using CollectorManagers, segments can be searched > concurrently and result in reduced latency. Opening this issue to explore the > effectiveness of using CollectorManagers in SolrIndexSearcher from latency > and throughput perspective. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-13350: Multithreaded search [solr]
asfgit closed pull request #2248: SOLR-13350: Multithreaded search URL: https://github.com/apache/solr/pull/2248 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-13350: Multithreaded search [solr]
chatman commented on PR #2248: URL: https://github.com/apache/solr/pull/2248#issuecomment-2096670563 bq. I think that's an anti-pattern or broken and isn't what I meant in JIRA. We could use a SynchronousQueue (with fairness) if we want to block for a thread -- probably what we should do. FYI that queue is the default for Executors.newCachedThreadPool(). The "caller runs" behavior I meant could be done via an ExecutorService delegate that catches RejectedException and simply runs the Runnable. @dsmiley instead of using a rejected tasks execution handler, I went with @noblepaul 's suggestion of having a reasonably large queue for the threadpool (number of threads * 1000). Beyond this, if tasks are submitted, it is okay to reject them. We can revisit these limits later as well. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [I] Add support to set capabilities on Solr cloud container's security context [solr-operator]
bentastic27 commented on issue #489: URL: https://github.com/apache/solr-operator/issues/489#issuecomment-2096660400 I'm running into this issue on EKS. @janhoy / @HoustonPutman Are these extra capabilties/privs ever actually needed? If not, we can likely hardcode in the changes made in this comment: https://github.com/apache/solr-operator/issues/671#issuecomment-1875441585 Here is an example of a values.yaml I'm trying to use for a Solr Cluster: ``` securityContext: allowPrivilegeEscalation: false capabilities: drop: ["ALL"] runAsNonRoot: true seccompProfile: type: RuntimeDefault allowPrivilegeEscalation: false podOptions: labels: sidecar.istio.io/inject: 'false' podSecurityContext: <<: *securityContext zk: provided: zookeeperPodPolicy: securityContext: <<: *securityContext labels: sidecar.istio.io/inject: 'false' ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[PR] SOLR-17271 PerReplicaState: Shard leader elections still impact state.json [solr]
noblepaul opened a new pull request, #2443: URL: https://github.com/apache/solr/pull/2443 WIP. do not commit -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-17271) PerReplicaState: Shard leader elections still impact state.json
[ https://issues.apache.org/jira/browse/SOLR-17271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17843821#comment-17843821 ] Noble Paul commented on SOLR-17271: --- [~jsweeney] are you sure it is shard election that is causing the state.json update? I think it is the DOWNNODE message that caused the state.json update May be I need to do more testing > PerReplicaState: Shard leader elections still impact state.json > --- > > Key: SOLR-17271 > URL: https://issues.apache.org/jira/browse/SOLR-17271 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Justin Sweeney >Priority: Major > > From testing it has been found that shard leader elections when there are > multiple replicas still touches state.json, even when PerReplicaState (PRS) > is enabled. With PRS, leader should be contained in the per replica state so > we should not have to modify state.json. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Resolved] (SOLR-17088) TestPrepRecovery.testLeaderNotResponding fails much more lately
[ https://issues.apache.org/jira/browse/SOLR-17088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Houston Putman resolved SOLR-17088. --- Fix Version/s: main (10.0) Resolution: Fixed > TestPrepRecovery.testLeaderNotResponding fails much more lately > --- > > Key: SOLR-17088 > URL: https://issues.apache.org/jira/browse/SOLR-17088 > Project: Solr > Issue Type: Test >Reporter: David Smiley >Assignee: Houston Putman >Priority: Minor > Fix For: main (10.0) > > Attachments: 2023-11-27 fail.log.txt > > Time Spent: 50m > Remaining Estimate: 0h > > I'll attach logs. I didn't try and root cause. [Increased in test frequency > lately|http://fucit.org/solr-jenkins-reports/history-trend-of-recent-failures.html#series/org.apache.solr.cloud.TestPrepRecovery.testLeaderNotResponding]. > All recent failures happen on main, not 9x. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Created] (SOLR-17277) Circuit Breakers should support a "Warn-only" mode
Jason Gerlowski created SOLR-17277: -- Summary: Circuit Breakers should support a "Warn-only" mode Key: SOLR-17277 URL: https://issues.apache.org/jira/browse/SOLR-17277 Project: Solr Issue Type: Improvement Security Level: Public (Default Security Level. Issues are Public) Components: Circuit Breakers Affects Versions: main (10.0) Reporter: Jason Gerlowski The threshold values used when configuring Circuit Breakers are incredibly important. Identifying when load begins to get "dangerous" is tricky: too stringent and users will get 429s unnecessarily, too lax and the circuit breakers won't be effective at preventing crashes. Conservative users may wish to evaluate potential thresholds without actually blocking traffic. Could Circuit Breakers offer some sort of "Dry Run" capability - where a request that would otherwise have been blocked merely logs a message? This would allow users to get feedback on their thresholds before they actually start sending CB-based 429s to users. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17088: Fix custom solr.xml usage in tests with extra jettys [solr]
HoustonPutman merged PR #2437: URL: https://github.com/apache/solr/pull/2437 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[PR] SOLR-17276 Scrape metrics at fixed rate in prometheus-exporter [solr]
rafalh opened a new pull request, #2442: URL: https://github.com/apache/solr/pull/2442 https://issues.apache.org/jira/browse/SOLR-17276 # Description Metrics are not scraped by prometheus-exporter at a fixed rate because it uses `scheduleWithFixedDelay` API to schedule the scraping proces. Duration of scraping influences the actual rate. # Solution Make sure Solr metrics are scraped at fixed rate by using `scheduleAtFixedRate` API. Note: there will be no overlapping executions, because JavaDoc says: "If any execution of this task takes longer than its period, then subsequent executions may start late, but will not concurrently execute.". # Tests None # Checklist Please review the following and check all that apply: - [ ] I have reviewed the guidelines for [How to Contribute](https://github.com/apache/solr/blob/main/CONTRIBUTING.md) and my code conforms to the standards described there to the best of my ability. - [ ] I have created a Jira issue and added the issue ID to my pull request title. - [ ] I have given Solr maintainers [access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork) to contribute to my PR branch. (optional but recommended) - [ ] I have developed this patch against the `main` branch. - [ ] I have run `./gradlew check`. - [ ] I have added tests for my changes. - [ ] I have added documentation for the [Reference Guide](https://github.com/apache/solr/tree/main/solr/solr-ref-guide) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Created] (SOLR-17276) Use fixed rate instead of fixed delay in prometheus-exporter
Rafał Harabień created SOLR-17276: - Summary: Use fixed rate instead of fixed delay in prometheus-exporter Key: SOLR-17276 URL: https://issues.apache.org/jira/browse/SOLR-17276 Project: Solr Issue Type: Improvement Security Level: Public (Default Security Level. Issues are Public) Components: contrib - prometheus-exporter Reporter: Rafał Harabień Attachments: image-2024-05-06-18-10-30-739.png Prometheus-exporter is supposed to read metrics from Solr servers every 60 seconds (scrape interval can be changed using --scrape-interval argument). But the truth is it does it every 60+X seconds where X is the time needed to read metrics from all Solr servers. In my case X is 1-2 s. If Prometheus scrapes the exporter every 60 seconds it can lead to duplicated samples (e.g. metrics will stay the same for 2 minutes). !image-2024-05-06-18-10-30-739.png! It's result of using [scheduler.scheduleWithFixedDelay|https://github.com/apache/solr/blob/2bb2ada0a372f4d101b78df8d43e0fc44c8edbf3/solr/prometheus-exporter/src/java/org/apache/solr/prometheus/collector/SchedulerMetricsCollector.java#L77] instead of scheduler.scheduleAtFixedRate. Note: function scheduled with scheduleAtFixedRate can still be started late if previous execution has not finished. There is no risk of overlapping executions. I am going to prepare a PR. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Created] (SOLR-17275) Major performance regression of CloudSolrClient in Solr 9.6.0 when using aliases
Rafał Harabień created SOLR-17275:
-
Summary: Major performance regression of CloudSolrClient in Solr
9.6.0 when using aliases
Key: SOLR-17275
URL: https://issues.apache.org/jira/browse/SOLR-17275
Project: Solr
Issue Type: Bug
Security Level: Public (Default Security Level. Issues are Public)
Components: SolrJ
Affects Versions: 9.6.0
Environment: SolrJ 9.6.0, Ubuntu 22.04, Java 17
Reporter: Rafał Harabień
Attachments: image-2024-05-06-17-23-42-236.png
I observe worse performance of CloudSolrClient after upgrading from SolrJ 9.5.0
to 9.6.0, especially on p99.
p99 jumped from ~25 ms to ~400 ms
p90 jumped from ~9.9 ms to ~22 ms
p75 jumped from ~7 ms to ~11 ms
p50 jumped from ~4.5 ms to ~7.5 ms
Screenshot from Grafana (at ~14:30 was deployed the new version):
!image-2024-05-06-17-23-42-236.png!
I've got a thread-dump and I can see many threads waiting in
[ZkStateReader.forceUpdateCollection|https://github.com/apache/solr/blob/f8e5a93c11267e13b7b43005a428bfb910ac6e57/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/ZkStateReader.java#L503]:
{noformat}
Thread info: "suggest-solrThreadPool-thread-52" prio=5 Id=600 BLOCKED on
org.apache.solr.common.cloud.ZkStateReader@62e6bc3d owned by
"suggest-solrThreadPool-thread-34" Id=582
at
app//org.apache.solr.common.cloud.ZkStateReader.forceUpdateCollection(ZkStateReader.java:506)
- blocked on org.apache.solr.common.cloud.ZkStateReader@62e6bc3d
at
app//org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.getState(ZkClientClusterStateProvider.java:155)
at
app//org.apache.solr.client.solrj.impl.CloudSolrClient.resolveAliases(CloudSolrClient.java:1207)
at
app//org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1099)
at
app//org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:892)
at
app//org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:820)
at
app//org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:255)
at
app//org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:927)
...
Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@1beb7ed3
{noformat}
At the same time qTime from Solr hasn't changed so I'm pretty sure it's a
client regression.
I've tried reproducing it locally and I can see
[forceUpdateCollection|https://github.com/apache/solr/blob/f8e5a93c11267e13b7b43005a428bfb910ac6e57/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/ZkStateReader.java#L503]
function being called for every request in my application. I can see that
[this|https://github.com/apache/solr/commit/8cf552aa3642be473c6a08ce44feceb9cbe396d7]
commit
changed the logic in ZkClientClusterStateProvider.getState so the mentioned
function gets called if clusterState.getCollectionRef [returns
null|https://github.com/apache/solr/blob/f8e5a93c11267e13b7b43005a428bfb910ac6e57/solr/solrj-zookeeper/src/java/org/apache/solr/client/solrj/impl/ZkClientClusterStateProvider.java#L151].
In 9.5.0 it wasn't the case (forceUpdateCollection was not called in this
place). I can see in the debugger that getCollectionRef only supports
collections and not aliases (collectionStates map contains only collections).
In my application all collections are referenced using aliases so I guess
that's why I can see the regression in Solr response time.
I am not familiar with the code enough to prepare a PR but I hope this insight
will be enough to fix this issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-13360: Collation code fails with non-increasing token order [solr]
epugh commented on PR #210: URL: https://github.com/apache/solr/pull/210#issuecomment-2096253924 Looks like the PR needs a test to demonstrate the bug and therefore exercise the fix. These things are hard to evaluate without a unit test. Would you be willing to add one? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17192: Add "field-limiting" URP to catch ill-designed schemas [solr]
gerlowskija commented on code in PR #2395:
URL: https://github.com/apache/solr/pull/2395#discussion_r1591039849
##
solr/core/src/java/org/apache/solr/update/processor/NumFieldLimitingUpdateRequestProcessorFactory.java:
##
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.update.processor;
+
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.util.plugin.SolrCoreAware;
+
+/**
+ * This factory generates an UpdateRequestProcessor which fails update
requests once a core has
+ * exceeded a configurable maximum number of fields. Meant as a safeguard to
help users notice
+ * potentially-dangerous schema design before performance and stability
problems start to occur.
+ *
+ * The URP uses the core's {@link SolrIndexSearcher} to judge the current
number of fields.
+ * Accordingly, it undercounts the number of fields in the core - missing all
fields added since the
+ * previous searcher was opened. As such, the URP's request-blocking is "best
effort" - it cannot be
+ * relied on as a precise limit on the number of fields.
+ *
+ * Additionally, the field-counting includes all documents present in the
index, including any
+ * deleted docs that haven't yet been purged via segment merging. Note that
this can differ
+ * significantly from the number of fields defined in managed-schema.xml -
especially when dynamic
+ * fields are enabled. The only way to reduce this field count is to delete
documents and wait until
+ * the deleted documents have been removed by segment merges. Users may of
course speed up this
+ * process by tweaking Solr's segment-merging, triggering an "optimize"
operation, etc.
+ *
+ * {@link NumFieldLimitingUpdateRequestProcessorFactory} accepts two
configuration parameters:
+ *
+ *
+ * maxFields - (required) The maximum number of fields
before update requests
+ * should be aborted. Once this limit has been exceeded, additional
update requests will fail
+ * until fields have been removed or the "maxFields" is increased.
+ * warnOnly - (optional) If true then the URP
logs verbose warnings
+ * about the limit being exceeded but doesn't abort update requests.
Defaults to false
+ *if not specified
+ *
+ *
+ * @since 9.6.0
+ */
+public class NumFieldLimitingUpdateRequestProcessorFactory extends
UpdateRequestProcessorFactory
+implements SolrCoreAware {
+
+ private static final String MAXIMUM_FIELDS_PARAM = "maxFields";
+ private static final String WARN_ONLY_PARAM = "warnOnly";
+
+ private NumFieldsMonitor numFieldsMonitor;
+ private int maximumFields;
+ private boolean warnOnly;
+
+ @Override
+ public void inform(final SolrCore core) {
+// register a commit callback for monitoring the number of fields in the
schema
+numFieldsMonitor = new NumFieldsMonitor(core);
+core.getUpdateHandler().registerCommitCallback(numFieldsMonitor);
+core.registerNewSearcherListener(numFieldsMonitor);
+ }
+
+ @Override
+ public void init(NamedList args) {
+warnOnly = args.indexOf(WARN_ONLY_PARAM, 0) > 0 ?
args.getBooleanArg(WARN_ONLY_PARAM) : false;
+
+if (args.indexOf(MAXIMUM_FIELDS_PARAM, 0) < 0) {
+ throw new IllegalArgumentException(
+ "The "
+ + MAXIMUM_FIELDS_PARAM
+ + " parameter is required for "
+ + getClass().getName()
+ + ", but no value was provided.");
+}
+final Object rawMaxFields = args.get(MAXIMUM_FIELDS_PARAM);
+if (rawMaxFields == null || !(rawMaxFields instanceof Integer)) {
+ throw new IllegalArgumentException(
+ MAXIMUM_FIELDS_PARAM + " must be configured as a non-null ");
+}
+maximumFields = (Integer) rawMaxFields;
+if (maximumFields <= 0) {
+ throw new IllegalArgumentException(MAXIMUM_FIELDS_PARAM + " must be a
positive integer");
+}
+ }
+
+ @Override
+ public UpdateRequestProcessor getInstance(
+ SolrQueryRequest req, SolrQueryResponse rsp, UpdateRequestProcessor
next) {
+
Re: [PR] SOLR-17192: Add "field-limiting" URP to catch ill-designed schemas [solr]
gerlowskija commented on code in PR #2395:
URL: https://github.com/apache/solr/pull/2395#discussion_r1591064705
##
solr/core/src/java/org/apache/solr/update/processor/NumFieldLimitingUpdateRequestProcessor.java:
##
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.update.processor;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.Locale;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.update.AddUpdateCommand;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class NumFieldLimitingUpdateRequestProcessor extends
UpdateRequestProcessor {
Review Comment:
Missed this earlier; done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-13360: Collation code fails with non-increasing token order [solr]
geffchang commented on PR #210: URL: https://github.com/apache/solr/pull/210#issuecomment-2096067466 When is the plan to fix/merge this issue? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17192: Add "field-limiting" URP to catch ill-designed schemas [solr]
gerlowskija commented on code in PR #2395:
URL: https://github.com/apache/solr/pull/2395#discussion_r1591039849
##
solr/core/src/java/org/apache/solr/update/processor/NumFieldLimitingUpdateRequestProcessorFactory.java:
##
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.update.processor;
+
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.util.plugin.SolrCoreAware;
+
+/**
+ * This factory generates an UpdateRequestProcessor which fails update
requests once a core has
+ * exceeded a configurable maximum number of fields. Meant as a safeguard to
help users notice
+ * potentially-dangerous schema design before performance and stability
problems start to occur.
+ *
+ * The URP uses the core's {@link SolrIndexSearcher} to judge the current
number of fields.
+ * Accordingly, it undercounts the number of fields in the core - missing all
fields added since the
+ * previous searcher was opened. As such, the URP's request-blocking is "best
effort" - it cannot be
+ * relied on as a precise limit on the number of fields.
+ *
+ * Additionally, the field-counting includes all documents present in the
index, including any
+ * deleted docs that haven't yet been purged via segment merging. Note that
this can differ
+ * significantly from the number of fields defined in managed-schema.xml -
especially when dynamic
+ * fields are enabled. The only way to reduce this field count is to delete
documents and wait until
+ * the deleted documents have been removed by segment merges. Users may of
course speed up this
+ * process by tweaking Solr's segment-merging, triggering an "optimize"
operation, etc.
+ *
+ * {@link NumFieldLimitingUpdateRequestProcessorFactory} accepts two
configuration parameters:
+ *
+ *
+ * maxFields - (required) The maximum number of fields
before update requests
+ * should be aborted. Once this limit has been exceeded, additional
update requests will fail
+ * until fields have been removed or the "maxFields" is increased.
+ * warnOnly - (optional) If true then the URP
logs verbose warnings
+ * about the limit being exceeded but doesn't abort update requests.
Defaults to false
+ *if not specified
+ *
+ *
+ * @since 9.6.0
+ */
+public class NumFieldLimitingUpdateRequestProcessorFactory extends
UpdateRequestProcessorFactory
+implements SolrCoreAware {
+
+ private static final String MAXIMUM_FIELDS_PARAM = "maxFields";
+ private static final String WARN_ONLY_PARAM = "warnOnly";
+
+ private NumFieldsMonitor numFieldsMonitor;
+ private int maximumFields;
+ private boolean warnOnly;
+
+ @Override
+ public void inform(final SolrCore core) {
+// register a commit callback for monitoring the number of fields in the
schema
+numFieldsMonitor = new NumFieldsMonitor(core);
+core.getUpdateHandler().registerCommitCallback(numFieldsMonitor);
+core.registerNewSearcherListener(numFieldsMonitor);
+ }
+
+ @Override
+ public void init(NamedList args) {
+warnOnly = args.indexOf(WARN_ONLY_PARAM, 0) > 0 ?
args.getBooleanArg(WARN_ONLY_PARAM) : false;
+
+if (args.indexOf(MAXIMUM_FIELDS_PARAM, 0) < 0) {
+ throw new IllegalArgumentException(
+ "The "
+ + MAXIMUM_FIELDS_PARAM
+ + " parameter is required for "
+ + getClass().getName()
+ + ", but no value was provided.");
+}
+final Object rawMaxFields = args.get(MAXIMUM_FIELDS_PARAM);
+if (rawMaxFields == null || !(rawMaxFields instanceof Integer)) {
+ throw new IllegalArgumentException(
+ MAXIMUM_FIELDS_PARAM + " must be configured as a non-null ");
+}
+maximumFields = (Integer) rawMaxFields;
+if (maximumFields <= 0) {
+ throw new IllegalArgumentException(MAXIMUM_FIELDS_PARAM + " must be a
positive integer");
+}
+ }
+
+ @Override
+ public UpdateRequestProcessor getInstance(
+ SolrQueryRequest req, SolrQueryResponse rsp, UpdateRequestProcessor
next) {
+
Re: [PR] Remove 9.4 from list of supported versions [solr-docker]
janhoy merged PR #21: URL: https://github.com/apache/solr-docker/pull/21 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17260 ReleaseWizard should remove old minor versions from Docker Hub [solr]
janhoy merged PR #2430: URL: https://github.com/apache/solr/pull/2430 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
