[GitHub] [solr] risdenk commented on a diff in pull request #1158: SOLR-16368: experiment with builder to simplify client creation

[GitBox]

risdenk commented on code in PR #1158:
URL: https://github.com/apache/solr/pull/1158#discussion_r1011120330


##
solr/solrj/src/test/org/apache/solr/client/solrj/embedded/SolrExampleStreamingBinaryTest.java:
##
@@ -35,9 +34,15 @@ public class SolrExampleStreamingBinaryTest extends 
SolrExampleStreamingTest {
 
   @Override
   public SolrClient createNewSolrClient() {
-ConcurrentUpdateSolrClient client = (ConcurrentUpdateSolrClient) 
super.createNewSolrClient();
-client.setParser(new BinaryResponseParser());
-client.setRequestWriter(new BinaryRequestWriter());
+
+SolrClient client =
+new ErrorTrackingConcurrentUpdateSolrClient.Builder(getServerUrl())
+.withQueueSize(2)
+.withThreadCount(5)

Review Comment:
   Queue size and thread count? They aren't in the original?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[GitHub] [solr] epugh commented on pull request #1158: SOLR-16368: experiment with builder to simplify client creation

[GitBox]

epugh commented on PR #1158:
URL: https://github.com/apache/solr/pull/1158#issuecomment-1299471494

   If this all looks in the right direction, then I can tackle 
`Http2SolrClient` and `ConcurrentUpdateHttp2SolrClient`, moving them to builder 
pattern.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[GitHub] [solr] epugh commented on pull request #1158: SOLR-16368: experiment with builder to simplify client creation

[GitBox]

epugh commented on PR #1158:
URL: https://github.com/apache/solr/pull/1158#issuecomment-1299465793

   Can't figure out how to use builder pattern in `TestSmileRequest` ;-(


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[GitHub] [solr] epugh commented on pull request #1158: SOLR-16368: experiment with builder to simplify client creation

[GitBox]

epugh commented on PR #1158:
URL: https://github.com/apache/solr/pull/1158#issuecomment-1299363456

   I'd love another set of eyes on eb13c64057c000326087915e5810c3eea5339adf 
@risdenk, I *think* I simplified the test which meant I could remove the 
`setParser` code...   But maybe I lost what the test was all about...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[GitHub] [solr] epugh commented on a diff in pull request #1053: SOLR-16392: Refactor and update v2 DELETEREPLICAPROP API

[GitBox]

epugh commented on code in PR #1053:
URL: https://github.com/apache/solr/pull/1053#discussion_r1010962959


##
solr/core/src/java/org/apache/solr/handler/admin/CollectionsHandler.java:
##
@@ -1298,19 +1298,25 @@ public Map execute(
   V2ApiUtils.squashIntoSolrResponseWithoutHeader(rsp, 
addReplicaPropResponse);
   return null;
 }),
-// XXX should this command support followAliases?
 DELETEREPLICAPROP_OP(
 DELETEREPLICAPROP,
 (req, rsp, h) -> {
-  Map map =
-  copy(
-  req.getParams().required(),
-  null,
-  COLLECTION_PROP,
-  PROPERTY_PROP,
-  SHARD_ID_PROP,
-  REPLICA_PROP);
-  return copy(req.getParams(), map, PROPERTY_PROP);
+  final RequiredSolrParams requiredParams = req.getParams().required();
+  final String propNameToDelete = requiredParams.get(PROPERTY_PROP);
+  final String trimmedPropNameToDelete =
+  propNameToDelete.startsWith(PROPERTY_PREFIX)
+  ? propNameToDelete.substring(PROPERTY_PREFIX.length())
+  : propNameToDelete;
+  final DeleteReplicaPropertyAPI deleteReplicaPropertyAPI =

Review Comment:
   I don't think I'm the person to introduce new Java constructs to the Solr 
code base ;-)  I think if the community is up for it, great...   I might 
advocate that we do a big sweep through so that all the places `var` makes 
sense, we put that in, so it's not "a bit one way here, a bit the other way 
there"  And maybe we even have a way of checking that!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[GitHub] [solr] dsmiley commented on a diff in pull request #1053: SOLR-16392: Refactor and update v2 DELETEREPLICAPROP API

[GitBox]

dsmiley commented on code in PR #1053:
URL: https://github.com/apache/solr/pull/1053#discussion_r1010941513


##
solr/core/src/test/org/apache/solr/handler/admin/api/DeleteReplicaPropertyAPITest.java:
##
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.handler.admin.api;
+
+import static 
org.apache.solr.cloud.api.collections.CollectionHandlingUtils.SHARD_UNIQUE;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyLong;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+import io.opentracing.noop.NoopSpan;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.solr.SolrTestCaseJ4;
+import org.apache.solr.cloud.OverseerSolrResponse;
+import 
org.apache.solr.cloud.api.collections.DistributedCollectionConfigSetCommandRunner;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.cloud.ZkNodeProps;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.core.CoreContainer;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.SolrQueryResponse;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.mockito.ArgumentCaptor;
+
+/** Unit tests for {@link DeleteReplicaPropertyAPI} */
+public class DeleteReplicaPropertyAPITest extends SolrTestCaseJ4 {
+
+  private CoreContainer mockCoreContainer;
+  private DistributedCollectionConfigSetCommandRunner mockCommandRunner;
+  private SolrQueryRequest mockQueryRequest;
+  private SolrQueryResponse queryResponse;
+  private ArgumentCaptor messageCapturer;
+
+  private DeleteReplicaPropertyAPI deleteReplicaPropApi;
+
+  @BeforeClass
+  public static void ensureWorkingMockito() {
+assumeWorkingMockito();
+  }
+
+  @Before
+  public void setUp() throws Exception {
+super.setUp();
+
+mockCoreContainer = mock(CoreContainer.class);
+mockCommandRunner = 
mock(DistributedCollectionConfigSetCommandRunner.class);
+when(mockCoreContainer.getDistributedCollectionCommandRunner())
+.thenReturn(Optional.of(mockCommandRunner));
+when(mockCommandRunner.runCollectionCommand(any(), any(), anyLong()))
+.thenReturn(new OverseerSolrResponse(new NamedList<>()));
+mockQueryRequest = mock(SolrQueryRequest.class);
+when(mockQueryRequest.getSpan()).thenReturn(NoopSpan.INSTANCE);
+queryResponse = new SolrQueryResponse();
+messageCapturer = ArgumentCaptor.forClass(ZkNodeProps.class);
+
+deleteReplicaPropApi =
+new DeleteReplicaPropertyAPI(mockCoreContainer, mockQueryRequest, 
queryResponse);
+  }
+
+  @Test
+  public void testReportsErrorWhenCalledInStandaloneMode() {
+when(mockCoreContainer.isZooKeeperAware()).thenReturn(false);
+
+final SolrException e =
+expectThrows(
+SolrException.class,
+() -> {
+  deleteReplicaPropApi.deleteReplicaProperty(
+  "someColl", "someShard", "someReplica", "somePropName");
+});
+assertEquals(400, e.code());
+assertTrue(
+"Exception message differed from expected: " + e.getMessage(),
+e.getMessage().contains("not running in SolrCloud mode"));
+  }
+
+  @Test
+  public void testCreatesValidOverseerMessage() throws Exception {

Review Comment:
   > Yep, that's exactly what this PR does! 
   
   Can you substantiate that please?  I'm talking about a unit test that only 
tests the V1 to V2 mapping without implementation details of how V2 does its 
job.  I could imagine mocking (yes me, endorsing mocking) to capture what 
parameters are passed to DeleteReplicaPropertyAPI.deleteReplicaProperty() to 
ensure they align with a V1 call.  Mention of the Overseer would be an unwanted 
implementation detail.



##
solr/core/src/test/org/apache/solr/handler/admin/api/DeleteReplicaPropertyAPITest.java:
##
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * Th

[GitHub] [solr] epugh commented on pull request #1158: SOLR-16368: experiment with builder to simplify client creation

[GitBox]

epugh commented on PR #1158:
URL: https://github.com/apache/solr/pull/1158#issuecomment-1299286090

   @risdenk I'd love some input on the way I did deprecation.  Since the whole 
class is kind of deprecated, I didn't provide a bunch of `@deprecation` in the 
javadocs, or add the `since` tag...   Do you think I should?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[jira] [Commented] (SOLR-16497) Allow finer grained locking in SolrCores

[Jira]

[ 
https://issues.apache.org/jira/browse/SOLR-16497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627331#comment-17627331
 ] 

Torsten Bøgh Köster commented on SOLR-16497:


[~dsmiley] take your time, the change is quite huge :-/

A little background on our Solr setup: We run on AWS in a SolrCloud setup. 
Inside AWS we have an Autoscaling Group for each Solr shard. The ASG scales out 
at a given CPU usage percentage (currently 50%). We have a custom Solr cluster 
manager that assigns replicas to new Solr instances joining the SolrCloud. The 
ASG assigns a shard tag to each of its instances. This tag is picked up by the 
cluster manager to place the correct shard on the instance.

Unfortunately the Solr Autoscaling framework never really worked for us so we 
had to build the cluster manager ourselves. On the other hand it handles domain 
specific taks like index creation & validation.

I'm happy to answer any questions :-)

> Allow finer grained locking in SolrCores
> 
>
> Key: SOLR-16497
> URL: https://issues.apache.org/jira/browse/SOLR-16497
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: search
>Affects Versions: 9.0, 8.11.2
>Reporter: Torsten Bøgh Köster
>Assignee: David Smiley
>Priority: Major
> Attachments: solrcores_locking.png, solrcores_locking_fixed.png
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Access to loaded SolrCore instances is a synchronized read and write 
> operation in SolrCores#getCoreFromAnyList. This method is touched by every 
> request as every HTTP request is assigned the SolrCore it operates on.
> h3. Background
> Under heavy load we discovered that application halts inside of Solr are 
> becoming a serious problem in high traffic environments. Using Java Flight 
> Recordings we discovered high accumulated applications halts on the 
> modifyLock in SolrCores. In our case this means that we can only utilize our 
> machines up to 25% cpu usage. With the fix applied, a utilization up to 80% 
> is perfectly doable.
> In our case this specific locking problem was masked by another locking 
> problem in the SlowCompositeReaderWrapper. We'll submit our fix to the 
> locking problem in the SlowCompositeReaderWrapper in a following issue.
> h3. Problem
> Our Solr instances utilizes the collapse component heavily. The instances run 
> with 32 cores and 32gb Java heap on a rather small index (4gb). The instances 
> scale out at 50% cpu load. We take Java Flight Recorder snapshots of 60 
> seconds
> as soon the cpu usage exceeds 50%.
>  !solrcores_locking.png|width=1024! 
> During our 60s Java Flight Recorder snapshot, the ~2k Jetty acceptor threads 
> accumulated more than 12h locking time inside SolrCores on the modifyLock 
> instance used as synchronized lock (see screenshot). With this fix the 
> locking access is reduced to write accesses only. We validated this using 
> another JFR snapshot:
>  !solrcores_locking_fixed.png|width=1024! 
> We ran this code for a couple of weeks in our live environment.
> h3. Solution
> The synchronized modifyLock is replaced by a ReentrantReadWriteLock. This 
> allows concurrent reads from the internal SolrCore instance list (cores) but 
> grants exclusive access to write operations on the instance list (cores).
> In Solr 9.x the cache inside the TransientSolrCoreCacheFactoryDefault adds a 
> cache overflow handling of the size based internal cache (SOLR-15964). As 
> soon as SolrCores are evicted from the internal cache, the cache behaviour 
> changes from a size based cache to a reference based cache via the cache's 
> eviction handler. SolrCore instances that are still referenced are inserted 
> back into the cache. This means that write operations to the cache (insert 
> SolrCore) can be issued during read operations in SolrCores. Hence these 
> operations have only a read lock which cannot be upgraded to a write lock 
> (dead lock).
> To overcome this, we moved the cache maintenance (including the eviction 
> handler) in TransientSolrCoreCacheFactoryDefault to a separate thread. This 
> thread can acquire a write lock but on the other hand a separate thread will 
> schedule a ping-pong behaviour in the eviction handler on a full cache with 
> SolrCores still referenced. To overcome this we made the overflow behaviour 
> transparent by adding an additional overflowCores instance. Here we add 
> evicted but still referenced cores from the transientCores cache.
> Furthermore we need to ensure that only a single transientSolrCoreCache 
> inside TransientSolrCoreCacheFactoryDefault is created. As we now allow 
> multiple read threads, we call the the getTransientCacheHandler() method 
> initially holding a write lock ins

[GitHub] [solr] epugh commented on a diff in pull request #1158: SOLR-16368: experiment with builder to simplify client creation

[GitBox]

epugh commented on code in PR #1158:
URL: https://github.com/apache/solr/pull/1158#discussion_r1010874223


##
solr/solrj/src/java/org/apache/solr/client/solrj/impl/SolrClientBuilder.java:
##
@@ -46,6 +49,16 @@ public B withResponseParser(ResponseParser responseParser) {
 return getThis();
   }
 
+  public B withRequestWriter(RequestWriter requestWriter) {

Review Comment:
   yeah, I think if folks agree with moving to immutable SolrClient then yes?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[GitHub] [solr-site] HoustonPutman merged pull request #80: Warn about scans in security section.

[GitBox]

HoustonPutman merged PR #80:
URL: https://github.com/apache/solr-site/pull/80


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[GitHub] [solr] epugh commented on a diff in pull request #1158: SOLR-16368: experiment with builder to simplify client creation

[GitBox]

epugh commented on code in PR #1158:
URL: https://github.com/apache/solr/pull/1158#discussion_r1010870504


##
solr/solrj/src/test/org/apache/solr/client/solrj/embedded/SolrExampleStreamingBinaryTest.java:
##
@@ -35,9 +34,15 @@ public class SolrExampleStreamingBinaryTest extends 
SolrExampleStreamingTest {
 
   @Override
   public SolrClient createNewSolrClient() {
-ConcurrentUpdateSolrClient client = (ConcurrentUpdateSolrClient) 
super.createNewSolrClient();
-client.setParser(new BinaryResponseParser());
-client.setRequestWriter(new BinaryRequestWriter());
+
+SolrClient client =
+new ErrorTrackingConcurrentUpdateSolrClient.Builder(getServerUrl())
+.withQueueSize(2)
+.withThreadCount(5)

Review Comment:
   can you elaborate on what is not needed?   Defining the client here versus 
just returning from the .build()?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[GitHub] [solr] sonatype-lift[bot] commented on a diff in pull request #1158: SOLR-16368: experiment with builder to simplify client creation

[GitBox]

sonatype-lift[bot] commented on code in PR #1158:
URL: https://github.com/apache/solr/pull/1158#discussion_r1010832334


##
solr/test-framework/src/java/org/apache/solr/SolrJettyTestBase.java:
##
@@ -156,8 +160,7 @@ public synchronized SolrClient getSolrClient() {
   public SolrClient createNewSolrClient() {
 try {
   // setup the client...
-  final String url = jetty.getBaseUrl().toString() + "/" + "collection1";
-  final SolrClient client = getHttpSolrClient(url, 
DEFAULT_CONNECTION_TIMEOUT);
+  final SolrClient client = getHttpSolrClient(getServerUrl(), 
DEFAULT_CONNECTION_TIMEOUT);

Review Comment:
   *THREAD_SAFETY_VIOLATION:*  Unprotected write. Non-private method 
`SolrJettyTestBase.createNewSolrClient()` indirectly mutates container 
`util.ObjectReleaseTracker.OBJECTS` via call to `Map.put(...)` outside of 
synchronization.
Reporting because another access to the same memory occurs on a background 
thread, although this access may not.
   
   ---
   
   ℹ️ Learn about @sonatype-lift commands
   
   You can reply with the following commands. For example, reply with 
***@sonatype-lift ignoreall*** to leave out all findings.
   | **Command** | **Usage** |
   | - | - |
   | `@sonatype-lift ignore` | Leave out the above finding from this PR |
   | `@sonatype-lift ignoreall` | Leave out all the existing findings from this 
PR |
   | `@sonatype-lift exclude ` | Exclude specified 
`file\|issue\|path\|tool` from Lift findings by updating your config.toml file |
   
   **Note:** When talking to LiftBot, you need to **refresh** the page to see 
its response.
   [Click here](https://github.com/apps/sonatype-lift/installations/new) 
to add LiftBot to another repo.
   
   
   
   ---
   
   Was this a good recommendation?
   [ [🙁 Not 
relevant](https://www.sonatype.com/lift-comment-rating?comment=349964066&lift_comment_rating=1)
 ] - [ [😕 Won't 
fix](https://www.sonatype.com/lift-comment-rating?comment=349964066&lift_comment_rating=2)
 ] - [ [😑 Not critical, will 
fix](https://www.sonatype.com/lift-comment-rating?comment=349964066&lift_comment_rating=3)
 ] - [ [🙂 Critical, will 
fix](https://www.sonatype.com/lift-comment-rating?comment=349964066&lift_comment_rating=4)
 ] - [ [😊 Critical, fixing 
now](https://www.sonatype.com/lift-comment-rating?comment=349964066&lift_comment_rating=5)
 ]



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[GitHub] [solr] risdenk commented on a diff in pull request #1158: SOLR-16368: experiment with builder to simplify client creation

[GitBox]

risdenk commented on code in PR #1158:
URL: https://github.com/apache/solr/pull/1158#discussion_r1010801052


##
solr/solrj/src/test/org/apache/solr/client/solrj/SolrSchemalessExampleTest.java:
##
@@ -130,20 +130,18 @@ public void testFieldMutating() throws Exception {
 
   @Override
   public SolrClient createNewSolrClient() {
-try {
-  // setup the server...
-  String url = jetty.getBaseUrl().toString() + "/collection1";
-  HttpSolrClient client = getHttpSolrClient(url, 
DEFAULT_CONNECTION_TIMEOUT);
-  client.setUseMultiPartPost(random().nextBoolean());
-
-  if (random().nextBoolean()) {
-client.setParser(new BinaryResponseParser());
-client.setRequestWriter(new BinaryRequestWriter());
-  }
-
-  return client;
-} catch (Exception ex) {
-  throw new RuntimeException(ex);
+// setup the server...
+String url = jetty.getBaseUrl().toString() + "/collection1";
+HttpSolrClient.Builder httpSolrClientBuilder = new 
HttpSolrClient.Builder(url);
+if (random().nextBoolean()) {
+  httpSolrClientBuilder
+  .withRequestWriter(new BinaryRequestWriter())
+  .withResponseParser(new BinaryResponseParser());
+}
+if (random().nextBoolean()) {
+  httpSolrClientBuilder.withUseMultiPartPost(true);
 }

Review Comment:
   don't need to the if statement
   
   ```suggestion
   httpSolrClientBuilder.withUseMultiPartPost(random().nextBoolean());
   ```



##
solr/solrj/src/test/org/apache/solr/client/solrj/SolrSchemalessExampleTest.java:
##
@@ -130,20 +130,18 @@ public void testFieldMutating() throws Exception {
 
   @Override
   public SolrClient createNewSolrClient() {
-try {
-  // setup the server...
-  String url = jetty.getBaseUrl().toString() + "/collection1";
-  HttpSolrClient client = getHttpSolrClient(url, 
DEFAULT_CONNECTION_TIMEOUT);
-  client.setUseMultiPartPost(random().nextBoolean());
-
-  if (random().nextBoolean()) {
-client.setParser(new BinaryResponseParser());
-client.setRequestWriter(new BinaryRequestWriter());
-  }
-
-  return client;
-} catch (Exception ex) {
-  throw new RuntimeException(ex);
+// setup the server...
+String url = jetty.getBaseUrl().toString() + "/collection1";

Review Comment:
   should be `getServerUrl()`?



##
solr/solrj/src/test/org/apache/solr/client/solrj/embedded/SolrExampleStreamingBinaryTest.java:
##
@@ -35,9 +34,15 @@ public class SolrExampleStreamingBinaryTest extends 
SolrExampleStreamingTest {
 
   @Override
   public SolrClient createNewSolrClient() {
-ConcurrentUpdateSolrClient client = (ConcurrentUpdateSolrClient) 
super.createNewSolrClient();
-client.setParser(new BinaryResponseParser());
-client.setRequestWriter(new BinaryRequestWriter());
+
+SolrClient client =
+new ErrorTrackingConcurrentUpdateSolrClient.Builder(getServerUrl())
+.withQueueSize(2)
+.withThreadCount(5)

Review Comment:
   not needed



##
solr/solrj/src/java/org/apache/solr/client/solrj/impl/SolrClientBuilder.java:
##
@@ -46,6 +49,16 @@ public B withResponseParser(ResponseParser responseParser) {
 return getThis();
   }
 
+  public B withRequestWriter(RequestWriter requestWriter) {
+this.requestWriter = requestWriter;
+return getThis();
+  }
+
+  public B withUseMultiPartPost(Boolean useMultiPartPost) {

Review Comment:
   should deprecate `setUseMultiPartPost`?



##
solr/solrj/src/java/org/apache/solr/client/solrj/impl/SolrClientBuilder.java:
##
@@ -46,6 +49,16 @@ public B withResponseParser(ResponseParser responseParser) {
 return getThis();
   }
 
+  public B withRequestWriter(RequestWriter requestWriter) {

Review Comment:
   should deprecate `setRequestWriter`?



##
solr/solrj/src/test/org/apache/solr/client/solrj/SolrExampleBinaryTest.java:
##
@@ -32,19 +32,15 @@ public static void beforeTest() throws Exception {
 
   @Override
   public SolrClient createNewSolrClient() {
-try {
-  // setup the server...
-  String url = jetty.getBaseUrl().toString() + "/collection1";
-  HttpSolrClient client = getHttpSolrClient(url, 
DEFAULT_CONNECTION_TIMEOUT);
-  client.setUseMultiPartPost(random().nextBoolean());
+// setup the server...
+String url = jetty.getBaseUrl().toString() + "/collection1";

Review Comment:
   should be `getServerUrl()`?



##
solr/solrj/src/test/org/apache/solr/client/solrj/SolrExampleXMLTest.java:
##
@@ -32,15 +32,14 @@ public static void beforeTest() throws Exception {
 
   @Override
   public SolrClient createNewSolrClient() {
-try {
-  String url = jetty.getBaseUrl().toString() + "/collection1";
-  HttpSolrClient client = getHttpSolrClient(url, 
DEFAULT_CONNECTION_TIMEOUT);
-  client.setUseMultiPartPost(random().nextBoolean());
-  client.setParser(new XMLResponseParser());
-  client.setRe

[GitHub] [solr] epugh opened a new pull request, #1158: SOLR-16368: experiment with builder to simplify client creation

[GitBox]

epugh opened a new pull request, #1158:
URL: https://github.com/apache/solr/pull/1158

   https://issues.apache.org/jira/browse/SOLR-16368
   
   
   # Description
   
   Part of working on reducing the use of legacy `HttpSolrClient` in the tests 
everywhere is seeing if mutating the client can be reduced by embracing the 
Builder pattern.
   
   # Solution
   Use the Builder where possible.
   
   I couldn't figure out how to untangle the logic in `TestRandomFlRTGCloud` 
and would love some eyes on that one!   
   
   # Tests
   
   ran the tests
   
   # Checklist
   
   Please review the following and check all that apply:
   
   - [ ] I have reviewed the guidelines for [How to 
Contribute](https://wiki.apache.org/solr/HowToContribute) and my code conforms 
to the standards described there to the best of my ability.
   - [ ] I have created a Jira issue and added the issue ID to my pull request 
title.
   - [ ] I have given Solr maintainers 
[access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork)
 to contribute to my PR branch. (optional but recommended)
   - [ ] I have developed this patch against the `main` branch.
   - [ ] I have run `./gradlew check`.
   - [ ] I have added tests for my changes.
   - [ ] I have added documentation for the [Reference 
Guide](https://github.com/apache/solr/tree/main/solr/solr-ref-guide)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[GitHub] [solr] justinrsweeney opened a new pull request, #1157: Modifying github actions to run on all branches including ones with slashes

[GitBox]

justinrsweeney opened a new pull request, #1157:
URL: https://github.com/apache/solr/pull/1157

   https://issues.apache.org/jira/browse/SOLR-16516
   
   # Description
   
   This change modifies the Github Actions to execute on any PR opened against 
any branch, including branches with a `/` in the branch name. This can be 
useful for organizations that have forked the Solr repo and want to run checks 
against additional branches.
   
   # Solution
   
   This is a simple change to the pattern used to be `**` instead of `*` to 
include branches with slashes. More info here: 
https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#filter-pattern-cheat-sheet.
   
   # Tests
   
   N/A
   
   # Checklist
   
   Please review the following and check all that apply:
   
   - [ ] I have reviewed the guidelines for [How to 
Contribute](https://wiki.apache.org/solr/HowToContribute) and my code conforms 
to the standards described there to the best of my ability.
   - [ ] I have created a Jira issue and added the issue ID to my pull request 
title.
   - [ ] I have given Solr maintainers 
[access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork)
 to contribute to my PR branch. (optional but recommended)
   - [ ] I have developed this patch against the `main` branch.
   - [ ] I have run `./gradlew check`.
   - [ ] I have added tests for my changes.
   - [ ] I have added documentation for the [Reference 
Guide](https://github.com/apache/solr/tree/main/solr/solr-ref-guide)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[jira] [Created] (SOLR-16516) Execute Github Action for Gradle Checks on All PRs

[Justin Sweeney (Jira)]

Justin Sweeney created SOLR-16516:
-

 Summary: Execute Github Action for Gradle Checks on All PRs
 Key: SOLR-16516
 URL: https://issues.apache.org/jira/browse/SOLR-16516
 Project: Solr
  Issue Type: Improvement
  Security Level: Public (Default Security Level. Issues are Public)
  Components: github
Affects Versions: 9.1
Reporter: Justin Sweeney


Currently the Solr Github Actions configuration that runs Gradle checks on PRs 
is setup to run on PRs of branches using the pattern `*`. This means the action 
will be run on all open PRs against any branch except branches containing `/`.

I'd propose that this should actually use the pattern `**` which will also run 
against branches containing `/`. At FullStory we are using branches with `/` in 
our fork so this is inhibiting checks running for us and I don't see a reason 
whey we wouldn't generally want that behavior given `/` is a reasonable 
character in branches.

More detail here on the branch filter syntax: 
https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#filter-pattern-cheat-sheet



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[jira] [Commented] (SOLR-16514) lucene query: only QueryParser run well

[David Smiley (Jira)]

[ 
https://issues.apache.org/jira/browse/SOLR-16514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627281#comment-17627281
 ] 

David Smiley commented on SOLR-16514:
-

BTW, very recently, Lucene stopped using JIRA; it's read-only.  So I would not 
have been able to migrate the issue even if I felt it to be a suitable "issue".

> lucene query: only QueryParser run well
> ---
>
> Key: SOLR-16514
> URL: https://issues.apache.org/jira/browse/SOLR-16514
> Project: Solr
>  Issue Type: Task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: clients - python
>Reporter: Nicola Paganotti
>Priority: Critical
>  Labels: newbie
>
> {code:java}
>     def searchGiustizia(self,startPagination,recPerPage):
>         indexPath = File(self.fileLucene).toPath()
>         directory = FSDirectory.open(indexPath)
>         searcher = IndexSearcher(DirectoryReader.open(directory))         
>  
> paQuery5 = TermQuery(Term("parte","TELESI RICCARDO"))
>         analyzer = StandardAnalyzer()        print 
> ("\n--")
>         start = datetime.now()
>         collector = TotalHitCountCollector()
>         searcher.search(paQuery5, collector)
>         print("found: ",collector.getTotalHits())        
>         duration = datetime.now() - start
>         print("time ",str(duration))
> print ("\n--") 
> {code}
> This function do a simple search in my Index of Apache Lucene. I use 
> TermQuery but it return no results. If I use QueryParser, Lucene find me 
> record with parte = "TELESI RICCARDO". Why doesn't TermQuery work?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[jira] [Commented] (SOLR-16496) provide option for Query Elevation Component to bypass filters

[David Smiley (Jira)]

[ 
https://issues.apache.org/jira/browse/SOLR-16496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627278#comment-17627278
 ] 

David Smiley commented on SOLR-16496:
-

We can debate the top-level query parameter's name but -- yes, like that.  It 
would be {{elevate.something}} probably {{elevate.excludeTags}}.  Note that 
JSON Faceting 
[uses|https://solr.apache.org/guide/solr/latest/query-guide/json-faceting-domain-changes.html#filter-exclusions]
 this {{excludeTags}} name.

> provide option for Query Elevation Component to bypass filters
> --
>
> Key: SOLR-16496
> URL: https://issues.apache.org/jira/browse/SOLR-16496
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SearchComponents - other
>Reporter: Rudi Seitz
>Priority: Major
>
> The Query Elevation Component respects the fq parameter.
> A document listed in elevate.xml or specified via the {{elevateIds}} 
> parameter must match the provided filter queries in order to be included in 
> the result set for a given query. Documents that don't match the filter 
> queries will be excluded regardless of whether they are supposed to be 
> "elevated."
> In some cases, this behavior is desirable; in other cases, it is not. For 
> example, an ecommerce landing page might filter products according to whether 
> they are in stock ({{{}fq=inStock:true{}}}) but might wish to show certain 
> promoted products regardless of inventory.
> This ticket asks for an {{elevateFilteredDocs}} parameter that could be set 
> to true to include elevated documents in the result set regardless of whether 
> they match the provided filter queries. The default would be false, in 
> accordance with the current behavior.
> This parameter would allow elevated documents to "bypass" the provided 
> filters, while keeping the filters in place for non-elevated documents.
> From an implementation standpoint, this parameter could be supported with 
> code in {{QueryElevationComponent#setQuery}} that updates the filter queries 
> in similar way to how the main query is updated. When 
> {{{}elevateFilteredDocs=true{}}}, each filter query would become a boolean 
> "OR" of the original filter query with a second query matching the elevated 
> documents by id.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[jira] [Commented] (SOLR-5776) Look at speeding up using SSL with tests.

[Eric Pugh (Jira)]

[ 
https://issues.apache.org/jira/browse/SOLR-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627271#comment-17627271
 ] 

Eric Pugh commented on SOLR-5776:
-

Just stumbled over this ticket...     [~janhoy]  if you think this is something 
we can deal with now, I'd be happy to pick it up if you want to create a Jira ;)

> Look at speeding up using SSL with tests.
> -
>
> Key: SOLR-5776
> URL: https://issues.apache.org/jira/browse/SOLR-5776
> Project: Solr
>  Issue Type: Test
>Reporter: Mark Miller
>Assignee: Mark Miller
>Priority: Major
> Fix For: 4.9, 6.0
>
> Attachments: SOLR-5776.patch, SOLR-5776.patch, SOLR-5776.patch
>
>
> We have to disable SSL on a bunch of tests now because it appears to sometime 
> be ridiculously slow - especially in slow envs (I never see timeouts on my 
> machine).
> I was talking to Robert about this, and he mentioned that there might be some 
> settings we could change to speed it up.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[GitHub] [solr] joshgog commented on a diff in pull request #1080: SOLR 15749 created a v2 equivalent of v1 'rename'

[GitBox]

joshgog commented on code in PR #1080:
URL: https://github.com/apache/solr/pull/1080#discussion_r1010708916


##
solr/core/src/test/org/apache/solr/handler/admin/api/V2CollectionAPIMappingTest.java:
##
@@ -87,13 +96,15 @@ public void testGetCollectionStatus() throws Exception {
 
 
   @Test
-  public void testRenameCollectionAllParams() throws Exception {
-final SolrParams v1Params = captureConvertedV1Params(
-"/collections/collName", "POST", "{\"rename\": {\"to\": 
\"targetColl\"}}");
+  public void testRenameCollectionAllParams(){
+final SolrQueryRequest request = 
runRenameCollectionsApi("/collections/collName");

Review Comment:
   I'm not sure what exactly you would like to be clarified. If its about mocks 
checkout `AddReplicaPropertyAPITest` which uses mocked` SolrQueryRequest` and 
`SolrQueryResponse`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[GitHub] [solr] joshgog commented on a diff in pull request #1080: SOLR 15749 created a v2 equivalent of v1 'rename'

[GitBox]

joshgog commented on code in PR #1080:
URL: https://github.com/apache/solr/pull/1080#discussion_r1010708916


##
solr/core/src/test/org/apache/solr/handler/admin/api/V2CollectionAPIMappingTest.java:
##
@@ -87,13 +96,15 @@ public void testGetCollectionStatus() throws Exception {
 
 
   @Test
-  public void testRenameCollectionAllParams() throws Exception {
-final SolrParams v1Params = captureConvertedV1Params(
-"/collections/collName", "POST", "{\"rename\": {\"to\": 
\"targetColl\"}}");
+  public void testRenameCollectionAllParams(){
+final SolrQueryRequest request = 
runRenameCollectionsApi("/collections/collName");

Review Comment:
   I'm not sure what exactly you would like to be clarified. If its about mock 
checkout `AddReplicaPropertyAPITest` which uses mocked` SolrQueryRequest` and 
`SolrQueryResponse`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[jira] [Commented] (SOLR-16496) provide option for Query Elevation Component to bypass filters

[Rudi Seitz (Jira)]

[ 
https://issues.apache.org/jira/browse/SOLR-16496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627263#comment-17627263
 ] 

Rudi Seitz commented on SOLR-16496:
---

Thanks for the feedback [~dsmiley] 

 

Here is the ref-guide example of how to exclude filters selectively while 
faceting:

{{q=mainquery&fq=status:public&fq=\{!tag=dt}doctype:pdf&facet=true&facet.field=\{!ex=dt}doctype}}

 

In the case of elevation, I agree, the same approach to tagging a filter would 
work nicely ({{{}fq=\{!tag=dt}doctype:pdf{}}}).

But do you have an opinion on how the QEC-specific parameter to exclude some 
tags should look?

 

In the case of faceting, the tags to exclude are specified via the local 
parameter ex, as in: {{facet.field=\{!ex=dt}doctype}}

 

In the case of elevation, I don't think there's an obvious query parameter to 
which a local parameter like ex could be attached. I also don't think "ex" is 
appropriate as a name for the parameter because, in the elevation scenario, the 
filters won't be "excluded" or "bypassed" entirely but rather 
updated/modified/broadened to include the elevated documents.

 

What do you think about providing the tags via a top-level query parameter like 
this?

{{q=mainquery&fq=status:public&fq=\{!tag=dt}doctype:pdf&broadenFiltersForElevation=dt}}

 

> provide option for Query Elevation Component to bypass filters
> --
>
> Key: SOLR-16496
> URL: https://issues.apache.org/jira/browse/SOLR-16496
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SearchComponents - other
>Reporter: Rudi Seitz
>Priority: Major
>
> The Query Elevation Component respects the fq parameter.
> A document listed in elevate.xml or specified via the {{elevateIds}} 
> parameter must match the provided filter queries in order to be included in 
> the result set for a given query. Documents that don't match the filter 
> queries will be excluded regardless of whether they are supposed to be 
> "elevated."
> In some cases, this behavior is desirable; in other cases, it is not. For 
> example, an ecommerce landing page might filter products according to whether 
> they are in stock ({{{}fq=inStock:true{}}}) but might wish to show certain 
> promoted products regardless of inventory.
> This ticket asks for an {{elevateFilteredDocs}} parameter that could be set 
> to true to include elevated documents in the result set regardless of whether 
> they match the provided filter queries. The default would be false, in 
> accordance with the current behavior.
> This parameter would allow elevated documents to "bypass" the provided 
> filters, while keeping the filters in place for non-elevated documents.
> From an implementation standpoint, this parameter could be supported with 
> code in {{QueryElevationComponent#setQuery}} that updates the filter queries 
> in similar way to how the main query is updated. When 
> {{{}elevateFilteredDocs=true{}}}, each filter query would become a boolean 
> "OR" of the original filter query with a second query matching the elevated 
> documents by id.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[jira] [Resolved] (SOLR-10368) Randomize TestCollectionAPI to use both replication schemes

[Houston Putman (Jira)]

 [ 
https://issues.apache.org/jira/browse/SOLR-10368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Houston Putman resolved SOLR-10368.
---
  Assignee: (was: Cao Manh Dat)
Resolution: Cannot Reproduce

The test currently is randomized between tlog and nrt replication. It looks 
like the work was done in SOLR-10233

> Randomize TestCollectionAPI to use both replication schemes
> ---
>
> Key: SOLR-10368
> URL: https://issues.apache.org/jira/browse/SOLR-10368
> Project: Solr
>  Issue Type: Test
>Reporter: Ishan Chattopadhyaya
>Priority: Major
>
> As per SOLR-9835, the TestCollectionAPI was modified to *necessarily* use the 
> new replication scheme for creating the test collection.
> https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;a=blobdiff;f=solr/core/src/test/org/apache/solr/cloud/TestCollectionAPI.java;h=8fbfee391e0f27700723a2b45a11abe987b6236d;hp=8905077e4c800c087394fd3793c0514276780718;hb=7830462d4b7da3acefff6353419e71cde62d5fee;hpb=faeb1fe8c16f9e02aa5c3bba295bc24325b94a07
> {code}
> +  req.setRealtimeReplicas(1);
> {code}
> This leaves the default replication mode with no coverage with respect to the 
> collections API when replication factor is > 1 and numShards > 1. We should 
> randomize the use of new and old replication schemes in this test (and any 
> other test that was similarly changed). Alternatively, we can have another 
> collection in the test for the default replication scheme with 
> replicationFactor and numShards > 1.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[jira] [Updated] (SOLR-16497) Allow finer grained locking in SolrCores

[Kevin Risden (Jira)]

 [ 
https://issues.apache.org/jira/browse/SOLR-16497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kevin Risden updated SOLR-16497:

Status: Patch Available  (was: Open)

> Allow finer grained locking in SolrCores
> 
>
> Key: SOLR-16497
> URL: https://issues.apache.org/jira/browse/SOLR-16497
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: search
>Affects Versions: 9.0, 8.11.2
>Reporter: Torsten Bøgh Köster
>Assignee: David Smiley
>Priority: Major
> Attachments: solrcores_locking.png, solrcores_locking_fixed.png
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Access to loaded SolrCore instances is a synchronized read and write 
> operation in SolrCores#getCoreFromAnyList. This method is touched by every 
> request as every HTTP request is assigned the SolrCore it operates on.
> h3. Background
> Under heavy load we discovered that application halts inside of Solr are 
> becoming a serious problem in high traffic environments. Using Java Flight 
> Recordings we discovered high accumulated applications halts on the 
> modifyLock in SolrCores. In our case this means that we can only utilize our 
> machines up to 25% cpu usage. With the fix applied, a utilization up to 80% 
> is perfectly doable.
> In our case this specific locking problem was masked by another locking 
> problem in the SlowCompositeReaderWrapper. We'll submit our fix to the 
> locking problem in the SlowCompositeReaderWrapper in a following issue.
> h3. Problem
> Our Solr instances utilizes the collapse component heavily. The instances run 
> with 32 cores and 32gb Java heap on a rather small index (4gb). The instances 
> scale out at 50% cpu load. We take Java Flight Recorder snapshots of 60 
> seconds
> as soon the cpu usage exceeds 50%.
>  !solrcores_locking.png|width=1024! 
> During our 60s Java Flight Recorder snapshot, the ~2k Jetty acceptor threads 
> accumulated more than 12h locking time inside SolrCores on the modifyLock 
> instance used as synchronized lock (see screenshot). With this fix the 
> locking access is reduced to write accesses only. We validated this using 
> another JFR snapshot:
>  !solrcores_locking_fixed.png|width=1024! 
> We ran this code for a couple of weeks in our live environment.
> h3. Solution
> The synchronized modifyLock is replaced by a ReentrantReadWriteLock. This 
> allows concurrent reads from the internal SolrCore instance list (cores) but 
> grants exclusive access to write operations on the instance list (cores).
> In Solr 9.x the cache inside the TransientSolrCoreCacheFactoryDefault adds a 
> cache overflow handling of the size based internal cache (SOLR-15964). As 
> soon as SolrCores are evicted from the internal cache, the cache behaviour 
> changes from a size based cache to a reference based cache via the cache's 
> eviction handler. SolrCore instances that are still referenced are inserted 
> back into the cache. This means that write operations to the cache (insert 
> SolrCore) can be issued during read operations in SolrCores. Hence these 
> operations have only a read lock which cannot be upgraded to a write lock 
> (dead lock).
> To overcome this, we moved the cache maintenance (including the eviction 
> handler) in TransientSolrCoreCacheFactoryDefault to a separate thread. This 
> thread can acquire a write lock but on the other hand a separate thread will 
> schedule a ping-pong behaviour in the eviction handler on a full cache with 
> SolrCores still referenced. To overcome this we made the overflow behaviour 
> transparent by adding an additional overflowCores instance. Here we add 
> evicted but still referenced cores from the transientCores cache.
> Furthermore we need to ensure that only a single transientSolrCoreCache 
> inside TransientSolrCoreCacheFactoryDefault is created. As we now allow 
> multiple read threads, we call the the getTransientCacheHandler() method 
> initially holding a write lock inside the load() method. Calling the method 
> only needs a write lock initially (for cache creation). For all other calls, 
> a read lock is sufficient. By default, the getTransientCacheHandler() 
> acquires a read lock. If a write is needed (e.g. for core creation), the 
> getTransientCacheHandlerInternal() is called. This method explicitly does not 
> use a lock in order to provide the flexibility to choose between a read-lock 
> and a write-lock. This ensures that a single instance of 
> transientSolrCoreCache is created.
> The lock signaling between SolrCore and CoreContainer gets replaced by a 
> Condition that is tied to the write lock.
> h3. Summary
> This change allows for a finer grained access to the list of open SolrCores. 
> The decreased blocking read access is noticeabl

[jira] [Updated] (SOLR-16515) Remove synchronized access to cachedOrdMaps in SlowCompositeReaderWrapper

[Kevin Risden (Jira)]

 [ 
https://issues.apache.org/jira/browse/SOLR-16515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kevin Risden updated SOLR-16515:

Status: Patch Available  (was: Open)

> Remove synchronized access to cachedOrdMaps in SlowCompositeReaderWrapper
> -
>
> Key: SOLR-16515
> URL: https://issues.apache.org/jira/browse/SOLR-16515
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: search
>Affects Versions: 9.0, 8.11.2
>Reporter: Torsten Bøgh Köster
>Priority: Major
> Attachments: slow-composite-reader-wrapper-after.jpg, 
> slow-composite-reader-wrapper-before.jpg
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> The  {{SlowCompositeReaderWrapper}}  uses synchronized read and write access 
> to its internal  {{cachedOrdMaps}} . By using a  {{ConcurrentHashMap}}  
> instead of a  {{LinkedHashMap}}  as the  underlying  {{cachedOrdMaps}}  
> implementation and the  {{ConcurrentHashMap#computeIfAbsent}}  method to 
> compute cache values, we were able to reduce locking contention significantly.
> h3. Background
> Under heavy load we discovered that application halts inside of Solr are 
> becoming a serious problem in high traffic environments. Using Java Flight 
> Recordings we discovered high accumulated applications halts on the  
> {{cachedOrdMaps}}  in  {{SlowCompositeReaderWrapper}} . Without this fix we 
> were able to utilize our machines only up to 25% cpu usage. With the fix 
> applied, a utilization up to 80% is perfectly doable.
> h3. Description
> Our Solr instances utilizes the  {{collapse}}  component heavily. The 
> instances run with 32 cores and 32gb Java heap on a rather small index (4gb). 
> The instances scale out at 50% cpu load. We take Java Flight Recorder 
> snapshots of 60 seconds
> as soon the cpu usage exceeds 50%.
>  !slow-composite-reader-wrapper-before.jpg|width=1024! 
> During our 60s Java Flight Recorder snapshot, the ~2k Jetty threads 
> accumulated more than 16h locking time inside the  
> {{SlowCompositeReaderWrapper}}  (see screenshot). With this fix applied, the 
> locking access is reduced to cache write accesses only. We validated this 
> using another JFR snapshot:
>  !slow-composite-reader-wrapper-after.jpg|width=1024! 
> h3. Solution
> We propose the following improvement inside the  
> {{SlowCompositeReaderWrapper}}  removing blocking  {{synchronized}}  access 
> to the internal  {{cachedOrdMaps}} . The implementation keeps the semantics 
> of the  {{getSortedDocValues}}  and  {{getSortedSetDocValues}}  methods but 
> moves the expensive part of  {{OrdinalMap#build}}  into a producer. We use 
> the producer to access the  {{ConcurrentHashMap}}  using the  
> {{ConcurrentHashMap#computeIfAbsent}}  method only.
> The current implementation uses the  {{synchronized}}  block not only to lock 
> access to the  {{cachedOrdMaps}}  but also to protect the critical section 
> between getting, building and putting the  {{OrdinalMap}}  into the cache. 
> Inside the critical section the decision is formed, whether a cacheable value 
> should be composed and added to the cache. 
> To support non-blocking read access to the cache, we move the building part 
> of the critical section into a producer  {{Function}} . The check whether we 
> have a cacheable value is made upfront. To properly make that decision we had 
> to take logic from  {{MultiDocValues#getSortedSetValues}}  and  
> {{MultiDocValues#getSortedValues}}  (the  {{SlowCompositeReaderWrapper}}  
> already contained duplicated code from those methods).
> h3. Summary
> This change removes most blocking access inside the  
> {{SlowCompositeReaderWrapper}}  and despite it's name it's now 
> capable of a much higher request throughput.
> This change has been composed together by Dennis Berger, Torsten Bøgh Köster 
> and Marco Petris.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[jira] [Commented] (SOLR-15733) Separate out a solrj-streaming module

[Joel Bernstein (Jira)]

[ 
https://issues.apache.org/jira/browse/SOLR-15733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627238#comment-17627238
 ] 

Joel Bernstein commented on SOLR-15733:
---

Ok, will do.

> Separate out a solrj-streaming module
> -
>
> Key: SOLR-15733
> URL: https://issues.apache.org/jira/browse/SOLR-15733
> Project: Solr
>  Issue Type: Sub-task
>Reporter: Jan Høydahl
>Assignee: Joel Bernstein
>Priority: Major
>  Time Spent: 8h 40m
>  Remaining Estimate: 0h
>
> The huge amount of streaming expression code under 
> {{org.apache.solr.client.solrj.io}} package can be shipped as an optional 
> jar. The {{solrj-jdbc}} (or {{solrj-sql}}) modules would then have a 
> dependency on this...



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[jira] [Commented] (SOLR-16497) Allow finer grained locking in SolrCores

[David Smiley (Jira)]

[ 
https://issues.apache.org/jira/browse/SOLR-16497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627220#comment-17627220
 ] 

David Smiley commented on SOLR-16497:
-

I'm definitely interested, as I expect [~broustant] may be.  It'll take me some 
days to get back to look closer.

You mentioned "The instances scale out at 50% cpu load.".  This is off-topic 
here but nonetheless can you share more on the approach on how Solr is managed 
dynamically in this way?  SolrCloud?  Autoscaling framework? custom stuff...

> Allow finer grained locking in SolrCores
> 
>
> Key: SOLR-16497
> URL: https://issues.apache.org/jira/browse/SOLR-16497
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: search
>Affects Versions: 9.0, 8.11.2
>Reporter: Torsten Bøgh Köster
>Priority: Major
> Attachments: solrcores_locking.png, solrcores_locking_fixed.png
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Access to loaded SolrCore instances is a synchronized read and write 
> operation in SolrCores#getCoreFromAnyList. This method is touched by every 
> request as every HTTP request is assigned the SolrCore it operates on.
> h3. Background
> Under heavy load we discovered that application halts inside of Solr are 
> becoming a serious problem in high traffic environments. Using Java Flight 
> Recordings we discovered high accumulated applications halts on the 
> modifyLock in SolrCores. In our case this means that we can only utilize our 
> machines up to 25% cpu usage. With the fix applied, a utilization up to 80% 
> is perfectly doable.
> In our case this specific locking problem was masked by another locking 
> problem in the SlowCompositeReaderWrapper. We'll submit our fix to the 
> locking problem in the SlowCompositeReaderWrapper in a following issue.
> h3. Problem
> Our Solr instances utilizes the collapse component heavily. The instances run 
> with 32 cores and 32gb Java heap on a rather small index (4gb). The instances 
> scale out at 50% cpu load. We take Java Flight Recorder snapshots of 60 
> seconds
> as soon the cpu usage exceeds 50%.
>  !solrcores_locking.png|width=1024! 
> During our 60s Java Flight Recorder snapshot, the ~2k Jetty acceptor threads 
> accumulated more than 12h locking time inside SolrCores on the modifyLock 
> instance used as synchronized lock (see screenshot). With this fix the 
> locking access is reduced to write accesses only. We validated this using 
> another JFR snapshot:
>  !solrcores_locking_fixed.png|width=1024! 
> We ran this code for a couple of weeks in our live environment.
> h3. Solution
> The synchronized modifyLock is replaced by a ReentrantReadWriteLock. This 
> allows concurrent reads from the internal SolrCore instance list (cores) but 
> grants exclusive access to write operations on the instance list (cores).
> In Solr 9.x the cache inside the TransientSolrCoreCacheFactoryDefault adds a 
> cache overflow handling of the size based internal cache (SOLR-15964). As 
> soon as SolrCores are evicted from the internal cache, the cache behaviour 
> changes from a size based cache to a reference based cache via the cache's 
> eviction handler. SolrCore instances that are still referenced are inserted 
> back into the cache. This means that write operations to the cache (insert 
> SolrCore) can be issued during read operations in SolrCores. Hence these 
> operations have only a read lock which cannot be upgraded to a write lock 
> (dead lock).
> To overcome this, we moved the cache maintenance (including the eviction 
> handler) in TransientSolrCoreCacheFactoryDefault to a separate thread. This 
> thread can acquire a write lock but on the other hand a separate thread will 
> schedule a ping-pong behaviour in the eviction handler on a full cache with 
> SolrCores still referenced. To overcome this we made the overflow behaviour 
> transparent by adding an additional overflowCores instance. Here we add 
> evicted but still referenced cores from the transientCores cache.
> Furthermore we need to ensure that only a single transientSolrCoreCache 
> inside TransientSolrCoreCacheFactoryDefault is created. As we now allow 
> multiple read threads, we call the the getTransientCacheHandler() method 
> initially holding a write lock inside the load() method. Calling the method 
> only needs a write lock initially (for cache creation). For all other calls, 
> a read lock is sufficient. By default, the getTransientCacheHandler() 
> acquires a read lock. If a write is needed (e.g. for core creation), the 
> getTransientCacheHandlerInternal() is called. This method explicitly does not 
> use a lock in order to provide the flexibility to choose between a read-lock 
> and a write-lock. This ensu

[jira] [Assigned] (SOLR-16497) Allow finer grained locking in SolrCores

[David Smiley (Jira)]

 [ 
https://issues.apache.org/jira/browse/SOLR-16497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Smiley reassigned SOLR-16497:
---

Assignee: David Smiley

> Allow finer grained locking in SolrCores
> 
>
> Key: SOLR-16497
> URL: https://issues.apache.org/jira/browse/SOLR-16497
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: search
>Affects Versions: 9.0, 8.11.2
>Reporter: Torsten Bøgh Köster
>Assignee: David Smiley
>Priority: Major
> Attachments: solrcores_locking.png, solrcores_locking_fixed.png
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Access to loaded SolrCore instances is a synchronized read and write 
> operation in SolrCores#getCoreFromAnyList. This method is touched by every 
> request as every HTTP request is assigned the SolrCore it operates on.
> h3. Background
> Under heavy load we discovered that application halts inside of Solr are 
> becoming a serious problem in high traffic environments. Using Java Flight 
> Recordings we discovered high accumulated applications halts on the 
> modifyLock in SolrCores. In our case this means that we can only utilize our 
> machines up to 25% cpu usage. With the fix applied, a utilization up to 80% 
> is perfectly doable.
> In our case this specific locking problem was masked by another locking 
> problem in the SlowCompositeReaderWrapper. We'll submit our fix to the 
> locking problem in the SlowCompositeReaderWrapper in a following issue.
> h3. Problem
> Our Solr instances utilizes the collapse component heavily. The instances run 
> with 32 cores and 32gb Java heap on a rather small index (4gb). The instances 
> scale out at 50% cpu load. We take Java Flight Recorder snapshots of 60 
> seconds
> as soon the cpu usage exceeds 50%.
>  !solrcores_locking.png|width=1024! 
> During our 60s Java Flight Recorder snapshot, the ~2k Jetty acceptor threads 
> accumulated more than 12h locking time inside SolrCores on the modifyLock 
> instance used as synchronized lock (see screenshot). With this fix the 
> locking access is reduced to write accesses only. We validated this using 
> another JFR snapshot:
>  !solrcores_locking_fixed.png|width=1024! 
> We ran this code for a couple of weeks in our live environment.
> h3. Solution
> The synchronized modifyLock is replaced by a ReentrantReadWriteLock. This 
> allows concurrent reads from the internal SolrCore instance list (cores) but 
> grants exclusive access to write operations on the instance list (cores).
> In Solr 9.x the cache inside the TransientSolrCoreCacheFactoryDefault adds a 
> cache overflow handling of the size based internal cache (SOLR-15964). As 
> soon as SolrCores are evicted from the internal cache, the cache behaviour 
> changes from a size based cache to a reference based cache via the cache's 
> eviction handler. SolrCore instances that are still referenced are inserted 
> back into the cache. This means that write operations to the cache (insert 
> SolrCore) can be issued during read operations in SolrCores. Hence these 
> operations have only a read lock which cannot be upgraded to a write lock 
> (dead lock).
> To overcome this, we moved the cache maintenance (including the eviction 
> handler) in TransientSolrCoreCacheFactoryDefault to a separate thread. This 
> thread can acquire a write lock but on the other hand a separate thread will 
> schedule a ping-pong behaviour in the eviction handler on a full cache with 
> SolrCores still referenced. To overcome this we made the overflow behaviour 
> transparent by adding an additional overflowCores instance. Here we add 
> evicted but still referenced cores from the transientCores cache.
> Furthermore we need to ensure that only a single transientSolrCoreCache 
> inside TransientSolrCoreCacheFactoryDefault is created. As we now allow 
> multiple read threads, we call the the getTransientCacheHandler() method 
> initially holding a write lock inside the load() method. Calling the method 
> only needs a write lock initially (for cache creation). For all other calls, 
> a read lock is sufficient. By default, the getTransientCacheHandler() 
> acquires a read lock. If a write is needed (e.g. for core creation), the 
> getTransientCacheHandlerInternal() is called. This method explicitly does not 
> use a lock in order to provide the flexibility to choose between a read-lock 
> and a write-lock. This ensures that a single instance of 
> transientSolrCoreCache is created.
> The lock signaling between SolrCore and CoreContainer gets replaced by a 
> Condition that is tied to the write lock.
> h3. Summary
> This change allows for a finer grained access to the list of open SolrCores. 
> The decreased blocking read access is noticeable in de

[jira] [Commented] (SOLR-16514) lucene query: only QueryParser run well

[Nicola Paganotti (Jira)]

[ 
https://issues.apache.org/jira/browse/SOLR-16514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627217#comment-17627217
 ] 

Nicola Paganotti commented on SOLR-16514:
-

I filled for Sole project because I'm looking for support about lucene ( solr 
Is based on lucene) and I haven't found a lucene project in Jira. I like jira 
software... Ok I accept his advise I will post in mailing list.

> lucene query: only QueryParser run well
> ---
>
> Key: SOLR-16514
> URL: https://issues.apache.org/jira/browse/SOLR-16514
> Project: Solr
>  Issue Type: Task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: clients - python
>Reporter: Nicola Paganotti
>Priority: Critical
>  Labels: newbie
>
> {code:java}
>     def searchGiustizia(self,startPagination,recPerPage):
>         indexPath = File(self.fileLucene).toPath()
>         directory = FSDirectory.open(indexPath)
>         searcher = IndexSearcher(DirectoryReader.open(directory))         
>  
> paQuery5 = TermQuery(Term("parte","TELESI RICCARDO"))
>         analyzer = StandardAnalyzer()        print 
> ("\n--")
>         start = datetime.now()
>         collector = TotalHitCountCollector()
>         searcher.search(paQuery5, collector)
>         print("found: ",collector.getTotalHits())        
>         duration = datetime.now() - start
>         print("time ",str(duration))
> print ("\n--") 
> {code}
> This function do a simple search in my Index of Apache Lucene. I use 
> TermQuery but it return no results. If I use QueryParser, Lucene find me 
> record with parte = "TELESI RICCARDO". Why doesn't TermQuery work?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[jira] [Resolved] (SOLR-16514) lucene query: only QueryParser run well

[David Smiley (Jira)]

 [ 
https://issues.apache.org/jira/browse/SOLR-16514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Smiley resolved SOLR-16514.
-
Resolution: Invalid

Hello.  You filed this issue for the Solr project but it appears it should have 
been for Lucene?  I have the ability to move/migrate the issue over there, but 
I think you should instead use a project mailing list to ask questions.  Both 
Lucene & Solr (and maybe all ASF projects) don't use Jira for questions.

> lucene query: only QueryParser run well
> ---
>
> Key: SOLR-16514
> URL: https://issues.apache.org/jira/browse/SOLR-16514
> Project: Solr
>  Issue Type: Task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: clients - python
>Reporter: Nicola Paganotti
>Priority: Critical
>  Labels: newbie
>
> {code:java}
>     def searchGiustizia(self,startPagination,recPerPage):
>         indexPath = File(self.fileLucene).toPath()
>         directory = FSDirectory.open(indexPath)
>         searcher = IndexSearcher(DirectoryReader.open(directory))         
>  
> paQuery5 = TermQuery(Term("parte","TELESI RICCARDO"))
>         analyzer = StandardAnalyzer()        print 
> ("\n--")
>         start = datetime.now()
>         collector = TotalHitCountCollector()
>         searcher.search(paQuery5, collector)
>         print("found: ",collector.getTotalHits())        
>         duration = datetime.now() - start
>         print("time ",str(duration))
> print ("\n--") 
> {code}
> This function do a simple search in my Index of Apache Lucene. I use 
> TermQuery but it return no results. If I use QueryParser, Lucene find me 
> record with parte = "TELESI RICCARDO". Why doesn't TermQuery work?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[jira] [Commented] (SOLR-15733) Separate out a solrj-streaming module

[David Smiley (Jira)]

[ 
https://issues.apache.org/jira/browse/SOLR-15733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627206#comment-17627206
 ] 

David Smiley commented on SOLR-15733:
-

BTW, next time remember to tidy up the commit message in the GitHub UI rather 
than accept the raw commit-by-commit internal work.

Out of curiosity, I measured the size of some SolrJ JAR files:
 * solr-solrj: 1.0M
 * solr-solrj-zookeeper: 210K
 * solr-solrj-streaming: 912K

And of course, there has been a drop in dependencies since 9.0.

> Separate out a solrj-streaming module
> -
>
> Key: SOLR-15733
> URL: https://issues.apache.org/jira/browse/SOLR-15733
> Project: Solr
>  Issue Type: Sub-task
>Reporter: Jan Høydahl
>Assignee: Joel Bernstein
>Priority: Major
>  Time Spent: 8h 40m
>  Remaining Estimate: 0h
>
> The huge amount of streaming expression code under 
> {{org.apache.solr.client.solrj.io}} package can be shipped as an optional 
> jar. The {{solrj-jdbc}} (or {{solrj-sql}}) modules would then have a 
> dependency on this...



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[GitHub] [solr-site] dsmiley commented on a diff in pull request #80: Warn about scans in security section.

[GitBox]

dsmiley commented on code in PR #80:
URL: https://github.com/apache/solr-site/pull/80#discussion_r1010529249


##
content/pages/security.md:
##
@@ -4,9 +4,21 @@ save_as: security.html
 template: security
 
 ## How to report a security issue
-If you believe you have discovered a vulnerability in Solr, you may first want 
to consult the [list of known false 
positives](https://cwiki.apache.org/confluence/display/SOLR/SolrSecurity#SolrSecurity-SolrandVulnerabilityScanningTools)
 to make sure you are reporting a real vulnerability.
-Then please disclose responsibly by following [these ASF 
guidelines](https://www.apache.org/security/) for reporting.
 
+### CVEs in Solr dependencies
+
+Solr depends on lots of other open-source software -- "dependencies".  If a 
CVE is published (a publicly identified vulnerability) against one of them, the 
Solr project will review it to see if it's actually exploitable in Solr -- 
usually they aren't.  Please review the [officially published non-exploitable 
vulnerabilities](https://cwiki.apache.org/confluence/display/SOLR/SolrSecurity#SolrSecurity-SolrandVulnerabilityScanningTools)
 before taking any steps.  If you **don't** see a CVE there, you should take 
the following steps:
+1. Search through the [Solr users mailing 
list](https://lists.apache.org/list.html?us...@solr.apache.org) to see if 
anyone else has brought up this dependency CVE.
+1. If no one has, then please do [subscribe to the users mailing 
list](https://solr.apache.org/community.html#mailing-lists-chat) and then send 
an email asking about the CVE.
+
+### Exploits found in Solr
+
+The Solr PMC will not accept the output of a vulnerability scan as a security 
report.

Review Comment:
   This line is only applicable to "CVEs in Solr dependencies".  You could 
begin that section with this sentence.



##
content/pages/security.md:
##
@@ -4,9 +4,21 @@ save_as: security.html
 template: security
 
 ## How to report a security issue
-If you believe you have discovered a vulnerability in Solr, you may first want 
to consult the [list of known false 
positives](https://cwiki.apache.org/confluence/display/SOLR/SolrSecurity#SolrSecurity-SolrandVulnerabilityScanningTools)
 to make sure you are reporting a real vulnerability.
-Then please disclose responsibly by following [these ASF 
guidelines](https://www.apache.org/security/) for reporting.
 
+### CVEs in Solr dependencies
+
+Solr depends on lots of other open-source software -- "dependencies".  If a 
CVE is published (a publicly identified vulnerability) against one of them, the 
Solr project will review it to see if it's actually exploitable in Solr -- 
usually they aren't.  Please review the [officially published non-exploitable 
vulnerabilities](https://cwiki.apache.org/confluence/display/SOLR/SolrSecurity#SolrSecurity-SolrandVulnerabilityScanningTools)
 before taking any steps.  If you **don't** see a CVE there, you should take 
the following steps:

Review Comment:
   do as you wish, but I prefer that when writing markdown/asciidoc, that 
sentences start on a new line.  Much of our Ref Guide thankfully was upgraded 
to this style by Cassandra.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[GitHub] [solr-site] dsmiley commented on a diff in pull request #80: Warn about scans in security section.

[GitBox]

dsmiley commented on code in PR #80:
URL: https://github.com/apache/solr-site/pull/80#discussion_r1010526385


##
content/pages/security.md:
##
@@ -7,6 +7,9 @@ template: security
 If you believe you have discovered a vulnerability in Solr, you may first want 
to consult the [list of known false 
positives](https://cwiki.apache.org/confluence/display/SOLR/SolrSecurity#SolrSecurity-SolrandVulnerabilityScanningTools)
 to make sure you are reporting a real vulnerability.
 Then please disclose responsibly by following [these ASF 
guidelines](https://www.apache.org/security/) for reporting.
 
+The Solr PMC will not accept the output of a vulnerability scan as a security 
report.
+Please do not email the security list with issues on Solr dependencies or 
outputs from vulnerability scanning tools.

Review Comment:
   Hopefully if the reporter believes it's exploitable, they still won't do 
that.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[GitHub] [solr-site] HoustonPutman commented on a diff in pull request #80: Warn about scans in security section.

[GitBox]

HoustonPutman commented on code in PR #80:
URL: https://github.com/apache/solr-site/pull/80#discussion_r1010523257


##
content/pages/security.md:
##
@@ -7,6 +7,9 @@ template: security
 If you believe you have discovered a vulnerability in Solr, you may first want 
to consult the [list of known false 
positives](https://cwiki.apache.org/confluence/display/SOLR/SolrSecurity#SolrSecurity-SolrandVulnerabilityScanningTools)
 to make sure you are reporting a real vulnerability.
 Then please disclose responsibly by following [these ASF 
guidelines](https://www.apache.org/security/) for reporting.
 
+The Solr PMC will not accept the output of a vulnerability scan as a security 
report.
+Please do not email the security list with issues on Solr dependencies or 
outputs from vulnerability scanning tools.

Review Comment:
   Kevin messaged me and had similar concerns. I am of Jan's opinion that if 
its public, its public and attackers will probably not be waiting to hear 
something about it on the Solr users mailing list. I could be wrong though.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[GitHub] [solr-site] HoustonPutman commented on pull request #80: Warn about scans in security section.

[GitBox]

HoustonPutman commented on PR #80:
URL: https://github.com/apache/solr-site/pull/80#issuecomment-1298618072

   Ok split up the dependency CVE and Solr exploits into separate sections. 
hopefully its more clear?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[GitHub] [solr-site] dsmiley commented on a diff in pull request #80: Warn about scans in security section.

[GitBox]

dsmiley commented on code in PR #80:
URL: https://github.com/apache/solr-site/pull/80#discussion_r1010507750


##
content/pages/security.md:
##
@@ -7,6 +7,9 @@ template: security
 If you believe you have discovered a vulnerability in Solr, you may first want 
to consult the [list of known false 
positives](https://cwiki.apache.org/confluence/display/SOLR/SolrSecurity#SolrSecurity-SolrandVulnerabilityScanningTools)
 to make sure you are reporting a real vulnerability.
 Then please disclose responsibly by following [these ASF 
guidelines](https://www.apache.org/security/) for reporting.
 
+The Solr PMC will not accept the output of a vulnerability scan as a security 
report.
+Please do not email the security list with issues on Solr dependencies or 
outputs from vulnerability scanning tools.

Review Comment:
   Shrug; but they may have found something exploitable.  It could be awkward 
to respond to this in a public list before we are ready to manage the timeline 
of the remediation.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[GitHub] [solr-site] dsmiley commented on a diff in pull request #80: Warn about scans in security section.

[GitBox]

dsmiley commented on code in PR #80:
URL: https://github.com/apache/solr-site/pull/80#discussion_r1010506370


##
content/pages/security.md:
##
@@ -7,6 +7,9 @@ template: security
 If you believe you have discovered a vulnerability in Solr, you may first want 
to consult the [list of known false 
positives](https://cwiki.apache.org/confluence/display/SOLR/SolrSecurity#SolrSecurity-SolrandVulnerabilityScanningTools)
 to make sure you are reporting a real vulnerability.

Review Comment:
   security@ is reasonable / prudent, I think.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[GitHub] [solr-site] HoustonPutman commented on a diff in pull request #80: Warn about scans in security section.

[GitBox]

HoustonPutman commented on code in PR #80:
URL: https://github.com/apache/solr-site/pull/80#discussion_r1010502343


##
content/pages/security.md:
##
@@ -7,6 +7,9 @@ template: security
 If you believe you have discovered a vulnerability in Solr, you may first want 
to consult the [list of known false 
positives](https://cwiki.apache.org/confluence/display/SOLR/SolrSecurity#SolrSecurity-SolrandVulnerabilityScanningTools)
 to make sure you are reporting a real vulnerability.

Review Comment:
   I like this, but I guess it begs the question where should they ask us. On 
the security@ or users@ list?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[GitHub] [solr-operator] janhoy commented on issue #484: Support Projected Volume in Solr Pod spec

[GitBox]

janhoy commented on issue #484:
URL: https://github.com/apache/solr-operator/issues/484#issuecomment-1298598915

   Turned out it is supported already, but with a slightly different yaml 
syntax:
   
   ```yaml
   solr:
 podOptions:
   volumes:
 # Map service account JWT token onto a volume
 - name: sa-token-vol
   defaultContainerMount:
 mountPath: /var/run/secrets/tokens
 name: sa-token-vol
   source:
 projected:
   sources:
 - serviceAccountToken:
 path: sa-token
 expirationSeconds: 86400
 audience: myAudience
   ```
   
   Closing


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[GitHub] [solr-operator] janhoy closed issue #484: Support Projected Volume in Solr Pod spec

[GitBox]

janhoy closed issue #484: Support Projected Volume in Solr Pod spec
URL: https://github.com/apache/solr-operator/issues/484


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[GitHub] [solr-operator] janhoy commented on pull request #485: Support for Projected Volume

[GitBox]

janhoy commented on PR #485:
URL: https://github.com/apache/solr-operator/pull/485#issuecomment-1298597724

   I tested the yaml above and it actually works - I now have the 
`/var/run/secrets/tokens/sa-token` file in my pod! Closing this. Perhaps I'll 
add another PR to document how to mount a projected volume.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[GitHub] [solr-operator] janhoy closed pull request #485: Support for Projected Volume

[GitBox]

janhoy closed pull request #485: Support for Projected Volume
URL: https://github.com/apache/solr-operator/pull/485


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[GitHub] [solr-operator] HoustonPutman closed issue #472: Helm zk.provided.zookeeperPodPolicy documentation issues

[GitBox]

HoustonPutman closed issue #472: Helm zk.provided.zookeeperPodPolicy 
documentation issues
URL: https://github.com/apache/solr-operator/issues/472


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[GitHub] [solr-operator] HoustonPutman commented on issue #472: Helm zk.provided.zookeeperPodPolicy documentation issues

[GitBox]

HoustonPutman commented on issue #472:
URL: https://github.com/apache/solr-operator/issues/472#issuecomment-1298594870

   Closed in https://github.com/apache/solr-operator/pull/482


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[GitHub] [solr-operator] HoustonPutman closed issue #479: Operator in CrashLoop with panic

[GitBox]

HoustonPutman closed issue #479: Operator in CrashLoop with panic
URL: https://github.com/apache/solr-operator/issues/479


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[GitHub] [solr-operator] HoustonPutman merged pull request #481: Fix bug with named PVCs

[GitBox]

HoustonPutman merged PR #481:
URL: https://github.com/apache/solr-operator/pull/481


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[GitHub] [solr-operator] HoustonPutman commented on pull request #485: Support for Projected Volume

[GitBox]

HoustonPutman commented on PR #485:
URL: https://github.com/apache/solr-operator/pull/485#issuecomment-1298587121

   But your PR is quite close, you've done a good job!
   
   You would have had two steps left:
   - Actually set the projected volume on the pods in the controller (you have 
just created a CRD entry and test, the test fails because you aren't actually 
using the information from the CRD to set the Pod).
 - 
https://github.com/apache/solr-operator/blob/main/controllers/util/solr_util.go#L237
 - 
https://github.com/apache/solr-operator/blob/main/controllers/util/prometheus_exporter_util.go#L158
   - Run `make prepare` to do the code/config generation and auto format 
everything.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[GitHub] [solr] tboeghk opened a new pull request, #1156: [SOLR-16515] Remove synchronized access to cachedOrdMaps in SlowCompositeReaderWrapper

[GitBox]

tboeghk opened a new pull request, #1156:
URL: https://github.com/apache/solr/pull/1156

   https://issues.apache.org/jira/browse/SOLR-16515
   
   The `SlowCompositeReaderWrapper` uses synchronized read and write access to 
its internal `cachedOrdMaps`. By using a `ConcurrentHashMap` instead of a 
`LinkedHashMap` as the  underlying `cachedOrdMaps` implementation and the 
`ConcurrentHashMap#computeIfAbsent` method to compute cache values, we were 
able to reduce locking contention significantly.
   
   ### Background
   
   Under heavy load we discovered that application halts inside of Solr are 
becoming a serious problem in high traffic environments. Using Java Flight 
Recordings we discovered high accumulated applications halts on the 
`cachedOrdMaps` in `SlowCompositeReaderWrapper`. Without this fix we were able 
to utilize our machines only up to 25% cpu usage. With the fix applied, a 
utilization up to 80% is perfectly doable.
   
   ### Description
   
   Our Solr instances utilizes the `collapse` component heavily. The instances 
run with 32 cores and 32gb Java heap on a rather small index (4gb). The 
instances scale out at 50% cpu load. We take Java Flight Recorder snapshots of 
60 seconds
   as soon the cpu usage exceeds 50%.
   
   https://user-images.githubusercontent.com/557264/196221280-7eeac492-d204-497a-907e-bc261cd90530.png";>
   
   During our 60s Java Flight Recorder snapshot, the ~2k Jetty threads 
accumulated more than 16h locking time inside the `SlowCompositeReaderWrapper` 
(see screenshot). With this fix applied, the locking access is reduced to cache 
write accesses only. We validated this using another JFR snapshot:
   
   https://user-images.githubusercontent.com/557264/196221323-3ab53d23-cf23-4aee-a395-b518d4b42e54.png";>
   
   ### Solution
   
   We propose the following improvement inside the `SlowCompositeReaderWrapper` 
removing blocking `synchronized` access to the internal `cachedOrdMaps`. The 
implementation keeps the semantics of the `getSortedDocValues` and 
`getSortedSetDocValues` methods but moves the expensive part of 
`OrdinalMap#build` into a producer. We use the producer to access the 
`ConcurrentHashMap` using the `ConcurrentHashMap#computeIfAbsent` method only.
   
   The current implementation uses the `synchronized` block not only to lock 
access to the `cachedOrdMaps` but also to protect the critical section between 
getting, building and putting the `OrdinalMap` into the cache. Inside the 
critical section the decision is formed, whether a cacheable value should be 
composed and added to the cache. 
   
   To support non-blocking read access to the cache, we move the building part 
of the critical section into a producer `Function`. The check whether we have a 
cacheable value is made upfront. To properly make that decision we had to take 
logic from `MultiDocValues#getSortedSetValues` and 
`MultiDocValues#getSortedValues` (the `SlowCompositeReaderWrapper` already 
contained duplicated code from those methods).
   
   ### Summary
   
   This change removes most blocking access inside the 
`SlowCompositeReaderWrapper` and despite it's name it's now capable of a much 
higher request throughput.
   
   This change has been composed together by Dennis Berger, Torsten Bøgh Köster 
and Marco Petris.
   
   # Checklist
   
   Please review the following and check all that apply:
   
   - [x] I have reviewed the guidelines for [How to 
Contribute](https://wiki.apache.org/solr/HowToContribute) and my code conforms 
to the standards described there to the best of my ability.
   - [x] I have created a Jira issue and added the issue ID to my pull request 
title.
   - [x] I have given Solr maintainers 
[access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork)
 to contribute to my PR branch. (optional but recommended)
   - [x] I have developed this patch against the `main` branch.
   - [x] I have run `./gradlew check`.
   - [x] I have added tests for my changes.
   - [ ] I have added documentation for the [Reference 
Guide](https://github.com/apache/solr/tree/main/solr/solr-ref-guide)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[GitHub] [solr-operator] HoustonPutman commented on pull request #485: Support for Projected Volume

[GitBox]

HoustonPutman commented on PR #485:
URL: https://github.com/apache/solr-operator/pull/485#issuecomment-1298581924

   > Digging some more, I see that in the spec, `corev1.ProjectedVolumeSource` 
is alredy a child of `corev1.VolumeSource`
   
   Yeah it looks like that is the case! It sounds like you had a use case for 
it, so do you want to give it a try and see how it works?
   
   Looks like its there in the documentation, so this isn't a thing in the 
v1.20 Kube APIs (the version of the code dependency that we are using) that was 
removed in a later version.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[jira] [Updated] (SOLR-16515) Remove synchronized access to cachedOrdMaps in SlowCompositeReaderWrapper

[Jira]

 [ 
https://issues.apache.org/jira/browse/SOLR-16515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Torsten Bøgh Köster updated SOLR-16515:
---
Description: 
The  {{SlowCompositeReaderWrapper}}  uses synchronized read and write access to 
its internal  {{cachedOrdMaps}} . By using a  {{ConcurrentHashMap}}  instead of 
a  {{LinkedHashMap}}  as the  underlying  {{cachedOrdMaps}}  implementation and 
the  {{ConcurrentHashMap#computeIfAbsent}}  method to compute cache values, we 
were able to reduce locking contention significantly.

h3. Background

Under heavy load we discovered that application halts inside of Solr are 
becoming a serious problem in high traffic environments. Using Java Flight 
Recordings we discovered high accumulated applications halts on the  
{{cachedOrdMaps}}  in  {{SlowCompositeReaderWrapper}} . Without this fix we 
were able to utilize our machines only up to 25% cpu usage. With the fix 
applied, a utilization up to 80% is perfectly doable.

h3. Description

Our Solr instances utilizes the  {{collapse}}  component heavily. The instances 
run with 32 cores and 32gb Java heap on a rather small index (4gb). The 
instances scale out at 50% cpu load. We take Java Flight Recorder snapshots of 
60 seconds
as soon the cpu usage exceeds 50%.

 !slow-composite-reader-wrapper-before.jpg|width=1024! 

During our 60s Java Flight Recorder snapshot, the ~2k Jetty threads accumulated 
more than 16h locking time inside the  {{SlowCompositeReaderWrapper}}  (see 
screenshot). With this fix applied, the locking access is reduced to cache 
write accesses only. We validated this using another JFR snapshot:

 !slow-composite-reader-wrapper-after.jpg|width=1024! 

h3. Solution

We propose the following improvement inside the  {{SlowCompositeReaderWrapper}} 
 removing blocking  {{synchronized}}  access to the internal  {{cachedOrdMaps}} 
. The implementation keeps the semantics of the  {{getSortedDocValues}}  and  
{{getSortedSetDocValues}}  methods but moves the expensive part of  
{{OrdinalMap#build}}  into a producer. We use the producer to access the  
{{ConcurrentHashMap}}  using the  {{ConcurrentHashMap#computeIfAbsent}}  method 
only.
The current implementation uses the  {{synchronized}}  block not only to lock 
access to the  {{cachedOrdMaps}}  but also to protect the critical section 
between getting, building and putting the  {{OrdinalMap}}  into the cache. 
Inside the critical section the decision is formed, whether a cacheable value 
should be composed and added to the cache. 
To support non-blocking read access to the cache, we move the building part of 
the critical section into a producer  {{Function}} . The check whether we have 
a cacheable value is made upfront. To properly make that decision we had to 
take logic from  {{MultiDocValues#getSortedSetValues}}  and  
{{MultiDocValues#getSortedValues}}  (the  {{SlowCompositeReaderWrapper}}  
already contained duplicated code from those methods).

h3. Summary

This change removes most blocking access inside the  
{{SlowCompositeReaderWrapper}}  and despite it's name it's now capable 
of a much higher request throughput.
This change has been composed together by Dennis Berger, Torsten Bøgh Köster 
and Marco Petris.

  was:
The  {{SlowCompositeReaderWrapper}}  uses synchronized read and write access to 
its internal  {{cachedOrdMaps}} . By using a  {{ConcurrentHashMap}}  instead of 
a  {{LinkedHashMap}}  as the  underlying  {{cachedOrdMaps}}  implementation and 
the  {{ConcurrentHashMap#computeIfAbsent}}  method to compute cache values, we 
were able to reduce locking contention significantly.

h3. Background

Under heavy load we discovered that application halts inside of Solr are 
becoming a serious problem in high traffic environments. Using Java Flight 
Recordings we discovered high accumulated applications halts on the  
{{cachedOrdMaps}}  in  {{SlowCompositeReaderWrapper}} . Without this fix we 
were able to utilize our machines only up to 25% cpu usage. With the fix 
applied, a utilization up to 80% is perfectly doable.

h3. Description

Our Solr instances utilizes the  {{collapse}}  component heavily. The instances 
run with 32 cores and 32gb Java heap on a rather small index (4gb). The 
instances scale out at 50% cpu load. We take Java Flight Recorder snapshots of 
60 seconds
as soon the cpu usage exceeds 50%.

 !slow-composite-reader-wrapper-before.jpg|height=1024px! 

During our 60s Java Flight Recorder snapshot, the ~2k Jetty threads accumulated 
more than 16h locking time inside the  {{SlowCompositeReaderWrapper}}  (see 
screenshot). With this fix applied, the locking access is reduced to cache 
write accesses only. We validated this using another JFR snapshot:

 !slow-composite-reader-wrapper-after.jpg|height=1024px! 

h3. Solution

We propose the following improvement inside the  {{SlowCompositeReaderWrapper}} 
 removing blocking  {{synchronized}}  access to the int

[jira] [Updated] (SOLR-16497) Allow finer grained locking in SolrCores

[Jira]

 [ 
https://issues.apache.org/jira/browse/SOLR-16497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Torsten Bøgh Köster updated SOLR-16497:
---
Description: 
Access to loaded SolrCore instances is a synchronized read and write operation 
in SolrCores#getCoreFromAnyList. This method is touched by every request as 
every HTTP request is assigned the SolrCore it operates on.

h3. Background

Under heavy load we discovered that application halts inside of Solr are 
becoming a serious problem in high traffic environments. Using Java Flight 
Recordings we discovered high accumulated applications halts on the modifyLock 
in SolrCores. In our case this means that we can only utilize our machines up 
to 25% cpu usage. With the fix applied, a utilization up to 80% is perfectly 
doable.

In our case this specific locking problem was masked by another locking problem 
in the SlowCompositeReaderWrapper. We'll submit our fix to the locking problem 
in the SlowCompositeReaderWrapper in a following issue.

h3. Problem

Our Solr instances utilizes the collapse component heavily. The instances run 
with 32 cores and 32gb Java heap on a rather small index (4gb). The instances 
scale out at 50% cpu load. We take Java Flight Recorder snapshots of 60 seconds
as soon the cpu usage exceeds 50%.

 !solrcores_locking.png|width=1024! 

During our 60s Java Flight Recorder snapshot, the ~2k Jetty acceptor threads 
accumulated more than 12h locking time inside SolrCores on the modifyLock 
instance used as synchronized lock (see screenshot). With this fix the locking 
access is reduced to write accesses only. We validated this using another JFR 
snapshot:

 !solrcores_locking_fixed.png|width=1024! 

We ran this code for a couple of weeks in our live environment.

h3. Solution

The synchronized modifyLock is replaced by a ReentrantReadWriteLock. This 
allows concurrent reads from the internal SolrCore instance list (cores) but 
grants exclusive access to write operations on the instance list (cores).

In Solr 9.x the cache inside the TransientSolrCoreCacheFactoryDefault adds a 
cache overflow handling of the size based internal cache (SOLR-15964). As soon 
as SolrCores are evicted from the internal cache, the cache behaviour changes 
from a size based cache to a reference based cache via the cache's eviction 
handler. SolrCore instances that are still referenced are inserted back into 
the cache. This means that write operations to the cache (insert SolrCore) can 
be issued during read operations in SolrCores. Hence these operations have only 
a read lock which cannot be upgraded to a write lock (dead lock).

To overcome this, we moved the cache maintenance (including the eviction 
handler) in TransientSolrCoreCacheFactoryDefault to a separate thread. This 
thread can acquire a write lock but on the other hand a separate thread will 
schedule a ping-pong behaviour in the eviction handler on a full cache with 
SolrCores still referenced. To overcome this we made the overflow behaviour 
transparent by adding an additional overflowCores instance. Here we add evicted 
but still referenced cores from the transientCores cache.

Furthermore we need to ensure that only a single transientSolrCoreCache inside 
TransientSolrCoreCacheFactoryDefault is created. As we now allow multiple read 
threads, we call the the getTransientCacheHandler() method initially holding a 
write lock inside the load() method. Calling the method only needs a write lock 
initially (for cache creation). For all other calls, a read lock is sufficient. 
By default, the getTransientCacheHandler() acquires a read lock. If a write is 
needed (e.g. for core creation), the getTransientCacheHandlerInternal() is 
called. This method explicitly does not use a lock in order to provide the 
flexibility to choose between a read-lock and a write-lock. This ensures that a 
single instance of transientSolrCoreCache is created.

The lock signaling between SolrCore and CoreContainer gets replaced by a 
Condition that is tied to the write lock.

h3. Summary

This change allows for a finer grained access to the list of open SolrCores. 
The decreased blocking read access is noticeable in decreased blocking times of 
the Solr application (see screenshot).

This change has been composed together by Dennis Berger, Torsten Bøgh Köster 
and Marco Petris.

  was:
Access to loaded SolrCore instances is a synchronized read and write operation 
in SolrCores#getCoreFromAnyList. This method is touched by every request as 
every HTTP request is assigned the SolrCore it operates on.

h3. Background

Under heavy load we discovered that application halts inside of Solr are 
becoming a serious problem in high traffic environments. Using Java Flight 
Recordings we discovered high accumulated applications halts on the modifyLock 
in SolrCores. In our case this means that we can only utilize our machines up 
to 25% cpu usage. With the fix appli

[jira] [Updated] (SOLR-16515) Remove synchronized access to cachedOrdMaps in SlowCompositeReaderWrapper

[Jira]

 [ 
https://issues.apache.org/jira/browse/SOLR-16515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Torsten Bøgh Köster updated SOLR-16515:
---
Description: 
The  {{SlowCompositeReaderWrapper}}  uses synchronized read and write access to 
its internal  {{cachedOrdMaps}} . By using a  {{ConcurrentHashMap}}  instead of 
a  {{LinkedHashMap}}  as the  underlying  {{cachedOrdMaps}}  implementation and 
the  {{ConcurrentHashMap#computeIfAbsent}}  method to compute cache values, we 
were able to reduce locking contention significantly.

h3. Background

Under heavy load we discovered that application halts inside of Solr are 
becoming a serious problem in high traffic environments. Using Java Flight 
Recordings we discovered high accumulated applications halts on the  
{{cachedOrdMaps}}  in  {{SlowCompositeReaderWrapper}} . Without this fix we 
were able to utilize our machines only up to 25% cpu usage. With the fix 
applied, a utilization up to 80% is perfectly doable.

h3. Description

Our Solr instances utilizes the  {{collapse}}  component heavily. The instances 
run with 32 cores and 32gb Java heap on a rather small index (4gb). The 
instances scale out at 50% cpu load. We take Java Flight Recorder snapshots of 
60 seconds
as soon the cpu usage exceeds 50%.

 !slow-composite-reader-wrapper-before.jpg|height=1024px! 

During our 60s Java Flight Recorder snapshot, the ~2k Jetty threads accumulated 
more than 16h locking time inside the  {{SlowCompositeReaderWrapper}}  (see 
screenshot). With this fix applied, the locking access is reduced to cache 
write accesses only. We validated this using another JFR snapshot:

 !slow-composite-reader-wrapper-after.jpg|height=1024px! 

h3. Solution

We propose the following improvement inside the  {{SlowCompositeReaderWrapper}} 
 removing blocking  {{synchronized}}  access to the internal  {{cachedOrdMaps}} 
. The implementation keeps the semantics of the  {{getSortedDocValues}}  and  
{{getSortedSetDocValues}}  methods but moves the expensive part of  
{{OrdinalMap#build}}  into a producer. We use the producer to access the  
{{ConcurrentHashMap}}  using the  {{ConcurrentHashMap#computeIfAbsent}}  method 
only.
The current implementation uses the  {{synchronized}}  block not only to lock 
access to the  {{cachedOrdMaps}}  but also to protect the critical section 
between getting, building and putting the  {{OrdinalMap}}  into the cache. 
Inside the critical section the decision is formed, whether a cacheable value 
should be composed and added to the cache. 
To support non-blocking read access to the cache, we move the building part of 
the critical section into a producer  {{Function}} . The check whether we have 
a cacheable value is made upfront. To properly make that decision we had to 
take logic from  {{MultiDocValues#getSortedSetValues}}  and  
{{MultiDocValues#getSortedValues}}  (the  {{SlowCompositeReaderWrapper}}  
already contained duplicated code from those methods).

h3. Summary

This change removes most blocking access inside the  
{{SlowCompositeReaderWrapper}}  and despite it's name it's now capable 
of a much higher request throughput.
This change has been composed together by Dennis Berger, Torsten Bøgh Köster 
and Marco Petris.

  was:
The  {{SlowCompositeReaderWrapper}}  uses synchronized read and write access to 
its internal  {{cachedOrdMaps}} . By using a  {{ConcurrentHashMap}}  instead of 
a  {{LinkedHashMap}}  as the  underlying  {{cachedOrdMaps}}  implementation and 
the  {{ConcurrentHashMap#computeIfAbsent}}  method to compute cache values, we 
were able to reduce locking contention significantly.

h3. Background

Under heavy load we discovered that application halts inside of Solr are 
becoming a serious problem in high traffic environments. Using Java Flight 
Recordings we discovered high accumulated applications halts on the  
{{cachedOrdMaps}}  in  {{SlowCompositeReaderWrapper}} . Without this fix we 
were able to utilize our machines only up to 25% cpu usage. With the fix 
applied, a utilization up to 80% is perfectly doable.

h3. Description

Our Solr instances utilizes the  {{collapse}}  component heavily. The instances 
run with 32 cores and 32gb Java heap on a rather small index (4gb). The 
instances scale out at 50% cpu load. We take Java Flight Recorder snapshots of 
60 seconds
as soon the cpu usage exceeds 50%.

 !slow-composite-reader-wrapper-before.jpg! 

During our 60s Java Flight Recorder snapshot, the ~2k Jetty threads accumulated 
more than 16h locking time inside the  {{SlowCompositeReaderWrapper}}  (see 
screenshot). With this fix applied, the locking access is reduced to cache 
write accesses only. We validated this using another JFR snapshot:

 !slow-composite-reader-wrapper-after.jpg! 

h3. Solution

We propose the following improvement inside the  {{SlowCompositeReaderWrapper}} 
 removing blocking  {{synchronized}}  access to the internal  {{cachedOrdMaps

[jira] [Created] (SOLR-16515) Remove synchronized access to cachedOrdMaps in SlowCompositeReaderWrapper

[Jira]

Torsten Bøgh Köster created SOLR-16515:
--

 Summary: Remove synchronized access to cachedOrdMaps in 
SlowCompositeReaderWrapper
 Key: SOLR-16515
 URL: https://issues.apache.org/jira/browse/SOLR-16515
 Project: Solr
  Issue Type: Improvement
  Security Level: Public (Default Security Level. Issues are Public)
  Components: search
Affects Versions: 8.11.2, 9.0
Reporter: Torsten Bøgh Köster
 Attachments: slow-composite-reader-wrapper-after.jpg, 
slow-composite-reader-wrapper-before.jpg

The  {{SlowCompositeReaderWrapper}}  uses synchronized read and write access to 
its internal  {{cachedOrdMaps}} . By using a  {{ConcurrentHashMap}}  instead of 
a  {{LinkedHashMap}}  as the  underlying  {{cachedOrdMaps}}  implementation and 
the  {{ConcurrentHashMap#computeIfAbsent}}  method to compute cache values, we 
were able to reduce locking contention significantly.

h3. Background

Under heavy load we discovered that application halts inside of Solr are 
becoming a serious problem in high traffic environments. Using Java Flight 
Recordings we discovered high accumulated applications halts on the  
{{cachedOrdMaps}}  in  {{SlowCompositeReaderWrapper}} . Without this fix we 
were able to utilize our machines only up to 25% cpu usage. With the fix 
applied, a utilization up to 80% is perfectly doable.

h3. Description

Our Solr instances utilizes the  {{collapse}}  component heavily. The instances 
run with 32 cores and 32gb Java heap on a rather small index (4gb). The 
instances scale out at 50% cpu load. We take Java Flight Recorder snapshots of 
60 seconds
as soon the cpu usage exceeds 50%.

 !slow-composite-reader-wrapper-before.jpg! 

During our 60s Java Flight Recorder snapshot, the ~2k Jetty threads accumulated 
more than 16h locking time inside the  {{SlowCompositeReaderWrapper}}  (see 
screenshot). With this fix applied, the locking access is reduced to cache 
write accesses only. We validated this using another JFR snapshot:

 !slow-composite-reader-wrapper-after.jpg! 

h3. Solution

We propose the following improvement inside the  {{SlowCompositeReaderWrapper}} 
 removing blocking  {{synchronized}}  access to the internal  {{cachedOrdMaps}} 
. The implementation keeps the semantics of the  {{getSortedDocValues}}  and  
{{getSortedSetDocValues}}  methods but moves the expensive part of  
{{OrdinalMap#build}}  into a producer. We use the producer to access the  
{{ConcurrentHashMap}}  using the  {{ConcurrentHashMap#computeIfAbsent}}  method 
only.
The current implementation uses the  {{synchronized}}  block not only to lock 
access to the  {{cachedOrdMaps}}  but also to protect the critical section 
between getting, building and putting the  {{OrdinalMap}}  into the cache. 
Inside the critical section the decision is formed, whether a cacheable value 
should be composed and added to the cache. 
To support non-blocking read access to the cache, we move the building part of 
the critical section into a producer  {{Function}} . The check whether we have 
a cacheable value is made upfront. To properly make that decision we had to 
take logic from  {{MultiDocValues#getSortedSetValues}}  and  
{{MultiDocValues#getSortedValues}}  (the  {{SlowCompositeReaderWrapper}}  
already contained duplicated code from those methods).

h3. Summary

This change removes most blocking access inside the  
{{SlowCompositeReaderWrapper}}  and despite it's name it's now capable 
of a much higher request throughput.
This change has been composed together by Dennis Berger, Torsten Bøgh Köster 
and Marco Petris.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[GitHub] [solr-operator] janhoy commented on pull request #485: Support for Projected Volume

[GitBox]

janhoy commented on PR #485:
URL: https://github.com/apache/solr-operator/pull/485#issuecomment-1298538512

   Digging some more, I see that in the spec, `corev1.ProjectedVolumeSource` is 
alredy a child of `corev1.VolumeSource`, so perhaps this already is supported 
by nesting `projected` inside `source` instead of the shortcut form documented 
in 
https://kubernetes.io/docs/concepts/storage/projected-volumes/#example-configuration-secret-downwardapi-configmap?
   
   ```yaml
   solr:
 podOptions:
   volumes:
 # Map service account JWT token onto a volume
 - name: sa-token-vol
   defaultContainerMount:
 mountPath: /var/run/secrets/tokens
 name: sa-token-vol
   source:
 projected:
   sources:
 - serviceAccountToken:
 path: sa-token
 expirationSeconds: 86400
 audience: myAudience
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[GitHub] [solr-operator] janhoy commented on pull request #485: Support for Projected Volume

[GitBox]

janhoy commented on PR #485:
URL: https://github.com/apache/solr-operator/pull/485#issuecomment-1298500481

   The test output is
   
   ```
   • Failure [1.304 seconds]
   SolrCloud controller - General
   /Users/janhoy/git/solr-operator/controllers/solrcloud_controller_test.go:35
 Solr Cloud with Projected Volume
 
/Users/janhoy/git/solr-operator/controllers/solrcloud_controller_test.go:707
   has the correct resources [It]
   
/Users/janhoy/git/solr-operator/controllers/solrcloud_controller_test.go:725
   
   Additional Volume from podOptions not loaded into pod properly.
   Expected
   <*v1.ProjectedVolumeSource | 0x0>: nil
   to equal
   <*v1.ProjectedVolumeSource | 0x106e1c780>: {
   Sources: [
   {
   Secret: nil,
   DownwardAPI: nil,
   ConfigMap: nil,
   ServiceAccountToken: {
   Audience: "My.Audience",
   ExpirationSeconds: 300,
   Path: "token",
   },
   },
   ],
   DefaultMode: nil,
   }
   
   
/Users/janhoy/git/solr-operator/controllers/solrcloud_controller_test.go:736
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[GitHub] [solr-operator] janhoy opened a new pull request, #485: Draft: Support for Projected Volume

[GitBox]

janhoy opened a new pull request, #485:
URL: https://github.com/apache/solr-operator/pull/485

   This is my first attempt at a contribution to the operator, implementing 
#484. But I cannot get the debugger working in IDEA so I'm not sure where it 
goes wrong.
   
   Fixes #484 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[jira] [Commented] (SOLR-15733) Separate out a solrj-streaming module

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/SOLR-15733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627157#comment-17627157
 ] 

ASF subversion and git services commented on SOLR-15733:


Commit db28836ac46372e353e087e171fa0b25875f2320 in solr's branch 
refs/heads/main from Joel Bernstein
[ https://gitbox.apache.org/repos/asf?p=solr.git;h=db28836ac46 ]

SOLR-15733: Separate out a solrj-streaming module (#1099)

* SOLR-15733: Initial commit WIP

* SOLR-15733: Move solrj.client.io

* SOLR-15733: Fix directories

* SOLR-15733: Move tests

* SOLR-15733: Remove commons math from main solrj

* SOLR-15733: Wire-in solrj-streaming

* SOLR-15733: Get solrj-streaming test classes to compile

* SOLR-15733: Get solrj-streaming tests passing

* Fix dependency declarations

* Fix ref guide to DaemonStream reference

* Add solrj refguide note

* Remove hsqldb test dependency from solr/solrj

* Cleanup solrj sql reference

Co-authored-by: Kevin Risden 

> Separate out a solrj-streaming module
> -
>
> Key: SOLR-15733
> URL: https://issues.apache.org/jira/browse/SOLR-15733
> Project: Solr
>  Issue Type: Sub-task
>Reporter: Jan Høydahl
>Assignee: Joel Bernstein
>Priority: Major
>  Time Spent: 8h 40m
>  Remaining Estimate: 0h
>
> The huge amount of streaming expression code under 
> {{org.apache.solr.client.solrj.io}} package can be shipped as an optional 
> jar. The {{solrj-jdbc}} (or {{solrj-sql}}) modules would then have a 
> dependency on this...



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[GitHub] [solr] joel-bernstein merged pull request #1099: SOLR-15733: Separate out a solrj-streaming module

[GitBox]

joel-bernstein merged PR #1099:
URL: https://github.com/apache/solr/pull/1099


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[GitHub] [solr] sonatype-lift[bot] commented on a diff in pull request #1144: SOLR-16488: Create a v2 equivalent for /admin/zookeeper v1 APIs

[GitBox]

sonatype-lift[bot] commented on code in PR #1144:
URL: https://github.com/apache/solr/pull/1144#discussion_r1010410293


##
solr/core/src/java/org/apache/solr/handler/admin/api/ZookeeperAPI.java:
##
@@ -0,0 +1,280 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.handler.admin.api;
+
+import static 
org.apache.solr.client.solrj.impl.BinaryResponseParser.BINARY_CONTENT_TYPE_V2;
+import static org.apache.solr.common.params.CommonParams.OMIT_HEADER;
+import static org.apache.solr.common.params.CommonParams.PATH;
+import static org.apache.solr.common.params.CommonParams.WT;
+import static 
org.apache.solr.security.PermissionNameProvider.Name.ZK_READ_PERM;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import io.swagger.v3.oas.annotations.Operation;
+import java.lang.invoke.MethodHandles;
+import java.util.HashMap;
+import java.util.Locale;
+import java.util.Map;
+import javax.inject.Inject;
+import javax.ws.rs.GET;
+import javax.ws.rs.Path;
+import javax.ws.rs.Produces;
+import org.apache.solr.api.JerseyResource;
+import org.apache.solr.cloud.ZkController;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.SolrException.ErrorCode;
+import org.apache.solr.common.cloud.SolrZkClient;
+import org.apache.solr.common.cloud.ZkDynamicConfig;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.core.CoreContainer;
+import org.apache.solr.handler.admin.ZookeeperInfoHandler.FilterType;
+import org.apache.solr.handler.admin.ZookeeperInfoHandler.PageOfCollections;
+import 
org.apache.solr.handler.admin.ZookeeperInfoHandler.PagedCollectionSupport;
+import org.apache.solr.handler.admin.ZookeeperInfoHandler.ZKPrinter;
+import org.apache.solr.handler.admin.ZookeeperStatusHandler;
+import org.apache.solr.jersey.PermissionName;
+import org.apache.solr.jersey.SolrJerseyResponse;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.zookeeper.KeeperException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Path("/cluster/zookeeper/")
+public class ZookeeperAPI extends JerseyResource {
+
+  private static final Logger log = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+  private final CoreContainer coreContainer;
+  private final SolrQueryRequest solrQueryRequest;
+  private final SolrQueryResponse solrQueryResponse;
+  private PagedCollectionSupport pagingSupport;
+
+  @Inject
+  public ZookeeperAPI(
+  CoreContainer coreContainer,
+  SolrQueryRequest solrQueryRequest,
+  SolrQueryResponse solrQueryResponse) {
+this.coreContainer = coreContainer;
+this.solrQueryRequest = solrQueryRequest;
+this.solrQueryResponse = solrQueryResponse;
+  }
+
+  @GET
+  @Path("/files")
+  @Produces({"application/json", "application/xml", BINARY_CONTENT_TYPE_V2})
+  @Operation(
+  summary = "List Zookeeper files.",
+  tags = {"zookeeperFiles"})
+  @PermissionName(ZK_READ_PERM)
+  public ZookeeperFilesResponse getFiles() throws Exception {
+final ZookeeperFilesResponse response = 
instantiateJerseyResponse(ZookeeperFilesResponse.class);
+final SolrParams params = solrQueryRequest.getParams();
+Map map = new HashMap<>(1);

Review Comment:
   
*[ModifiedButNotUsed](https://errorprone.info/bugpattern/ModifiedButNotUsed):*  
A collection or proto builder was created, but its values were never accessed.
   
   ---
   
   ℹ️ Learn about @sonatype-lift commands
   
   You can reply with the following commands. For example, reply with 
***@sonatype-lift ignoreall*** to leave out all findings.
   | **Command** | **Usage** |
   | - | - |
   | `@sonatype-lift ignore` | Leave out the above finding from this PR |
   | `@sonatype-lift ignoreall` | Leave out all the existing findings from this 
PR |
   | `@sonatype-lift exclude ` | Exclude specified 
`file\|issue\|path\|tool` from Lift findings by updating your config.toml file |
   
   **Note:** When talking to LiftBot, you need to **refresh** the page to see 
its response.
   [Click here](https://github.com/apps/sonatype-lift/installations/new) 
to add LiftBot to another repo.
   
   
   
  

[GitHub] [solr-operator] janhoy commented on issue #484: Support Projected Volume in Solr Pod spec

[GitBox]

janhoy commented on issue #484:
URL: https://github.com/apache/solr-operator/issues/484#issuecomment-1298419826

   I'm not into the codebase or Go in general, but would like to attempt a PR 
if I get some assistance. I'll post a PR with my WIP progress soon.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[GitHub] [solr] tboeghk commented on pull request #1118: [SOLR-16489] CaffeineCache puts thread into infinite loop

[GitBox]

tboeghk commented on PR #1118:
URL: https://github.com/apache/solr/pull/1118#issuecomment-1298315534

   @ben-manes we can confirm that your change with `3.1.2-SNAPSHOT` is working! 
   
   We removed our fix from the SolrCache and can confirm that no threads inside 
Solr are spinning anymore. We see the following exceptions in the Solr log (I 
had to remove some key details as they are sensitive):
   
   ```
   java.lang.IllegalStateException: An invalid state was detected that occurs 
if the key's equals or hashCode was modified while it resided in the map. This 
violation of the Map contract can lead to non-deterministic behavior (key: 
((indexed_dimensions:dame^5000.0 | MatchNoDocsQuery("") | MatchNoDocsQuery("") 
| selling_points:dame^1000.0 | MatchNoDocsQuery("") | MatchNoDocsQuery("") | 
indexed_filter_values:dame^1.0 | target_group:dame^5.0 | [...] 
text:trigema^1.0)~0.2)~3).
   at 
com.github.benmanes.caffeine.cache.Caffeine.requireState(Caffeine.java:204)
   at 
com.github.benmanes.caffeine.cache.BoundedLocalCache.requireIsAlive(BoundedLocalCache.java:288)
   at 
com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$put$8(BoundedLocalCache.java:2293)
   at 
java.base/java.util.concurrent.ConcurrentHashMap.computeIfPresent(ConcurrentHashMap.java:1852)
   at 
com.github.benmanes.caffeine.cache.BoundedLocalCache.put(BoundedLocalCache.java:2292)
   at 
com.github.benmanes.caffeine.cache.BoundedLocalCache.put(BoundedLocalCache.java:2212)
   at 
com.github.benmanes.caffeine.cache.LocalAsyncCache$AsMapView.put(LocalAsyncCache.java:689)
   at org.apache.solr.search.CaffeineCache.put(CaffeineCache.java:267)
   at 
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1437)
   at 
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:596)
   [...]
   ```
   
   We'll check what exact `Query` implementation is inserted into the cache.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[jira] [Commented] (SOLR-16497) Allow finer grained locking in SolrCores

[Jira]

[ 
https://issues.apache.org/jira/browse/SOLR-16497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627057#comment-17627057
 ] 

Torsten Bøgh Köster commented on SOLR-16497:


GitHub PR is finally done. Adoption to Solr 9.x took some time but tests are 
finally green :-)

> Allow finer grained locking in SolrCores
> 
>
> Key: SOLR-16497
> URL: https://issues.apache.org/jira/browse/SOLR-16497
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: search
>Affects Versions: 9.0, 8.11.2
>Reporter: Torsten Bøgh Köster
>Priority: Major
> Attachments: solrcores_locking.png, solrcores_locking_fixed.png
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Access to loaded SolrCore instances is a synchronized read and write 
> operation in SolrCores#getCoreFromAnyList. This method is touched by every 
> request as every HTTP request is assigned the SolrCore it operates on.
> h3. Background
> Under heavy load we discovered that application halts inside of Solr are 
> becoming a serious problem in high traffic environments. Using Java Flight 
> Recordings we discovered high accumulated applications halts on the 
> modifyLock in SolrCores. In our case this means that we can only utilize our 
> machines up to 25% cpu usage. With the fix applied, a utilization up to 80% 
> is perfectly doable.
> In our case this specific locking problem was masked by another locking 
> problem in the SlowCompositeReaderWrapper. We'll submit our fix to the 
> locking problem in the SlowCompositeReaderWrapper in a following issue.
> h3. Problem
> Our Solr instances utilizes the collapse component heavily. The instances run 
> with 32 cores and 32gb Java heap on a rather small index (4gb). The instances 
> scale out at 50% cpu load. We take Java Flight Recorder snapshots of 60 
> seconds
> as soon the cpu usage exceeds 50%.
>  !solrcores_locking.png|height=1024px! 
> During our 60s Java Flight Recorder snapshot, the ~2k Jetty acceptor threads 
> accumulated more than 12h locking time inside SolrCores on the modifyLock 
> instance used as synchronized lock (see screenshot). With this fix the 
> locking access is reduced to write accesses only. We validated this using 
> another JFR snapshot:
>  !solrcores_locking_fixed.png|height=1024px! 
> We ran this code for a couple of weeks in our live environment.
> h3. Solution
> The synchronized modifyLock is replaced by a ReentrantReadWriteLock. This 
> allows concurrent reads from the internal SolrCore instance list (cores) but 
> grants exclusive access to write operations on the instance list (cores).
> In Solr 9.x the cache inside the TransientSolrCoreCacheFactoryDefault adds a 
> cache overflow handling of the size based internal cache (SOLR-15964). As 
> soon as SolrCores are evicted from the internal cache, the cache behaviour 
> changes from a size based cache to a reference based cache via the cache's 
> eviction handler. SolrCore instances that are still referenced are inserted 
> back into the cache. This means that write operations to the cache (insert 
> SolrCore) can be issued during read operations in SolrCores. Hence these 
> operations have only a read lock which cannot be upgraded to a write lock 
> (dead lock).
> To overcome this, we moved the cache maintenance (including the eviction 
> handler) in TransientSolrCoreCacheFactoryDefault to a separate thread. This 
> thread can acquire a write lock but on the other hand a separate thread will 
> schedule a ping-pong behaviour in the eviction handler on a full cache with 
> SolrCores still referenced. To overcome this we made the overflow behaviour 
> transparent by adding an additional overflowCores instance. Here we add 
> evicted but still referenced cores from the transientCores cache.
> Furthermore we need to ensure that only a single transientSolrCoreCache 
> inside TransientSolrCoreCacheFactoryDefault is created. As we now allow 
> multiple read threads, we call the the getTransientCacheHandler() method 
> initially holding a write lock inside the load() method. Calling the method 
> only needs a write lock initially (for cache creation). For all other calls, 
> a read lock is sufficient. By default, the getTransientCacheHandler() 
> acquires a read lock. If a write is needed (e.g. for core creation), the 
> getTransientCacheHandlerInternal() is called. This method explicitly does not 
> use a lock in order to provide the flexibility to choose between a read-lock 
> and a write-lock. This ensures that a single instance of 
> transientSolrCoreCache is created.
> The lock signaling between SolrCore and CoreContainer gets replaced by a 
> Condition that is tied to the write lock.
> h3. Summary
> This change allows for a finer

[jira] [Updated] (SOLR-16497) Allow finer grained locking in SolrCores

[Jira]

 [ 
https://issues.apache.org/jira/browse/SOLR-16497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Torsten Bøgh Köster updated SOLR-16497:
---
Description: 
Access to loaded SolrCore instances is a synchronized read and write operation 
in SolrCores#getCoreFromAnyList. This method is touched by every request as 
every HTTP request is assigned the SolrCore it operates on.

h3. Background

Under heavy load we discovered that application halts inside of Solr are 
becoming a serious problem in high traffic environments. Using Java Flight 
Recordings we discovered high accumulated applications halts on the modifyLock 
in SolrCores. In our case this means that we can only utilize our machines up 
to 25% cpu usage. With the fix applied, a utilization up to 80% is perfectly 
doable.

In our case this specific locking problem was masked by another locking problem 
in the SlowCompositeReaderWrapper. We'll submit our fix to the locking problem 
in the SlowCompositeReaderWrapper in a following issue.

h3. Problem

Our Solr instances utilizes the collapse component heavily. The instances run 
with 32 cores and 32gb Java heap on a rather small index (4gb). The instances 
scale out at 50% cpu load. We take Java Flight Recorder snapshots of 60 seconds
as soon the cpu usage exceeds 50%.

 !solrcores_locking.png|height=1024px! 

During our 60s Java Flight Recorder snapshot, the ~2k Jetty acceptor threads 
accumulated more than 12h locking time inside SolrCores on the modifyLock 
instance used as synchronized lock (see screenshot). With this fix the locking 
access is reduced to write accesses only. We validated this using another JFR 
snapshot:

 !solrcores_locking_fixed.png|height=1024px! 

We ran this code for a couple of weeks in our live environment.

h3. Solution

The synchronized modifyLock is replaced by a ReentrantReadWriteLock. This 
allows concurrent reads from the internal SolrCore instance list (cores) but 
grants exclusive access to write operations on the instance list (cores).

In Solr 9.x the cache inside the TransientSolrCoreCacheFactoryDefault adds a 
cache overflow handling of the size based internal cache (SOLR-15964). As soon 
as SolrCores are evicted from the internal cache, the cache behaviour changes 
from a size based cache to a reference based cache via the cache's eviction 
handler. SolrCore instances that are still referenced are inserted back into 
the cache. This means that write operations to the cache (insert SolrCore) can 
be issued during read operations in SolrCores. Hence these operations have only 
a read lock which cannot be upgraded to a write lock (dead lock).

To overcome this, we moved the cache maintenance (including the eviction 
handler) in TransientSolrCoreCacheFactoryDefault to a separate thread. This 
thread can acquire a write lock but on the other hand a separate thread will 
schedule a ping-pong behaviour in the eviction handler on a full cache with 
SolrCores still referenced. To overcome this we made the overflow behaviour 
transparent by adding an additional overflowCores instance. Here we add evicted 
but still referenced cores from the transientCores cache.

Furthermore we need to ensure that only a single transientSolrCoreCache inside 
TransientSolrCoreCacheFactoryDefault is created. As we now allow multiple read 
threads, we call the the getTransientCacheHandler() method initially holding a 
write lock inside the load() method. Calling the method only needs a write lock 
initially (for cache creation). For all other calls, a read lock is sufficient. 
By default, the getTransientCacheHandler() acquires a read lock. If a write is 
needed (e.g. for core creation), the getTransientCacheHandlerInternal() is 
called. This method explicitly does not use a lock in order to provide the 
flexibility to choose between a read-lock and a write-lock. This ensures that a 
single instance of transientSolrCoreCache is created.

The lock signaling between SolrCore and CoreContainer gets replaced by a 
Condition that is tied to the write lock.

h3. Summary

This change allows for a finer grained access to the list of open SolrCores. 
The decreased blocking read access is noticeable in decreased blocking times of 
the Solr application (see screenshot).

This change has been composed together by Dennis Berger, Torsten Bøgh Köster 
and Marco Petris.

  was:
Access to loaded SolrCore instances is a synchronized read and write operation 
in SolrCores#getCoreFromAnyList. This method is touched by every request as 
every HTTP request is assigned the SolrCore it operates on.

h3. Background

Under heavy load we discovered that application halts inside of Solr are 
becoming a serious problem in high traffic environments. Using Java Flight 
Recordings we discovered high accumulated applications halts on the modifyLock 
in SolrCores. In our case this means that we can only utilize our machines up 
to 25% cpu usage. With the fix

[GitHub] [solr] tboeghk opened a new pull request, #1155: [SOLR-16497] Allow finer grained locking in SolrCores

[GitBox]

tboeghk opened a new pull request, #1155:
URL: https://github.com/apache/solr/pull/1155

   https://issues.apache.org/jira/browse/SOLR-16497
   
   Access to loaded `SolrCore` instances is a synchronized read and write 
operation in `SolrCores#getCoreFromAnyList`. This method is touched by every 
request as every HTTP request is assigned the `SolrCore` it operates on.
   
   ### Background
   
   Under heavy load we discovered that application halts inside of Solr are 
becoming a serious problem in high traffic environments. Using Java Flight 
Recordings we discovered high accumulated applications halts on the 
`modifyLock` in `SolrCores`. In our case this means that we can only utilize 
our machines up to 25% cpu usage. With the fix applied, a utilization up to 80% 
is perfectly doable.
   
   > In our case this specific locking problem was masked by another locking 
problem in the `SlowCompositeReaderWrapper`. > We'll submit our fix to the 
locking problem in the `SlowCompositeReaderWrapper` in a following issue.
   
   ### Description
   
   Our Solr instances utilizes the `collapse` component heavily. The instances 
run with 32 cores and 32gb Java heap on a rather small index (4gb). The 
instances scale out at 50% cpu load. We take Java Flight Recorder snapshots of 
60 seconds
   as soon the cpu usage exceeds 50%.
   
   https://user-images.githubusercontent.com/557264/196221505-c0d819ea-574a-41f5-80ae-1b0218225b32.png";>
   
   During our 60s Java Flight Recorder snapshot, the ~2k Jetty acceptor threads 
accumulated more than 12h locking time inside `SolrCores` on the `modifyLock` 
instance. The `modifyLock` instance is used as a synchronized lock (see 
screenshot). With this fix applied, the locking access is reduced to write 
accesses only. We validated this using another JFR snapshot:
   
   https://user-images.githubusercontent.com/557264/196221528-362e5d7f-022a-4aa8-9cd7-844f59a61102.png";>
   
   We ran this code for a couple of weeks in our live environment in a 
backported version on a Solr version 8.11.2. This fix is built against the 
`main` branch.
   
   ### Solution
   
   The synchronized `modifyLock` is replaced by a 
[`ReentrantReadWriteLock`](https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/locks/ReentrantReadWriteLock.html).
 This allows concurrent reads from the internal `SolrCore` instance list 
(`cores`) but grants exclusive access to write operations on the instance list 
(`cores`).
   
   In Solr 9.x the cache inside the `TransientSolrCoreCacheFactoryDefault` adds 
a cache overflow handling of the size based internal cache 
([SOLR-15964](https://issues.apache.org/jira/browse/SOLR-15964)). As soon as 
`SolrCore`s are evicted from the internal cache, the cache behaviour changes 
from a size based cache to a reference based cache via the cache's eviction 
handler. `SolrCore` instances that are still referenced are inserted back into 
the cache. This means that write operations to the cache (insert `SolrCore`) 
can be issued during read operations in `SolrCores`. Hence these operations 
have only a read lock which cannot be upgraded to a write lock (dead lock).
   
   To overcome this, we moved the cache maintenance (including the eviction 
handler) in `TransientSolrCoreCacheFactoryDefault` to a separate thread. This 
thread can acquire a write lock but on the other hand a separate thread will 
schedule a ping-pong behaviour in the eviction handler on a full cache with 
`SolrCore`s still referenced. To overcome this we made the overflow behaviour 
transparent by adding an additional `overflowCores` instance. Here we add 
evicted but still referenced cores from the `transientCores` cache.
   
   Furthermore we need to ensure that only a single `transientSolrCoreCache` 
inside `TransientSolrCoreCacheFactoryDefault` is created. As we now allow 
multiple read threads, we call the the `getTransientCacheHandler()` method 
initially holding a write lock inside the `load()` method. Calling the method 
only needs a write lock initially (for cache creation). For all other  calls, a 
read lock is sufficient. By default, the `getTransientCacheHandler()` acquires 
a read lock. If a write is needed (e.g. for core creation), the 
`getTransientCacheHandlerInternal()` is called. This method explicitly does not 
use a lock in order to provide the flexibility to choose between a read-lock 
and a write-lock. This ensures that a single instance of 
`transientSolrCoreCache` is created.
   
   The lock signaling between `SolrCore` and `CoreContainer` gets replaced by a
   
[`Condition`](https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/locks/Condition.html)
 that is tied to the write lock. 
   
   This change allows for a finer grained access to the list of open 
`SolrCores`. The decreased blocking read access is noticeable in decreased 
blocking times of the Solr application (see screenshot).
   
   # Checklist
   
   Please review the following and check all that apply:
   

[GitHub] [solr] sonatype-lift[bot] commented on a diff in pull request #1144: SOLR-16488: Create a v2 equivalent for /admin/zookeeper v1 APIs

[GitBox]

sonatype-lift[bot] commented on code in PR #1144:
URL: https://github.com/apache/solr/pull/1144#discussion_r1010266780


##
solr/core/src/java/org/apache/solr/handler/admin/api/ZookeeperAPI.java:
##
@@ -0,0 +1,287 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.handler.admin.api;
+
+import static 
org.apache.solr.client.solrj.impl.BinaryResponseParser.BINARY_CONTENT_TYPE_V2;
+import static org.apache.solr.common.params.CommonParams.OMIT_HEADER;
+import static org.apache.solr.common.params.CommonParams.PATH;
+import static org.apache.solr.common.params.CommonParams.WT;
+import static 
org.apache.solr.security.PermissionNameProvider.Name.ZK_READ_PERM;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import io.swagger.v3.oas.annotations.Operation;
+import java.lang.invoke.MethodHandles;
+import java.util.HashMap;
+import java.util.Locale;
+import java.util.Map;
+import javax.inject.Inject;
+import javax.ws.rs.GET;
+import javax.ws.rs.Path;
+import javax.ws.rs.Produces;
+import org.apache.solr.api.JerseyResource;
+import org.apache.solr.cloud.ZkController;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.SolrException.ErrorCode;
+import org.apache.solr.common.cloud.SolrZkClient;
+import org.apache.solr.common.cloud.ZkDynamicConfig;
+import org.apache.solr.common.params.MapSolrParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.core.CoreContainer;
+import org.apache.solr.handler.admin.ZookeeperInfoHandler.FilterType;
+import org.apache.solr.handler.admin.ZookeeperInfoHandler.PageOfCollections;
+import 
org.apache.solr.handler.admin.ZookeeperInfoHandler.PagedCollectionSupport;
+import org.apache.solr.handler.admin.ZookeeperInfoHandler.ZKPrinter;
+import org.apache.solr.handler.admin.ZookeeperStatusHandler;
+import org.apache.solr.jersey.PermissionName;
+import org.apache.solr.jersey.SolrJerseyResponse;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.RawResponseWriter;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.zookeeper.KeeperException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Path("/cluster/zookeeper/")
+public class ZookeeperAPI extends JerseyResource {
+
+  private static final Logger log = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+  private final CoreContainer coreContainer;
+  private final SolrQueryRequest solrQueryRequest;
+
+  private final SolrQueryResponse solrQueryResponse;
+
+  private PagedCollectionSupport pagingSupport;
+
+  @Inject
+  public ZookeeperAPI(
+  CoreContainer coreContainer,
+  SolrQueryRequest solrQueryRequest,
+  SolrQueryResponse solrQueryResponse) {
+this.coreContainer = coreContainer;
+this.solrQueryRequest = solrQueryRequest;
+this.solrQueryResponse = solrQueryResponse;
+  }
+
+  @GET
+  @Path("/files")
+  @Produces({"application/json", "application/xml", BINARY_CONTENT_TYPE_V2})
+  @Operation(
+  summary = "List Zookeeper files.",
+  tags = {"zookeeperFiles"})
+  @PermissionName(ZK_READ_PERM)
+  public ZookeeperFilesResponse getFiles() throws Exception {
+final ZookeeperFilesResponse response = 
instantiateJerseyResponse(ZookeeperFilesResponse.class);
+final SolrParams params = solrQueryRequest.getParams();
+Map map = new HashMap<>(1);
+map.put(WT, "raw");
+map.put(OMIT_HEADER, "true");
+solrQueryRequest.setParams(SolrParams.wrapDefaults(new MapSolrParams(map), 
params));
+synchronized (this) {
+  if (pagingSupport == null) {
+pagingSupport = new PagedCollectionSupport();
+ZkController zkController = coreContainer.getZkController();
+if (zkController != null) {
+  // get notified when the ZK session expires (so we can clear the 
cached collections and
+  // rebuild)
+  zkController.addOnReconnectListener(pagingSupport);
+}
+  }
+}
+
+String path = params.get(PATH);
+
+if (params.get("addr") != null) {
+  throw new SolrException(ErrorCode.BAD_REQUEST, "Illegal parameter 
\"addr\"");
+}
+
+boolean detail = false;

[GitHub] [solr] joshgog commented on pull request #1144: SOLR-16488: Create a v2 equivalent for /admin/zookeeper v1 APIs

[GitBox]

joshgog commented on PR #1144:
URL: https://github.com/apache/solr/pull/1144#issuecomment-1298227224

   > I have changed the response property to NamedList and the result I'm 
getting is now
   > 
   > 
`{"responseHeader":{"status":0,"QTime":267},"zookeeperFiles":{"immutableCopy":{"immutableCopy":{"immutableCopy":{"immutableCopy":{"immutableCopy":{"immutableCopy":{"immutableCopy":{"immutableCopy":{"immutableCopy":{"immutableCopy":{"immutableCopy":{"immutableCopy":{"immutableCopy":{"immutableCopy":{"immutableCopy":{"immutableCopy":{"immutableCopy":{"immutableCopy":{"immutableCopy":{"immutableCopy":{"immutableCopy":{"immutableCopy":{"immutableCopy":{"immutableCopy":{"immutableCopy":{"immutableCopy":{"immutableCopy":{"immutableCopy":{"immutableCopy":`
   
   Since the handler reponse is an OutputStream I'm thinking of getting the 
OutputStrem into a String that would be returned as the response.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[jira] [Created] (SOLR-16514) lucene query: only QueryParser run well

[Nicola Paganotti (Jira)]

Nicola Paganotti created SOLR-16514:
---

 Summary: lucene query: only QueryParser run well
 Key: SOLR-16514
 URL: https://issues.apache.org/jira/browse/SOLR-16514
 Project: Solr
  Issue Type: Task
  Security Level: Public (Default Security Level. Issues are Public)
  Components: clients - python
Reporter: Nicola Paganotti


{code:java}
    def searchGiustizia(self,startPagination,recPerPage):
        indexPath = File(self.fileLucene).toPath()
        directory = FSDirectory.open(indexPath)
        searcher = IndexSearcher(DirectoryReader.open(directory))       
   
paQuery5 = TermQuery(Term("parte","TELESI RICCARDO"))
        analyzer = StandardAnalyzer()        print 
("\n--")
        start = datetime.now()
        collector = TotalHitCountCollector()
        searcher.search(paQuery5, collector)
        print("found: ",collector.getTotalHits())        
        duration = datetime.now() - start
        print("time ",str(duration))
print ("\n--") 
{code}
This function do a simple search in my Index of Apache Lucene. I use TermQuery 
but it return no results. If I use QueryParser, Lucene find me record with 
parte = "TELESI RICCARDO". Why doesn't TermQuery work?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[GitHub] [solr] AAnakhe commented on a diff in pull request #1080: SOLR 15749 created a v2 equivalent of v1 'rename'

[GitBox]

AAnakhe commented on code in PR #1080:
URL: https://github.com/apache/solr/pull/1080#discussion_r1010192031


##
solr/core/src/test/org/apache/solr/handler/admin/api/V2CollectionAPIMappingTest.java:
##
@@ -87,13 +96,15 @@ public void testGetCollectionStatus() throws Exception {
 
 
   @Test
-  public void testRenameCollectionAllParams() throws Exception {
-final SolrParams v1Params = captureConvertedV1Params(
-"/collections/collName", "POST", "{\"rename\": {\"to\": 
\"targetColl\"}}");
+  public void testRenameCollectionAllParams(){
+final SolrQueryRequest request = 
runRenameCollectionsApi("/collections/collName");

Review Comment:
   > `Test result returns java.lang.AssertionError: expected: but 
was:`
   > 
   > What if we mock `SolrQueryRequest` and `SolrQueryResponse` , pass them to 
the`public void renameCollection(SolrQueryRequest req, SolrQueryResponse rsp)` 
method of the API then perform the assertions
   
   Please I need more clarification on this



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org

[GitHub] [solr] AAnakhe commented on a diff in pull request #1080: SOLR 15749 created a v2 equivalent of v1 'rename'

[GitBox]

AAnakhe commented on code in PR #1080:
URL: https://github.com/apache/solr/pull/1080#discussion_r1010155274


##
solr/core/src/test/org/apache/solr/handler/admin/api/V2CollectionAPIMappingTest.java:
##
@@ -87,13 +96,15 @@ public void testGetCollectionStatus() throws Exception {
 
 
   @Test
-  public void testRenameCollectionAllParams() throws Exception {
-final SolrParams v1Params = captureConvertedV1Params(
-"/collections/collName", "POST", "{\"rename\": {\"to\": 
\"targetColl\"}}");
+  public void testRenameCollectionAllParams(){
+final SolrQueryRequest request = 
runRenameCollectionsApi("/collections/collName");

Review Comment:
   I will try to implement that, thanks 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org