[
https://issues.apache.org/jira/browse/SOLR-18488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18130424#comment-18130424
]
Robert Lucarini commented on SOLR-18488:
----------------------------------------
I looked into the Jetty 12.1.14 internals behind this, and the diagnosis in the
description needs a correction.
_The selector does run on a dedicated thread, in practice._
ManagedSelector.doStart() submits \{{_strategy::produce}} to the executor once.
That task is the select loop and runs for the client’s lifetime. There is no
per-readiness-cycle dispatch and no wait for a free slot (the default h2sc
executor is an unbounded cached pool anyway). The h2sc-* threads in
\{{SelectorProducer.produce()}} seen in thread dumps are that long-running
loop. They are only named differently from the JDK client’s SelectorManager
thread.
_Option (b) isn’t achievable through the Builder._ ManagedSelector takes both
the select loop and I/O task dispatch from \{{selectorManager.getExecutor()}},
which is the same object as \{{HttpClient.setExecutor()}}. A separate executor
would only rename the thread.
A follow-up A/B on the same workload (60 VUs, HTTP/1.1, JDK client vs Jetty
client) narrowed things down:
* _The regression reproduces, but only in the tail._ p99 was +66% (it was +69%
in the original run). The median was unchanged (91-94 ms in both). Something
delays a subset of requests by about 130 ms; there’s no fixed per-request
overhead.
* _Connection limits are ruled out._ Jetty caps each HTTP/1.1 destination at
64 connections unless \{{withMaxConnectionsPerHost}} is set, but connections to
Solr peaked at about 25. The Jetty client actually held fewer connections than
the JDK client (median 11-14 vs 25), with no sign of pool queueing.
* _GC is ruled out._ Max GC pauses were 11-37 ms in both arms, and allocation
rates were similar.
One untested lead: a plain ExecutorService gets wrapped as NoTryExecutor, so
AdaptiveExecutionStrategy never uses Execute-Produce-Consume, and every
blocking response task is handed off to another thread. That would mainly
predict a small cost on every request, though, which the unchanged median
argues against.
So neither of the proposed Builder options (selector count, dedicated selector
thread) is likely to address this. The root cause of the tail latency is still
open, and profiling is needed before proposing an API change.
> HttpJettySolrClient: selector work is scheduled on the shared client executor
> instead of a dedicated thread, adding latency under load
> --------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SOLR-18488
> URL: https://issues.apache.org/jira/browse/SOLR-18488
> Project: Solr
> Issue Type: Improvement
> Components: clients - java
> Affects Versions: 10.0
> Reporter: Robert Lucarini
> Priority: Major
>
> HttpJettySolrClient's ManagedSelector does not run its NIO select loop on a
> dedicated thread. Instead, each selector's select() call is "produced" as a
> task via AdaptiveExecutionStrategy and dispatched onto the same
> general-purpose executor pool used for all other client work (connection
> setup, request writing, response parsing). Confirmed via live thread dumps
> under sustained concurrent load: with the default useHttp1_1(true)
> configuration (2 selectors), both selector threads consistently appear as
> h2sc-* executor-pool threads running
> ManagedSelector$SelectorProducer.produce() →
> AdaptiveExecutionStrategy.produceTask(), not as separately-named,
> always-running threads.
> This differs from HttpJdkSolrClient's underlying
> jdk.internal.net.http.HttpClientImpl, which creates exactly one dedicated
> SelectorManager thread at construction time that runs continuously for the
> life of the client and never competes with request-processing work for a
> thread-pool slot.
> Under high-concurrency load (60 VUs, small/frequent requests), this appears
> to add a measurable, consistent latency overhead compared to
> HttpJdkSolrClient — real A/B benchmark, same workload, same cluster, 5 reps:
> p99 latency increased 61-119% per rep (overall +68.8%, CI [35.6%, 99.2%])
> when switching from HttpJdkSolrClient to HttpJettySolrClient, with no other
> configuration changes. This is plausibly explained by the extra "produce →
> wait for a free executor thread to pick up the select task" dispatch hop
> occurring on every I/O-readiness cycle, which the JDK client's
> dedicated-thread design avoids entirely.
> There is currently no way to configure this: the ClientConnector/selector
> construction happens inside HttpJettySolrClient.createHttpClient(), which is
> private, and HttpJettySolrClient.Builder exposes no method to control
> selector count, execution strategy, or whether selection runs on a dedicated
> thread vs. the shared executor.
> Ask: expose a Builder option to either (a) increase the selector count
> independently of useHttp1_1, or (b) run selection on a dedicated thread
> rather than the shared client executor — similar in spirit to the existing
> useHttp1_1/withMaxConnectionsPerHost builder options.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
