[
https://issues.apache.org/jira/browse/SOLR-16531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17655580#comment-17655580
]
Jason Gerlowski commented on SOLR-16531:
----------------------------------------
Alright, wanted to post another update on my progress to close out the week.
h3. Some (Temporarily) Dead Ends
I've spent most of the time since my last comment attempting to swap out Jersey
for a different JAX-RS implementation like RESTEasy. The "swap" ended up being
more time-consuming than anticipated, as our JAX-RS integration relies on some
Jersey-specific classes that don't have clear counterparts in other JAX-RS
impls. It should still be do-able, due to the increased scope and the limited
time before the Jan 11th date, I'm pausing my pursuit of this avenue for now.
I'd love other eyes on this though if anyone's willing in the meantime. The
code is available on [this
branch|https://github.com/gerlowskija/solr/tree/resteasy-trial-run]. The main
holdup at this point is finding a replacement for the ApplicationHandler class
used within CoreContainer and SolrCore to wrap Jersey applications.
h3. Some Progress!
On a more promising note, I had much more success spiking out a Jersey
integration that creates an "app" for each unique configset (instead of for
each and every Solr core). The code is nowhere near committable, but hacking an
approach together ended up being pretty straightforward (see the code
[here|https://github.com/gerlowskija/solr/tree/jaxrs-with-fix-and-appcaching]).
And as we'd guessed/hoped, creating a Jersey app-per-configset does show
massive improvements in the time spent on Jersey init.
As measured by a [JMH
benchmark|https://github.com/gerlowskija/solr/blob/benchmark_attempt/solr/benchmark/src/java/org/apache/solr/bench/index/SolrStartup.java],
node startup spends 28% of its CPU cycles on Jersey init when we create a
Jersey app-per-core. When we switch to app-per-configset, this drops down to
0.24%! Of course, these numbers are all pretty dependent on the benchmark
parameters (the number of cores, the number of configsets, etc.), but it
definitely proves out app-per-configset idea. I've attached flamegraphs below
showing the results.
[^vanilla-jaxrs-integration-flamegraph.html]
[^jaxrs-integration-with-app-per-configset-flamegraph.html]
h3. Next Steps and Questions
# App-per-configset is definitely the most promising path forward IMO, and we
now have some concrete proof for that. So my plan at this point is to continue
working on that. Obviously this won't be done in time for the January 11th date
we agreed to earlier. I remain willing to abide by that date since it's what we
agreed to, but I wonder whether this approach (combined with the other bits and
bobs of progress made so far) shows enough promise that it'd make sense to
extend the deadline a bit further, especially if the "per-configset" approach
is do-able for 9.2 (which continues to appear far off, given the nascent 9.1.1
release in its early stages).
# Ishan - any chance you were able to kick off a solr-bench run with the
commits/branches I pointed you at in my comment above? No rush, but excited to
get a sanity check from someone else's machine, since I've had trouble locally
with the benchmarks at higher 'numCollections'.
# Would love feedback from anyone on the RESTEasy and per-configset branches I
shared links to above!
> Performance degradation due to introduction of JAX-RS
> -----------------------------------------------------
>
> Key: SOLR-16531
> URL: https://issues.apache.org/jira/browse/SOLR-16531
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Reporter: Ishan Chattopadhyaya
> Priority: Blocker
> Fix For: 9.2
>
> Attachments: Screenshot from 2022-11-09 11-20-44.png,
> jaxrs-integration-with-app-per-configset-flamegraph.html,
> results-with-patch.tar.gz, vanilla-jaxrs-integration-flamegraph.html
>
>
> During performance benchmarking on branch_9x, I observed a slowdown in
> restart performance since commits in SOLR-16347. See attached screenshot.
> CC [~gerlowskija].
> http://mostly.cool/cluster-test-with-patch.html
> The benchmark is here:
> https://github.com/fullstorydev/solr-bench/blob/ishan/repeatable-jenkins/suites/cluster-test.json.
> This suite was run after retro-actively applying the parallelStream patch
> from SOLR-16414:
> https://github.com/apache/solr/commit/b33161d0cdd976fc0c3dc78c4afafceb4db671cf.diff
>
> Effort to automate these benchmarks is WIP and tracked here: SOLR-16525.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
