arup-chauhan opened a new pull request, #5273: URL: https://github.com/apache/solr/pull/5273
https://issues.apache.org/jira/browse/SOLR-17665 ## Description SOLR-17665 reports search overhead when the node's search executor is enabled even though the request uses `multiThreaded=false`. The goal of this draft is to investigate that overhead and, if reproduced on current main, isolate sequential requests from the search worker executor while preserving Solr search behavior. This is an early work-in-progress PR. The current patch prepares the benchmark comparison; it does not yet implement request executor isolation or demonstrate a remaining performance regression. ## Current changes - Extend `SolrBenchState` to configure `indexSearcherExecutorThreads` explicitly in the embedded cluster's `solr.xml`. The default benchmark cluster XML does not expose the `solr.searchThreads` property. - Add independent `searchThreads` and `multiThreaded` parameters to `ExitableDirectoryReaderSearch`. - Add document-count and timeout parameters so short setup checks can run without timeout instrumentation. - Log completed search-pool tasks at iteration boundaries, outside the timed query loop. This is a coarse activity diagnostic; it does not identify which query phase submitted work. - Use serial batched indexing to reduce variation in document ordering and retain the existing benchmark's cache disabling. The comparison separates three configurations: | Configuration | searchThreads | multiThreaded | Purpose | |---|---:|---|---| | Pool disabled | 0 | false | Baseline | | Pool enabled, sequential request | 4 | false | Investigate SOLR-17665 overhead | | Pool enabled, parallel request | 4 | true | Check the parallel configuration | This builds on the benchmark approach discussed in SOLR-17841. It targets the remaining executor-selection question in SOLR-17665, rather than reimplementing the DocSet collector improvement in #4724. That improvement is already present in this branch's baseline. The older benchmark PR #4616 remains separate. ## Next steps - [x] Configure the benchmark pool independently from the request flag. - [x] Run small trials and observe worker-pool activity. - [ ] Establish a representative workload, validate result equivalence, and run repeated pool-off versus pool-on comparisons with `multiThreaded=false`. - [ ] Profile any reproducible overhead, including query rewrite and weight creation. - [ ] Evaluate a Solr-specific request execution searcher without a worker executor for sequential requests. Preserve Solr-specific query behavior, caches, scoring, query limits, cancellation, and reader lifecycle. - [ ] Add regression coverage and compare sequential overhead and parallel performance before marking this ready for review. The request-searcher design is a hypothesis, not an implemented solution. It follows suggestions already made in SOLR-17665 and architectural comparisons with Elasticsearch and OpenSearch. No implementation from either project is included in this patch. Automatic selection based on load or query history is outside this draft's current scope. ## Validation so far - `./gradlew :solr:benchmark:compileJava tidy` - `./gradlew --no-daemon check -x test` - `./gradlew --no-daemon updateLicenses` - Three JMH smoke runs of `ExitableDirectoryReaderSearch.testShortQuery`: 1,000 documents, `verifyEDRInUse=false`, `timeAllowed=0`, one fork, one 1-second warmup iteration, one 1-second measurement iteration, one client thread, fixed seed `17665`, and a separate work directory for each configuration above. All three smoke runs completed. The parallel configuration recorded completed search-pool tasks; the two sequential configurations recorded zero. Zero tasks in this small workload does not establish that all query types avoid worker execution. These are setup checks, not reportable performance measurements, and they do not reproduce the original regression. The full unit-test suite has not been run. Prepared with significant assistance from OpenAI Codex for investigation, benchmark changes, and this description. This remains a draft for contributor review and further experiments. ## Checklist - [x] Linked the existing JIRA issue and included its ID in the title. - [x] Developed against `main`. - [x] Ran formatting, benchmark compilation, and `check -x test`. - [ ] Added regression tests for the eventual production change. - [ ] Completed performance investigation and implementation. - [ ] Added any end-user documentation and changelog needed for the final change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
