[
https://issues.apache.org/jira/browse/SOLR-15715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17449042#comment-17449042
]
Ishan Chattopadhyaya commented on SOLR-15715:
---------------------------------------------
{quote}{{That is exactly what I want to understand – how do you scale these
nodes up? Are these autoscaled? What is the implication of scaling these nodes
on the metadata storage (i.e. ZK state) and the overseer?}}
{quote}
It is possible to use Kubernetes or AWS autoscaling based on QPS or other load
metrics to provision more of these query aggregation nodes. If this feature
leverages the Node Roles feature (SIP-15), then the implication of having many
query aggregation nodes would be that there will be many ephemeral nodes added
to the /node-roles subtree (nested under coordinator role).
{quote}{{You are concentrating a load (that was previously distributed) on a
few select nodes, so the probability of needing to scale these nodes is high. }}
{quote}
An important benefit of isolating the cost of query aggregation is that while
super expensive queries (queries of death) can still take down aggregation
nodes, but data nodes still continue to function. Regarding the "need is high"
for scaling up these query aggregation nodes, I think it is dependent on the
workload and I'm not at liberty to disclose the exact details of FullStory's
production workload.
{quote}I would still like to see a simulated (and reproducible) benchmark that
targets the above mentioned scenario and demonstrates the said feature's
handling of the same.
{quote}
The benchmark I posted above is one such simulated benchmark which is
reproducible (I'll share the steps to reproduce it once we have a PR opened).
There might be many more reproducible benchmarks to come for this feature, each
highlighting different aspects of this solution.
> Dedicated query aggregator nodes in the solr cluster.
> ------------------------------------------------------
>
> Key: SOLR-15715
> URL: https://issues.apache.org/jira/browse/SOLR-15715
> Project: Solr
> Issue Type: New Feature
> Components: SearchComponents - other
> Affects Versions: 8.10.1
> Reporter: Hitesh Khamesra
> Priority: Major
> Attachments: coordinator-poc.jpg, coordinator-poc.pdf,
> regular-node.jpg, regular-node.pdf
>
>
> We have a large collection with 1000s of shards in the solr cluster. We have
> observed that distributed solr query takes many resources(thread, memory,
> etc.) on the solr data node(node which contains indexes). Thus we need
> dedicated query nodes to execute distributed queries on large solr
> collection. That would reduce the memory/cpu pressure from solr data nodes.
> Elastis search has similar functionality
> [here|https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html#coordinating-node]
>
> [~noble.paul] [~ichattopadhyaya]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
