[
https://issues.apache.org/jira/browse/SOLR-11443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16199756#comment-16199756
]
Cao Manh Dat edited comment on SOLR-11443 at 10/11/17 3:34 AM:
---------------------------------------------------------------
Updated patch
- One more optimization, instead of using {{poll()}} we can remove nodes in
batch by using {{Zk.multi()}}, which saves us a lot of communications time
between Overseer and Zk. I introduced a new method {{void
DistributedQueue.remove(Collection<String> paths)}}, which remove nodes in
batch. ([~dragonsinth] : what do you think about the implementation of this
method?)
- Increased the cap of Overseer queue to 100000, cause I think Overseer can
handle more messages.
- The patch including changes from SOLR-11447.
I run the benchmark which number of state messages to 100000, here are result
Before optimize :
{code}
72804 INFO (TEST-OverseerTest.testPerformance-seed#[CA3B138EDD8B0BD5]) [ ]
o.a.s.c.OverseerTest op: state, success: 100001, failure: 0
72810 INFO (TEST-OverseerTest.testPerformance-seed#[CA3B138EDD8B0BD5]) [ ]
o.a.s.c.OverseerTest avgRequestsPerSecond: 2332.563593723015
{code}
After optimize :
{code}
42739 INFO (TEST-OverseerTest.testPerformance-seed#[7E989E665D42FFFB]) [ ]
o.a.s.c.OverseerTest op: state, success: 100001, failure: 0
42742 INFO (TEST-OverseerTest.testPerformance-seed#[7E989E665D42FFFB]) [ ]
o.a.s.c.OverseerTest avgRequestsPerSecond: 11454.952022695608
{code}
was (Author: caomanhdat):
Updated patch
- One more optimization, instead of using {{poll()}} we can remove nodes in
batch by using {{Zk.multi()}}, which saves us a lot of communications time
between Overseer and Zk. I introduced a new method {{void
DistributedQueue.remove(Collection<String> paths)}}, which remove nodes in
batch. ([~dragonsinth] : what do you think?)
- Increased the cap of Overseer queue to 100000, cause I think Overseer can
handle more messages.
- The patch including changes from SOLR-11447.
I run the benchmark which number of state messages to 100000, here are result
Before optimize :
{code}
72804 INFO (TEST-OverseerTest.testPerformance-seed#[CA3B138EDD8B0BD5]) [ ]
o.a.s.c.OverseerTest op: state, success: 100001, failure: 0
72810 INFO (TEST-OverseerTest.testPerformance-seed#[CA3B138EDD8B0BD5]) [ ]
o.a.s.c.OverseerTest avgRequestsPerSecond: 2332.563593723015
{code}
After optimize :
{code}
42739 INFO (TEST-OverseerTest.testPerformance-seed#[7E989E665D42FFFB]) [ ]
o.a.s.c.OverseerTest op: state, success: 100001, failure: 0
42742 INFO (TEST-OverseerTest.testPerformance-seed#[7E989E665D42FFFB]) [ ]
o.a.s.c.OverseerTest avgRequestsPerSecond: 11454.952022695608
{code}
> Remove the usage of workqueue for Overseer
> ------------------------------------------
>
> Key: SOLR-11443
> URL: https://issues.apache.org/jira/browse/SOLR-11443
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Reporter: Cao Manh Dat
> Assignee: Cao Manh Dat
> Attachments: SOLR-11443.patch, SOLR-11443.patch
>
>
> If we can remove the usage of workqueue, We can save a lot of IO blocking in
> Overseer, hence boost performance a lot.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
