[
https://issues.apache.org/jira/browse/SOLR-11443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16194627#comment-16194627
]
Cao Manh Dat edited comment on SOLR-11443 at 10/6/17 2:13 PM:
--------------------------------------------------------------
Patch for this ticket. The idea here is simple, we peek for 1000 messages in
the queue, processed them, write new clusterstate to ZK, then poll out these
messages. So we only poll out processed messages when new clusterstate is
written.
In case of Overseer get restarted, all the uncommitted messages still in the
queue ( no need for workqueue, only keep them for backward compatible ), we
will reprocess them and still achieve the desired state.
Here are some benchmark number ( OverseerTest.testPerformance() )
Before optimize : {{avgRequestsPerSecond: 1551.8934622998179}}
After optmize : {{avgRequestsPerSecond: 3425.594762960455}}
was (Author: caomanhdat):
Patch for this ticket. The idea here is simple, we peek for 1000 messages in
the queue, processed them, write new clusterstate to ZK, then poll out these
messages. So we only poll out processed messages when new clusterstate is
written.
In case of Overseer get restarted, all the uncommitted messages still in the
queue, we will reprocess them and still achieve the desired state.
Here are some benchmark number ( OverseerTest.testPerformance() )
Before optimize : {{avgRequestsPerSecond: 1551.8934622998179}}
After optmize : {{avgRequestsPerSecond: 3425.594762960455}}
> Remove the usage of workqueue for Overseer
> ------------------------------------------
>
> Key: SOLR-11443
> URL: https://issues.apache.org/jira/browse/SOLR-11443
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Reporter: Cao Manh Dat
> Assignee: Cao Manh Dat
> Attachments: SOLR-11443.patch
>
>
> If we can remove the usage of workqueue, We can save a lot of IO blocking in
> Overseer, hence boost performance a lot.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
