[jira] [Comment Edited] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads

[Ryan Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849096#comment-15849096
 ] 

Ryan Zhang edited comment on ZOOKEEPER-2024 at 2/1/17 11:22 PM:


We imported this commit but quickly found a bug somewhere and the commit 
processor crashed because the commit is not the one that is waiting

smf1-chn-23-sr1.prod.twitter.com:  2017-01-25 02:14:30,147 - ERROR 
[CommitProcessor:4] - Got cxid 0x2c2 expected 0x3a0 for client session id 
53b398c5800b6
smf1-chn-23-sr1.prod.twitter.com:  2017-01-25 02:14:30,147 - ERROR 
[CommitProcessor:4] - Severe unrecoverable error, from thread : 
CommitProcessor:4
smf1-chn-23-sr1.prod.twitter.com:  2017-02-01 22:31:16,011 - ERROR 
[CommitProcessor:4] - Got cxid 0x35db2 expected 0x3624c for client session id 
1006589aec2087e
smf1-chn-23-sr1.prod.twitter.com:  2017-02-01 22:31:16,011 - ERROR 
[CommitProcessor:4] - Severe unrecoverable error, from thread : 
CommitProcessor:4
smf1-chn-23-sr1.prod.twitter.com:  2017-02-01 22:50:19,204 - ERROR 
[CommitProcessor:4] - Got cxid 0x0 expected 0x6126 for client session id 
4003f40105a00e7
smf1-chn-23-sr1.prod.twitter.com:  2017-02-01 22:50:19,205 - ERROR 
[CommitProcessor:4] - Severe unrecoverable error, from thread : 
CommitProcessor:4
smf1-chn-23-sr1.prod.twitter.com:  2017-02-01 22:53:03,738 - ERROR 
[CommitProcessor:4] - Got cxid 0x2fe94 expected 0x2fe96 for client session id 
563a1644b008b
smf1-chn-23-sr1.prod.twitter.com:  2017-02-01 22:53:03,738 - ERROR 
[CommitProcessor:4] - Severe unrecoverable error, from thread : 
CommitProcessor:4
smf1-chn-23-sr1.prod.twitter.com:  2017-02-01 23:06:05,795 - ERROR 
[CommitProcessor:4] - Got cxid 0x9c98c expected 0x9cb5f for client session id 
10045820b228ba2
smf1-chn-23-sr1.prod.twitter.com:  2017-02-01 23:06:05,795 - ERROR 
[CommitProcessor:4] - Severe unrecoverable error, from thread : 
CommitProcessor:4


I am adding more logs on the leader to try to debug but just wonder if anyone 
has seen this too? 


was (Author: nerdyyatrice):
We imported this commit but quickly found a bug somewhere and the commit 
processor crashed because the commit is not the one that is waiting

smf1-chn-23-sr1.prod.twitter.com:  2017-01-25 02:14:30,147 - ERROR 
[CommitProcessor:4] - Got cxid 0x2c2 expected 0x3a0 for client session id 
53b398c5800b6
smf1-chn-23-sr1.prod.twitter.com:  2017-01-25 02:14:30,147 - ERROR 
[CommitProcessor:4] - Severe unrecoverable error, from thread : 
CommitProcessor:4
smf1-chn-23-sr1.prod.twitter.com:  2017-02-01 22:31:16,011 - ERROR 
[CommitProcessor:4] - Got cxid 0x35db2 expected 0x3624c for client session id 
1006589aec2087e
smf1-chn-23-sr1.prod.twitter.com:  2017-02-01 22:31:16,011 - ERROR 
[CommitProcessor:4] - Severe unrecoverable error, from thread : 
CommitProcessor:4
smf1-chn-23-sr1.prod.twitter.com:  2017-02-01 22:50:19,204 - ERROR 
[CommitProcessor:4] - Got cxid 0x0 expected 0x6126 for client session id 
4003f40105a00e7
smf1-chn-23-sr1.prod.twitter.com:  2017-02-01 22:50:19,205 - ERROR 
[CommitProcessor:4] - Severe unrecoverable error, from thread : 
CommitProcessor:4
smf1-chn-23-sr1.prod.twitter.com:  2017-02-01 22:53:03,738 - ERROR 
[CommitProcessor:4] - Got cxid 0x2fe94 expected 0x2fe96 for client session id 
563a1644b008b
smf1-chn-23-sr1.prod.twitter.com:  2017-02-01 22:53:03,738 - ERROR 
[CommitProcessor:4] - Severe unrecoverable error, from thread : 
CommitProcessor:4
smf1-chn-23-sr1.prod.twitter.com:  2017-02-01 23:06:05,795 - ERROR 
[CommitProcessor:4] - Got cxid 0x9c98c expected 0x9cb5f for client session id 
10045820b228ba2
smf1-chn-23-sr1.prod.twitter.com:  2017-02-01 23:06:05,795 - ERROR 
[CommitProcessor:4] - Severe unrecoverable error, from thread : 
CommitProcessor:4


just wonder if anyone has seen this?

> Major throughput improvement with mixed workloads
> -
>
> Key: ZOOKEEPER-2024
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024
> Project: ZooKeeper
>  Issue Type: Improvement
>  Components: quorum, server
>Reporter: Kfir Lev-Ari
>Assignee: Kfir Lev-Ari
> Fix For: 3.6.0
>
> Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, 
> ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, 
> ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, 
> ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, 
> ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch
>
>
> The patch is applied to the commit processor, and solves two problems:
> 1. Stalling - once the commit processor encounters a local write request, it 
> stalls local processing of all sessions until it receives a commit of that 
> request from the leader. 
> In mixed workloads, this severely hampers performance as it does not allow 
> read-only sessions to proceed at faster 

[jira] [Comment Edited] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads

[Raul Gutierrez Segales (JIRA)]

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15213812#comment-15213812
 ] 

Raul Gutierrez Segales edited comment on ZOOKEEPER-2024 at 3/28/16 5:03 AM:


I pinged [~shralex] privately about this, but posting here as well: the 
indentation for comments (along with some extra tabs) went crazy on the last 
version of the patch (see the attached screenshot).

Could we please fix the indentation and remove the tabs/whitespaces? Thanks!


was (Author: rgs):
I pinged [~shralex] privately about this, but posting here as well: the 
indentation for comments (along with some extra tabs) went crazy on the last 
version of the patch.

Could we please fix the indentation and remove the tabs/whitespaces? Thanks!

> Major throughput improvement with mixed workloads
> -
>
> Key: ZOOKEEPER-2024
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024
> Project: ZooKeeper
>  Issue Type: Improvement
>  Components: quorum, server
>Reporter: Kfir Lev-Ari
>Assignee: Kfir Lev-Ari
> Fix For: 3.5.3
>
> Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, 
> ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, 
> ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, 
> ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, 
> ZOOKEEPER-2024.patch, indentation-tabs.png
>
>
> The patch is applied to the commit processor, and solves two problems:
> 1. Stalling - once the commit processor encounters a local write request, it 
> stalls local processing of all sessions until it receives a commit of that 
> request from the leader. 
> In mixed workloads, this severely hampers performance as it does not allow 
> read-only sessions to proceed at faster speed than read-write ones.
> 2. Starvation - as long as there are read requests to process, older remote 
> committed write requests are starved. 
> This occurs due to a bug fix 
> (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing 
> of local read requests before handling any committed write. The problem is 
> only manifested under high local read load. 
> Our solution solves these two problems. It improves throughput in mixed 
> workloads (in our tests, by up to 8x), and reduces latency, especially higher 
> percentiles (i.e., slowest requests). 
> The main idea is to separate sessions that inherently need to stall in order 
> to enforce order semantics, from ones that do not need to stall. To this end, 
> we add data structures for buffering and managing pending requests of stalled 
> sessions; these requests are moved out of the critical path to these data 
> structures, allowing continued processing of unaffected sessions. 
> Please see the docs:  
> 1) https://goo.gl/m1cINJ - includes a detailed description of the new commit 
> processor algorithm.
> 2) The attached patch implements our solution, and a collection of related 
> unit tests (https://reviews.apache.org/r/25160)
> 3) https://goo.gl/W0xDUP - performance results. 
> See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)