[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15855362#comment-15855362 ] Alexander Shraer commented on ZOOKEEPER-2024: - yep, thanks > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Fix For: 3.6.0 > > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > Please see the docs: > 1) https://goo.gl/m1cINJ - includes a detailed description of the new commit > processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) https://goo.gl/W0xDUP - performance results. > (See https://issues.apache.org/jira/browse/ZOOKEEPER-2023 for the > corresponding new system test that produced these performance measurements) > > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15855358#comment-15855358 ] Ryan Zhang commented on ZOOKEEPER-2024: --- created https://issues.apache.org/jira/browse/ZOOKEEPER-2684, still trying to figure out the process. Just to confirm, is that link's content up to date? > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Fix For: 3.6.0 > > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > Please see the docs: > 1) https://goo.gl/m1cINJ - includes a detailed description of the new commit > processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) https://goo.gl/W0xDUP - performance results. > (See https://issues.apache.org/jira/browse/ZOOKEEPER-2023 for the > corresponding new system test that produced these performance measurements) > > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15855005#comment-15855005 ] Alexander Shraer commented on ZOOKEEPER-2024: - Hi Ryan, please create a separate JIRA and assign it to yourself. You can then create a pull request via github, https://cwiki.apache.org/confluence/display/ZOOKEEPER/HowToContribute Looking forward to your jira - I'm curious what the bug is :) Thanks! Alex > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Fix For: 3.6.0 > > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > Please see the docs: > 1) https://goo.gl/m1cINJ - includes a detailed description of the new commit > processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) https://goo.gl/W0xDUP - performance results. > (See https://issues.apache.org/jira/browse/ZOOKEEPER-2023 for the > corresponding new system test that produced these performance measurements) > > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15854986#comment-15854986 ] Ryan Zhang commented on ZOOKEEPER-2024: --- I found the bug, should I create another issue or apply a patch here? I am super confused on the process. > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Fix For: 3.6.0 > > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > Please see the docs: > 1) https://goo.gl/m1cINJ - includes a detailed description of the new commit > processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) https://goo.gl/W0xDUP - performance results. > (See https://issues.apache.org/jira/browse/ZOOKEEPER-2023 for the > corresponding new system test that produced these performance measurements) > > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15849096#comment-15849096 ] Ryan Zhang commented on ZOOKEEPER-2024: --- We imported this commit but quickly found a bug somewhere and the commit processor crashed because the commit is not the one that is waiting smf1-chn-23-sr1.prod.twitter.com: 2017-01-25 02:14:30,147 - ERROR [CommitProcessor:4] - Got cxid 0x2c2 expected 0x3a0 for client session id 53b398c5800b6 smf1-chn-23-sr1.prod.twitter.com: 2017-01-25 02:14:30,147 - ERROR [CommitProcessor:4] - Severe unrecoverable error, from thread : CommitProcessor:4 smf1-chn-23-sr1.prod.twitter.com: 2017-02-01 22:31:16,011 - ERROR [CommitProcessor:4] - Got cxid 0x35db2 expected 0x3624c for client session id 1006589aec2087e smf1-chn-23-sr1.prod.twitter.com: 2017-02-01 22:31:16,011 - ERROR [CommitProcessor:4] - Severe unrecoverable error, from thread : CommitProcessor:4 smf1-chn-23-sr1.prod.twitter.com: 2017-02-01 22:50:19,204 - ERROR [CommitProcessor:4] - Got cxid 0x0 expected 0x6126 for client session id 4003f40105a00e7 smf1-chn-23-sr1.prod.twitter.com: 2017-02-01 22:50:19,205 - ERROR [CommitProcessor:4] - Severe unrecoverable error, from thread : CommitProcessor:4 smf1-chn-23-sr1.prod.twitter.com: 2017-02-01 22:53:03,738 - ERROR [CommitProcessor:4] - Got cxid 0x2fe94 expected 0x2fe96 for client session id 563a1644b008b smf1-chn-23-sr1.prod.twitter.com: 2017-02-01 22:53:03,738 - ERROR [CommitProcessor:4] - Severe unrecoverable error, from thread : CommitProcessor:4 smf1-chn-23-sr1.prod.twitter.com: 2017-02-01 23:06:05,795 - ERROR [CommitProcessor:4] - Got cxid 0x9c98c expected 0x9cb5f for client session id 10045820b228ba2 smf1-chn-23-sr1.prod.twitter.com: 2017-02-01 23:06:05,795 - ERROR [CommitProcessor:4] - Severe unrecoverable error, from thread : CommitProcessor:4 just wonder if anyone has seen this? > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Fix For: 3.6.0 > > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > Please see the docs: > 1) https://goo.gl/m1cINJ - includes a detailed description of the new commit > processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) https://goo.gl/W0xDUP - performance results. > (See https://issues.apache.org/jira/browse/ZOOKEEPER-2023 for the > corresponding new system test that produced these performance measurements) > > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15738520#comment-15738520 ] Kfir Lev-Ari commented on ZOOKEEPER-2024: - [~randgalt], I've made ZooNet public now, you can find it [here|https://github.com/kfirlevari/ZooNet]. I also wrote ZooNet a systest, as in [ZOOKEEPER-2023|https://issues.apache.org/jira/browse/ZOOKEEPER-2023], one that supports multiple ZKs. For simplicity, I've also added a small script that runs the ZooNet systest ([this|https://github.com/kfirlevari/ZooNet/blob/trunk/README.md] explains how to run it). Obviously, this implementation can be improved, e.g., better support for async ops (see [here|https://github.com/kfirlevari/ZooNet/issues/1] for details). Let me know what you think ;) Cheers, Kfir > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Fix For: 3.6.0 > > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > Please see the docs: > 1) https://goo.gl/m1cINJ - includes a detailed description of the new commit > processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) https://goo.gl/W0xDUP - performance results. > (See https://issues.apache.org/jira/browse/ZOOKEEPER-2023 for the > corresponding new system test that produced these performance measurements) > > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15738339#comment-15738339 ] Jordan Zimmerman commented on ZOOKEEPER-2024: - [~kfirlevari] Where is the code for ZooNet mentioned in the white paper? I'd love to add this to Apache Curator. > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Fix For: 3.6.0 > > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > Please see the docs: > 1) https://goo.gl/m1cINJ - includes a detailed description of the new commit > processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) https://goo.gl/W0xDUP - performance results. > (See https://issues.apache.org/jira/browse/ZOOKEEPER-2023 for the > corresponding new system test that produced these performance measurements) > > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15304590#comment-15304590 ] Edward Ribeiro commented on ZOOKEEPER-2024: --- Congratulations, [~kfirlevari], it's awesome work!!! :D By the way, what do you think about migrating both the algorithm description and performance results docs you linked above to the official documentation (src/docs/src/documentation/content/xdocs/)? I guess zookeeperInternals.xml is a good fit for it. Just a suggestion for future reference. Does it make sense, [~rgs], [~fpj]? Cheers, Ed. > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Fix For: 3.6.0 > > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > Please see the docs: > 1) https://goo.gl/m1cINJ - includes a detailed description of the new commit > processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) https://goo.gl/W0xDUP - performance results. > (See https://issues.apache.org/jira/browse/ZOOKEEPER-2023 for the > corresponding new system test that produced these performance measurements) > > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15304517#comment-15304517 ] Kfir Lev-Ari commented on ZOOKEEPER-2024: - Thank you very much [~cnauroth] for your review! ;) I also wish to thank everyone here for their part in making this patch part of ZK. > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Fix For: 3.6.0 > > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > Please see the docs: > 1) https://goo.gl/m1cINJ - includes a detailed description of the new commit > processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) https://goo.gl/W0xDUP - performance results. > (See https://issues.apache.org/jira/browse/ZOOKEEPER-2023 for the > corresponding new system test that produced these performance measurements) > > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15302608#comment-15302608 ] Chris Nauroth commented on ZOOKEEPER-2024: -- I took another review pass, and here is my +1. This is belated of course, but I wanted to show my appreciation for the solid work and perseverance by [~kfirlevari]. Thank you so much for the contribution. I agree with waiting for an explicit +1 from reviewers, and I also apologize for my own delay in reviewing this. > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Fix For: 3.6.0 > > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > Please see the docs: > 1) https://goo.gl/m1cINJ - includes a detailed description of the new commit > processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) https://goo.gl/W0xDUP - performance results. > (See https://issues.apache.org/jira/browse/ZOOKEEPER-2023 for the > corresponding new system test that produced these performance measurements) > > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15284205#comment-15284205 ] Edward Bortnikov commented on ZOOKEEPER-2024: - Yay, great news Alex! We did not believe this commit would make it before 2024 :) On Monday, May 16, 2016 12:36 AM, Alexander Shraer (JIRA) wrote: [ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Shraer updated ZOOKEEPER-2024: Resolution: Fixed Status: Resolved (was: Patch Available) Committer to trunk. -- This message was sent by Atlassian JIRA (v6.3.4#6332) > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Fix For: 3.6.0 > > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > Please see the docs: > 1) https://goo.gl/m1cINJ - includes a detailed description of the new commit > processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) https://goo.gl/W0xDUP - performance results. > (See https://issues.apache.org/jira/browse/ZOOKEEPER-2023 for the > corresponding new system test that produced these performance measurements) > > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15284008#comment-15284008 ] Flavio Junqueira commented on ZOOKEEPER-2024: - I appreciate the concern with the community and fully agree about jira turnaround. However, the limited time the contributors of this project have had have been focusing on solving critical issues that affect ZooKeeper in production. This jira was lined up for trunk, so it really was lower in my priority list because I'm investing time in the blockers for 3.5.2, and I think it is the same for several of the other contributors. We do wait for people to +1 it, no matter how irritating the wait might be (and it often is), sometimes even pinging them outside the list. Specifically to this jira, there aren't that many active people here, so you could easily have just pinged everyone. In fact, I think it is just [~rgs] and I. I really want to make a big deal out of this because I don't want to see this kind of behavior again in other jiras. I like much better the approach much of getting reviewers to clearly state their approval, and the fact that a jira doesn't get the priority you'd like, doesn't mean you should bypass. Please wait for the votes next time and ping people accordingly. And let's work on your suggestion of not holding jiras for long, I'm all for it. [~kfirlevari] I apologize for the wait and thanks for contributing. > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Fix For: 3.6.0 > > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > Please see the docs: > 1) https://goo.gl/m1cINJ - includes a detailed description of the new commit > processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) https://goo.gl/W0xDUP - performance results. > (See https://issues.apache.org/jira/browse/ZOOKEEPER-2023 for the > corresponding new system test that produced these performance measurements) > > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15284001#comment-15284001 ] Alexander Shraer commented on ZOOKEEPER-2024: - [~fpj], ideally I'd also like it to be the case. But, many people have reviewed and gave comments on this patch, and we can't possibly wait for all of them. Specifically your comments have been addressed 13 Apr and since then we've been pinging on this thread. If there's something wrong with the patch then by all means revert it, but dragging the review process for months and years (this patch is been at works since 2014!) is not the way this community should work. > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Fix For: 3.6.0 > > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > Please see the docs: > 1) https://goo.gl/m1cINJ - includes a detailed description of the new commit > processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) https://goo.gl/W0xDUP - performance results. > (See https://issues.apache.org/jira/browse/ZOOKEEPER-2023 for the > corresponding new system test that produced these performance measurements) > > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15283997#comment-15283997 ] Flavio Junqueira commented on ZOOKEEPER-2024: - [~shralex] this is is not how we typically commit code in this community. According to the bylaws, you have done nothing wrong, but we if we look at many of the other jiras, specially the ones with significant changes, we wait until people who have reviewed it give their vote. Please ping people before proceeding, it is not very nice to bypass like this and we are all trying out best to keep the project in the best shape possible, not only satisfy individual preferences. This is in trunk, so we have time to fix it, but I'd appreciate if folks had a look at it because if there is something wrong and enough -1 votes we will revert it. > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Fix For: 3.5.3 > > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > Please see the docs: > 1) https://goo.gl/m1cINJ - includes a detailed description of the new commit > processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) https://goo.gl/W0xDUP - performance results. > (See https://issues.apache.org/jira/browse/ZOOKEEPER-2023 for the > corresponding new system test that produced these performance measurements) > > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15283995#comment-15283995 ] Raul Gutierrez Segales commented on ZOOKEEPER-2024: --- Cool [~shralex]! Thanks for the hard work. Will be good to know how this behaves in prod as early adopters step forward :-) > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Fix For: 3.5.3 > > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > Please see the docs: > 1) https://goo.gl/m1cINJ - includes a detailed description of the new commit > processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) https://goo.gl/W0xDUP - performance results. > (See https://issues.apache.org/jira/browse/ZOOKEEPER-2023 for the > corresponding new system test that produced these performance measurements) > > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15283992#comment-15283992 ] Hudson commented on ZOOKEEPER-2024: --- SUCCESS: Integrated in ZooKeeper-trunk #2925 (See [https://builds.apache.org/job/ZooKeeper-trunk/2925/]) ZOOKEEPER-2024 Major throughput improvement with mixed workloads (Kfir Lev-Ari via shralex) (shralex: [http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1743978]) * trunk/CHANGES.txt * trunk/src/java/main/org/apache/zookeeper/server/quorum/CommitProcessor.java * trunk/src/java/test/org/apache/zookeeper/server/quorum/CommitProcessorConcurrencyTest.java * trunk/src/java/test/org/apache/zookeeper/server/quorum/CommitProcessorTest.java > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Fix For: 3.5.3 > > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > Please see the docs: > 1) https://goo.gl/m1cINJ - includes a detailed description of the new commit > processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) https://goo.gl/W0xDUP - performance results. > (See https://issues.apache.org/jira/browse/ZOOKEEPER-2023 for the > corresponding new system test that produced these performance measurements) > > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15283989#comment-15283989 ] Alexander Shraer commented on ZOOKEEPER-2024: - Hi Raul, no, I'm the only one that approved. Last revision to the patch was almost 2 months ago, and Kfir addressed all comments by Flavio, you and others. Flavio's last comments were addressed a month ago. This patch had many eyes on it and evaluated like no other I'm aware of :) I decided that its fine to commit at this point. > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Fix For: 3.5.3 > > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > Please see the docs: > 1) https://goo.gl/m1cINJ - includes a detailed description of the new commit > processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) https://goo.gl/W0xDUP - performance results. > (See https://issues.apache.org/jira/browse/ZOOKEEPER-2023 for the > corresponding new system test that produced these performance measurements) > > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15283979#comment-15283979 ] Raul Gutierrez Segales commented on ZOOKEEPER-2024: --- Well, that was too late I guess... Did Flavio +1 it offline? > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Fix For: 3.5.3 > > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > Please see the docs: > 1) https://goo.gl/m1cINJ - includes a detailed description of the new commit > processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) https://goo.gl/W0xDUP - performance results. > (See https://issues.apache.org/jira/browse/ZOOKEEPER-2023 for the > corresponding new system test that produced these performance measurements) > > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15283976#comment-15283976 ] Raul Gutierrez Segales commented on ZOOKEEPER-2024: --- [~shralex]: thanks for the follow-up Alex. I am doing another pass now. A small nit though: the tabs and extra whitespaces are still there, which make the patch hard to read via git diff (and git show) (and possibly in many editors). If you have a git checkout of the repo, try git diff if your patch is not committed to a branch, otherwise git show. And you'll see something like: http://imgur.com/a/8rclv > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Fix For: 3.5.3 > > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > Please see the docs: > 1) https://goo.gl/m1cINJ - includes a detailed description of the new commit > processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) https://goo.gl/W0xDUP - performance results. > (See https://issues.apache.org/jira/browse/ZOOKEEPER-2023 for the > corresponding new system test that produced these performance measurements) > > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15283955#comment-15283955 ] Hadoop QA commented on ZOOKEEPER-2024: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12804079/ZOOKEEPER-2024.patch against trunk revision 1742472. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 11 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3162//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3162//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3162//console This message is automatically generated. > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Fix For: 3.5.3 > > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > Please see the docs: > 1) https://goo.gl/m1cINJ - includes a detailed description of the new commit > processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) https://goo.gl/W0xDUP - performance results. > (See https://issues.apache.org/jira/browse/ZOOKEEPER-2023 for the > corresponding new system test that produced these performance measurements) > > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15281006#comment-15281006 ] Alexander Shraer commented on ZOOKEEPER-2024: - Hey [~fpj], any remaining concerns or ok to commit this to trunk ? > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Fix For: 3.5.3 > > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > Please see the docs: > 1) https://goo.gl/m1cINJ - includes a detailed description of the new commit > processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) https://goo.gl/W0xDUP - performance results. > (See https://issues.apache.org/jira/browse/ZOOKEEPER-2023 for the > corresponding new system test that produced these performance measurements) > > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255726#comment-15255726 ] Kfir Lev-Ari commented on ZOOKEEPER-2024: - Hi all, Please let me know if there is anything else that is missing in order for this patch to be submitted. Thanks ;) > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Fix For: 3.5.3 > > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > Please see the docs: > 1) https://goo.gl/m1cINJ - includes a detailed description of the new commit > processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) https://goo.gl/W0xDUP - performance results. > (See https://issues.apache.org/jira/browse/ZOOKEEPER-2023 for the > corresponding new system test that produced these performance measurements) > > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15238870#comment-15238870 ] Kfir Lev-Ari commented on ZOOKEEPER-2024: - [~fpj], thanks for pushing for the latency numbers, they turned out to be quite good and I already had it in the logs. I've updated in the evaluation document the throughput graphs - now they contain latency data for each scenario. Its not the most elegant way to present it, but here is the summary. As expected, the latency in read-only workload is equivalent in both algorithms, as well as in write-only workloads. The new algorithm significantly reduces read and write latency in mixed workloads in which the write percentage is below 30 (for reads up to 96% improvement, for writes up to 89%). In cases of mixed workload in which the R/W clients have write intensive workload (of 100%-70%) the latency of the RO is reduced due to the starvation problem in the original algorithm (up to -184% reduction). I.e., the readers in the original algorithm have lower latency, but they read older values and postpone the update of the local DB by remote writes. See the median latency in the 3 servers scenario, 50% R/W Clients, 50% RO Clients, 0% reads column. > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Fix For: 3.5.3 > > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > Please see the docs: > 1) https://goo.gl/m1cINJ - includes a detailed description of the new commit > processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) https://goo.gl/W0xDUP - performance results. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214288#comment-15214288 ] Hadoop QA commented on ZOOKEEPER-2024: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12795628/ZOOKEEPER-2024.patch against trunk revision 1736259. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 11 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3127//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3127//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3127//console This message is automatically generated. > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Fix For: 3.5.3 > > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > Please see the docs: > 1) https://goo.gl/m1cINJ - includes a detailed description of the new commit > processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) https://goo.gl/W0xDUP - performance results. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214278#comment-15214278 ] Flavio Junqueira commented on ZOOKEEPER-2024: - bq. I can address your formatting comments though, would that be ok? Sure, sounds good to me if you prefer to do it. bq. It is pretty clear that latency improves too, isn't it ? I'd rather check it or have it checked to avoid anything unexpected. We should do more than just reason about it. Also, you mention reads, but I'd like to make sure that writes won't suffer due to something unexpected. It shouldn't be a big deal to check it out. I'm willing to help out after 3.5.2 is out, right now there are other more pressing jiras that require my attention. bq. Synchronization Thanks for the pointer, let me think a bit more about this one and get back to you. > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Fix For: 3.5.3 > > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > Please see the docs: > 1) https://goo.gl/m1cINJ - includes a detailed description of the new commit > processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) https://goo.gl/W0xDUP - performance results. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15213865#comment-15213865 ] Hadoop QA commented on ZOOKEEPER-2024: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12795577/ZOOKEEPER-2024.patch against trunk revision 1736259. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 11 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3126//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3126//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3126//console This message is automatically generated. > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Fix For: 3.5.3 > > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, indentation-tabs.png > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > Please see the docs: > 1) https://goo.gl/m1cINJ - includes a detailed description of the new commit > processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) https://goo.gl/W0xDUP - performance results. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15213841#comment-15213841 ] Hadoop QA commented on ZOOKEEPER-2024: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12795573/indentation-tabs.png against trunk revision 1736259. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3125//console This message is automatically generated. > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Fix For: 3.5.3 > > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, indentation-tabs.png > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > Please see the docs: > 1) https://goo.gl/m1cINJ - includes a detailed description of the new commit > processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) https://goo.gl/W0xDUP - performance results. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15213673#comment-15213673 ] Hadoop QA commented on ZOOKEEPER-2024: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12795562/ZOOKEEPER-2024.patch against trunk revision 1736259. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 11 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3124//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3124//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3124//console This message is automatically generated. > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Fix For: 3.5.3 > > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > Please see the docs: > 1) https://goo.gl/m1cINJ - includes a detailed description of the new commit > processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) https://goo.gl/W0xDUP - performance results. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15213521#comment-15213521 ] Alexander Shraer commented on ZOOKEEPER-2024: - Thanks for the comments, Flavio. Due to some personal circumstances, Kfir may not have time to do new experiments any time soon. I can address your formatting comments though, would that be ok? Wrt latency, Kfir checked it at some point but doesn't have recent data. It is pretty clear that latency improves too, isn't it ? instead of being blocked by any write each request may only be blocked by previous writes in the same client session, so there should be a huge win for latency as well. I don't see any reason why latency could become worse due to this change. Synchronization: https://goo.gl/m1cINJ gives more details, please see pseudocode on the last page. The idea is that synchronization on new incoming requests is now separate from synchronization on "thread pool is available to process more requests", since we may have work to do even if there is no "new" incoming requests. Waiting for empty pool before and after a write: This is the same as before the patch (check out the condition for waking up the thread right now). Due to synchronization issues in FinalRequestProcessor (potential races between setting watches and updating data), commitRequestProcessor has always either sent it many reads or a single write. We kept this in the patch. I think this part can be made more efficient, e.g., using read/write locks. We had discussions about this offline with a few contributors, and I plan to open a separate Jira for this. > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Fix For: 3.5.3 > > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > Please see the docs: > 1) https://goo.gl/m1cINJ - includes a detailed description of the new commit > processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) https://goo.gl/W0xDUP - performance results. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15213485#comment-15213485 ] Flavio Junqueira commented on ZOOKEEPER-2024: - [~kfirlevari] thanks for the work here so far, nice improvement. your results focus on throughput, though, have you checked latency? > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Fix For: 3.5.3 > > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > Please see the docs: > 1) https://goo.gl/m1cINJ - includes a detailed description of the new commit > processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) https://goo.gl/W0xDUP - performance results. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15213230#comment-15213230 ] Alexander Shraer commented on ZOOKEEPER-2024: - Thanks a lot [~rakeshr] and [~rgs] for reviewing. I added a couple of clarification questions for you to reviewboard. > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Fix For: 3.5.3 > > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > Please see the docs: > 1) https://goo.gl/m1cINJ - includes a detailed description of the new commit > processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) https://goo.gl/W0xDUP - performance results. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15213147#comment-15213147 ] Raul Gutierrez Segales commented on ZOOKEEPER-2024: --- [~kfirlevari]: I am doing another round of reviews and I see that a lot of white-spaces and tabs sneaked in... do you mind removing them please? They are pretty easy to spot via reviewboard or (if you have a git via svn checkout) with git diff.. Thanks! > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Fix For: 3.5.3 > > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > Please see the docs: > 1) https://goo.gl/m1cINJ - includes a detailed description of the new commit > processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) https://goo.gl/W0xDUP - performance results. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15213108#comment-15213108 ] Hadoop QA commented on ZOOKEEPER-2024: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12795517/ZOOKEEPER-2024.patch against trunk revision 1736259. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 11 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3122//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3122//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3122//console This message is automatically generated. > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Fix For: 3.5.3 > > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > Please see the docs: > 1) https://goo.gl/m1cINJ - includes a detailed description of the new commit > processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) https://goo.gl/W0xDUP - performance results. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15212835#comment-15212835 ] Hadoop QA commented on ZOOKEEPER-2024: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12795502/ZOOKEEPER-2024.patch against trunk revision 1736259. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 11 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3121//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3121//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3121//console This message is automatically generated. > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Fix For: 3.5.3 > > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > Please see the docs: > 1) https://goo.gl/m1cINJ - includes a detailed description of the new commit > processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) https://goo.gl/W0xDUP - performance results. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15211635#comment-15211635 ] Rakesh R commented on ZOOKEEPER-2024: - Thanks [~kfirlevari] and [~shralex] for the continuous effort. I've added few review comments in [RB|https://reviews.apache.org/r/25160/], let me find some more time in coming days and will push this to trunk. > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Fix For: 3.5.3 > > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > Please see the docs: > 1) https://goo.gl/m1cINJ - includes a detailed description of the new commit > processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) https://goo.gl/W0xDUP - performance results. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205455#comment-15205455 ] Alexander Shraer commented on ZOOKEEPER-2024: - Hi All, We've kept this patch waiting for more a month now. Given the thorough review/revisions it got on reviewboard and the extensive testing and performance measurements attached here, I feel pretty confident about this feature and would like to commit it to trunk. I'll wait a few more days to let people comment/review. Cheers, Alex > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Fix For: 3.5.3 > > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > Please see the docs: > 1) https://goo.gl/m1cINJ - includes a detailed description of the new commit > processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) https://goo.gl/W0xDUP - performance results. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15159561#comment-15159561 ] Alexander Shraer commented on ZOOKEEPER-2024: - [~rgs], this patch does not change the Final Request Processor where watches are triggered, or the interaction of the Commit Processor with it, so I don't believe it introduces new races. In particular the thread pool in Commit Processor is still being cleared before and after each write is entered to the pool (on any thread). > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Fix For: 3.5.2 > > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > Please see the docs: > 1) https://goo.gl/m1cINJ - includes a detailed description of the new commit > processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) https://goo.gl/W0xDUP - performance results. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15159543#comment-15159543 ] Raul Gutierrez Segales commented on ZOOKEEPER-2024: --- +1 on aiming for trunk first. [~shralex]: one remaining concern (will re-read the code today) is with losing watches. We had a very similar patch at Twitter but we ended up with some weird corner cases in which watches were lost. cc: [~svoutil], [~tnarg] > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Fix For: 3.5.2 > > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > Please see the docs: > 1) https://goo.gl/m1cINJ - includes a detailed description of the new commit > processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) https://goo.gl/W0xDUP - performance results. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15159514#comment-15159514 ] Alexander Shraer commented on ZOOKEEPER-2024: - Sure, lets focus on getting this into trunk, and defer the release question. > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Fix For: 3.5.2 > > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > Please see the docs: > 1) https://goo.gl/m1cINJ - includes a detailed description of the new commit > processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) https://goo.gl/W0xDUP - performance results. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15159316#comment-15159316 ] Flavio Junqueira commented on ZOOKEEPER-2024: - Thanks for working on this [~kfirlevari]. I think this is great, but this should go into trunk, not the 3.5 branch. We already have enough to stabilize in 3.5 and even though this is a nice improvement, this isn't a pressing one. > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Fix For: 3.5.2 > > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > Please see the docs: > 1) https://goo.gl/m1cINJ - includes a detailed description of the new commit > processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) https://goo.gl/W0xDUP - performance results. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15157917#comment-15157917 ] Hadoop QA commented on ZOOKEEPER-2024: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12788955/ZOOKEEPER-2024.patch against trunk revision 1729259. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 11 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3048//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3048//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3048//console This message is automatically generated. > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Fix For: 3.5.2 > > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > Please see the docs: > 1) https://goo.gl/m1cINJ - includes a detailed description of the new commit > processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) https://goo.gl/W0xDUP - performance results. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15154706#comment-15154706 ] Alexander Shraer commented on ZOOKEEPER-2024: - Hi All, The patch is now in a very good shape, thanks a lot Kfir for the hard work. This simplifies the previous commit processor algorithm, has significant performance gains (up to x10 in the attached graphs, but with non-saturated workloads Kfir observed even higher gains), as well as solves the write starvation problem in read-only workloads. Assuming Hudson tests pass, I vote +1. It would be great if others can take a look and vote too. The attached documents and review board are up to date. Cheers, Alex > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > Please see the docs: > 1) https://goo.gl/m1cINJ - includes a detailed description of the new commit > processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) https://goo.gl/W0xDUP - performance results. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15145177#comment-15145177 ] Hadoop QA commented on ZOOKEEPER-2024: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12787705/ZOOKEEPER-2024.patch against trunk revision 1729259. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 11 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3046//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3046//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3046//console This message is automatically generated. > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > Please see the docs: > 1) > https://docs.google.com/document/d/1oXJiSt9VqL35hCYQRmFuC63ETd0F_g6uApzocgkFe3Y/edit?usp=sharing > - includes a detailed description of the new commit processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) > https://docs.google.com/spreadsheets/d/11mmobkIf-0czIyEEwgytwqRme5OH8tmZcb4EBcsMZ_w/edit?usp=sharing > - new performance results. > https://docs.google.com/spreadsheets/d/1vmdfsq4WLr92BQO-CGcualE0KhAtjIu3bCaVwYajLo8/edit?usp=sharing > - shows (old) performance results of running system tests on the patched ZK > using the patched system test from > https://issues.apache.org/jira/browse/ZOOKEEPER-2023. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15144536#comment-15144536 ] Hadoop QA commented on ZOOKEEPER-2024: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12787650/ZOOKEEPER-2024.patch against trunk revision 1729259. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 11 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3045//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3045//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3045//console This message is automatically generated. > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > Please see the docs: > 1) > https://docs.google.com/document/d/1oXJiSt9VqL35hCYQRmFuC63ETd0F_g6uApzocgkFe3Y/edit?usp=sharing > - includes a detailed description of the new commit processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) > https://docs.google.com/spreadsheets/d/11mmobkIf-0czIyEEwgytwqRme5OH8tmZcb4EBcsMZ_w/edit?usp=sharing > - new performance results. > https://docs.google.com/spreadsheets/d/1vmdfsq4WLr92BQO-CGcualE0KhAtjIu3bCaVwYajLo8/edit?usp=sharing > - shows (old) performance results of running system tests on the patched ZK > using the patched system test from > https://issues.apache.org/jira/browse/ZOOKEEPER-2023. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15142287#comment-15142287 ] Hadoop QA commented on ZOOKEEPER-2024: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12787423/ZOOKEEPER-2024.patch against trunk revision 1729259. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 11 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3044//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3044//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3044//console This message is automatically generated. > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > In order to avoid starvation, our solution prioritizes committed write > requests over reads, and enforces fairness among read requests of sessions. > Please see the docs: > 1) > https://docs.google.com/document/d/1oXJiSt9VqL35hCYQRmFuC63ETd0F_g6uApzocgkFe3Y/edit?usp=sharing > - includes a detailed description of the new commit processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) > https://docs.google.com/spreadsheets/d/11mmobkIf-0czIyEEwgytwqRme5OH8tmZcb4EBcsMZ_w/edit?usp=sharing > - new performance results. > https://docs.google.com/spreadsheets/d/1vmdfsq4WLr92BQO-CGcualE0KhAtjIu3bCaVwYajLo8/edit?usp=sharing > - shows (old) performance results of running system tests on the patched ZK > using the patched system test from > https://issues.apache.org/jira/browse/ZOOKEEPER-2023. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15141685#comment-15141685 ] Hadoop QA commented on ZOOKEEPER-2024: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12787326/ZOOKEEPER-2024.patch against trunk revision 1729259. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 11 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3043//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3043//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3043//console This message is automatically generated. > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > In order to avoid starvation, our solution prioritizes committed write > requests over reads, and enforces fairness among read requests of sessions. > Please see the docs: > 1) > https://docs.google.com/document/d/1oXJiSt9VqL35hCYQRmFuC63ETd0F_g6uApzocgkFe3Y/edit?usp=sharing > - includes a detailed description of the new commit processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) > https://docs.google.com/spreadsheets/d/11mmobkIf-0czIyEEwgytwqRme5OH8tmZcb4EBcsMZ_w/edit?usp=sharing > - new performance results. > https://docs.google.com/spreadsheets/d/1vmdfsq4WLr92BQO-CGcualE0KhAtjIu3bCaVwYajLo8/edit?usp=sharing > - shows (old) performance results of running system tests on the patched ZK > using the patched system test from > https://issues.apache.org/jira/browse/ZOOKEEPER-2023. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15116355#comment-15116355 ] Hadoop QA commented on ZOOKEEPER-2024: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12784276/ZOOKEEPER-2024.patch against trunk revision 1726354. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 11 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3015//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3015//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3015//console This message is automatically generated. > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > In order to avoid starvation, our solution prioritizes committed write > requests over reads, and enforces fairness among read requests of sessions. > Please see the docs: > 1) > https://docs.google.com/document/d/1oXJiSt9VqL35hCYQRmFuC63ETd0F_g6uApzocgkFe3Y/edit?usp=sharing > - includes a detailed description of the new commit processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) > https://docs.google.com/spreadsheets/d/11mmobkIf-0czIyEEwgytwqRme5OH8tmZcb4EBcsMZ_w/edit?usp=sharing > - new performance results. > https://docs.google.com/spreadsheets/d/1vmdfsq4WLr92BQO-CGcualE0KhAtjIu3bCaVwYajLo8/edit?usp=sharing > - shows (old) performance results of running system tests on the patched ZK > using the patched system test from > https://issues.apache.org/jira/browse/ZOOKEEPER-2023. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15116336#comment-15116336 ] Hadoop QA commented on ZOOKEEPER-2024: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12784274/ZOOKEEPER-2024.patch against trunk revision 1726354. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 11 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3014//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3014//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3014//console This message is automatically generated. > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > In order to avoid starvation, our solution prioritizes committed write > requests over reads, and enforces fairness among read requests of sessions. > Please see the docs: > 1) > https://docs.google.com/document/d/1oXJiSt9VqL35hCYQRmFuC63ETd0F_g6uApzocgkFe3Y/edit?usp=sharing > - includes a detailed description of the new commit processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) > https://docs.google.com/spreadsheets/d/11mmobkIf-0czIyEEwgytwqRme5OH8tmZcb4EBcsMZ_w/edit?usp=sharing > - new performance results. > https://docs.google.com/spreadsheets/d/1vmdfsq4WLr92BQO-CGcualE0KhAtjIu3bCaVwYajLo8/edit?usp=sharing > - shows (old) performance results of running system tests on the patched ZK > using the patched system test from > https://issues.apache.org/jira/browse/ZOOKEEPER-2023. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15116273#comment-15116273 ] Hadoop QA commented on ZOOKEEPER-2024: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12784265/ZOOKEEPER-2024.patch against trunk revision 1726354. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 11 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3013//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3013//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3013//console This message is automatically generated. > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > In order to avoid starvation, our solution prioritizes committed write > requests over reads, and enforces fairness among read requests of sessions. > Please see the docs: > 1) > https://docs.google.com/document/d/1oXJiSt9VqL35hCYQRmFuC63ETd0F_g6uApzocgkFe3Y/edit?usp=sharing > - includes a detailed description of the new commit processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) > https://docs.google.com/spreadsheets/d/11mmobkIf-0czIyEEwgytwqRme5OH8tmZcb4EBcsMZ_w/edit?usp=sharing > - new performance results. > https://docs.google.com/spreadsheets/d/1vmdfsq4WLr92BQO-CGcualE0KhAtjIu3bCaVwYajLo8/edit?usp=sharing > - shows (old) performance results of running system tests on the patched ZK > using the patched system test from > https://issues.apache.org/jira/browse/ZOOKEEPER-2023. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182103#comment-14182103 ] Kfir Lev-Ari commented on ZOOKEEPER-2024: - [~tnarg] Nope, FinalRequestProcessor still runs as single writer or multiple readers, without any concurrency between writes and reads. The idea here is to keep everything outside of CommitProcessor as before, and only change the scheduling that is done inside the CommitProcessor, while keeping ZK's semantics. Basically, if we have an outstanding write, we can put it aside until it is committed, along with all of the following requests that are ordered after it. i.e., the requests that belong to the same session or other writes from different sessions. Meaning that only read requests that has no order relation with any pending write can be processed (we placed such read requests in the read ready queue). > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > In order to avoid starvation, our solution prioritizes committed write > requests over reads, and enforces fairness among read requests of sessions. > Please see the docs: > 1) > https://docs.google.com/document/d/1oXJiSt9VqL35hCYQRmFuC63ETd0F_g6uApzocgkFe3Y/edit?usp=sharing > - includes a detailed description of the new commit processor algorithm. > https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) > https://docs.google.com/spreadsheets/d/11mmobkIf-0czIyEEwgytwqRme5OH8tmZcb4EBcsMZ_w/edit?usp=sharing > - new performance results. > https://docs.google.com/spreadsheets/d/1vmdfsq4WLr92BQO-CGcualE0KhAtjIu3bCaVwYajLo8/edit?usp=sharing > - shows (old) performance results of running system tests on the patched ZK > using the patched system test from > https://issues.apache.org/jira/browse/ZOOKEEPER-2023. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14180253#comment-14180253 ] Grant Monroe commented on ZOOKEEPER-2024: - [~kfirlevari] This patch allows writes to be concurrent w/ reads in the FinalRequestProcessor right? Won't this require a ReaderWriterLock in FinalRequestProcessor so that the zxid in the response is consistent w/ what was read? > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > In order to avoid starvation, our solution prioritizes committed write > requests over reads, and enforces fairness among read requests of sessions. > Please see the docs: > 1) > https://docs.google.com/document/d/1oXJiSt9VqL35hCYQRmFuC63ETd0F_g6uApzocgkFe3Y/edit?usp=sharing > - includes a detailed description of the new commit processor algorithm. > https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) > https://docs.google.com/spreadsheets/d/11mmobkIf-0czIyEEwgytwqRme5OH8tmZcb4EBcsMZ_w/edit?usp=sharing > - new performance results. > https://docs.google.com/spreadsheets/d/1vmdfsq4WLr92BQO-CGcualE0KhAtjIu3bCaVwYajLo8/edit?usp=sharing > - shows (old) performance results of running system tests on the patched ZK > using the patched system test from > https://issues.apache.org/jira/browse/ZOOKEEPER-2023. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14150496#comment-14150496 ] Kfir Lev-Ari commented on ZOOKEEPER-2024: - Please see the following doc for updated comparison results (throughput & latency graphs) - https://docs.google.com/document/d/1HXtgdYx1yJE4Bs02-Q4QCRjuAEVH3yOa4wgraSaR0dw/edit?usp=sharing The logs can be found here - https://docs.google.com/spreadsheets/d/1mLZP5FmAXTtgOf60BZkhxOmLzNFrT02wBWNQopx_7Dw/edit?usp=sharing I've modified generate load system test (diff can be found here https://reviews.apache.org/r/25935/) in order to get the latency per each type of client (MW vs RO). > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > In order to avoid starvation, our solution prioritizes committed write > requests over reads, and enforces fairness among read requests of sessions. > Please see the docs: > 1) > https://docs.google.com/document/d/1oXJiSt9VqL35hCYQRmFuC63ETd0F_g6uApzocgkFe3Y/edit?usp=sharing > - includes a detailed description of the new commit processor algorithm. > https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) > https://docs.google.com/spreadsheets/d/11mmobkIf-0czIyEEwgytwqRme5OH8tmZcb4EBcsMZ_w/edit?usp=sharing > - new performance results. > https://docs.google.com/spreadsheets/d/1vmdfsq4WLr92BQO-CGcualE0KhAtjIu3bCaVwYajLo8/edit?usp=sharing > - shows (old) performance results of running system tests on the patched ZK > using the patched system test from > https://issues.apache.org/jira/browse/ZOOKEEPER-2023. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14129844#comment-14129844 ] Hadoop QA commented on ZOOKEEPER-2024: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12667975/ZOOKEEPER-2024.patch against trunk revision 1623916. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 11 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2330//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2330//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2330//console This message is automatically generated. > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > In order to avoid starvation, our solution prioritizes committed write > requests over reads, and enforces fairness among read requests of sessions. > Please see the docs: > 1) > https://docs.google.com/document/d/1oXJiSt9VqL35hCYQRmFuC63ETd0F_g6uApzocgkFe3Y/edit?usp=sharing > - includes a detailed description of the new commit processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) > https://docs.google.com/spreadsheets/d/11mmobkIf-0czIyEEwgytwqRme5OH8tmZcb4EBcsMZ_w/edit?usp=sharing > - new performance results. > https://docs.google.com/spreadsheets/d/1vmdfsq4WLr92BQO-CGcualE0KhAtjIu3bCaVwYajLo8/edit?usp=sharing > - shows (old) performance results of running system tests on the patched ZK > using the patched system test from > https://issues.apache.org/jira/browse/ZOOKEEPER-2023. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14129152#comment-14129152 ] Alexander Shraer commented on ZOOKEEPER-2024: - please also fix the new javac warning: CommitProcessor.java:298: warning: [cast] redundant cast to java.util.Map.Entry> [javac] Map.Entry> sessionPair = (Map.Entry>)it.next(); > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > In order to avoid starvation, our solution prioritizes committed write > requests over reads, and enforces fairness among read requests of sessions. > Please see the docs: > 1) > https://docs.google.com/document/d/1oXJiSt9VqL35hCYQRmFuC63ETd0F_g6uApzocgkFe3Y/edit?usp=sharing > - includes a detailed description of the new commit processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) > https://docs.google.com/spreadsheets/d/11mmobkIf-0czIyEEwgytwqRme5OH8tmZcb4EBcsMZ_w/edit?usp=sharing > - new performance results. > https://docs.google.com/spreadsheets/d/1vmdfsq4WLr92BQO-CGcualE0KhAtjIu3bCaVwYajLo8/edit?usp=sharing > - shows (old) performance results of running system tests on the patched ZK > using the patched system test from > https://issues.apache.org/jira/browse/ZOOKEEPER-2023. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14129134#comment-14129134 ] Hadoop QA commented on ZOOKEEPER-2024: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12667836/ZOOKEEPER-2024.patch against trunk revision 1623916. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 11 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 3 javac compiler warnings (more than the trunk's current 2 warnings). +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2327//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2327//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2327//console This message is automatically generated. > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > In order to avoid starvation, our solution prioritizes committed write > requests over reads, and enforces fairness among read requests of sessions. > Please see the docs: > 1) > https://docs.google.com/document/d/1oXJiSt9VqL35hCYQRmFuC63ETd0F_g6uApzocgkFe3Y/edit?usp=sharing > - includes a detailed description of the new commit processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) > https://docs.google.com/spreadsheets/d/11mmobkIf-0czIyEEwgytwqRme5OH8tmZcb4EBcsMZ_w/edit?usp=sharing > - new performance results. > https://docs.google.com/spreadsheets/d/1vmdfsq4WLr92BQO-CGcualE0KhAtjIu3bCaVwYajLo8/edit?usp=sharing > - shows (old) performance results of running system tests on the patched ZK > using the patched system test from > https://issues.apache.org/jira/browse/ZOOKEEPER-2023. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127319#comment-14127319 ] Alexander Shraer commented on ZOOKEEPER-2024: - Hi Kfir, Thanks for addressing my comments. I added some more for the second test file. The rest looks good to me! Would be great if more people review this. > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > In order to avoid starvation, our solution prioritizes committed write > requests over reads, and enforces fairness among read requests of sessions. > Please see the docs: > 1) > https://docs.google.com/document/d/1oXJiSt9VqL35hCYQRmFuC63ETd0F_g6uApzocgkFe3Y/edit?usp=sharing > - includes a detailed description of the new commit processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) > https://docs.google.com/spreadsheets/d/11mmobkIf-0czIyEEwgytwqRme5OH8tmZcb4EBcsMZ_w/edit?usp=sharing > - new performance results. > https://docs.google.com/spreadsheets/d/1vmdfsq4WLr92BQO-CGcualE0KhAtjIu3bCaVwYajLo8/edit?usp=sharing > - shows (old) performance results of running system tests on the patched ZK > using the patched system test from > https://issues.apache.org/jira/browse/ZOOKEEPER-2023. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14126796#comment-14126796 ] Hadoop QA commented on ZOOKEEPER-2024: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12667376/ZOOKEEPER-2024.patch against trunk revision 1621313. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 11 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 3 javac compiler warnings (more than the trunk's current 2 warnings). +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2318//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2318//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2318//console This message is automatically generated. > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > In order to avoid starvation, our solution prioritizes committed write > requests over reads, and enforces fairness among read requests of sessions. > Please see the docs: > 1) > https://docs.google.com/document/d/1oXJiSt9VqL35hCYQRmFuC63ETd0F_g6uApzocgkFe3Y/edit?usp=sharing > - includes a detailed description of the new commit processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) > https://docs.google.com/spreadsheets/d/11mmobkIf-0czIyEEwgytwqRme5OH8tmZcb4EBcsMZ_w/edit?usp=sharing > - new performance results. > https://docs.google.com/spreadsheets/d/1vmdfsq4WLr92BQO-CGcualE0KhAtjIu3bCaVwYajLo8/edit?usp=sharing > - shows (old) performance results of running system tests on the patched ZK > using the patched system test from > https://issues.apache.org/jira/browse/ZOOKEEPER-2023. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14125301#comment-14125301 ] Hadoop QA commented on ZOOKEEPER-2024: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12667138/ZOOKEEPER-2024.patch against trunk revision 1621313. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 11 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 3 javac compiler warnings (more than the trunk's current 2 warnings). +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2317//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2317//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2317//console This message is automatically generated. > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > In order to avoid starvation, our solution prioritizes committed write > requests over reads, and enforces fairness among read requests of sessions. > Please see the docs: > 1) > https://docs.google.com/document/d/1oXJiSt9VqL35hCYQRmFuC63ETd0F_g6uApzocgkFe3Y/edit?usp=sharing > - includes a detailed description of the new commit processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) > https://docs.google.com/spreadsheets/d/11mmobkIf-0czIyEEwgytwqRme5OH8tmZcb4EBcsMZ_w/edit?usp=sharing > - new performance results. > https://docs.google.com/spreadsheets/d/1vmdfsq4WLr92BQO-CGcualE0KhAtjIu3bCaVwYajLo8/edit?usp=sharing > - shows (old) performance results of running system tests on the patched ZK > using the patched system test from > https://issues.apache.org/jira/browse/ZOOKEEPER-2023. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14125281#comment-14125281 ] Kfir Lev-Ari commented on ZOOKEEPER-2024: - Hi Alex, thanks. Please verify the updated version ;) > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > In order to avoid starvation, our solution prioritizes committed write > requests over reads, and enforces fairness among read requests of sessions. > Please see the docs: > 1) > https://docs.google.com/document/d/1oXJiSt9VqL35hCYQRmFuC63ETd0F_g6uApzocgkFe3Y/edit?usp=sharing > - includes a detailed description of the new commit processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) > https://docs.google.com/spreadsheets/d/11mmobkIf-0czIyEEwgytwqRme5OH8tmZcb4EBcsMZ_w/edit?usp=sharing > - new performance results. > https://docs.google.com/spreadsheets/d/1vmdfsq4WLr92BQO-CGcualE0KhAtjIu3bCaVwYajLo8/edit?usp=sharing > - shows (old) performance results of running system tests on the patched ZK > using the patched system test from > https://issues.apache.org/jira/browse/ZOOKEEPER-2023. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14125067#comment-14125067 ] Alexander Shraer commented on ZOOKEEPER-2024: - Hi Kfir, I reviewed the changes to CommitProcessor, great job! Mostly had minor comments. In general, please add more comments to the code. Maybe also make some bigger comments based on your doc that can help explain the flow. In the reviewboard all comments are marked as open, even those you've fixed - please change the status of a comment once you address it. I didn't read the tests yet, but intend to do it soon. Thanks, Alex > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > In order to avoid starvation, our solution prioritizes committed write > requests over reads, and enforces fairness among read requests of sessions. > Please see the docs: > 1) > https://docs.google.com/document/d/1oXJiSt9VqL35hCYQRmFuC63ETd0F_g6uApzocgkFe3Y/edit?usp=sharing > - includes a detailed description of the new commit processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) > https://docs.google.com/spreadsheets/d/11mmobkIf-0czIyEEwgytwqRme5OH8tmZcb4EBcsMZ_w/edit?usp=sharing > - new performance results. > https://docs.google.com/spreadsheets/d/1vmdfsq4WLr92BQO-CGcualE0KhAtjIu3bCaVwYajLo8/edit?usp=sharing > - shows (old) performance results of running system tests on the patched ZK > using the patched system test from > https://issues.apache.org/jira/browse/ZOOKEEPER-2023. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14121461#comment-14121461 ] Hadoop QA commented on ZOOKEEPER-2024: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12666492/ZOOKEEPER-2024.patch against trunk revision 1621313. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 11 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 3 javac compiler warnings (more than the trunk's current 2 warnings). +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2312//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2312//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2312//console This message is automatically generated. > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch, > ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > In order to avoid starvation, our solution prioritizes committed write > requests over reads, and enforces fairness among read requests of sessions. > Please see the docs: > 1) > https://docs.google.com/document/d/1oXJiSt9VqL35hCYQRmFuC63ETd0F_g6uApzocgkFe3Y/edit?usp=sharing > - includes a detailed description of the new commit processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) > https://docs.google.com/spreadsheets/d/11mmobkIf-0czIyEEwgytwqRme5OH8tmZcb4EBcsMZ_w/edit?usp=sharing > - new performance results. > https://docs.google.com/spreadsheets/d/1vmdfsq4WLr92BQO-CGcualE0KhAtjIu3bCaVwYajLo8/edit?usp=sharing > - shows (old) performance results of running system tests on the patched ZK > using the patched system test from > https://issues.apache.org/jira/browse/ZOOKEEPER-2023. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14120258#comment-14120258 ] Grant Monroe commented on ZOOKEEPER-2024: - [~hdeng] I believe your concerns regarding server side buffering and scalability are addressed by the per-connection outstandingLimit, which is set to either 1000 or the value of the zookeeper.globalOutstandingLimit system property. By reading requests off the wire and moving them through the processor pipeline we improve throughput, but the total outstanding requests per connection is bounded. > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > In order to avoid starvation, our solution prioritizes committed write > requests over reads, and enforces fairness among read requests of sessions. > Please see the docs: > 1) > https://docs.google.com/document/d/1oXJiSt9VqL35hCYQRmFuC63ETd0F_g6uApzocgkFe3Y/edit?usp=sharing > - includes a detailed description of the new commit processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) > https://docs.google.com/spreadsheets/d/1vmdfsq4WLr92BQO-CGcualE0KhAtjIu3bCaVwYajLo8/edit?usp=sharing > - shows performance results of running system tests on the patched ZK using > the patched system test from > https://issues.apache.org/jira/browse/ZOOKEEPER-2023. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14119695#comment-14119695 ] Hadoop QA commented on ZOOKEEPER-2024: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12666183/ZOOKEEPER-2024.patch against trunk revision 1621313. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 11 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 3 javac compiler warnings (more than the trunk's current 2 warnings). +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2311//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2311//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2311//console This message is automatically generated. > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > In order to avoid starvation, our solution prioritizes committed write > requests over reads, and enforces fairness among read requests of sessions. > Please see the docs: > 1) > https://docs.google.com/document/d/1oXJiSt9VqL35hCYQRmFuC63ETd0F_g6uApzocgkFe3Y/edit?usp=sharing > - includes a detailed description of the new commit processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) > https://docs.google.com/spreadsheets/d/1vmdfsq4WLr92BQO-CGcualE0KhAtjIu3bCaVwYajLo8/edit?usp=sharing > - shows performance results of running system tests on the patched ZK using > the patched system test from > https://issues.apache.org/jira/browse/ZOOKEEPER-2023. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14119686#comment-14119686 ] Kfir Lev-Ari commented on ZOOKEEPER-2024: - I've uploaded a new patch, according to [~tnarg] remarks. [~hdeng]] - regarding the two issues, I agree with what [~shralex] wrote. The buffering on the server side is basically moving requests from one queue to another, while processing requests that currently wait for no good reason. I'm not sure why you think it limits scalability. Allowing users to set their preferred order sounds like a separate feature. Regarding starvation, I think that the focus should be about making sure that in the suggested solution, every request is eventually processed. In the current ZK, local reads may cause a committed write, (that is, a write that doesn't belong to a local session), to wait forever. > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Attachments: ZOOKEEPER-2024.patch, ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > In order to avoid starvation, our solution prioritizes committed write > requests over reads, and enforces fairness among read requests of sessions. > Please see the docs: > 1) > https://docs.google.com/document/d/1oXJiSt9VqL35hCYQRmFuC63ETd0F_g6uApzocgkFe3Y/edit?usp=sharing > - includes a detailed description of the new commit processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) > https://docs.google.com/spreadsheets/d/1vmdfsq4WLr92BQO-CGcualE0KhAtjIu3bCaVwYajLo8/edit?usp=sharing > - shows performance results of running system tests on the patched ZK using > the patched system test from > https://issues.apache.org/jira/browse/ZOOKEEPER-2023. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14119087#comment-14119087 ] Alexander Shraer commented on ZOOKEEPER-2024: - Hongchao, It seems that what you mean by unaffected requests is different from what Kfir means. In his proposal all client's requests within a sessions are dependent, so a write blocks all further reads from the same session. This is the current ZK semantics. IMO your proposal to handle it on the client side will either change semantics or alternatively will be worse, not better, for performance. > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Attachments: ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > In order to avoid starvation, our solution prioritizes committed write > requests over reads, and enforces fairness among read requests of sessions. > Please see the docs: > 1) > https://docs.google.com/document/d/1oXJiSt9VqL35hCYQRmFuC63ETd0F_g6uApzocgkFe3Y/edit?usp=sharing > - includes a detailed description of the new commit processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) > https://docs.google.com/spreadsheets/d/1vmdfsq4WLr92BQO-CGcualE0KhAtjIu3bCaVwYajLo8/edit?usp=sharing > - shows performance results of running system tests on the patched ZK using > the patched system test from > https://issues.apache.org/jira/browse/ZOOKEEPER-2023. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14119077#comment-14119077 ] Hongchao Deng commented on ZOOKEEPER-2024: -- Okay. I am feeling more clear what I wanna say. Let me summarize it. h3. Stalling I totally agree that unaffected requests should not block each other. But this seems to be specific user activity optimizations. Even if user doesn't wanna handle it, we can still handle in client code. So why do buffering on server side. The server approach means more workload on server side and limits scalability. h3. Starvation I think it's good to provide options to enable read or write preferred ordering with a default one being sequential (no preference). Since this is [~kfirlevari] proposal, could you clarify the tradeoff / motivation to push it on server side? And welcome to ask me questions if anything confusing I made. > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Attachments: ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > In order to avoid starvation, our solution prioritizes committed write > requests over reads, and enforces fairness among read requests of sessions. > Please see the docs: > 1) > https://docs.google.com/document/d/1oXJiSt9VqL35hCYQRmFuC63ETd0F_g6uApzocgkFe3Y/edit?usp=sharing > - includes a detailed description of the new commit processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) > https://docs.google.com/spreadsheets/d/1vmdfsq4WLr92BQO-CGcualE0KhAtjIu3bCaVwYajLo8/edit?usp=sharing > - shows performance results of running system tests on the patched ZK using > the patched system test from > https://issues.apache.org/jira/browse/ZOOKEEPER-2023. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14118981#comment-14118981 ] Alexander Shraer commented on ZOOKEEPER-2024: - I see. Well, ZK provides guarantees per-session, so you won't have any guarantees across sessions. Which may be fine if the operations are really independent. But I think your intuition makes sense. Instead of multiple sessions (where you loose all semantics) one could specify relaxed per-operation or per session semantics. I've actually suggested this in the past here: http://wiki.apache.org/hadoop/ZooKeeper/MountRemoteZookeeper The specific ZK property that requires blocking reads after writes of the same session is prefix/fifo of client requests. So even if the requests are concurrent (i.e., one didn't complete before the other started) ZK guarantees that they will complete in invocation order and that if one fails, the ones invoked after it will fail too. IIRC the feedback I got was that users expect this fifo property (its reflects program order) and relaxing it may not be a good idea. But personally I still think that this can be a good optional feature (either per operation or per session). For example, a client's program: write(v, 1) is invoked write(v, 1) completes write(v, 2) is invoked read(v) is invoked currently c must return 2. but perhaps the user could say read(v, "I want sequential consistency without fifo") which could return either 1 or 2. This could improve the throughput further. In any case, something like this is probably out of scope for Kfir's proposal but with Kfir's patch in place it would probably be easier to add in the future. > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Attachments: ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > In order to avoid starvation, our solution prioritizes committed write > requests over reads, and enforces fairness among read requests of sessions. > Please see the docs: > 1) > https://docs.google.com/document/d/1oXJiSt9VqL35hCYQRmFuC63ETd0F_g6uApzocgkFe3Y/edit?usp=sharing > - includes a detailed description of the new commit processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) > https://docs.google.com/spreadsheets/d/1vmdfsq4WLr92BQO-CGcualE0KhAtjIu3bCaVwYajLo8/edit?usp=sharing > - shows performance results of running system tests on the patched ZK using > the patched system test from > https://issues.apache.org/jira/browse/ZOOKEEPER-2023. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14118904#comment-14118904 ] Hongchao Deng commented on ZOOKEEPER-2024: -- bq. imagine a bunch of reads blocked by a write on a client ... A technique I can think of on client side is buffering all requests and then multiplexing those requests into multiple sessions to achieve non-blocking. Nonetheless my point isn't about how to handle heavy read traffic. It is about putting the load on client side as much as possible instead of server side to achieve scalability. This sounds like throughput tuning for user optimized activity -- If a user know it is heavy read and there would be non conflicted requests, should he just create different sessions for those requests to achieve non-blocking? If ZK can help users to take care of it, my intuition goes directly to client side. Right? > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Attachments: ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > In order to avoid starvation, our solution prioritizes committed write > requests over reads, and enforces fairness among read requests of sessions. > Please see the docs: > 1) > https://docs.google.com/document/d/1oXJiSt9VqL35hCYQRmFuC63ETd0F_g6uApzocgkFe3Y/edit?usp=sharing > - includes a detailed description of the new commit processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) > https://docs.google.com/spreadsheets/d/1vmdfsq4WLr92BQO-CGcualE0KhAtjIu3bCaVwYajLo8/edit?usp=sharing > - shows performance results of running system tests on the patched ZK using > the patched system test from > https://issues.apache.org/jira/browse/ZOOKEEPER-2023. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14118819#comment-14118819 ] Alexander Shraer commented on ZOOKEEPER-2024: - Your suggestion is much more different from the current implementation than Kfir's patch. But I also suspect that blocking on the client side would have a performance impact - imagine a bunch of reads blocked by a write on a client - we could have processed the reads immediately after the commit of the write reached the local server, instead the client will first need to get the commit and only then send the reads to the local server, so this is added latency for all those reads. I suspect that by not keeping the server's pipeline full this method would also have a significant impact on throughput... > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Attachments: ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > In order to avoid starvation, our solution prioritizes committed write > requests over reads, and enforces fairness among read requests of sessions. > Please see the docs: > 1) > https://docs.google.com/document/d/1oXJiSt9VqL35hCYQRmFuC63ETd0F_g6uApzocgkFe3Y/edit?usp=sharing > - includes a detailed description of the new commit processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) > https://docs.google.com/spreadsheets/d/1vmdfsq4WLr92BQO-CGcualE0KhAtjIu3bCaVwYajLo8/edit?usp=sharing > - shows performance results of running system tests on the patched ZK using > the patched system test from > https://issues.apache.org/jira/browse/ZOOKEEPER-2023. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14118776#comment-14118776 ] Hongchao Deng commented on ZOOKEEPER-2024: -- bq. Read requests can only be processed if they're not blocked by a previous write in the same session. So we need to be able to check whether the session is stalled waiting for a write to commit or not. Now I recall what I meant by "second type". This is it. So why not put the problem on client side? Clients can buffer all requests and make use of different sessions to serve without "being blocked". The tradeoff here is client side v.s. server side. Isn't pushing the load on client side instead of server side a better method? > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Attachments: ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > In order to avoid starvation, our solution prioritizes committed write > requests over reads, and enforces fairness among read requests of sessions. > Please see the docs: > 1) > https://docs.google.com/document/d/1oXJiSt9VqL35hCYQRmFuC63ETd0F_g6uApzocgkFe3Y/edit?usp=sharing > - includes a detailed description of the new commit processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) > https://docs.google.com/spreadsheets/d/1vmdfsq4WLr92BQO-CGcualE0KhAtjIu3bCaVwYajLo8/edit?usp=sharing > - shows performance results of running system tests on the patched ZK using > the patched system test from > https://issues.apache.org/jira/browse/ZOOKEEPER-2023. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14118741#comment-14118741 ] Alexander Shraer commented on ZOOKEEPER-2024: - Read requests can only be processed if they're not blocked by a previous write in the same session. So we need to be able to check whether the session is stalled waiting for a write to commit or not. The idea is to move pending requests of such blocked sessions into a separate data structure (where they are indexed in a more convenient way) instead of just keeping them in the usual request queue and having to traverse the queue multiple times. > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Attachments: ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > In order to avoid starvation, our solution prioritizes committed write > requests over reads, and enforces fairness among read requests of sessions. > Please see the docs: > 1) > https://docs.google.com/document/d/1oXJiSt9VqL35hCYQRmFuC63ETd0F_g6uApzocgkFe3Y/edit?usp=sharing > - includes a detailed description of the new commit processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) > https://docs.google.com/spreadsheets/d/1vmdfsq4WLr92BQO-CGcualE0KhAtjIu3bCaVwYajLo8/edit?usp=sharing > - shows performance results of running system tests on the patched ZK using > the patched system test from > https://issues.apache.org/jira/browse/ZOOKEEPER-2023. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14118710#comment-14118710 ] Hongchao Deng commented on ZOOKEEPER-2024: -- [~kfirlevari] bq. During that period, the session of that request is stalled, but the other sessions might have requests that do not need to be committed, and therefore can be processed. Agreed with this point. It would be nice to be fixed/improved ASAP. bq. To this end, we add data structures for buffering and managing pending requests of stalled sessions This is my questioned point. It seems to put more load on server and potentially limit scalability. And I doubt if this is necessary. For example, as you mentioned: bq. this severely hampers performance as it does not allow read-only sessions to proceed at faster speed than read-write ones. read requests could be processed directly. Why does it need to buffer other sessions' write requests? So IMO, a buffer-less method seems the best way to handle separate sessions, right? > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari >Assignee: Kfir Lev-Ari > Attachments: ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > In order to avoid starvation, our solution prioritizes committed write > requests over reads, and enforces fairness among read requests of sessions. > Please see the docs: > 1) > https://docs.google.com/document/d/1oXJiSt9VqL35hCYQRmFuC63ETd0F_g6uApzocgkFe3Y/edit?usp=sharing > - includes a detailed description of the new commit processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests (https://reviews.apache.org/r/25160) > 3) > https://docs.google.com/spreadsheets/d/1vmdfsq4WLr92BQO-CGcualE0KhAtjIu3bCaVwYajLo8/edit?usp=sharing > - shows performance results of running system tests on the patched ZK using > the patched system test from > https://issues.apache.org/jira/browse/ZOOKEEPER-2023. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14116403#comment-14116403 ] Kfir Lev-Ari commented on ZOOKEEPER-2024: - [~hdeng]. I didn't understand the second type of separation that you stated - can you please clarify the meaning of "unaffected from each other"? By "stalled session" I referred to the time phase in ZK's algorithm that starts with a proposal that is sent to the leader, and ends when it is returned as committed. During that period, the session of that request is stalled, but the other sessions might have requests that do not need to be committed, and therefore can be processed. I'm not sure how it relates to the client side as you suggested. Regarding the second issue - well, I look at it from a technical point of view - it takes much longer to process committed requests than local read requests, (for each committed request, a proposal is sent to the leader, the leader sends messages to the quorum etc..). So I saw it as the natural choice - give the committed an higher priority once it is ready. Anyway, as you wrote, it is possible to make it configurable, but I'm not sure that I'll call it a bug, because in this solution there is no starvation. > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari > Attachments: ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > In order to avoid starvation, our solution prioritizes committed write > requests over reads, and enforces fairness among read requests of sessions. > Please see the docs: > 1) > https://docs.google.com/document/d/1oXJiSt9VqL35hCYQRmFuC63ETd0F_g6uApzocgkFe3Y/edit?usp=sharing > - includes a detailed description of the new commit processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests > 3) > https://docs.google.com/spreadsheets/d/1vmdfsq4WLr92BQO-CGcualE0KhAtjIu3bCaVwYajLo8/edit?usp=sharing > - shows performance results of running system tests on the patched ZK using > the patched system test from > https://issues.apache.org/jira/browse/ZOOKEEPER-2023. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114409#comment-14114409
]
Hongchao Deng commented on ZOOKEEPER-2024:
--
[~kfirlevari]. Thanks for sharing this novel idea.
I have some questions. Can you help clarify?
{quote}
The main idea is to separate sessions that inherently need to stall in order to
enforce order semantics, from ones that do not need to stall. To this end, we
add data structures for buffering and managing pending requests of stalled
sessions; these requests are moved out of the critical path to these data
structures, allowing continued processing of unaffected sessions.
{quote}
There seems to be two kinds separation: 1. sessions unaffected from each other
and 2. requests in one session unaffected from each other. Right?
Another question, why does it need to buffer "pending requests of stalled
sessions"? More specifically, why on server not client side? I assumed you were
meaning the second type of separation.
{quote}
This occurs due to a bug fix
(https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing
of local read requests before handling any committed write.
In order to avoid starvation, our solution prioritizes committed write requests
over reads, and enforces fairness among read requests of sessions.
{quote}
This sounds like a bug. We shouldn't give any preference to either read or
write. A user should configure on client side to tackle the need of either
heavy write or heavy read. Right?
> Major throughput improvement with mixed workloads
> -
>
> Key: ZOOKEEPER-2024
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024
> Project: ZooKeeper
> Issue Type: Improvement
> Components: quorum, server
>Reporter: Kfir Lev-Ari
> Attachments: ZOOKEEPER-2024.patch
>
>
> The patch is applied to the commit processor, and solves two problems:
> 1. Stalling - once the commit processor encounters a local write request, it
> stalls local processing of all sessions until it receives a commit of that
> request from the leader.
> In mixed workloads, this severely hampers performance as it does not allow
> read-only sessions to proceed at faster speed than read-write ones.
> 2. Starvation - as long as there are read requests to process, older remote
> committed write requests are starved.
> This occurs due to a bug fix
> (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing
> of local read requests before handling any committed write. The problem is
> only manifested under high local read load.
> Our solution solves these two problems. It improves throughput in mixed
> workloads (in our tests, by up to 8x), and reduces latency, especially higher
> percentiles (i.e., slowest requests).
> The main idea is to separate sessions that inherently need to stall in order
> to enforce order semantics, from ones that do not need to stall. To this end,
> we add data structures for buffering and managing pending requests of stalled
> sessions; these requests are moved out of the critical path to these data
> structures, allowing continued processing of unaffected sessions.
> In order to avoid starvation, our solution prioritizes committed write
> requests over reads, and enforces fairness among read requests of sessions.
> Please see the docs:
> 1)
> https://docs.google.com/document/d/1oXJiSt9VqL35hCYQRmFuC63ETd0F_g6uApzocgkFe3Y/edit?usp=sharing
> - includes a detailed description of the new commit processor algorithm.
> 2) The attached patch implements our solution, and a collection of related
> unit tests
> 3)
> https://docs.google.com/spreadsheets/d/1vmdfsq4WLr92BQO-CGcualE0KhAtjIu3bCaVwYajLo8/edit?usp=sharing
> - shows performance results of running system tests on the patched ZK using
> the patched system test from
> https://issues.apache.org/jira/browse/ZOOKEEPER-2023.
> See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114129#comment-14114129 ] Kfir Lev-Ari commented on ZOOKEEPER-2024: - Sure, thanks! ;) https://reviews.apache.org/r/25160 Also, as Alex recommended, cc: [~thawan], [~fpj] > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari > Attachments: ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > In order to avoid starvation, our solution prioritizes committed write > requests over reads, and enforces fairness among read requests of sessions. > Please see the docs: > 1) > https://docs.google.com/document/d/1oXJiSt9VqL35hCYQRmFuC63ETd0F_g6uApzocgkFe3Y/edit?usp=sharing > - includes a detailed description of the new commit processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests > 3) > https://docs.google.com/spreadsheets/d/1vmdfsq4WLr92BQO-CGcualE0KhAtjIu3bCaVwYajLo8/edit?usp=sharing > - shows performance results of running system tests on the patched ZK using > the patched system test from > https://issues.apache.org/jira/browse/ZOOKEEPER-2023. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114084#comment-14114084 ] Raul Gutierrez Segales commented on ZOOKEEPER-2024: --- This looks pretty interesting [~kfirlevari]. Mind creating a reviewboard to streamline the review (given the complexity and length of the patch)? Thanks! > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari > Attachments: ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > In order to avoid starvation, our solution prioritizes committed write > requests over reads, and enforces fairness among read requests of sessions. > Please see the docs: > 1) > https://docs.google.com/document/d/1oXJiSt9VqL35hCYQRmFuC63ETd0F_g6uApzocgkFe3Y/edit?usp=sharing > - includes a detailed description of the new commit processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests > 3) > https://docs.google.com/spreadsheets/d/1vmdfsq4WLr92BQO-CGcualE0KhAtjIu3bCaVwYajLo8/edit?usp=sharing > - shows performance results of running system tests on the patched ZK using > the patched system test from > https://issues.apache.org/jira/browse/ZOOKEEPER-2023. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (ZOOKEEPER-2024) Major throughput improvement with mixed workloads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14113949#comment-14113949 ] Hadoop QA commented on ZOOKEEPER-2024: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12664994/ZOOKEEPER-2024.patch against trunk revision 1620111. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 11 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 3 javac compiler warnings (more than the trunk's current 2 warnings). +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2303//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2303//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2303//console This message is automatically generated. > Major throughput improvement with mixed workloads > - > > Key: ZOOKEEPER-2024 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2024 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum, server >Reporter: Kfir Lev-Ari > Attachments: ZOOKEEPER-2024.patch > > > The patch is applied to the commit processor, and solves two problems: > 1. Stalling - once the commit processor encounters a local write request, it > stalls local processing of all sessions until it receives a commit of that > request from the leader. > In mixed workloads, this severely hampers performance as it does not allow > read-only sessions to proceed at faster speed than read-write ones. > 2. Starvation - as long as there are read requests to process, older remote > committed write requests are starved. > This occurs due to a bug fix > (https://issues.apache.org/jira/browse/ZOOKEEPER-1505) that forces processing > of local read requests before handling any committed write. The problem is > only manifested under high local read load. > Our solution solves these two problems. It improves throughput in mixed > workloads (in our tests, by up to 8x), and reduces latency, especially higher > percentiles (i.e., slowest requests). > The main idea is to separate sessions that inherently need to stall in order > to enforce order semantics, from ones that do not need to stall. To this end, > we add data structures for buffering and managing pending requests of stalled > sessions; these requests are moved out of the critical path to these data > structures, allowing continued processing of unaffected sessions. > In order to avoid starvation, our solution prioritizes committed write > requests over reads, and enforces fairness among read requests of sessions. > Please see the docs: > 1) > https://docs.google.com/document/d/1oXJiSt9VqL35hCYQRmFuC63ETd0F_g6uApzocgkFe3Y/edit?usp=sharing > - includes a detailed description of the new commit processor algorithm. > 2) The attached patch implements our solution, and a collection of related > unit tests > 3) > https://docs.google.com/spreadsheets/d/1vmdfsq4WLr92BQO-CGcualE0KhAtjIu3bCaVwYajLo8/edit?usp=sharing > - shows performance results of running system tests on the patched ZK using > the patched system test from > https://issues.apache.org/jira/browse/ZOOKEEPER-2023. > See also https://issues.apache.org/jira/browse/ZOOKEEPER-1609 -- This message was sent by Atlassian JIRA (v6.2#6252)
