[ https://issues.apache.org/jira/browse/ZOOKEEPER-2684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15866318#comment-15866318 ]
Ryan Zhang commented on ZOOKEEPER-2684: --------------------------------------- Unfortunately, this is only half of the story. I tested the build and see the following crash atla-avu-37-sr1.prod.twttr.net: 2017-02-14 15:04:06,180 - ERROR [CommitProcessor:5] - Severe unrecoverable error, from thread : CommitProcessor:5 atla-avu-37-sr1.prod.twttr.net: org.apache.zookeeper.KeeperException$RuntimeInconsistencyException: KeeperErrorCode = RuntimeInconsistency atla-avu-37-sr1.prod.twttr.net: at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:276) atla-cbn-19-sr1.prod.twttr.net: atla-cbn-19-sr1.prod.twttr.net: 2017-02-14 17:55:04,020 - ERROR [CommitProcessor:3] - Severe unrecoverable error, from thread : CommitProcessor:3 atla-cbn-19-sr1.prod.twttr.net: org.apache.zookeeper.KeeperException$RuntimeInconsistencyException: KeeperErrorCode = RuntimeInconsistency atla-cbn-19-sr1.prod.twttr.net: at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:276) atla-cbn-19-sr1.prod.twttr.net: 2017-02-14 18:02:12,019 - ERROR [CommitProcessor:3] - Got request sessionid:0x30069d48d4909da type:error cxid:0x9ca zxid:0x38204ef60ad txntype:-1 reqpath:n/a atla-cbn-19-sr1.prod.twttr.net: but we are expecting request sessionid:0x30069d48d4909da type:create cxid:0x9c9 zxid:0xfffffffffffffffe txntype:unknown reqpath:/loadgen/load_ atla-cbn-19-sr1.prod.twttr.net: atla-cbn-19-sr1.prod.twttr.net: 2017-02-14 18:02:12,019 - ERROR [CommitProcessor:3] - Severe unrecoverable error, from thread : CommitProcessor:3 atla-cbn-19-sr1.prod.twttr.net: org.apache.zookeeper.KeeperException$RuntimeInconsistencyException: KeeperErrorCode = RuntimeInconsistency atla-cbn-19-sr1.prod.twttr.net: at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:276) atla-cbo-26-sr4.prod.twttr.net: atla-cbo-26-sr4.prod.twttr.net: 2017-02-14 17:55:42,323 - ERROR [CommitProcessor:4] - Severe unrecoverable error, from thread : CommitProcessor:4 atla-cbo-26-sr4.prod.twttr.net: org.apache.zookeeper.KeeperException$RuntimeInconsistencyException: KeeperErrorCode = RuntimeInconsistency atla-cbo-26-sr4.prod.twttr.net: at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:276) atla-cbo-26-sr4.prod.twttr.net: 2017-02-14 18:12:25,515 - ERROR [CommitProcessor:4] - Got request sessionid:0x40069edcb5b7b7f type:create cxid:0x36 zxid:0x38205009d26 txntype:1 reqpath:n/a atla-cbo-26-sr4.prod.twttr.net: but we are expecting request sessionid:0x40069edcb5b7b7f type:delete cxid:0x30 zxid:0xfffffffffffffffe txntype:unknown reqpath:/loadgen/load_1759491424 atla-cbo-26-sr4.prod.twttr.net: atla-cbo-26-sr4.prod.twttr.net: 2017-02-14 18:12:25,516 - ERROR [CommitProcessor:4] - Severe unrecoverable error, from thread : CommitProcessor:4 atla-cbo-26-sr4.prod.twttr.net: org.apache.zookeeper.KeeperException$RuntimeInconsistencyException: KeeperErrorCode = RuntimeInconsistency atla-cbo-26-sr4.prod.twttr.net: at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:276) atla-cbs-11-sr2.prod.twttr.net: atla-cbs-11-sr2.prod.twttr.net: 2017-02-14 17:58:06,370 - ERROR [CommitProcessor:2] - Severe unrecoverable error, from thread : CommitProcessor:2 atla-cbs-11-sr2.prod.twttr.net: org.apache.zookeeper.KeeperException$RuntimeInconsistencyException: KeeperErrorCode = RuntimeInconsistency atla-cbs-11-sr2.prod.twttr.net: at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:276) atla-cbs-11-sr2.prod.twttr.net: 2017-02-14 18:09:41,178 - ERROR [CommitProcessor:2] - Got request sessionid:0x20069efe2da8afa type:error cxid:0x295 zxid:0x38204fc2277 txntype:-1 reqpath:n/a atla-cbs-11-sr2.prod.twttr.net: but we are expecting request sessionid:0x20069efe2da8afa type:delete cxid:0x291 zxid:0xfffffffffffffffe txntype:unknown reqpath:/loadgen/load_1759352322 atla-cbs-11-sr2.prod.twttr.net: atla-cbs-11-sr2.prod.twttr.net: 2017-02-14 18:09:41,178 - ERROR [CommitProcessor:2] - Severe unrecoverable error, from thread : CommitProcessor:2 atla-cbs-11-sr2.prod.twttr.net: org.apache.zookeeper.KeeperException$RuntimeInconsistencyException: KeeperErrorCode = RuntimeInconsistency atla-cbs-11-sr2.prod.twttr.net: at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:276) atla-cdc-11-sr2.prod.twttr.net: atla-cdc-11-sr2.prod.twttr.net: 2017-02-14 17:50:49,359 - ERROR [CommitProcessor:1] - Severe unrecoverable error, from thread : CommitProcessor:1 atla-cdc-11-sr2.prod.twttr.net: org.apache.zookeeper.KeeperException$RuntimeInconsistencyException: KeeperErrorCode = RuntimeInconsistency atla-cdc-11-sr2.prod.twttr.net: at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:276) atla-cdc-11-sr2.prod.twttr.net: 2017-02-14 18:03:46,193 - ERROR [CommitProcessor:1] - Got request sessionid:0x10038835caa7017 type:create cxid:0x21 zxid:0x38204f21112 txntype:1 reqpath:n/a atla-cdc-11-sr2.prod.twttr.net: but we are expecting request sessionid:0x10038835caa7017 type:delete cxid:0x1d zxid:0xfffffffffffffffe txntype:unknown reqpath:/loadgen/load_1758999588 Stay tuned, I am adding back the verbose logs > Fix a crashing bug in the mixed workloads commit processor > ---------------------------------------------------------- > > Key: ZOOKEEPER-2684 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2684 > Project: ZooKeeper > Issue Type: Bug > Components: server > Affects Versions: 3.6.0 > Environment: with pretty heavy load on a real cluster > Reporter: Ryan Zhang > Assignee: Ryan Zhang > Priority: Blocker > Attachments: ZOOKEEPER-2684.patch > > > We deployed our build with ZOOKEEPER-2024 and it quickly started to crash > with the following error > atla-buh-05-sr1.prod.twttr.net: 2017-01-18 22:24:42,305 - ERROR > [CommitProcessor:2] > -org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:268) > – Got cxid 0x119fa expected 0x11fc5 for client session id 1009079ba470055 > atla-buh-05-sr1.prod.twttr.net: 2017-01-18 22:32:04,746 - ERROR > [CommitProcessor:2] > -org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:268) > – Got cxid 0x698 expected 0x928 for client session id 4002eeb3fd0009d > atla-buh-05-sr1.prod.twttr.net: 2017-01-18 22:34:46,648 - ERROR > [CommitProcessor:2] > -org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:268) > – Got cxid 0x8904 expected 0x8f34 for client session id 51b8905c90251 > atla-buh-05-sr1.prod.twttr.net: 2017-01-18 22:43:46,834 - ERROR > [CommitProcessor:2] > -org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:268) > – Got cxid 0x3a8d expected 0x3ebc for client session id 2051af11af900cc > clearly something is not right in the new commit processor per session queue > implementation. -- This message was sent by Atlassian JIRA (v6.3.15#6346)