Re: unexpected server response ZRUNTIMEINCONSISTENCY

I can't help you with a solution, but we are also seeing this. We had not made the connection to a reconfig causing this, but a quick look at some logs indicates it may be happening to us during a reconfig as well. That gives us a thread to dig more-- we'll update if we find anything. Best, Jare

Re: [jira] [Commented] (ZOOKEEPER-2151) FollowerZookeeperServer has thousands of outstanding requests stuck in CommitProcessor

icate. Thanks for the help! On Tue, Mar 24, 2015 at 9:07 PM, Jared Cantwell (JIRA) wrote: > > [ > https://issues.apache.org/jira/browse/ZOOKEEPER-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14379204#comment-14379204 > ] > >

[jira] [Resolved] (ZOOKEEPER-2151) FollowerZookeeperServer has thousands of outstanding requests stuck in CommitProcessor

[ https://issues.apache.org/jira/browse/ZOOKEEPER-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jared Cantwell resolved ZOOKEEPER-2151. --- Resolution: Duplicate Fix Version/s: 3.5.0 > FollowerZookeeperServer

[jira] [Commented] (ZOOKEEPER-2151) FollowerZookeeperServer has thousands of outstanding requests stuck in CommitProcessor

[ https://issues.apache.org/jira/browse/ZOOKEEPER-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14379204#comment-14379204 ] Jared Cantwell commented on ZOOKEEPER-2151: --- Wow, that looks like

[jira] [Commented] (ZOOKEEPER-2151) FollowerZookeeperServer has thousands of outstanding requests stuck in CommitProcessor

[ https://issues.apache.org/jira/browse/ZOOKEEPER-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14379157#comment-14379157 ] Jared Cantwell commented on ZOOKEEPER-2151: --- Raul Gutierrez Segales:

[jira] [Created] (ZOOKEEPER-2151) FollowerZookeeperServer has thousands of outstanding requests stuck in CommitProcessor

Jared Cantwell created ZOOKEEPER-2151: - Summary: FollowerZookeeperServer has thousands of outstanding requests stuck in CommitProcessor Key: ZOOKEEPER-2151 URL: https://issues.apache.org/jira/browse/ZOOKEEPER

Re: One ensemble node shows massive number of 'Outstanding' requests

t has already passed a COMMIT message to the CommitProcessor for every request that is stuck in the queuedRequests list and nextPending member of CommitProcessor. I guess I should open a JIRA for this at this point? ~Jared On Tue, Mar 24, 2015 at 8:49 AM, Jared Cantwell wrote: > I do not see any

Re: One ensemble node shows massive number of 'Outstanding' requests

m/ml/hotspot-runtime-dev-java/2013-09/msg6.html > > https://bbossola.wordpress.com/2013/09/04/jvm-issue-concurrency-is-affected-by-changing-the-date-of-the-system/ > > Patrick > > > On Mon, Mar 23, 2015 at 5:00 PM, Jared Cantwell > wrote: > > Greetings, > > >

Re: One ensemble node shows massive number of 'Outstanding' requests

Greetings, We just saw this problem again, and this time we were able to capture a core file of the jvm using gdb. I've run it through jstack and jmap to get a heap profile. I can see that the FollowerZookeeperServer has a requestsInProcess member that is ~24K. I can also see that the CommitPro

[jira] [Commented] (ZOOKEEPER-1865) Fix retry logic in Learner.connectToLeader()

[ https://issues.apache.org/jira/browse/ZOOKEEPER-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14299516#comment-14299516 ] Jared Cantwell commented on ZOOKEEPER-1865: --- Are the core unit tests st

[jira] [Updated] (ZOOKEEPER-1865) Fix retry logic in Learner.connectToLeader()

[ https://issues.apache.org/jira/browse/ZOOKEEPER-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jared Cantwell updated ZOOKEEPER-1865: -- Attachment: (was: ZOOKEEPER-1865-nanoTime-noUT.patch) > Fix retry logic

[jira] [Updated] (ZOOKEEPER-1865) Fix retry logic in Learner.connectToLeader()

[ https://issues.apache.org/jira/browse/ZOOKEEPER-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jared Cantwell updated ZOOKEEPER-1865: -- Attachment: (was: ZOOKEEPER-1865-nanoTime-noUT.patch) > Fix retry logic

[jira] [Updated] (ZOOKEEPER-1865) Fix retry logic in Learner.connectToLeader()

[ https://issues.apache.org/jira/browse/ZOOKEEPER-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jared Cantwell updated ZOOKEEPER-1865: -- Attachment: ZOOKEEPER-1865-nanoTime.patch > Fix retry logic

[jira] [Updated] (ZOOKEEPER-1865) Fix retry logic in Learner.connectToLeader()

[ https://issues.apache.org/jira/browse/ZOOKEEPER-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jared Cantwell updated ZOOKEEPER-1865: -- Attachment: ZOOKEEPER-1865-nanoTime-noUT.patch Corrected diff format. > Fix re

[jira] [Commented] (ZOOKEEPER-1865) Fix retry logic in Learner.connectToLeader()

[ https://issues.apache.org/jira/browse/ZOOKEEPER-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14294503#comment-14294503 ] Jared Cantwell commented on ZOOKEEPER-1865: --- Camille, we didn't

[jira] [Updated] (ZOOKEEPER-1865) Fix retry logic in Learner.connectToLeader()

[ https://issues.apache.org/jira/browse/ZOOKEEPER-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jared Cantwell updated ZOOKEEPER-1865: -- Attachment: ZOOKEEPER-1865-nanoTime-noUT.patch Alternate patch using nanoTime

[jira] [Commented] (ZOOKEEPER-1865) Fix retry logic in Learner.connectToLeader()

[ https://issues.apache.org/jira/browse/ZOOKEEPER-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280916#comment-14280916 ] Jared Cantwell commented on ZOOKEEPER-1865: --- We just hit this in

[jira] [Commented] (ZOOKEEPER-1934) Stale data received from sync'd ensemble peer

[ https://issues.apache.org/jira/browse/ZOOKEEPER-1934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14135644#comment-14135644 ] Jared Cantwell commented on ZOOKEEPER-1934: --- I am working with Marshal

[jira] [Commented] (ZOOKEEPER-1096) Leader communication should listen on specified IP, not wildcard address

[ https://issues.apache.org/jira/browse/ZOOKEEPER-1096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13753879#comment-13753879 ] Jared Cantwell commented on ZOOKEEPER-1096: --- It was changed as par

[jira] [Commented] (ZOOKEEPER-1096) Leader communication should listen on specified IP, not wildcard address

[ https://issues.apache.org/jira/browse/ZOOKEEPER-1096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13753820#comment-13753820 ] Jared Cantwell commented on ZOOKEEPER-1096: --- I am simply saying

[jira] [Commented] (ZOOKEEPER-1096) Leader communication should listen on specified IP, not wildcard address

[ https://issues.apache.org/jira/browse/ZOOKEEPER-1096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13753686#comment-13753686 ] Jared Cantwell commented on ZOOKEEPER-1096: --- 3.5.0 solves some of

[jira] [Updated] (ZOOKEEPER-1096) Leader communication should listen on specified IP, not wildcard address

[ https://issues.apache.org/jira/browse/ZOOKEEPER-1096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jared Cantwell updated ZOOKEEPER-1096: -- Attachment: ZOOKEEPER-1096.patch Fixed some prefixes. > Leader communicat

[jira] [Updated] (ZOOKEEPER-1096) Leader communication should listen on specified IP, not wildcard address

[ https://issues.apache.org/jira/browse/ZOOKEEPER-1096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jared Cantwell updated ZOOKEEPER-1096: -- Attachment: ZOOKEEPER-1096.patch > Leader communication should listen on specif

[jira] [Created] (ZOOKEEPER-1096) Leader communication should listen on specified IP, not wildcard address

Issue Type: Improvement Components: server Affects Versions: 3.3.3, 3.4.0 Reporter: Jared Cantwell Priority: Minor Server should specify the local address that is used for leader communication (and not use the default of listening on all interfaces). This

[jira] Commented: (ZOOKEEPER-965) Need a multi-update command to allow multiple znodes to be updated safely

[ https://issues.apache.org/jira/browse/ZOOKEEPER-965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004296#comment-13004296 ] Jared Cantwell commented on ZOOKEEPER-965: -- Hey Ted, Has there been

[jira] Commented: (ZOOKEEPER-882) Startup loads last transaction from snapshot

[ https://issues.apache.org/jira/browse/ZOOKEEPER-882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12972685#action_12972685 ] Jared Cantwell commented on ZOOKEEPER-882: -- Also, somewhat related to

[jira] Commented: (ZOOKEEPER-882) Startup loads last transaction from snapshot

[ https://issues.apache.org/jira/browse/ZOOKEEPER-882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12972667#action_12972667 ] Jared Cantwell commented on ZOOKEEPER-882: -- Flavio, so I figured out what

[jira] Updated: (ZOOKEEPER-882) Startup loads last transaction from snapshot

[ https://issues.apache.org/jira/browse/ZOOKEEPER-882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jared Cantwell updated ZOOKEEPER-882: - Attachment: ZOOKEEPER-882.patch Attaching patch described in previous comment

[jira] Commented: (ZOOKEEPER-882) Startup loads last transaction from snapshot

[ https://issues.apache.org/jira/browse/ZOOKEEPER-882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12972553#action_12972553 ] Jared Cantwell commented on ZOOKEEPER-882: -- Flavio, I hit a wall with th