[jira] [Commented] (ZOOKEEPER-3156) ZOOKEEPER-2184 causes kerberos principal to not have resolved host name
[ https://issues.apache.org/jira/browse/ZOOKEEPER-3156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16632482#comment-16632482 ] Robert Joseph Evans commented on ZOOKEEPER-3156: OK I have a patch up for 3.4 and if it looks good I am happy to port it to all release lines. [https://github.com/apache/zookeeper/pull/648] I have a few questions though. # I have canonicalization on by default, is that what we want? # Is there some place that I can document the new System property that I added? > ZOOKEEPER-2184 causes kerberos principal to not have resolved host name > --- > > Key: ZOOKEEPER-3156 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3156 > Project: ZooKeeper > Issue Type: Bug > Components: java client >Affects Versions: 3.6.0, 3.4.13, 3.5.5 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Prior to ZOOKEEPER-2184 the zookeeper client would canonicalize a configured > host name before creating the SASL client which is used to create the > principal name. After ZOOKEEPER-2184 that canonicalization does not happen > so the principal that the ZK client tries to use when it is configured to > talk to a CName is different between 3.4.13 and all previous versions of ZK. > > For example > > zk1.mycluster.mycompany.com maps to real-node.mycompany.com. > > 3.4.13 will want the server to have > [zookeeper/zk1.mycluster@kdc.mycompany.com|mailto:zookeeper/zk1.mycluster@kdc.mycompany.com] > 3.4.12 wants the server to have > [zookeeper/real-node.mycompany@kdc.mycompany.com|mailto:zookeeper/real-node.mycompany@kdc.mycompany.com] > > This makes 3.4.13 incompatible with many ZK setups currently in existence. > It would be nice to have that resolution be optional because in some cases it > might be nice to have a single principal tied to the cname. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ZOOKEEPER-3156) ZOOKEEPER-2184 causes kerberos principal to not have resolved host name
[
https://issues.apache.org/jira/browse/ZOOKEEPER-3156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16632211#comment-16632211
]
Robert Joseph Evans commented on ZOOKEEPER-3156:
[~andorm],
Sorry I was on a plane all day yesterday so I didn't respond sooner. I have
been doing some debugging trying to get a patch, but I have not come up with
one yet. I originally wanted to see if I could match the behavior from before
by moving the host name resolution code later in the process, but despite the
data structures being the same it is not possible as some information is lost
in the current process. I think having a flag that would optionally call
{{getCannicalHostName}} would be great. If someone else has a patch I am happy
to test it out, otherwise I will put together a patch myself.
Be aware though that a global flag is not 100% backwards compatible, and there
is a very small possibility that a single process will want to talk to multiple
different zookeeper ensembles, i.e. apache storm + older kafka client, which
could run into an issue where each ensemble wants something different. It is a
small enough chance, and there are ways to work around it so I am fine with it,
but it is possible.
> ZOOKEEPER-2184 causes kerberos principal to not have resolved host name
> ---
>
> Key: ZOOKEEPER-3156
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3156
> Project: ZooKeeper
> Issue Type: Bug
> Components: java client
>Affects Versions: 3.6.0, 3.4.13, 3.5.5
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Blocker
>
> Prior to ZOOKEEPER-2184 the zookeeper client would canonicalize a configured
> host name before creating the SASL client which is used to create the
> principal name. After ZOOKEEPER-2184 that canonicalization does not happen
> so the principal that the ZK client tries to use when it is configured to
> talk to a CName is different between 3.4.13 and all previous versions of ZK.
>
> For example
>
> zk1.mycluster.mycompany.com maps to real-node.mycompany.com.
>
> 3.4.13 will want the server to have
> [zookeeper/zk1.mycluster@kdc.mycompany.com|mailto:zookeeper/zk1.mycluster@kdc.mycompany.com]
> 3.4.12 wants the server to have
> [zookeeper/real-node.mycompany@kdc.mycompany.com|mailto:zookeeper/real-node.mycompany@kdc.mycompany.com]
>
> This makes 3.4.13 incompatible with many ZK setups currently in existence.
> It would be nice to have that resolution be optional because in some cases it
> might be nice to have a single principal tied to the cname.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ZOOKEEPER-3156) ZOOKEEPER-2184 causes kerberos principal to not have resolved host name
[ https://issues.apache.org/jira/browse/ZOOKEEPER-3156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated ZOOKEEPER-3156: --- Affects Version/s: 3.5.5 3.6.0 > ZOOKEEPER-2184 causes kerberos principal to not have resolved host name > --- > > Key: ZOOKEEPER-3156 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3156 > Project: ZooKeeper > Issue Type: Bug > Components: java client >Affects Versions: 3.6.0, 3.4.13, 3.5.5 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > > Prior to ZOOKEEPER-2184 the zookeeper client would canonicalize a configured > host name before creating the SASL client which is used to create the > principal name. After ZOOKEEPER-2184 that canonicalization does not happen > so the principal that the ZK client tries to use when it is configured to > talk to a CName is different between 3.4.13 and all previous versions of ZK. > > For example > > zk1.mycluster.mycompany.com maps to real-node.mycompany.com. > > 3.4.13 will want the server to have > [zookeeper/zk1.mycluster@kdc.mycompany.com|mailto:zookeeper/zk1.mycluster@kdc.mycompany.com] > 3.4.12 wants the server to have > [zookeeper/real-node.mycompany@kdc.mycompany.com|mailto:zookeeper/real-node.mycompany@kdc.mycompany.com] > > This makes 3.4.13 incompatible with many ZK setups currently in existence. > It would be nice to have that resolution be optional because in some cases it > might be nice to have a single principal tied to the cname. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ZOOKEEPER-3156) ZOOKEEPER-2184 causes kerberos principal to not have resolved host name
Robert Joseph Evans created ZOOKEEPER-3156: -- Summary: ZOOKEEPER-2184 causes kerberos principal to not have resolved host name Key: ZOOKEEPER-3156 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3156 Project: ZooKeeper Issue Type: Bug Components: java client Affects Versions: 3.4.13 Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans Prior to ZOOKEEPER-2184 the zookeeper client would canonicalize a configured host name before creating the SASL client which is used to create the principal name. After ZOOKEEPER-2184 that canonicalization does not happen so the principal that the ZK client tries to use when it is configured to talk to a CName is different between 3.4.13 and all previous versions of ZK. For example zk1.mycluster.mycompany.com maps to real-node.mycompany.com. 3.4.13 will want the server to have [zookeeper/zk1.mycluster@kdc.mycompany.com|mailto:zookeeper/zk1.mycluster@kdc.mycompany.com] 3.4.12 wants the server to have [zookeeper/real-node.mycompany@kdc.mycompany.com|mailto:zookeeper/real-node.mycompany@kdc.mycompany.com] This makes 3.4.13 incompatible with many ZK setups currently in existence. It would be nice to have that resolution be optional because in some cases it might be nice to have a single principal tied to the cname. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ZOOKEEPER-3092) Pluggable metrics system for ZooKeeper
[ https://issues.apache.org/jira/browse/ZOOKEEPER-3092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16547844#comment-16547844 ] Robert Joseph Evans commented on ZOOKEEPER-3092: +1 This is something that I had a customer ask for just yesterday. I love the service loader idea, or something that can keep the classpath unchanged unless someone wants to use it. One thing to keep in mind too is that a lot of metrics systems, like codahale/dropwizard, do not support dimensions. It is especially helpful if you want to do something slightly more advanced like showing a breakdown of load by user or connections by the host they are coming from. This is not something I think we have to have up front, but there are use cases where it can be very helpful. Flink forked dropwizard to be able to support them. Hadoop metrics2 has this in there too as context. For storm this is something that I am battling with right now because we opted for just using pure dropwizard and are regretting it. Please learn from my mistakes. > Pluggable metrics system for ZooKeeper > -- > > Key: ZOOKEEPER-3092 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3092 > Project: ZooKeeper > Issue Type: New Feature > Components: metric system >Reporter: Michael Han >Priority: Major > > ZooKeeper should provide a pluggable metrics system such that various metrics > can be collected and reported using different approaches that fit production > monitoring / alert / debugging needs. > Historically ZooKeeper provides four letter words and JMX which exposes > certain stats / metrics but they are not very flexible in terms of > programmatically accessing metrics and connecting metrics to different > reporting systems. > There are other projects that's already doing this which can be used for > reference, such as bookkeeper metrics service providers and hadoop metrics2. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ZOOKEEPER-2845) Data inconsistency issue due to retain database in leader election
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362234#comment-16362234 ] Robert Joseph Evans commented on ZOOKEEPER-2845: [~lvfangmin], Thanks for pushing on this. I had missed an error case in the follower. I have updated the patch to hopefully fix all of the issues, but please have a look at it. > Data inconsistency issue due to retain database in leader election > -- > > Key: ZOOKEEPER-2845 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2845 > Project: ZooKeeper > Issue Type: Bug > Components: quorum >Affects Versions: 3.4.10, 3.5.3, 3.6.0 >Reporter: Fangmin Lv >Assignee: Robert Joseph Evans >Priority: Critical > > In ZOOKEEPER-2678, the ZKDatabase is retained to reduce the unavailable time > during leader election. In ZooKeeper ensemble, it's possible that the > snapshot is ahead of txn file (due to slow disk on the server, etc), or the > txn file is ahead of snapshot due to no commit message being received yet. > If snapshot is ahead of txn file, since the SyncRequestProcessor queue will > be drained during shutdown, the snapshot and txn file will keep consistent > before leader election happening, so this is not an issue. > But if txn is ahead of snapshot, it's possible that the ensemble will have > data inconsistent issue, here is the simplified scenario to show the issue: > Let's say we have a 3 servers in the ensemble, server A and B are followers, > and C is leader, and all the snapshot and txn are up to T0: > 1. A new request reached to leader C to create Node N, and it's converted to > txn T1 > 2. Txn T1 was synced to disk in C, but just before the proposal reaching out > to the followers, A and B restarted, so the T1 didn't exist in A and B > 3. A and B formed a new quorum after restart, let's say B is the leader > 4. C changed to looking state due to no enough followers, it will sync with > leader B with last Zxid T0, which will have an empty diff sync > 5. Before C take snapshot it restarted, it replayed the txns on disk which > includes T1, now it will have Node N, but A and B doesn't have it. > Also I included the a test case to reproduce this issue consistently. > We have a totally different RetainDB version which will avoid this issue by > doing consensus between snapshot and txn files before leader election, will > submit for review. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ZOOKEEPER-2845) Data inconsistency issue due to retain database in leader election
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361234#comment-16361234 ] Robert Joseph Evans commented on ZOOKEEPER-2845: [~lvfangmin], You are right I did miss the ID changing on the reload as part of my tests. I will spend some more time debugging. My patch does fix the test case that was uploaded, but I want to be sure I understand the issue well enough to see what situations might not be fixed by it. > Data inconsistency issue due to retain database in leader election > -- > > Key: ZOOKEEPER-2845 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2845 > Project: ZooKeeper > Issue Type: Bug > Components: quorum >Affects Versions: 3.4.10, 3.5.3, 3.6.0 >Reporter: Fangmin Lv >Assignee: Robert Joseph Evans >Priority: Critical > > In ZOOKEEPER-2678, the ZKDatabase is retained to reduce the unavailable time > during leader election. In ZooKeeper ensemble, it's possible that the > snapshot is ahead of txn file (due to slow disk on the server, etc), or the > txn file is ahead of snapshot due to no commit message being received yet. > If snapshot is ahead of txn file, since the SyncRequestProcessor queue will > be drained during shutdown, the snapshot and txn file will keep consistent > before leader election happening, so this is not an issue. > But if txn is ahead of snapshot, it's possible that the ensemble will have > data inconsistent issue, here is the simplified scenario to show the issue: > Let's say we have a 3 servers in the ensemble, server A and B are followers, > and C is leader, and all the snapshot and txn are up to T0: > 1. A new request reached to leader C to create Node N, and it's converted to > txn T1 > 2. Txn T1 was synced to disk in C, but just before the proposal reaching out > to the followers, A and B restarted, so the T1 didn't exist in A and B > 3. A and B formed a new quorum after restart, let's say B is the leader > 4. C changed to looking state due to no enough followers, it will sync with > leader B with last Zxid T0, which will have an empty diff sync > 5. Before C take snapshot it restarted, it replayed the txns on disk which > includes T1, now it will have Node N, but A and B doesn't have it. > Also I included the a test case to reproduce this issue consistently. > We have a totally different RetainDB version which will avoid this issue by > doing consensus between snapshot and txn files before leader election, will > submit for review. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ZOOKEEPER-2845) Data inconsistency issue due to retain database in leader election
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16360880#comment-16360880 ] Robert Joseph Evans commented on ZOOKEEPER-2845: [~lvfangmin], I will spend some more time debugging it because I could have made a mistake, but that is not what I saw from the unit test you provided. When I logged the zxid used for leader election both before and after clearing the DB it didn't change, but like I said I could have missed something and I am not a regular contributor so I will go back and try to do it again. > Data inconsistency issue due to retain database in leader election > -- > > Key: ZOOKEEPER-2845 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2845 > Project: ZooKeeper > Issue Type: Bug > Components: quorum >Affects Versions: 3.4.10, 3.5.3, 3.6.0 >Reporter: Fangmin Lv >Assignee: Robert Joseph Evans >Priority: Critical > > In ZOOKEEPER-2678, the ZKDatabase is retained to reduce the unavailable time > during leader election. In ZooKeeper ensemble, it's possible that the > snapshot is ahead of txn file (due to slow disk on the server, etc), or the > txn file is ahead of snapshot due to no commit message being received yet. > If snapshot is ahead of txn file, since the SyncRequestProcessor queue will > be drained during shutdown, the snapshot and txn file will keep consistent > before leader election happening, so this is not an issue. > But if txn is ahead of snapshot, it's possible that the ensemble will have > data inconsistent issue, here is the simplified scenario to show the issue: > Let's say we have a 3 servers in the ensemble, server A and B are followers, > and C is leader, and all the snapshot and txn are up to T0: > 1. A new request reached to leader C to create Node N, and it's converted to > txn T1 > 2. Txn T1 was synced to disk in C, but just before the proposal reaching out > to the followers, A and B restarted, so the T1 didn't exist in A and B > 3. A and B formed a new quorum after restart, let's say B is the leader > 4. C changed to looking state due to no enough followers, it will sync with > leader B with last Zxid T0, which will have an empty diff sync > 5. Before C take snapshot it restarted, it replayed the txns on disk which > includes T1, now it will have Node N, but A and B doesn't have it. > Also I included the a test case to reproduce this issue consistently. > We have a totally different RetainDB version which will avoid this issue by > doing consensus between snapshot and txn files before leader election, will > submit for review. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ZOOKEEPER-2845) Data inconsistency issue due to retain database in leader election
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16358899#comment-16358899 ] Robert Joseph Evans commented on ZOOKEEPER-2845: [~lvfangmin], So how does clearing the DB prevent it from re-applying the transactions in the transaction log? > Data inconsistency issue due to retain database in leader election > -- > > Key: ZOOKEEPER-2845 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2845 > Project: ZooKeeper > Issue Type: Bug > Components: quorum >Affects Versions: 3.4.10, 3.5.3, 3.6.0 >Reporter: Fangmin Lv >Assignee: Robert Joseph Evans >Priority: Critical > > In ZOOKEEPER-2678, the ZKDatabase is retained to reduce the unavailable time > during leader election. In ZooKeeper ensemble, it's possible that the > snapshot is ahead of txn file (due to slow disk on the server, etc), or the > txn file is ahead of snapshot due to no commit message being received yet. > If snapshot is ahead of txn file, since the SyncRequestProcessor queue will > be drained during shutdown, the snapshot and txn file will keep consistent > before leader election happening, so this is not an issue. > But if txn is ahead of snapshot, it's possible that the ensemble will have > data inconsistent issue, here is the simplified scenario to show the issue: > Let's say we have a 3 servers in the ensemble, server A and B are followers, > and C is leader, and all the snapshot and txn are up to T0: > 1. A new request reached to leader C to create Node N, and it's converted to > txn T1 > 2. Txn T1 was synced to disk in C, but just before the proposal reaching out > to the followers, A and B restarted, so the T1 didn't exist in A and B > 3. A and B formed a new quorum after restart, let's say B is the leader > 4. C changed to looking state due to no enough followers, it will sync with > leader B with last Zxid T0, which will have an empty diff sync > 5. Before C take snapshot it restarted, it replayed the txns on disk which > includes T1, now it will have Node N, but A and B doesn't have it. > Also I included the a test case to reproduce this issue consistently. > We have a totally different RetainDB version which will avoid this issue by > doing consensus between snapshot and txn files before leader election, will > submit for review. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ZOOKEEPER-2845) Data inconsistency issue due to retain database in leader election
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16358627#comment-16358627 ] Robert Joseph Evans commented on ZOOKEEPER-2845: [~lvfangmin], Perhaps I don't understand the issue well enough which is totally possible because I am not a frequent contributor and the path for all of the request processors is kind of complex. My understanding is that the SyncRequestProcessor handles writing out edits to the edit log and snapshots, there are a few other places where this happens at startup though. The SyncRequestProcessor writes out edits as they arrive and will flush them to disk periodically in batches. It also takes snapshots periodically. The in memory portion appears to be updated by the FinalRequestProcessor prior to a quorum of acks being received. So yes there is the possibility that something is written to the transaction log that is not applied to memory. This means that when ZKDatabase.clear() is called it should actually fast forward the in memory changes to match those in the edit log + snapshot. So you are saying that 1) proposals come in, are written to the transaction log, but the in memory database is not updated yet. 2) the server does a soft restart for some reason and some transactions appear to be lost (because the in memory DB was not fast forwarded). 3) more transactions come in (possibly conflicting with the first set of transactions). 4) before a snapshot can happen the leader or follower restarts and has to reconstruct the in memory DB from edits + snapshot. This would then reapply the edits that originally appeared to be lost. This does look like it might happen, so I will look into that as well. But the test in [https://github.com/apache/zookeeper/pull/310] didn't appear to trigger this. I could be wrong because I concentrated most of my debugging on the original leader and what was happening with it, instead of the followers and what was happening with them. I also didn't understand how clearing the leader's in memory database caused an edit to be lost, if the edits are being written out to disk before the in memory DB is updated. What I saw was that 1) a bunch of edits and leaders/followers being restarted that didn't really do much of anything. 2) the original leader lost a connection to the followers. 3a) A transaction was written to the leader in memory DB but it didn't get a quorum of acks 3b) The followers restarted and formed a new quorum 4) The original leader timed out and joined the new quorum 5) As part of the sync when the old leader joined the new quorum it got a diff (not a snap), but it had an edit that was not a part of the new leader so it was off from the others. I could see this second part happening even without my change so I don't really understand how that clearing the database would prevent it. My thinking was that it was a race condition where the edits in the edit log were not flushed yet, and as such when we cleared the DB they were lost. But I didn't confirm this. > Data inconsistency issue due to retain database in leader election > -- > > Key: ZOOKEEPER-2845 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2845 > Project: ZooKeeper > Issue Type: Bug > Components: quorum >Affects Versions: 3.4.10, 3.5.3, 3.6.0 >Reporter: Fangmin Lv >Assignee: Robert Joseph Evans >Priority: Critical > > In ZOOKEEPER-2678, the ZKDatabase is retained to reduce the unavailable time > during leader election. In ZooKeeper ensemble, it's possible that the > snapshot is ahead of txn file (due to slow disk on the server, etc), or the > txn file is ahead of snapshot due to no commit message being received yet. > If snapshot is ahead of txn file, since the SyncRequestProcessor queue will > be drained during shutdown, the snapshot and txn file will keep consistent > before leader election happening, so this is not an issue. > But if txn is ahead of snapshot, it's possible that the ensemble will have > data inconsistent issue, here is the simplified scenario to show the issue: > Let's say we have a 3 servers in the ensemble, server A and B are followers, > and C is leader, and all the snapshot and txn are up to T0: > 1. A new request reached to leader C to create Node N, and it's converted to > txn T1 > 2. Txn T1 was synced to disk in C, but just before the proposal reaching out > to the followers, A and B restarted, so the T1 didn't exist in A and B > 3. A and B formed a new quorum after restart, let's say B is the leader > 4. C changed to looking state due to no enough followers, it will sync with > leader B with last Zxid T0, which will have an empty diff sync > 5. Before C take snapshot it restarted, it
[jira] [Comment Edited] (ZOOKEEPER-2845) Data inconsistency issue due to retain database in leader election
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16347026#comment-16347026
]
Robert Joseph Evans edited comment on ZOOKEEPER-2845 at 1/31/18 3:39 PM:
-
I have a fix that I will be posting shortly. I need to clean up the patch and
make sure that I get pull requests ready for all of the branches that
ZOOKEEPER-2926 went into.
The following table describes the situation that allows a node to get into an
inconsistent state.
|| ||N1||N2||N3||
|Start with cluster in sync N1 is leader|0x0 0x5|0x0 0x5|0x0 0x5|
|N2 and N3 go down|0x0 0x5| | |
|Proposal to N1 (fails with no quorum)|0x0 0x6| | |
|N2 and N3 return, but N1 is restarting. N2 elected leader| |0x1 0x0|0x1 0x0|
|A proposal is accepted| |0x1 0x1|0x1 0x1|
|N1 returns and is trying to sync with the new leader N2|0x0 0x6|0x1 0x1|0x1
0x1|
At this point the code in {{LearnerHandler.syncFollower}} takes over to bring
N1 into sync with N2 the new leader.
That code checks the following in order
# Is there a {{forceSync}}? Not in this case
# Are the two zxids in sync already? No {{0x0 0x6 != 0x1 0x1}}
# is the peer zxid > the local zxid (and peer didn't just rotate to a new
epoch)? No {{0x0 0x6 < 0x1 0x1}}
# is the peer zxid in between the max committed log and the min committed log?
In this case yes it is, but it shouldn't be. The max committed log is {{0x1
0x1}}. The min committed log is {{0x0 0x5}} or something likely below it
because it is based off of distance in the edit log. The issue is that once
the epoch changes, {{0x0}} to {{0x1}}, the leader has no idea if the edits are
in its edit log without explicitly checking for them.
The reason that ZOOKEEPER-2926 exposed this is because previously when a leader
was elected the in memory DB was dropped and everything was reread from disk.
When this happens the {{0x0 0x6}} proposal was lost. But it is not guaranteed
to be lost in all cases. In theory a snapshot could be taken triggered by that
proposal, either on the leader, or on a follower that also received the
proposal, but does not join the new quorum in time. As such ZOOKEEPER-2926
really just extended the window of an already existing race. But it extended
it almost indefinitely so it is much more likely to happen.
My fix is to update {{LearnerHandler.syncFollower}} to only send a {{DIFF}} if
the epochs are the same. If they are not the same we don't know if something
we inserted that we don't know about.
was (Author: revans2):
I have a fix that I will be posting shortly. I need to clean up the patch and
make sure that I get pull requests ready for all of the branches that
ZOOKEEPER-2926 went into.
The following table describes the situation that allows a node to get into an
inconsistent state.
|| ||N1||N2||N3||
|Start with cluster in sync N1 is leader|0x0 0x5|0x0 0x5|0x0 0x5|
|N2 and N3 go down|0x0 0x5| | |
|Proposal to N1 (fails with no quorum)|0x0 0x6| | |
|N2 and N3 return, but N1 is restarting. N2 elected leader| |0x1 0x0|0x1 0x0|
|A proposal is accepted| |0x1 0x1|0x1 0x1|
|N1 returns and is trying to sync with the new leader N2|0x0 0x6|0x1 0x1|0x1
0x1|
At this point the code in {{LearnerHandler.syncFollower}} takes over to bring
N1 into sync with N2 the new leader.
That code checks the following in order
# Is there a {{forceSync}}? Not in this case
# Are the two zxids in sync already? No {{0x0 0x6 != 0x1 0x1}}
# is the peer zxid > the local zxid (and peer didn't just rotate to a new
epoch)? No {{0x0 0x6 < 0x1 0x1}}
# is the peer zxid in between the max committed log and the min committed log?
In this case yes it is, but it shouldn't be. The max committed log is {{0x1
0x1}}. The min committed log is {{0x0 0x5}} or something likely below it
because it is based off of distance in the edit log. The issue is that once
the epoch changes, {{0x0}} to {{0x1}}, the leader has no idea if the edits are
in its edit log without explicitly checking for them.
The reason that ZOOKEEPER-2926 exposed this is because previously when a leader
was elected the in memory DB was dropped and everything was reread from disk.
When this happens the {{0x0 0x6}} proposal was lost. But it is not guaranteed
to be lost in all cases. In theory a snapshot could be taken triggered by that
proposal, either on the leader, or on a follower that also allied the proposal,
but does not join the new quorum in time. As such ZOOKEEPER-2926 really just
extended the window of an already existing race. But it extended it almost
indefinitely so it is much more likely to happen.
My fix is to update {{LearnerHandler.syncFollower}} to only send a {{DIFF}} if
the epochs are the same. If they are not the same we don't know if something
we inserted that we don't know about.
> Data inconsistency issue due to retain
[jira] [Commented] (ZOOKEEPER-2845) Data inconsistency issue due to retain database in leader election
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16347026#comment-16347026
]
Robert Joseph Evans commented on ZOOKEEPER-2845:
I have a fix that I will be posting shortly. I need to clean up the patch and
make sure that I get pull requests ready for all of the branches that
ZOOKEEPER-2926 went into.
The following table describes the situation that allows a node to get into an
inconsistent state.
|| ||N1||N2||N3||
|Start with cluster in sync N1 is leader|0x0 0x5|0x0 0x5|0x0 0x5|
|N2 and N3 go down|0x0 0x5| | |
|Proposal to N1 (fails with no quorum)|0x0 0x6| | |
|N2 and N3 return, but N1 is restarting. N2 elected leader| |0x1 0x0|0x1 0x0|
|A proposal is accepted| |0x1 0x1|0x1 0x1|
|N1 returns and is trying to sync with the new leader N2|0x0 0x6|0x1 0x1|0x1
0x1|
At this point the code in {{LearnerHandler.syncFollower}} takes over to bring
N1 into sync with N2 the new leader.
That code checks the following in order
# Is there a {{forceSync}}? Not in this case
# Are the two zxids in sync already? No {{0x0 0x6 != 0x1 0x1}}
# is the peer zxid > the local zxid (and peer didn't just rotate to a new
epoch)? No {{0x0 0x6 < 0x1 0x1}}
# is the peer zxid in between the max committed log and the min committed log?
In this case yes it is, but it shouldn't be. The max committed log is {{0x1
0x1}}. The min committed log is {{0x0 0x5}} or something likely below it
because it is based off of distance in the edit log. The issue is that once
the epoch changes, {{0x0}} to {{0x1}}, the leader has no idea if the edits are
in its edit log without explicitly checking for them.
The reason that ZOOKEEPER-2926 exposed this is because previously when a leader
was elected the in memory DB was dropped and everything was reread from disk.
When this happens the {{0x0 0x6}} proposal was lost. But it is not guaranteed
to be lost in all cases. In theory a snapshot could be taken triggered by that
proposal, either on the leader, or on a follower that also allied the proposal,
but does not join the new quorum in time. As such ZOOKEEPER-2926 really just
extended the window of an already existing race. But it extended it almost
indefinitely so it is much more likely to happen.
My fix is to update {{LearnerHandler.syncFollower}} to only send a {{DIFF}} if
the epochs are the same. If they are not the same we don't know if something
we inserted that we don't know about.
> Data inconsistency issue due to retain database in leader election
> --
>
> Key: ZOOKEEPER-2845
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2845
> Project: ZooKeeper
> Issue Type: Bug
> Components: quorum
>Affects Versions: 3.4.10, 3.5.3, 3.6.0
>Reporter: Fangmin Lv
>Assignee: Robert Joseph Evans
>Priority: Critical
>
> In ZOOKEEPER-2678, the ZKDatabase is retained to reduce the unavailable time
> during leader election. In ZooKeeper ensemble, it's possible that the
> snapshot is ahead of txn file (due to slow disk on the server, etc), or the
> txn file is ahead of snapshot due to no commit message being received yet.
> If snapshot is ahead of txn file, since the SyncRequestProcessor queue will
> be drained during shutdown, the snapshot and txn file will keep consistent
> before leader election happening, so this is not an issue.
> But if txn is ahead of snapshot, it's possible that the ensemble will have
> data inconsistent issue, here is the simplified scenario to show the issue:
> Let's say we have a 3 servers in the ensemble, server A and B are followers,
> and C is leader, and all the snapshot and txn are up to T0:
> 1. A new request reached to leader C to create Node N, and it's converted to
> txn T1
> 2. Txn T1 was synced to disk in C, but just before the proposal reaching out
> to the followers, A and B restarted, so the T1 didn't exist in A and B
> 3. A and B formed a new quorum after restart, let's say B is the leader
> 4. C changed to looking state due to no enough followers, it will sync with
> leader B with last Zxid T0, which will have an empty diff sync
> 5. Before C take snapshot it restarted, it replayed the txns on disk which
> includes T1, now it will have Node N, but A and B doesn't have it.
> Also I included the a test case to reproduce this issue consistently.
> We have a totally different RetainDB version which will avoid this issue by
> doing consensus between snapshot and txn files before leader election, will
> submit for review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (ZOOKEEPER-2845) Data inconsistency issue due to retain database in leader election
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans reassigned ZOOKEEPER-2845: -- Assignee: Robert Joseph Evans > Data inconsistency issue due to retain database in leader election > -- > > Key: ZOOKEEPER-2845 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2845 > Project: ZooKeeper > Issue Type: Bug > Components: quorum >Affects Versions: 3.4.10, 3.5.3, 3.6.0 >Reporter: Fangmin Lv >Assignee: Robert Joseph Evans >Priority: Critical > > In ZOOKEEPER-2678, the ZKDatabase is retained to reduce the unavailable time > during leader election. In ZooKeeper ensemble, it's possible that the > snapshot is ahead of txn file (due to slow disk on the server, etc), or the > txn file is ahead of snapshot due to no commit message being received yet. > If snapshot is ahead of txn file, since the SyncRequestProcessor queue will > be drained during shutdown, the snapshot and txn file will keep consistent > before leader election happening, so this is not an issue. > But if txn is ahead of snapshot, it's possible that the ensemble will have > data inconsistent issue, here is the simplified scenario to show the issue: > Let's say we have a 3 servers in the ensemble, server A and B are followers, > and C is leader, and all the snapshot and txn are up to T0: > 1. A new request reached to leader C to create Node N, and it's converted to > txn T1 > 2. Txn T1 was synced to disk in C, but just before the proposal reaching out > to the followers, A and B restarted, so the T1 didn't exist in A and B > 3. A and B formed a new quorum after restart, let's say B is the leader > 4. C changed to looking state due to no enough followers, it will sync with > leader B with last Zxid T0, which will have an empty diff sync > 5. Before C take snapshot it restarted, it replayed the txns on disk which > includes T1, now it will have Node N, but A and B doesn't have it. > Also I included the a test case to reproduce this issue consistently. > We have a totally different RetainDB version which will avoid this issue by > doing consensus between snapshot and txn files before leader election, will > submit for review. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ZOOKEEPER-2699) Restrict 4lw commands based on client IP
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15872094#comment-15872094 ] Robert Joseph Evans commented on ZOOKEEPER-2699: I'm not sure this will fix the issue. IP address spoofing is rather simple to do. You are not guaranteed to get a result back, but for 4 letter commands doing DoS you really don't care all that much. In fact it might be better because you don't have to worry about your node being bogged down with responses from someone else. If we restrict it to the loopback device or something like that, it is much more likely to restrict bad users. > Restrict 4lw commands based on client IP > > > Key: ZOOKEEPER-2699 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2699 > Project: ZooKeeper > Issue Type: Bug > Components: security, server >Reporter: Mohammad Arshad >Assignee: Mohammad Arshad > > Currently 4lw commands are executed without authentication and can be > accessed from any IP which has access to ZooKeeper server. ZOOKEEPER-2693 > attempts to limit the 4lw commands which are enabled by default or enabled by > configuration. > In addition to ZOOKEEPER-2693 we should also restrict 4lw commands based on > client IP as well. It is required for following scenario > # User wants to enable all the 4lw commands > # User wants to limit the access of the commands which are considered to be > safe by default. > > *Implementation:* > we can introduce new property 4lw.commands.host.whitelist > # By default we allow all the hosts, but off course only on the 4lw exposed > commands as per the ZOOKEEPER-2693 > # It can be configured to allow individual IPs(192.168.1.2,192.168.1.3 etc.) > # It can also be configured to allow group of IPs like 192.168.1.* -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (ZOOKEEPER-2678) Large databases take a long time to regain a quorum
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15866743#comment-15866743 ] Robert Joseph Evans commented on ZOOKEEPER-2678: Thanks everyone. I appreciate the great feedback and the quick turnaround time on this. > Large databases take a long time to regain a quorum > --- > > Key: ZOOKEEPER-2678 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2678 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.4.9, 3.5.2 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Fix For: 3.4.10, 3.6.0 > > > I know this is long but please here me out. > I recently inherited a massive zookeeper ensemble. The snapshot is 3.4 GB on > disk. Because of its massive size we have been running into a number of > issues. There are lots of problems that we hope to fix with tuning GC etc, > but the big one right now that is blocking us making a lot of progress on the > rest of them is that when we lose a quorum because the leader left, for what > ever reason, it can take well over 5 mins for a new quorum to be established. > So we cannot tune the leader without risking downtime. > We traced down where the time was being spent and found that each server was > clearing the database so it would be read back in again before leader > election even started. Then as part of the sync phase each server will write > out a snapshot to checkpoint the progress it made as part of the sync. > I will be putting up a patch shortly with some proposed changes in it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (ZOOKEEPER-2678) Large databases take a long time to regain a quorum
Robert Joseph Evans created ZOOKEEPER-2678: -- Summary: Large databases take a long time to regain a quorum Key: ZOOKEEPER-2678 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2678 Project: ZooKeeper Issue Type: Bug Components: server Affects Versions: 3.5.2, 3.4.9 Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans I know this is long but please here me out. I recently inherited a massive zookeeper ensemble. The snapshot is 3.4 GB on disk. Because of its massive size we have been running into a number of issues. There are lots of problems that we hope to fix with tuning GC etc, but the big one right now that is blocking us making a lot of progress on the rest of them is that when we lose a quorum because the leader left, for what ever reason, it can take well over 5 mins for a new quorum to be established. So we cannot tune the leader without risking downtime. We traced down where the time was being spent and found that each server was clearing the database so it would be read back in again before leader election even started. Then as part of the sync phase each server will write out a snapshot to checkpoint the progress it made as part of the sync. I will be putting up a patch shortly with some proposed changes in it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-1716) jute/Utils.fromCSVBuffer cannot parse data returnd by toCSVBuffer
[
https://issues.apache.org/jira/browse/ZOOKEEPER-1716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14505236#comment-14505236
]
Robert Joseph Evans commented on ZOOKEEPER-1716:
Looks like it works to me. Thanks for the fix. +1 (although non-binding)
> jute/Utils.fromCSVBuffer cannot parse data returnd by toCSVBuffer
> --
>
> Key: ZOOKEEPER-1716
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1716
> Project: ZooKeeper
> Issue Type: Bug
> Components: jute
>Affects Versions: 3.5.0
>Reporter: Robert Joseph Evans
> Attachments: ZOOKEEPER-1716.patch
>
>
> I was trying to use org.apache.zookeeper.server.LogFormatter to analyze the
> access pattern of a particular application. As part of this I wanted to get
> the size of the data that was being written into ZK.
> I ran into an issue where in some cases the hex data had an odd length. I
> looked into it and found that the buffer is being written out using
> Integer.toHexString(barr[idx])
> Looking at the javadoce for toHexString it indicates that it does not pad the
> bits at all, and will output the twos compliment of the number if it is
> negative. I then looked at how the data was being parsed and it assumed that
> every byte consisted of exactly two characters, which is not true.
> {code}
> Utils.toCSVBuffer(new byte[] {0xff}) returns "#"
> Utils.toCSVBuffer(new byte[] {0x01}) returns "#1"
> If I combine those
> Utils.fromCSVBuffer(Utils.toCSVBuffer(new byte[] {0xff, 0x01, 0xff})) will
> return {0xff, 0xff, 0xff, 0xff, 0x1f, 0xff, 0xff, 0xff}
> {code}
> I think what we want is something like
> {code}
> static final char[] NIBBLE_TO_HEX = {
> '0', '1', '2', '3', '4', '5', '6', '7',
> '8', '9', 'a', 'b', 'c', 'd', 'e', 'f'
> };
> static String toCSVBuffer(byte barr[]) {
> if (barr == null || barr.length == 0) {
> return "";
> }
> StringBuilder sb = new StringBuilder(barr.length + 1);
> sb.append('#');
> for(int idx = 0; idx < barr.length; idx++) {
> byte b = barr[idx];
> sb.append(NIBBLE_TO_HEX[b&0x0f]);
> sb.append(NIBBLE_TO_HEX[(b&0xf0)>>4]);
> }
> return sb.toString();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2106) Error when reading from leader causes JVM to hang
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14275529#comment-14275529
]
Robert Joseph Evans commented on ZOOKEEPER-2106:
Actually digging further it looks like the ensemble would have recovered
eventually. But it was happening so often that we thought it was not
recovering. I'll dig in deeper to try to get the root cause of the drops, and
open a new JIRA if it is related to ZK. Sorry to have filed a JIRA without
seeing all of what was going on.
> Error when reading from leader causes JVM to hang
> -
>
> Key: ZOOKEEPER-2106
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2106
> Project: ZooKeeper
> Issue Type: Bug
>Affects Versions: 3.4.5
>Reporter: Robert Joseph Evans
>Priority: Critical
>
> I tried looking through existing JIRA for something like this, but the
> closest I came was ZOOKEEPER-2104. It looks very similar, but I don't know
> if it really is the same thing. Essentially we had a 5 node ensemble for a
> large storm cluster. For a few of the nodes at the same time they get an
> error that looks like.
> {code}
> WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@762] - Connection broken for
> id 2, my id = 4, error =
> java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:392)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:747)
> WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@765] - Interrupting SendWorker
> WARN [SendWorker:2:QuorumCnxManager$SendWorker@679] - Interrupted while
> waiting for message on queue
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:831)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:62)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:667)
> WARN [SendWorker:2:QuorumCnxManager$SendWorker@688] - Send worker leaving
> thread
> WARN [QuorumPeer[myid=4]/0.0.0.0:50512:Follower@89] - Exception when
> following the leader
> java.net.SocketException: Connection reset
> at java.net.SocketInputStream.read(SocketInputStream.java:189)
> at java.net.SocketInputStream.read(SocketInputStream.java:121)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
> at java.io.DataInputStream.readInt(DataInputStream.java:387)
> at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
> at
> org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
> at
> org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:108)
> at
> org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:152)
> at
> org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:85)
> at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:740)
> INFO [QuorumPeer[myid=4]/0.0.0.0:50512:Follower@166] - shutdown called
> java.lang.Exception: shutdown Follower
> at
> org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166)
> at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:744)
> {code}
> After that all of the connections are shut down
> {code}
> INFO [QuorumPeer[myid=4]/0.0.0.0:50512:NIOServerCnxn@1001] - Closed socket
> connection for client ...
> {code}
> but it does not manage to have the JVM shut down
> {code}
> INFO [QuorumPeer[myid=4]/0.0.0.0:50512:FollowerZooKeeperServer@139] -
> Shutting down
> INFO [QuorumPeer[myid=4]/0.0.0.0:50512:ZooKeeperServer@419] - shutting down
> INFO [QuorumPeer[myid=4]/0.0.0.0:50512:FollowerRequestProcessor@105] -
> Shutting down
> INFO [QuorumPeer[myid=4]/0.0.0.0:50512:CommitProcessor@181] - Shutting down
> INFO [FollowerRequestProcessor:4:FollowerRequestProcessor@95] -
> FollowerRequestProcessor exited loop!
> INFO [QuorumPeer[myid=4]/0.0.0.0:50512:FinalRequestProcessor@415] - shutdown
> of request processor complete
> INFO [CommitProcessor:4:CommitProcessor@150] - CommitProcessor exited loop!
> WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:50512:NIOServerCnxn@354] -
> Exception causing close of session 0x0 due to java.io.IOException:
> ZooKeeperServer not running
> INF
[jira] [Resolved] (ZOOKEEPER-2106) Error when reading from leader causes JVM to hang
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans resolved ZOOKEEPER-2106.
Resolution: Invalid
> Error when reading from leader causes JVM to hang
> -
>
> Key: ZOOKEEPER-2106
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2106
> Project: ZooKeeper
> Issue Type: Bug
>Affects Versions: 3.4.5
>Reporter: Robert Joseph Evans
>Priority: Critical
>
> I tried looking through existing JIRA for something like this, but the
> closest I came was ZOOKEEPER-2104. It looks very similar, but I don't know
> if it really is the same thing. Essentially we had a 5 node ensemble for a
> large storm cluster. For a few of the nodes at the same time they get an
> error that looks like.
> {code}
> WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@762] - Connection broken for
> id 2, my id = 4, error =
> java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:392)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:747)
> WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@765] - Interrupting SendWorker
> WARN [SendWorker:2:QuorumCnxManager$SendWorker@679] - Interrupted while
> waiting for message on queue
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:831)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:62)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:667)
> WARN [SendWorker:2:QuorumCnxManager$SendWorker@688] - Send worker leaving
> thread
> WARN [QuorumPeer[myid=4]/0.0.0.0:50512:Follower@89] - Exception when
> following the leader
> java.net.SocketException: Connection reset
> at java.net.SocketInputStream.read(SocketInputStream.java:189)
> at java.net.SocketInputStream.read(SocketInputStream.java:121)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
> at java.io.DataInputStream.readInt(DataInputStream.java:387)
> at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
> at
> org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
> at
> org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:108)
> at
> org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:152)
> at
> org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:85)
> at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:740)
> INFO [QuorumPeer[myid=4]/0.0.0.0:50512:Follower@166] - shutdown called
> java.lang.Exception: shutdown Follower
> at
> org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166)
> at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:744)
> {code}
> After that all of the connections are shut down
> {code}
> INFO [QuorumPeer[myid=4]/0.0.0.0:50512:NIOServerCnxn@1001] - Closed socket
> connection for client ...
> {code}
> but it does not manage to have the JVM shut down
> {code}
> INFO [QuorumPeer[myid=4]/0.0.0.0:50512:FollowerZooKeeperServer@139] -
> Shutting down
> INFO [QuorumPeer[myid=4]/0.0.0.0:50512:ZooKeeperServer@419] - shutting down
> INFO [QuorumPeer[myid=4]/0.0.0.0:50512:FollowerRequestProcessor@105] -
> Shutting down
> INFO [QuorumPeer[myid=4]/0.0.0.0:50512:CommitProcessor@181] - Shutting down
> INFO [FollowerRequestProcessor:4:FollowerRequestProcessor@95] -
> FollowerRequestProcessor exited loop!
> INFO [QuorumPeer[myid=4]/0.0.0.0:50512:FinalRequestProcessor@415] - shutdown
> of request processor complete
> INFO [CommitProcessor:4:CommitProcessor@150] - CommitProcessor exited loop!
> WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:50512:NIOServerCnxn@354] -
> Exception causing close of session 0x0 due to java.io.IOException:
> ZooKeeperServer not running
> INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:50512:NIOServerCnxn@1001] -
> Closed socket connection for client /... (no session established for client)
> INFO [QuorumPeer[myid=4]/0.0.0.0:50512:SyncRequestProcessor@175] - Shutting
> down
> INFO [SyncThread:4:SyncRequestProcessor@155] - SyncRequestProcessor exited!
> INFO [QuorumPeer[myid=4]/0.0.0.0:50512:QuorumPeer
[jira] [Created] (ZOOKEEPER-2106) Error when reading from leader causes JVM to hang
Robert Joseph Evans created ZOOKEEPER-2106:
--
Summary: Error when reading from leader causes JVM to hang
Key: ZOOKEEPER-2106
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2106
Project: ZooKeeper
Issue Type: Bug
Affects Versions: 3.4.5
Reporter: Robert Joseph Evans
Priority: Critical
I tried looking through existing JIRA for something like this, but the closest
I came was ZOOKEEPER-2104. It looks very similar, but I don't know if it
really is the same thing. Essentially we had a 5 node ensemble for a large
storm cluster. For a few of the nodes at the same time they get an error that
looks like.
{code}
WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@762] - Connection broken for id
2, my id = 4, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at
org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:747)
WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@765] - Interrupting SendWorker
WARN [SendWorker:2:QuorumCnxManager$SendWorker@679] - Interrupted while
waiting for message on queue
java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at
java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at
org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:831)
at
org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:62)
at
org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:667)
WARN [SendWorker:2:QuorumCnxManager$SendWorker@688] - Send worker leaving
thread
WARN [QuorumPeer[myid=4]/0.0.0.0:50512:Follower@89] - Exception when following
the leader
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:189)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
at
org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
at
org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:108)
at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:152)
at
org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:85)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:740)
INFO [QuorumPeer[myid=4]/0.0.0.0:50512:Follower@166] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:744)
{code}
After that all of the connections are shut down
{code}
INFO [QuorumPeer[myid=4]/0.0.0.0:50512:NIOServerCnxn@1001] - Closed socket
connection for client ...
{code}
but it does not manage to have the JVM shut down
{code}
INFO [QuorumPeer[myid=4]/0.0.0.0:50512:FollowerZooKeeperServer@139] - Shutting
down
INFO [QuorumPeer[myid=4]/0.0.0.0:50512:ZooKeeperServer@419] - shutting down
INFO [QuorumPeer[myid=4]/0.0.0.0:50512:FollowerRequestProcessor@105] -
Shutting down
INFO [QuorumPeer[myid=4]/0.0.0.0:50512:CommitProcessor@181] - Shutting down
INFO [FollowerRequestProcessor:4:FollowerRequestProcessor@95] -
FollowerRequestProcessor exited loop!
INFO [QuorumPeer[myid=4]/0.0.0.0:50512:FinalRequestProcessor@415] - shutdown
of request processor complete
INFO [CommitProcessor:4:CommitProcessor@150] - CommitProcessor exited loop!
WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:50512:NIOServerCnxn@354] -
Exception causing close of session 0x0 due to java.io.IOException:
ZooKeeperServer not running
INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:50512:NIOServerCnxn@1001] - Closed
socket connection for client /... (no session established for client)
INFO [QuorumPeer[myid=4]/0.0.0.0:50512:SyncRequestProcessor@175] - Shutting
down
INFO [SyncThread:4:SyncRequestProcessor@155] - SyncRequestProcessor exited!
INFO [QuorumPeer[myid=4]/0.0.0.0:50512:QuorumPeer@670] - LOOKING
{code}
after that all connections to that node initiate, and then are shut down with
ZooKeeperServer not running. It seems to stay in this state indefinitely until
the process is manually restarted. After that it recovers.
We have seen this happen on multiple servers at the same time resulting in the
entire ensemble being unusable.
[jira] [Updated] (ZOOKEEPER-1782) zookeeper.superUser is not as super as superDigest
[
https://issues.apache.org/jira/browse/ZOOKEEPER-1782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated ZOOKEEPER-1782:
---
Attachment: zk-1782.patch
This version includes a unit test. The test fails without the update and works
with it.
I have run all of the java unit tests and they all pass.
> zookeeper.superUser is not as super as superDigest
> --
>
> Key: ZOOKEEPER-1782
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1782
> Project: ZooKeeper
> Issue Type: Bug
>Affects Versions: 3.4.5
>Reporter: Robert Joseph Evans
> Attachments: zk-1782.patch, zk-1782.patch
>
>
> The zookeeper.superUser system property does not fully grant super user
> privileges, like zookeeper.DigestAuthenticationProvider.superDigest does.
> zookeeper.superUser only has as many privileges as the sasl ACLs on the znode
> being accessed. This means that if a znode only has digest ACLs
> zookeeper.superUser is ignored. Or if a znode has a single sasl ACL that
> only has read privileges zookeeper.superUser only has read privileges.
> The reason for this is that SASLAuthenticationProvider implements the
> superUser check in the matches method, instead of having the super user
> include a new Id("super","") as Digest does.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (ZOOKEEPER-1782) zookeeper.superUser is not as super as superDigest
[
https://issues.apache.org/jira/browse/ZOOKEEPER-1782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated ZOOKEEPER-1782:
---
Attachment: zk-1782.patch
This is a simple patch that should fix the issue. I still need to add in some
unit tests, but I am not super familiar with the code so I thought I would
throw this up first to get feedback on it.
> zookeeper.superUser is not as super as superDigest
> --
>
> Key: ZOOKEEPER-1782
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1782
> Project: ZooKeeper
> Issue Type: Bug
>Affects Versions: 3.4.5
>Reporter: Robert Joseph Evans
> Attachments: zk-1782.patch
>
>
> The zookeeper.superUser system property does not fully grant super user
> privileges, like zookeeper.DigestAuthenticationProvider.superDigest does.
> zookeeper.superUser only has as many privileges as the sasl ACLs on the znode
> being accessed. This means that if a znode only has digest ACLs
> zookeeper.superUser is ignored. Or if a znode has a single sasl ACL that
> only has read privileges zookeeper.superUser only has read privileges.
> The reason for this is that SASLAuthenticationProvider implements the
> superUser check in the matches method, instead of having the super user
> include a new Id("super","") as Digest does.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (ZOOKEEPER-1782) zookeeper.superUser is not as super as superDigest
Robert Joseph Evans created ZOOKEEPER-1782:
--
Summary: zookeeper.superUser is not as super as superDigest
Key: ZOOKEEPER-1782
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1782
Project: ZooKeeper
Issue Type: Bug
Affects Versions: 3.4.5
Reporter: Robert Joseph Evans
The zookeeper.superUser system property does not fully grant super user
privileges, like zookeeper.DigestAuthenticationProvider.superDigest does.
zookeeper.superUser only has as many privileges as the sasl ACLs on the znode
being accessed. This means that if a znode only has digest ACLs
zookeeper.superUser is ignored. Or if a znode has a single sasl ACL that only
has read privileges zookeeper.superUser only has read privileges.
The reason for this is that SASLAuthenticationProvider implements the superUser
check in the matches method, instead of having the super user include a new
Id("super","") as Digest does.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (ZOOKEEPER-1716) jute/Utils.fromCSVBuffer cannot parse data returnd by toCSVBuffer
[
https://issues.apache.org/jira/browse/ZOOKEEPER-1716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated ZOOKEEPER-1716:
---
Description:
I was trying to use org.apache.zookeeper.server.LogFormatter to analyze the
access pattern of a particular application. As part of this I wanted to get
the size of the data that was being written into ZK.
I ran into an issue where in some cases the hex data had an odd length. I
looked into it and found that the buffer is being written out using
Integer.toHexString(barr[idx])
Looking at the javadoce for toHexString it indicates that it does not pad the
bits at all, and will output the twos compliment of the number if it is
negative. I then looked at how the data was being parsed and it assumed that
every byte consisted of exactly two characters, which is not true.
{code}
Utils.toCSVBuffer(new byte[] {0xff}) returns "#"
Utils.toCSVBuffer(new byte[] {0x01}) returns "#1"
If I combine those
Utils.fromCSVBuffer(Utils.toCSVBuffer(new byte[] {0xff, 0x01, 0xff})) will
return {0xff, 0xff, 0xff, 0xff, 0x1f, 0xff, 0xff, 0xff}
{code}
I think what we want is something like
{code}
static final char[] NIBBLE_TO_HEX = {
'0', '1', '2', '3', '4', '5', '6', '7',
'8', '9', 'a', 'b', 'c', 'd', 'e', 'f'
};
static String toCSVBuffer(byte barr[]) {
if (barr == null || barr.length == 0) {
return "";
}
StringBuilder sb = new StringBuilder(barr.length + 1);
sb.append('#');
for(int idx = 0; idx < barr.length; idx++) {
byte b = barr[idx];
sb.append(NIBBLE_TO_HEX[b&0x0f]);
sb.append(NIBBLE_TO_HEX[(b&0xf0)>>4]);
}
return sb.toString();
}
{code}
was:
I was trying to use org.apache.zookeeper.server.LogFormatter to analyze the
access pattern of a particular application. As part of this I wanted to get
the size of the data that was being written into ZK.
I ran into an issue where in some cases the hex data had an odd length. I
looked into it and found that the buffer is being written out using
Integer.toHexString(barr[idx])
Looking at the javadoce for toHexString it indicates that it does not pad the
bits at all, and will output the twos compliment of the number if it is
negative. I then looked at how the data was being parsed and it assumed that
every byte consisted of exactly two characters, which is not true.
Utils.toCSVBuffer(new byte[] {0xff}) returns "#"
Utils.toCSVBuffer(new byte[] {0x01}) returns "#1"
If I combine those
Utils.fromCSVBuffer(Utils.toCSVBuffer(new byte[] {0xff, 0x01, 0xff})) will
return {0xff, 0xff, 0xff, 0xff, 0x1f, 0xff, 0xff, 0xff}
I think what we want is something like
static final char[] NIBBLE_TO_HEX = {
'0', '1', '2', '3', '4', '5', '6', '7',
'8', '9', 'a', 'b', 'c', 'd', 'e', 'f'
};
static String toCSVBuffer(byte barr[]) {
if (barr == null || barr.length == 0) {
return "";
}
StringBuilder sb = new StringBuilder(barr.length + 1);
sb.append('#');
for(int idx = 0; idx < barr.length; idx++) {
byte b = barr[idx];
sb.append(NIBBLE_TO_HEX[b&0x0f]);
sb.append(NIBBLE_TO_HEX[(b&0xf0)>>4]);
}
return sb.toString();
}
> jute/Utils.fromCSVBuffer cannot parse data returnd by toCSVBuffer
> --
>
> Key: ZOOKEEPER-1716
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1716
> Project: ZooKeeper
> Issue Type: Bug
> Components: jute
>Affects Versions: 3.4.5
>Reporter: Robert Joseph Evans
>
> I was trying to use org.apache.zookeeper.server.LogFormatter to analyze the
> access pattern of a particular application. As part of this I wanted to get
> the size of the data that was being written into ZK.
> I ran into an issue where in some cases the hex data had an odd length. I
> looked into it and found that the buffer is being written out using
> Integer.toHexString(barr[idx])
> Looking at the javadoce for toHexString it indicates that it does not pad the
> bits at all, and will output the twos compliment of the number if it is
> negative. I then looked at how the data was being parsed and it assumed that
> every byte consisted of exactly two characters, which is not true.
> {code}
> Utils.toCSVBuffer(new byte[] {0xff}) returns "#"
> Utils.toCSVBuffer(new byte[] {0x01}) returns "#1"
> If I combine those
> Utils.fromCSVBuffer(Utils.toCSVBuffer(new byte[] {0xff, 0x01, 0xff})) will
> return {0xff, 0xff, 0xff, 0xff, 0x1f, 0xff, 0xff, 0xff}
> {code}
> I think what we want is something like
> {code}
> static final char[] NIBBLE_TO_HEX = {
> '0', '1', '2', '3', '4', '5', '6', '7',
> '8', '9', 'a', 'b', 'c', 'd', 'e', 'f'
> };
> static String toCSVBuffer(byte barr[]) {
> if (barr == null || barr.length == 0) {
> return "";
> }
> StringBu
[jira] [Created] (ZOOKEEPER-1716) jute/Utils.fromCSVBuffer cannot parse data returnd by toCSVBuffer
Robert Joseph Evans created ZOOKEEPER-1716:
--
Summary: jute/Utils.fromCSVBuffer cannot parse data returnd by
toCSVBuffer
Key: ZOOKEEPER-1716
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1716
Project: ZooKeeper
Issue Type: Bug
Components: jute
Affects Versions: 3.4.5
Reporter: Robert Joseph Evans
I was trying to use org.apache.zookeeper.server.LogFormatter to analyze the
access pattern of a particular application. As part of this I wanted to get
the size of the data that was being written into ZK.
I ran into an issue where in some cases the hex data had an odd length. I
looked into it and found that the buffer is being written out using
Integer.toHexString(barr[idx])
Looking at the javadoce for toHexString it indicates that it does not pad the
bits at all, and will output the twos compliment of the number if it is
negative. I then looked at how the data was being parsed and it assumed that
every byte consisted of exactly two characters, which is not true.
Utils.toCSVBuffer(new byte[] {0xff}) returns "#"
Utils.toCSVBuffer(new byte[] {0x01}) returns "#1"
If I combine those
Utils.fromCSVBuffer(Utils.toCSVBuffer(new byte[] {0xff, 0x01, 0xff})) will
return {0xff, 0xff, 0xff, 0xff, 0x1f, 0xff, 0xff, 0xff}
I think what we want is something like
static final char[] NIBBLE_TO_HEX = {
'0', '1', '2', '3', '4', '5', '6', '7',
'8', '9', 'a', 'b', 'c', 'd', 'e', 'f'
};
static String toCSVBuffer(byte barr[]) {
if (barr == null || barr.length == 0) {
return "";
}
StringBuilder sb = new StringBuilder(barr.length + 1);
sb.append('#');
for(int idx = 0; idx < barr.length; idx++) {
byte b = barr[idx];
sb.append(NIBBLE_TO_HEX[b&0x0f]);
sb.append(NIBBLE_TO_HEX[(b&0xf0)>>4]);
}
return sb.toString();
}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
