[jira] Updated: (ZOOKEEPER-784) server-side functionality for read-only mode
[ https://issues.apache.org/jira/browse/ZOOKEEPER-784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Doroshenko updated ZOOKEEPER-784: Attachment: ZOOKEEPER-784.patch Added read-only functionality to client side: to create RO client, one have to pass additional boolean param to ZooKeeper's ctor; ctors with old signatures create usual clients. Now, if server is partitioned, it doesn't allow usual clients to connect. Decided not to create separate ticket for client-side change since it's anyway related to this one. server-side functionality for read-only mode Key: ZOOKEEPER-784 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-784 Project: Zookeeper Issue Type: Sub-task Reporter: Sergey Doroshenko Assignee: Sergey Doroshenko Attachments: ZOOKEEPER-784.patch, ZOOKEEPER-784.patch, ZOOKEEPER-784.patch As per http://wiki.apache.org/hadoop/ZooKeeper/GSoCReadOnlyMode , create ReadOnlyZooKeeperServer which comes into play when peer is partitioned. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-661) Add Ruby bindings
[ https://issues.apache.org/jira/browse/ZOOKEEPER-661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12879391#action_12879391 ] Patrick Hunt commented on ZOOKEEPER-661: Cool, thanks all! It would be great to see all this unified and in contrib, but seeing activity on this front regardless of where/how it's hosted is awesome! Thanks! Add Ruby bindings - Key: ZOOKEEPER-661 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-661 Project: Zookeeper Issue Type: New Feature Components: contrib-bindings Environment: MRI Ruby 1.9 JRuby 1.4 Reporter: Andrew Reynhout Priority: Minor Add Ruby bindings to the ZooKeeper distribution. Ruby presents special threading difficulties for asynchronous ZK calls (aget, watchers, etc). It looks like the simplest workaround is to patch the ZK C API. Proposed approach will be described in comment. Please use this ticket for discussion and suggestions. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-784) server-side functionality for read-only mode
[ https://issues.apache.org/jira/browse/ZOOKEEPER-784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12879405#action_12879405 ] Patrick Hunt commented on ZOOKEEPER-784: I read through the wiki page, I think this is an awesome feature! A few high level questions came to mind: 1) please be clear in the wiki page on b/w compatibility. There's some discussion of this, but what I mean is be very explicit regarding client-server compatibility, server-server compatibility. Have individual sections for this: a) new client new server (obv compat) b) new server new server (ditto) c) new client (using r/o client mode) old server (?) d) new server - old server (?) 2) be clear on how the client library will search for r/o server vs w/r server (if ensemble has internal partitioning) be clear on how this will work if no servers at all are available. In particular I'm interested in how the client will backoff vs hammering (constantly connecting/disconnecting to servers) 3) add use cases that explain to users exactly how this will benefit them. go through some actual use cases such as leader election, group membership, and dynamic config. You should clearly spellout how the user's client code will handle these cases when using r/o mode. You might for example rewrite the recipes from the recipes page for the r/o supported version. Detail any special considerations the user might need to consider. For example, in the case of leader election you might detail how this is handled from zk perspective (client attached to r/o server) but also detail the considerations the user should have - such as the leader may have changed, what might the user have to consider. (leader can change if client A is connected to r/o server while the rest of the service (other clients of zk) might be connected to a r/w server, during which time the users's leadership changes). etc... We'll need this anyway for the user docs, doing it upfront on the jira will help you both for that and while developing the feature/tests. server-side functionality for read-only mode Key: ZOOKEEPER-784 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-784 Project: Zookeeper Issue Type: Sub-task Reporter: Sergey Doroshenko Assignee: Sergey Doroshenko Attachments: ZOOKEEPER-784.patch, ZOOKEEPER-784.patch, ZOOKEEPER-784.patch As per http://wiki.apache.org/hadoop/ZooKeeper/GSoCReadOnlyMode , create ReadOnlyZooKeeperServer which comes into play when peer is partitioned. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-784) server-side functionality for read-only mode
[ https://issues.apache.org/jira/browse/ZOOKEEPER-784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12879407#action_12879407 ] Patrick Hunt commented on ZOOKEEPER-784: For b/w compat I forgot: old client - new server as well as new client not using the r/o feature against new/old server. I might have missed others... server-side functionality for read-only mode Key: ZOOKEEPER-784 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-784 Project: Zookeeper Issue Type: Sub-task Reporter: Sergey Doroshenko Assignee: Sergey Doroshenko Attachments: ZOOKEEPER-784.patch, ZOOKEEPER-784.patch, ZOOKEEPER-784.patch As per http://wiki.apache.org/hadoop/ZooKeeper/GSoCReadOnlyMode , create ReadOnlyZooKeeperServer which comes into play when peer is partitioned. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-702) GSoC 2010: Failure Detector Model
[ https://issues.apache.org/jira/browse/ZOOKEEPER-702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12879450#action_12879450 ] Abmar Barros commented on ZOOKEEPER-702: Once ZooKeeper guarantees delivery order, I can ignore the seq field in heartbeat and pings messages. Now I am implementing the methods on the interface I proposed and two issues came to mind: * ZooKeeper uses application messages as hearbeats, so actual pings and heartbeats are used only at idle states. The proposed methods must be adapted to consider this scenario. * Regarding that a single thread is being used for the app and the FD, there is a delay between the time a monitored must be pinged and the time it is actually pinged. The impact of this delay in the FD QoS must be analyzed. GSoC 2010: Failure Detector Model - Key: ZOOKEEPER-702 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-702 Project: Zookeeper Issue Type: Wish Reporter: Henry Robinson Assignee: Abmar Barros Attachments: bertier-pseudo.txt, chen-pseudo.txt, phiaccrual-pseudo.txt, ZOOKEEPER-702.patch Failure Detector Module Possible Mentor Henry Robinson (henry at apache dot org) Requirements Java, some distributed systems knowledge, comfort implementing distributed systems protocols Description ZooKeeper servers detects the failure of other servers and clients by counting the number of 'ticks' for which it doesn't get a heartbeat from other machines. This is the 'timeout' method of failure detection and works very well; however it is possible that it is too aggressive and not easily tuned for some more unusual ZooKeeper installations (such as in a wide-area network, or even in a mobile ad-hoc network). This project would abstract the notion of failure detection to a dedicated Java module, and implement several failure detectors to compare and contrast their appropriateness for ZooKeeper. For example, Apache Cassandra uses a phi-accrual failure detector (http://ddsg.jaist.ac.jp/pub/HDY+04.pdf) which is much more tunable and has some very interesting properties. This is a great project if you are interested in distributed algorithms, or want to help re-factor some of ZooKeeper's internal code. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: Meetup entry for Zookeeper Contributors meeting (day after summit)
Hi Patrick, Will this be hosted over web? Thanks. -Vishal On Fri, Jun 11, 2010 at 3:33 PM, Patrick Hunt ph...@apache.org wrote: There's now a meetup entry for anyone interested in attending the ZooKeeper contributors meeting happening at Yahoo offices (sunnyvale) the day after the summit. Lunch will be served. Please RSVP if you plan to attend so that we can be sure to have enough space/vittles. http://www.meetup.com/Hadoop-Contributors/calendar/13771414/ Regards, Patrick
[jira] Commented: (ZOOKEEPER-335) zookeeper servers should commit the new leader txn to their logs.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12879520#action_12879520 ] Vishal K commented on ZOOKEEPER-335: Hi, We are running into this bug very often (almost 60-75% hit rate) while testing our newly developed application over ZK. This is almost a blocker for us. Will the fix be simplified if backward compatibility was not an issue? Thanks. zookeeper servers should commit the new leader txn to their logs. - Key: ZOOKEEPER-335 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-335 Project: Zookeeper Issue Type: Bug Components: server Affects Versions: 3.1.0 Reporter: Mahadev konar Assignee: Mahadev konar Priority: Blocker Fix For: 3.4.0 currently the zookeeper followers do not commit the new leader election. This will cause problems in a failure scenarios with a follower acking to the same leader txn id twice, which might be two different intermittent leaders and allowing them to propose two different txn's of the same zxid. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-712) Bookie recovery
[ https://issues.apache.org/jira/browse/ZOOKEEPER-712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erwin Tam updated ZOOKEEPER-712: Attachment: ZOOKEEPER-712.patch Bookie recovery --- Key: ZOOKEEPER-712 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-712 Project: Zookeeper Issue Type: New Feature Components: contrib-bookkeeper Reporter: Flavio Paiva Junqueira Assignee: Erwin Tam Attachments: ZOOKEEPER-712.patch Recover the ledger fragments of a bookie once it crashes. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: Meetup entry for Zookeeper Contributors meeting (day after summit)
Yahoo tells me that we'll have a conference call bridge number, I have not gotten that detail yet though. When I have it I'll update the meetup. Patrick On 06/16/2010 02:36 PM, Vishal K wrote: Hi Patrick, Will this be hosted over web? Thanks. -Vishal On Fri, Jun 11, 2010 at 3:33 PM, Patrick Huntph...@apache.org wrote: There's now a meetup entry for anyone interested in attending the ZooKeeper contributors meeting happening at Yahoo offices (sunnyvale) the day after the summit. Lunch will be served. Please RSVP if you plan to attend so that we can be sure to have enough space/vittles. http://www.meetup.com/Hadoop-Contributors/calendar/13771414/ Regards, Patrick
[jira] Commented: (ZOOKEEPER-335) zookeeper servers should commit the new leader txn to their logs.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12879548#action_12879548 ] Patrick Hunt commented on ZOOKEEPER-335: We are unable to reproduce this issue. If you can provide the server logs (all servers) and attach them to this jira it would be very helpful. Some detail on the approx time of the issue so we can correlate to the logs would help too (summary of what you did/do to cause it, etc... anything that might help us nail this one down). Detail on ZK version, OS, Java version, HW info, etc... would also be of use to us. zookeeper servers should commit the new leader txn to their logs. - Key: ZOOKEEPER-335 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-335 Project: Zookeeper Issue Type: Bug Components: server Affects Versions: 3.1.0 Reporter: Mahadev konar Assignee: Mahadev konar Priority: Blocker Fix For: 3.4.0 currently the zookeeper followers do not commit the new leader election. This will cause problems in a failure scenarios with a follower acking to the same leader txn id twice, which might be two different intermittent leaders and allowing them to propose two different txn's of the same zxid. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-712) Bookie recovery
[ https://issues.apache.org/jira/browse/ZOOKEEPER-712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erwin Tam updated ZOOKEEPER-712: Status: Patch Available (was: Open) First pass at implementing bookie recovery as an admin tool. The patch uploaded has two new files: BookieRecoveryTest.java and BookKeeperTools.java (with a corresponding new directory under bookkeeper/tools). The main API that BookKeeperTools.java implements is the following: public void asyncRecoverBookieData(final InetSocketAddress bookieSrc, final InetSocketAddress bookieDest, final RecoverCallback cb, final Object context); The synchronous version of this API is here: public void recoverBookieData(final InetSocketAddress bookieSrc, final InetSocketAddress bookieDest) throws InterruptedException; This is to recover all of the bookie ledger data that was present on a dead input bookieSrc. The other input 'bookieDest' is optional and if passed, we will recover all data to that bookie server specifically. Otherwise for each ledger being recovered that was stored on the bookieSrc, we choose one of the other available bookie servers and re-replicate the ledger fragment entries to there. A command line way to invoke this is done via a main method in BookKeeperTools which expects the following 2-3 input parameters. USAGE: BookKeeperTools zkServers bookieSrc [bookieDest] zkServers is a comma separated list of host:port pairs for the ZooKeeper servers in the cluster. bookieSrc is the host:port for the bookie server we are recovering data from. bookieDest is the host:port (optional) for the bookie server we want to recover the data to. Current limitations: There is no way to know what each ledger's digest type or key/password are. That metadata isn't stored anywhere we can access readily. Thus for now, we are assuming that all ledgers to be recovered have been created with the same digest type and password. These values are set via java system properties called digestType and passwd. Once we have a way to store and retrieve this information (via ZK most likely and stored by the Bookie servers when new ledgers are created), we can modify this aspect of the bookie recovery tool. Pseudocode: Inputs: zkServers - Used to create a ZK client so we can read in all of the BK metadata needed to perform the bookie recovery. bookieSrc - Used to match against ledger metadata indicating which bookie servers comprise the ensembles that make up a ledger. bookieDest - Optionally used to write the recovered ledger data to directly. 1. Sync with ZK 2. Read from ZK to get all available bookie servers (only if bookieDest was not passed). 3. Read from ZK to get all active ledgers. 4. For each ledger, open it to obtain the LedgerHandle. 5. For each ledger fragment, see if the dead input bookieSrc is a part of the ensemble for it. 6. For each ledger fragment that needs to be recovered, find the entries that were actually stored on the bookieSrc. Since we stripe data across bookies in the ensemble, not all ledger entries in the fragment would have been stored on the bookieSrc. 7. For each of the ledger entries in the fragment that were stored on the bookieSrc, use the BookKeeper client to read it. Using the client will take care of choosing one of the other bookies where this data is available since the bookieSrc server is dead. 8. Choosing a new bookieDest (either passed in explicitly or chosen at random currently among the available bookie servers), write this ledger entry directly to there using the BookieClient (the layer that talks to the bookie servers directly). 9. Once all ledger entries for all fragments for a ledger have been recovered and re-replicated, update ZK so the ledger's metadata now points to the new bookie server where this data has been re-replicated to. We choose a single new bookie for each ledger to store the recovered data. Bookie recovery --- Key: ZOOKEEPER-712 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-712 Project: Zookeeper Issue Type: New Feature Components: contrib-bookkeeper Reporter: Flavio Paiva Junqueira Assignee: Erwin Tam Attachments: ZOOKEEPER-712.patch Recover the ledger fragments of a bookie once it crashes. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
