RE: zookeeper dev f2f meeting before or after the hadoop summit
Hi Ben, I think this is a great idea. I am an engineer at facebook and I will be starting working on Zookeeper in the next week or so and it would be great to meet other developers in the zookeeper community. Thanks Vishal -Original Message- From: Mahadev Konar [mailto:maha...@apache.org] Sent: Monday, May 02, 2011 4:24 PM To: dev@zookeeper.apache.org Cc: zookeeper-...@hadoop.apache.org Subject: Re: zookeeper dev f2f meeting before or after the hadoop summit Nice idea Ben. I'd say before the Hadoop Summit. Folks are usually running around to catch there flight etc after the Summit. thanks mahadev On Fri, Apr 29, 2011 at 1:21 PM, Benjamin Reed br...@apache.org wrote: hey do you think it would be a good idea to have a f2f meeting before or after the hadoop summit? the pig development group does this and the meetings are really good. it is open for everyone but its really to advertise and coordinate core development efforts. i believe they setup a conf # for people that can't make it. ben -- thanks mahadev @mahadevkonar
[jira] [Commented] (ZOOKEEPER-1027) chroot not transparent in zoo_create()
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13032543#comment-13032543 ] Mahadev konar commented on ZOOKEEPER-1027: -- +1 the patch looks good to me. chroot not transparent in zoo_create() -- Key: ZOOKEEPER-1027 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1027 Project: ZooKeeper Issue Type: Bug Components: c client Affects Versions: 3.3.3 Environment: Linux, ZooKeeper 3.3.3, C-client, java 1.6.0_17-b04, hotspot server vm Reporter: Thijs Terlouw Assignee: Thijs Terlouw Fix For: 3.3.3, 3.4.0 Attachments: ZOOKEEPER-1027-TRUNK_WITH_TESTS.patch I've recently started to use the chroot functionality (introduced in 3.2.0) as part of my connect string.It mostly works as expected, but there is one case that is unexpected: when I create a path with zoo_create() I can retrieve the created path. This is very useful when you set the ZOO_SEQUENCE flag. Unfortunately the returned path includes the chroot as part of the path. This was unexpected to me: I expected that the chroot would be totally transparent. The documentation for zoo_create() says: path_buffer : Buffer which will be filled with the path of the new node (this might be different than the supplied path because of the ZOO_SEQUENCE flag). This gave me the impression that this flag is the only reason the returned path is different from the created path, but apparently it's not. Is this a bug or intended behavior? I workaround this issue now by remembering the chroot in my wrapper code and after a call to zoo_create() i check if the returned path starts with the chroot. If it does, I remove it. My use case is to create a path with a sequence number and then delete this path later. Unfortunately I cannot delete the path because it has the chroot prepended to it, and thus it will result in two chroots. I believe this only affects the create functions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1027) chroot not transparent in zoo_create()
[
https://issues.apache.org/jira/browse/ZOOKEEPER-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13032549#comment-13032549
]
Mahadev konar commented on ZOOKEEPER-1027:
--
one minor nit:
{noformat}
LOG_ERROR((server path %s does not include chroot path %s,
server_path, zh-chroot));
{noformat}
I think we should comment this out as well, since this wont be an error on a
create call. What do you think Thijs?
chroot not transparent in zoo_create()
--
Key: ZOOKEEPER-1027
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1027
Project: ZooKeeper
Issue Type: Bug
Components: c client
Affects Versions: 3.3.3
Environment: Linux, ZooKeeper 3.3.3, C-client, java 1.6.0_17-b04,
hotspot server vm
Reporter: Thijs Terlouw
Assignee: Thijs Terlouw
Fix For: 3.4.0
Attachments: ZOOKEEPER-1027-TRUNK_WITH_TESTS.patch
I've recently started to use the chroot functionality (introduced in
3.2.0) as part of my connect string.It mostly works as expected, but
there is one case that is unexpected: when I create a path with
zoo_create() I can retrieve the created path. This is very useful when
you set the ZOO_SEQUENCE flag. Unfortunately the returned path
includes the chroot as part of the path. This was unexpected to me: I
expected that the chroot would be totally transparent. The
documentation for zoo_create() says:
path_buffer : Buffer which will be filled with the path of the new
node (this might be different than the supplied path because of the
ZOO_SEQUENCE flag).
This gave me the impression that this flag is the only reason the
returned path is different from the created path, but apparently it's
not. Is this a bug or intended behavior?
I workaround this issue now by remembering the chroot in
my wrapper code and after a call to zoo_create() i check if the returned
path starts with the chroot. If it does, I remove it.
My use case is to create a path with a sequence number and then delete
this path later. Unfortunately I cannot delete the path because it has
the chroot prepended to it, and thus it will result in two chroots.
I believe this only affects the create functions.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: zookeeper dev f2f meeting before or after the hadoop summit
I'm looking forward to meeting other ZK devs. Thanks to Ben for suggesting it. Either before or after the Summit is fine for me. -Eugene
Re: zookeeper dev f2f meeting before or after the hadoop summit
Ben, last year Yahoo was nice enough to host us. Are they doing again this year? I heard some of the other hadoop related projects were going to get together, should we try to sync up with them? Whichever way you go this sounds like a good idea to me. Patrick On Thu, May 12, 2011 at 11:03 AM, Eugene Koontz ekoo...@hiro-tan.org wrote: I'm looking forward to meeting other ZK devs. Thanks to Ben for suggesting it. Either before or after the Summit is fine for me. -Eugene
[jira] [Commented] (ZOOKEEPER-965) Need a multi-update command to allow multiple znodes to be updated safely
[ https://issues.apache.org/jira/browse/ZOOKEEPER-965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13032575#comment-13032575 ] Stephen Tyree commented on ZOOKEEPER-965: - Great job all around! Some comments on the C library portion of the patch: - The zoo_acheck comment includes documentation on a member 'stat' that is not an argument - The zoo_check comment has the parameter stat listed as _stat - In create_completion_entry, you commented out the else case where you set c-clist to NULL. Did you mean to leave this in there? - In op_result_string_completion, you memcpy from the value onto data-value based on the size of value. Is data-value guaranteed to be larger than value? - You only assert result in op_result_string_completion, but none of the other op_result_* functions - Do you not need to set result-err in op_result_multi_completion? - In zoo_amulti, any reason int index = 0; isn't a few lines down right before it's used? - Slick zoo_amulti code, very nicely done. - In testCreate you compare results[*].err to 0, which will work, but shouldn't you compare it to ZOK? Or do I misunderstand op_result_t. - In testDelete, your comment should say // '/multi2' should have been deleted All-in-all the code looks great in the C client. Nicely done. Need a multi-update command to allow multiple znodes to be updated safely - Key: ZOOKEEPER-965 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-965 Project: ZooKeeper Issue Type: Bug Affects Versions: 3.3.3 Reporter: Ted Dunning Assignee: Ted Dunning Fix For: 3.4.0 Attachments: ZOOKEEPER-965.patch, ZOOKEEPER-965.patch, ZOOKEEPER-965.patch, ZOOKEEPER-965.patch, ZOOKEEPER-965.patch, ZOOKEEPER-965.patch, ZOOKEEPER-965.patch, ZOOKEEPER-965.patch, ZOOKEEPER-965.patch, ZOOKEEPER-965.patch The basic idea is to have a single method called multi that will accept a list of create, delete, update or check objects each of which has a desired version or file state in the case of create. If all of the version and existence constraints can be satisfied, then all updates will be done atomically. Two API styles have been suggested. One has a list as above and the other style has a Transaction that allows builder-like methods to build a set of updates and a commit method to finalize the transaction. This can trivially be reduced to the first kind of API so the list based API style should be considered the primitive and the builder style should be implemented as syntactic sugar. The total size of all the data in all updates and creates in a single transaction should be limited to 1MB. Implementation-wise this capability can be done using standard ZK internals. The changes include: - update to ZK clients to all the new call - additional wire level request - on the server, in the code that converts transactions to idempotent form, the code should be slightly extended to convert a list of operations to idempotent form. - on the client, a down-rev server that rejects the multi-update should be detected gracefully and an informative exception should be thrown. To facilitate shared development, I have established a github repository at https://github.com/tdunning/zookeeper and am happy to extend committer status to anyone who agrees to donate their code back to Apache. The final patch will be attached to this bug as normal. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1059) stat command isses on non-existing node causes NPE
[
https://issues.apache.org/jira/browse/ZOOKEEPER-1059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13032584#comment-13032584
]
Mahadev konar commented on ZOOKEEPER-1059:
--
bhallamudi,
Would you mind putting up a patch for it?
stat command isses on non-existing node causes NPE
---
Key: ZOOKEEPER-1059
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1059
Project: ZooKeeper
Issue Type: Bug
Components: java client
Reporter: Bhallamudi Venkata Siva Kamesh
Fix For: 3.4.0
*stat* command issues on non existing zookeeper node,causes NPE to the client.
{noformat}
[zk: localhost:2181(CONNECTED) 2] stat /invalidPath
Exception in thread main java.lang.NullPointerException
at
org.apache.zookeeper.ZooKeeperMain.printStat(ZooKeeperMain.java:131)
at
org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:723)
at
org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:582)
at
org.apache.zookeeper.ZooKeeperMain.executeLine(ZooKeeperMain.java:354)
at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:312)
at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:271)
{noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (ZOOKEEPER-1059) stat command isses on non-existing node causes NPE
[
https://issues.apache.org/jira/browse/ZOOKEEPER-1059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahadev konar updated ZOOKEEPER-1059:
-
Affects Version/s: (was: 3.4.0)
Fix Version/s: 3.4.0
stat command isses on non-existing node causes NPE
---
Key: ZOOKEEPER-1059
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1059
Project: ZooKeeper
Issue Type: Bug
Components: java client
Reporter: Bhallamudi Venkata Siva Kamesh
Fix For: 3.4.0
*stat* command issues on non existing zookeeper node,causes NPE to the client.
{noformat}
[zk: localhost:2181(CONNECTED) 2] stat /invalidPath
Exception in thread main java.lang.NullPointerException
at
org.apache.zookeeper.ZooKeeperMain.printStat(ZooKeeperMain.java:131)
at
org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:723)
at
org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:582)
at
org.apache.zookeeper.ZooKeeperMain.executeLine(ZooKeeperMain.java:354)
at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:312)
at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:271)
{noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (ZOOKEEPER-1057) zookeeper c-client, connection to offline server fails to successfully fallback to second zk host
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated ZOOKEEPER-1057: - Fix Version/s: 3.4.0 zookeeper c-client, connection to offline server fails to successfully fallback to second zk host - Key: ZOOKEEPER-1057 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1057 Project: ZooKeeper Issue Type: Bug Components: c client Affects Versions: 3.3.1, 3.3.2, 3.3.3 Environment: snowdutyrise-lm ~/- uname -a Darwin snowdutyrise-lm 9.8.0 Darwin Kernel Version 9.8.0: Wed Jul 15 16:55:01 PDT 2009; root:xnu-1228.15.4~1/RELEASE_I386 i386 also observed on: 2.6.35-28-server 49-Ubuntu SMP Tue Mar 1 14:55:37 UTC 2011 Reporter: Woody Anderson Fix For: 3.3.4, 3.4.0 Hello, I'm a contributor for the node.js zookeeper module: https://github.com/yfinkelstein/node-zookeeper i'm using zk 3.3.3 for the purposes of this issue, but i have validated it fails on 3.3.1 and 3.3.2 i'm having an issue when trying to connect when one of my zookeeper servers is offline. if the first server attempted is online, all is good. if the offline server is attempted first, then the client is never able to connect to _any_ server. inside zookeeper.c a connection loss (-4) is received, the socket is closed and buffers are cleaned up, it then attempts the next server in the list, creates a new socket (which gets the same fd as the previously closed socket) and connecting fails, and it continues to fail seemingly forever. The nature of this fail is not that it gets -4 connection loss errors, but that zookeeper_interest doesn't find anything going on on the socket before the user provided timeout kicks things out. I don't want to have to wait 5 minutes, even if i could make myself. this is the message that follows the connection loss: 2011-04-27 23:18:28,355:13485:ZOO_ERROR@handle_socket_error_msg@1530: Socket [127.0.0.1:5020] zk retcode=-7, errno=60(Operation timed out): connection timed out (exceeded timeout by 3ms) 2011-04-27 23:18:28,355:13485:ZOO_ERROR@yield@213: yield:zookeeper_interest returned error: -7 - operation timeout While investigating, i decided to comment out close(zh-fd) in handle_error (zookeeper.c#1153) now everything works (obviously i'm leaking an fd). Connection the the second host works immediately. this is the behavior i'm looking for, though i clearly don't want to leak the fd, so i'm wondering why the fd re-use is causing this issue. close() is not returning an error (i checked even though current code assumes success). i'm on osx 10.6.7 i tried adding a setsockopt so_linger (though i didn't want that to be a solution), it didn't work. full debug traces are included in issue here: https://github.com/yfinkelstein/node-zookeeper/issues/6 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-965) Need a multi-update command to allow multiple znodes to be updated safely
[ https://issues.apache.org/jira/browse/ZOOKEEPER-965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13032647#comment-13032647 ] Ted Dunning commented on ZOOKEEPER-965: --- Stephen, Nice work on the code review. Marshall will have to comment on your points, but it is clear that you read the code well and deserve a stroke for that! Need a multi-update command to allow multiple znodes to be updated safely - Key: ZOOKEEPER-965 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-965 Project: ZooKeeper Issue Type: Bug Affects Versions: 3.3.3 Reporter: Ted Dunning Assignee: Ted Dunning Fix For: 3.4.0 Attachments: ZOOKEEPER-965.patch, ZOOKEEPER-965.patch, ZOOKEEPER-965.patch, ZOOKEEPER-965.patch, ZOOKEEPER-965.patch, ZOOKEEPER-965.patch, ZOOKEEPER-965.patch, ZOOKEEPER-965.patch, ZOOKEEPER-965.patch, ZOOKEEPER-965.patch The basic idea is to have a single method called multi that will accept a list of create, delete, update or check objects each of which has a desired version or file state in the case of create. If all of the version and existence constraints can be satisfied, then all updates will be done atomically. Two API styles have been suggested. One has a list as above and the other style has a Transaction that allows builder-like methods to build a set of updates and a commit method to finalize the transaction. This can trivially be reduced to the first kind of API so the list based API style should be considered the primitive and the builder style should be implemented as syntactic sugar. The total size of all the data in all updates and creates in a single transaction should be limited to 1MB. Implementation-wise this capability can be done using standard ZK internals. The changes include: - update to ZK clients to all the new call - additional wire level request - on the server, in the code that converts transactions to idempotent form, the code should be slightly extended to convert a list of operations to idempotent form. - on the client, a down-rev server that rejects the multi-update should be detected gracefully and an informative exception should be thrown. To facilitate shared development, I have established a github repository at https://github.com/tdunning/zookeeper and am happy to extend committer status to anyone who agrees to donate their code back to Apache. The final patch will be attached to this bug as normal. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-965) Need a multi-update command to allow multiple znodes to be updated safely
[ https://issues.apache.org/jira/browse/ZOOKEEPER-965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13032676#comment-13032676 ] Marshall McMullen commented on ZOOKEEPER-965: - Ted, I pushed my latest changes to fix the things Stephen identified in his review. Need a multi-update command to allow multiple znodes to be updated safely - Key: ZOOKEEPER-965 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-965 Project: ZooKeeper Issue Type: Bug Affects Versions: 3.3.3 Reporter: Ted Dunning Assignee: Ted Dunning Fix For: 3.4.0 Attachments: ZOOKEEPER-965.patch, ZOOKEEPER-965.patch, ZOOKEEPER-965.patch, ZOOKEEPER-965.patch, ZOOKEEPER-965.patch, ZOOKEEPER-965.patch, ZOOKEEPER-965.patch, ZOOKEEPER-965.patch, ZOOKEEPER-965.patch, ZOOKEEPER-965.patch The basic idea is to have a single method called multi that will accept a list of create, delete, update or check objects each of which has a desired version or file state in the case of create. If all of the version and existence constraints can be satisfied, then all updates will be done atomically. Two API styles have been suggested. One has a list as above and the other style has a Transaction that allows builder-like methods to build a set of updates and a commit method to finalize the transaction. This can trivially be reduced to the first kind of API so the list based API style should be considered the primitive and the builder style should be implemented as syntactic sugar. The total size of all the data in all updates and creates in a single transaction should be limited to 1MB. Implementation-wise this capability can be done using standard ZK internals. The changes include: - update to ZK clients to all the new call - additional wire level request - on the server, in the code that converts transactions to idempotent form, the code should be slightly extended to convert a list of operations to idempotent form. - on the client, a down-rev server that rejects the multi-update should be detected gracefully and an informative exception should be thrown. To facilitate shared development, I have established a github repository at https://github.com/tdunning/zookeeper and am happy to extend committer status to anyone who agrees to donate their code back to Apache. The final patch will be attached to this bug as normal. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (ZOOKEEPER-1055) check for duplicate ACLs in addACL() and create()
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koontz updated ZOOKEEPER-1055: - Attachment: ZOOKEEPER-1055.patch This is not a fix yet; but is simply a unit test that shows the bug (this test fails on trunk). check for duplicate ACLs in addACL() and create() - Key: ZOOKEEPER-1055 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1055 Project: ZooKeeper Issue Type: Bug Reporter: Eugene Koontz Assignee: Eugene Koontz Attachments: ZOOKEEPER-1055.patch actual result: [zk: (CONNECTED) 0] create /test2 'test2' digest:test:test:cdrwa,digest:test:test:cdrwa Created /test2 [zk: (CONNECTED) 1] getAcl /test2 'digest,'test:test : cdrwa 'digest,'test:test : cdrwa [zk: (CONNECTED) 2] but getAcl should only have a single entry. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (ZOOKEEPER-1055) check for duplicate ACLs in addACL() and create()
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koontz updated ZOOKEEPER-1055: - Attachment: (was: ZOOKEEPER-1055.patch) check for duplicate ACLs in addACL() and create() - Key: ZOOKEEPER-1055 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1055 Project: ZooKeeper Issue Type: Bug Reporter: Eugene Koontz Assignee: Eugene Koontz Attachments: ZOOKEEPER-1055.patch actual result: [zk: (CONNECTED) 0] create /test2 'test2' digest:test:test:cdrwa,digest:test:test:cdrwa Created /test2 [zk: (CONNECTED) 1] getAcl /test2 'digest,'test:test : cdrwa 'digest,'test:test : cdrwa [zk: (CONNECTED) 2] but getAcl should only have a single entry. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1055) check for duplicate ACLs in addACL() and create()
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13032688#comment-13032688 ] Eugene Koontz commented on ZOOKEEPER-1055: -- Correction: sometimes fails on trunk, sometimes succeeds. check for duplicate ACLs in addACL() and create() - Key: ZOOKEEPER-1055 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1055 Project: ZooKeeper Issue Type: Bug Reporter: Eugene Koontz Assignee: Eugene Koontz Attachments: ZOOKEEPER-1055.patch actual result: [zk: (CONNECTED) 0] create /test2 'test2' digest:test:test:cdrwa,digest:test:test:cdrwa Created /test2 [zk: (CONNECTED) 1] getAcl /test2 'digest,'test:test : cdrwa 'digest,'test:test : cdrwa [zk: (CONNECTED) 2] but getAcl should only have a single entry. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-965) Need a multi-update command to allow multiple znodes to be updated safely
[ https://issues.apache.org/jira/browse/ZOOKEEPER-965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13032733#comment-13032733 ] jirapos...@reviews.apache.org commented on ZOOKEEPER-965: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/739/ --- Review request for zookeeper and Benjamin Reed. Summary --- This mega-patch adds the multi-op capability to ZK. This allows a batch of create, delete, update or version-check operations to succeed or fail together. Both C and java bindings are provided. This addresses bug ZOOKEEPER-965. https://issues.apache.org/jira/browse/ZOOKEEPER-965 Diffs - CHANGES.txt d22f1f0 src/c/Makefile.am 65830fe src/c/configure.ac cdea898 src/c/include/proto.h 843032f src/c/include/zookeeper.h c055edb src/c/src/zookeeper.c db715b0 src/c/tests/TestMulti.cc PRE-CREATION src/java/main/org/apache/zookeeper/MultiResponse.java PRE-CREATION src/java/main/org/apache/zookeeper/MultiTransactionRecord.java PRE-CREATION src/java/main/org/apache/zookeeper/Op.java PRE-CREATION src/java/main/org/apache/zookeeper/OpResult.java PRE-CREATION src/java/main/org/apache/zookeeper/Transaction.java PRE-CREATION src/java/main/org/apache/zookeeper/ZooDefs.java 832976f src/java/main/org/apache/zookeeper/ZooKeeper.java 00b4012 src/java/main/org/apache/zookeeper/server/DataTree.java d16537e src/java/main/org/apache/zookeeper/server/FinalRequestProcessor.java 2538cf7 src/java/main/org/apache/zookeeper/server/NIOServerCnxn.java 68970d2 src/java/main/org/apache/zookeeper/server/PrepRequestProcessor.java 50f208d src/java/main/org/apache/zookeeper/server/Request.java a5c57e2 src/java/main/org/apache/zookeeper/server/RequestProcessor.java 5c3e8ff src/java/main/org/apache/zookeeper/server/TraceFormatter.java 8ece929 src/java/main/org/apache/zookeeper/server/package.html 3ec7656 src/java/main/org/apache/zookeeper/server/quorum/CommitProcessor.java 49affd5 src/java/main/org/apache/zookeeper/server/quorum/QuorumPeer.java 9774fb2 src/java/main/org/apache/zookeeper/server/util/SerializeUtils.java 0ad4dd6 src/java/test/org/apache/zookeeper/MultiResponseTest.java PRE-CREATION src/java/test/org/apache/zookeeper/MultiTransactionRecordTest.java PRE-CREATION src/java/test/org/apache/zookeeper/test/MultiTransactionTest.java PRE-CREATION src/zookeeper.jute 7d96f32 Diff: https://reviews.apache.org/r/739/diff Testing --- A number of unit tests have been implemented. Test coverage is very good. A sample application has been written to do simple operations on a graph of nodes. Thanks, Ted Need a multi-update command to allow multiple znodes to be updated safely - Key: ZOOKEEPER-965 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-965 Project: ZooKeeper Issue Type: Bug Affects Versions: 3.3.3 Reporter: Ted Dunning Assignee: Ted Dunning Fix For: 3.4.0 Attachments: ZOOKEEPER-965.patch, ZOOKEEPER-965.patch, ZOOKEEPER-965.patch, ZOOKEEPER-965.patch, ZOOKEEPER-965.patch, ZOOKEEPER-965.patch, ZOOKEEPER-965.patch, ZOOKEEPER-965.patch, ZOOKEEPER-965.patch, ZOOKEEPER-965.patch The basic idea is to have a single method called multi that will accept a list of create, delete, update or check objects each of which has a desired version or file state in the case of create. If all of the version and existence constraints can be satisfied, then all updates will be done atomically. Two API styles have been suggested. One has a list as above and the other style has a Transaction that allows builder-like methods to build a set of updates and a commit method to finalize the transaction. This can trivially be reduced to the first kind of API so the list based API style should be considered the primitive and the builder style should be implemented as syntactic sugar. The total size of all the data in all updates and creates in a single transaction should be limited to 1MB. Implementation-wise this capability can be done using standard ZK internals. The changes include: - update to ZK clients to all the new call - additional wire level request - on the server, in the code that converts transactions to idempotent form, the code should be slightly extended to convert a list of operations to idempotent form. - on the client, a down-rev server that rejects the multi-update should be detected gracefully and an informative exception should be thrown. To facilitate shared development, I have established a github repository at
Review Request: ZOOKEEPER-965 - Add multi operation
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/739/ --- Review request for zookeeper and Benjamin Reed. Summary --- This mega-patch adds the multi-op capability to ZK. This allows a batch of create, delete, update or version-check operations to succeed or fail together. Both C and java bindings are provided. This addresses bug ZOOKEEPER-965. https://issues.apache.org/jira/browse/ZOOKEEPER-965 Diffs - CHANGES.txt d22f1f0 src/c/Makefile.am 65830fe src/c/configure.ac cdea898 src/c/include/proto.h 843032f src/c/include/zookeeper.h c055edb src/c/src/zookeeper.c db715b0 src/c/tests/TestMulti.cc PRE-CREATION src/java/main/org/apache/zookeeper/MultiResponse.java PRE-CREATION src/java/main/org/apache/zookeeper/MultiTransactionRecord.java PRE-CREATION src/java/main/org/apache/zookeeper/Op.java PRE-CREATION src/java/main/org/apache/zookeeper/OpResult.java PRE-CREATION src/java/main/org/apache/zookeeper/Transaction.java PRE-CREATION src/java/main/org/apache/zookeeper/ZooDefs.java 832976f src/java/main/org/apache/zookeeper/ZooKeeper.java 00b4012 src/java/main/org/apache/zookeeper/server/DataTree.java d16537e src/java/main/org/apache/zookeeper/server/FinalRequestProcessor.java 2538cf7 src/java/main/org/apache/zookeeper/server/NIOServerCnxn.java 68970d2 src/java/main/org/apache/zookeeper/server/PrepRequestProcessor.java 50f208d src/java/main/org/apache/zookeeper/server/Request.java a5c57e2 src/java/main/org/apache/zookeeper/server/RequestProcessor.java 5c3e8ff src/java/main/org/apache/zookeeper/server/TraceFormatter.java 8ece929 src/java/main/org/apache/zookeeper/server/package.html 3ec7656 src/java/main/org/apache/zookeeper/server/quorum/CommitProcessor.java 49affd5 src/java/main/org/apache/zookeeper/server/quorum/QuorumPeer.java 9774fb2 src/java/main/org/apache/zookeeper/server/util/SerializeUtils.java 0ad4dd6 src/java/test/org/apache/zookeeper/MultiResponseTest.java PRE-CREATION src/java/test/org/apache/zookeeper/MultiTransactionRecordTest.java PRE-CREATION src/java/test/org/apache/zookeeper/test/MultiTransactionTest.java PRE-CREATION src/zookeeper.jute 7d96f32 Diff: https://reviews.apache.org/r/739/diff Testing --- A number of unit tests have been implemented. Test coverage is very good. A sample application has been written to do simple operations on a graph of nodes. Thanks, Ted
[jira] [Updated] (ZOOKEEPER-1055) check for duplicate ACLs in addACL() and create()
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koontz updated ZOOKEEPER-1055: - Attachment: ZOOKEEPER-1055.patch check for duplicate ACLs in addACL() and create() - Key: ZOOKEEPER-1055 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1055 Project: ZooKeeper Issue Type: Bug Affects Versions: 3.4.0 Reporter: Eugene Koontz Assignee: Eugene Koontz Fix For: 3.4.0 Attachments: ZOOKEEPER-1055.patch, ZOOKEEPER-1055.patch actual result: [zk: (CONNECTED) 0] create /test2 'test2' digest:test:test:cdrwa,digest:test:test:cdrwa Created /test2 [zk: (CONNECTED) 1] getAcl /test2 'digest,'test:test : cdrwa 'digest,'test:test : cdrwa [zk: (CONNECTED) 2] but getAcl should only have a single entry. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1055) check for duplicate ACLs in addACL() and create()
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13032778#comment-13032778 ] Eugene Koontz commented on ZOOKEEPER-1055: -- With patch, unit test passes consistently. check for duplicate ACLs in addACL() and create() - Key: ZOOKEEPER-1055 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1055 Project: ZooKeeper Issue Type: Bug Affects Versions: 3.4.0 Reporter: Eugene Koontz Assignee: Eugene Koontz Fix For: 3.4.0 Attachments: ZOOKEEPER-1055.patch, ZOOKEEPER-1055.patch actual result: [zk: (CONNECTED) 0] create /test2 'test2' digest:test:test:cdrwa,digest:test:test:cdrwa Created /test2 [zk: (CONNECTED) 1] getAcl /test2 'digest,'test:test : cdrwa 'digest,'test:test : cdrwa [zk: (CONNECTED) 2] but getAcl should only have a single entry. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1046) Creating a new sequential node results in a ZNODEEXISTS error
[
https://issues.apache.org/jira/browse/ZOOKEEPER-1046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13032777#comment-13032777
]
Vishal K commented on ZOOKEEPER-1046:
-
I will upload a test soon.
Creating a new sequential node results in a ZNODEEXISTS error
-
Key: ZOOKEEPER-1046
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1046
Project: ZooKeeper
Issue Type: Bug
Components: server
Affects Versions: 3.3.2, 3.3.3
Environment: A 3 node-cluster running Debian squeeze.
Reporter: Jeremy Stribling
Labels: sequence
Fix For: 3.4.0
Attachments: ZOOKEEPER-1046.patch, ZOOKEEPER-1046.patch,
ZOOKEEPER-1046.tgz
On several occasions, I've seen a create() with the sequential flag set fail
with a ZNODEEXISTS error, and I don't think that should ever be possible. In
past runs, I've been able to closely inspect the state of the system with the
command line client, and saw that the parent znode's cversion is smaller than
the sequential number of existing children znode under that parent. In one
example:
{noformat}
[zk:ip:port(CONNECTED) 3] stat /zkrsm
cZxid = 0x5
ctime = Mon Jan 17 18:28:19 PST 2011
mZxid = 0x5
mtime = Mon Jan 17 18:28:19 PST 2011
pZxid = 0x1d819
cversion = 120710
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 0
numChildren = 2955
{noformat}
However, the znode /zkrsm/002d_record120804 existed on disk.
In a recent run, I was able to capture the Zookeeper logs, and I will attach
them to this JIRA. The logs are named as nodeX.zxid_prefixes.log, and each
new log represents an application process restart.
Here's the scenario:
# There's a cluster with nodes 1,2,3 using zxid 0x3.
# All three nodes restart, forming a cluster of zxid 0x4.
# Node 3 restarts, leading to a cluster of 0x5.
At this point, it seems like node 1 is the leader of the 0x5 epoch. In its
log (node1.0x4-0x5.log) you can see the first (of many) instances of the
following message:
{noformat}
2011-04-11 21:16:12,607 16649 [ProcessThread:-1] INFO
org.apache.zookeeper.server.PrepRequestProcessor - Got user-level
KeeperException when processing sessionid:0x512f466bd44e0002 type:create
cxid:0x4da376ab zxid:0xfffe txntype:unknown reqpath:n/a Error
Path:/zkrsm/00b2_record0001761440 Error:KeeperErrorCode =
NodeExists for /zkrsm/00b2_record0001761440
{noformat}
This then repeats forever as my application isn't expecting to ever get this
error message on a sequential node create, and just continually retries. The
message even transfers over to node3.0x5-0x6.log once the 0x6 epoch comes
into play.
I don't see anything terribly fishy in the transition between the epochs; the
correct snapshots seem to be getting transferred, etc. Unfortunately I don't
have a ZK snapshot/log that exhibits the problem when starting with a fresh
system.
Some oddities you might notice in these logs:
* Between epochs 0x3 and 0x4, the zookeeper IDs of the nodes changed due to a
bug in our application code. (They are assigned randomly, but are supposed
to be consistent across restarts.)
* We manage node membership dynamically, and our application restarts the
ZooKeeperServer classes whenever a new node wants to join (without restarting
the entire application process). This is why you'll see messages like the
following in node1.0x4-0x5.log before a new election begins:
{noformat}
2011-04-11 21:16:00,762 4804 [QuorumPeer:/0.0.0.0:2888] INFO
org.apache.zookeeper.server.quorum.Learner - shutdown called
{noformat}
* There is in fact one of these dynamic membership changes in
node1.0x4-0x5.log, just before the 0x4 epoch is formed. I'm not sure how
this would be related though, as no transactions are done during this period.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1055) check for duplicate ACLs in addACL() and create()
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13032779#comment-13032779 ] Eugene Koontz commented on ZOOKEEPER-1055: -- I noticed that we use SelectionKey.attach() to attach authInfo information to ClientCnxns. A bit of googling led me to this blog post: http://jfarcand.wordpress.com/2006/07/06/tricks-and-tips-with-nio-part-ii-why-selectionkey-attach-is-evil/; I'm not sure if this blog post is relevant, but thought I'd mention it. check for duplicate ACLs in addACL() and create() - Key: ZOOKEEPER-1055 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1055 Project: ZooKeeper Issue Type: Bug Affects Versions: 3.4.0 Reporter: Eugene Koontz Assignee: Eugene Koontz Fix For: 3.4.0 Attachments: ZOOKEEPER-1055.patch, ZOOKEEPER-1055.patch actual result: [zk: (CONNECTED) 0] create /test2 'test2' digest:test:test:cdrwa,digest:test:test:cdrwa Created /test2 [zk: (CONNECTED) 1] getAcl /test2 'digest,'test:test : cdrwa 'digest,'test:test : cdrwa [zk: (CONNECTED) 2] but getAcl should only have a single entry. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1027) chroot not transparent in zoo_create()
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13032808#comment-13032808 ] Thijs Terlouw commented on ZOOKEEPER-1027: -- I think just for 3.4 is good, because it changes the behavior of the C-client a little bit and that should probably not happen in a small patch release such as 3.3.3 About the LOG_ERROR() message: I didn't look in detail at the code-path for creating, so I didn't consider this case. It's a non-functional change, so removing it is no problem for me :) chroot not transparent in zoo_create() -- Key: ZOOKEEPER-1027 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1027 Project: ZooKeeper Issue Type: Bug Components: c client Affects Versions: 3.3.3 Environment: Linux, ZooKeeper 3.3.3, C-client, java 1.6.0_17-b04, hotspot server vm Reporter: Thijs Terlouw Assignee: Thijs Terlouw Fix For: 3.4.0 Attachments: ZOOKEEPER-1027-TRUNK_WITH_TESTS.patch I've recently started to use the chroot functionality (introduced in 3.2.0) as part of my connect string.It mostly works as expected, but there is one case that is unexpected: when I create a path with zoo_create() I can retrieve the created path. This is very useful when you set the ZOO_SEQUENCE flag. Unfortunately the returned path includes the chroot as part of the path. This was unexpected to me: I expected that the chroot would be totally transparent. The documentation for zoo_create() says: path_buffer : Buffer which will be filled with the path of the new node (this might be different than the supplied path because of the ZOO_SEQUENCE flag). This gave me the impression that this flag is the only reason the returned path is different from the created path, but apparently it's not. Is this a bug or intended behavior? I workaround this issue now by remembering the chroot in my wrapper code and after a call to zoo_create() i check if the returned path starts with the chroot. If it does, I remove it. My use case is to create a path with a sequence number and then delete this path later. Unfortunately I cannot delete the path because it has the chroot prepended to it, and thus it will result in two chroots. I believe this only affects the create functions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Issues on java lock recipe
Hello,
My colleagues and I are working with the java lock recipe implementation.
We think we found two bugs in the code:
1) The first one is reported on this jira
topichttps://issues.apache.org/jira/browse/ZOOKEEPER-645.
The issue is that the znodes used to control the lock are ordered by
sessionID first and then by the sequence number. As earlier connected
clients appear to have lower sessionID values than those connected latter,
who connects first gets the lock disregarding anyone who has already the
lock.
We've posted a patch on that jira but it has not yet been reviewed.
2) The other bug is when you try to unlock. When calling unlock(), either
having the lock held or waiting for it, the znode lock is removed. However,
if you are not holding the lock, there's still a zookeeper watcher waiting
on the next znode with lower sequence number, which is necessary to avoid
the heard effect on the recipe implementation. When watcher tells the lock
implementation that the watched znode has been removed, the lock recipe
calls lock(). What happens then is that a new znode lock is created so the
client is again (unwilling) waiting for the lock (or eventually is now
holding the lock). If that client doesn't do anything (such as unlocking it
over and over until eventually getting the lock and doing an ultimate
unlock) there would be a deadlock due to the client having the lock without
knowing it.
We've come up with a patch for this (it's attached to this email). Our
question is: should we post this patch on the same jira topic mentioned on
the beginning of this email or should we open a new topic for this issue?
Thanks,
Andre Esteve
http://www.lsd.ic.unicamp.br/mc715-1s2011/index.php/Main_Page (wiki in
Portuguese about our (and other's) works using zookeeper)
Index: WriteLock.java
===
--- WriteLock.java (revision 1102068)
+++ WriteLock.java (working copy)
@@ -152,7 +152,10 @@
LOG.debug(Watcher fired on path: + event.getPath() + state: +
event.getState() + type + event.getType());
try {
-lock();
+// avoid locking when not waiting for it
+if (id != null) {
+lock();
+}
} catch (Exception e) {
LOG.warn(Failed to acquire lock: + e, e);
}
[jira] [Created] (ZOOKEEPER-1062) Net-ZooKeeper: Net::ZooKeeper consumes 100% cpu on wait
Net-ZooKeeper: Net::ZooKeeper consumes 100% cpu on wait --- Key: ZOOKEEPER-1062 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1062 Project: ZooKeeper Issue Type: Bug Components: contrib-bindings Affects Versions: 3.3.1 Reporter: Patrick Hunt Reported by a user on the CDH user list (user reports that the listed fix addressed this issue for him): Net::ZooKeeper consumes 100% cpu when wait is used. At my initial inspection, it seems to be related to implementation mistake in pthread_cond_timedwait. https://rt.cpan.org/Public/Bug/Display.html?id=61290 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
