[jira] Created: (ZOOKEEPER-1018) The connection permutation in get_addrs uses a weak and inefficient shuffle
The connection permutation in get_addrs uses a weak and inefficient shuffle
---
Key: ZOOKEEPER-1018
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1018
Project: ZooKeeper
Issue Type: Improvement
Components: c client
Affects Versions: 3.3.2
Reporter: Stephen Tyree
Priority: Minor
After determining all of the addresses in the get_addrs function in the C
client, the connection is permuted using the following code:
setup_random();
/* Permute */
for(i = 0; i < zh->addrs_count; i++) {
struct sockaddr_storage *s1 = zh->addrs + random()%zh->addrs_count;
struct sockaddr_storage *s2 = zh->addrs + random()%zh->addrs_count;
if (s1 != s2) {
struct sockaddr_storage t = *s1;
*s1 = *s2;
*s2 = t;
}
}
Not only does this shuffle produce an uneven permutation, but it is half as
efficient as the Fisher-Yates shuffle which produces an unbiased one. It seems
like it would be a simple fix to increase the randomness and efficiency of the
shuffle by switching over to using Fisher-Yates.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (ZOOKEEPER-1018) The connection permutation in get_addrs uses a weak and inefficient shuffle
[
https://issues.apache.org/jira/browse/ZOOKEEPER-1018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen Tyree updated ZOOKEEPER-1018:
-
Attachment: zookeeper.c.patch
Patch to implement the change. I'm not sure if I diff'd this correctly, but
this is it.
> The connection permutation in get_addrs uses a weak and inefficient shuffle
> ---
>
> Key: ZOOKEEPER-1018
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1018
> Project: ZooKeeper
> Issue Type: Improvement
> Components: c client
>Affects Versions: 3.3.2
>Reporter: Stephen Tyree
>Priority: Minor
> Attachments: zookeeper.c.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> After determining all of the addresses in the get_addrs function in the C
> client, the connection is permuted using the following code:
> setup_random();
> /* Permute */
> for(i = 0; i < zh->addrs_count; i++) {
> struct sockaddr_storage *s1 = zh->addrs +
> random()%zh->addrs_count;
> struct sockaddr_storage *s2 = zh->addrs +
> random()%zh->addrs_count;
> if (s1 != s2) {
> struct sockaddr_storage t = *s1;
> *s1 = *s2;
> *s2 = t;
> }
> }
> Not only does this shuffle produce an uneven permutation, but it is half as
> efficient as the Fisher-Yates shuffle which produces an unbiased one. It
> seems like it would be a simple fix to increase the randomness and efficiency
> of the shuffle by switching over to using Fisher-Yates.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Created: (ZOOKEEPER-1020) Implement function in C client to determine which host you're currently connected to.
Implement function in C client to determine which host you're currently
connected to.
-
Key: ZOOKEEPER-1020
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1020
Project: ZooKeeper
Issue Type: New Feature
Components: c client
Reporter: Stephen Tyree
Priority: Minor
On occasion it might be useful to determine which host your Zookeeper client is
currently connected to, be it for debugging purposes or otherwise. A possible
signature for that function:
const char* zoo_get_connected_host(zhandle_t *zh, char *buffer, size_t
buffer_size, unsigned short *port);
Clients could use it like below:
char buffer[33];
unsigned short port = 0;
if (!zoo_get_connected_host(zh, buffer, sizeof(buffer), &port))
return EXIT_FAILURE;
printf("The connected host is: %s:%d\n", buffer, port);
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (ZOOKEEPER-1020) Implement function in C client to determine which host you're currently connected to.
[
https://issues.apache.org/jira/browse/ZOOKEEPER-1020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen Tyree updated ZOOKEEPER-1020:
-
Attachment: ZOOKEEPER-1020.patch
Patch to add functionality.
> Implement function in C client to determine which host you're currently
> connected to.
> -
>
> Key: ZOOKEEPER-1020
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1020
> Project: ZooKeeper
> Issue Type: New Feature
> Components: c client
>Reporter: Stephen Tyree
>Priority: Minor
> Attachments: ZOOKEEPER-1020.patch
>
>
> On occasion it might be useful to determine which host your Zookeeper client
> is currently connected to, be it for debugging purposes or otherwise. A
> possible signature for that function:
> const char* zoo_get_connected_host(zhandle_t *zh, char *buffer, size_t
> buffer_size, unsigned short *port);
> Clients could use it like below:
> char buffer[33];
> unsigned short port = 0;
> if (!zoo_get_connected_host(zh, buffer, sizeof(buffer), &port))
> return EXIT_FAILURE;
> printf("The connected host is: %s:%d\n", buffer, port);
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (ZOOKEEPER-1020) Implement function in C client to determine which host you're currently connected to.
[
https://issues.apache.org/jira/browse/ZOOKEEPER-1020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen Tyree updated ZOOKEEPER-1020:
-
Attachment: ZOOKEEPER-1020.patch
This time with proper spacing.
> Implement function in C client to determine which host you're currently
> connected to.
> -
>
> Key: ZOOKEEPER-1020
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1020
> Project: ZooKeeper
> Issue Type: New Feature
> Components: c client
>Reporter: Stephen Tyree
>Priority: Minor
> Attachments: ZOOKEEPER-1020.patch
>
>
> On occasion it might be useful to determine which host your Zookeeper client
> is currently connected to, be it for debugging purposes or otherwise. A
> possible signature for that function:
> const char* zoo_get_connected_host(zhandle_t *zh, char *buffer, size_t
> buffer_size, unsigned short *port);
> Clients could use it like below:
> char buffer[33];
> unsigned short port = 0;
> if (!zoo_get_connected_host(zh, buffer, sizeof(buffer), &port))
> return EXIT_FAILURE;
> printf("The connected host is: %s:%d\n", buffer, port);
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (ZOOKEEPER-1020) Implement function in C client to determine which host you're currently connected to.
[
https://issues.apache.org/jira/browse/ZOOKEEPER-1020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen Tyree updated ZOOKEEPER-1020:
-
Attachment: (was: ZOOKEEPER-1020.patch)
> Implement function in C client to determine which host you're currently
> connected to.
> -
>
> Key: ZOOKEEPER-1020
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1020
> Project: ZooKeeper
> Issue Type: New Feature
> Components: c client
>Reporter: Stephen Tyree
>Priority: Minor
> Attachments: ZOOKEEPER-1020.patch
>
>
> On occasion it might be useful to determine which host your Zookeeper client
> is currently connected to, be it for debugging purposes or otherwise. A
> possible signature for that function:
> const char* zoo_get_connected_host(zhandle_t *zh, char *buffer, size_t
> buffer_size, unsigned short *port);
> Clients could use it like below:
> char buffer[33];
> unsigned short port = 0;
> if (!zoo_get_connected_host(zh, buffer, sizeof(buffer), &port))
> return EXIT_FAILURE;
> printf("The connected host is: %s:%d\n", buffer, port);
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (ZOOKEEPER-1018) The connection permutation in get_addrs uses a weak and inefficient shuffle
[
https://issues.apache.org/jira/browse/ZOOKEEPER-1018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13006962#comment-13006962
]
Stephen Tyree commented on ZOOKEEPER-1018:
--
Although I am less familiar with the Java code, I found this snippet in
org/main/apache/zookeeper/client/StaticHostProvider.java:
Collections.shuffle(this.serverAddresses);
This tells me that the Java library lacks this problem. Do you want me to
package this up more neatly with an svn diff?
> The connection permutation in get_addrs uses a weak and inefficient shuffle
> ---
>
> Key: ZOOKEEPER-1018
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1018
> Project: ZooKeeper
> Issue Type: Improvement
> Components: c client
>Affects Versions: 3.3.2
>Reporter: Stephen Tyree
>Priority: Minor
> Attachments: zookeeper.c.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> After determining all of the addresses in the get_addrs function in the C
> client, the connection is permuted using the following code:
> setup_random();
> /* Permute */
> for(i = 0; i < zh->addrs_count; i++) {
> struct sockaddr_storage *s1 = zh->addrs +
> random()%zh->addrs_count;
> struct sockaddr_storage *s2 = zh->addrs +
> random()%zh->addrs_count;
> if (s1 != s2) {
> struct sockaddr_storage t = *s1;
> *s1 = *s2;
> *s2 = t;
> }
> }
> Not only does this shuffle produce an uneven permutation, but it is half as
> efficient as the Fisher-Yates shuffle which produces an unbiased one. It
> seems like it would be a simple fix to increase the randomness and efficiency
> of the shuffle by switching over to using Fisher-Yates.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (ZOOKEEPER-1018) The connection permutation in get_addrs uses a weak and inefficient shuffle
[
https://issues.apache.org/jira/browse/ZOOKEEPER-1018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13006969#comment-13006969
]
Stephen Tyree commented on ZOOKEEPER-1018:
--
Running 1000 times with the old way, I get the following distribution:
308 The connected host is: (address 1):2181
369 The connected host is: (address 2):2181
324 The connected host is: (address 3):2181
Running 1000 times with the new way, I get the following distribution:
333 The connected host is: (address 1):2181
325 The connected host is: (address 2):2181
343 The connected host is: (address 3):2181
Hardly scientific, but an indication that the new way is better or at the least
as good.
> The connection permutation in get_addrs uses a weak and inefficient shuffle
> ---
>
> Key: ZOOKEEPER-1018
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1018
> Project: ZooKeeper
> Issue Type: Improvement
> Components: c client
>Affects Versions: 3.3.2
>Reporter: Stephen Tyree
>Priority: Minor
> Attachments: zookeeper.c.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> After determining all of the addresses in the get_addrs function in the C
> client, the connection is permuted using the following code:
> setup_random();
> /* Permute */
> for(i = 0; i < zh->addrs_count; i++) {
> struct sockaddr_storage *s1 = zh->addrs +
> random()%zh->addrs_count;
> struct sockaddr_storage *s2 = zh->addrs +
> random()%zh->addrs_count;
> if (s1 != s2) {
> struct sockaddr_storage t = *s1;
> *s1 = *s2;
> *s2 = t;
> }
> }
> Not only does this shuffle produce an uneven permutation, but it is half as
> efficient as the Fisher-Yates shuffle which produces an unbiased one. It
> seems like it would be a simple fix to increase the randomness and efficiency
> of the shuffle by switching over to using Fisher-Yates.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (ZOOKEEPER-1018) The connection permutation in get_addrs uses a weak and inefficient shuffle
[
https://issues.apache.org/jira/browse/ZOOKEEPER-1018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen Tyree updated ZOOKEEPER-1018:
-
Attachment: ZOOKEEPER-1018.patch
Patch to implement the enhancement.
> The connection permutation in get_addrs uses a weak and inefficient shuffle
> ---
>
> Key: ZOOKEEPER-1018
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1018
> Project: ZooKeeper
> Issue Type: Improvement
> Components: c client
>Affects Versions: 3.3.2
>Reporter: Stephen Tyree
>Priority: Minor
> Attachments: ZOOKEEPER-1018.patch, zookeeper.c.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> After determining all of the addresses in the get_addrs function in the C
> client, the connection is permuted using the following code:
> setup_random();
> /* Permute */
> for(i = 0; i < zh->addrs_count; i++) {
> struct sockaddr_storage *s1 = zh->addrs +
> random()%zh->addrs_count;
> struct sockaddr_storage *s2 = zh->addrs +
> random()%zh->addrs_count;
> if (s1 != s2) {
> struct sockaddr_storage t = *s1;
> *s1 = *s2;
> *s2 = t;
> }
> }
> Not only does this shuffle produce an uneven permutation, but it is half as
> efficient as the Fisher-Yates shuffle which produces an unbiased one. It
> seems like it would be a simple fix to increase the randomness and efficiency
> of the shuffle by switching over to using Fisher-Yates.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (ZOOKEEPER-1018) The connection permutation in get_addrs uses a weak and inefficient shuffle
[
https://issues.apache.org/jira/browse/ZOOKEEPER-1018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen Tyree updated ZOOKEEPER-1018:
-
Attachment: (was: zookeeper.c.patch)
> The connection permutation in get_addrs uses a weak and inefficient shuffle
> ---
>
> Key: ZOOKEEPER-1018
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1018
> Project: ZooKeeper
> Issue Type: Improvement
> Components: c client
>Affects Versions: 3.3.2
>Reporter: Stephen Tyree
>Priority: Minor
> Attachments: ZOOKEEPER-1018.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> After determining all of the addresses in the get_addrs function in the C
> client, the connection is permuted using the following code:
> setup_random();
> /* Permute */
> for(i = 0; i < zh->addrs_count; i++) {
> struct sockaddr_storage *s1 = zh->addrs +
> random()%zh->addrs_count;
> struct sockaddr_storage *s2 = zh->addrs +
> random()%zh->addrs_count;
> if (s1 != s2) {
> struct sockaddr_storage t = *s1;
> *s1 = *s2;
> *s2 = t;
> }
> }
> Not only does this shuffle produce an uneven permutation, but it is half as
> efficient as the Fisher-Yates shuffle which produces an unbiased one. It
> seems like it would be a simple fix to increase the randomness and efficiency
> of the shuffle by switching over to using Fisher-Yates.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (ZOOKEEPER-1020) Implement function in C client to determine which host you're currently connected to.
[
https://issues.apache.org/jira/browse/ZOOKEEPER-1020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13007045#comment-13007045
]
Stephen Tyree commented on ZOOKEEPER-1020:
--
Posting discussion with Benjamin Reed while on the IRC channel:
13:19 < breed_zk> it looks good, but i think it might be better to use
sockaddr instead of converting to a string. we should return a null if we
aren't connected. your
patch really tells the host that you should be connected to
rather than which one you are connected to. maybe use getpeername
13:21 < styree> Ok, so the function would return the sockaddr of the
connection, letting the client do any conversion they want to?
13:21 < styree> And calling getpeername on the fd used to talk to Zookeeper
13:24 < styree> Using the sockaddr would remove the need to seperate the
function into getting the host and the port, but how do we feel about the
function name?
13:31 < breed_zk> yes
13:32 < breed_zk> one sec
13:32 < styree> k
13:35 < breed_zk> there is this, perhaps silly, distinction we make with the
naming in C. zoo_ is for standard zookeeper calls, where zookeeper_ is for the
management
API, so by that logic i think it should be called
zookeeper_get_connected_host.
So I'm going to rename the function zookeeper_get_connected_host and have it
return the underlying sockaddr for the connection based on getpeername.
> Implement function in C client to determine which host you're currently
> connected to.
> -
>
> Key: ZOOKEEPER-1020
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1020
> Project: ZooKeeper
> Issue Type: New Feature
> Components: c client
>Reporter: Stephen Tyree
>Priority: Minor
> Attachments: ZOOKEEPER-1020.patch
>
>
> On occasion it might be useful to determine which host your Zookeeper client
> is currently connected to, be it for debugging purposes or otherwise. A
> possible signature for that function:
> const char* zoo_get_connected_host(zhandle_t *zh, char *buffer, size_t
> buffer_size, unsigned short *port);
> Clients could use it like below:
> char buffer[33];
> unsigned short port = 0;
> if (!zoo_get_connected_host(zh, buffer, sizeof(buffer), &port))
> return EXIT_FAILURE;
> printf("The connected host is: %s:%d\n", buffer, port);
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (ZOOKEEPER-1020) Implement function in C client to determine which host you're currently connected to.
[
https://issues.apache.org/jira/browse/ZOOKEEPER-1020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen Tyree updated ZOOKEEPER-1020:
-
Attachment: (was: ZOOKEEPER-1020.patch)
> Implement function in C client to determine which host you're currently
> connected to.
> -
>
> Key: ZOOKEEPER-1020
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1020
> Project: ZooKeeper
> Issue Type: New Feature
> Components: c client
>Reporter: Stephen Tyree
>Priority: Minor
> Attachments: ZOOKEEPER-1020.patch
>
>
> On occasion it might be useful to determine which host your Zookeeper client
> is currently connected to, be it for debugging purposes or otherwise. A
> possible signature for that function:
> const char* zoo_get_connected_host(zhandle_t *zh, char *buffer, size_t
> buffer_size, unsigned short *port);
> Clients could use it like below:
> char buffer[33];
> unsigned short port = 0;
> if (!zoo_get_connected_host(zh, buffer, sizeof(buffer), &port))
> return EXIT_FAILURE;
> printf("The connected host is: %s:%d\n", buffer, port);
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (ZOOKEEPER-1020) Implement function in C client to determine which host you're currently connected to.
[
https://issues.apache.org/jira/browse/ZOOKEEPER-1020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen Tyree updated ZOOKEEPER-1020:
-
Attachment: ZOOKEEPER-1020.patch
Updated the patch based on Benjamin Reed's comments.
> Implement function in C client to determine which host you're currently
> connected to.
> -
>
> Key: ZOOKEEPER-1020
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1020
> Project: ZooKeeper
> Issue Type: New Feature
> Components: c client
>Reporter: Stephen Tyree
>Priority: Minor
> Attachments: ZOOKEEPER-1020.patch
>
>
> On occasion it might be useful to determine which host your Zookeeper client
> is currently connected to, be it for debugging purposes or otherwise. A
> possible signature for that function:
> const char* zoo_get_connected_host(zhandle_t *zh, char *buffer, size_t
> buffer_size, unsigned short *port);
> Clients could use it like below:
> char buffer[33];
> unsigned short port = 0;
> if (!zoo_get_connected_host(zh, buffer, sizeof(buffer), &port))
> return EXIT_FAILURE;
> printf("The connected host is: %s:%d\n", buffer, port);
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (ZOOKEEPER-1018) The connection permutation in get_addrs uses a weak and inefficient shuffle
[
https://issues.apache.org/jira/browse/ZOOKEEPER-1018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13007304#comment-13007304
]
Stephen Tyree commented on ZOOKEEPER-1018:
--
Is the problem that I didn't run 'svn diff' from the root? If so I'll do that
and resubmit.
> The connection permutation in get_addrs uses a weak and inefficient shuffle
> ---
>
> Key: ZOOKEEPER-1018
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1018
> Project: ZooKeeper
> Issue Type: Improvement
> Components: c client
>Affects Versions: 3.3.2
>Reporter: Stephen Tyree
>Priority: Minor
> Attachments: ZOOKEEPER-1018.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> After determining all of the addresses in the get_addrs function in the C
> client, the connection is permuted using the following code:
> setup_random();
> /* Permute */
> for(i = 0; i < zh->addrs_count; i++) {
> struct sockaddr_storage *s1 = zh->addrs +
> random()%zh->addrs_count;
> struct sockaddr_storage *s2 = zh->addrs +
> random()%zh->addrs_count;
> if (s1 != s2) {
> struct sockaddr_storage t = *s1;
> *s1 = *s2;
> *s2 = t;
> }
> }
> Not only does this shuffle produce an uneven permutation, but it is half as
> efficient as the Fisher-Yates shuffle which produces an unbiased one. It
> seems like it would be a simple fix to increase the randomness and efficiency
> of the shuffle by switching over to using Fisher-Yates.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (ZOOKEEPER-1018) The connection permutation in get_addrs uses a weak and inefficient shuffle
[
https://issues.apache.org/jira/browse/ZOOKEEPER-1018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13007307#comment-13007307
]
Stephen Tyree commented on ZOOKEEPER-1018:
--
Also, should I be adding tests? Or should I be listing what I did to test this?
> The connection permutation in get_addrs uses a weak and inefficient shuffle
> ---
>
> Key: ZOOKEEPER-1018
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1018
> Project: ZooKeeper
> Issue Type: Improvement
> Components: c client
>Affects Versions: 3.3.2
>Reporter: Stephen Tyree
>Priority: Minor
> Attachments: ZOOKEEPER-1018.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> After determining all of the addresses in the get_addrs function in the C
> client, the connection is permuted using the following code:
> setup_random();
> /* Permute */
> for(i = 0; i < zh->addrs_count; i++) {
> struct sockaddr_storage *s1 = zh->addrs +
> random()%zh->addrs_count;
> struct sockaddr_storage *s2 = zh->addrs +
> random()%zh->addrs_count;
> if (s1 != s2) {
> struct sockaddr_storage t = *s1;
> *s1 = *s2;
> *s2 = t;
> }
> }
> Not only does this shuffle produce an uneven permutation, but it is half as
> efficient as the Fisher-Yates shuffle which produces an unbiased one. It
> seems like it would be a simple fix to increase the randomness and efficiency
> of the shuffle by switching over to using Fisher-Yates.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (ZOOKEEPER-1020) Implement function in C client to determine which host you're currently connected to.
[
https://issues.apache.org/jira/browse/ZOOKEEPER-1020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen Tyree updated ZOOKEEPER-1020:
-
Attachment: ZOOKEEPER-1020.patch
This time the diff is from the root directory of trunk.
> Implement function in C client to determine which host you're currently
> connected to.
> -
>
> Key: ZOOKEEPER-1020
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1020
> Project: ZooKeeper
> Issue Type: New Feature
> Components: c client
>Reporter: Stephen Tyree
>Priority: Minor
> Attachments: ZOOKEEPER-1020.patch, ZOOKEEPER-1020.patch
>
>
> On occasion it might be useful to determine which host your Zookeeper client
> is currently connected to, be it for debugging purposes or otherwise. A
> possible signature for that function:
> const char* zoo_get_connected_host(zhandle_t *zh, char *buffer, size_t
> buffer_size, unsigned short *port);
> Clients could use it like below:
> char buffer[33];
> unsigned short port = 0;
> if (!zoo_get_connected_host(zh, buffer, sizeof(buffer), &port))
> return EXIT_FAILURE;
> printf("The connected host is: %s:%d\n", buffer, port);
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (ZOOKEEPER-1018) The connection permutation in get_addrs uses a weak and inefficient shuffle
[
https://issues.apache.org/jira/browse/ZOOKEEPER-1018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen Tyree updated ZOOKEEPER-1018:
-
Attachment: ZOOKEEPER-1018.patch
> The connection permutation in get_addrs uses a weak and inefficient shuffle
> ---
>
> Key: ZOOKEEPER-1018
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1018
> Project: ZooKeeper
> Issue Type: Improvement
> Components: c client
>Affects Versions: 3.3.2
>Reporter: Stephen Tyree
>Priority: Minor
> Attachments: ZOOKEEPER-1018.patch, ZOOKEEPER-1018.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> After determining all of the addresses in the get_addrs function in the C
> client, the connection is permuted using the following code:
> setup_random();
> /* Permute */
> for(i = 0; i < zh->addrs_count; i++) {
> struct sockaddr_storage *s1 = zh->addrs +
> random()%zh->addrs_count;
> struct sockaddr_storage *s2 = zh->addrs +
> random()%zh->addrs_count;
> if (s1 != s2) {
> struct sockaddr_storage t = *s1;
> *s1 = *s2;
> *s2 = t;
> }
> }
> Not only does this shuffle produce an uneven permutation, but it is half as
> efficient as the Fisher-Yates shuffle which produces an unbiased one. It
> seems like it would be a simple fix to increase the randomness and efficiency
> of the shuffle by switching over to using Fisher-Yates.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (ZOOKEEPER-1018) The connection permutation in get_addrs uses a weak and inefficient shuffle
[
https://issues.apache.org/jira/browse/ZOOKEEPER-1018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13007494#comment-13007494
]
Stephen Tyree commented on ZOOKEEPER-1018:
--
Updated, this time taking the diff from the root directory.
> The connection permutation in get_addrs uses a weak and inefficient shuffle
> ---
>
> Key: ZOOKEEPER-1018
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1018
> Project: ZooKeeper
> Issue Type: Improvement
> Components: c client
>Affects Versions: 3.3.2
>Reporter: Stephen Tyree
>Priority: Minor
> Attachments: ZOOKEEPER-1018.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> After determining all of the addresses in the get_addrs function in the C
> client, the connection is permuted using the following code:
> setup_random();
> /* Permute */
> for(i = 0; i < zh->addrs_count; i++) {
> struct sockaddr_storage *s1 = zh->addrs +
> random()%zh->addrs_count;
> struct sockaddr_storage *s2 = zh->addrs +
> random()%zh->addrs_count;
> if (s1 != s2) {
> struct sockaddr_storage t = *s1;
> *s1 = *s2;
> *s2 = t;
> }
> }
> Not only does this shuffle produce an uneven permutation, but it is half as
> efficient as the Fisher-Yates shuffle which produces an unbiased one. It
> seems like it would be a simple fix to increase the randomness and efficiency
> of the shuffle by switching over to using Fisher-Yates.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (ZOOKEEPER-1018) The connection permutation in get_addrs uses a weak and inefficient shuffle
[
https://issues.apache.org/jira/browse/ZOOKEEPER-1018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen Tyree updated ZOOKEEPER-1018:
-
Attachment: (was: ZOOKEEPER-1018.patch)
> The connection permutation in get_addrs uses a weak and inefficient shuffle
> ---
>
> Key: ZOOKEEPER-1018
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1018
> Project: ZooKeeper
> Issue Type: Improvement
> Components: c client
>Affects Versions: 3.3.2
>Reporter: Stephen Tyree
>Priority: Minor
> Attachments: ZOOKEEPER-1018.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> After determining all of the addresses in the get_addrs function in the C
> client, the connection is permuted using the following code:
> setup_random();
> /* Permute */
> for(i = 0; i < zh->addrs_count; i++) {
> struct sockaddr_storage *s1 = zh->addrs +
> random()%zh->addrs_count;
> struct sockaddr_storage *s2 = zh->addrs +
> random()%zh->addrs_count;
> if (s1 != s2) {
> struct sockaddr_storage t = *s1;
> *s1 = *s2;
> *s2 = t;
> }
> }
> Not only does this shuffle produce an uneven permutation, but it is half as
> efficient as the Fisher-Yates shuffle which produces an unbiased one. It
> seems like it would be a simple fix to increase the randomness and efficiency
> of the shuffle by switching over to using Fisher-Yates.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (ZOOKEEPER-1020) Implement function in C client to determine which host you're currently connected to.
[
https://issues.apache.org/jira/browse/ZOOKEEPER-1020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen Tyree updated ZOOKEEPER-1020:
-
Attachment: (was: ZOOKEEPER-1020.patch)
> Implement function in C client to determine which host you're currently
> connected to.
> -
>
> Key: ZOOKEEPER-1020
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1020
> Project: ZooKeeper
> Issue Type: New Feature
> Components: c client
>Reporter: Stephen Tyree
>Priority: Minor
> Attachments: ZOOKEEPER-1020.patch
>
>
> On occasion it might be useful to determine which host your Zookeeper client
> is currently connected to, be it for debugging purposes or otherwise. A
> possible signature for that function:
> const char* zoo_get_connected_host(zhandle_t *zh, char *buffer, size_t
> buffer_size, unsigned short *port);
> Clients could use it like below:
> char buffer[33];
> unsigned short port = 0;
> if (!zoo_get_connected_host(zh, buffer, sizeof(buffer), &port))
> return EXIT_FAILURE;
> printf("The connected host is: %s:%d\n", buffer, port);
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (ZOOKEEPER-1020) Implement function in C client to determine which host you're currently connected to.
[
https://issues.apache.org/jira/browse/ZOOKEEPER-1020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen Tyree updated ZOOKEEPER-1020:
-
Attachment: ZOOKEEPER-1020.patch
This time with a unit test.
> Implement function in C client to determine which host you're currently
> connected to.
> -
>
> Key: ZOOKEEPER-1020
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1020
> Project: ZooKeeper
> Issue Type: New Feature
> Components: c client
>Reporter: Stephen Tyree
>Priority: Minor
> Attachments: ZOOKEEPER-1020.patch
>
>
> On occasion it might be useful to determine which host your Zookeeper client
> is currently connected to, be it for debugging purposes or otherwise. A
> possible signature for that function:
> const char* zoo_get_connected_host(zhandle_t *zh, char *buffer, size_t
> buffer_size, unsigned short *port);
> Clients could use it like below:
> char buffer[33];
> unsigned short port = 0;
> if (!zoo_get_connected_host(zh, buffer, sizeof(buffer), &port))
> return EXIT_FAILURE;
> printf("The connected host is: %s:%d\n", buffer, port);
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (ZOOKEEPER-1020) Implement function in C client to determine which host you're currently connected to.
[
https://issues.apache.org/jira/browse/ZOOKEEPER-1020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen Tyree updated ZOOKEEPER-1020:
-
Attachment: (was: ZOOKEEPER-1020.patch)
> Implement function in C client to determine which host you're currently
> connected to.
> -
>
> Key: ZOOKEEPER-1020
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1020
> Project: ZooKeeper
> Issue Type: New Feature
> Components: c client
>Reporter: Stephen Tyree
>Priority: Minor
> Attachments: ZOOKEEPER-1020.patch
>
>
> On occasion it might be useful to determine which host your Zookeeper client
> is currently connected to, be it for debugging purposes or otherwise. A
> possible signature for that function:
> const char* zoo_get_connected_host(zhandle_t *zh, char *buffer, size_t
> buffer_size, unsigned short *port);
> Clients could use it like below:
> char buffer[33];
> unsigned short port = 0;
> if (!zoo_get_connected_host(zh, buffer, sizeof(buffer), &port))
> return EXIT_FAILURE;
> printf("The connected host is: %s:%d\n", buffer, port);
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (ZOOKEEPER-1018) The connection permutation in get_addrs uses a weak and inefficient shuffle
[
https://issues.apache.org/jira/browse/ZOOKEEPER-1018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen Tyree updated ZOOKEEPER-1018:
-
Attachment: (was: ZOOKEEPER-1018.patch)
> The connection permutation in get_addrs uses a weak and inefficient shuffle
> ---
>
> Key: ZOOKEEPER-1018
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1018
> Project: ZooKeeper
> Issue Type: Improvement
> Components: c client
>Affects Versions: 3.3.2
>Reporter: Stephen Tyree
>Assignee: Stephen Tyree
>Priority: Minor
> Attachments: ZOOKEEPER-1018.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> After determining all of the addresses in the get_addrs function in the C
> client, the connection is permuted using the following code:
> setup_random();
> /* Permute */
> for(i = 0; i < zh->addrs_count; i++) {
> struct sockaddr_storage *s1 = zh->addrs +
> random()%zh->addrs_count;
> struct sockaddr_storage *s2 = zh->addrs +
> random()%zh->addrs_count;
> if (s1 != s2) {
> struct sockaddr_storage t = *s1;
> *s1 = *s2;
> *s2 = t;
> }
> }
> Not only does this shuffle produce an uneven permutation, but it is half as
> efficient as the Fisher-Yates shuffle which produces an unbiased one. It
> seems like it would be a simple fix to increase the randomness and efficiency
> of the shuffle by switching over to using Fisher-Yates.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (ZOOKEEPER-1018) The connection permutation in get_addrs uses a weak and inefficient shuffle
[
https://issues.apache.org/jira/browse/ZOOKEEPER-1018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen Tyree updated ZOOKEEPER-1018:
-
Attachment: ZOOKEEPER-1018.patch
The reason the unit tests failed was due to the change in how random numbers
impacted the order of servers. The code for shuffling addresses is correct, and
making everything work just required changing a few numbers in the Mock_random
generator.
> The connection permutation in get_addrs uses a weak and inefficient shuffle
> ---
>
> Key: ZOOKEEPER-1018
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1018
> Project: ZooKeeper
> Issue Type: Improvement
> Components: c client
>Affects Versions: 3.3.2
>Reporter: Stephen Tyree
>Assignee: Stephen Tyree
>Priority: Minor
> Attachments: ZOOKEEPER-1018.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> After determining all of the addresses in the get_addrs function in the C
> client, the connection is permuted using the following code:
> setup_random();
> /* Permute */
> for(i = 0; i < zh->addrs_count; i++) {
> struct sockaddr_storage *s1 = zh->addrs +
> random()%zh->addrs_count;
> struct sockaddr_storage *s2 = zh->addrs +
> random()%zh->addrs_count;
> if (s1 != s2) {
> struct sockaddr_storage t = *s1;
> *s1 = *s2;
> *s2 = t;
> }
> }
> Not only does this shuffle produce an uneven permutation, but it is half as
> efficient as the Fisher-Yates shuffle which produces an unbiased one. It
> seems like it would be a simple fix to increase the randomness and efficiency
> of the shuffle by switching over to using Fisher-Yates.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1018) The connection permutation in get_addrs uses a weak and inefficient shuffle
[
https://issues.apache.org/jira/browse/ZOOKEEPER-1018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13012985#comment-13012985
]
Stephen Tyree commented on ZOOKEEPER-1018:
--
What is the status on this? Is adjusting the unit test for shuffling sufficient
or is there more I need to do here?
> The connection permutation in get_addrs uses a weak and inefficient shuffle
> ---
>
> Key: ZOOKEEPER-1018
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1018
> Project: ZooKeeper
> Issue Type: Improvement
> Components: c client
>Affects Versions: 3.3.2
>Reporter: Stephen Tyree
>Assignee: Stephen Tyree
>Priority: Minor
> Fix For: 3.4.0
>
> Attachments: ZOOKEEPER-1018.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> After determining all of the addresses in the get_addrs function in the C
> client, the connection is permuted using the following code:
> setup_random();
> /* Permute */
> for(i = 0; i < zh->addrs_count; i++) {
> struct sockaddr_storage *s1 = zh->addrs +
> random()%zh->addrs_count;
> struct sockaddr_storage *s2 = zh->addrs +
> random()%zh->addrs_count;
> if (s1 != s2) {
> struct sockaddr_storage t = *s1;
> *s1 = *s2;
> *s2 = t;
> }
> }
> Not only does this shuffle produce an uneven permutation, but it is half as
> efficient as the Fisher-Yates shuffle which produces an unbiased one. It
> seems like it would be a simple fix to increase the randomness and efficiency
> of the shuffle by switching over to using Fisher-Yates.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (ZOOKEEPER-1051) SIGPIPE in Zookeeper 0.3.* when send'ing after cluster disconnection
SIGPIPE in Zookeeper 0.3.* when send'ing after cluster disconnection Key: ZOOKEEPER-1051 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1051 Project: ZooKeeper Issue Type: Bug Components: c client Affects Versions: 3.3.2, 3.3.3, 3.4.0 Reporter: Stephen Tyree Priority: Minor In libzookeeper_mt, if your process is going rather slowly (such as when running it in Valgrind's Memcheck) or you are using gdb with breakpoints, you can occasionally get SIGPIPE when trying to send a message to the cluster. For example: ==12788== ==12788== Process terminating with default action of signal 13 (SIGPIPE) ==12788==at 0x3F5180DE91: send (in /lib64/libpthread-2.5.so) ==12788==by 0x7F060AA: ??? (in /usr/lib64/libzookeeper_mt.so.2.0.0) ==12788==by 0x7F06E5B: zookeeper_process (in /usr/lib64/libzookeeper_mt.so.2.0.0) ==12788==by 0x7F0D38E: ??? (in /usr/lib64/libzookeeper_mt.so.2.0.0) ==12788==by 0x3F5180673C: start_thread (in /lib64/libpthread-2.5.so) ==12788==by 0x3F50CD3F6C: clone (in /lib64/libc-2.5.so) ==12788== This is probably not the behavior we would like, since we handle server disconnections after a failed call to send. To fix this, there are a few options we could use. For BSD environments, we can tell a socket to never send SIGPIPE with send using setsockopt: setsockopt(sd, SOL_SOCKET, SO_NOSIGPIPE, (void *)&set, sizeof(int)); For Linux environments, we can add a MSG_NOSIGNAL flag to every send call that says to not send SIGPIPE on a bad file descriptor. For more information, see: http://stackoverflow.com/questions/108183/how-to-prevent-sigpipes-or-handle-them-properly -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (ZOOKEEPER-1056) Questions and Improvements for the C client codebase
Questions and Improvements for the C client codebase Key: ZOOKEEPER-1056 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1056 Project: ZooKeeper Issue Type: Bug Components: c client Affects Versions: 3.4.0 Reporter: Stephen Tyree Priority: Minor Having been using the C client for a few months now, I thought I'd look through the code and see if anything could be improved and/or fixed in order to be a good citizen. Here are some observations and questions I was hoping people could elaborate on. - There appears to be a bug in sub_string (zookeeper.c). The third argument being passed into strncmp is a conditional due to misplaced parenthesis, meaning the length is either 0 or 1. This likely leads to many, many false positives of chroots matching paths. - There appears to be a bug in queue_session_event, where we check for cptr->buffer not being NULL after already dereferencing it - In both queue_buffer and queue_completion_nolock, we assert a conditional that we just checked for - What is the policy on whether the result of memory allocations are checked for, assert'd against or ignored? This is done inconsistently. - What is the policy on whether pointers are checked/set against NULL versus 0? This is done inconsistently. - Some functions, such as zoo_wget_children2_, exhibit needlessly high cyclomatic complexity - What is the policy on line length restrictions? Some functions go through hurdles to enforce 80 characters while others do no such thing. - What is the policy on indentation and spacing of if statements and blocks of code? This is done inconsistently. If any or all of these turn out to be issues that need to be fixed I'd be more than happy to do so. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-965) Need a multi-update command to allow multiple znodes to be updated safely
[ https://issues.apache.org/jira/browse/ZOOKEEPER-965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032575#comment-13032575 ] Stephen Tyree commented on ZOOKEEPER-965: - Great job all around! Some comments on the C library portion of the patch: - The zoo_acheck comment includes documentation on a member 'stat' that is not an argument - The zoo_check comment has the parameter stat listed as _stat - In create_completion_entry, you commented out the else case where you set c->clist to NULL. Did you mean to leave this in there? - In op_result_string_completion, you memcpy from the value onto data->value based on the size of value. Is data->value guaranteed to be larger than value? - You only assert result in op_result_string_completion, but none of the other op_result_* functions - Do you not need to set result->err in op_result_multi_completion? - In zoo_amulti, any reason "int index = 0;" isn't a few lines down right before it's used? - Slick zoo_amulti code, very nicely done. - In testCreate you compare results[*].err to 0, which will work, but shouldn't you compare it to ZOK? Or do I misunderstand op_result_t. - In testDelete, your comment should say "// '/multi2' should have been deleted" All-in-all the code looks great in the C client. Nicely done. > Need a multi-update command to allow multiple znodes to be updated safely > - > > Key: ZOOKEEPER-965 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-965 > Project: ZooKeeper > Issue Type: Bug >Affects Versions: 3.3.3 >Reporter: Ted Dunning >Assignee: Ted Dunning > Fix For: 3.4.0 > > Attachments: ZOOKEEPER-965.patch, ZOOKEEPER-965.patch, > ZOOKEEPER-965.patch, ZOOKEEPER-965.patch, ZOOKEEPER-965.patch, > ZOOKEEPER-965.patch, ZOOKEEPER-965.patch, ZOOKEEPER-965.patch, > ZOOKEEPER-965.patch, ZOOKEEPER-965.patch > > > The basic idea is to have a single method called "multi" that will accept a > list of create, delete, update or check objects each of which has a desired > version or file state in the case of create. If all of the version and > existence constraints can be satisfied, then all updates will be done > atomically. > Two API styles have been suggested. One has a list as above and the other > style has a "Transaction" that allows builder-like methods to build a set of > updates and a commit method to finalize the transaction. This can trivially > be reduced to the first kind of API so the list based API style should be > considered the primitive and the builder style should be implemented as > syntactic sugar. > The total size of all the data in all updates and creates in a single > transaction should be limited to 1MB. > Implementation-wise this capability can be done using standard ZK internals. > The changes include: > - update to ZK clients to all the new call > - additional wire level request > - on the server, in the code that converts transactions to idempotent form, > the code should be slightly extended to convert a list of operations to > idempotent form. > - on the client, a down-rev server that rejects the multi-update should be > detected gracefully and an informative exception should be thrown. > To facilitate shared development, I have established a github repository at > https://github.com/tdunning/zookeeper and am happy to extend committer > status to anyone who agrees to donate their code back to Apache. The final > patch will be attached to this bug as normal. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1117) zookeeper 3.3.3 fails to build with gcc >= 4.6.1 on Debian/Ubuntu
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13060021#comment-13060021 ] Stephen Tyree commented on ZOOKEEPER-1117: -- That warning exists for a good reason, and marking it not as an error is only glosses over the problem. A better solution would be to move the code which initializes port into the block that only runs on Cygwin since that's where it's used. > zookeeper 3.3.3 fails to build with gcc >= 4.6.1 on Debian/Ubuntu > - > > Key: ZOOKEEPER-1117 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1117 > Project: ZooKeeper > Issue Type: Bug > Components: c client >Affects Versions: 3.3.3 > Environment: Ubuntu Developement Release (11.10/Oneiric Ocelot), > Debian Unstable (sid) >Reporter: James Page >Priority: Minor > Attachments: disable-cc-errors > > > zookeeper 3.3.3 (and 3.3.1) fails to build on Debian and Ubuntu systems with > gcc >= 4.6.1: > /bin/bash ./libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I. > -I./include -I./tests -I./generated -Wall -Werror -g -O2 -D_GNU_SOURCE -MT > zookeeper.lo -MD -MP -MF .deps/zookeeper.Tpo -c -o zookeeper.lo `test -f > 'src/zookeeper.c' || echo './'`src/zookeeper.c > libtool: compile: gcc -DHAVE_CONFIG_H -I. -I./include -I./tests > -I./generated -Wall -Werror -g -O2 -D_GNU_SOURCE -MT zookeeper.lo -MD -MP -MF > .deps/zookeeper.Tpo -c src/zookeeper.c -fPIC -DPIC -o .libs/zookeeper.o > src/zookeeper.c: In function 'getaddrs': > src/zookeeper.c:455:13: error: variable 'port' set but not used > [-Werror=unused-but-set-variable] > cc1: all warnings being treated as errors > See http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=625441 for more > information. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (ZOOKEEPER-1051) SIGPIPE in Zookeeper 0.3.* when send'ing after cluster disconnection
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Tyree reassigned ZOOKEEPER-1051: Assignee: Stephen Tyree > SIGPIPE in Zookeeper 0.3.* when send'ing after cluster disconnection > > > Key: ZOOKEEPER-1051 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1051 > Project: ZooKeeper > Issue Type: Bug > Components: c client >Affects Versions: 3.3.2, 3.3.3, 3.4.0 >Reporter: Stephen Tyree >Assignee: Stephen Tyree >Priority: Minor > Attachments: ZOOKEEPER-1051.patch > > Original Estimate: 2h > Remaining Estimate: 2h > > In libzookeeper_mt, if your process is going rather slowly (such as when > running it in Valgrind's Memcheck) or you are using gdb with breakpoints, you > can occasionally get SIGPIPE when trying to send a message to the cluster. > For example: > ==12788== > ==12788== Process terminating with default action of signal 13 (SIGPIPE) > ==12788==at 0x3F5180DE91: send (in /lib64/libpthread-2.5.so) > ==12788==by 0x7F060AA: ??? (in /usr/lib64/libzookeeper_mt.so.2.0.0) > ==12788==by 0x7F06E5B: zookeeper_process (in > /usr/lib64/libzookeeper_mt.so.2.0.0) > ==12788==by 0x7F0D38E: ??? (in /usr/lib64/libzookeeper_mt.so.2.0.0) > ==12788==by 0x3F5180673C: start_thread (in /lib64/libpthread-2.5.so) > ==12788==by 0x3F50CD3F6C: clone (in /lib64/libc-2.5.so) > ==12788== > This is probably not the behavior we would like, since we handle server > disconnections after a failed call to send. To fix this, there are a few > options we could use. For BSD environments, we can tell a socket to never > send SIGPIPE with send using setsockopt: > setsockopt(sd, SOL_SOCKET, SO_NOSIGPIPE, (void *)&set, sizeof(int)); > For Linux environments, we can add a MSG_NOSIGNAL flag to every send call > that says to not send SIGPIPE on a bad file descriptor. > For more information, see: > http://stackoverflow.com/questions/108183/how-to-prevent-sigpipes-or-handle-them-properly -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (ZOOKEEPER-1051) SIGPIPE in Zookeeper 0.3.* when send'ing after cluster disconnection
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Tyree updated ZOOKEEPER-1051: - Attachment: ZOOKEEPER-1051.patch Patch which fixes the issue for Linux and Linux alone. Do we support Zookeeper for other *nixes? If so, there are likely to be platform specific solutions to the problem for them as well. > SIGPIPE in Zookeeper 0.3.* when send'ing after cluster disconnection > > > Key: ZOOKEEPER-1051 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1051 > Project: ZooKeeper > Issue Type: Bug > Components: c client >Affects Versions: 3.3.2, 3.3.3, 3.4.0 >Reporter: Stephen Tyree >Assignee: Stephen Tyree >Priority: Minor > Attachments: ZOOKEEPER-1051.patch > > Original Estimate: 2h > Remaining Estimate: 2h > > In libzookeeper_mt, if your process is going rather slowly (such as when > running it in Valgrind's Memcheck) or you are using gdb with breakpoints, you > can occasionally get SIGPIPE when trying to send a message to the cluster. > For example: > ==12788== > ==12788== Process terminating with default action of signal 13 (SIGPIPE) > ==12788==at 0x3F5180DE91: send (in /lib64/libpthread-2.5.so) > ==12788==by 0x7F060AA: ??? (in /usr/lib64/libzookeeper_mt.so.2.0.0) > ==12788==by 0x7F06E5B: zookeeper_process (in > /usr/lib64/libzookeeper_mt.so.2.0.0) > ==12788==by 0x7F0D38E: ??? (in /usr/lib64/libzookeeper_mt.so.2.0.0) > ==12788==by 0x3F5180673C: start_thread (in /lib64/libpthread-2.5.so) > ==12788==by 0x3F50CD3F6C: clone (in /lib64/libc-2.5.so) > ==12788== > This is probably not the behavior we would like, since we handle server > disconnections after a failed call to send. To fix this, there are a few > options we could use. For BSD environments, we can tell a socket to never > send SIGPIPE with send using setsockopt: > setsockopt(sd, SOL_SOCKET, SO_NOSIGPIPE, (void *)&set, sizeof(int)); > For Linux environments, we can add a MSG_NOSIGNAL flag to every send call > that says to not send SIGPIPE on a bad file descriptor. > For more information, see: > http://stackoverflow.com/questions/108183/how-to-prevent-sigpipes-or-handle-them-properly -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (ZOOKEEPER-1051) SIGPIPE in Zookeeper 0.3.* when send'ing after cluster disconnection
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Tyree updated ZOOKEEPER-1051: - Attachment: ZOOKEEPER-1051.patch Realized I missed a couple of spots, so I generalized the logic in a function. Haven't the ability to test the change on Windows. > SIGPIPE in Zookeeper 0.3.* when send'ing after cluster disconnection > > > Key: ZOOKEEPER-1051 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1051 > Project: ZooKeeper > Issue Type: Bug > Components: c client >Affects Versions: 3.3.2, 3.3.3, 3.4.0 >Reporter: Stephen Tyree >Assignee: Stephen Tyree >Priority: Minor > Attachments: ZOOKEEPER-1051.patch, ZOOKEEPER-1051.patch > > Original Estimate: 2h > Remaining Estimate: 2h > > In libzookeeper_mt, if your process is going rather slowly (such as when > running it in Valgrind's Memcheck) or you are using gdb with breakpoints, you > can occasionally get SIGPIPE when trying to send a message to the cluster. > For example: > ==12788== > ==12788== Process terminating with default action of signal 13 (SIGPIPE) > ==12788==at 0x3F5180DE91: send (in /lib64/libpthread-2.5.so) > ==12788==by 0x7F060AA: ??? (in /usr/lib64/libzookeeper_mt.so.2.0.0) > ==12788==by 0x7F06E5B: zookeeper_process (in > /usr/lib64/libzookeeper_mt.so.2.0.0) > ==12788==by 0x7F0D38E: ??? (in /usr/lib64/libzookeeper_mt.so.2.0.0) > ==12788==by 0x3F5180673C: start_thread (in /lib64/libpthread-2.5.so) > ==12788==by 0x3F50CD3F6C: clone (in /lib64/libc-2.5.so) > ==12788== > This is probably not the behavior we would like, since we handle server > disconnections after a failed call to send. To fix this, there are a few > options we could use. For BSD environments, we can tell a socket to never > send SIGPIPE with send using setsockopt: > setsockopt(sd, SOL_SOCKET, SO_NOSIGPIPE, (void *)&set, sizeof(int)); > For Linux environments, we can add a MSG_NOSIGNAL flag to every send call > that says to not send SIGPIPE on a bad file descriptor. > For more information, see: > http://stackoverflow.com/questions/108183/how-to-prevent-sigpipes-or-handle-them-properly -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1051) SIGPIPE in Zookeeper 0.3.* when send'ing after cluster disconnection
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090337#comment-13090337 ] Stephen Tyree commented on ZOOKEEPER-1051: -- There isn't an equivalent issue on Windows, only for POSIX-ish platforms that signal on failed socket sends. This means other Unix-like platforms could experience similar issues, but it appears in this case that for now we aren't going to concern ourselves with them. Is there anything left to do here or are we good for submission? > SIGPIPE in Zookeeper 0.3.* when send'ing after cluster disconnection > > > Key: ZOOKEEPER-1051 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1051 > Project: ZooKeeper > Issue Type: Bug > Components: c client >Affects Versions: 3.3.2, 3.3.3, 3.4.0 >Reporter: Stephen Tyree >Assignee: Stephen Tyree >Priority: Minor > Attachments: ZOOKEEPER-1051.patch, ZOOKEEPER-1051.patch > > Original Estimate: 2h > Remaining Estimate: 2h > > In libzookeeper_mt, if your process is going rather slowly (such as when > running it in Valgrind's Memcheck) or you are using gdb with breakpoints, you > can occasionally get SIGPIPE when trying to send a message to the cluster. > For example: > ==12788== > ==12788== Process terminating with default action of signal 13 (SIGPIPE) > ==12788==at 0x3F5180DE91: send (in /lib64/libpthread-2.5.so) > ==12788==by 0x7F060AA: ??? (in /usr/lib64/libzookeeper_mt.so.2.0.0) > ==12788==by 0x7F06E5B: zookeeper_process (in > /usr/lib64/libzookeeper_mt.so.2.0.0) > ==12788==by 0x7F0D38E: ??? (in /usr/lib64/libzookeeper_mt.so.2.0.0) > ==12788==by 0x3F5180673C: start_thread (in /lib64/libpthread-2.5.so) > ==12788==by 0x3F50CD3F6C: clone (in /lib64/libc-2.5.so) > ==12788== > This is probably not the behavior we would like, since we handle server > disconnections after a failed call to send. To fix this, there are a few > options we could use. For BSD environments, we can tell a socket to never > send SIGPIPE with send using setsockopt: > setsockopt(sd, SOL_SOCKET, SO_NOSIGPIPE, (void *)&set, sizeof(int)); > For Linux environments, we can add a MSG_NOSIGNAL flag to every send call > that says to not send SIGPIPE on a bad file descriptor. > For more information, see: > http://stackoverflow.com/questions/108183/how-to-prevent-sigpipes-or-handle-them-properly -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1451) C API improperly logs getaddrinfo failures on Linux when using glibc
[
https://issues.apache.org/jira/browse/ZOOKEEPER-1451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13260537#comment-13260537
]
Stephen Tyree commented on ZOOKEEPER-1451:
--
Is there anything else necessary here or are we good to go?
> C API improperly logs getaddrinfo failures on Linux when using glibc
>
>
> Key: ZOOKEEPER-1451
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1451
> Project: ZooKeeper
> Issue Type: Bug
> Components: c client
>Affects Versions: 3.4.3
> Environment: Linux when using glibc
>Reporter: Stephen Tyree
>Assignee: Stephen Tyree
>Priority: Trivial
> Attachments: ZOOKEEPER-1451.patch
>
>
> This is how the code currently logs getaddrinfo errors:
> {quote}
> errno = getaddrinfo_errno(rc);
> LOG_ERROR(("getaddrinfo: %s\n", strerror(errno)));
> {quote}
> On Linux, specifically when using glibc, there is a better function for
> logging getaddrinfo errors called gai_strerror. An example:
> {quote}
> LOG_ERROR(("getaddrinfo: %s\n", gai_strerror(rc)));
> {quote}
> It doesn't miss a lot of cases like the errno based version does.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (ZOOKEEPER-1461) Zookeeper C client doesn't check for NULL before dereferencing in prepend_string
Stephen Tyree created ZOOKEEPER-1461: Summary: Zookeeper C client doesn't check for NULL before dereferencing in prepend_string Key: ZOOKEEPER-1461 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1461 Project: ZooKeeper Issue Type: Improvement Components: c client Affects Versions: 3.3.5 Reporter: Stephen Tyree prepend_string, called before any checks for NULL in the c client for many API functions, has this line (zookeeper 3.3.5): if (zh->chroot == NULL) That means that before you check for NULL, you are dereferencing the pointer. This bug does not exist in the 3.4.* branch for whatever reason, but it still remains in the 3.3.* line. A patch which fixes it would make the line as follows: if (zh == NULL || zh->chroot == NULL) I would do that for you, but I don't know how to patch the 3.3.5 branch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (ZOOKEEPER-1461) Zookeeper C client doesn't check for NULL before dereferencing in prepend_string
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Tyree updated ZOOKEEPER-1461: - Attachment: ZOOKEEPER-1461.PATCH > Zookeeper C client doesn't check for NULL before dereferencing in > prepend_string > > > Key: ZOOKEEPER-1461 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1461 > Project: ZooKeeper > Issue Type: Improvement > Components: c client >Affects Versions: 3.3.5 >Reporter: Stephen Tyree > Fix For: 3.3.6 > > Attachments: ZOOKEEPER-1461.PATCH > > Original Estimate: 0h > Remaining Estimate: 0h > > prepend_string, called before any checks for NULL in the c client for many > API functions, has this line (zookeeper 3.3.5): > if (zh->chroot == NULL) > That means that before you check for NULL, you are dereferencing the pointer. > This bug does not exist in the 3.4.* branch for whatever reason, but it still > remains in the 3.3.* line. A patch which fixes it would make the line as > follows: > if (zh == NULL || zh->chroot == NULL) > I would do that for you, but I don't know how to patch the 3.3.5 branch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1461) Zookeeper C client doesn't check for NULL before dereferencing in prepend_string
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13266142#comment-13266142 ] Stephen Tyree commented on ZOOKEEPER-1461: -- Was it trying to apply the patch to trunk or the 3.3 branch? > Zookeeper C client doesn't check for NULL before dereferencing in > prepend_string > > > Key: ZOOKEEPER-1461 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1461 > Project: ZooKeeper > Issue Type: Improvement > Components: c client >Affects Versions: 3.3.5 >Reporter: Stephen Tyree >Assignee: Stephen Tyree > Fix For: 3.3.6 > > Attachments: ZOOKEEPER-1461.PATCH > > Original Estimate: 0h > Remaining Estimate: 0h > > prepend_string, called before any checks for NULL in the c client for many > API functions, has this line (zookeeper 3.3.5): > if (zh->chroot == NULL) > That means that before you check for NULL, you are dereferencing the pointer. > This bug does not exist in the 3.4.* branch for whatever reason, but it still > remains in the 3.3.* line. A patch which fixes it would make the line as > follows: > if (zh == NULL || zh->chroot == NULL) > I would do that for you, but I don't know how to patch the 3.3.5 branch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (ZOOKEEPER-1446) C API makes it difficult to implement a timed wait_until_connected method correctly
[
https://issues.apache.org/jira/browse/ZOOKEEPER-1446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen Tyree updated ZOOKEEPER-1446:
-
Attachment: ZOOKEEPER-1446.patch
An example patch. Was unable to run the unit test I wrote (because my setup is
being a jerk), but I'm more looking for input on the design.
> C API makes it difficult to implement a timed wait_until_connected method
> correctly
> ---
>
> Key: ZOOKEEPER-1446
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1446
> Project: ZooKeeper
> Issue Type: Bug
> Components: c client
>Affects Versions: 3.4.3, 3.3.5
>Reporter: Stephen Tyree
>Priority: Minor
> Attachments: ZOOKEEPER-1446.patch
>
>
> When using the C API, one might feel inclined to create a
> zookeeper_wait_until_connected method which waits for some amount for a
> connected state event to occur. The code might look like the following
> (didn't actually compile this):
> //--
> static pthread_mutex_t kConnectedMutex = PTHREAD_MUTEX_INITIALIZER;
> static pthread_cond_t kConnectedCondvar = PTHREAD_COND_INITIALIZER;
> int zookeeper_wait_until_connected(zhandle_t* zk, const struct timespec*
> timeout)
> {
> struct timespec abstime;
> clock_gettime(TIMER_ABSTIME, &abstime);
> abstime->tv_sec += timeout->tv_sec;
> abstime->tv_nsec += timeout->tv_nsec;
> pthread_mutex_lock(&kConnectedMutex);
> if (zoo_state(zk) == ZOO_CONNECTED_STATE) {
> return 1;
> }
> pthread_cond_timedwait(&kConnectedCondvar, &kConnectedMutex, &abstime);
> int state = zoo_state(zk);
> return (state == ZOO_CONNECTED_STATE);
> }
> void zookeeper_session_callback(zhandle_t* zh, int type, int state, const
> char* path, void* arg)
> {
> pthread_mutex_lock(&kConnectedMutex);
> if (type == ZOO_SESSION_EVENT && state == ZOO_CONNECTED_STATE) {
> pthread_cond_broadcast(&kConnectedCondvar);
> }
> }
> //-
> That would work fine (assuming I didn't screw anything up), except that
> pthread_cond_timedwait can spuriously wakeup, making you not actually wait
> the desired timeout. The solution to this is to loop until the condition is
> met, which might look like the following:
> //---
> int state = zoo_state(zk);
> int result = 0;
> while ((state == ZOO_CONNECTING_STATE || state == ZOO_ASSOCIATING_STATE) &&
> result != ETIMEDOUT) {
> result = pthread_cond_timedwait(&kConnectedCondvar, &kConnectedMutex,
> &abstime);
> state = zoo_state(zk);
> }
> //---
> That would work fine, except the state might be valid and connecting, yet not
> ZOO_CONNECTING_STATE or ZOO_ASSOCIATING_STATE, it might be 0 or, as
> implemented recently courtesy of zookeeper-1108, 999. Checking for those
> states causes your code to rely upon an implementation detail of zookeeper, a
> problem highlighted by that implementation detail changing recently. Is there
> any way this behavior can become documented (via a ZOO_INITIALIZED_STATE or
> something like that), or is there any way this behavior can be supported by
> the library itself?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1479) C Client: zoo_add_auth() doesn't wake up the IO thread
[
https://issues.apache.org/jira/browse/ZOOKEEPER-1479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13289443#comment-13289443
]
Stephen Tyree commented on ZOOKEEPER-1479:
--
Should it? Zookeeper operations are well-ordered for one session, so any
subsequent operation is going to have the authentication you provide via
zoo_add_auth whether or not the I/O thread is woken up.
> C Client: zoo_add_auth() doesn't wake up the IO thread
> --
>
> Key: ZOOKEEPER-1479
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1479
> Project: ZooKeeper
> Issue Type: Bug
> Components: c client
>Affects Versions: 3.4.3
>Reporter: Michi Mutsuzaki
> Fix For: 3.5.0
>
>
> It can take up to sessionTimeout / 3 for the IO thread to send out the auth
> packet. The {{zoo_add_auth()}} function should call {{adaptor_send_queue(zh,
> 0)}} after {{calling send_last_auth_info(zh)}}.
> --Michi
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1479) C Client: zoo_add_auth() doesn't wake up the IO thread
[
https://issues.apache.org/jira/browse/ZOOKEEPER-1479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13289643#comment-13289643
]
Stephen Tyree commented on ZOOKEEPER-1479:
--
Aside from being used in unit tests, is there a use case for synchronous
zoo_add_auth? The downside of this is that anyone using zoo_add_auth for
synchronous work does marginally more work than they did before for no benefit,
which is something that ideally would be avoided. If this is such an issue for
developers (as I could imagine it being), maybe for development builds of
Zookeeper there could be a preprocessor flag, say
ZOOKEEPER_SYNCHRONOUS_ADD_AUTH, which wakes the IO thread following the call.
Then release builds could simply not use the flag, and we end up getting the
best of both worlds.
On the one hand, this is ugly as sin and reeks of leaving printf's throughout
your code to aid in debugging. But on the other, this gives the best of both
worlds. Thoughts?
> C Client: zoo_add_auth() doesn't wake up the IO thread
> --
>
> Key: ZOOKEEPER-1479
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1479
> Project: ZooKeeper
> Issue Type: Bug
> Components: c client
>Affects Versions: 3.4.3
>Reporter: Michi Mutsuzaki
> Fix For: 3.5.0
>
>
> It can take up to sessionTimeout / 3 for the IO thread to send out the auth
> packet. The {{zoo_add_auth()}} function should call {{adaptor_send_queue(zh,
> 0)}} after {{calling send_last_auth_info(zh)}}.
> --Michi
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1519) Zookeeper Async calls can reference free()'d memory
[
https://issues.apache.org/jira/browse/ZOOKEEPER-1519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422447#comment-13422447
]
Stephen Tyree commented on ZOOKEEPER-1519:
--
Duplicating the data wouldn't help in circumstances where the data is a
structure contain malloc'd memory, i.e.:
struct Foo {
const char* bar;
};
> Zookeeper Async calls can reference free()'d memory
> ---
>
> Key: ZOOKEEPER-1519
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1519
> Project: ZooKeeper
> Issue Type: Bug
> Components: c client
>Affects Versions: 3.3.3
> Environment: Ubuntu 11.10, Ubuntu packaged Zookeeper 3.3.3 with some
> backported fixes.
>Reporter: Mark Gius
>
> zoo_acreate() and zoo_aset() take a char * argument for data and prepare a
> call to zookeeper. This char * doesn't seem to be duplicated at any point,
> making it possible that the caller of the asynchronous function might
> potentially free() the char * argument before the zookeeper library completes
> its request. This is unlikely to present a real problem unless the freed
> memory is re-used before zookeeper consumes it. I've been unable to
> reproduce this issue using pure C as a result.
> However, ZKPython is a whole different story. Consider this snippet:
> ok = zookeeper.acreate(handle, path, json.dumps(value),
> acl, flags, callback)
> assert ok == zookeeper.OK
> In this snippet, json.dumps() allocates a string which is passed into the
> acreate(). When acreate() returns, the zookeeper request has been
> constructed with a pointer to the string allocated by json.dumps(). Also
> when acreate() returns, that string is now referenced by 0 things (ZKPython
> doesn't bump the refcount) and the string is eligible for garbage collection
> and re-use. The Zookeeper request now has a pointer to dangerous freed
> memory.
> I've been seeing odd behavior in our development environments for some time
> now, where it appeared as though two separate JSON payloads had been joined
> together. Python has been allocating a new JSON string in the middle of the
> old string that an incomplete zookeeper async call had not yet processed.
> I am not sure if this is a behavior that should be documented, or if the C
> binding implementation needs to be updated to create copies of the data
> payload provided for aset and acreate.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (ZOOKEEPER-2025) Single-node ejection caused apparent reconnection storm, leading to cluster unresponsiveness
Stephen Tyree created ZOOKEEPER-2025: Summary: Single-node ejection caused apparent reconnection storm, leading to cluster unresponsiveness Key: ZOOKEEPER-2025 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2025 Project: ZooKeeper Issue Type: Bug Components: c client, server Affects Versions: 3.4.5 Reporter: Stephen Tyree Description will be included in an attached PDF. The two main questions we have are: 1: What would be the cause of the "Unreasonable Length" error in our context, and how might we prevent it from occurring? 2: What can we do to prevent the reconnection storm that led to the cluster becoming unresponsive? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (ZOOKEEPER-2025) Single-node ejection caused apparent reconnection storm, leading to cluster unresponsiveness
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Tyree updated ZOOKEEPER-2025: - Attachment: zookeeper_issues.pdf Attached PDF describing the issues we observed. > Single-node ejection caused apparent reconnection storm, leading to cluster > unresponsiveness > > > Key: ZOOKEEPER-2025 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2025 > Project: ZooKeeper > Issue Type: Bug > Components: c client, server >Affects Versions: 3.4.5 >Reporter: Stephen Tyree > Attachments: zookeeper_issues.pdf > > > Description will be included in an attached PDF. > The two main questions we have are: > 1: What would be the cause of the "Unreasonable Length" error in our context, > and how might we prevent it from occurring? > 2: What can we do to prevent the reconnection storm that led to the cluster > becoming unresponsive? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (ZOOKEEPER-2025) Single-node ejection caused apparent reconnection storm, leading to cluster unresponsiveness
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14116504#comment-14116504 ] Stephen Tyree commented on ZOOKEEPER-2025: -- The error message is the same, but stack trace is very different. In https://issues.apache.org/jira/browse/ZOOKEEPER-1513, the stack trace appears to be during initialization, whereas in the stack trace we observed, it is during normal operation, presumably when a follower is forwarding a write to the leader. I'm not expert in the code though, so I could be misreading that. > Single-node ejection caused apparent reconnection storm, leading to cluster > unresponsiveness > > > Key: ZOOKEEPER-2025 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2025 > Project: ZooKeeper > Issue Type: Bug > Components: c client, server >Affects Versions: 3.4.5 >Reporter: Stephen Tyree > Attachments: zookeeper_issues.pdf > > > Description will be included in an attached PDF. > The two main questions we have are: > 1: What would be the cause of the "Unreasonable Length" error in our context, > and how might we prevent it from occurring? > 2: What can we do to prevent the reconnection storm that led to the cluster > becoming unresponsive? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (ZOOKEEPER-1705) Certain implementations of C's rand() function coupled with the shuffle in libzookeeper_mt's getaddrs() produce a biased distribution of connections.
Stephen Tyree created ZOOKEEPER-1705: Summary: Certain implementations of C's rand() function coupled with the shuffle in libzookeeper_mt's getaddrs() produce a biased distribution of connections. Key: ZOOKEEPER-1705 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1705 Project: ZooKeeper Issue Type: Bug Components: c client Reporter: Stephen Tyree Priority: Minor Using libzookeeper_mt on an unsupported platform (OpenVMS) with a 5 server connection string, the fourth server in the connection string gets selected approximately only 6% of the time. This appears to be due to some strange properties of the LCG used in OpenVMS's C rand() function. Linux does not exhibit this behavior, but I can't speak for Windows, BSD, etc. It would be prudent, if libzookeeper_mt's behavior is intended to be the same on every platform it operates on (not that OpenVMS is one of those platforms), to use a PRNG of its own choosing. Integrating a defined PRNG, such as the mersenne twister, would give all platforms the same, correct behavior. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1400) Allow logging via callback instead of raw FILE pointer
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13657228#comment-13657228 ] Stephen Tyree commented on ZOOKEEPER-1400: -- This was a great patch. Any way we could rework it so it applies cleanly and get it tested/submitted? > Allow logging via callback instead of raw FILE pointer > -- > > Key: ZOOKEEPER-1400 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1400 > Project: ZooKeeper > Issue Type: Improvement > Components: c client >Affects Versions: 3.5.0 > Environment: Linux >Reporter: Marshall McMullen >Assignee: Marshall McMullen > Fix For: 3.5.0 > > Attachments: ZOOKEEPER-1400-1.patch, ZOOKEEPER-1400.patch > > > The existing logging framework inside the C client uses a raw FILE*. Using a > FILE* is very limiting and potentially dangerous. A safer alternative is to > just provide a callback that the C client will call for each message. In our > environment, we saw some really nasty issues with multiple threads all > connecting to zookeeper via the C Client related to the use of a raw FILE*. > Specifically, if the FILE * is closed and that file descriptor is reused by > the kernel before the C client is notified then the C client will use it's > static global logStream pointer for subsequent logging messages. That FILE* > is now a loose cannon! In our environment, we saw zookeeper log messages > ending up in other sockets and even in our core data path. Clearly this is > dangerous. In our particular case, we'd omitted a call to > zoo_set_log_stream(NULL) to notify C client that the FILE* has been closed. > However, even with that bug fixed, there's still a race condition where log > messages in flight may be sent before the C client is notified of the FILE > closure, and the same problem can happen. > Other issues we've seen involved multiple threads, wherein one would close > the FILE*, and that's a global change that affects all threads connected > within that process. That's a pretty nasty limitation as well. > My proposed change is to allow setting a callback for log messages. A > callback is used in preference to a raw FILE*. If no callback is set, then it > will fallback to the existing FILE*. If that's not set, then it falls back to > stderr as it always has. > While refactoring this code, I removed the need for the double parens in all > the LOG macros as that wasn't necessary and didn't fit with my new approach. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1400) Allow logging via callback instead of raw FILE pointer
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13657520#comment-13657520 ] Stephen Tyree commented on ZOOKEEPER-1400: -- Looked over your change Marshall, thanks for the extremely quick response! Some questions: - Why in create_completion_entry do you LOG_ERROR to LOG_STREAM instead of LOGCALLBACK? Sometimes logging to the callback and sometimes not seems like confusing behavior for an end user. - Why did you change resolve_hosts to be static? If it works with it being static, then it probably should be static, just curious of the reasoning. - You use variadic macros with log_message, and a GCC extension of them at that. Assuming you switch to the C99 way of doing them, does the Zookeeper library target C99? Would this work on Windows? > Allow logging via callback instead of raw FILE pointer > -- > > Key: ZOOKEEPER-1400 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1400 > Project: ZooKeeper > Issue Type: Improvement > Components: c client >Affects Versions: 3.5.0 > Environment: Linux >Reporter: Marshall McMullen >Assignee: Marshall McMullen > Fix For: 3.5.0 > > Attachments: ZOOKEEPER-1400.patch, ZOOKEEPER-1400.patch > > > The existing logging framework inside the C client uses a raw FILE*. Using a > FILE* is very limiting and potentially dangerous. A safer alternative is to > just provide a callback that the C client will call for each message. In our > environment, we saw some really nasty issues with multiple threads all > connecting to zookeeper via the C Client related to the use of a raw FILE*. > Specifically, if the FILE * is closed and that file descriptor is reused by > the kernel before the C client is notified then the C client will use it's > static global logStream pointer for subsequent logging messages. That FILE* > is now a loose cannon! In our environment, we saw zookeeper log messages > ending up in other sockets and even in our core data path. Clearly this is > dangerous. In our particular case, we'd omitted a call to > zoo_set_log_stream(NULL) to notify C client that the FILE* has been closed. > However, even with that bug fixed, there's still a race condition where log > messages in flight may be sent before the C client is notified of the FILE > closure, and the same problem can happen. > Other issues we've seen involved multiple threads, wherein one would close > the FILE*, and that's a global change that affects all threads connected > within that process. That's a pretty nasty limitation as well. > My proposed change is to allow setting a callback for log messages. A > callback is used in preference to a raw FILE*. If no callback is set, then it > will fallback to the existing FILE*. If that's not set, then it falls back to > stderr as it always has. > While refactoring this code, I removed the need for the double parens in all > the LOG macros as that wasn't necessary and didn't fit with my new approach. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1400) Allow logging via callback instead of raw FILE pointer
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13658336#comment-13658336 ] Stephen Tyree commented on ZOOKEEPER-1400: -- re: question 3, this is in your latest patch: +#define LOG_ERROR(_cb, _msg...) if(logLevel>=ZOO_LOG_LEVEL_ERROR) \ +log_message(_cb, ZOO_LOG_LEVEL_ERROR, __LINE__, __func__, _msg) Specifically the line _msg... and its usage in log_message, from the GCC documentation (http://gcc.gnu.org/onlinedocs/cpp/Variadic-Macros.html): "If your macro is complicated, you may want a more descriptive name for the variable argument than __VA_ARGS__. CPP permits this, as an extension. You may write an argument name immediately before the ‘...’; that name is used for the variable argument." Does that clarify my question, or am I crazy? > Allow logging via callback instead of raw FILE pointer > -- > > Key: ZOOKEEPER-1400 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1400 > Project: ZooKeeper > Issue Type: Improvement > Components: c client >Affects Versions: 3.5.0 > Environment: Linux >Reporter: Marshall McMullen >Assignee: Marshall McMullen > Fix For: 3.5.0 > > Attachments: ZOOKEEPER-1400.patch, ZOOKEEPER-1400.patch, > ZOOKEEPER-1400.patch > > > The existing logging framework inside the C client uses a raw FILE*. Using a > FILE* is very limiting and potentially dangerous. A safer alternative is to > just provide a callback that the C client will call for each message. In our > environment, we saw some really nasty issues with multiple threads all > connecting to zookeeper via the C Client related to the use of a raw FILE*. > Specifically, if the FILE * is closed and that file descriptor is reused by > the kernel before the C client is notified then the C client will use it's > static global logStream pointer for subsequent logging messages. That FILE* > is now a loose cannon! In our environment, we saw zookeeper log messages > ending up in other sockets and even in our core data path. Clearly this is > dangerous. In our particular case, we'd omitted a call to > zoo_set_log_stream(NULL) to notify C client that the FILE* has been closed. > However, even with that bug fixed, there's still a race condition where log > messages in flight may be sent before the C client is notified of the FILE > closure, and the same problem can happen. > Other issues we've seen involved multiple threads, wherein one would close > the FILE*, and that's a global change that affects all threads connected > within that process. That's a pretty nasty limitation as well. > My proposed change is to allow setting a callback for log messages. A > callback is used in preference to a raw FILE*. If no callback is set, then it > will fallback to the existing FILE*. If that's not set, then it falls back to > stderr as it always has. > While refactoring this code, I removed the need for the double parens in all > the LOG macros as that wasn't necessary and didn't fit with my new approach. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
