[jira] [Created] (BOOKKEEPER-36) Client backpressure
Client backpressure --- Key: BOOKKEEPER-36 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-36 Project: Bookkeeper Issue Type: Improvement Components: bookkeeper-client Reporter: Flavio Junqueira Assignee: Flavio Junqueira Priority: Critical The way we currently throttle on the client is by counting the number of outstanding operation on LedgerHandle, and having the application select what an appropriate value is. This is not a good way of doing it because the application has to guess what a good value is. We need to implement some form of backpressure instead to make sure we throttle only when the system is saturated. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1140) server shutdown is not stopping threads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072912#comment-13072912 ] Patrick Hunt commented on ZOOKEEPER-1140: - Comment on list from lakshman_ch: In QuorumPeer, when the peer is in LOOKING state we are starting ReadOnlyZooKeeperServer in a separate thread. And we are shutting down this server even before startup which has no effect. Also, as this is not a blocking call QP keeps on spawning new servers. 1) ReadOnlyZooKeeperServer.startup() need not be called in separate a thread. 2) ReadOnlyZooKeeperServer.startup() is not a blocking call. Need to introduce a method like Leader.lead(), Follower.followLeader() 3) Shutdown should be called only after the a/m blocking call is returned. server shutdown is not stopping threads --- Key: ZOOKEEPER-1140 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1140 Project: ZooKeeper Issue Type: Bug Components: server, tests Affects Versions: 3.4.0 Reporter: Patrick Hunt Priority: Blocker Fix For: 3.4.0 Near the end of QuorumZxidSyncTest there are tons of threads running - 115 ProcessThread threads, similar numbers of SessionTracker. Also I see ~100 ReadOnlyRequestProcessor - why is this running as a separate thread? (henry/flavio?) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Out of memory running ZK unit tests against trunk
Hi Laxman, you want to take a stab at it? https://issues.apache.org/jira/browse/ZOOKEEPER-1140 Can you followup with Flavio/Henry about the readonly issue? Shouldn't such a feature only be enabled when R/O support is enabled? (my assumption is that it should be off by default, on via configuration option) Patrick On Fri, Jul 29, 2011 at 7:00 AM, Laxman lakshman...@huawei.com wrote: In QuorumPeer, when the peer is in LOOKING state we are starting ReadOnlyZooKeeperServer in a separate thread. And we are shutting down this server even before startup which has no effect. Also, as this is not a blocking call QP keeps on spawning new servers. 1) ReadOnlyZooKeeperServer.startup() need not be called in separate a thread. 2) ReadOnlyZooKeeperServer.startup() is not a blocking call. Need to introduce a method like Leader.lead(), Follower.followLeader() 3) Shutdown should be called only after the a/m blocking call is returned. -Original Message- From: Patrick Hunt [mailto:ph...@apache.org] Sent: Friday, July 29, 2011 6:24 AM To: dev@zookeeper.apache.org Subject: Re: Out of memory running ZK unit tests against trunk Near the end of this test (QuorumZxidSyncTest) there are tons of threads running - 115 ProcessThread threads, similar numbers of SessionTracker. Also I see ~100 ReadOnlyRequestProcessor - why is this running as a separate thread? (henry/flavio?) Regardless, I'll enter a 3.4.0 blocker to clean this up - I suspect that the server shutdown is not shutting down fully for some reason. Patrick On Thu, Jul 28, 2011 at 5:28 PM, Mahadev Konar maha...@hortonworks.com wrote: Nice find Pat. I cant see a reason on why that should happen. Can we just do a stack dump and compare? thanks mahadev On Thu, Jul 28, 2011 at 1:54 PM, Patrick Hunt ph...@apache.org wrote: I tracked this down to a low ulimit setting on the particular jenkins host where this was failing (max processes). Specifically the following test was failing on trunk, but not on branch 3_3, which concerns me ./src/java/test/org/apache/zookeeper/test/QuorumZxidSyncTest.java there haven't been any real changes to this test between versions, any insight into why the server is using more threads in trunk vs branch33? Patrick On Fri, Jul 22, 2011 at 10:58 AM, Patrick Hunt ph...@apache.org wrote: I've never seen this before, but in my CI environment (sun jdk 1.6.0_20) I'm seeing some intermittent failures such as the following. Has anyone added/modified tests for 3.4.0 that might be using more threads/memory than previously? Creating ZK clients but not closing them, etc... java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:597) at org.apache.zookeeper.server.NIOServerCnxnFactory.start(NIOServerCnxnFactory. java:114) at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:406) at org.apache.zookeeper.test.QuorumBase.startServers(QuorumBase.java:186) at org.apache.zookeeper.test.QuorumBase.setUp(QuorumBase.java:103) at org.apache.zookeeper.test.QuorumBase.setUp(QuorumBase.java:67) at org.apache.zookeeper.test.QuorumZxidSyncTest.setUp(QuorumZxidSyncTest.java:3 9) Patrick
[jira] [Commented] (ZOOKEEPER-1140) server shutdown is not stopping threads
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072914#comment-13072914 ] Patrick Hunt commented on ZOOKEEPER-1140: - Shouldn't R/O support only be enabled via an explicit configuration option? i.e. off by default. server shutdown is not stopping threads --- Key: ZOOKEEPER-1140 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1140 Project: ZooKeeper Issue Type: Bug Components: server, tests Affects Versions: 3.4.0 Reporter: Patrick Hunt Priority: Blocker Fix For: 3.4.0 Near the end of QuorumZxidSyncTest there are tons of threads running - 115 ProcessThread threads, similar numbers of SessionTracker. Also I see ~100 ReadOnlyRequestProcessor - why is this running as a separate thread? (henry/flavio?) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (ZOOKEEPER-1034) perl bindings should automatically find the zookeeper c-client headers
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Harteau updated ZOOKEEPER-1034: Attachment: ant.test-nopatch 'ant test' output from trunk, after removing ZOOKEEPER-1033 and ZOOKEEPER-1034 [nrh@toki ~/projects/apache-zookeeper] grep -i fail ant.test-nopatch|grep -v 'Failures: 0' [junit] Tests run: 1, Failures: 1, Errors: 0, Time elapsed: 91.533 sec [junit] Test org.apache.zookeeper.test.ClientPortBindTest FAILED [junit] Tests run: 3, Failures: 2, Errors: 0, Time elapsed: 81.774 sec [junit] Test org.apache.zookeeper.test.FollowerResyncConcurrencyTest FAILED [junit] Tests run: 15, Failures: 1, Errors: 0, Time elapsed: 156.346 sec [junit] Test org.apache.zookeeper.test.QuorumTest FAILED BUILD FAILED /Users/nrh/projects/apache-zookeeper/build.xml:870: Tests failed! perl bindings should automatically find the zookeeper c-client headers -- Key: ZOOKEEPER-1034 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1034 Project: ZooKeeper Issue Type: Bug Components: contrib Affects Versions: 3.3.3 Reporter: Nicholas Harteau Assignee: Nicholas Harteau Priority: Minor Fix For: 3.4.0 Attachments: ZOOKEEPER-1034-trunk.patch, ant.test-nopatch Installing Net::ZooKeeper from cpan or the zookeeper distribution tarballs will always fail due to not finding c-client header files. In conjunction with ZOOKEEPER-1033 update perl bindings to look for c-client header files in INCDIR/zookeeper/ a.k.a. make installs of Net::ZooKeeper via cpan/cpanm/whatever *just work*, assuming you've already got the zookeeper c client installed. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-938) Support Kerberos authentication of clients.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073106#comment-13073106 ] Eugene Koontz commented on ZOOKEEPER-938: - Hi Mahadev, I am uploading my latest patch now, but am also doing integration tests with HBase. If those turn up any problems I will refresh the patch again. Support Kerberos authentication of clients. --- Key: ZOOKEEPER-938 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-938 Project: ZooKeeper Issue Type: New Feature Components: java client, server Reporter: Eugene Koontz Assignee: Eugene Koontz Fix For: 3.4.0, 3.5.0 Attachments: NIOServerCnxn.patch, ZOOKEEPER-938.patch, ZOOKEEPER-938.patch, ZOOKEEPER-938.patch, ZOOKEEPER-938.patch, ZOOKEEPER-938.patch, ZOOKEEPER-938.patch, ZOOKEEPER-938.patch, ZOOKEEPER-938.patch, ZOOKEEPER-938.patch, ZOOKEEPER-938.patch, ZOOKEEPER-938.patch, jaas.conf, sasl.patch Support Kerberos authentication of clients. The following usage would let an admin use Kerberos authentication to assign ACLs to authenticated clients. 1. Admin logs into zookeeper (not necessarily through Kerberos however). 2. Admin decides that a new node called '/mynode' should be owned by the user 'zkclient' and have full permissions on this. 3. Admin does: zk create /mynode content sasl:zkcli...@foofers.org:cdrwa 4. User 'zkclient' logins to kerberos using the command line utility 'kinit'. 5. User connects to zookeeper server using a Kerberos-enabled version of zkClient (ZookeeperMain). 6. Behind the scenes, the client and server exchange authentication information. User is now authenticated as 'zkclient'. 7. User accesses /mynode with permissions 'cdrwa'. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-938) Support Kerberos authentication of clients.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073122#comment-13073122 ] Mahadev konar commented on ZOOKEEPER-938: - thanks a lot Eugene! Support Kerberos authentication of clients. --- Key: ZOOKEEPER-938 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-938 Project: ZooKeeper Issue Type: New Feature Components: java client, server Reporter: Eugene Koontz Assignee: Eugene Koontz Fix For: 3.4.0, 3.5.0 Attachments: NIOServerCnxn.patch, ZOOKEEPER-938.patch, ZOOKEEPER-938.patch, ZOOKEEPER-938.patch, ZOOKEEPER-938.patch, ZOOKEEPER-938.patch, ZOOKEEPER-938.patch, ZOOKEEPER-938.patch, ZOOKEEPER-938.patch, ZOOKEEPER-938.patch, ZOOKEEPER-938.patch, ZOOKEEPER-938.patch, ZOOKEEPER-938.patch, jaas.conf, sasl.patch Support Kerberos authentication of clients. The following usage would let an admin use Kerberos authentication to assign ACLs to authenticated clients. 1. Admin logs into zookeeper (not necessarily through Kerberos however). 2. Admin decides that a new node called '/mynode' should be owned by the user 'zkclient' and have full permissions on this. 3. Admin does: zk create /mynode content sasl:zkcli...@foofers.org:cdrwa 4. User 'zkclient' logins to kerberos using the command line utility 'kinit'. 5. User connects to zookeeper server using a Kerberos-enabled version of zkClient (ZookeeperMain). 6. Behind the scenes, the client and server exchange authentication information. User is now authenticated as 'zkclient'. 7. User accesses /mynode with permissions 'cdrwa'. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
