So I think the current plan is: 1. Add timeout in zkclient 2. Ask zkclient to release new version (we need it for few other things too) 3. Rebase on new zkclient 4. Fix this jira and the few others than were waiting for the new zkclient
Does that make sense? Gwen On Mon, Feb 2, 2015 at 8:33 PM, Jaikiran Pai <jai.forums2...@gmail.com> wrote: > I just heard back from Stefan, who manages the ZkClient repo and he seems to > be open to have these changes be part of ZkClient project. I'll be creating > a pull request for that project to have it reviewed and merged. Although I > haven't heard of exact release plans, Stefan's reply did indicate that the > project could be released after this change is merged. > > -Jaikiran > > On Tuesday 03 February 2015 09:03 AM, Jaikiran Pai wrote: >> >> Thanks for pointing to that repo! >> >> I just had a look at it and it appears that the project isn't much active >> (going by the lack of activity). The latest contribution is from Gwen and >> that was around 3 months back. I haven't found release plans for that >> project or a place to ask about it (filing an issue doesn't seem right to >> ask this question). So I'll get in touch with the repo owner and see what >> his plans for the project are. >> >> -Jaikiran >> >> On Monday 02 February 2015 11:33 PM, Gwen Shapira wrote: >>> >>> I did! >>> >>> Thanks for clarifying :) >>> >>> The client that is part of Zookeeper itself actually does support >>> timeouts. >>> >>> On Mon, Feb 2, 2015 at 9:54 AM, Guozhang Wang <wangg...@gmail.com> wrote: >>>> >>>> Hi Jaikiran, >>>> >>>> I think Gwen was talking about contributing to ZkClient project: >>>> >>>> https://github.com/sgroschupf/zkclient >>>> >>>> Guozhang >>>> >>>> >>>> On Sun, Feb 1, 2015 at 5:30 AM, Jaikiran Pai <jai.forums2...@gmail.com> >>>> wrote: >>>> >>>>> Hi Gwen, >>>>> >>>>> Yes, the KafkaZkClient is a wrapper around ZkClient and not a complete >>>>> replacement. >>>>> >>>>> As for contributing to Zookeeper, yes that indeed in on my mind, but I >>>>> haven't yet had a chance to really look deeper into Zookeeper or get in >>>>> touch with their dev team to try and explain this potential improvement >>>>> to >>>>> them. I have no objection to contributing this or something similar to >>>>> Zookeeper directly. I think I should be able to bring this up in the >>>>> Zookeeper dev forum, sometime soon in the next few weekends. >>>>> >>>>> -Jaikiran >>>>> >>>>> >>>>> On Sunday 01 February 2015 11:40 AM, Gwen Shapira wrote: >>>>> >>>>>> It looks like the new KafkaZkClient is a wrapper around ZkClient, but >>>>>> not a replacement. Did I get it right? >>>>>> >>>>>> I think a wrapper for ZkClient can be useful - for example KAFKA-1664 >>>>>> can also use one. >>>>>> >>>>>> However, I'm wondering why not contribute the fix directly to ZKClient >>>>>> project and ask for a release that contains the fix? >>>>>> This will benefit other users of the project who may also need a >>>>>> timeout (thats pretty basic...) >>>>>> >>>>>> As an alternative, if we don't want to collaborate with ZKClient for >>>>>> some reason, forking the project into Kafka will probably give us more >>>>>> control than wrappers and without much downside. >>>>>> >>>>>> Just a thought. >>>>>> >>>>>> Gwen >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Sat, Jan 31, 2015 at 6:32 AM, Jaikiran Pai >>>>>> <jai.forums2...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Neha, Ewen (and others), my initial attempt to solve this is uploaded >>>>>>> here >>>>>>> https://reviews.apache.org/r/30477/. It solves the shutdown problem >>>>>>> and >>>>>>> now >>>>>>> the server shuts down even when Zookeeper has gone down before the >>>>>>> Kafka >>>>>>> server. >>>>>>> >>>>>>> I went with the approach of introducing a custom (enhanced) ZkClient >>>>>>> which >>>>>>> for now allows time outs to be optionally specified for certain >>>>>>> operations. >>>>>>> I intentionally haven't forced the use of this new KafkaZkClient all >>>>>>> over >>>>>>> the code and instead for now have just used it in the KafkaServer. >>>>>>> >>>>>>> Does this patch look like something worth using? >>>>>>> >>>>>>> -Jaikiran >>>>>>> >>>>>>> >>>>>>> On Thursday 29 January 2015 10:41 PM, Neha Narkhede wrote: >>>>>>> >>>>>>>> Ewen is right. ZkClient APIs are blocking and the right fix for this >>>>>>>> seems >>>>>>>> to be patching ZkClient. At some point, if we find ourselves >>>>>>>> fiddling >>>>>>>> too >>>>>>>> much with ZkClient, it wouldn't hurt to write our own little >>>>>>>> zookeeper >>>>>>>> client wrapper. >>>>>>>> >>>>>>>> On Thu, Jan 29, 2015 at 12:57 AM, Ewen Cheslack-Postava >>>>>>>> <e...@confluent.io> >>>>>>>> wrote: >>>>>>>> >>>>>>>> Looks like a bug to me -- the underlying ZK library wraps a lot of >>>>>>>>> >>>>>>>>> blocking >>>>>>>>> method implementations with waitUntilConnected() calls without any >>>>>>>>> timeouts. Ideally we could just add a version of >>>>>>>>> ZkUtils.getController() >>>>>>>>> with a timeout, but I don't see an easy way to accomplish that with >>>>>>>>> ZkClient. >>>>>>>>> >>>>>>>>> There's at least one other call to ZkUtils besides the one in the >>>>>>>>> stacktrace you gave that would cause the same issue, possibly more >>>>>>>>> that >>>>>>>>> aren't directly called in that method. One ugly solution would be >>>>>>>>> to >>>>>>>>> use >>>>>>>>> an >>>>>>>>> extra thread during shutdown to trigger timeouts, but I'd imagine >>>>>>>>> we >>>>>>>>> probably have other threads that could end up blocking in similar >>>>>>>>> ways. >>>>>>>>> >>>>>>>>> I filed https://issues.apache.org/jira/browse/KAFKA-1907 to track >>>>>>>>> the >>>>>>>>> issue. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Jan 26, 2015 at 6:35 AM, Jaikiran Pai < >>>>>>>>> jai.forums2...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> The main culprit is this thread which goes into "forever retry >>>>>>>>>> >>>>>>>>>> connection >>>>>>>>>> to a closed zookeeper" when I shutdown Kafka (via a Ctrl + C) >>>>>>>>>> after >>>>>>>>>> zookeeper has already been shutdown. I have attached the complete >>>>>>>>>> thread >>>>>>>>>> dump, but I don't know if it will be delivered to the mailing >>>>>>>>>> list. >>>>>>>>>> >>>>>>>>>> "Thread-2" prio=10 tid=0xb3305000 nid=0x4758 waiting on condition >>>>>>>>>> [0x6ad69000] >>>>>>>>>> java.lang.Thread.State: TIMED_WAITING (parking) >>>>>>>>>> at sun.misc.Unsafe.park(Native Method) >>>>>>>>>> - parking to wait for <0x70a93368> (a >>>>>>>>>> java.util.concurrent.locks. >>>>>>>>>> AbstractQueuedSynchronizer$ConditionObject) >>>>>>>>>> at java.util.concurrent.locks.LockSupport.parkUntil( >>>>>>>>>> LockSupport.java:267) >>>>>>>>>> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ >>>>>>>>>> ConditionObject.awaitUntil(AbstractQueuedSynchronizer.java:2130) >>>>>>>>>> at >>>>>>>>>> org.I0Itec.zkclient.ZkClient.waitForKeeperState(ZkClient.java:636) >>>>>>>>>> at >>>>>>>>>> org.I0Itec.zkclient.ZkClient.waitUntilConnected(ZkClient.java:619) >>>>>>>>>> at >>>>>>>>>> org.I0Itec.zkclient.ZkClient.waitUntilConnected(ZkClient.java:615) >>>>>>>>>> at >>>>>>>>>> >>>>>>>>> org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:679) >>>>>>>>> >>>>>>>>>> at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:766) >>>>>>>>>> at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:761) >>>>>>>>>> at >>>>>>>>>> kafka.utils.ZkUtils$.readDataMaybeNull(ZkUtils.scala:456) >>>>>>>>>> at kafka.utils.ZkUtils$.getController(ZkUtils.scala:65) >>>>>>>>>> at kafka.server.KafkaServer.kafka$server$KafkaServer$$ >>>>>>>>>> controlledShutdown(KafkaServer.scala:194) >>>>>>>>>> at kafka.server.KafkaServer$$anonfun$shutdown$1.apply$mcV$ >>>>>>>>>> sp(KafkaServer.scala:269) >>>>>>>>>> at kafka.utils.Utils$.swallow(Utils.scala:172) >>>>>>>>>> at kafka.utils.Logging$class.swallowWarn(Logging.scala:92) >>>>>>>>>> at kafka.utils.Utils$.swallowWarn(Utils.scala:45) >>>>>>>>>> at kafka.utils.Logging$class.swallow(Logging.scala:94) >>>>>>>>>> at kafka.utils.Utils$.swallow(Utils.scala:45) >>>>>>>>>> at kafka.server.KafkaServer.shutdown(KafkaServer.scala:269) >>>>>>>>>> at kafka.server.KafkaServerStartable.shutdown( >>>>>>>>>> KafkaServerStartable.scala:42) >>>>>>>>>> at kafka.Kafka$$anon$1.run(Kafka.scala:42) >>>>>>>>>> >>>>>>>>>> -Jaikiran >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Monday 26 January 2015 05:46 AM, Neha Narkhede wrote: >>>>>>>>>> >>>>>>>>>> For a clean shutdown, the broker tries to talk to the controller >>>>>>>>>> and >>>>>>>>>> also >>>>>>>>>> issues reads to zookeeper. Possibly that is where it tries to >>>>>>>>>>> >>>>>>>>>>> reconnect >>>>>>>>>>> >>>>>>>>>> to >>>>>>>>>> zk. It will help to look at the thread dump. >>>>>>>>>>> >>>>>>>>>>> Thanks >>>>>>>>>>> Neha >>>>>>>>>>> >>>>>>>>>>> On Fri, Jan 23, 2015 at 8:53 PM, Jaikiran Pai < >>>>>>>>>>> jai.forums2...@gmail.com >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> I was just playing around with the RC2 of 0.8.2 and noticed >>>>>>>>>>> that >>>>>>>>>>> if I >>>>>>>>>>> >>>>>>>>>>>> shutdown zookeeper first I can't shutdown Kafka server at all >>>>>>>>>>>> since >>>>>>>>>>>> it >>>>>>>>>>>> goes >>>>>>>>>>>> into a never ending attempt to reconnect with zookeeper. I had >>>>>>>>>>>> to >>>>>>>>>>>> kill >>>>>>>>>>>> the >>>>>>>>>>>> Kafka process to stop it. I tried it against trunk too and there >>>>>>>>>>>> too I >>>>>>>>>>>> see >>>>>>>>>>>> the same issue. Should I file a JIRA for this and see if I can >>>>>>>>>>>> come >>>>>>>>>>>> up >>>>>>>>>>>> with >>>>>>>>>>>> a patch? >>>>>>>>>>>> >>>>>>>>>>>> FWIW, here's the unending (and IMO too frequent) attempts at >>>>>>>>>>>> trying >>>>>>>>>>>> to >>>>>>>>>>>> reconnect. I've a thread dump too which shows that the other >>>>>>>>>>>> thread >>>>>>>>>>>> >>>>>>>>>>> which >>>>>>>>>> >>>>>>>>>> is trying to complete a controlled shutdown of Kafka is blocked >>>>>>>>>>>> >>>>>>>>>>>> forever >>>>>>>>>>>> for >>>>>>>>>>>> the zookeeper to be up. I can attach it to the JIRA. >>>>>>>>>>>> >>>>>>>>>>>> 2015-01-24 10:15:46,278] WARN Session 0x14b1a4136800000 for >>>>>>>>>>>> server >>>>>>>>>>>> >>>>>>>>>>> null, >>>>>>>>>> >>>>>>>>>> unexpected error, closing socket connection and attempting >>>>>>>>>> reconnect >>>>>>>>>>>> >>>>>>>>>>>> (org.apache.zookeeper.ClientCnxn) >>>>>>>>>>>> java.net.ConnectException: Connection refused >>>>>>>>>>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native >>>>>>>>>>>> Method) >>>>>>>>>>>> at sun.nio.ch.SocketChannelImpl.finishConnect( >>>>>>>>>>>> SocketChannelImpl.java:739) >>>>>>>>>>>> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport( >>>>>>>>>>>> ClientCnxnSocketNIO.java:361) >>>>>>>>>>>> at org.apache.zookeeper.ClientCnxn$SendThread.run( >>>>>>>>>>>> ClientCnxn.java:1081) >>>>>>>>>>>> [2015-01-24 10:15:47,437] INFO Opening socket connection to >>>>>>>>>>>> server >>>>>>>>>>>> localhost/127.0.0.1:2181. Will not attempt to authenticate using >>>>>>>>>>>> SASL >>>>>>>>>>>> (unknown error) (org.apache.zookeeper.ClientCnxn) >>>>>>>>>>>> [2015-01-24 10:15:47,438] WARN Session 0x14b1a4136800000 for >>>>>>>>>>>> server >>>>>>>>>>>> >>>>>>>>>>> null, >>>>>>>>>> >>>>>>>>>> unexpected error, closing socket connection and attempting >>>>>>>>>> reconnect >>>>>>>>>>>> >>>>>>>>>>>> (org.apache.zookeeper.ClientCnxn) >>>>>>>>>>>> java.net.ConnectException: Connection refused >>>>>>>>>>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native >>>>>>>>>>>> Method) >>>>>>>>>>>> at sun.nio.ch.SocketChannelImpl.finishConnect( >>>>>>>>>>>> SocketChannelImpl.java:739) >>>>>>>>>>>> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport( >>>>>>>>>>>> ClientCnxnSocketNIO.java:361) >>>>>>>>>>>> at org.apache.zookeeper.ClientCnxn$SendThread.run( >>>>>>>>>>>> ClientCnxn.java:1081) >>>>>>>>>>>> [2015-01-24 10:15:49,056] INFO Opening socket connection to >>>>>>>>>>>> server >>>>>>>>>>>> localhost/127.0.0.1:2181. Will not attempt to authenticate using >>>>>>>>>>>> SASL >>>>>>>>>>>> (unknown error) (org.apache.zookeeper.ClientCnxn) >>>>>>>>>>>> [2015-01-24 10:15:49,057] WARN Session 0x14b1a4136800000 for >>>>>>>>>>>> server >>>>>>>>>>>> >>>>>>>>>>> null, >>>>>>>>>> >>>>>>>>>> unexpected error, closing socket connection and attempting >>>>>>>>>> reconnect >>>>>>>>>>>> >>>>>>>>>>>> (org.apache.zookeeper.ClientCnxn) >>>>>>>>>>>> java.net.ConnectException: Connection refused >>>>>>>>>>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native >>>>>>>>>>>> Method) >>>>>>>>>>>> at sun.nio.ch.SocketChannelImpl.finishConnect( >>>>>>>>>>>> SocketChannelImpl.java:739) >>>>>>>>>>>> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport( >>>>>>>>>>>> ClientCnxnSocketNIO.java:361) >>>>>>>>>>>> at org.apache.zookeeper.ClientCnxn$SendThread.run( >>>>>>>>>>>> ClientCnxn.java:1081) >>>>>>>>>>>> [2015-01-24 10:15:50,801] INFO Opening socket connection to >>>>>>>>>>>> server >>>>>>>>>>>> localhost/127.0.0.1:2181. Will not attempt to authenticate using >>>>>>>>>>>> SASL >>>>>>>>>>>> (unknown error) (org.apache.zookeeper.ClientCnxn) >>>>>>>>>>>> [2015-01-24 10:15:50,802] WARN Session 0x14b1a4136800000 for >>>>>>>>>>>> server >>>>>>>>>>>> >>>>>>>>>>> null, >>>>>>>>>> >>>>>>>>>> unexpected error, closing socket connection and attempting >>>>>>>>>> reconnect >>>>>>>>>>>> >>>>>>>>>>>> (org.apache.zookeeper.ClientCnxn) >>>>>>>>>>>> java.net.ConnectException: Connection refused >>>>>>>>>>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native >>>>>>>>>>>> Method) >>>>>>>>>>>> at sun.nio.ch.SocketChannelImpl.finishConnect( >>>>>>>>>>>> SocketChannelImpl.java:739) >>>>>>>>>>>> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport( >>>>>>>>>>>> ClientCnxnSocketNIO.java:361) >>>>>>>>>>>> at org.apache.zookeeper.ClientCnxn$SendThread.run( >>>>>>>>>>>> ClientCnxn.java:1081) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -Jaikiran >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Ewen >>>>>>>>> >>>>>>>>> >>>> >>>> -- >>>> -- Guozhang >> >> >
