I didn't think client ID could be negative. Please open an issue in Jira and assign this to me.
-Jordan > On Sep 20, 2017, at 1:09 PM, Mark Johnson <[email protected]> > wrote: > > Think I found the problem. In the code that determines the EphemeralType > it is looking at the owner (which is the client ID or connection ID): > > EphemeralType.java: > > public static EphemeralType get(long ephemeralOwner) { > if (ephemeralOwner == CONTAINER_EPHEMERAL_OWNER) { > return CONTAINER; > } > if (ephemeralOwner < 0) { > return TTL; > } > return (ephemeralOwner == 0) ? VOID : NORMAL; > } > > However my connection ID is: > > header.getClientId(): -720548323429908480 > > This causes the code to think this is a TTL Ephemeral node instead of a > NORMAL Ephemeral node. > > This also explains why this is random - if my client ID is non-negative > then the node gets added correctly. > > Can someone verify that this is a bug? Can a ticket be created for this? > > Thanks in advance, > Mark > > > On Wed, Sep 20, 2017 at 1:17 AM, Mark Johnson <[email protected]> > wrote: > >> Hi Abraham, >> >> I believe I'm using nio - from the log: >> >> 2017-09-19 15:25:51,761 [myid:010000001020] - INFO >> [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with >> 10s sessionless connection timeout, 1 selector thread(s), 8 worker threads, >> and 64 kB direct buffers. >> >> And here's my zoo.cfg: >> >> tickTime=2000 >> initLimit=100 >> syncLimit=5 >> dataDir=/var/lib/zookeeper >> clientPort=2181 >> autopurge.snapRetainCount=3 >> # Purge task interval in hours >> autopurge.purgeInterval=1 >> minSessionTimeout=4000 >> maxSessionTimeout=60000 >> standaloneEnabled=false >> reconfigEnabled=true >> skipACL=yes >> dynamicConfigFile=/opt/zookeeper/conf/zoo.cfg.dynamic >> >> *However, the problem is not removal of the ephemeral nodes, it's that >> they're not getting created in the first place,* When I use the CLI to >> create an ephemeral node via: >> >> create -e /testnode >> >> It's not marked as an ephemeral node and hangs around forever. From stat >> /testnode: >> >> ephemeralOwner = 0x0 >> >> Sometimes Zookeeper will come up and create ephemeral nodes and they work >> correctly (deleted after the connection is terminated). But usually (with >> the version I'm using- 3.5.3-beta2) they are not created. >> >> Are you familiar with the code? I've downloaded the source and have been >> looking for the place where the znodes are actually created but have not >> found that section. If you can point me to the right area of the code I >> can probably troubleshoot this issue. >> >> Thanks for any light you can shed on this problem! >> >> Mark >> >> >> On Tue, Sep 19, 2017 at 6:18 PM, Abraham Fine <[email protected]> wrote: >> >>> Hi Mark- >>> >>> I'm assuming you are using nio and not netty, please let me know if that >>> is not the case. >>> >>> Have you possibly set zookeeper.nio.sessionlessCnxnTimeout? This value >>> is 10s by default, so it may take a little more than ten seconds by >>> default for ephemeral nodes to be deleted. >>> >>> Thanks, >>> Abe >>> >>> >>> On Tue, Sep 19, 2017, at 08:16, Mark Johnson wrote: >>>> Hi, >>>> >>>> I'm running version 3.5.3-BETA of Zookeeper in a Docker container and >>>> usually cannot create Ephemeral znodes, either through the CLI or though >>>> a Java program. It appears the owner is not set. Here's an example >>>> session: >>>> >>>> ============================================================ >>> ================================================ >>>> [root@Appliance ~]# docker exec -it 9b7cd1e80b5b /bin/bash >>>> zookeeper@9b7cd1e80b5b:/opt/zookeeper-3.5.3-beta$ bin/zkCli.sh >>>> Connecting to localhost:2181 >>>> 2017-09-19 10:03:31,373 [myid:] - INFO [main:Environment@109] - Client >>>> environment:zookeeper.version=3.5.3-beta-8ce24f9e675cbefffb8 >>> f21a47e06b42864475a60, >>>> built on 04/03/2017 16:19 GMT >>>> 2017-09-19 10:03:31,378 [myid:] - INFO [main:Environment@109] - Client >>>> environment:host.name=9b7cd1e80b5b >>>> 2017-09-19 10:03:31,378 [myid:] - INFO [main:Environment@109] - Client >>>> environment:java.version=1.8.0_144 >>>> 2017-09-19 10:03:31,380 [myid:] - INFO [main:Environment@109] - Client >>>> environment:java.vendor=Oracle Corporation >>>> 2017-09-19 10:03:31,380 [myid:] - INFO [main:Environment@109] - Client >>>> environment:java.home=/usr/lib/jvm/java-8-oracle/jre >>>> 2017-09-19 10:03:31,380 [myid:] - INFO [main:Environment@109] - Client >>>> environment:java.class.path=/opt/zookeeper-3.5.3-beta/bin/.. >>> /build/classes:/opt/zookeeper-3.5.3-beta/bin/../build/lib/*. >>> jar:/opt/zookeeper-3.5.3-beta/bin/../lib/slf4j-log4j12-1.7.5.jar >>>> :/opt/zookeeper-3.5.3-beta/bin/../lib/slf4j-api-1.7.5.jar:/ >>> opt/zookeeper-3.5.3-beta/bin/../lib/netty-3.10.5.Final.jar:/ >>> opt/zookeeper-3.5.3-beta/bin/../lib/log4j-1.2.17.jar:/opt/ >>> zookeeper-3.5.3-beta/bin/../lib/jline-2.11.jar:/opt/zookee >>> per-3.5.3-beta/bin/.. >>>> /lib/jetty-util-9.2.18.v20160721.jar:/opt/zookeeper-3.5.3- >>> beta/bin/../lib/jetty-servlet-9.2.18.v20160721.jar:/opt/ >>> zookeeper-3.5.3-beta/bin/../lib/jetty-server-9.2.18.v20160 >>> 721.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/jetty-security >>> -9.2.18.v20160721.jar:/op >>>> t/zookeeper-3.5.3-beta/bin/../lib/jetty-io-9.2.18.v20160721. >>> jar:/opt/zookeeper-3.5.3-beta/bin/../lib/jetty-http-9.2.18.v >>> 20160721.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/javax.serv >>> let-api-3.1.0.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/ >>> jackson-mapper-asl-1. >>>> 9.11.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/jackson-core-a >>> sl-1.9.11.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/commons-c >>> li-1.2.jar:/opt/zookeeper-3.5.3-beta/bin/../zookeeper-3.5.3- >>> beta.jar:/opt/zookeeper-3.5.3-beta/bin/../src/java/lib/*.jar >>> :/opt/zookeeper- >>>> 3.5.3-beta/bin/../conf: >>>> 2017-09-19 10:03:31,380 [myid:] - INFO [main:Environment@109] - Client >>>> environment:java.library.path=/usr/java/packages/lib/amd64:/ >>> usr/lib64:/lib64:/lib:/usr/lib >>>> 2017-09-19 10:03:31,380 [myid:] - INFO [main:Environment@109] - Client >>>> environment:java.io.tmpdir=/tmp >>>> 2017-09-19 10:03:31,380 [myid:] - INFO [main:Environment@109] - Client >>>> environment:java.compiler=<NA> >>>> 2017-09-19 10:03:31,380 [myid:] - INFO [main:Environment@109] - Client >>>> environment:os.name=Linux >>>> 2017-09-19 10:03:31,380 [myid:] - INFO [main:Environment@109] - Client >>>> environment:os.arch=amd64 >>>> 2017-09-19 10:03:31,381 [myid:] - INFO [main:Environment@109] - Client >>>> environment:os.version=3.10.0-514.26.2.el7.x86_64 >>>> 2017-09-19 10:03:31,381 [myid:] - INFO [main:Environment@109] - Client >>>> environment:user.name=zookeeper >>>> 2017-09-19 10:03:31,381 [myid:] - INFO [main:Environment@109] - Client >>>> environment:user.home=/home/zookeeper >>>> 2017-09-19 10:03:31,381 [myid:] - INFO [main:Environment@109] - Client >>>> environment:user.dir=/opt/zookeeper-3.5.3-beta >>>> 2017-09-19 10:03:31,381 [myid:] - INFO [main:Environment@109] - Client >>>> environment:os.memory.free=233MB >>>> 2017-09-19 10:03:31,382 [myid:] - INFO [main:Environment@109] - Client >>>> environment:os.memory.max=240MB >>>> 2017-09-19 10:03:31,383 [myid:] - INFO [main:Environment@109] - Client >>>> environment:os.memory.total=240MB >>>> 2017-09-19 10:03:31,386 [myid:] - INFO [main:ZooKeeper@865] - >>>> Initiating client connection, connectString=localhost:2181 >>>> sessionTimeout=30000 >>>> watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@28ba21f3 >>>> 2017-09-19 10:03:31,397 [myid:] - INFO [main:ClientCnxnSocket@236] - >>>> jute.maxbuffer value is 4194304 Bytes >>>> 2017-09-19 10:03:31,410 [myid:localhost:2181] - INFO >>>> [main-SendThread(localhost:2181):ClientCnxn$SendThread@1113] - Opening >>>> socket connection to server localhost/127.0.0.1:2181. Will not attempt >>>> to authenticate using SASL (unknown error) >>>> Welcome to ZooKeeper! >>>> JLine support is enabled >>>> 2017-09-19 10:03:31,493 [myid:localhost:2181] - INFO >>>> [main-SendThread(localhost:2181):ClientCnxn$SendThread@948] - Socket >>>> connection established, initiating session, client: /127.0.0.1:44882, >>>> server: localhost/127.0.0.1:2181 >>>> 2017-09-19 10:03:31,506 [myid:localhost:2181] - INFO >>>> [main-SendThread(localhost:2181):ClientCnxn$SendThread@1381] - Session >>>> establishment complete on server localhost/127.0.0.1:2181, sessionid = >>>> 0xeb0000480b3c005c, negotiated timeout = 30000 >>>> >>>> WATCHER:: >>>> >>>> WatchedEvent state:SyncConnected type:None path:null >>>> [zk: localhost:2181(CONNECTED) 0] create -e /testnode >>>> Created /testnode >>>> [zk: localhost:2181(CONNECTED) 1] stat /testnode >>>> cZxid = 0x2000002c4 >>>> ctime = Tue Sep 19 10:03:50 CDT 2017 >>>> mZxid = 0x2000002c4 >>>> mtime = Tue Sep 19 10:03:50 CDT 2017 >>>> pZxid = 0x2000002c4 >>>> cversion = 0 >>>> dataVersion = 0 >>>> aclVersion = 0 >>>> *ephemeralOwner = 0x0* >>>> dataLength = 0 >>>> numChildren = 0 >>>> [zk: localhost:2181(CONNECTED) 2] >>>> ============================================================ >>> ============================================== >>>> >>>> If I disconnect and reconnect, the znode is still present as well. If I >>>> restart Zookeeper, many times this will resolve itself somehow and >>>> creation of ephemeral nodes will work correctly. I have also tried this >>>> in a clustered configuration and get cases where one Zookeeper will >>>> create ephemeral nodes correctly and another Zookeeper will exhibit the >>>> above issue. >>>> >>>> Please let me know what other information would be of use or other >>>> things I could be looking for. >>>> >>>> Any help appreciated. >>>> >>>> Mark >>>> >>>> >>>> >>> >> >>
