Hi, I am facing an issue with sequence value when adding new nodes using PERSISTENT_SEQUENTIAL flag:
[zk: localhost:2181(CONNECTED) 5] create -s /tokens/ "test" 'Node already exists: /tokens/ Basically, adding new nodes with -s flag has started to fail because zookeeper claims that they already exist. Adding 'test' to the path fixes this problem: [zk: localhost:2181(CONNECTED) 13] create -s /tokens/test test Created /tokens/test0000039810 So the sequence is 0000039810. There already exists a path /tokens/0000039810 so the previous command fails with 'Node already exists" because it tried to add /tokens/0000039810. The sequence used is wrong. Here is stat: [zk: localhost:2181(CONNECTED) 12] get /tokens null cZxid = 0x10000001e ctime = Wed May 14 13:33:31 CEST 2014 mZxid = 0x10000001e mtime = Wed May 14 13:33:31 CEST 2014 pZxid = 0x2100000044 cversion = 71987 dataVersion = 0 aclVersion = 0 ephemeralOwner = 0x0 dataLength = 0 numChildren = 7633 Current solution would be to add around 150 fake nodes up to /tokens/test0000039944 to increase the counter. /tokens/0000039944 is the last path that exists. According to zookeeper documentation: When creating a znode you can also request that ZooKeeper append a monotonicly increasing counter to the end of path. This counter is unique to the parent znode. The counter has a format of %010d -- that is 10 digits with 0 (zero) padding (the counter is formatted in this way to simplify sorting), i.e. "<path>0000000001". See Queue Recipe for an example use of this feature. Note: the counter used to store the next sequence number is a signed int (4bytes) maintained by the parent node, the counter will overflow when incremented beyond 2147483647 (resulting in a name "<path>-2147483647"). My questions are: 1. Where exactly is the counter stored/maintained by the parent node and how is it possible to change it. 2. How this could happen that parent node has wrong sequence? I am pretty sure that all creates where done via PERSISTENT_SEQUENTIAL. It is a 3-node cluster. Could restart break it? Zookeeper version is 3.4.6 but it was upgraded from 3.3.4. Upgrade did not break it as it was working yesterday… Thanks in advance!
