[
https://issues.apache.org/jira/browse/ZOOKEEPER-2052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14160006#comment-14160006
]
Yip Ng commented on ZOOKEEPER-2052:
-----------------------------------
Hi Hongchao:
As illustrated in the junit testcase, the first multi failure was the culprit
to this issue. When the first multi with delete executes, it stores the
pending changes for the path but in this specific case, there were no
outstanding changes for those paths. The getPendingChanges() method mistakenly
saved the records that was obtained by the ZKDatabase with zxid = -1 (these are
not outstanding changes). When multi delete fails(there were ephemeral nodes
under /mymetadata/resources), it attempts to remove the added change records
from the multi operation and restore the outstanding changes for the paths.
But there were none to begin with, so the rollback actually added new change
records. When the multi op rolls back, one would expect that there shouldn't
be any *new* change records.
When client B's session expired, all its ephemeral nodes get removed, so the
subsequent multi delete should succeed; however, due to this lingering change
record in the zks.outstandingChanges, it prevents any deletion to
/mymetadata/resources since the ChangeRecord still have the count for the
ephemeral nodes.
Hence, the solution is to save pending changes only if there is an actual
outstanding change for that path. (zxid != -1)
> Unable to delete a node when the node has no children
> -----------------------------------------------------
>
> Key: ZOOKEEPER-2052
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2052
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
> Affects Versions: 3.4.6
> Environment: Red Hat Enterprise Linux 6.1 x86_64, standalone or 3
> node ensemble (v3.4.6), 2 Java clients (v3.4.6)
> Reporter: Yip Ng
> Attachments: ZOOKEEPER-2052.patch, ZOOKEEPER-2052.patch, zookeeper.log
>
>
> We stumbled upon a ZooKeeper bug where a node with no children cannot be
> removed on our 3 node ZooKeeper ensemble or standalone ZooKeeper on Red Hat
> Enterprise Linux x86_64 environment. Here is an example scenario/setup:
> o Standalone ZooKeeper or 3 node ensemble (v3.4.6)
> o 2 Java clients (v3.4.6)
> - Client A creates a persistent node (e.g.: /metadata/resources)
> - Client B creates ephemeral nodes under this persistent node
> o Client A attempts to remove the /metadata/resources node via multi op
> delete but fails since there are children
> o Client B's session expired, all the ephemeral nodes are removed
> o Client A attempts to recursively remove /metadata/resources node via
> multi op, this is expected to succeed but got the following exception:
> org.apache.zookeeper.KeeperException$NotEmptyException:
> KeeperErrorCode = Directory not empty
> (Note that Client B is the only client that creates these ephemeral nodes)
> o After this, we use zkCli.sh to inspect the problematic node but the
> zkCli.sh shows the /metadata/resources node indeed have no children but it
> will not allow /metadata/resources node to get deleted. (shown below)
> [zk: localhost:2181(CONNECTED) 0] ls /
> [zookeeper, metadata]
> [zk: localhost:2181(CONNECTED) 1] ls /metadata
> [resources]
> [zk: localhost:2181(CONNECTED) 2] get /metadata/resources
> null
> cZxid = 0x3
> ctime = Wed Oct 01 22:04:11 PDT 2014
> mZxid = 0x3
> mtime = Wed Oct 01 22:04:11 PDT 2014
> pZxid = 0x9
> cversion = 2
> dataVersion = 0
> aclVersion = 0
> ephemeralOwner = 0x0
> dataLength = 0
> numChildren = 0
> [zk: localhost:2181(CONNECTED) 3] delete /metadata/resources
> Node not empty: /metadata/resources
> [zk: localhost:2181(CONNECTED) 4] get /metadata/resources
> null
> cZxid = 0x3
> ctime = Wed Oct 01 22:04:11 PDT 2014
> mZxid = 0x3
> mtime = Wed Oct 01 22:04:11 PDT 2014
> pZxid = 0x9
> cversion = 2
> dataVersion = 0
> aclVersion = 0
> ephemeralOwner = 0x0
> dataLength = 0
> numChildren = 0
> o The only ways to remove this node is to either:
> a) Restart the ZooKeeper server
> b) set data to /metadata/resources then followed by a subsequent delete.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
