Salvatore Casolaro created ZOOKEEPER-5090:
---------------------------------------------
Summary: SnapshotComparer includes ephemeral znodes in comparison
output
Key: ZOOKEEPER-5090
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-5090
Project: ZooKeeper
Issue Type: Bug
Components: server
Affects Versions: 3.10.0
Environment: Operating system: Windows 11, amd64
Java version: 17.0.12
ZooKeeper version: 3.10.0-SNAPSHOT
Branch: master
Commit: 9102d0be12e66a5c592d8cbf0c6865f68ffa9252
Reporter: Salvatore Casolaro
h2. Description
SnapshotComparer includes ephemeral znodes in its comparison output, although
its class documentation states that it only outputs information about permanent
nodes and ignores both sessions and ephemeral nodes.
The issue was reproduced on the Apache ZooKeeper master branch at commit
9102d0be12e66a5c592d8cbf0c6865f68ffa9252.
h2. Expected behavior
Znodes whose ephemeralOwner is non-zero should be excluded from the comparison
data used by SnapshotComparer. Consequently, their paths and payload sizes
should not appear in the reported differences or affect aggregate deltas.
The official ZooKeeper 3.9.5 documentation states:
{quote}This tool only outputs information about permanent nodes, ignoring both
sessions and ephemeral nodes.
{quote}
Documentation:
[ZooKeeper 3.9.5
documentation|https://zookeeper.apache.org/doc/r3.9.5/zookeeperTools.html#zkSnapshotComparer]
h2. Actual behavior
When two valid snapshots containing different ephemeral znodes are compared,
SnapshotComparer reports the ephemeral paths as nodes found only in the left or
right tree.
The payload sizes of the ephemeral znodes are also included in the aggregate
size calculations and affect the delta reported for the root node.
Relevant output:
{code:java}
Node found in both trees. Delta: -2 bytes, 0 descendants
Node /effimero_sinistro found only in left tree. Descendant size: 13.
Descendant count: 0
Node /ephemeral_test found only in right tree. Descendant size: 11. Descendant
count: 0
All layers compared.
{code}
h2. Steps to reproduce
# Create a valid snapshot containing an ephemeral znode named
/effimero_sinistro with a 13-byte payload.
# Create another valid snapshot containing an ephemeral znode named
/ephemeral_test with an 11-byte payload.
# Verify with SnapshotFormatter that both nodes have a non-zero ephemeralOwner.
# Compare the snapshots using SnapshotComparer with both thresholds set to zero:
{code:java}
-l left_ephemeral.snap -r right_ephemeral.snap -b 0 -n 0
{code}
# Observe that SnapshotComparer reports both ephemeral paths and includes their
payload sizes in the aggregate root delta.
h2. Snapshot evidence
The dump of the left snapshot confirms:
{code:java}
/effimero_sinistro
ephemeralOwner = 0x10000101fa30000
dataLength = 13
Session:
0x10000101fa30000, timeout = 30000, ephemeralCount = 1
{code}
The dump of the right snapshot confirms:
{code:java}
/ephemeral_test
ephemeralOwner = 0x10000101fa30001
dataLength = 11
Session:
0x10000101fa30001, timeout = 30000, ephemeralCount = 1
{code}
Both ephemeralOwner values are non-zero and correspond to sessions stored in
the respective snapshots.
h2. Reproducibility
The black-box regression test was first executed three times and failed
consistently.
The issue was subsequently reproduced on a verification branch based on
upstream/master at the time of testing:
{code:java}
ZooKeeper version: 3.10.0-SNAPSHOT
Branch: master
Commit: 9102d0be12e66a5c592d8cbf0c6865f68ffa9252
{code}
SnapshotComparer completes the comparison successfully, but the regression test
fails because the output contains the ephemeral znode path:
{code:java}
java.lang.AssertionError:
Output must not contain: /effimero_sinistro
{code}
h2. Possible cause
The issue appears to originate in TreeInfo.TreeNode.populateChildren().
The method retrieves each DataNode and adds it to the internal TreeInfo without
checking whether the node has a non-zero ephemeralOwner:
{code:java}
DataNode childNode = dataTree.getNode(childPath);
long size;
synchronized (childNode) {
size = childNode.data == null ? 0 : childNode.data.length;
}
TreeNode childTreeNode = new TreeNode(childPath, size);
childTreeNode.populateChildren(childPath, dataTree, treeInfo, currentDepth + 1);
children.add(childTreeNode);
{code}
As a result, ephemeral znodes are included in the internal comparison tree.
Their paths can be reported directly, and their payload sizes can propagate to
ancestor aggregates.
In this reproduction, the 13-byte left ephemeral payload and the 11-byte right
ephemeral payload produce the observed -2 byte delta for the root node.
h2. Regression test
A black-box JUnit 4 test invokes SnapshotComparer.main(String[]) with the two
snapshot files. It verifies that the comparison completes successfully but does
not include either ephemeral path in its output.
The test is currently marked with @Ignore in the complete experimental test
suite because it exposes the reported defect. The @Ignore annotation was
temporarily removed when reproducing the failure.
The relevant assertions are:
{code:java}
assertCompletedSuccessfully(result);
assertOutputDoesNotContain(result, LEFT_EPHEMERAL_PATH);
assertOutputDoesNotContain(result, RIGHT_EPHEMERAL_PATH);
{code}
The minimal snapshot files, sanitized snapshot dumps, and the complete
regression test can be provided if needed.
h2. Related issue
SnapshotComparer was originally introduced in ZOOKEEPER-3427.
I searched Jira and the Apache ZooKeeper GitHub repository for existing issues
and pull requests concerning SnapshotComparer, ephemeral nodes, ephemeralOwner,
permanent-node filtering, and session filtering, but I did not find an existing
report covering this behavior.
h2. Academic context
This issue was identified during an academic software testing activity
supervised by Prof. Guglielmo De Angelis.
GitHub profile:
[Prof. Guglielmo De Angelis on GitHub|https://github.com/gulyx]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
