Devesh Kumar Singh created HDDS-16320:
-----------------------------------------
Summary: Recon incorrectly marks healthy EC containers as
REPLICA_MISMATCH by comparing checksums across different EC replica indexes
Key: HDDS-16320
URL: https://issues.apache.org/jira/browse/HDDS-16320
Project: Apache Ozone
Issue Type: Bug
Components: Ozone Recon
Affects Versions: 2.2.1
Reporter: Devesh Kumar Singh
Assignee: Devesh Kumar Singh
Recon incorrectly classifies healthy erasure-coded containers as
REPLICA_MISMATCH.
For an EC container, each replica index contains a different data or parity
fragment. Consequently, checksums belonging to different EC replica indexes are
not expected to be identical.
Recon currently compares dataChecksum across all replicas of a container
without considering the EC replica index. Therefore, a healthy EC container
with all required indexes can be reported as REPLICA_MISMATCH.
*### Steps to reproduce*
1. Start an Ozone docker cluster with at least five healthy datanodes and
Recon.
2. Create a volume and an EC bucket:
{code:java}
ozone sh volume create /recon-ec-vol
ozone sh bucket create /recon-ec-vol/recon-ec-bucket \
--type EC \
--replication rs-3-2-1024k
{code}
3. Write a key:
{code:java}
ozone freon ockg \
--volume recon-ec-vol \
--bucket recon-ec-bucket \
--number-of-tests 1 \
--size 1MB \
--threads 1 \
--prefix reconec \
--type EC \
--replication rs-3-2-1024k
{code}
4. Obtain the container ID:
{code:java}
KEY_PATH=/recon-ec-vol/recon-ec-bucket/reconec/0
CID=$(ozone sh key info "$KEY_PATH" |
jq -er '.ozoneKeyLocations[0].containerID')
echo "$CID"
{code}
5. Close the container so that its checksum information is available:
{code:java}
ozone admin container close "$CID"
{code}
6. Wait until the container has five replicas, and inspect their indexes and
checksums:
{code:java}
ozone admin container info "$CID" --json |
jq -r '
.replicas[] |
[
.replicaIndex,
.datanodeDetails.hostName,
.state,
.dataChecksum
] | @tsv'
{code}
Verify that:
- Replica indexes 1-5 are present.
- The indexes are hosted on five distinct datanodes.
- The replicas are healthy.
- Different indexes have different dataChecksum values.
7. Verify that the key remains readable:
{code:java}
ozone sh key get --force "$KEY_PATH" /tmp/recon-ec-download
test "$(stat -c %s /tmp/recon-ec-download)" -eq 1048576
{code}
8. Wait for the Recon container-health scan and query REPLICA_MISMATCH:
{code:java}
curl -s \
'http://localhost:9888/api/v1/containers/unhealthy/REPLICA_MISMATCH' |
jq --argjson cid "$CID" \
'[.. | objects | select(.containerID? == $cid)]'
{code}
* ### Actual result*
Recon reports the healthy EC container as:
REPLICA_MISMATCH
Data checksum mismatch across replicas
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
