[
https://issues.apache.org/jira/browse/HDDS-16780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16780:
------------------------------
Description:
{code}
[ERROR] Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 82.46 s
<<< FAILURE! -- in
org.apache.hadoop.ozone.container.replication.TestPerVolumePushReplication
[ERROR]
org.apache.hadoop.ozone.container.replication.TestPerVolumePushReplication.testDecommissionWithPerVolumePools
-- Time elapsed: 31.25 s <<< ERROR!
java.util.concurrent.TimeoutException:
Timed out waiting for condition.
at
org.apache.ozone.test.GenericTestUtils.waitFor(GenericTestUtils.java:137)
at
org.apache.hadoop.ozone.container.replication.TestPerVolumePushReplication.testDecommissionWithPerVolumePools(TestPerVolumePushReplication.java:271)
{code}
SCM log during the decommission (197 occurrences):
{code}
ERROR replication.UnhealthyReplicationProcessor
(UnhealthyReplicationProcessor.java:processAll(125)) - Error processing Health
result of class: class
org.apache.hadoop.hdds.scm.container.replication.ContainerHealthResult$UnderReplicatedHealthResult
...
java.lang.IllegalArgumentException: Capacity cannot be negative.
at
org.apache.hadoop.hdds.scm.container.placement.metrics.SCMNodeStat.<init>(SCMNodeStat.java:45)
at
org.apache.hadoop.hdds.scm.node.SCMNodeManager.getNodeStatInternal(SCMNodeManager.java:1172)
at
org.apache.hadoop.hdds.scm.node.SCMNodeManager.getNodeStat(SCMNodeManager.java:1147)
at
org.apache.hadoop.hdds.scm.container.replication.ReplicationManagerUtil.excludeFullNodes(ReplicationManagerUtil.java:245)
at
org.apache.hadoop.hdds.scm.container.replication.ReplicationManagerUtil.getExcludedAndUsedNodes(ReplicationManagerUtil.java:223)
at
org.apache.hadoop.hdds.scm.container.replication.ECUnderReplicationHandler.processAndSendCommands(ECUnderReplicationHandler.java:130)
{code}
h3. Root cause
The integration test {{ozone-site.xml}} sets
{{hdds.datanode.du.factory.classname}} to {{MockSpaceUsageCheckFactory$None}},
which reports {{Long.MAX_VALUE}} capacity for every volume.
{{TestPerVolumePushReplication}} starts datanodes with 2 data volumes, so
{{SCMNodeManager#getNodeStatInternal}} overflows when it sums the storage
reports of a node, and {{SCMNodeStat}} throws {{Capacity cannot be negative}}.
{{ReplicationManagerUtil#excludeFullNodes}} calls {{getNodeStat}} for every
datanode with a pending replica ADD. So while any replication is in flight,
processing of every other under replicated container fails, and the
decommission effectively replicates one container per ReplicationManager cycle
(about 3s). The decommissioning node held 12 containers; it had reached 9 of
them when the 30s {{waitForDnToReachOpState}} wait expired.
h3. Fix
The same overflow affects any mini cluster test with more than one volume per
datanode, and {{SCMNodeManager#getStats}}, which sums over all datanodes. Lower
{{MockSpaceUsageSource.unlimited()}} to {{Long.MAX_VALUE / 1024}} (about 8 PB
per volume), which leaves room to sum 1024 volumes without overflow.
- https://github.com/apache/ozone/actions/runs/37686266652/job/113019002301
was:
{code}
[ERROR] Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 82.46 s
<<< FAILURE! -- in
org.apache.hadoop.ozone.container.replication.TestPerVolumePushReplication
[ERROR]
org.apache.hadoop.ozone.container.replication.TestPerVolumePushReplication.testDecommissionWithPerVolumePools
-- Time elapsed: 31.25 s <<< ERROR!
java.util.concurrent.TimeoutException:
Timed out waiting for condition.
at
org.apache.ozone.test.GenericTestUtils.waitFor(GenericTestUtils.java:137)
at
org.apache.hadoop.ozone.container.replication.TestPerVolumePushReplication.testDecommissionWithPerVolumePools(TestPerVolumePushReplication.java:271)
{code}
SCM log during the decommission (197 occurrences):
{code}
ERROR replication.UnhealthyReplicationProcessor
(UnhealthyReplicationProcessor.java:processAll(125)) - Error processing Health
result of class: class
org.apache.hadoop.hdds.scm.container.replication.ContainerHealthResult$UnderReplicatedHealthResult
...
java.lang.IllegalArgumentException: Capacity cannot be negative.
at
org.apache.hadoop.hdds.scm.container.placement.metrics.SCMNodeStat.<init>(SCMNodeStat.java:45)
at
org.apache.hadoop.hdds.scm.node.SCMNodeManager.getNodeStatInternal(SCMNodeManager.java:1172)
at
org.apache.hadoop.hdds.scm.node.SCMNodeManager.getNodeStat(SCMNodeManager.java:1147)
at
org.apache.hadoop.hdds.scm.container.replication.ReplicationManagerUtil.excludeFullNodes(ReplicationManagerUtil.java:245)
at
org.apache.hadoop.hdds.scm.container.replication.ReplicationManagerUtil.getExcludedAndUsedNodes(ReplicationManagerUtil.java:223)
at
org.apache.hadoop.hdds.scm.container.replication.ECUnderReplicationHandler.processAndSendCommands(ECUnderReplicationHandler.java:130)
{code}
h3. Root cause
The integration test {{ozone-site.xml}} sets
{{hdds.datanode.du.factory.classname}} to {{MockSpaceUsageCheckFactory$None}},
which reports {{Long.MAX_VALUE}} capacity for every volume.
{{TestPerVolumePushReplication}} starts datanodes with 2 data volumes, so
{{SCMNodeManager#getNodeStatInternal}} overflows when it sums the storage
reports of a node, and {{SCMNodeStat}} throws {{Capacity cannot be negative}}.
{{ReplicationManagerUtil#excludeFullNodes}} calls {{getNodeStat}} for every
datanode with a pending replica ADD. So while any replication is in flight,
processing of every other under replicated container fails, and the
decommission effectively replicates one container per ReplicationManager cycle
(about 3s). The decommissioning node held 12 containers; it had reached 9 of
them when the 30s {{waitForDnToReachOpState}} wait expired.
h3. Fix
Use {{MockSpaceUsageCheckFactory$HalfTera}} (fixed 512 GB per volume) in this
test, so the per node sum stays in range.
- https://github.com/apache/ozone/actions/runs/37686266652/job/113019002301
> Intermittent failure in
> TestPerVolumePushReplication#testDecommissionWithPerVolumePools
> ---------------------------------------------------------------------------------------
>
> Key: HDDS-16780
> URL: https://issues.apache.org/jira/browse/HDDS-16780
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: SCM, test
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Labels: pull-request-available
>
> {code}
> [ERROR] Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 82.46
> s <<< FAILURE! -- in
> org.apache.hadoop.ozone.container.replication.TestPerVolumePushReplication
> [ERROR]
> org.apache.hadoop.ozone.container.replication.TestPerVolumePushReplication.testDecommissionWithPerVolumePools
> -- Time elapsed: 31.25 s <<< ERROR!
> java.util.concurrent.TimeoutException:
> Timed out waiting for condition.
> at
> org.apache.ozone.test.GenericTestUtils.waitFor(GenericTestUtils.java:137)
> at
> org.apache.hadoop.ozone.container.replication.TestPerVolumePushReplication.testDecommissionWithPerVolumePools(TestPerVolumePushReplication.java:271)
> {code}
> SCM log during the decommission (197 occurrences):
> {code}
> ERROR replication.UnhealthyReplicationProcessor
> (UnhealthyReplicationProcessor.java:processAll(125)) - Error processing
> Health result of class: class
> org.apache.hadoop.hdds.scm.container.replication.ContainerHealthResult$UnderReplicatedHealthResult
> ...
> java.lang.IllegalArgumentException: Capacity cannot be negative.
> at
> org.apache.hadoop.hdds.scm.container.placement.metrics.SCMNodeStat.<init>(SCMNodeStat.java:45)
> at
> org.apache.hadoop.hdds.scm.node.SCMNodeManager.getNodeStatInternal(SCMNodeManager.java:1172)
> at
> org.apache.hadoop.hdds.scm.node.SCMNodeManager.getNodeStat(SCMNodeManager.java:1147)
> at
> org.apache.hadoop.hdds.scm.container.replication.ReplicationManagerUtil.excludeFullNodes(ReplicationManagerUtil.java:245)
> at
> org.apache.hadoop.hdds.scm.container.replication.ReplicationManagerUtil.getExcludedAndUsedNodes(ReplicationManagerUtil.java:223)
> at
> org.apache.hadoop.hdds.scm.container.replication.ECUnderReplicationHandler.processAndSendCommands(ECUnderReplicationHandler.java:130)
> {code}
> h3. Root cause
> The integration test {{ozone-site.xml}} sets
> {{hdds.datanode.du.factory.classname}} to
> {{MockSpaceUsageCheckFactory$None}}, which reports {{Long.MAX_VALUE}}
> capacity for every volume. {{TestPerVolumePushReplication}} starts datanodes
> with 2 data volumes, so {{SCMNodeManager#getNodeStatInternal}} overflows when
> it sums the storage reports of a node, and {{SCMNodeStat}} throws {{Capacity
> cannot be negative}}.
> {{ReplicationManagerUtil#excludeFullNodes}} calls {{getNodeStat}} for every
> datanode with a pending replica ADD. So while any replication is in flight,
> processing of every other under replicated container fails, and the
> decommission effectively replicates one container per ReplicationManager
> cycle (about 3s). The decommissioning node held 12 containers; it had reached
> 9 of them when the 30s {{waitForDnToReachOpState}} wait expired.
> h3. Fix
> The same overflow affects any mini cluster test with more than one volume per
> datanode, and {{SCMNodeManager#getStats}}, which sums over all datanodes.
> Lower {{MockSpaceUsageSource.unlimited()}} to {{Long.MAX_VALUE / 1024}}
> (about 8 PB per volume), which leaves room to sum 1024 volumes without
> overflow.
> - https://github.com/apache/ozone/actions/runs/37686266652/job/113019002301
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
