[
https://issues.apache.org/jira/browse/HDDS-17622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17622:
------------------------------
Affects Version/s: 2.2.1
2.0.0
1.4.0
2.1.2
2.1.0
1.4.1
2.1.1
2.2.2
1.3.0
2.2.0
> RackScatter placement fails although a rack has a writable datanode, because
> a new rack gets only one round of random picks
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17622
> URL: https://issues.apache.org/jira/browse/HDDS-17622
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 1.3.0, 1.4.0, 2.0.0, 1.4.1, 2.1.0, 2.2.0, 2.1.1, 2.1.2,
> 2.2.1, 2.2.2
> Reporter: Siyao Meng
> Priority: Minor
> Attachments:
> CR-4-retry-skipped-rack-in-rackscatter-unique-rack-phase.patch,
> TestBugCR4RackScatterSingleRoundOfPicks.java
>
>
> h3. Mechanism
> {{SCMContainerPlacementRackScatter.chooseDatanodesInternal}} first picks one
> node on each rack that holds no replica yet ("unique racks"), then fills the
> rest with a same-rack fallback. For each rack, {{chooseNode}} draws up to 5
> random nodes ({{INNER_LOOP_MAX_RETRY}}) and excludes each one that
> {{isValidNode}} rejects. {{chooseNodesFromRacks}} can retry a rack whose
> picks all missed in a later round ({{skippedRacks}}, ending after 4 rounds
> without progress, {{OUTER_LOOP_MAX_RETRY}} + 1), but the unique-rack phase
> calls it with {{maxOuterLoopIterations = maxReplicasPerRack}}. For EC on at
> least as many racks as replicas that is 1, so a rack gets a single round of 5
> picks. If those miss, the method throws {{FAILED_TO_FIND_HEALTHY_NODES}}
> ("Chosen nodes size from Unique Racks: 0, but required nodes to choose from
> Unique Racks: 1 do not match"), and the same-rack fallback is not reached.
> Full, STALE, decommissioning and maintenance datanodes stay in the network
> topology and are not in the excluded list, so on a rack where most datanodes
> cannot take a replica, 5 picks often miss the ones that can.
> h3. Trigger
> An EC container needs a target on a new rack, most datanodes of that rack are
> full or not writable, and it is the only rack without a replica, for example
> a cluster with exactly as many racks as the EC width (rs-3-2 on 5 racks).
> With more candidate racks, the first round moves on to the next rack, and the
> placement fails only if every candidate rack misses. For example rs-3-2 with
> index 1 to 4 on four racks and index 5 missing, and a fifth rack with 10
> datanodes of which one is writable: 5 picks without replacement miss it with
> probability 0.5. With 20 unusable datanodes and one writable it is 16/21
> (about 0.76).
> h3. Impact
> The placement fails although a valid target exists.
> {{ReplicationManagerUtil.getTargetDatanodes}} turns it into
> {{FAILED_TO_FIND_SUITABLE_NODE}}, {{ECUnderReplicationHandler}} rethrows, and
> the container is requeued for the next pass, which draws again. Repair still
> completes, but with failure probability p it takes p/(1-p) extra passes on
> average (the under-replicated queue is processed every 30 s by default): one
> more pass in the first layout above and about three in the second, while the
> container runs with reduced redundancy. Each failed pass also logs an ERROR
> with a stack trace, raises the container's requeue count (which sorts it
> behind others with the same remaining redundancy), and runs the handler's
> remedies for a missing target, such as deleting an UNHEALTHY replica if the
> container has one. The same applies to every caller of the EC policy, for
> example copies for decommission and mis-replication, and EC pipeline creation
> on the write path ({{ECPipelineProvider}} calls the same {{chooseDatanodes}}
> with no used nodes), where {{WritableECContainerProvider}} hides most
> failures by falling back to existing open pipelines and a second allocation
> attempt. Not measured end to end; only the per placement failure rate was run.
> HDDS-16634 reports the same kind of probe limit in
> {{SCMContainerPlacementRackAware}}, measured as a slower decommission.
> h3. Reproduction
> [^TestBugCR4RackScatterSingleRoundOfPicks.java] ({{hadoop-hdds/server-scm}},
> target path and Maven command in the class comment) builds the example above
> with a real {{NetworkTopologyImpl}} and the real policy (only {{NodeManager}}
> is a mock that returns the datanodes): rack4 has 5 full, 4 STALE and 1
> writable datanode, and rack0 to rack3 each have the used node and 2 free
> writable nodes. It runs 200 placements for one target through
> {{ReplicationManagerUtil.getTargetDatanodes}}. On unmodified source about
> half failed (102 of 200 in one run) with {{FAILED_TO_FIND_SUITABLE_NODE}},
> the other 98 chose the writable datanode of rack4, and none used the free
> nodes of rack0 to rack3. The test passes when the defect is present. With the
> patch all 200 chose the writable datanode and the test fails at the assertion
> on the failure count.
> h3. Patch
> [^CR-4-retry-skipped-rack-in-rackscatter-unique-rack-phase.patch], against
> e4ae41fc4047c99af983020b4f33bd297c9d0ac5. It also applies to master at
> 3fc2387040f.
> The unique-rack phase no longer limits {{chooseNodesFromRacks}} to
> {{maxReplicasPerRack}} rounds (the same-rack fallback already passed
> {{Integer.MAX_VALUE}}), so the existing skipped-rack retry runs: a rack whose
> picks all missed is tried again in the next round, with the rejected nodes
> still excluded, until 4 rounds bring no progress. The round limit parameter,
> now unused, is removed. The per rack limit is unchanged, so no rack gets more
> than {{maxReplicasPerRack}} nodes, and the final
> {{validateContainerPlacement}} check still applies. A placement that
> succeeded at the old limit is unchanged; only placements that used to throw
> get more picks. A rack with no usable node (as in HDDS-17362) still ends in
> the same exception, after 20 picks instead of 5, so such a failing placement
> now logs up to 4 times as many INFO lines from {{isValidNode}} and
> {{chooseNode}} on each pass. This is bounded and rack scoped, and the
> same-rack fallback already behaves this way.
> This narrows the failure, it does not remove it: a rack is now searched up to
> 20 picks instead of 5. Picks are made without replacement, so a rack of up to
> 20 datanodes is always searched completely, and the remaining failure needs a
> larger rack that is almost all unusable. With 20 unusable datanodes and one
> writable, 91 of 2000 placements still failed (1/21 expected), against about
> 76% before. Choosing only among datanodes that can take a replica would
> remove it; that is the direction of HDDS-16634 and out of scope here. Falling
> through to the same-rack fallback instead of throwing, as HDDS-16629 did for
> {{SCMContainerPlacementRackAware}} when no other rack has a writable node,
> was not taken: here another rack does have one, and the fallback would put
> the replica on a rack that already holds one, leaving the container
> mis-replicated.
> Covered by {{chooseOnlyWritableNodeOfMostlyUnusableRack}} in the existing
> {{TestSCMContainerPlacementRackScatter}}: one writable datanode among 10 on
> the only new rack must be chosen in 20 of 20 placements, and when it is made
> STALE too the placement still fails with {{FAILED_TO_FIND_HEALTHY_NODES}}.
> Without the change it fails with the "Unique Racks: 0" exception above. With
> the change, the placement policy, pipeline placement and Replication Manager
> handler suites in {{hadoop-hdds/server-scm}} pass (651 tests) and checkstyle
> is clean.
> Found by code review of the SCM EC container replication and placement repair
> paths, as part of the TLA+ verification effort under HDDS-15926, on commit
> e4ae41fc4047c99af983020b4f33bd297c9d0ac5. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. Related but not
> duplicates: HDDS-16634 (the probe limit in
> {{SCMContainerPlacementRackAware}}), HDDS-17362 (RackScatter counts a rack of
> out-of-service datanodes as available; the patch does not change that case
> beyond the extra picks described above). The attached patch is a proposal for
> review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
