[
https://issues.apache.org/jira/browse/HDDS-17016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17016:
------------------------------
Attachment: MC-12-enforce-volume-failure-tolerance-on-every-check.patch
> Datanode with more failed volumes than tolerated keeps serving after restart,
> then stops on a later volume check that finds nothing wrong
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17016
> URL: https://issues.apache.org/jira/browse/HDDS-17016
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments:
> MC-12-enforce-volume-failure-tolerance-on-every-check.patch,
> TestBugMC12ToleranceAtStartup.java
>
>
> h3. Mechanism
> {{MutableVolumeSet.hasEnoughVolumes}} is only called from
> {{handleVolumeFailures}}, and {{checkAllVolumes}} calls that only when the
> check itself finds a new failure. Volumes that are failed directly through
> {{failVolume}} are never compared with
> {{hdds.datanode.failed.data.volumes.tolerated}} (or its metadata and DB
> variants) at that point. At startup these are volumes rejected by
> {{initializeVolumeSet}} (inconsistent or unloadable root), by
> {{VersionEndpointTask.checkVolumeSet}} and by {{ContainerReader}}; at runtime
> {{ScmHAFinalizeUpgradeActionDatanode}} does the same. The startup
> {{checkAllVolumes}} in {{OzoneContainer.start}} then finds no new failure, so
> the datanode starts and serves although it is over the limit.
> The callback of {{checkVolumeAsync}} calls {{handleVolumeFailures}} even when
> the check found nothing. The next on demand check of any healthy volume
> (started by an I/O error on a container) therefore finds the set over the
> limit and calls {{handleFatalVolumeFailures}}, stopping the datanode at an
> arbitrary later time.
> This dates from the introduction of the tolerance: the startup exception that
> HDDS-9022 replaced with {{handleFatalVolumeFailures}} was also only thrown
> from {{handleVolumeFailures}}.
> h3. Trigger
> A finite tolerance and a restart with more unusable volumes than tolerated,
> for example volumes whose {{VERSION}} file is missing or corrupt. With the
> default {{-1}} only one corner case is affected: {{ContainerReader}} at
> startup or {{ScmHAFinalizeUpgradeActionDatanode}} at runtime fails every data
> volume directly. The datanode then keeps running with no data volume, because
> no healthy volume is left to start an on demand check.
> h3. Impact
> * The datanode serves with more failed volumes than the operator allowed,
> until any later failure handling runs.
> * It is then stopped by a check that found nothing wrong, for example the on
> demand check of a healthy volume.
> * Shown at unit level with a real volume set and checker and a mocked
> {{DatanodeStateMachine}}; not exercised on a datanode registered with SCM.
> h3. Reproduction
> PASS on unmodified source. [^TestBugMC12ToleranceAtStartup.java]
> ({{hadoop-hdds/container-service}}, target path and Maven command in the
> class comment) sets the data volume tolerance to 1, formats three volumes
> through {{StorageVolumeUtil.checkVolume}}, removes the {{VERSION}} file of
> two, rebuilds the volume set with a real {{StorageVolumeChecker}} and runs
> {{checkAllVolumes()}} as {{OzoneContainer.start}} does. Output: {{healthy=1
> failed=2 tolerated=1}}, and the startup check makes no fatal call.
> {{StorageVolumeUtil.onFailure}} on the healthy volume then logs "Not enough
> volumes ... MaxVolumeFailuresTolerated: 1, ActiveVolumes: 1, FailedVolumes:
> 2" and calls {{handleFatalVolumeFailures}} while the checked volume is still
> healthy. A passing test means the defect is present.
> h3. Patch
> [^MC-12-enforce-volume-failure-tolerance-on-every-check.patch], against
> cce1ca28944886218132f01c0e271c1fd0700078. It also applies to master at
> 2536fe9cbc3.
> {{MutableVolumeSet.checkAllVolumes}} now also calls
> {{handleFatalVolumeFailures}} when the check found no new failure but
> {{hasEnoughVolumes()}} is false. {{OzoneContainer.start}} runs it on the
> data, metadata and DB volume sets after all startup paths that fail volumes
> directly, so an over limit datanode stops at startup, and the periodic check
> covers direct failures at runtime. Stopping at startup is the intended
> behaviour: HDDS-5219 threw "Don't have enough good volumes on startup", and
> {{TestDatanodeHddsVolumeFailureToleration.testDNCorrectlyHandlesVolumeFailureOnStartup}}
> expects a restart with two bad volumes and tolerance 1 to exit. Both only
> work when the startup check itself finds the failures; the patch applies the
> same rule to volumes that failed before that check. Calling
> {{handleVolumeFailures}} unconditionally was not chosen because it would also
> run the failed volume listener on every periodic check.
> Covered by {{testVolumesFailedOnLoadCountTowardsTolerance}} in the existing
> {{TestVolumeSetDiskChecks}}: three volumes, two of them failed by
> {{initializeVolumeSet}}, a checker that reports no failure. With tolerance 1
> it expects one fatal call and without the change fails with "Wanted but not
> invoked: datanodeStateMachine.handleFatalVolumeFailures()"; with tolerance 2
> and with the default {{-1}} it expects none and passes either way. With the
> change the volume, container, reader, upgrade, handler and state machine
> suites (309 tests) and {{TestDatanodeHddsVolumeFailureToleration}} pass, and
> checkstyle is clean. With the patch the reproduction fails at its check that
> startup made no fatal call.
> *Upgrade note.* A datanode with a finite
> {{hdds.datanode.failed.data.volumes.tolerated}} (or the metadata or DB
> variant) that has more unusable volumes than tolerated now exits at startup
> with "DatanodeStateMachine Shutdown due to too many bad volumes", instead of
> serving until a later volume check stops it. Such a datanode will not come
> back after the upgrade restart, so a rolling upgrade stalls at it. Before
> upgrading, compare {{NumFailedVolumes}} of
> {{VolumeHealthMetrics-DATA_VOLUME}} (and {{VolumeHealthMetrics-META_VOLUME}},
> {{VolumeHealthMetrics-DB_VOLUME}}) with the tolerance on each such datanode,
> then repair or remove the failed directories or raise the tolerance. With the
> default {{-1}} a datanode now also exits when every data volume has been
> failed directly, instead of running with none.
> Found by TLA+ model checking and code review of the datanode volume and
> per-disk RocksDB lifecycle paths under HDDS-15926, on commit
> cce1ca28944886218132f01c0e271c1fd0700078. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. Related but not
> duplicates: HDDS-5219 (introduced the tolerance), HDDS-9022 (replaced the
> startup exception with the fatal handler), HDDS-14871 (volume check latch
> timeout) and HDDS-11770 (proposed default 0, abandoned). The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
