[
https://issues.apache.org/jira/browse/HDDS-15077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDDS-15077:
-----------------------------------
Description:
HDDS-13022 changed the way that exclusiveSize is tracked internally. Here's the
explanation. We should add them into the code and maybe System Internals doc.
h1. exclusiveSize
Set by: KeyDeletingService (key cleanup background service)
Meaning: The amount of data (in bytes, unreplicated) that is exclusively
referenced by this snapshot, as determined during key-level deep cleaning —
i.e., when deleted keys are processed and it's determined that only this
snapshot holds a reference to certain data blocks.
h1. exclusiveReplicatedSize
Set by: KeyDeletingService
Meaning: Same as exclusiveSize, but accounts for replication factor. For
example, if a block is 1 MB stored with RATIS/3 replication, exclusiveSize = 1
MB but exclusiveReplicatedSize = 3 MB.
h1. exclusiveSizeDeltaFromDirDeepCleaning
Set by: SnapshotDirectoryCleaningService (directory cleanup background service)
Meaning: The additional exclusive size (unreplicated) discovered during
directory-level deep cleaning — i.e., when deleted directories are recursively
processed and their contents are found to be exclusively held by this snapshot.
This is kept separate from exclusiveSize to avoid the two services overwriting
each other.
h1. exclusiveReplicatedSizeDeltaFromDirDeepCleaning
Set by: SnapshotDirectoryCleaningService
Meaning: Same as exclusiveSizeDeltaFromDirDeepCleaning, but scaled by the
replication factor.
h1. Why Are They Split?
Before this PR, both services wrote to the same exclusiveSize field. Since they
run independently and asynchronously, one service could overwrite the other's
result, causing the exclusive size to be understated (a correctness bug).
By using separate fields, each service owns its own variable. The true total
exclusive size is then computed as:
{code}
totalExclusiveSize = exclusiveSize + exclusiveSizeDeltaFromDirDeepCleaning
totalExclusiveReplicatedSize = exclusiveReplicatedSize +
exclusiveReplicatedSizeDeltaFromDirDeepCleaning
{code}
was:
HDDS-13022 changed the way that exclusiveSize is tracked internally. Here's the
explanation. We should add them into the code and maybe System Internals doc.
h1. exclusiveSize
Set by: KeyDeletingService (key cleanup background service)
Meaning: The amount of data (in bytes, unreplicated) that is exclusively
referenced by this snapshot, as determined during key-level deep cleaning —
i.e., when deleted keys are processed and it's determined that only this
snapshot holds a reference to certain data blocks.
h1. exclusiveReplicatedSize
Set by: KeyDeletingService
Meaning: Same as exclusiveSize, but accounts for replication factor. For
example, if a block is 1 MB stored with RATIS/3 replication, exclusiveSize = 1
MB but exclusiveReplicatedSize = 3 MB.
h1. exclusiveSizeDeltaFromDirDeepCleaning
Set by: SnapshotDirectoryCleaningService (directory cleanup background service)
Meaning: The additional exclusive size (unreplicated) discovered during
directory-level deep cleaning — i.e., when deleted directories are recursively
processed and their contents are found to be exclusively held by this snapshot.
This is kept separate from exclusiveSize to avoid the two services overwriting
each other.
h1. exclusiveReplicatedSizeDeltaFromDirDeepCleaning
Set by: SnapshotDirectoryCleaningService
Meaning: Same as exclusiveSizeDeltaFromDirDeepCleaning, but scaled by the
replication factor.
> Explain "exclusiveSize" and "exclusiveSizeDeltaFromDirDeepCleaning"
> -------------------------------------------------------------------
>
> Key: HDDS-15077
> URL: https://issues.apache.org/jira/browse/HDDS-15077
> Project: Apache Ozone
> Issue Type: Task
> Reporter: Wei-Chiu Chuang
> Priority: Major
>
> HDDS-13022 changed the way that exclusiveSize is tracked internally. Here's
> the explanation. We should add them into the code and maybe System Internals
> doc.
> h1. exclusiveSize
> Set by: KeyDeletingService (key cleanup background service)
> Meaning: The amount of data (in bytes, unreplicated) that is exclusively
> referenced by this snapshot, as determined during key-level deep cleaning —
> i.e., when deleted keys are processed and it's determined that only this
> snapshot holds a reference to certain data blocks.
> h1. exclusiveReplicatedSize
> Set by: KeyDeletingService
> Meaning: Same as exclusiveSize, but accounts for replication factor. For
> example, if a block is 1 MB stored with RATIS/3 replication, exclusiveSize =
> 1 MB but exclusiveReplicatedSize = 3 MB.
> h1. exclusiveSizeDeltaFromDirDeepCleaning
> Set by: SnapshotDirectoryCleaningService (directory cleanup background
> service)
> Meaning: The additional exclusive size (unreplicated) discovered during
> directory-level deep cleaning — i.e., when deleted directories are
> recursively processed and their contents are found to be exclusively held by
> this snapshot. This is kept separate from exclusiveSize to avoid the two
> services overwriting each other.
> h1. exclusiveReplicatedSizeDeltaFromDirDeepCleaning
> Set by: SnapshotDirectoryCleaningService
> Meaning: Same as exclusiveSizeDeltaFromDirDeepCleaning, but scaled by the
> replication factor.
>
>
> h1. Why Are They Split?
> Before this PR, both services wrote to the same exclusiveSize field. Since
> they run independently and asynchronously, one service could overwrite the
> other's result, causing the exclusive size to be understated (a correctness
> bug).
> By using separate fields, each service owns its own variable. The true total
> exclusive size is then computed as:
> {code}
> totalExclusiveSize = exclusiveSize + exclusiveSizeDeltaFromDirDeepCleaning
> totalExclusiveReplicatedSize = exclusiveReplicatedSize +
> exclusiveReplicatedSizeDeltaFromDirDeepCleaning
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
