[
https://issues.apache.org/jira/browse/HDDS-11267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arafat Khan updated HDDS-11267:
-------------------------------
Description:
The issue involves datanodes in Ozone reporting negative container sizes for
the {{usedBytes}} and block count metrics. This occurs when the Ozone Manager
sends duplicate block deletion requests to the Storage Container Manager. Due
to a delay in processing the original request, OM may mistakenly send a
duplicate request. The datanode, upon receiving the duplicate request, attempts
to delete blocks that have already been deleted, but still updates the metrics,
leading to negative values. The proposed solution is to modify the deletion
process in the datanode to track and ignore duplicate block deletion requests,
ensuring metrics are not updated incorrectly.
Recon Reported the following negative sized containers:-
{
"state" : "CLOSED",
"stateEnterTime" : "2024-06-25T16:38:50.672Z",
"replicationConfig" : {
"data" : 6,
"parity" : 3,
"ecChunkSize" : 1048576,
"codec" : "RS" },
"usedBytes" : -222298112,
"numberOfKeys" : -17,
"lastUsed" : "2024-06-27T18:04:32.845329Z",
"owner" : "ozone1",
"containerID" : 1023,
"deleteTransactionId" : 9846,
"sequenceId" : 0,
"deleted" : false,
"open" : false}
was:The issue involves datanodes in Ozone reporting negative container sizes
for the {{usedBytes}} and block count metrics. This occurs when the Ozone
Manager sends duplicate block deletion requests to the Storage Container
Manager. Due to a delay in processing the original request, OM may mistakenly
send a duplicate request. The datanode, upon receiving the duplicate request,
attempts to delete blocks that have already been deleted, but still updates the
metrics, leading to negative values. The proposed solution is to modify the
deletion process in the datanode to track and ignore duplicate block deletion
requests, ensuring metrics are not updated incorrectly.
> Ozone Datanode Reporting Negative Container values for UsedBytes and
> BlockCount parameters
> ------------------------------------------------------------------------------------------
>
> Key: HDDS-11267
> URL: https://issues.apache.org/jira/browse/HDDS-11267
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Datanode, Ozone Recon
> Reporter: Arafat Khan
> Assignee: Arafat Khan
> Priority: Major
>
> The issue involves datanodes in Ozone reporting negative container sizes for
> the {{usedBytes}} and block count metrics. This occurs when the Ozone Manager
> sends duplicate block deletion requests to the Storage Container Manager. Due
> to a delay in processing the original request, OM may mistakenly send a
> duplicate request. The datanode, upon receiving the duplicate request,
> attempts to delete blocks that have already been deleted, but still updates
> the metrics, leading to negative values. The proposed solution is to modify
> the deletion process in the datanode to track and ignore duplicate block
> deletion requests, ensuring metrics are not updated incorrectly.
> Recon Reported the following negative sized containers:-
> {
> "state" : "CLOSED",
> "stateEnterTime" : "2024-06-25T16:38:50.672Z",
> "replicationConfig" : {
> "data" : 6,
> "parity" : 3,
> "ecChunkSize" : 1048576,
> "codec" : "RS" },
> "usedBytes" : -222298112,
> "numberOfKeys" : -17,
> "lastUsed" : "2024-06-27T18:04:32.845329Z",
> "owner" : "ozone1",
> "containerID" : 1023,
> "deleteTransactionId" : 9846,
> "sequenceId" : 0,
> "deleted" : false,
> "open" : false}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@ozone.apache.org
For additional commands, e-mail: issues-h...@ozone.apache.org
