[
https://issues.apache.org/jira/browse/HDDS-11036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18102640#comment-18102640
]
Chia-Chuan Ho commented on HDDS-11036:
--------------------------------------
Recon can lose derived events if it crashes after applying OM deltas locally
but before tasks process them.
Replaying old deltas may be unsafe because the WAL or historical values may no
longer be available. Would it be acceptable to recover by: # Rebuilding all
tasks from a consistent local OM DB checkpoint.
# Fetching a fresh OM snapshot first if the local DB is unusable.
Current master already provides /triggerdbsync/om/reinit via HDDS-15863.
Should HDDS-11036 extend it with fresh-snapshot recovery and stronger sync
coordination?
> Ozone Recon Data Recovery Tool
> ------------------------------
>
> Key: HDDS-11036
> URL: https://issues.apache.org/jira/browse/HDDS-11036
> Project: Apache Ozone
> Issue Type: Improvement
> Components: Ozone Recon
> Reporter: Devesh Kumar Singh
> Priority: Major
>
> Recon do periodic sync with OM to keep OM namespace metadata like keys,
> volumes, buckets up-to date and use it for various other tasks and derive
> data out of it using various tasks (ContainerKeyMapperTask, NSSummaryTask,
> OMDBInsightTask etc).
> If due to unexpected errors at OM or Recon , Recon may lag or OM namespace
> metadata may not be in sync either due to
> # RocksDB lock issues.
> # Recon has fetched OM DB snapshot since last sequence number, but before
> processing those incremental events (PUT, DELETE, UPDATE), if some unexpected
> error caused any processed or derived data failed to save in recon DB, Recon
> will not fetch that incremental data again and will not process those events
> again.
> In such cases, Recon should have a recovery tool to fetch the OM DB snapshot
> from last fetched sequence number and then explicitly process and run tasks
> on fetched events for Recon.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
