[
https://issues.apache.org/jira/browse/HDDS-15028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDDS-15028:
-----------------------------------
Description:
https://ozone.apache.org/docs/core-concepts/architecture/storage-container-manager/#how-to-exit-safe-mode
Typically SCM should exit safe mode by itself. We need a troubleshooting doc to
identify the problem that prevent automatic safe mode exit.
Symptoms:
Client application attempting to write files, but the request is rejected with
exception message "SafeModePrecheck failed for allocateBlock"
Get safe mode status:
'ozone admin safemode --verbose' command, or check leader SCM web UI. At the
bottom is the Safemode rules statuses, and the SCM stays in the safe mode if
any of the rules is not satisifed.
1. In a SCM-HA environment, only one SCM is up. In which case, start the other
two SCM.
2. insufficient number of Datanodes are up. Too many datanodes are not
available, either they are down, or they have not completed startup process.
For example, datanodes are still verifying containers and volumes.
3. Insufficient number of healthy containers. Containers may not be replicated
sufficiently, or some containers are corrupt. Run 'ozone admin container
report' command to check what happens. See Container Replication Report page at
https://ozone.apache.org/docs/administrator-guide/operations/container-replication-report
for details.
If admin is absolutely sure the cluster is healthy, manually exit the safe mode
using command 'ozone admin safemode --force-exit'
was:
https://ozone.apache.org/docs/core-concepts/architecture/storage-container-manager/#how-to-exit-safe-mode
Typically SCM should exit safe mode by itself. We need a troubleshooting doc to
identify the problem that prevent automatic safe mode exit.
Symptoms:
Client application attempting to write files, but the request is rejected with
exception message "SafeModePrecheck failed for allocateBlock"
Get safe mode status:
'ozone admin safemode --verbose' command, or check leader OM web UI.
1. In a SCM-HA environment, only one SCM is up. In which case, start the other
two SCM.
2. insufficient number of Datanodes are up. Too many datanodes are not
available, either they are down, or they have not completed startup process.
For example, datanodes are still verifying containers and volumes.
3. Insufficient number of healthy containers. Containers may not be replicated
sufficiently, or some containers are corrupt. Run 'ozone admin container
report' command to check what happens. See Container Replication Report page at
https://ozone.apache.org/docs/administrator-guide/operations/container-replication-report
for details.
If admin is absolutely sure the cluster is healthy, manually exit the safe mode
using command 'ozone admin safemode --force-exit'
> [Website v2][Troubleshooting] How to exit safe mode
> ---------------------------------------------------
>
> Key: HDDS-15028
> URL: https://issues.apache.org/jira/browse/HDDS-15028
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: documentation
> Reporter: Wei-Chiu Chuang
> Priority: Major
>
> https://ozone.apache.org/docs/core-concepts/architecture/storage-container-manager/#how-to-exit-safe-mode
> Typically SCM should exit safe mode by itself. We need a troubleshooting doc
> to identify the problem that prevent automatic safe mode exit.
> Symptoms:
> Client application attempting to write files, but the request is rejected
> with exception message "SafeModePrecheck failed for allocateBlock"
> Get safe mode status:
> 'ozone admin safemode --verbose' command, or check leader SCM web UI. At the
> bottom is the Safemode rules statuses, and the SCM stays in the safe mode if
> any of the rules is not satisifed.
> 1. In a SCM-HA environment, only one SCM is up. In which case, start the
> other two SCM.
> 2. insufficient number of Datanodes are up. Too many datanodes are not
> available, either they are down, or they have not completed startup process.
> For example, datanodes are still verifying containers and volumes.
> 3. Insufficient number of healthy containers. Containers may not be
> replicated sufficiently, or some containers are corrupt. Run 'ozone admin
> container report' command to check what happens. See Container Replication
> Report page at
> https://ozone.apache.org/docs/administrator-guide/operations/container-replication-report
> for details.
> If admin is absolutely sure the cluster is healthy, manually exit the safe
> mode using command 'ozone admin safemode --force-exit'
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
