This is an automated email from the ASF dual-hosted git repository.
jojochuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ozone-site.git
The following commit(s) were added to refs/heads/master by this push:
new 3814834113 HDDS-15917. Add client connectivity troubleshooting
runbook. (#498)
3814834113 is described below
commit 3814834113671e1c3dd02f4720ef6900405d5dca
Author: Wei-Chiu Chuang <[email protected]>
AuthorDate: Tue Jul 21 12:46:15 2026 -0600
HDDS-15917. Add client connectivity troubleshooting runbook. (#498)
Generated-by: Cursor <[email protected]>, Composer 2.5
---
.../01-installation/02-kubernetes.md | 4 ++
.../02-configuration/99-appendix.md | 2 +-
docs/06-troubleshooting/18-client-connectivity.md | 69 ++++++++++++++++++++++
docs/06-troubleshooting/README.mdx | 2 -
docusaurus.config.js | 4 +-
5 files changed, 75 insertions(+), 6 deletions(-)
diff --git a/docs/02-quick-start/01-installation/02-kubernetes.md
b/docs/02-quick-start/01-installation/02-kubernetes.md
index d57508b9dd..cdf350e82d 100644
--- a/docs/02-quick-start/01-installation/02-kubernetes.md
+++ b/docs/02-quick-start/01-installation/02-kubernetes.md
@@ -130,3 +130,7 @@ Now you can access any of the services. By default the
services are not publishe
kubectl port-forward s3g-0 9878:9878
kubectl port-forward scm-0 9876:9876
```
+
+:::note Client connectivity
+If metadata operations work but client-to-Datanode data traffic fails, or
Datanode pods restart often, set `hdds.datanode.use.datanode.hostname=true` on
clients and Datanodes. See [Troubleshooting client
connectivity](../../troubleshooting/client-connectivity).
+:::
diff --git a/docs/05-administrator-guide/02-configuration/99-appendix.md
b/docs/05-administrator-guide/02-configuration/99-appendix.md
index 5e0d51bbdb..35779bcc61 100644
--- a/docs/05-administrator-guide/02-configuration/99-appendix.md
+++ b/docs/05-administrator-guide/02-configuration/99-appendix.md
@@ -181,7 +181,7 @@ This page provides a comprehensive overview of the
configuration keys available
| `hdds.datanode.slow.op.warning.threshold` | 500ms | `OZONE`, `DATANODE`,
`PERFORMANCE` | Thresholds for printing slow-operation audit logs. |
| `hdds.datanode.storage.utilization.critical.threshold` | 0.95 | `OZONE`,
`SCM`, `MANAGEMENT` | If a datanode overall storage utilization exceeds more
than this value, the datanode will be marked out of space. |
| `hdds.datanode.storage.utilization.warning.threshold` | 0.75 | `OZONE`,
`SCM`, `MANAGEMENT` | If a datanode overall storage utilization exceeds more
than this value, a warning will be logged while processing the nodeReport in
SCM. |
-| `hdds.datanode.use.datanode.hostname` | false | `OZONE`, `DATANODE` |
Whether Datanodes should use Datanode hostnames when connecting to other
Datanodes for data transfer. |
+| `hdds.datanode.use.datanode.hostname` | false | `OZONE`, `DATANODE` |
Whether Datanodes should use Datanode hostnames when connecting to other
Datanodes for data transfer. See [Troubleshooting client
connectivity](../../troubleshooting/client-connectivity). |
| `hdds.datanode.volume.choosing.policy` |
org.apache.hadoop.ozone.container.common.volume.CapacityVolumeChoosingPolicy |
`OZONE`, `CONTAINER`, `STORAGE`, `MANAGEMENT` | The class name of the policy
for choosing volumes in the list of directories. Defaults to
org.apache.hadoop.ozone.container.common.volume.CapacityVolumeChoosingPolicy.
This volume choosing policy randomly chooses two volumes with remaining space
and then picks the one with lower utilization. |
| `hdds.datanode.volume.min.free.space` | -1 | `OZONE`, `CONTAINER`,
`STORAGE`, `MANAGEMENT` | Minimum free space (bytes) applied together with
min.free.space.percent (reported to SCM in heartbeat as freeSpaceToSpare) and
min.free.space.hard.limit.percent (local write enforcement). The effective
value for each tier is max(this bytes, capacity * ratio). |
| `hdds.datanode.volume.min.free.space.hard.limit.percent` | 0.015 | `OZONE`,
`CONTAINER`, `STORAGE`, `MANAGEMENT` | Minimum fraction of volume capacity
reserved for local enforcement: writes fail when available space would drop
below max(this ratio * capacity, `hdds.datanode.volume.min.free.space`). Should
be <= min.free.space.percent so SCM can plan for a larger headroom than the
DN enforces locally. |
diff --git a/docs/06-troubleshooting/18-client-connectivity.md
b/docs/06-troubleshooting/18-client-connectivity.md
new file mode 100644
index 0000000000..11f7d104ed
--- /dev/null
+++ b/docs/06-troubleshooting/18-client-connectivity.md
@@ -0,0 +1,69 @@
+---
+sidebar_label: Client connectivity
+---
+
+# Troubleshooting client connectivity
+
+Use this page when **Ozone clients cannot connect to Datanodes** for data
transfer, even though metadata operations through the Ozone Manager still work.
+
+## When to use this page
+
+- **Kubernetes** — Datanode pods were restarted; data reads or writes fail, or
pipelines misbehave after restarts.
+- **Port-forward or remote clients** — You reach OM or S3 from outside the
cluster network; metadata works but client-to-Datanode data traffic fails.
+- **Multi-homed servers** — Datanodes have more than one network; clients
cannot reach the Datanode **IP** SCM registered.
+
+## Symptoms
+
+- **Metadata operations succeed** (for example `ozone sh key info`, listing
keys, or S3 object metadata), but **data operations fail** (reads, writes, or
other client-to-Datanode traffic).
+- Logs mention `UNAVAILABLE`, `Connection refused`, or a Datanode **IP** that
is not reachable from the client host.
+- After Kubernetes pod restarts, write throughput drops or never recovers.
+
+## Why hostname mode helps
+
+With the default (`hdds.datanode.use.datanode.hostname=false`), the client
opens gRPC to the **IP address stored in SCM** for each Datanode. That value is
used directly; the client does not look it up in DNS again at connect time. If
the IP is stale or not routable from your client, data operations fail even
when hostnames would work.
+
+With `hdds.datanode.use.datanode.hostname=true`, the client connects using the
**hostname** from SCM, and the JVM resolves that name when opening the channel.
+
+## Fix
+
+**Step 1.** Add this to **Ozone clients** and **every Datanode**
(`ozone-site.xml`):
+
+```xml
+<property>
+ <name>hdds.datanode.use.datanode.hostname</name>
+ <value>true</value>
+</property>
+```
+
+**Step 2.** Ensure Datanode **hostnames** resolve on the network where data is
read and written:
+
+- **Kubernetes:** use stable pod DNS (StatefulSet with a headless Service).
+- **Multi-homed:** set `hdds.datanode.hostname` to the storage FQDN if needed.
Make sure DNS or `/etc/hosts` on clients resolves that name on the **data**
network.
+
+**Step 3.** Restart clients and Datanodes (or roll pods) so the new setting is
loaded.
+
+:::note Ozone version
+For **reads**, this setting is fully honored from **Ozone 2.1** onward. On
older releases, run read workloads inside the cluster network or upgrade. See
[HDDS-13124](https://issues.apache.org/jira/browse/HDDS-13124).
+:::
+
+## Verify
+
+Run from the **same host and network** as your application:
+
+```bash
+ozone sh key info <volume>/<bucket>/test.txt
+ozone sh key cat <volume>/<bucket>/test.txt
+```
+
+The first command checks metadata (OM). The second checks data read
(Datanode). On Kubernetes, repeat after restarting a Datanode pod.
+
+## Still stuck?
+
+- From the client host, check that the Datanode hostname shown in SCM or Recon
resolves (`getent hosts <hostname>`).
+- **Multi-homed only:** if SCM shows the wrong IP, review
`hdds.datanode.dns.interface` in the [configuration
appendix](../administrator-guide/configuration/appendix).
+- If **OM or SCM** RPC fails (not Datanode data ports), that is a separate
issue. See [Client
Failover](../administrator-guide/configuration/high-availability/client-failover).
+
+## See also
+
+- [HDDS-5916](https://issues.apache.org/jira/browse/HDDS-5916) — Kubernetes
pipeline issues after pod IP changes
+- [HDDS-13124](https://issues.apache.org/jira/browse/HDDS-13124) — read path
and hostname configuration
diff --git a/docs/06-troubleshooting/README.mdx
b/docs/06-troubleshooting/README.mdx
index d89e8722c9..88bbd41582 100644
--- a/docs/06-troubleshooting/README.mdx
+++ b/docs/06-troubleshooting/README.mdx
@@ -1,7 +1,5 @@
# Troubleshooting
-**TODO:** Allow inclusion of the troubleshooting folder by updating the
excludes list in `docusaurus.config.js`.
-
import DocCardList from '@theme/DocCardList';
This section contains guides to help troubleshoot common issues in an Ozone
deployment.
diff --git a/docusaurus.config.js b/docusaurus.config.js
index 3c98447fd6..5c4739f8f5 100644
--- a/docusaurus.config.js
+++ b/docusaurus.config.js
@@ -240,10 +240,8 @@ const config = {
sidebarPath: undefined,
editUrl:
'https://github.com/apache/ozone-site/tree/master',
- // TODO: The following sections are currently hidden. Ensure that a
section contains a few pages
- // of publishable quality before enabling visibility for that
section.
+ // TODO: Ensure remaining sections contain publishable pages before
enabling.
exclude: [
- '**/06-troubleshooting/**',
]
},
blog: {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
