This is an automated email from the ASF dual-hosted git repository. ArafatKhan2198 pushed a commit to branch jira2 in repository https://gitbox.apache.org/repos/asf/ozone-site.git
commit c9e97dde745f3fddd6b7f9693aef4bb5ae07ed79 Author: arafat <[email protected]> AuthorDate: Fri Jul 24 20:55:37 2026 +0530 HDDS-15965. Add Recon Web UI cluster pages: Datanodes, Pipelines, Containers. --- .../02-recon/01-recon-web-ui/05-datanodes.mdx | 143 ++++++++++++++++++ .../02-recon/01-recon-web-ui/06-pipelines.mdx | 115 +++++++++++++++ .../02-recon/01-recon-web-ui/07-containers.mdx | 164 +++++++++++++++++++++ .../ozone_recon_containers.png | Bin .../ozone_recon_datanodes.png | Bin .../ozone_recon_pipelines.png | Bin 6 files changed, 422 insertions(+) diff --git a/docs/05-administrator-guide/03-operations/09-observability/02-recon/01-recon-web-ui/05-datanodes.mdx b/docs/05-administrator-guide/03-operations/09-observability/02-recon/01-recon-web-ui/05-datanodes.mdx new file mode 100644 index 0000000000..fb3b3fbaa9 --- /dev/null +++ b/docs/05-administrator-guide/03-operations/09-observability/02-recon/01-recon-web-ui/05-datanodes.mdx @@ -0,0 +1,143 @@ +--- +sidebar_label: Datanodes +--- + +# Recon UI — Datanodes Page + + + +## 1. Page Overview + +The **Datanodes** page lists every datanode Recon knows about and shows the +health, operational state, storage usage, pipeline membership, container counts, +and build information for each one. It is the main place to check datanode +health and capacity across the cluster. + +Besides browsing, the page lets you stop tracking datanodes that are no longer +part of the cluster (removing dead nodes from Recon). + +## 2. When to Use This Page + +- To confirm all datanodes are healthy and in service. +- To find stale or dead datanodes and investigate them. +- To see which datanodes are being decommissioned or are in maintenance. +- To check per-datanode storage usage and spot nodes that are filling up. +- To see which pipelines a datanode participates in and how many it leads. +- To clean up Recon's view by removing datanodes that are permanently gone. + +## 3. How to Access the Page + +Open the **Datanodes** entry in the left navigation menu, or go to the +`/Datanodes` route directly. You can also arrive here from the **Datanodes** +health card on the Overview page. + +## 4. Information Displayed + +The page header shows the title and an **Auto Reload** panel (see Available +Actions). The main area is a table of datanodes, one row per node. + +### Table columns + +- **Hostname** — the datanode host (always shown, pinned to the left). Sortable. +- **State** — datanode health, with an icon: + - **HEALTHY** (green check) — heartbeating normally. + - **STALE** (hourglass) — heartbeats delayed. + - **DEAD** (red cross) — not heartbeating. + Filterable and sortable. +- **Operational State** — the administrative state: + - **IN_SERVICE** (green check) — serving normally. + - **DECOMMISSIONING** (hourglass) — decommission in progress. + - **DECOMMISSIONED** (warning) — decommission complete. + - **ENTERING_MAINTENANCE** (hourglass) — entering maintenance. + - **IN_MAINTENANCE** (warning) — under maintenance. + Filterable and sortable. +- **UUID** — the datanode's unique ID. For datanodes that are actively + decommissioning, this cell shows a decommission summary instead of the plain + ID. +- **Storage Capacity** — a bar showing used, committed, and remaining storage + for the node. Sortable by remaining storage. +- **Last Heartbeat** — how long ago the node last reported in; `NA` if never. +- **Pipeline ID(s)** — the number of pipelines the node is in; hover to see the + list, each with its replication type/factor and an indicator of whether this + node is the pipeline leader. +- **Leader Count** — the number of Ratis pipelines in which this datanode is + elected leader. Sortable. +- **Containers** — the number of containers on the node. Sortable. +- **Open Container** — the number of open containers per pipeline. Sortable. +- **Version** — the Ozone version running on the node. Sortable. +- **Setup Time** — when the node was set up; `NA` if unavailable. +- **Revision** — the build revision. Sortable. +- **Build Date** — the build date. Sortable. +- **Network Location** — the node's rack/network location. Sortable. + +## 5. Available Actions + +- **Columns** selector — choose which columns are visible. **Hostname** is + always shown. +- **Search** — filter the shown rows by **Hostname**, **UUID**, or **Version**; + matching is a simple contains match on the loaded rows. Disabled when the + table is empty. +- **State / Operational State filters** — filter to specific values from those + column headers. +- **Sort** — click a sortable column header. +- **Pagination** — page through results; page size is adjustable and the footer + shows the range and total (for example, `1-10 of 24 Datanodes`). +- **Row selection + Remove** — select one or more datanodes to reveal a red + **Remove** button. Only **DEAD** datanodes can be selected; healthy or stale + nodes cannot. Choosing **Remove** opens a **Stop Tracking Datanode** + confirmation dialog before Recon stops tracking the selected node(s). +- **Auto Reload** panel — an **Auto Refresh** toggle (refreshes every 60 seconds + and remembers your choice for the session; polling starts automatically on + this page) and a manual **reload** button showing the last refreshed time. +- **Pipeline popover** — hover over the Pipeline ID(s) cell to see the node's + pipelines and leadership. + +## 6. How to Interpret the Information + +- **State HEALTHY:** the node is heartbeating normally. **STALE:** heartbeats + are late — the node may be under load or having network issues. **DEAD:** the + node has stopped heartbeating and its data is treated as unavailable. +- **Operational State:** IN_SERVICE is the normal state. DECOMMISSIONING / + DECOMMISSIONED mean the node is being or has been retired. ENTERING_MAINTENANCE + / IN_MAINTENANCE mean it is temporarily out for maintenance. +- **Storage Capacity bar:** watch for nodes that are nearly full or unbalanced + relative to others. Committed space is reserved but not yet written. +- **Last Heartbeat:** a large value (or `NA`) alongside a STALE/DEAD state + confirms the node is not reporting. +- **Leader Count:** a very uneven leader distribution across nodes can indicate + imbalance in Ratis leadership. +- **Remove eligibility:** the fact that only DEAD nodes can be selected is a + safeguard — you cannot accidentally stop tracking a live node. + +## 7. Common Use Cases + +1. **Health sweep.** Filter **State** to STALE and DEAD to immediately see any + datanodes that are not healthy, then check their Last Heartbeat. +2. **Capacity check.** Sort by **Storage Capacity** to find the fullest nodes + and plan rebalancing or expansion. +3. **Clean up retired nodes.** After a node is permanently removed from the + cluster and shows as DEAD, select it and use **Remove** to stop Recon from + tracking it. + +## 8. Important Notes and Limitations + +- **Data source and freshness.** Datanode information comes from Recon's own SCM + view, which is built from datanode heartbeats and reports. State and storage + figures are only as current as the last heartbeat, and **Auto Refresh** only + re-queries Recon. +- **Remove only affects Recon.** Removing a datanode stops Recon from tracking + it (it is removed from Recon's memory and its record in Recon's database). It + does not decommission or delete the node in the cluster, and only **DEAD** + nodes can be removed. If a removed node comes back and heartbeats, it can + reappear. +- **Decommission details** for actively decommissioning nodes are shown in the + UUID cell as a summary. +- **Search and filters apply to the loaded rows.** +- On an error, the page reports a data-fetch error and the table stays empty. + +## 9. Related Pages + +- **Overview** — the Datanodes health card and cluster-wide totals. +- **Pipelines** — details of the pipelines a datanode participates in. +- **Containers** — container-level detail, including missing containers. +- **Capacity** — cluster-wide storage and capacity breakdown. diff --git a/docs/05-administrator-guide/03-operations/09-observability/02-recon/01-recon-web-ui/06-pipelines.mdx b/docs/05-administrator-guide/03-operations/09-observability/02-recon/01-recon-web-ui/06-pipelines.mdx new file mode 100644 index 0000000000..b478a06d5d --- /dev/null +++ b/docs/05-administrator-guide/03-operations/09-observability/02-recon/01-recon-web-ui/06-pipelines.mdx @@ -0,0 +1,115 @@ +--- +sidebar_label: Pipelines +--- + +# Recon UI — Pipelines Page + + + +## 1. Page Overview + +The **Pipelines** page lists the pipelines in your Ozone cluster. A pipeline is +the group of datanodes that work together to store replicas of container data. +For each pipeline the page shows its replication type and factor, status, the +datanodes involved, the current leader, how long it has been alive, and +leader-election metrics. + +It is a read-only listing used to review pipeline health, replication, and +datanode membership. + +## 2. When to Use This Page + +- To see all pipelines and their current status. +- To confirm replication type and factor across pipelines. +- To find which datanodes make up a pipeline and which one is the leader. +- To investigate pipelines that are not OPEN or that show frequent leader + elections. +- To look up a specific pipeline by its ID. + +## 3. How to Access the Page + +Open the **Pipelines** entry in the left navigation menu, or go to the +`/Pipelines` route directly. You can also arrive here from the **Pipelines** +count card on the Overview page. + +## 4. Information Displayed + +The page header shows the title and an **Auto Reload** panel (see Available +Actions). The main area is a table of pipelines, one row per pipeline. + +### Table columns + +- **Pipeline ID** — the pipeline's unique identifier (always shown, pinned to + the left). Sortable. +- **Replication Type & Factor** — the replication scheme (for example RATIS) and + factor (for example 3), with a replication icon. Sortable. +- **Status** — the pipeline state: OPEN, CLOSING, QUASI_CLOSED, CLOSED, + UNHEALTHY, INVALID, DELETED, or DORMANT. Filterable and sortable. +- **Containers** — the number of containers in the pipeline. Sortable. +- **Datanodes** — the datanodes that make up the pipeline, shown by hostname; + hover a hostname to see its UUID. +- **Leader** — the datanode currently acting as the pipeline leader. Sortable. +- **Last Leader Election** — elapsed time since the current leader was elected. + Shows `NA` unless a metrics provider such as Prometheus is configured. +- **Lifetime** — how long the pipeline has been alive. Sortable. +- **No. of Elections** — the number of leader elections in the pipeline. Shows + `NA` unless a metrics provider such as Prometheus is configured. + +## 5. Available Actions + +- **Columns** selector — choose which columns are visible. **Pipeline ID** is + always shown. +- **Search** — filter the shown rows by **Pipeline ID**; matching is a simple + contains match on the loaded rows. Disabled when the table is empty. +- **Status filter** — filter to specific pipeline states from the Status column + header. +- **Sort** — click a sortable column header. +- **Pagination** — page through results; page size is adjustable and the footer + shows the range and total (for example, `1-10 of 42 pipelines`). +- **Auto Reload** panel — an **Auto Refresh** toggle (refreshes every 60 seconds + and remembers your choice for the session) and a manual **reload** button + showing the last refreshed time. +- **Datanode tooltip** — hover a datanode hostname to see its UUID. + +## 6. How to Interpret the Information + +- **Status OPEN:** the pipeline is active and accepting writes. CLOSING / + QUASI_CLOSED / CLOSED indicate the pipeline is being or has been shut down. + UNHEALTHY, INVALID, DELETED, or DORMANT indicate pipelines that are not in + normal service and may warrant investigation. +- **Replication Type & Factor:** confirms how many replicas the pipeline + maintains (for example RATIS with factor 3 keeps three copies). +- **Datanodes / Leader:** shows exactly which nodes hold the pipeline and which + one leads it. Cross-reference with the Datanodes page if a member node is + unhealthy. +- **Last Leader Election / No. of Elections:** frequent elections can indicate + instability (for example a flaky leader node or network issues). These values + are only populated when a metrics provider is configured; otherwise they show + `NA` and are not an error. + +## 7. Common Use Cases + +1. **Spot unhealthy pipelines.** Filter **Status** to values other than OPEN to + quickly find pipelines that are closing, unhealthy, or invalid. +2. **Trace a pipeline's datanodes.** Search for a Pipeline ID, then read the + Datanodes and Leader columns to see which nodes hold its data. +3. **Investigate leader churn.** With a metrics provider configured, sort by + **No. of Elections** to find pipelines with excessive leader elections and + correlate with datanode health. + +## 8. Important Notes and Limitations + +- **Data source and freshness.** Pipeline information comes from Recon's own SCM + view, built from datanode reports. It is only as current as the last refresh, + and **Auto Refresh** only re-queries Recon. +- **Metrics columns need a metrics provider.** **Last Leader Election** and + **No. of Elections** are populated only when a metrics service such as + Prometheus is configured for the cluster. Without it, these columns show `NA`. +- **Search and filters apply to the loaded rows.** +- On an error, the page reports a data-fetch error and the table stays empty. + +## 9. Related Pages + +- **Datanodes** — health and details of the datanodes that make up pipelines. +- **Containers** — the containers stored within pipelines. +- **Overview** — cluster-wide totals, including the pipeline count. diff --git a/docs/05-administrator-guide/03-operations/09-observability/02-recon/01-recon-web-ui/07-containers.mdx b/docs/05-administrator-guide/03-operations/09-observability/02-recon/01-recon-web-ui/07-containers.mdx new file mode 100644 index 0000000000..d40e3e7a20 --- /dev/null +++ b/docs/05-administrator-guide/03-operations/09-observability/02-recon/01-recon-web-ui/07-containers.mdx @@ -0,0 +1,164 @@ +--- +sidebar_label: Containers +--- + +# Recon UI — Containers Page + + + +## 1. Page Overview + +The **Containers** page focuses on **unhealthy containers** — containers whose +replicas are missing, under- or over-replicated, mis-replicated, mismatched, or +quasi-closed. A highlights bar summarizes the counts, tabs break the containers +down by problem type, and each container can be expanded to see the keys it +holds. An Export tab lets you download the full list of a given unhealthy state +as a CSV. + +It is the main page for diagnosing container-level replication and data- +availability problems. + +## 2. When to Use This Page + +- To check whether any containers are missing (a data-availability risk). +- To find under-replicated or over-replicated containers and track recovery. +- To identify mis-replicated or replica-mismatched containers. +- To see which keys are affected by an unhealthy container. +- To export a full list of containers in a given state for offline analysis or + ticketing. + +## 3. How to Access the Page + +Open the **Containers** entry in the left navigation menu, or go to the +`/Containers` route directly. You can also arrive here from the **Containers** +health card on the Overview page. + +## 4. Information Displayed + +The page header shows the title and an **Auto Reload** panel (see Available +Actions). + +### Highlights card + +A summary bar of cluster-wide container counts: + +- **Total Containers** — all containers in the cluster. +- **Missing** — containers with no available replicas. +- **Under-Replicated** — fewer replicas than expected. +- **Over-Replicated** — more replicas than expected. +- **Mis-Replicated** — replicas placed in a way that violates placement rules. +- **Mismatched Replicas** — replicas that disagree (for example, differing data + checksums). +- **Quasi Closed** — containers stuck in the quasi-closed state. + +### Tabs + +Each of the first six tabs shows a table of containers in that state: **Missing**, +**Under-Replicated**, **Over-Replicated**, **Mis-Replicated**, **Mismatched +Replicas**, and **Quasi Closed**. The seventh tab is **Export**. + +### Container table columns + +- **Container ID** — the container identifier. Sortable. +- **No. of Blocks** — number of blocks/keys in the container. Sortable. +- **Actual/Expected Replica(s)** — the count of replicas present versus expected + (for example `2 / 3`). +- **Datanodes** — the number of datanodes holding replicas; hover to list the + hosts (and data checksums when available). +- **Pipeline ID** — the pipeline the container belongs to. +- **Unhealthy Since** — when the container entered this unhealthy state. On the + Quasi Closed tab this column is labeled **State Enter Time**. + +### Expanded row — keys in a container + +Clicking a container row expands it and loads the keys stored in that container, +in a table with **Volume**, **Bucket**, **Key**, **Size**, **Date Created**, +**Date Modified**, and **Path**. + +### Export tab + +- A **Container State** selector and an **Export CSV** button to start an export + job for one state. +- **Active Exports** table (shown when jobs are running/queued): Job ID, State, + Status, Queue Position, Submitted, Started, and a live **Progress** bar with a + processed/total record count. +- **Completed Exports** table: Job ID, State, Status, Records, Submitted, + Started, Completed, and an **Action** column with **Download** (showing the + number of downloads remaining) and **Delete**. Failed jobs show the error + message instead of a download button. + +## 5. Available Actions + +- **Tabs** — switch between unhealthy states; each tab loads its own data on + first open. +- **Columns** selector — choose visible columns. **Container ID** is always + shown. +- **Search** — filter loaded rows by **Container ID** or **Pipeline ID** + (contains match). Disabled when the table is empty. +- **Sort** — click a sortable column header. +- **Row expand** — click a row to load and view the keys in that container. +- **Pagination** — **Previous** / **Next** buttons with a **Rows per page** + selector (10, 25, 50, 100). Paging uses container IDs, so it steps forward and + backward through the list rather than jumping to arbitrary pages. +- **Auto Reload** panel — an **Auto Refresh** toggle (refreshes the active tab + and highlights every 60 seconds) and a manual **reload** button. +- **Export actions** — start an export for a chosen state, watch progress, + download the resulting CSV (limited number of downloads), and delete finished + jobs. Only one active/completed export per state is allowed at a time; the + button is disabled and explains why if one already exists. + +## 6. How to Interpret the Information + +- **Missing > 0:** the most serious state — those containers have no reachable + replicas, so their data is currently unavailable. Investigate immediately. +- **Under-Replicated:** fewer copies than required; the system normally + re-replicates automatically, but persistent entries may indicate datanode or + capacity problems. +- **Over-Replicated:** more copies than required; usually self-corrects, wasting + some space until it does. +- **Mis-Replicated:** replica placement violates the configured placement policy + (for example rack rules), even if the count is correct. +- **Mismatched Replicas:** replicas exist but disagree — check the per-datanode + checksums in the Datanodes popover. +- **Quasi Closed:** containers that could not be cleanly closed; the State Enter + Time shows how long they have been stuck. +- **Actual/Expected `2 / 3`:** actual is less than expected → under-replicated; + greater → over-replicated. +- **Export status:** QUEUED (waiting), RUNNING (in progress, with a progress + bar), COMPLETED (downloadable), FAILED (hover for the error). + +## 7. Common Use Cases + +1. **Data-availability check.** Open the page, read the **Missing** highlight, + and if non-zero open the Missing tab and expand rows to see exactly which + keys are affected. +2. **Track replication recovery.** Watch the Under-Replicated count and tab over + time (with Auto Refresh on) to confirm the cluster is healing after a datanode + outage. +3. **Hand off to another team.** Use the Export tab to generate a CSV of all + containers in a given state and attach it to a ticket for follow-up. + +## 8. Important Notes and Limitations + +- **Data source and freshness.** Unhealthy-container information comes from + Recon's own database, populated by Recon's periodic container-health checks; + the keys shown when expanding a container come from Recon's copy of the OM + database. Figures are only as current as those background tasks, and **Auto + Refresh** only re-queries Recon. +- **The tables list unhealthy containers**, not all containers. Total container + count appears only in the Highlights bar. +- **Missing count on the Overview page is capped at 1001**; this page shows the + detailed per-state lists. +- **Search applies to the loaded page only** (contains match on Container ID or + Pipeline ID). +- **Exports are limited:** one active/completed job per state at a time, a + bounded number of downloads per completed job, and a bounded queue (starting an + export may be rejected if the queue is full). +- On an error, the page shows a data-fetch error; export polling errors are + silent by design. + +## 9. Related Pages + +- **Overview** — the Containers health card and total container count. +- **Datanodes** — health of the datanodes that hold container replicas. +- **Pipelines** — the pipelines containers belong to. diff --git a/docs/05-administrator-guide/03-operations/09-observability/02-recon/ozone_recon_containers.png b/docs/05-administrator-guide/03-operations/09-observability/02-recon/01-recon-web-ui/ozone_recon_containers.png similarity index 100% rename from docs/05-administrator-guide/03-operations/09-observability/02-recon/ozone_recon_containers.png rename to docs/05-administrator-guide/03-operations/09-observability/02-recon/01-recon-web-ui/ozone_recon_containers.png diff --git a/docs/05-administrator-guide/03-operations/09-observability/02-recon/ozone_recon_datanodes.png b/docs/05-administrator-guide/03-operations/09-observability/02-recon/01-recon-web-ui/ozone_recon_datanodes.png similarity index 100% rename from docs/05-administrator-guide/03-operations/09-observability/02-recon/ozone_recon_datanodes.png rename to docs/05-administrator-guide/03-operations/09-observability/02-recon/01-recon-web-ui/ozone_recon_datanodes.png diff --git a/docs/05-administrator-guide/03-operations/09-observability/02-recon/ozone_recon_pipelines.png b/docs/05-administrator-guide/03-operations/09-observability/02-recon/01-recon-web-ui/ozone_recon_pipelines.png similarity index 100% rename from docs/05-administrator-guide/03-operations/09-observability/02-recon/ozone_recon_pipelines.png rename to docs/05-administrator-guide/03-operations/09-observability/02-recon/01-recon-web-ui/ozone_recon_pipelines.png --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
