w3ll1ngt commented on code in PR #13130: URL: https://github.com/apache/ignite/pull/13130#discussion_r3603375836
########## docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc: ########## @@ -47,3 +47,345 @@ queries with JOINs at massive scale and expect significant performance benefits. * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes. +== How to assess cluster health + +Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention. + +It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes. + +The key properties are: +- The cluster serves operations allowed by its intended state. +- Data remains consistent (`idle_verify` passes, meaning partitions are intact and there are no conflicts). +- Expected nodes are present. +- No pending transactions and no split brain. +- No lost partitions. +- Transient activity such as rebalancing or queue growth converges instead of accumulating. + +But the thing is, there is no universal single health metric. + +Use several signals together. +For example, `SELECT 1` can confirm that a client or SQL query path is reachable, but it does not check user partitions, backup consistency, baseline membership, or all server nodes. +Similarly, `control.sh --cache idle_verify` is an important consistency check, but it is not a complete health check. + +=== Check the Intended State and Node Membership +Start with the link:monitoring-metrics/cluster-states[cluster state]. + +Run: + +[source,shell] +---- +./control.sh --state +---- + +Relevant output: + +[source,text] +---- +Command [STATE] started +Arguments: --state +-------------------------------------------------------------------------------- +Cluster state: ACTIVE +Command [STATE] finished with code: 0 +---- + +`ACTIVE` is expected for normal read-write operation. +`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally enabled. +`INACTIVE` is acceptable only when it matches the current operation, for example planned maintenance; an inactive cluster does not serve the data workload. +The criterion is whether the actual state matches the state that was intentionally set for the deployment. + +If baseline topology is enabled, check the baseline as well: + +Run: + +[source,shell] +---- +./control.sh --baseline +---- + +Relevant output for a cluster where all baseline nodes are online: + +[source,text] +---- +Cluster state: ACTIVE +Current topology version: 3 +Baseline auto adjustment disabled: softTimeout=300000 + +Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1) + +Baseline nodes: + ConsistentId=node-1, State=ONLINE, Order=1 + ConsistentId=node-2, State=ONLINE, Order=2 + ConsistentId=node-3, State=ONLINE, Order=3 +-------------------------------------------------------------------------------- +Number of baseline nodes: 3 + +Other nodes not found. +---- + +If an online server node has joined the cluster but is not in the baseline, the command shows it under `Other nodes`: + +[source,text] +---- +Other nodes: + ConsistentId=node-4, Order=4 +Number of other nodes: 1 +---- + +The baseline contains server nodes that are intended to store data. +Client nodes are not part of the baseline. +An online server node in `Other nodes` is not always an error: the node may have been prepared intentionally but not yet introduced into the data topology. +If the node is expected to store data, first check the link:clustering/baseline-topology#baseline-topology-autoadjustment[baseline auto-adjustment policy] and the deployment plan, then use the documented baseline change procedure. Review Comment: extended to fulfil this thought ########## docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc: ########## @@ -47,3 +47,345 @@ queries with JOINs at massive scale and expect significant performance benefits. * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes. +== How to assess cluster health + +Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention. + +It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes. + +The key properties are: +- The cluster serves operations allowed by its intended state. +- Data remains consistent (`idle_verify` passes, meaning partitions are intact and there are no conflicts). +- Expected nodes are present. +- No pending transactions and no split brain. +- No lost partitions. +- Transient activity such as rebalancing or queue growth converges instead of accumulating. + +But the thing is, there is no universal single health metric. + +Use several signals together. +For example, `SELECT 1` can confirm that a client or SQL query path is reachable, but it does not check user partitions, backup consistency, baseline membership, or all server nodes. +Similarly, `control.sh --cache idle_verify` is an important consistency check, but it is not a complete health check. + +=== Check the Intended State and Node Membership +Start with the link:monitoring-metrics/cluster-states[cluster state]. + +Run: + +[source,shell] +---- +./control.sh --state +---- + +Relevant output: + +[source,text] +---- +Command [STATE] started +Arguments: --state +-------------------------------------------------------------------------------- +Cluster state: ACTIVE +Command [STATE] finished with code: 0 +---- + +`ACTIVE` is expected for normal read-write operation. +`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally enabled. +`INACTIVE` is acceptable only when it matches the current operation, for example planned maintenance; an inactive cluster does not serve the data workload. +The criterion is whether the actual state matches the state that was intentionally set for the deployment. + +If baseline topology is enabled, check the baseline as well: + +Run: + +[source,shell] +---- +./control.sh --baseline +---- + +Relevant output for a cluster where all baseline nodes are online: + +[source,text] +---- +Cluster state: ACTIVE +Current topology version: 3 +Baseline auto adjustment disabled: softTimeout=300000 + +Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1) + +Baseline nodes: + ConsistentId=node-1, State=ONLINE, Order=1 + ConsistentId=node-2, State=ONLINE, Order=2 + ConsistentId=node-3, State=ONLINE, Order=3 +-------------------------------------------------------------------------------- +Number of baseline nodes: 3 + +Other nodes not found. +---- + +If an online server node has joined the cluster but is not in the baseline, the command shows it under `Other nodes`: + +[source,text] +---- +Other nodes: + ConsistentId=node-4, Order=4 +Number of other nodes: 1 +---- + +The baseline contains server nodes that are intended to store data. +Client nodes are not part of the baseline. +An online server node in `Other nodes` is not always an error: the node may have been prepared intentionally but not yet introduced into the data topology. +If the node is expected to store data, first check the link:clustering/baseline-topology#baseline-topology-autoadjustment[baseline auto-adjustment policy] and the deployment plan, then use the documented baseline change procedure. +Changing the baseline can start link:data-rebalancing[rebalancing]. This is often not an easy procedure for a cluster with data in persistence storage. + +Frequent unexpected topology changes are not normal and usually indicate node instability, network issues, or misconfigured discovery. However, a temporary topology change does not make the cluster unhealthy by itself. +Server `JOIN`, `LEFT`, and `FAIL` events change cluster membership and the major topology version. Review Comment: done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
