This is an automated email from the ASF dual-hosted git repository. asf-gitbox-commits pushed a commit to branch branch_10_1 in repository https://gitbox.apache.org/repos/asf/solr.git
commit 92dd58aa2b9c1f299a31e782a2f12fae9b40605d Author: Eric Pugh <[email protected]> AuthorDate: Sun Sep 27 13:36:43 2026 -0400 SOLR-18417: Have a proper page on healthchecks (#4886) (cherry picked from commit ef22eb5cecda5d33b09364b8714f7d470db1b940) (cherry picked from commit 3225d759123f506acb2473ac5942cd6d53c25255) --- .../pages/implicit-requesthandlers.adoc | 20 +-- .../modules/deployment-guide/deployment-nav.adoc | 1 + .../deployment-guide/pages/healthcheck.adoc | 169 +++++++++++++++++++++ .../pages/taking-solr-to-production.adoc | 2 + 4 files changed, 174 insertions(+), 18 deletions(-) diff --git a/solr/solr-ref-guide/modules/configuration-guide/pages/implicit-requesthandlers.adoc b/solr/solr-ref-guide/modules/configuration-guide/pages/implicit-requesthandlers.adoc index c99fb342feb..6522beb7f10 100644 --- a/solr/solr-ref-guide/modules/configuration-guide/pages/implicit-requesthandlers.adoc +++ b/solr/solr-ref-guide/modules/configuration-guide/pages/implicit-requesthandlers.adoc @@ -40,24 +40,8 @@ This handler must have a collection name in the path to the endpoint. |`solr/<collection>/admin/file` |{solr-javadocs}/core/org/apache/solr/handler/admin/ShowFileRequestHandler.html[ShowFileRequestHandler] |`_ADMIN_FILE` |=== -Health:: Report the health of the node. -+ -[cols="3*.",frame=none,grid=cols,options="header"] -|=== -|API Endpoints |Class & Javadocs |Paramset -|v1: `solr/admin/info/health` - -v2: `api/node/health` |v1: {solr-javadocs}/core/org/apache/solr/handler/admin/HealthCheckHandler.html[HealthCheckHandler] - -v2: {solr-javadocs}/core/org/apache/solr/handler/admin/api/NodeHealth.html[NodeHealth] | -|=== -+ -In SolrCloud mode the handler checks that the node is connected to ZooKeeper and is listed in live nodes. -The optional `requireHealthyCores=true` parameter additionally requires that all local replicas be in an active state, which is useful for rolling-restart probes. -+ -In user-managed (leader-follower) mode the handler checks replication lag. -The optional `maxGenerationLag=<n>` parameter specifies the maximum number of Lucene commit generations by which a follower is allowed to trail its leader; the endpoint returns HTTP 503 if any core exceeds this threshold. -See xref:deployment-guide:user-managed-index-replication.adoc#monitoring-follower-replication-lag[Monitoring Follower Replication Lag] for details and examples. +Health:: Report the health of the node, via the `api/node/health` endpoint ({solr-javadocs}/core/org/apache/solr/handler/admin/api/NodeHealth.html[NodeHealth]). +See xref:deployment-guide:healthcheck.adoc[] for details, parameters, and examples. Logging:: Retrieve and modify registered loggers. + diff --git a/solr/solr-ref-guide/modules/deployment-guide/deployment-nav.adoc b/solr/solr-ref-guide/modules/deployment-guide/deployment-nav.adoc index 83bb2e286ed..9b81acaf3e7 100644 --- a/solr/solr-ref-guide/modules/deployment-guide/deployment-nav.adoc +++ b/solr/solr-ref-guide/modules/deployment-guide/deployment-nav.adoc @@ -58,6 +58,7 @@ * Monitoring Solr ** xref:configuring-logging.adoc[] ** xref:ping.adoc[] +** xref:healthcheck.adoc[] ** xref:metrics-reporting.adoc[] ** xref:performance-statistics-reference.adoc[] ** xref:plugins-stats-screen.adoc[] diff --git a/solr/solr-ref-guide/modules/deployment-guide/pages/healthcheck.adoc b/solr/solr-ref-guide/modules/deployment-guide/pages/healthcheck.adoc new file mode 100644 index 00000000000..7e5bc472214 --- /dev/null +++ b/solr/solr-ref-guide/modules/deployment-guide/pages/healthcheck.adoc @@ -0,0 +1,169 @@ += Healthchecks +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +Solr provides two distinct tools for checking whether things are working, and they answer different questions. +Confusing them is easy, since both are casually called a "healthcheck", so this page describes each and helps you pick the right one. + +[cols="1,2,2"] +|=== +| |Node Health Endpoint |`bin/solr healthcheck` command + +|Answers +|"Is this node alive and part of the cluster?" +|"Does this collection actually have data, and is every replica in sync?" + +|Scope +|A single node +|A single collection, across all its shards and replicas + +|Checks the index/data? +|No -- metadata only +|Yes -- runs real queries + +|Interface +|HTTP endpoint +|Command-line tool + +|Typical use +|Load balancer / orchestrator liveness or readiness probe +|Manual or scripted diagnosis after deployment, restart, or an incident +|=== + +== Node Healthcheck + +The `api/node/health` endpoint ({solr-javadocs}/core/org/apache/solr/handler/admin/api/NodeHealth.html[NodeHealth]) reports whether a single Solr node is alive and able to participate in the cluster. +It is designed to be cheap enough to poll frequently from a load balancer or an orchestrator like Kubernetes. + +[IMPORTANT] +==== +None of these checks touch the index. +A node with a collection that has zero documents, or a collection whose query results are wrong, will still report a healthy `200 OK` here. +If you need to confirm a collection actually has data, see <<Collection Data Healthcheck>> below. +==== + +What the endpoint checks -- and how it fails -- depends on whether the node is running in SolrCloud mode or user-managed (leader-follower) mode. + +=== SolrCloud Mode + +In SolrCloud mode, the endpoint returns HTTP `200 OK` only if all of the following are true, and HTTP `503 Unavailable` otherwise: + +* The node's core container has finished starting up. +* The node is connected to ZooKeeper. +* The node is listed in ZooKeeper's `live_nodes`. + +[source,bash] +---- +curl "http://localhost:8983/api/node/health"; +---- + +Healthy node: + +[source,json] +---- +{ + "responseHeader":{"status":0,"QTime":1}, + "status":"OK" +} +---- + +Unhealthy node (HTTP 503): + +[source,json] +---- +{ + "responseHeader":{"status":503,"QTime":1}, + "status":"FAILURE", + "error":{"msg":"Not connected to ZK"} +} +---- + +==== Rolling Restarts: `requireHealthyCores` + +Add `requireHealthyCores=true` to additionally require that every local replica belonging to an active shard has finished initializing -- i.e., none are in the `RECOVERING` or `DOWN` state. +This is useful as a readiness probe during a rolling restart, so an orchestrator doesn't move on to the next node while the one it just restarted still has replicas recovering. + +[source,bash] +---- +curl "http://localhost:8983/api/node/health?requireHealthyCores=true"; +---- + +=== User-Managed (Leader-Follower) Mode + +In user-managed replication, the endpoint instead checks how far behind each local follower core is from its leader, in Lucene commit generations. +Set `maxGenerationLag=<n>` to fail the healthcheck once a follower falls more than `<n>` generations behind; without it, the check simply reports `OK` once a follower has replicated at least once, even if it later falls arbitrarily far behind. + +[source,bash] +---- +curl "http://localhost:8983/api/node/health?maxGenerationLag=100"; +---- + +Healthy node (all followers within the allowed lag): + +[source,json] +---- +{ + "responseHeader":{"status":0,"QTime":1}, + "status":"OK" +} +---- + +Unhealthy node (a follower has fallen too far behind, HTTP 503): + +[source,json] +---- +{ + "responseHeader":{"status":503,"QTime":1}, + "status":"FAILURE", + "error":{"msg":"Cores violating maxGenerationLag:100.\nCore collection1 is lagging by 137 generations"} +} +---- + +See xref:user-managed-index-replication.adoc#monitoring-follower-replication-lag[Monitoring Follower Replication Lag] for details. + +== Collection Healthcheck + +While the Node Health endpoint checks process liveness, the `bin/solr healthcheck` tool verifies whether a specific collection is functioning properly. It queries every replica directly (non-distributed) to compare document counts, confirm each shard has a leader, and verify all replicas are in the ACTIVE state. + +Because it executes real queries against individual replicas, it is a heavier operation than the Node Health endpoint and is meant as an on-demand CLI tool rather than a continuously polled HTTP healthcheck for load balancers. Use it after deployments, restarts, or incidents to confirm collection integrity. +[source,bash] +---- +bin/solr healthcheck -c gettingstarted +---- + +[source,json] +---- +{ + "collection":"gettingstarted", + "status":"healthy", + "numDocs":42, + "numShards":2, + "shards":[ + { + "shard":"shard1", + "status":"healthy", + "replicas":[ + {"name":"core_node1", "url":"...", "numDocs":21, "status":"active", "leader":true} + ] + } + ] +} +---- + +TIP: Healthy, well-functioning replicas often disagree on exact document counts due to slightly staggered autoCommit periods or in-flight updates. Small, momentary count discrepancies between replicas are normal and should not automatically be treated as an issue unless the mismatch persists. + +See xref:solr-control-script-reference.adoc#healthcheck[the `healthcheck` command reference] for the full list of options. diff --git a/solr/solr-ref-guide/modules/deployment-guide/pages/taking-solr-to-production.adoc b/solr/solr-ref-guide/modules/deployment-guide/pages/taking-solr-to-production.adoc index e91be0fd62c..f3574a4b443 100644 --- a/solr/solr-ref-guide/modules/deployment-guide/pages/taking-solr-to-production.adoc +++ b/solr/solr-ref-guide/modules/deployment-guide/pages/taking-solr-to-production.adoc @@ -205,6 +205,8 @@ If the `status` command is not successful, look for error messages in `/var/solr Additionally, if you can't connect to running Solr from an external location, check your `SOLR_HOST_BIND` setting to make sure Solr is accessible to the network. +Once Solr is running, see xref:healthcheck.adoc[] for how to wire up a load balancer or orchestrator health probe, and how to confirm a collection is actually serving data rather than just that the node is up. + == Fine-Tune Your Production Setup === Dynamic Defaults for ConcurrentMergeScheduler
