consistent-hash and balance random select a natural target on the
ring, but may fall back to another server when that target cannot be
used.
This may happen after a failed attempt:
- option redispatch
- hash-balance-factor considers the target over quota
- hash-preserve-affinity maxconn/maxqueue considers it saturated
This is a signal we want to track as a standing operational metric.
This adds that visibility in two forms:
- a new "lbbypass" stats counter (show stat / CSV / Prometheus),
incremented atomically on both the bypassed server and the
backend whenever a bypass occurs, mirroring the existing
"retries"/"redispatches" counter pattern (struct
be_counters_shared_tg, ME_NEW_BE_SHARED stats column). Since this
struct is part of the shm-stats-file mapping, SHM_STATS_FILE_VER_MINOR
is bumped from 3 to 4.
- a new "txn.lb_bypass_reason" sample fetch, usable in log-format,
ACLs or set-var, returning one of "redispatch",
"hash-balance-factor", "preserve-affinity-maxconn" or
"preserve-affinity-maxqueue", or failing (undefined) when no
bypass occurred.
The reason is classified in chash_get_server_hash() (lb_chash.c):
the naive ring pick is saved before the existing eligibility-walk
loop runs, and if the final pick differs from it, the same four
conditions the loop already checks are re-evaluated once against the
naive pick to determine which one tripped. This happens inside the
same LBPRM_LOCK read-lock critical section the loop already uses, so
there is no new race window and no duplicated locking. The loop
itself is unmodified.
chash_get_server_hash() and the 8 get_server_{sh,uh,ph,ph_post,hh,
rch,expr,rnd}() wrappers in backend.c gain a trailing, NULL-safe
"enum lb_bypass_reason *reason_out" parameter. assign_server() threads
a local reason through whichever wrapper it calls and stores the
result on the new struct stream field "lb_bypass_reason", which the
sample fetch reads back. get_server_rnd() (balance random) draws
multiple times and keeps a prev/curr pair already; a parallel
prev_reason/curr_reason pair was added so the reported reason always
matches whichever draw is ultimately returned.
Two things are intentionally NOT covered by this patch:
- A naive target being skipped because it is in drain mode or
marked unhealthy. Such servers are proactively removed from the
consistent-hash ring itself (chash_dequeue_srv()) the moment
their state changes, so chash_get_server_hash() never sees them
as ring candidates and has nothing to compare against. Detecting
this would require maintaining a second, always-populated ring
including unhealthy servers, which is a larger, separate change.
- chash_get_next_server(), the no-hash-key fallback path used when
e.g. a configured header/cookie hash key is absent from the
request. It has no concept of a "naive target" to compare
against and only implements a maxconn-only skip; requests going
through it always report no bypass.
Both limitations are documented in doc/configuration.txt next to the
new sample fetch.
This is purely additive: no existing server-selection decision is
changed by this patch, only observed and reported.
---
doc/configuration.txt | 27 +++++++++++-
doc/management.txt | 3 ++
include/haproxy/backend-t.h | 14 ++++++
include/haproxy/backend.h | 24 +++++++----
include/haproxy/counters-t.h | 1 +
include/haproxy/lb_chash.h | 3 +-
include/haproxy/stats-file-t.h | 2 +-
include/haproxy/stats-t.h | 1 +
include/haproxy/stream-t.h | 1 +
src/backend.c | 79 ++++++++++++++++++++++------------
src/lb_chash.c | 46 +++++++++++++++++++-
src/log.c | 2 +-
src/stats-proxy.c | 1 +
src/stream.c | 27 ++++++++++++
14 files changed, 188 insertions(+), 43 deletions(-)
diff --git a/doc/configuration.txt b/doc/configuration.txt
index 419781bd7..2ae9442b9 100644
--- a/doc/configuration.txt
+++ b/doc/configuration.txt
@@ -8627,7 +8627,12 @@ hash-balance-factor <factor>
This setting is also used by "balance random" which internally relies on the
consistent hashing mechanism.
- See also : "balance" and "hash-type".
+ Each time a request is reassigned to another server because of this
+ setting, it is counted in the "lbbypass" stats counter (both on the
+ bypassed server and on the backend), and "txn.lb_bypass_reason" reports
+ "hash-balance-factor" for that request.
+
+ See also : "balance", "hash-type" and "txn.lb_bypass_reason".
hash-preserve-affinity { always | maxconn | maxqueue }
Specify a method for assigning streams to servers with hash load balancing
@@ -8657,7 +8662,14 @@ hash-preserve-affinity { always | maxconn | maxqueue }
picked by following the hashing ring. This has no effect on
servers that do not set both "maxconn" and "maxqueue".
- See also : "maxconn", "maxqueue", "hash-balance-factor"
+ Each time a request is reassigned to another server because of this
+ setting, it is counted in the "lbbypass" stats counter (both on the
+ bypassed server and on the backend), and "txn.lb_bypass_reason" reports
+ "preserve-affinity-maxconn" or "preserve-affinity-maxqueue" for that
+ request.
+
+ See also : "maxconn", "maxqueue", "hash-balance-factor" and
+ "txn.lb_bypass_reason".
hash-type <method> <function> <modifier>
Specify a method to use for mapping hashes to servers
@@ -25538,6 +25550,7 @@ src_updt_conn_cnt([<table>])
integer
srv_id integer
srv_name string
txn.conn_retries integer
+txn.lb_bypass_reason string
txn.redispatched boolean
-------------------------------------------------+-------------
@@ -26757,6 +26770,16 @@ txn.conn_retries : integer
connection is not fully established. For HTTP connections, the value may be
affected by L7 retries.
+txn.lb_bypass_reason : string
+ Returns the reason the hash/random load-balancing algorithm bypassed the
+ naturally hashed target server in favor of an alternate one, if any. One
+ of "redispatch", "hash-balance-factor", "preserve-affinity-maxconn", or
+ "preserve-affinity-maxqueue". The fetch fails (undefined) if no bypass
+ occurred for this request. Note that bypasses caused by the naive target
+ being in drain mode or marked unhealthy are not currently detected or
+ reported by this fetch. See also "hash-balance-factor" and
+ "hash-preserve-affinity".
+
txn.redispatched : boolean
Returns true if the connection has experienced redispatch upon retry
according
to "option redispatch" configuration. This value is subject to change while
diff --git a/doc/management.txt b/doc/management.txt
index 5d0a7322d..55e2117f5 100644
--- a/doc/management.txt
+++ b/doc/management.txt
@@ -1387,6 +1387,9 @@ Here is the list of static fields using the proxy
statistics domain:
process started
118. cache_hint_hits [.FB.]: cumulative number of cache hits having served
early hints
+ 119. lbbypass [..BS]: cumulative number of times the hash/random
load-balancing
+ algorithm bypassed the naturally hashed/drawn target server in favor of
+ an alternate one
For all other statistics domains, the presence or the order of the fields are
not guaranteed. In this case, the header line should always be used to parse
diff --git a/include/haproxy/backend-t.h b/include/haproxy/backend-t.h
index 53c3f8748..1c3825a32 100644
--- a/include/haproxy/backend-t.h
+++ b/include/haproxy/backend-t.h
@@ -134,6 +134,20 @@
/* various constants */
+/* Reasons why the hash/random load-balancing algorithm bypassed the naively
+ * computed ring target in favor of an alternate server. Drives the
+ * "lb_bypass" counters and the "txn.lb_bypass_reason" sample fetch. Keep in
+ * sync with lb_bypass_reason_str[] in src/stream.c.
+ */
+enum lb_bypass_reason {
+ LB_BYPASS_NONE = 0, /* no bypass: naive target was used */
+ LB_BYPASS_REDISPATCH, /* naive target == stream's failed
previous server ("avoid") */
+ LB_BYPASS_HASH_BALANCE_FACTOR, /* naive target over its
hash-balance-factor quota */
+ LB_BYPASS_AFFINITY_MAXCONN, /* naive target saturated,
hash-preserve-affinity maxconn */
+ LB_BYPASS_AFFINITY_MAXQUEUE, /* naive target's queue full,
hash-preserve-affinity maxqueue */
+ LB_BYPASS_REASONS /* not a real value; array bound */
+};
+
/* The scale factor between user weight and effective weight allows smooth
* weight modulation even with small weights (eg: 1). It should not be too high
* though because it limits the number of servers in FWRR mode in order to
diff --git a/include/haproxy/backend.h b/include/haproxy/backend.h
index 5d2cb657c..fd13d04c6 100644
--- a/include/haproxy/backend.h
+++ b/include/haproxy/backend.h
@@ -34,14 +34,22 @@ extern struct list lb_ops_list;
void lb_ops_register(struct lb_ops *ops);
-struct server *get_server_sh(struct proxy *px, const char *addr, int len,
const struct server *avoid);
-struct server *get_server_uh(struct proxy *px, char *uri, int uri_len, const
struct server *avoid);
-struct server *get_server_ph(struct proxy *px, const char *uri, int uri_len,
const struct server *avoid);
-struct server *get_server_ph_post(struct stream *s, const struct server
*avoid);
-struct server *get_server_hh(struct stream *s, const struct server *avoid);
-struct server *get_server_rch(struct stream *s, const struct server *avoid);
-struct server *get_server_expr(struct stream *s, const struct server *avoid);
-struct server *get_server_rnd(struct stream *s, const struct server *avoid);
+struct server *get_server_sh(struct proxy *px, const char *addr, int len,
const struct server *avoid,
+ enum lb_bypass_reason *reason_out);
+struct server *get_server_uh(struct proxy *px, char *uri, int uri_len, const
struct server *avoid,
+ enum lb_bypass_reason *reason_out);
+struct server *get_server_ph(struct proxy *px, const char *uri, int uri_len,
const struct server *avoid,
+ enum lb_bypass_reason *reason_out);
+struct server *get_server_ph_post(struct stream *s, const struct server *avoid,
+ enum lb_bypass_reason *reason_out);
+struct server *get_server_hh(struct stream *s, const struct server *avoid,
+ enum lb_bypass_reason *reason_out);
+struct server *get_server_rch(struct stream *s, const struct server *avoid,
+ enum lb_bypass_reason *reason_out);
+struct server *get_server_expr(struct stream *s, const struct server *avoid,
+ enum lb_bypass_reason *reason_out);
+struct server *get_server_rnd(struct stream *s, const struct server *avoid,
+ enum lb_bypass_reason *reason_out);
int assign_server(struct stream *s);
int assign_server_address(struct stream *s);
diff --git a/include/haproxy/counters-t.h b/include/haproxy/counters-t.h
index 85dacb3d2..5de87bebc 100644
--- a/include/haproxy/counters-t.h
+++ b/include/haproxy/counters-t.h
@@ -159,6 +159,7 @@ struct be_counters_shared_tg {
long long redispatches; /* retried and redispatched
connections (BE only) */
long long retries; /* retried and redispatched
connections (BE only) */
+ long long lb_bypass; /* times hash/random LB
bypassed the naive ring target (BE only) */
long long failed_resp; /* failed responses (BE only) */
long long failed_conns; /* failed connect() attempts
(BE only) */
unsigned int last_sess; /* last session time */
diff --git a/include/haproxy/lb_chash.h b/include/haproxy/lb_chash.h
index fd8ad111f..891cbad84 100644
--- a/include/haproxy/lb_chash.h
+++ b/include/haproxy/lb_chash.h
@@ -29,7 +29,8 @@
struct proxy;
struct server;
struct server *chash_get_next_server(struct proxy *p, struct server
*srvtoavoid);
-struct server *chash_get_server_hash(struct proxy *p, unsigned int hash, const
struct server *avoid);
+struct server *chash_get_server_hash(struct proxy *p, unsigned int hash, const
struct server *avoid,
+ enum lb_bypass_reason *reason_out);
#endif /* _HAPROXY_LB_CHASH_H */
diff --git a/include/haproxy/stats-file-t.h b/include/haproxy/stats-file-t.h
index 0343ca9fe..f20fd0581 100644
--- a/include/haproxy/stats-file-t.h
+++ b/include/haproxy/stats-file-t.h
@@ -15,7 +15,7 @@ enum stfile_domain {
};
#define SHM_STATS_FILE_VER_MAJOR 1
-#define SHM_STATS_FILE_VER_MINOR 3
+#define SHM_STATS_FILE_VER_MINOR 4
#define SHM_STATS_FILE_HEARTBEAT_TIMEOUT 60 /* passed this delay (seconds)
process which has not
* sent heartbeat will be
considered down
diff --git a/include/haproxy/stats-t.h b/include/haproxy/stats-t.h
index f381d5030..5eae0c562 100644
--- a/include/haproxy/stats-t.h
+++ b/include/haproxy/stats-t.h
@@ -499,6 +499,7 @@ enum stat_idx_px {
ST_I_PX_RES_IN,
ST_I_PX_RES_OUT,
ST_I_PX_CACHE_HINT_HITS,
+ ST_I_PX_LBBYPASS,
/* must always be the last one */
ST_I_PX_MAX
};
diff --git a/include/haproxy/stream-t.h b/include/haproxy/stream-t.h
index 24bc09fae..931eff58c 100644
--- a/include/haproxy/stream-t.h
+++ b/include/haproxy/stream-t.h
@@ -275,6 +275,7 @@ struct stream {
int conn_retries; /* number of connect retries performed
*/
unsigned int conn_exp; /* wake up time for connect, queue,
turn-around, ... */
unsigned int conn_err_type; /* first error detected, one of
STRM_ET_* */
+ unsigned int lb_bypass_reason; /* one of LB_BYPASS_* (backend-t.h),
set by assign_server() */
uint32_t rules_bcount; /* number of rules evaluated since last
yield */
diff --git a/src/backend.c b/src/backend.c
index 1c374676e..99920c1a0 100644
--- a/src/backend.c
+++ b/src/backend.c
@@ -178,7 +178,8 @@ void update_backend_weight(struct proxy *px)
* If any server is found, it will be returned. If no valid server is found,
* NULL is returned.
*/
-struct server *get_server_sh(struct proxy *px, const char *addr, int len,
const struct server *avoid)
+struct server *get_server_sh(struct proxy *px, const char *addr, int len,
const struct server *avoid,
+ enum lb_bypass_reason *reason_out)
{
unsigned int h, l;
@@ -203,7 +204,7 @@ struct server *get_server_sh(struct proxy *px, const char
*addr, int len, const
h = full_hash(h);
hash_done:
if ((px->lbprm.algo & BE_LB_LKUP) == BE_LB_LKUP_CHTREE)
- return chash_get_server_hash(px, h, avoid);
+ return chash_get_server_hash(px, h, avoid, reason_out);
else
return map_get_server_hash(px, h);
}
@@ -222,7 +223,8 @@ struct server *get_server_sh(struct proxy *px, const char
*addr, int len, const
* algorithm out of a tens because it gave him the best results.
*
*/
-struct server *get_server_uh(struct proxy *px, char *uri, int uri_len, const
struct server *avoid)
+struct server *get_server_uh(struct proxy *px, char *uri, int uri_len, const
struct server *avoid,
+ enum lb_bypass_reason *reason_out)
{
unsigned int hash = 0;
int c;
@@ -256,7 +258,7 @@ struct server *get_server_uh(struct proxy *px, char *uri,
int uri_len, const str
hash_done:
if ((px->lbprm.algo & BE_LB_LKUP) == BE_LB_LKUP_CHTREE)
- return chash_get_server_hash(px, hash, avoid);
+ return chash_get_server_hash(px, hash, avoid, reason_out);
else
return map_get_server_hash(px, hash);
}
@@ -270,7 +272,8 @@ struct server *get_server_uh(struct proxy *px, char *uri,
int uri_len, const str
* is returned. If any server is found, it will be returned. If no valid server
* is found, NULL is returned.
*/
-struct server *get_server_ph(struct proxy *px, const char *uri, int uri_len,
const struct server *avoid)
+struct server *get_server_ph(struct proxy *px, const char *uri, int uri_len,
const struct server *avoid,
+ enum lb_bypass_reason *reason_out)
{
unsigned int hash = 0;
const char *start, *end;
@@ -310,7 +313,7 @@ struct server *get_server_ph(struct proxy *px, const char
*uri, int uri_len, con
hash = gen_hash(px, start, (end - start));
if ((px->lbprm.algo & BE_LB_LKUP) ==
BE_LB_LKUP_CHTREE)
- return chash_get_server_hash(px, hash,
avoid);
+ return chash_get_server_hash(px, hash,
avoid, reason_out);
else
return map_get_server_hash(px, hash);
}
@@ -329,7 +332,8 @@ struct server *get_server_ph(struct proxy *px, const char
*uri, int uri_len, con
/*
* this does the same as the previous server_ph, but check the body contents
*/
-struct server *get_server_ph_post(struct stream *s, const struct server *avoid)
+struct server *get_server_ph_post(struct stream *s, const struct server *avoid,
+ enum lb_bypass_reason *reason_out)
{
unsigned int hash = 0;
struct channel *req = &s->req;
@@ -387,7 +391,7 @@ struct server *get_server_ph_post(struct stream *s, const
struct server *avoid)
hash = gen_hash(px, start, (end - start));
if ((px->lbprm.algo & BE_LB_LKUP) ==
BE_LB_LKUP_CHTREE)
- return chash_get_server_hash(px, hash,
avoid);
+ return chash_get_server_hash(px, hash,
avoid, reason_out);
else
return map_get_server_hash(px, hash);
}
@@ -414,7 +418,8 @@ struct server *get_server_ph_post(struct stream *s, const
struct server *avoid)
* is found, NULL is returned. When lbprm.arg_opt1 is set, the hash will only
* apply to the middle part of a domain name ("use_domain_only" option).
*/
-struct server *get_server_hh(struct stream *s, const struct server *avoid)
+struct server *get_server_hh(struct stream *s, const struct server *avoid,
+ enum lb_bypass_reason *reason_out)
{
unsigned int hash = 0;
struct proxy *px = s->be;
@@ -481,13 +486,14 @@ struct server *get_server_hh(struct stream *s, const
struct server *avoid)
hash_done:
if ((px->lbprm.algo & BE_LB_LKUP) == BE_LB_LKUP_CHTREE)
- return chash_get_server_hash(px, hash, avoid);
+ return chash_get_server_hash(px, hash, avoid, reason_out);
else
return map_get_server_hash(px, hash);
}
/* RDP Cookie HASH. */
-struct server *get_server_rch(struct stream *s, const struct server *avoid)
+struct server *get_server_rch(struct stream *s, const struct server *avoid,
+ enum lb_bypass_reason *reason_out)
{
unsigned int hash = 0;
struct proxy *px = s->be;
@@ -524,7 +530,7 @@ struct server *get_server_rch(struct stream *s, const
struct server *avoid)
hash_done:
if ((px->lbprm.algo & BE_LB_LKUP) == BE_LB_LKUP_CHTREE)
- return chash_get_server_hash(px, hash, avoid);
+ return chash_get_server_hash(px, hash, avoid, reason_out);
else
return map_get_server_hash(px, hash);
}
@@ -532,7 +538,8 @@ struct server *get_server_rch(struct stream *s, const
struct server *avoid)
/* sample expression HASH. Returns NULL if the sample is not found or if there
* are no server, relying on the caller to fall back to round robin instead.
*/
-struct server *get_server_expr(struct stream *s, const struct server *avoid)
+struct server *get_server_expr(struct stream *s, const struct server *avoid,
+ enum lb_bypass_reason *reason_out)
{
struct proxy *px = s->be;
struct sample *smp;
@@ -563,28 +570,34 @@ struct server *get_server_expr(struct stream *s, const
struct server *avoid)
hash_done:
if ((px->lbprm.algo & BE_LB_LKUP) == BE_LB_LKUP_CHTREE)
- return chash_get_server_hash(px, hash, avoid);
+ return chash_get_server_hash(px, hash, avoid, reason_out);
else
return map_get_server_hash(px, hash);
}
/* random value */
-struct server *get_server_rnd(struct stream *s, const struct server *avoid)
+struct server *get_server_rnd(struct stream *s, const struct server *avoid,
+ enum lb_bypass_reason *reason_out)
{
unsigned int hash = 0;
struct proxy *px = s->be;
struct server *prev, *curr;
+ enum lb_bypass_reason prev_reason, curr_reason = LB_BYPASS_NONE;
int draws = px->lbprm.arg_opt1; // number of draws
/* tot_weight appears to mean srv_count */
- if (px->lbprm.tot_weight == 0)
+ if (px->lbprm.tot_weight == 0) {
+ if (reason_out)
+ *reason_out = LB_BYPASS_NONE;
return NULL;
+ }
curr = NULL;
do {
prev = curr;
+ prev_reason = curr_reason;
hash = statistical_prng();
- curr = chash_get_server_hash(px, hash, avoid);
+ curr = chash_get_server_hash(px, hash, avoid, &curr_reason);
if (!curr)
break;
@@ -595,14 +608,18 @@ struct server *get_server_rnd(struct stream *s, const
struct server *avoid)
uint64_t wcurr = (uint64_t)curr->served *
prev->cur_eweight;
uint64_t wprev = (uint64_t)prev->served *
curr->cur_eweight;
- if (wcurr > wprev)
+ if (wcurr > wprev) {
curr = prev;
+ curr_reason = prev_reason;
+ }
else if (wcurr == wprev && curr->counters.shared.tg &&
prev->counters.shared.tg) {
/* same load: pick the lowest weighted request
rate */
wcurr =
read_freq_ctr_period_estimate(&curr->counters.shared.tg[tgid -
1]->sess_per_sec, MS_TO_TICKS(1000));
wprev =
read_freq_ctr_period_estimate(&prev->counters.shared.tg[tgid -
1]->sess_per_sec, MS_TO_TICKS(1000));
- if (wprev * curr->cur_eweight < wcurr *
prev->cur_eweight)
+ if (wprev * curr->cur_eweight < wcurr *
prev->cur_eweight) {
curr = prev;
+ curr_reason = prev_reason;
+ }
}
}
} while (--draws > 0);
@@ -614,6 +631,9 @@ struct server *get_server_rnd(struct stream *s, const
struct server *avoid)
(curr->queueslength || (curr->maxconn && curr->served >=
srv_dynamic_maxconn(curr))))
curr = NULL;
+ if (reason_out)
+ *reason_out = curr ? curr_reason : LB_BYPASS_NONE;
+
return curr;
}
@@ -647,12 +667,14 @@ int assign_server(struct stream *s)
struct connection *conn = NULL;
struct server *conn_slot;
struct server *srv = NULL, *prev_srv;
+ enum lb_bypass_reason bypass_reason = LB_BYPASS_NONE;
int err;
err = SRV_STATUS_INTERNAL;
if (unlikely(s->pend_pos || s->flags & SF_ASSIGNED))
goto out_err;
+ s->lb_bypass_reason = LB_BYPASS_NONE;
prev_srv = objt_server(s->target);
conn_slot = s->srv_conn;
@@ -734,7 +756,7 @@ int assign_server(struct stream *s)
if ((s->be->lbprm.algo & BE_LB_KIND) == BE_LB_KIND_RR) {
/* static-rr (map) or random (chash) */
if ((s->be->lbprm.algo & BE_LB_PARM) ==
BE_LB_RR_RANDOM)
- srv = get_server_rnd(s, prev_srv);
+ srv = get_server_rnd(s, prev_srv,
&bypass_reason);
else
srv = map_get_server_rr(s->be,
prev_srv);
break;
@@ -753,12 +775,12 @@ int assign_server(struct stream *s)
if (src && src->ss_family == AF_INET) {
srv = get_server_sh(s->be,
(void *)&((struct
sockaddr_in *)src)->sin_addr,
- 4, prev_srv);
+ 4, prev_srv,
&bypass_reason);
}
else if (src && src->ss_family == AF_INET6) {
srv = get_server_sh(s->be,
(void *)&((struct
sockaddr_in6 *)src)->sin6_addr,
- 16, prev_srv);
+ 16, prev_srv,
&bypass_reason);
}
break;
@@ -776,7 +798,7 @@ int assign_server(struct stream *s)
if (!isttest(uri))
uri = ist("");
}
- srv = get_server_uh(s->be, uri.ptr,
uri.len, prev_srv);
+ srv = get_server_uh(s->be, uri.ptr,
uri.len, prev_srv, &bypass_reason);
}
break;
@@ -786,27 +808,27 @@ int assign_server(struct stream *s)
struct ist uri;
uri =
htx_sl_req_uri(http_get_stline(htxbuf(&s->req.buf)));
- srv = get_server_ph(s->be, uri.ptr,
uri.len, prev_srv);
+ srv = get_server_ph(s->be, uri.ptr,
uri.len, prev_srv, &bypass_reason);
if (!srv && s->txn.http->meth ==
HTTP_METH_POST)
- srv = get_server_ph_post(s,
prev_srv);
+ srv = get_server_ph_post(s,
prev_srv, &bypass_reason);
}
break;
case BE_LB_HASH_HDR:
/* Header Parameter hashing */
if (IS_HTX_STRM(s) &&
s->txn.http->req.msg_state >= HTTP_MSG_BODY)
- srv = get_server_hh(s, prev_srv);
+ srv = get_server_hh(s, prev_srv,
&bypass_reason);
break;
case BE_LB_HASH_RDP:
/* RDP Cookie hashing */
- srv = get_server_rch(s, prev_srv);
+ srv = get_server_rch(s, prev_srv,
&bypass_reason);
break;
case BE_LB_HASH_SMP:
/* sample expression hashing */
- srv = get_server_expr(s, prev_srv);
+ srv = get_server_expr(s, prev_srv,
&bypass_reason);
break;
default:
@@ -852,6 +874,7 @@ int assign_server(struct stream *s)
if (srv->counters.shared.tg)
_HA_ATOMIC_INC(&srv->counters.shared.tg[tgid -
1]->cum_lbconn);
}
+ s->lb_bypass_reason = bypass_reason;
stream_set_srv_target(s, srv);
}
else {
diff --git a/src/lb_chash.c b/src/lb_chash.c
index 12a933a21..2f9fecfc8 100644
--- a/src/lb_chash.c
+++ b/src/lb_chash.c
@@ -19,11 +19,13 @@
#include <import/eb32tree.h>
#include <haproxy/api.h>
#include <haproxy/backend.h>
+#include <haproxy/counters.h>
#include <haproxy/errors.h>
#include <haproxy/guid.h>
#include <haproxy/proxy.h>
#include <haproxy/queue.h>
#include <haproxy/server.h>
+#include <haproxy/thread.h>
#include <haproxy/tools.h>
#include <haproxy/xxhash.h>
@@ -411,17 +413,28 @@ static int chash_server_is_eligible(struct server *s)
* It will also skip server <avoid> if the hash result ends on this one.
* If no valid server is found, NULL is returned.
*
+ * If <reason_out> is not NULL, it is always set, either to LB_BYPASS_NONE if
+ * the naively hashed server was used as-is, or to the reason why it had to be
+ * skipped in favor of another one on the ring (redispatch, hash-balance-factor
+ * or hash-preserve-affinity). Servers skipped because they are down or in
+ * drain mode are not reported through <reason_out>, as they never appear as
+ * ring candidates in the first place.
+ *
* The lbprm's lock will be used in R/O mode. The server's lock is not used.
*/
-struct server *chash_get_server_hash(struct proxy *p, unsigned int hash, const
struct server *avoid)
+struct server *chash_get_server_hash(struct proxy *p, unsigned int hash, const
struct server *avoid,
+ enum lb_bypass_reason *reason_out)
{
struct eb32_node *next, *prev;
- struct server *nsrv, *psrv;
+ struct server *nsrv, *psrv, *naive_srv;
struct eb_root *root;
unsigned int dn, dp;
int loop;
int hashafnty;
+ if (reason_out)
+ *reason_out = LB_BYPASS_NONE;
+
HA_RWLOCK_RDLOCK(LBPRM_LOCK, &p->lbprm.lock);
if (p->srv_act)
@@ -465,6 +478,8 @@ struct server *chash_get_server_hash(struct proxy *p,
unsigned int hash, const s
nsrv = psrv;
}
+ naive_srv = nsrv;
+
loop = 0;
hashafnty = p->options3 & PR_O3_HASHAFNTY_MASK;
@@ -486,6 +501,33 @@ struct server *chash_get_server_hash(struct proxy *p,
unsigned int hash, const s
nsrv = eb32_entry(next, struct tree_occ, node)->server;
}
+ if (nsrv != naive_srv) {
+ enum lb_bypass_reason reason;
+
+ if (naive_srv == avoid)
+ reason = LB_BYPASS_REDISPATCH;
+ else if (p->lbprm.hash_balance_factor &&
!chash_server_is_eligible(naive_srv))
+ reason = LB_BYPASS_HASH_BALANCE_FACTOR;
+ else if (hashafnty == PR_O3_HASHAFNTY_MAXCONN &&
naive_srv->maxconn &&
+ naive_srv->served >= srv_dynamic_maxconn(naive_srv))
+ reason = LB_BYPASS_AFFINITY_MAXCONN;
+ else if (hashafnty == PR_O3_HASHAFNTY_MAXQUEUE &&
naive_srv->maxconn && naive_srv->maxqueue &&
+ naive_srv->served + naive_srv->queueslength >=
+ srv_dynamic_maxconn(naive_srv) + naive_srv->maxqueue)
+ reason = LB_BYPASS_AFFINITY_MAXQUEUE;
+ else
+ reason = LB_BYPASS_NONE; /* defensive; the loop above
guarantees one clause matched */
+
+ if (reason != LB_BYPASS_NONE) {
+ if (p->be_counters.shared.tg)
+ _HA_ATOMIC_INC(&p->be_counters.shared.tg[tgid -
1]->lb_bypass);
+ if (naive_srv->counters.shared.tg)
+
_HA_ATOMIC_INC(&naive_srv->counters.shared.tg[tgid - 1]->lb_bypass);
+ }
+ if (reason_out)
+ *reason_out = reason;
+ }
+
out:
HA_RWLOCK_RDUNLOCK(LBPRM_LOCK, &p->lbprm.lock);
return nsrv;
diff --git a/src/log.c b/src/log.c
index 7e2ee16ec..1a41c31e5 100644
--- a/src/log.c
+++ b/src/log.c
@@ -3165,7 +3165,7 @@ static inline void __do_send_log_backend(struct proxy
*be, struct log_header hdr
unsigned int hash;
hash = statistical_prng(); /* random */
- srv = chash_get_server_hash(be, hash, NULL);
+ srv = chash_get_server_hash(be, hash, NULL, NULL);
}
else if ((be->lbprm.algo & BE_LB_ALGO) == BE_LB_ALGO_LH) {
struct sample result;
diff --git a/src/stats-proxy.c b/src/stats-proxy.c
index 862c79319..c4becf615 100644
--- a/src/stats-proxy.c
+++ b/src/stats-proxy.c
@@ -94,6 +94,7 @@ const struct stat_col stat_cols_px[ST_I_PX_MAX] = {
[ST_I_PX_ERESP] = ME_NEW_BE_SHARED("eresp",
"response_errors_total", FN_COUNTER, FF_U64, failed_resp,
STATS_PX_CAP___BS, "Total number of invalid responses since the worker
process started"),
[ST_I_PX_WRETR] = ME_NEW_BE_SHARED("wretr",
"retry_warnings_total", FN_COUNTER, FF_U64, retries,
STATS_PX_CAP___BS, "Total number of server connection retries since the
worker process started"),
[ST_I_PX_WREDIS] = ME_NEW_BE_SHARED("wredis",
"redispatch_warnings_total", FN_COUNTER, FF_U64, redispatches,
STATS_PX_CAP___BS, "Total number of server redispatches due to connection
failures since the worker process started"),
+ [ST_I_PX_LBBYPASS] = ME_NEW_BE_SHARED("lbbypass",
"lb_bypass_total", FN_COUNTER, FF_U64, lb_bypass,
STATS_PX_CAP___BS, "Total number of times the hash/random load-balancing
algorithm bypassed the naive hash target in favor of an alternate server"),
[ST_I_PX_STATUS] = { .name = "status",
.alt_name = "status", .desc =
"Frontend/listen status: OPEN/WAITING/FULL/STOP; backend: UP/DOWN; server: last
check status", .cap = STATS_PX_CAP_LFBS },
[ST_I_PX_WEIGHT] = { .name = "weight",
.alt_name = "weight", .desc = "Server's
effective weight, or sum of active servers' effective weights for a backend",
.cap = STATS_PX_CAP___BS },
[ST_I_PX_ACT] = { .name = "act",
.alt_name = "active_servers", .desc = "Total number
of active UP servers with a non-zero weight", .cap = STATS_PX_CAP___BS },
diff --git a/src/stream.c b/src/stream.c
index 4c425230f..16fe1b6eb 100644
--- a/src/stream.c
+++ b/src/stream.c
@@ -4892,6 +4892,32 @@ static int smp_fetch_redispatched(const struct arg
*args, struct sample *smp, co
return 1;
}
+/* indexed by enum lb_bypass_reason; LB_BYPASS_NONE is never looked up, see
+ * smp_fetch_lb_bypass_reason() below.
+ */
+static const char *const lb_bypass_reason_str[LB_BYPASS_REASONS] = {
+ [LB_BYPASS_REDISPATCH] = "redispatch",
+ [LB_BYPASS_HASH_BALANCE_FACTOR] = "hash-balance-factor",
+ [LB_BYPASS_AFFINITY_MAXCONN] = "preserve-affinity-maxconn",
+ [LB_BYPASS_AFFINITY_MAXQUEUE] = "preserve-affinity-maxqueue",
+};
+
+static int smp_fetch_lb_bypass_reason(const struct arg *args, struct sample
*smp, const char *km, void *private)
+{
+ smp->flags = SMP_F_VOL_TXN;
+ smp->data.type = SMP_T_STR;
+ if (!smp->strm)
+ return 0;
+ if (smp->strm->lb_bypass_reason == LB_BYPASS_NONE ||
+ smp->strm->lb_bypass_reason >= LB_BYPASS_REASONS)
+ return 0;
+
+ smp->data.u.str.area = (char
*)lb_bypass_reason_str[smp->strm->lb_bypass_reason];
+ smp->data.u.str.data = strlen(smp->data.u.str.area);
+ smp->flags |= SMP_F_CONST;
+ return 1;
+}
+
/* Note: must not be declared <const> as its list will be overwritten.
* Please take care of keeping this list alphabetically sorted.
*/
@@ -4909,6 +4935,7 @@ static struct sample_fetch_kw_list smp_kws = {ILH, {
{ "term_events", smp_fetch_tevts, 0, NULL,
SMP_T_STR, SMP_USE_INTRN, },
{ "txn.conn_retries", smp_fetch_conn_retries, 0, NULL,
SMP_T_SINT, SMP_USE_L4SRV, },
{ "txn.id32", smp_fetch_id32, 0, NULL,
SMP_T_SINT, SMP_USE_INTRN, },
+ { "txn.lb_bypass_reason",smp_fetch_lb_bypass_reason, 0, NULL,
SMP_T_STR, SMP_USE_L4SRV, },
{ "txn.redispatched", smp_fetch_redispatched, 0, NULL,
SMP_T_BOOL, SMP_USE_L4SRV, },
{ "txn.sess_term_state",smp_fetch_sess_term_state, 0, NULL,
SMP_T_STR, SMP_USE_INTRN, },
{ "waiting_entity", smp_fetch_waiting_entity, 0, NULL,
SMP_T_STR, SMP_USE_INTRN, },
--
2.52.0
