Re: Github PR: Fix typo in description of `-st` parameter

[Willy Tarreau]

Hi Jorrit,

On Tue, Oct 25, 2016 at 10:58:54PM +, Lukas Tribus wrote:
> Author: Jorrit Schippers 
> Number of patches: 1
> Patch titles: 
> Fix typo in description of `-st` parameter
> 
> Link: https://github.com/haproxy/haproxy/pull/65

Patch adjusted and merged. Please in the future check CONTRIBUTING and
put your description in the commit message itself.

Thanks!
Willy

Github PR: Fix typo in description of `-st` parameter

[Lukas Tribus]

Dear list!

This is an automated relay of the Github pull request: Fix typo in description 
of `-st` parameter

Author: Jorrit Schippers 
Number of patches: 1
Patch titles: 
Fix typo in description of `-st` parameter

Link: https://github.com/haproxy/haproxy/pull/65
Edit locally: wget https://github.com/haproxy/haproxy/pull/65.patch && vi 
65.patch
Apply locally: curl https://github.com/haproxy/haproxy/pull/65.patch | git am -
Description: The description of the `-sf` parameter seems to have been used as 
the source of the description of the `-st` parameter. The word wait from the 
`-sf` description was not removed, creating an invalid sentence.

This github pull request will be closed automatically; patch should be reviewed 
on this list.

Re: [PATCH] DOC: fix the entry for hash-balance-factor config option

[Willy Tarreau]

On Tue, Oct 25, 2016 at 05:04:12PM -0400, Andrew Rodland wrote:
> It was accidentally added as "balance-factor". Fix it and
> re-alphabetize.

merged, thanks Andrew!

Willy

[ANNOUNCE] haproxy-1.7-dev5

[Willy Tarreau]

Hi,

HAProxy 1.7-dev5 was released on 2016/10/25. It added 65 new commits
after version 1.7-dev4.

Things have been calming down since last release, which is quite good.
Among the changes I've noticed when preparing this release, the following
ones caught my attention:
  - minimum supported OpenBSD version increased from 3.0 to 5.7 for the
"openbsd" make target. This brings accept4().

  - API change for 51degrees, if you use it on the development version you
will have to read the notes to know how to upgrade your lib to the
newest version (or switch back to 1.6-stable which preserved the
compatibility with your current version)

  - (hopefully) much safer and better architected DNS response parser

  - long-awaited "tcp-request session" rule-set (just reminds me that
I forgot to update the ROADMAP file to mark it done). In short it
will mostly be used with track-sc actions when the frontend uses
the PROXY protocol.

  - "noreuseport" option on bind lines to disable SO_REUSEPORT. May
sometimes be helpful.

  - logs now ignore the idle and handshake times by default so that
the request time really represents the time from the first char
to the complete request. Idle and handshake times are available
separately.

  - upgraded peers protocol to 2.1 (backwards compatible) to support
initial synchronization with expiration dates. It will solve the
problem that fast-reloaders are seeing with keys that never expire.

  - bounded-load consistent hashing : limits the load on the target
server by spreading excess requests to neighbours. Andrew, I just
noticed that you documented "balance-factor" instead of
"hash-balance-factor", care to send me an updated patch ? (don't
forget to move the block to maintain alphabetical order). Thanks.

  - CLI keyword registration : does nothing for now but will make it
possible to spread the various keywords to their respective files
as we did with sample fetches and converters, instead of constantly
overloading dumpstats.c.

  - on the CLI we can now change a server's address, port and check port
at run time. Some related work is still in progress.

  - IP_BIND_ADDRESS_NO_PORT is used when setting the source address of
outgoing connections on systems where it's supported (Linux >= 4.2).

  - ah and the systemd saga patches to fix the zombie listening processes
during fast reloads.

Now let's stop speaking about the past and speak about the future instead.

First we may still have a few pending bugs. I got a private report about
a few zombie processes across a reload without systemd where each process
used to have CLOSE_WAIT connections on the peers port, so we seem to still
have an issue there. Reports are more than welcome.

Second, I know that we still have a few devs in progress. Baptiste is
finishing some really interesting changes to allow to start with a down
server that doesn't resolve that can be enabled at run time. I think everyone
understand what this allows to do. We should be able to start creating farms
with pre-provisionned servers that are configured on the fly so that many
users don't have to reload the process all the time.

Third, I used to have in my "list to santa claus" some drafts about a
protocol making it possible to let haproxy communicate with external
components to retrieve some info. The idea started with the problems
caused by most ldap libs not working fine in event-driven systems
(often at least the connect() is blocking). The concept would be to
reuse the co-sockets that Thierry created for the Lua network
communication for external validation. We can imagine sample fetches,
converters, actions, various things working more or less like an RPC,
with a standardized protocol that servers can easily implement without
getting their fingers dirty inside haproxy. Sort of a light ICAP. And
the good thing is that Christopher who did all the filters joined us
and accepted the challenge to try to reuse his filter infrastructure
to get this done for this release. Yeah Chris, feel the pressure now :-)

We won't be making an airport control tower first but something extensible,
so the initial design is more important than the functional coverage. I
still intend to release by the end of this month *if possible*, and given
that the end of this month is early next week, it's likely that it may
slip a little bit. But "a little bit" doesn't mean "let's take this
opportunity to make the first implementation even better" but "let's
ensure the first version works well enough". Obviously I'll take more
care of a well-designed protocol than an exact release date, but for
me the days of 1.5-dev are long dead and I'd rather release without
it that release 2 months late.

Fourth, while integrating the various patches I received recently, I
noticed that we still have a lot of ugly stuff in the code. If some
people are interested in contributing some time and/or code without
knowing whe

[PATCH] DOC: fix the entry for hash-balance-factor config option

[Andrew Rodland]

It was accidentally added as "balance-factor". Fix it and
re-alphabetize.

Signed-off-by: Andrew Rodland 
---
 doc/configuration.txt | 54 +--
 1 file changed, 27 insertions(+), 27 deletions(-)

diff --git a/doc/configuration.txt b/doc/configuration.txt
index a93c103..793c79e 100644
--- a/doc/configuration.txt
+++ b/doc/configuration.txt
@@ -2205,32 +2205,6 @@ balance url_param  [check_post]
   See also : "dispatch", "cookie", "transparent", "hash-type" and "http_proxy".
 
 
-balance-factor 
-  Specify the balancing factor for bounded-load consistent hashing
-  May be used in sections :   defaults | frontend | listen | backend
- yes   |no|   no   |   yes
-  Arguments :
- is the control for the maximum number of concurrent requests to
- send to a server, expressed as a percentage of the average number
-of concurrent requests across all of the active servers.
-
-  Specifying a "balance-factor" for a server with "hash-type consistent"
-  enables an algorithm that prevents any one server from getting too many
-  requests at once, even if some hash buckets receive many more requests than
-  others. Setting  to 0 (the default) disables the feature. Otherwise,
-   is a percentage greater than 100. For example, if  is 150,
-  then no server will be allowed to have a load more than 1.5 times the 
average.
-  If server weights are used, they will be respected.
-
-  If the first-choice server is disqualified, the algorithm will choose another
-  server based on the request hash, until a server with additional capacity is
-  found. A higher  allows more imbalance between the servers, while a
-  lower  means that more servers will be checked on average, affecting
-  performance. Reasonable values are from 125 to 200.
-
-  See also : "balance" and "hash-type".
-
-
 bind []: [, ...] [param*]
 bind / [, ...] [param*]
   Define one or several listening addresses and/or ports in a frontend.
@@ -3300,6 +3274,32 @@ grace 
   simplify it.
 
 
+hash-balance-factor 
+  Specify the balancing factor for bounded-load consistent hashing
+  May be used in sections :   defaults | frontend | listen | backend
+ yes   |no|   no   |   yes
+  Arguments :
+ is the control for the maximum number of concurrent requests to
+ send to a server, expressed as a percentage of the average number
+of concurrent requests across all of the active servers.
+
+  Specifying a "hash-balance-factor" for a server with "hash-type consistent"
+  enables an algorithm that prevents any one server from getting too many
+  requests at once, even if some hash buckets receive many more requests than
+  others. Setting  to 0 (the default) disables the feature. Otherwise,
+   is a percentage greater than 100. For example, if  is 150,
+  then no server will be allowed to have a load more than 1.5 times the 
average.
+  If server weights are used, they will be respected.
+
+  If the first-choice server is disqualified, the algorithm will choose another
+  server based on the request hash, until a server with additional capacity is
+  found. A higher  allows more imbalance between the servers, while a
+  lower  means that more servers will be checked on average, affecting
+  performance. Reasonable values are from 125 to 200.
+
+  See also : "balance" and "hash-type".
+
+
 hash-type   
   Specify a method to use for mapping hashes to servers
   May be used in sections :   defaults | frontend | listen | backend
@@ -3384,7 +3384,7 @@ hash-type   
   default function is "sdbm", the selection of a function should be based on
   the range of the values being hashed.
 
-  See also : "balance", "balance-factor", "server"
+  See also : "balance", "hash-balance-factor", "server"
 
 
 http-check disable-on-404
-- 
2.9.3

Re: [PATCH] MINOR: build: Allow linking to device-atlas library file

[Willy Tarreau]

Hi David,

On Mon, Oct 10, 2016 at 02:31:03PM +0100, David CARLIER wrote:
> Hi willy,
> 
> I think that would be useful to backport it, even when 1.7 will be
> out, the 1.6 branch will be still used.

Oops sorry I missed your e-mail and discovered that I hadn't pushed
Bertrand's patch that I already merged. Now done and backported.

Cheers,
Willy

Re: [PATCH 0/4] Consistent hashing with bounded loads

[Willy Tarreau]

Hi Andrew,

On Tue, Oct 25, 2016 at 12:47:17PM -0400, Andrew Rodland wrote:
> Here's a resend of my previous patch, incorporating some changes suggested by 
> Willy. As my mailer ate my patches last time around, and I'm not completely 
> sure that it won't do it again, I'm attaching them to this email as well as 
> sending them inline in the following emails.

That's perfect, I've just applied your whole series. Thanks for polishing
it that fast :-)

For those interested, having tested it I can say it works remarkably well,
and fills the gap between consistent hashing and static hashing in that
the main criticism against consistent hashing often is its poor balancing
characteristics. Here it can smoothly pick another server to maintain the
balancing, I expect good results on large content caches (eg: video).

Cheers,
Willy

[PATCH 4/4] MEDIUM: server: Implement bounded-load hash algorithm

[Andrew Rodland]

The consistent hash lookup is done as normal, then if balancing is
enabled, we progress through the hash ring until we find a server that
doesn't have "too much" load. In the case of equal weights for all
servers, the allowed number of requests for a server is either the
floor or the ceil of (num_requests * hash-balance-factor / num_servers);
with unequal weights things are somewhat more complicated, but the
spirit is the same -- a server should not be able to go too far above
(its relative weight times) the average load. Using the hash ring to
make the second/third/etc. choice maintains as much locality as
possible given the load limit.

Signed-off-by: Andrew Rodland 
---
 src/lb_chash.c | 47 ++-
 1 file changed, 46 insertions(+), 1 deletion(-)

diff --git a/src/lb_chash.c b/src/lb_chash.c
index a62dfb5..84a2ef3 100644
--- a/src/lb_chash.c
+++ b/src/lb_chash.c
@@ -242,6 +242,34 @@ static void chash_update_server_weight(struct server *srv)
 }
 
 /*
+ * This function implements the "Consistent Hashing with Bounded Loads" 
algorithm
+ * of Mirrokni, Thorup, and Zadimoghaddam (arxiv:1608.01350), adapted for use 
with
+ * unequal server weights.
+ */
+int chash_server_is_eligible(struct server *s)
+{
+   /* The total number of slots to allocate is the total number of 
outstanding requests
+* (including the one we're about to make) times the 
load-balance-factor, rounded up.
+*/
+   unsigned tot_slots = ((s->proxy->served + 1) * 
s->proxy->lbprm.chash.balance_factor + 99) / 100;
+   unsigned slots_per_weight = tot_slots / s->proxy->lbprm.tot_weight;
+   unsigned remainder = tot_slots % s->proxy->lbprm.tot_weight;
+
+   /* Allocate a whole number of slots per weight unit... */
+   unsigned slots = s->eweight * slots_per_weight;
+
+   /* And then distribute the rest among servers proportionally to their 
weight. */
+   slots += ((s->cumulative_weight + s->eweight) * remainder) / 
s->proxy->lbprm.tot_weight
+   - (s->cumulative_weight * remainder) / 
s->proxy->lbprm.tot_weight;
+
+   /* But never leave a server with 0. */
+   if (slots == 0)
+   slots = 1;
+
+   return s->served < slots;
+}
+
+/*
  * This function returns the running server from the CHASH tree, which is at
  * the closest distance from the value of . Doing so ensures that even
  * with a well imbalanced hash, if some servers are close to each other, they
@@ -254,6 +282,7 @@ struct server *chash_get_server_hash(struct proxy *p, 
unsigned int hash)
struct server *nsrv, *psrv;
struct eb_root *root;
unsigned int dn, dp;
+   int loop;
 
if (p->srv_act)
root = &p->lbprm.chash.act;
@@ -287,7 +316,23 @@ struct server *chash_get_server_hash(struct proxy *p, 
unsigned int hash)
dp = hash - prev->key;
dn = next->key - hash;
 
-   return (dp <= dn) ? psrv : nsrv;
+   if (dp <= dn) {
+   next = prev;
+   nsrv = psrv;
+   }
+
+   loop = 0;
+   while (p->lbprm.chash.balance_factor && 
!chash_server_is_eligible(nsrv)) {
+   next = eb32_next(next);
+   if (!next) {
+   next = eb32_first(root);
+   if (++loop > 1) // protection against accidental loop
+   break;
+   }
+   nsrv = eb32_entry(next, struct tree_occ, node)->server;
+   }
+
+   return nsrv;
 }
 
 /* Return next server from the CHASH tree in backend . If the tree is empty,
-- 
2.9.3

[PATCH 3/4] MINOR: server: compute a "cumulative weight" to allow chash balancing to hit its target

[Andrew Rodland]

For active servers, this is the sum of the eweights of all active
servers before this one in the backend, and
[srv->cumulative_weight .. srv_cumulative_weight + srv_eweight) is a
space occupied by this server in the range [0 .. lbprm.tot_wact), and
likewise for backup servers with tot_wbck. This allows choosing a
server or a range of servers proportional to their weight, by simple
integer comparison.

Signed-off-by: Andrew Rodland 
---
 include/types/server.h | 1 +
 src/backend.c  | 2 ++
 2 files changed, 3 insertions(+)

diff --git a/include/types/server.h b/include/types/server.h
index ce13820..5d89212 100644
--- a/include/types/server.h
+++ b/include/types/server.h
@@ -203,6 +203,7 @@ struct server {
unsigned wscore;/* weight score, used during 
srv map computation */
unsigned prev_eweight;  /* eweight before last change */
unsigned rweight;   /* remainer of weight in the 
current LB tree */
+   unsigned cumulative_weight; /* weight of servers prior to 
this one in the same group, for chash balancing */
unsigned npos, lpos;/* next and last positions in 
the LB tree */
struct eb32_node lb_node;   /* node used for tree-based 
load balancing */
struct eb_root *lb_tree;/* we want to know in what tree 
the server is */
diff --git a/src/backend.c b/src/backend.c
index faf872c..573f054 100644
--- a/src/backend.c
+++ b/src/backend.c
@@ -115,9 +115,11 @@ void recount_servers(struct proxy *px)
!(px->options & PR_O_USE_ALL_BK))
px->lbprm.fbck = srv;
px->srv_bck++;
+   srv->cumulative_weight = px->lbprm.tot_wbck;
px->lbprm.tot_wbck += srv->eweight;
} else {
px->srv_act++;
+   srv->cumulative_weight = px->lbprm.tot_wact;
px->lbprm.tot_wact += srv->eweight;
}
}
-- 
2.9.3

[PATCH 2/4] MINOR: backend: add hash-balance-factor option for hash-type consistent

[Andrew Rodland]

0 will mean no balancing occurs; otherwise it represents the ratio
between the highest-loaded server and the average load, times 100 (i.e.
a value of 150 means a 1.5x ratio), assuming equal weights.

Signed-off-by: Andrew Rodland 
---
 doc/configuration.txt| 28 +++-
 include/types/lb_chash.h |  1 +
 src/cfgparse.c   | 15 +++
 3 files changed, 43 insertions(+), 1 deletion(-)

diff --git a/doc/configuration.txt b/doc/configuration.txt
index d047524..a93c103 100644
--- a/doc/configuration.txt
+++ b/doc/configuration.txt
@@ -2205,6 +2205,32 @@ balance url_param  [check_post]
   See also : "dispatch", "cookie", "transparent", "hash-type" and "http_proxy".
 
 
+balance-factor 
+  Specify the balancing factor for bounded-load consistent hashing
+  May be used in sections :   defaults | frontend | listen | backend
+ yes   |no|   no   |   yes
+  Arguments :
+ is the control for the maximum number of concurrent requests to
+ send to a server, expressed as a percentage of the average number
+of concurrent requests across all of the active servers.
+
+  Specifying a "balance-factor" for a server with "hash-type consistent"
+  enables an algorithm that prevents any one server from getting too many
+  requests at once, even if some hash buckets receive many more requests than
+  others. Setting  to 0 (the default) disables the feature. Otherwise,
+   is a percentage greater than 100. For example, if  is 150,
+  then no server will be allowed to have a load more than 1.5 times the 
average.
+  If server weights are used, they will be respected.
+
+  If the first-choice server is disqualified, the algorithm will choose another
+  server based on the request hash, until a server with additional capacity is
+  found. A higher  allows more imbalance between the servers, while a
+  lower  means that more servers will be checked on average, affecting
+  performance. Reasonable values are from 125 to 200.
+
+  See also : "balance" and "hash-type".
+
+
 bind []: [, ...] [param*]
 bind / [, ...] [param*]
   Define one or several listening addresses and/or ports in a frontend.
@@ -3358,7 +3384,7 @@ hash-type   
   default function is "sdbm", the selection of a function should be based on
   the range of the values being hashed.
 
-  See also : "balance", "server"
+  See also : "balance", "balance-factor", "server"
 
 
 http-check disable-on-404
diff --git a/include/types/lb_chash.h b/include/types/lb_chash.h
index 5991ce9..b711636 100644
--- a/include/types/lb_chash.h
+++ b/include/types/lb_chash.h
@@ -30,6 +30,7 @@ struct lb_chash {
struct eb_root act; /* weighted chash entries of active servers */
struct eb_root bck; /* weighted chash entries of backup servers */
struct eb32_node *last; /* last node found in case of round robin (or 
NULL) */
+   int balance_factor; /* load balancing factor * 100, 0 if disabled */
 };
 
 #endif /* _TYPES_LB_CHASH_H */
diff --git a/src/cfgparse.c b/src/cfgparse.c
index cc2f507..229b3ce 100644
--- a/src/cfgparse.c
+++ b/src/cfgparse.c
@@ -1983,6 +1983,7 @@ void init_default_instance()
defproxy.maxconn = cfg_maxpconn;
defproxy.conn_retries = CONN_RETRIES;
defproxy.redispatch_after = 0;
+   defproxy.lbprm.chash.balance_factor = 0;
 
defproxy.defsrv.check.inter = DEF_CHKINTR;
defproxy.defsrv.check.fastinter = 0;
@@ -2825,6 +2826,7 @@ int cfg_parse_listen(const char *file, int linenum, char 
**args, int kwm)
 
if (curproxy->cap & PR_CAP_BE) {
curproxy->lbprm.algo = defproxy.lbprm.algo;
+   curproxy->lbprm.chash.balance_factor = 
defproxy.lbprm.chash.balance_factor;
curproxy->fullconn = defproxy.fullconn;
curproxy->conn_retries = defproxy.conn_retries;
curproxy->redispatch_after = defproxy.redispatch_after;
@@ -5958,6 +5960,19 @@ stats_error_parsing:
}
}
}
+   else if (strcmp(args[0], "hash-balance-factor") == 0) {
+   if (*(args[1]) == 0) {
+   Alert("parsing [%s:%d] : '%s' expects an integer 
argument.\n", file, linenum, args[0]);
+   err_code |= ERR_ALERT | ERR_FATAL;
+   goto out;
+   }
+   curproxy->lbprm.chash.balance_factor = atol(args[1]);
+   if (curproxy->lbprm.chash.balance_factor != 0 && 
curproxy->lbprm.chash.balance_factor <= 100) {
+   Alert("parsing [%s:%d] : '%s' must be 0 or greater than 
100.\n", file, linenum, args[0]);
+   err_code |= ERR_ALERT | ERR_FATAL;
+   goto out;
+   }
+   }
else if (strcmp(args[0], "unique-id-format") == 0) {
if (!*(args[1])) {
Alert("parsing [%s:%d] : 

[PATCH 1/4] MINOR: proxy: add 'served' field to proxy, equal to total of all servers'

[Andrew Rodland]

This will allow lb_chash to determine the total active sessions for a
proxy without any computation.

Signed-off-by: Andrew Rodland 
---
 include/types/proxy.h | 1 +
 src/queue.c   | 1 +
 src/stream.c  | 2 ++
 3 files changed, 4 insertions(+)

diff --git a/include/types/proxy.h b/include/types/proxy.h
index 2f4f9b9..028b3a7 100644
--- a/include/types/proxy.h
+++ b/include/types/proxy.h
@@ -277,6 +277,7 @@ struct proxy {
} tcp_rep;
struct server *srv, defsrv; /* known servers; default 
server configuration */
int srv_act, srv_bck;   /* # of servers eligible for LB 
(UP|!checked) AND (enabled+weight!=0) */
+   int served; /* # of active sessions 
currently being served */
struct lbprm lbprm; /* load-balancing parameters */
char *cookie_domain;/* domain used to insert the 
cookie */
char *cookie_name;  /* name of the cookie to look 
for */
diff --git a/src/queue.c b/src/queue.c
index 1f27c49..08a6c3d 100644
--- a/src/queue.c
+++ b/src/queue.c
@@ -126,6 +126,7 @@ struct stream *pendconn_get_next_strm(struct server *srv, 
struct proxy *px)
strm->target = &srv->obj_type;
stream_add_srv_conn(strm, srv);
srv->served++;
+   srv->proxy->served++;
if (px->lbprm.server_take_conn)
px->lbprm.server_take_conn(srv);
 
diff --git a/src/stream.c b/src/stream.c
index 151bcb0..738a23c 100644
--- a/src/stream.c
+++ b/src/stream.c
@@ -2515,6 +2515,7 @@ void sess_change_server(struct stream *sess, struct 
server *newsrv)
 
if (sess->srv_conn) {
sess->srv_conn->served--;
+   sess->srv_conn->proxy->served--;
if (sess->srv_conn->proxy->lbprm.server_drop_conn)

sess->srv_conn->proxy->lbprm.server_drop_conn(sess->srv_conn);
stream_del_srv_conn(sess);
@@ -2522,6 +2523,7 @@ void sess_change_server(struct stream *sess, struct 
server *newsrv)
 
if (newsrv) {
newsrv->served++;
+   newsrv->proxy->served++;
if (newsrv->proxy->lbprm.server_take_conn)
newsrv->proxy->lbprm.server_take_conn(newsrv);
stream_add_srv_conn(sess, newsrv);
-- 
2.9.3

[PATCH 0/4] Consistent hashing with bounded loads

[Andrew Rodland]

Hi,

Here's a resend of my previous patch, incorporating some changes suggested by 
Willy. As my mailer ate my patches last time around, and I'm not completely 
sure that it won't do it again, I'm attaching them to this email as well as 
sending them inline in the following emails.

Thanks,

Andrew>From faaef5c05fd829579dc71c2f3d91a165a3d17a3b Mon Sep 17 00:00:00 2001
From: Andrew Rodland 
Date: Tue, 20 Sep 2016 13:40:00 -0400
Subject: [PATCH 1/4] MINOR: proxy: add 'served' field to proxy, equal to total
 of all servers'
X-Bogosity: Ham, tests=bogofilter, spamicity=0.00, version=1.2.4

This will allow lb_chash to determine the total active sessions for a
proxy without any computation.

Signed-off-by: Andrew Rodland 
---
 include/types/proxy.h | 1 +
 src/queue.c   | 1 +
 src/stream.c  | 2 ++
 3 files changed, 4 insertions(+)

diff --git a/include/types/proxy.h b/include/types/proxy.h
index 2f4f9b9..028b3a7 100644
--- a/include/types/proxy.h
+++ b/include/types/proxy.h
@@ -277,6 +277,7 @@ struct proxy {
 	} tcp_rep;
 	struct server *srv, defsrv;		/* known servers; default server configuration */
 	int srv_act, srv_bck;			/* # of servers eligible for LB (UP|!checked) AND (enabled+weight!=0) */
+	int served;/* # of active sessions currently being served */
 	struct lbprm lbprm;			/* load-balancing parameters */
 	char *cookie_domain;			/* domain used to insert the cookie */
 	char *cookie_name;			/* name of the cookie to look for */
diff --git a/src/queue.c b/src/queue.c
index 1f27c49..08a6c3d 100644
--- a/src/queue.c
+++ b/src/queue.c
@@ -126,6 +126,7 @@ struct stream *pendconn_get_next_strm(struct server *srv, struct proxy *px)
 	strm->target = &srv->obj_type;
 	stream_add_srv_conn(strm, srv);
 	srv->served++;
+	srv->proxy->served++;
 	if (px->lbprm.server_take_conn)
 		px->lbprm.server_take_conn(srv);
 
diff --git a/src/stream.c b/src/stream.c
index 151bcb0..738a23c 100644
--- a/src/stream.c
+++ b/src/stream.c
@@ -2515,6 +2515,7 @@ void sess_change_server(struct stream *sess, struct server *newsrv)
 
 	if (sess->srv_conn) {
 		sess->srv_conn->served--;
+		sess->srv_conn->proxy->served--;
 		if (sess->srv_conn->proxy->lbprm.server_drop_conn)
 			sess->srv_conn->proxy->lbprm.server_drop_conn(sess->srv_conn);
 		stream_del_srv_conn(sess);
@@ -2522,6 +2523,7 @@ void sess_change_server(struct stream *sess, struct server *newsrv)
 
 	if (newsrv) {
 		newsrv->served++;
+		newsrv->proxy->served++;
 		if (newsrv->proxy->lbprm.server_take_conn)
 			newsrv->proxy->lbprm.server_take_conn(newsrv);
 		stream_add_srv_conn(sess, newsrv);
-- 
2.9.3

>From 799186e5bcd94befabb18f84bc4500fb6cca07d5 Mon Sep 17 00:00:00 2001
From: Andrew Rodland 
Date: Tue, 20 Sep 2016 13:41:44 -0400
Subject: [PATCH 2/4] MINOR: backend: add hash-balance-factor option for
 hash-type consistent
X-Bogosity: Ham, tests=bogofilter, spamicity=0.00, version=1.2.4

0 will mean no balancing occurs; otherwise it represents the ratio
between the highest-loaded server and the average load, times 100 (i.e.
a value of 150 means a 1.5x ratio), assuming equal weights.

Signed-off-by: Andrew Rodland 
---
 doc/configuration.txt| 28 +++-
 include/types/lb_chash.h |  1 +
 src/cfgparse.c   | 15 +++
 3 files changed, 43 insertions(+), 1 deletion(-)

diff --git a/doc/configuration.txt b/doc/configuration.txt
index d047524..a93c103 100644
--- a/doc/configuration.txt
+++ b/doc/configuration.txt
@@ -2205,6 +2205,32 @@ balance url_param  [check_post]
   See also : "dispatch", "cookie", "transparent", "hash-type" and "http_proxy".
 
 
+balance-factor 
+  Specify the balancing factor for bounded-load consistent hashing
+  May be used in sections :   defaults | frontend | listen | backend
+ yes   |no|   no   |   yes
+  Arguments :
+ is the control for the maximum number of concurrent requests to
+ send to a server, expressed as a percentage of the average number
+	 of concurrent requests across all of the active servers.
+
+  Specifying a "balance-factor" for a server with "hash-type consistent"
+  enables an algorithm that prevents any one server from getting too many
+  requests at once, even if some hash buckets receive many more requests than
+  others. Setting  to 0 (the default) disables the feature. Otherwise,
+   is a percentage greater than 100. For example, if  is 150,
+  then no server will be allowed to have a load more than 1.5 times the average.
+  If server weights are used, they will be respected.
+
+  If the first-choice server is disqualified, the algorithm will choose another
+  server based on the request hash, until a server with additional capacity is
+  found. A higher  allows more imbalance between the servers, while a
+  lower  means that more servers will be checked on average, affecting
+  performance. Reasonable values are from 125 to 200.
+
+  See also : "balance" and "hash-type".
+
+
 bind []: [, ...] [param*]
 

Re: HAProxy reloads lets old and outdated processes

[Willy Tarreau]

Hi Pierre,

> Apart from that, we exchanged off-list with Willy about the submitted patch.
> It seems that if fixes the issue. I now have only one instance bound to the
> TCP sockets after the reloads, the others are there just to terminate the
> existing connections.

And thank you for the quick tests in the live environment, that was very
helpful. Here's the patch series I have added to address the issue (there
were other abominations in this wrapper that had to be dealt with) :

  7643d09 BUG/MINOR: systemd: make the wrapper return a non-null status code on 
error
  4351ea6 BUG/MINOR: systemd: always restore signals before execve()
  3747ea0 BUG/MINOR: systemd: check return value of calloc()
  a785269 MINOR: systemd: report it when execve() fails
  b957109 BUG/MEDIUM: systemd: let the wrapper know that haproxy has completed 
or failed

I intend to backport this soon into 1.6 and even 1.5. Normally the
wrapper is expected to be exactly the same so the patches should
apply (unless we missed some fixes of course). Those facing the issue
on these versions are welcome to test, if more issues remain we'll have
to address them anyway.

cheers,
Willy

RE: HAProxy reloads lets old and outdated processes

[Pierre Cheynier]

Hi,


I didn't subscribed to the list and noticed that there was several exchanges on 
this thread that I didn't read so far.


To share a bit more of our context:


* we do not reload every 2ms, this was the setting used to be able to reproduce 
easily and in a short period of time. Our reload average is more around 5 to 
10s, which seems consistent to me on relatively big setups (I'm talking about 1 
hundred of physical nodes per DC that makes run up to 1 thousand of app 
instances).


* true, it's something that becomes very common as long as I/PaaS-style 
architectures are adopted. On our side we work with Apache Mesos and schedulers 
that add/remove backends as long as the end-user scale his application or if 
node/app fails, are under maintenance etc.


By the way, I noticed that a lot of these "trending" projects are using HAProxy 
as their external load balancing stack (and most of them are also usually run 
over systemd-based distros), so it seems to me that this will fix some setups 
(that apparently rely on Yelp approach to 'safely restart' their haproxy - but 
induce latencies).


Apart from that, we exchanged off-list with Willy about the submitted patch. It 
seems that if fixes the issue. I now have only one instance bound to the TCP 
sockets after the reloads, the others are there just to terminate the existing 
connections.


Pierre

Re: Can I specify a wildcard redirect

[Andrew Smalley]

Hello Jürgen

Thank you for your reply saying its the same line you already have

I did this on a single VIP assuming you just wanted to rewrite /de to / and
have everything below /de/page-x become /page-x

If this is the case it works well and does not produce a redirect loop.

Try it out and see how it works on its own.



Regards

Andrew Smalley

Loadbalancer.org Ltd.



On 25 October 2016 at 15:18, Jürgen Haas  wrote:

> Thanks Andrew,
>
> That's the same regex that I have in my backend definition. But I also
> need the ACLs to make sure that the redirect only happens on a specific
> host and with a specific beginning of a path. Otherwise that would be
> redirected every time and end up in an infinite loop, doesn't it?
>
> Thanks
> Jürgen
>
> Am 25.10.2016 um 15:47 schrieb Andrew Smalley:
> > Hello Jürgen
> >
> > Sorry for the delay in replying to you.
> >
> > after a little playing I have come up with this single line without an
> > ACL which seems to do what you want.
> >
> > It will redirect http://domain.com/de/this/that/other/dir
> >
> >
> > To
> >
> > http://domain.com/this/that/other/dir
> >
> >
> > reqrep ^([^\ :]*)\ /de/(.*) \1\ /\2
> >
> > Regards
> >
> > Andrew Smalley
> >
> > Loadbalancer.org Ltd.
> >
> >
> >
> > On 25 October 2016 at 10:35, Jürgen Haas
> >  > > wrote:
> >
> > Hi Andrew,
> >
> > just not having luck with this. Here is my rule which is certainly
> used
> > when e.g. calling https://www.arocom.de/de/team but it doesn't
> redirect
> > to https://www.arocom.de/team
> >
> > Any idea what's wrong?
> >
> > backend backend_aweb2_https
> >   acl r_host hdr(host) -i -n www.arocom.de 
> >   acl r_path path_beg /de/
> >   reqirep "^([^\ :]*)\ /de/(.+)" "\1\ /\2" if r_host r_path
> >   redirect prefix / code 301 if r_host r_path
> >   http-response add-header X-Via aweb2
> >   server server_aweb2 1.2.3.4:80  maxconn 100
> >
> > Thanks
> > Jürgen
> >
> >
> > Am 24.10.2016 um 11:23 schrieb Andrew Smalley:
> > > Hello Jürgen
> > >
> > > In that case I think you will want something like
> > >
> > >
> > > |acl de_url path_beg /de reqrep ^([^\ :]*)\ /de/\d+/(.+)/? \1\ /\2
> > > redirect prefix / code 301 if de_url |
> > >
> > >
> > >
> > > Regards
> > >
> > > Andrew Smalley
> > >
> > > Loadbalancer.org Ltd.
> > >
> > >
> > >
> > > On 24 October 2016 at 10:19, Jürgen Haas
> > >  > 
> > >  >  xmd5yjdbdmrexy1tmh2...@public.gmane.org>>>
> > wrote:
> > >
> > > Hi Andrew,
> > >
> > > Thanks for your quick reply and yes, I'm using the manual
> almost daily.
> > > But my question is not covered, I guess.
> > >
> > > Also your example is not working as it is always redirecting
> to the
> > > front page, but we would require wildcards.
> > >
> > > Examples:
> > >
> > > http://www.example.com/de/page-one  page-one>
> > >  > > =>
> > > http://www.example.com/page-one
> >   > >
> > > http://www.example.com/de/page-two
> > 
> > >  > > =>
> > > http://www.example.com/page-two
> >   > >
> > >
> > > In other words, we just want to remove the "/de" subsctring
> from the
> > > URL. Is that possible?
> > >
> > >
> > > Thanks
> > > Jürgen
> > >
> > >
> > >
> > > Am 24.10.2016 um 11:00 schrieb Andrew Smalley:
> > > > Hello Jürgen
> > > >
> > > > Below is a link to the haproxy manual which will tell you
> exactly what
> > > > you wish to know.
> > > >
> > > > https://www.haproxy.com/doc/aloha/7.0/haproxy/http_
> redirection.html
> >  redirection.html>
> > >  redirection.html
> >  >>
> > > >
> > > > and something like this will be what you are looking to do
> > > >
> > > > |acl is_de path_beg -i /de acl is_domain hdr(host) -i
> www.domain.com  
> > > > 

Re: Can I specify a wildcard redirect

[Jürgen Haas]

Thanks Andrew,

That's the same regex that I have in my backend definition. But I also
need the ACLs to make sure that the redirect only happens on a specific
host and with a specific beginning of a path. Otherwise that would be
redirected every time and end up in an infinite loop, doesn't it?

Thanks
Jürgen

Am 25.10.2016 um 15:47 schrieb Andrew Smalley:
> Hello Jürgen
> 
> Sorry for the delay in replying to you.
> 
> after a little playing I have come up with this single line without an
> ACL which seems to do what you want.
> 
> It will redirect http://domain.com/de/this/that/other/dir
> 
> 
> To
> 
> http://domain.com/this/that/other/dir
> 
> 
> reqrep ^([^\ :]*)\ /de/(.*) \1\ /\2
> 
> Regards
> 
> Andrew Smalley
> 
> Loadbalancer.org Ltd.
> 
> 
> 
> On 25 October 2016 at 10:35, Jürgen Haas
>  > wrote:
> 
> Hi Andrew,
> 
> just not having luck with this. Here is my rule which is certainly used
> when e.g. calling https://www.arocom.de/de/team but it doesn't redirect
> to https://www.arocom.de/team
> 
> Any idea what's wrong?
> 
> backend backend_aweb2_https
>   acl r_host hdr(host) -i -n www.arocom.de 
>   acl r_path path_beg /de/
>   reqirep "^([^\ :]*)\ /de/(.+)" "\1\ /\2" if r_host r_path
>   redirect prefix / code 301 if r_host r_path
>   http-response add-header X-Via aweb2
>   server server_aweb2 1.2.3.4:80  maxconn 100
> 
> Thanks
> Jürgen
> 
> 
> Am 24.10.2016 um 11:23 schrieb Andrew Smalley:
> > Hello Jürgen
> >
> > In that case I think you will want something like
> >
> >
> > |acl de_url path_beg /de reqrep ^([^\ :]*)\ /de/\d+/(.+)/? \1\ /\2
> > redirect prefix / code 301 if de_url |
> >
> >
> >
> > Regards
> >
> > Andrew Smalley
> >
> > Loadbalancer.org Ltd.
> >
> >
> >
> > On 24 October 2016 at 10:19, Jürgen Haas
> >  
> >  
> >>
> wrote:
> >
> > Hi Andrew,
> >
> > Thanks for your quick reply and yes, I'm using the manual almost 
> daily.
> > But my question is not covered, I guess.
> >
> > Also your example is not working as it is always redirecting to the
> > front page, but we would require wildcards.
> >
> > Examples:
> >
> > http://www.example.com/de/page-one 
> 
> >  > =>
> > http://www.example.com/page-one
>   >
> > http://www.example.com/de/page-two
> 
> >  > =>
> > http://www.example.com/page-two
>   >
> >
> > In other words, we just want to remove the "/de" subsctring from the
> > URL. Is that possible?
> >
> >
> > Thanks
> > Jürgen
> >
> >
> >
> > Am 24.10.2016 um 11:00 schrieb Andrew Smalley:
> > > Hello Jürgen
> > >
> > > Below is a link to the haproxy manual which will tell you exactly 
> what
> > > you wish to know.
> > >
> > > 
> https://www.haproxy.com/doc/aloha/7.0/haproxy/http_redirection.html
> 
> >  >
> > >
> > > and something like this will be what you are looking to do
> > >
> > > |acl is_de path_beg -i /de acl is_domain hdr(host) -i 
> www.domain.com  
> > >  redirect code 301 location
> > > http://www.domain.com/ if is_domain is_de|
> > >
> > >
> > >
> > > Regards
> > >
> > > Andrew Smalley
> > >
> > > Loadbalancer.org Ltd.
> > >
> > >
> > >
> > > On 24 October 2016 at 09:53, Jürgen Haas
> > >  
> 
> >  
> >
> > > 

Re: Can I specify a wildcard redirect

[Andrew Smalley]

Hello Jürgen

Sorry for the delay in replying to you.

after a little playing I have come up with this single line without an ACL
which seems to do what you want.

It will redirect http://domain.com/de/this/that/other/dir


To

http://domain.com/this/that/other/dir


reqrep ^([^\ :]*)\ /de/(.*) \1\ /\2

Regards

Andrew Smalley

Loadbalancer.org Ltd.



On 25 October 2016 at 10:35, Jürgen Haas  wrote:

> Hi Andrew,
>
> just not having luck with this. Here is my rule which is certainly used
> when e.g. calling https://www.arocom.de/de/team but it doesn't redirect
> to https://www.arocom.de/team
>
> Any idea what's wrong?
>
> backend backend_aweb2_https
>   acl r_host hdr(host) -i -n www.arocom.de
>   acl r_path path_beg /de/
>   reqirep "^([^\ :]*)\ /de/(.+)" "\1\ /\2" if r_host r_path
>   redirect prefix / code 301 if r_host r_path
>   http-response add-header X-Via aweb2
>   server server_aweb2 1.2.3.4:80 maxconn 100
>
> Thanks
> Jürgen
>
>
> Am 24.10.2016 um 11:23 schrieb Andrew Smalley:
> > Hello Jürgen
> >
> > In that case I think you will want something like
> >
> >
> > |acl de_url path_beg /de reqrep ^([^\ :]*)\ /de/\d+/(.+)/? \1\ /\2
> > redirect prefix / code 301 if de_url |
> >
> >
> >
> > Regards
> >
> > Andrew Smalley
> >
> > Loadbalancer.org Ltd.
> >
> >
> >
> > On 24 October 2016 at 10:19, Jürgen Haas
> >  > > wrote:
> >
> > Hi Andrew,
> >
> > Thanks for your quick reply and yes, I'm using the manual almost
> daily.
> > But my question is not covered, I guess.
> >
> > Also your example is not working as it is always redirecting to the
> > front page, but we would require wildcards.
> >
> > Examples:
> >
> > http://www.example.com/de/page-one
> >  =>
> > http://www.example.com/page-one 
> > http://www.example.com/de/page-two
> >  =>
> > http://www.example.com/page-two 
> >
> > In other words, we just want to remove the "/de" subsctring from the
> > URL. Is that possible?
> >
> >
> > Thanks
> > Jürgen
> >
> >
> >
> > Am 24.10.2016 um 11:00 schrieb Andrew Smalley:
> > > Hello Jürgen
> > >
> > > Below is a link to the haproxy manual which will tell you exactly
> what
> > > you wish to know.
> > >
> > > https://www.haproxy.com/doc/aloha/7.0/haproxy/http_
> redirection.html
> >  >
> > >
> > > and something like this will be what you are looking to do
> > >
> > > |acl is_de path_beg -i /de acl is_domain hdr(host) -i
> www.domain.com 
> > >  redirect code 301 location
> > > http://www.domain.com/ if is_domain is_de|
> > >
> > >
> > >
> > > Regards
> > >
> > > Andrew Smalley
> > >
> > > Loadbalancer.org Ltd.
> > >
> > >
> > >
> > > On 24 October 2016 at 09:53, Jürgen Haas
> > >  > 
> > >  >  xmd5yjdbdmrexy1tmh2...@public.gmane.org>>>
> > wrote:
> > >
> > > Hi all,
> > >
> > > one of my clients is looking for a wildcard redirect to get
> redirects
> > > from www.example.com/de/* 
> >  to
> > > www.example.com/* 
> > 
> > >
> > > I know how to do just the opposite, but for this one I
> > couldn't find a
> > > solution in the documentation.
> > >
> > > Any chance that can be done?
> > >
> > >
> > > Thanks
> > > Jürgen
> > >
> > >
> >
> >
> >
>
>
>

Re: Detection of PROXY protocol version and Citrix CIP

[Willy Tarreau]

Hi Hugo,

I'm CCing Bertrand who implemented the netscaler CIP protocol.

On Mon, Oct 17, 2016 at 01:36:40PM -0700, Hugo Slabbert wrote:
> The PROXY protocol spec specifically indicates that a receiver should not
> try to guess whether or not a PROXY protocol header is present[1]:
> 
> > The receiver MUST be configured to only receive the protocol described
> > in this specification and MUST not try to guess whether the protocol
> > header is present or not. This means that the protocol explicitly
> > prevents port sharing between public and private access. Otherwise it
> > would open a major security breach by allowing untrusted parties to
> > spoof their connection addresses. The receiver SHOULD ensure proper
> > access filtering so that only trusted proxies are allowed to use this
> > protocol.
> 
> I have an (unfortunate) scenario where traffic to the same backend may be
> arriving through either an HAProxy *or* a Citrix NetScaler where source IP
> information needs to be passed to the backend, at least during a DNS swing
> from one to the other while records TTL out.
> 
> The spec as I read it intends to protect against a malicious actor sending
> traffic with a PROXY protocol header set with an arbitrary client IP.  In
> other words, the "MUST NOT" directive is there to ensure that backends that
> are not *explicitly* configured to receive PROXY protocol traffic can't be
> tricked into treating traffic from a random host as PROXY protocol traffic.
> 
> Is that correct?

Yes that's correct.

> Is it against the spirit or letter of the spec, though, to test not for the
> absence or presence of the PROXY protocol header itself but rather (a)
> whether the PROXY protocol version is 1 or 2 and (b) if another client
> source IP preservation/representation protocol, like the Citrix Client IP
> option, is in use?  Or does that still expose a possible exploit vector?
> Protections could still be provided to the backend servers in either case
> such that PROXY protocol traffic or Citrix Client IP traffic would only be
> permitted from known/trusted proxies.

No it's not against the spirit, I think it's even a good idea, just one
that we probably never thought about. As you understood, what matters is
the person configuring haproxy knows what front components can be trusted
or not. Once the connection comes from a trusted IP, I think you don't
care about the protocol, it could be accepted.

> I haven't been able to brainstorm a weakness/exploit in that type of setup,
> as it would basically amount to (pseudo-code):
> 
> if header.type == "proxyv1":
> # do PROXY protocol v1 stuff
> elif header.type == "proxyv2":
> # do PROXY protocol v2 stuff
> elif header.type == "cip":
> # do Citrix Client IP stuff
> else:
> # no good; drop it like it's hot
> 
> ...with the network configured to permit only trusted proxies to talk
> directly to the backends.

By the way you can currently do this using "expect-proxy layer4" and
"expect-netscaler-cip layer4" in tcp-request rules. If you already
know your clients addresses (which I'm sure you do), instead of leaving
the entry hardcoded you could put such rules instead.

Now I'm having a question : shouldn't we simply move the netscaler CIP
parser to the proxy protocol parser and use a single option ? (ie accept-
proxy would validate both of them, and possibly future ones if needed).

It's important to decide before we release 1.7 :-)

Thanks,
Willy

Re: Latency spikes

[Willy Tarreau]

On Mon, Oct 24, 2016 at 11:46:53AM +0300, Dmitry Maslov wrote:
> Hello,
> 
> I'm keep digging on this problem and that's what I've found so far:
> I have another one frontend/backend pair handled by the same instance, and
> it looks like whenever this backend goes slow, everything goes slow.
> Why one slow backend affects others? Any recommendations on how can I avoid
> this?

Is your global maxconn set ? It might be possible that you're limiting the
amount of incoming connections to a lower value than what you need. And
this typically happens when things slow down because connections continue
to accumulate.

Willy

Re: Issue with windows IE 11 and Edge

[Willy Tarreau]

On Mon, Oct 24, 2016 at 03:14:59PM +, James Stroehmann wrote:
> We have updated our test instance of HAProxy to 1.6.9 but are still able to
> reproduce the issue.
> 
> 
> From: Stroehmann, James
> Sent: Friday, October 07, 2016 2:44 PM
> To: haproxy@formilux.org
> Subject: Issue with windows IE 11 and Edge
> 
> We have a website that is setup like this:
> 
> CDN -> HAPROXY -> ELB -> APACHE -> TOMCAT
> 
> We are seeing with IE11 and Edge clients they will randomly lose their
> session and get kicked out to a login screen. Firefox, Chrome and older IE
> clients appear unaffected. Testing at all the different layers, we can
> reproduce against the HAProxy and CDN layers, but not the ELB/Apache/Tomcat
> layers, which leads us to believe the problem lies at the HAProxy layer.We
> are using HA-Proxy version 1.6.3 2015/12/25, compiled from source. I'm not
> seeing any errors in the haproxy logs, any ideas as to where to start
> troubleshooting?

Well technically speaking what can be said is that adding CDN+haproxy
triggers the problem. Without your config it's very hard to have any
idea about this. It will be important to see how you perform your
stickiness, how health checks are performed (because if your apache
servers are regularly seen up and down from haproxy, for sure the users
will be redistributed to another location). Also have you looked at your
haproxy stats to see if your servers are seen down from time to time ?
And what do you see in haproxy logs for users affected with the problem ?
The termination flags should almost always be "--". If you often see
something different, it might indicate an issue somewhere in the chain.
Most often with cookie insertion, you'll see "--NI" for new users getting
a new cookie, and "--VN" for existing visitors posting a valid cookie.
Anything else should be studied.

Regards,
Willy

Re: HAProxy reloads lets old and outdated processes

[Willy Tarreau]

Hi Holger,

On Tue, Oct 25, 2016 at 12:38:26PM +0200, Holger Just wrote:
> Hey Willy,
> 
> Willy Tarreau wrote:
> > I absolutely despise systemd and each time I have to work on the
> > wrapper I feel like I'm going to throw up. So for me working on this
> > crap is a huge pain each time. But I'm really fed up with seeing
> > people having problems in this crazy environment because one
> > clueless guy decided that he knew better than all others how a daemon
> > should reload, so whatever we can do to make our users' lifes easier
> > in captivity should at least be considered.
> 
> Just to be sure, I don't like systemd for mostly the reasons you
> mentioned. However, I do use the systemd wrapper to reliably run HAProxy
> under runit for a couple of years now.

This feedback is actually useful because if we later decide to improve
the design, we must absolutely take such other use cases into consideration.

> Since they (similar to most service managers) also expect a service to
> have one stable parent process even after reloading,

In my opinion this is a flawed expectation. I really don't understand how
a *service* manager confuses a *service* and a *process*. There is not a
1-to-1 relation between the two at all. A service is something that is
delivered. We don't care how it is delivered. Some kernel daemons do not
even involve any process at all. Further, certains settings like IP
forwarding are services in my opinion yet they just end up being a sysctl.

> the systemd wrapper
> acts as a nice workaround to facilitate reloading.

I've been using it from time to time to get multi-process *and* stdout/stderr
available for debugging.

> The same wrapper
> allows simple service handling with Solaris's SMF and is a much better
> solution than the crude python script I wrote a couple of years ago for
> this simple process.

Good point.

> I guess what I've been trying to say is: the wrapper is absolutely
> useful for about any process manager, not just systemd and I would love
> to see it stay compatible with other process managers like runit.

Do you see any specific value that I'm overlooing in using the wrapper
as it is now compared to having it natively integrated into haproxy ?
The only value I'm seeing is a possibly lower memory consumption for very
large configurations after the copy-on-write happens in the children. I
think it's a small benefit to be honnest.

Thanks for your feedback,
Willy

Re: HAProxy reloads lets old and outdated processes

[Holger Just]

Hey Willy,

Willy Tarreau wrote:
> I absolutely despise systemd and each time I have to work on the
> wrapper I feel like I'm going to throw up. So for me working on this
> crap is a huge pain each time. But I'm really fed up with seeing
> people having problems in this crazy environment because one
> clueless guy decided that he knew better than all others how a daemon
> should reload, so whatever we can do to make our users' lifes easier
> in captivity should at least be considered.

Just to be sure, I don't like systemd for mostly the reasons you
mentioned. However, I do use the systemd wrapper to reliably run HAProxy
under runit for a couple of years now.

Since they (similar to most service managers) also expect a service to
have one stable parent process even after reloading, the systemd wrapper
acts as a nice workaround to facilitate reloading. The same wrapper
allows simple service handling with Solaris's SMF and is a much better
solution than the crude python script I wrote a couple of years ago for
this simple process.

I guess what I've been trying to say is: the wrapper is absolutely
useful for about any process manager, not just systemd and I would love
to see it stay compatible with other process managers like runit.

Thanks for the great work Willy, here and on the Kernel.

Regards, Holger

Re: http-reuse always, work quite well

[Pavlos Parissis]

On 22/10/2016 08:08 πμ, Willy Tarreau wrote:
> Hi Pavlos,
> 
> On Fri, Oct 21, 2016 at 03:01:52PM +0200, Pavlos Parissis wrote:
>>> I'm not surprized that always works better, but my point is that if it's
>>> much better it can be useful to stay with it, but if it's only 1% better
>>> it's not worth it.
>>>
>>
>> It is way better:-), see Marcin's response.
> 
> Ah sorry, I missed it. Indeed it looks much better, but we don't have
> the reference (no reuse) on this graph.

I will try to rerun the test tomorrow, which runs on production servers with
real user traffic:-)

> If the no reuse shows 10 times
> higher average times, then it means "safe" reuse brings a 10 times
> improvement and "always" brings 20 times so it's a matter of choice.
> However if safe does approximately the same as no reuse, for sure
> "always" is almost needed.
> 
> while "always" is optimal, strictly speaking it's
> not very clean if the clients are not always willing to retry a failed
> first request, and browsers typically fall into that category. A real
> world case can be a request dequeued to a connection that just closes.

 What is the response of HAProxy to clients in this case? HTTP 50N?
>>>
>>> No, the client-side connection will simply be aborted so that the client
>>> can decide whether to retry or not.
>>
>> Connection will be aborted by haproxy sending TCP RST?
> 
> As much as possible yes. The principle is to let the client retry the
> request (since it is the only one knowing whether it's safe or not).
> 
>>> I'd suggest a rule of thumb (maybe this should be added to the doc) : watch
>>> your logs over a long period. If you don't see queue timeouts, nor request
>>> timeouts, it's probably safe enough to use "always".
>>
>> Which field on the log do we need to watch? Tq?
> 
> Tw (time spent waiting in the queue), Tc (time spent getting a connection),
> and of course the termination flags, everything with a C or Q on the second
> char needs to be analysed.
> 

I looked at the logs for a period of 11hours and found zero occurrences of C or
Q. I also didn't noticed any change on Tw and Tc timers. I will keep an eye.

>>> Each time you notice
>>> one of them, there is a small risk of impacting another client. It's not
>>> rocket science but the risks depend on the same parameters.
>>
>>
>> Thanks a lot for yet another reach in information replies.
> 
> You're welcome. Please note that the reuse mechanism is not perfect and
> can still be improved. So do not hesitate to report any issue you find,
> we definitely need real-world feedback like this. I cannot promise that
> every issue will be fixed, but at least we need to consider them and see
> what can be done.
> 

Acked, I will report any issues we may find.

Thanks,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: Can I specify a wildcard redirect

[Jürgen Haas]

Hi Andrew,

just not having luck with this. Here is my rule which is certainly used
when e.g. calling https://www.arocom.de/de/team but it doesn't redirect
to https://www.arocom.de/team

Any idea what's wrong?

backend backend_aweb2_https
  acl r_host hdr(host) -i -n www.arocom.de
  acl r_path path_beg /de/
  reqirep "^([^\ :]*)\ /de/(.+)" "\1\ /\2" if r_host r_path
  redirect prefix / code 301 if r_host r_path
  http-response add-header X-Via aweb2
  server server_aweb2 1.2.3.4:80 maxconn 100

Thanks
Jürgen


Am 24.10.2016 um 11:23 schrieb Andrew Smalley:
> Hello Jürgen
> 
> In that case I think you will want something like
> 
> 
> |acl de_url path_beg /de reqrep ^([^\ :]*)\ /de/\d+/(.+)/? \1\ /\2
> redirect prefix / code 301 if de_url |
> 
> 
> 
> Regards
> 
> Andrew Smalley
> 
> Loadbalancer.org Ltd.
> 
> 
> 
> On 24 October 2016 at 10:19, Jürgen Haas
>  > wrote:
> 
> Hi Andrew,
> 
> Thanks for your quick reply and yes, I'm using the manual almost daily.
> But my question is not covered, I guess.
> 
> Also your example is not working as it is always redirecting to the
> front page, but we would require wildcards.
> 
> Examples:
> 
> http://www.example.com/de/page-one
>  =>
> http://www.example.com/page-one 
> http://www.example.com/de/page-two
>  =>
> http://www.example.com/page-two 
> 
> In other words, we just want to remove the "/de" subsctring from the
> URL. Is that possible?
> 
> 
> Thanks
> Jürgen
> 
> 
> 
> Am 24.10.2016 um 11:00 schrieb Andrew Smalley:
> > Hello Jürgen
> >
> > Below is a link to the haproxy manual which will tell you exactly what
> > you wish to know.
> >
> > https://www.haproxy.com/doc/aloha/7.0/haproxy/http_redirection.html
> 
> >
> > and something like this will be what you are looking to do
> >
> > |acl is_de path_beg -i /de acl is_domain hdr(host) -i www.domain.com 
> 
> >  redirect code 301 location
> > http://www.domain.com/ if is_domain is_de|
> >
> >
> >
> > Regards
> >
> > Andrew Smalley
> >
> > Loadbalancer.org Ltd.
> >
> >
> >
> > On 24 October 2016 at 09:53, Jürgen Haas
> >  
> >  
> >>
> wrote:
> >
> > Hi all,
> >
> > one of my clients is looking for a wildcard redirect to get 
> redirects
> > from www.example.com/de/* 
>  to
> > www.example.com/* 
> 
> >
> > I know how to do just the opposite, but for this one I
> couldn't find a
> > solution in the documentation.
> >
> > Any chance that can be done?
> >
> >
> > Thanks
> > Jürgen
> >
> >
> 
> 
> 




signature.asc
Description: OpenPGP digital signature

Re: HAProxy reloads lets old and outdated processes

[Willy Tarreau]

Hi Jarno,

On Tue, Oct 25, 2016 at 11:43:44AM +0300, Jarno Huuskonen wrote:
> This is probably a bit off topic, but there's sd_notify call
> (and [Service] Type=notify)
> where service can notify systemd that it's done starting/reloading
> configuration:
> https://www.freedesktop.org/software/systemd/man/sd_notify.html

Thank you, that can be useful for future improvements on the wrapper.
And no, it's not off-topic, quite the opposite.

> I don't know if systemd-wrapper would call:
> sd_notify(0, "RELOADING=1");
> ... restart
> sd_notify(0, "READY=1");
> if this would prevent systemd from trying to do multiple reloads before
> haproxy has finished starting.

Yes, however for now the problem is that the wrapper doesn't even know
whether or not haproxy has finished restarting since it stays attached.

I'm starting to think that the wrapper was a good idea to address short
term incompatibilities, but over the long term we may have to think again
about a master-worker architecture that would address this. And this
combined with previous Simon's work on the socket server could possibly
also help address the RST on close issue.

Thanks,
willy

Re: HAProxy reloads lets old and outdated processes

[Willy Tarreau]

Hi Pavlos,

On Tue, Oct 25, 2016 at 10:34:14AM +0200, Pavlos Parissis wrote:
> Well, I have full confidence on the quality of your code

You should not, keep in mind that I still hold and by far the record
on the number of bugs in this project :-)

> (assuming you will
> polish the patch to handle errors as you mentioned :-) ) and I am willing to
> test it on our environment when arrives on HAPEE.

OK but anyway I don't intend to backport it until it's confirmed as OK
by those who currently face problems. There may be other stupid issues
I have no clue about when using systemd.

> But, we will never hit the
> conditions which triggers this behavior as our configuration tool for haproxy
> doesn't allow to reload very often, we allow 1 reload per min (this is
> configurable of course). We did that in order to address also the case of too
> many live processes for a cluster of haproxies which has a lot of long-lived 
> TCP
> connections [1].

I think that's reasonable. I've also seen other methods in the past which
I found were pretty efficient, especially in websocket environments : the
principle was to gracefully shut down the previous process and kill the
oldest ones (or just keep a certain number). Of course there can be
variations, but all of these methods try to address the same issues.

> > I'm really open to suggestions. I absolutely despise systemd and each time
> > I have to work on the wrapper I feel like I'm going to throw up. So for me
> > working on this crap is a huge pain each time. But I'm really fed up with
> > seeing people having problems in this crazy environment because one clueless
> > guy decided that he knew better than all others how a daemon should reload,
> > so whatever we can do to make our users' lifes easier in captivity should
> > at least be considered.
> > 
> 
> Have you considered to report this to systemd? May be they have a solution, I
> don't know.

It's impossible to report any single bug to them. Everytime it's someone else
doing something wrong because *they* are right. I reported a machine getting
dead at boot due to a dead RTC battery and this stupid systemd spinning in
loops on the time. I was told it was a kernel bug. The kernel bug is not being
able to store a date prior to 1970 on a 32-bit time_t... I even proposed a
patch to address the issue, it was rejected to put pressure on the kernel. In
the end another workaround was found in the kernel to address this crap. So the
systemd team just wants to adapt operating systems to their vision of what an OS
should be, and they use us (the users) to put pressure on other ones. BTW it's
how we ended up having this systemd-wrapper ourselves. We should have called it
"die-crappy-systemd" instead...

> To sum up, go ahead with the patch as it addresses real problems of users and
> you can count on me testing HAPEE in our environment.

Thank you! I'm waiting for Pierre's testing to confirm he cannot kill it
either anymore.

Willy

Re: HAProxy reloads lets old and outdated processes

[Jarno Huuskonen]

Hi,

On Sat, Oct 22, Willy Tarreau wrote:
> Another important point, when you say you restart every 2ms, are you
> certain you have a way to ensure that everything is completely started
> before you issue your signal to kill the old process ? I'm asking because
> thanks to the principle that the wrapper must stay in foreground (smart
> design choice from systemd), there's no way for a service manager to
> know whether all processes are fully started or not. With a normal init,
> when the process returns, all sub-processes have been created.

This is probably a bit off topic, but there's sd_notify call
(and [Service] Type=notify)
where service can notify systemd that it's done starting/reloading
configuration:
https://www.freedesktop.org/software/systemd/man/sd_notify.html

I don't know if systemd-wrapper would call:
sd_notify(0, "RELOADING=1");
... restart
sd_notify(0, "READY=1");
if this would prevent systemd from trying to do multiple reloads before
haproxy has finished starting.

-Jarno

-- 
Jarno Huuskonen

Re: HAProxy reloads lets old and outdated processes

[Pavlos Parissis]

On 25/10/2016 01:21 πμ, Willy Tarreau wrote:
> Hi guys,
> 
> On Tue, Oct 25, 2016 at 12:42:26AM +0200, Lukas Tribus wrote:
>> Not fixing *real world issues* because we don't agree with the use-case or
>> there is a design misconception somewhere else is dangerous. We don't have
>> to support every single obscure use-case out there, that's not what I am
>> saying, but systemd is a reality; as is docker and periodic reloads.
> (...)
> 
> Thank you both for your insights. There are indeed valid points in both
> of your arguments. I too am afraid of breaking things for people who do
> really abuse, but at the same time we cannot blame the users who get
> caught by systemd lying to them. I really don't care about people who
> would reload every 2 ms to be honnest, but I'm concerned about people
> shooting themselves in the foot because they use very large configs or
> because as you say Lukas, they accidently run the command twice. This
> is something we used to deal with in the past, it's hard to lose this
> robustness. I've seen configs taking minutes to start (300k backends),
> reduced to a few seconds after the backends were moved to a binary
> tree. But these seconds remain a period of uncertainty and that's not
> nice for the user.
> 
> I think the patch I sent this evening covers both of your concerns. It's
> quite simple, relies on a process *closing* a file descriptor, which also
> covers a dying/crashing process (because I never trust any design consisting
> in saying "I promise I will tell you when I die"). And it doesn't
> significantly change things. I'm really interested in feedback on it.
> Pavlos, please honnestly tell me if it really scares you and I'd like
> to address this (even if that means doing it differently). Let's consider
> that I want something backportable into HAPEE, you know that users there
> can be very demanding regarding reliability :-)
> 

Well, I have full confidence on the quality of your code (assuming you will
polish the patch to handle errors as you mentioned :-) ) and I am willing to
test it on our environment when arrives on HAPEE. But, we will never hit the
conditions which triggers this behavior as our configuration tool for haproxy
doesn't allow to reload very often, we allow 1 reload per min (this is
configurable of course). We did that in order to address also the case of too
many live processes for a cluster of haproxies which has a lot of long-lived TCP
connections [1].


> I'm really open to suggestions. I absolutely despise systemd and each time
> I have to work on the wrapper I feel like I'm going to throw up. So for me
> working on this crap is a huge pain each time. But I'm really fed up with
> seeing people having problems in this crazy environment because one clueless
> guy decided that he knew better than all others how a daemon should reload,
> so whatever we can do to make our users' lifes easier in captivity should
> at least be considered.
> 

Have you considered to report this to systemd? May be they have a solution, I
don't know.

To sum up, go ahead with the patch as it addresses real problems of users and
you can count on me testing HAPEE in our environment.

Cheers,
Pavlos

[1] I have mentioned before that we balance rsyslog traffic from 20K clients
and every time we reload haproxy we see old processes staying alive for days.
This is because frontend/backend runs in TCP mode and rsyslog daemon on clients
doesn't terminate the connection till it is restarted, it opens 1
long-lived TCP connection against frontend. haproxy can't close the connection
on shutdown as it does with HTTP mode as it doesn't understand the protocol and
tries to play nice and graceful with the clients, I will wait for you to close
the connection.



signature.asc
Description: OpenPGP digital signature

Re: HAProxy reloads lets old and outdated processes

[Pavlos Parissis]

Good morning,

Got my coffee ready before I read and reply:-)

On 25/10/2016 12:42 πμ, Lukas Tribus wrote:
> Hello,
> 
> 
> Am 24.10.2016 um 22:32 schrieb Pavlos Parissis:
>> 
>> IMHO: Ask the users to not perform reloads every 2miliseconds. It is
>> insane. You may spend X hours on this which will make the code a bot more
>> complex and cause possible breakages somewhere else.
> 
> Not fixing *real world issues* because we don't agree with the use-case or
> there is a design misconception somewhere else is dangerous. We don't have to
> support every single obscure use-case out there, that's not what I am saying,
> but systemd is a reality; as is docker and periodic reloads.
> 
> You are talking about 2 milliseconds, but that is just the testcase here,
> think about how long haproxy would need to start when it has to load
> thousands of certificates. Probably more than a few seconds (I don't have any
> clue), and it would be pretty easy to create a mess of processes, not because
> of docker/cloud orchestration/whatever, but in SSH by hitting reload two
> times in a row manually.
> 
> I don't want to be scared of hitting reload two times even if I'm on a
> systemd based box with heavy SSL traffic. In fact, I *do* wanna be able to
> reload haproxy every 2 ms, not because I need it, but because the alternative
> would mean I need to remember to be "always careful" about that "strange
> issue with systemd which is not our fault" and make sure my colleague is not
> doing the same thing I'm doing and we reload simultaneously. I don't want to
> run my infrastructure like a house of cards.
> 
> This is not limited to fancy new cloud orchestration technologies and it is
> not a minor issue either.
> 
> 

All valid points. The bottom line is to trust the reload process that wont cause
unexpected behavior regardless the frequency of reloads and wallclock of a
single reload.


> 
>> I am pretty sure 90% of the cases which require so often reload are the
>> ones which try to integrate HAProxy with docker stuff, where servers in the
>> pools are treated as ephemeral nodes, appear and disappear very often and
>> at high volume.
> 
> Not sure if I understand you here correctly, but this sounds like you are 
> implying that we shouldn't spend time fixing issues related to docker (and 
> similar technologies). I have to disagree.
> 

Οn the contrary, I have requested for ETA of DNS SRV functionality, which allows
to extend and shrink the backend without reload, and I have also requested for
the ability to add/remove servers via the socket. All these because I need to
support docker on my environments:-)

The high frequency of reloads on docker environment is the result of missing the
above 2 functionalities.

> 
> We may not like systemd and we may not like docker. But that doesn't mean
> its not worth looking into those issues.
> 

Οn the contrary, I *do* love systemd. I am not joking here.

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature