Re: graceful shutdown maximum time
On Fri, Aug 18, 2017, at 02:53 PM, William Lallemand wrote: > On Fri, Aug 18, 2017 at 02:26:29PM -0700, Justin Karneges wrote: > > Is there a way to configure HAProxy to have a maximum graceful shutdown > > time? For example it would stop listening for new connections > > immediately, and then after a specified amount of time it would close > > all connections and terminate, regardless of what kind of activity might > > be going on within those connections. > > The "hard-stop-after" keyword do what you want: > > https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#hard-stop-after Cool! Looks like a relatively new feature. I'll check it out. Thanks, Justin
graceful shutdown maximum time
Hi, When HAProxy is shut down gracefully, my understanding is that it waits for all open connections to be closed before it will terminate. However, if the connections don't ever close then HAProxy may never shut down (or perhaps it takes a very long time, I'm not sure). This is mainly a problem with long-lived connections that remain continuously active, so timeouts won't trigger either. Is there a way to configure HAProxy to have a maximum graceful shutdown time? For example it would stop listening for new connections immediately, and then after a specified amount of time it would close all connections and terminate, regardless of what kind of activity might be going on within those connections. Otherwise my plan is to create a cron job that kills old haproxy processes on a delay. Would be nice if there were a better option though. Thanks for any suggestion. Justin
Re: proxy protocol for varnish 3.0.5
On 12/30/2013 10:00 AM, Baptiste wrote: On Mon, Dec 30, 2013 at 6:36 PM, Emmanuel Hocdet m...@gandi.net wrote: Hi, I have made a patch to add proxy protocol to V arnish 3.0 you can find it at http://varnish.hocdet.net Emmanuel Brilliant!!! I'm going to play with it soon :) Dumb question: what is the advantage of the proxy protocol for http (as would be the case with varnish)? I assumed the proxy protocol was used to enable load balancing of non-http protocols. Justin
copy header
Hi folks, I have a need to capture the exact headers sent by a client. The problem is I also use options forwardfor and http-server-close, so the headers sent to my backend are altered by HAProxy. My plan is to copy any affected headers under new names, using a special prefix. This way my backend application has a way to differentiate between modified headers and original headers. HAProxy doesn't seem to have a command for copying headers. There is only reqadd for adding and reqrep for replacing. However, I can fake a copy by injecting a newline in the middle of a replacement value: reqirep ^(Connection:)(.*) Connection:\2\nOrig-\1\2 If the client provides: Connection: Keep-Alive Then the above rule (along with http-server-close) will cause HAproxy to send the request with: Connection: Keep-Alive Orig-Connection: Keep-Alive Connection: close My backend application can then be configured to capture the Connection header only if it is prefixed. This seems to work, but I am posting to the mailing list first just to confirm if this isn't an abuse of reqirep. Thanks, Justin
Re: HAProxy with native SSL support !
On Tuesday, September 04, 2012 01:37:17 AM Willy Tarreau wrote: After several months of efforts by the Exceliance team, we managed to rework all the buffer and connection layers in order to get SSL working on both sides of HAProxy. Very cool. Since HAProxy is event-driven, is anything done to avoid expensive crypto operations from blocking the event loop? Maybe that is what you intend for remote sockets ? Justin
Re: HAProxy with native SSL support !
On Tuesday, September 04, 2012 08:41:44 AM Willy Tarreau wrote: On Mon, Sep 03, 2012 at 11:21:51PM -0700, Justin Karneges wrote: On Tuesday, September 04, 2012 01:37:17 AM Willy Tarreau wrote: After several months of efforts by the Exceliance team, we managed to rework all the buffer and connection layers in order to get SSL working on both sides of HAProxy. Very cool. Since HAProxy is event-driven, is anything done to avoid expensive crypto operations from blocking the event loop? Maybe that is what you intend for remote sockets ? Exactly ! That said, as I've always said, the problem is not the relative cost of crypto but the computation time. On small devices, if an operation can take 10ms, it clearly is a problem. However on larger devices, if no operation takes more than 100 microseconds, then the added latency is minimal compared to system latency. That does not mean it's not a problem, just that when you're at 100% CPU, you might still have enough scheduling capacity to process the rest of the traffic without too much impact. But clearly when we can offload SSL to other processes, that will be much better. This can already be done BTW, but you have to chain proxies. For instance : global nbproc 4 listen front bind-process 1,2 bind :443 ssl haproxy.pem server http 127.0.0.1: send-proxy listen http bind-process 3 bind 127.0.0.1: accept-proxy mode http server back 127.0.0.1:8443 send-proxy listen back bind-process 4 bind 127.0.0.1:8443 accept-proxy server out 192.168.0.1:443 ssl You see, https is accepted on processes 1 and 2, which decipher the traffic and pass it to process 3, preserving IP information. This process does the HTTP handling and passes the traffic to process 4 which will re-cipher it before going to the server. Admittedly this is ugly and that's why we'd like to implement the remote sockets soon, so that you can write something like this : listen http bind-process 3 bind :443 ssl haproxy.pem process 1,2 mode http server out 192.168.0.1:443 ssl process 4 Also, using socketpair() internally will be much faster and have a lower overhead than TCP connect/accept. Usually, the most expensive operations in TLS are the public key ones at the start of a negotiation (and possibly a renegotiation, though I'm not sure of protocol details there). However, pretty much all other TLS traffic should be fast to execute. I wonder if there is a way to differentiate these steps? This is how you'd ensure that a bunch of new https connections do not cause slowdown of existing https connections. I have tried in the past to use OpenSSL in an event-driven application in a way that does not block the event loop. This was for a local desktop application that used smart cards, where an OpenSSL call might block for 5 seconds while it hits the hardware. Since the expected number of TLS sessions in the application was small, I simply made one thread per TLS session, and funneled all results back to the event-driven main thread. Of course a thread per connection approach would not work if you want to handle thousands of connections like HAProxy. One day I wanted to go back and rewrite my code such that I could send operations to separate threads or keep them in the main thread based on a given TLS session state. This way, I could make it so only the TLS handshakes (with the expensive RSA operations and such) would need concurrency. Then the number of threads needed would be the number of expected simultaneous handshakes, which would be a far more manageable number (20?) than expected simultaneous sessions (thousands). This is not an answer for you, but just a brain dump of my thoughts on the subject. Justin
Re: patch: on-marked-up option
On Sunday, June 03, 2012 02:19:54 PM Willy Tarreau wrote: I think that overall it's fine. I'm going to add minor restrictions to the UP server before killing backup connections : - ensure that it's not backup (otherwise it can make things worse when you have multiple backup servers) - ensure that its weight is 0, otherwise it will not get any traffic I'll also add some basic doc for this new option. Sounds great. And I'm glad to see the dev11 release. Thanks! Justin
Re: mysql failover and forcing disconnects
On Wednesday, May 23, 2012 11:57:14 PM Willy Tarreau wrote: There is an option at the server level which is on-marked-down shutdown-session. It achieves exactly what you want, it will kill all connections to a server which is detected as down. Perfect! 2) If the master eventually comes back, all connections that ended up routing to the slave will stay on the slave indefinitely. The only solution I have for this is to restart mysql on the slave, which kicks everyone off causing them to reconnect and get routed back to the master. This is acceptable if restoring master required some kind of manual maintenance, since I'd already be getting my hands dirty anyway. However, if master disappears and comes back due to brief network outage that resolves itself automatically, it's unfortunate that I'd still have to manually react to this by kicking everyone off the slave. There is no universal solution for this. As haproxy doesn't inspect the mysql traffic, it cannot know when a connection remains idle and unused. Making it arbitrarily kill connections to a working server would be the worst thing to do, as it would kill connections on which a transaction is waiting for being completed. Well, the network could fail at anytime and have a similar effect. I'm not sure if killing all connections to the backup is really any worse than killing all connections to the non-backup (via on-marked-down). Either way a bunch of client errors may occur, but for a scenario that is hopefully rare. Maybe an on-marked-up shutdown-backup-sessions option would be good. Justin
Re: mysql failover and forcing disconnects
On Thursday, May 24, 2012 01:59:32 AM Willy Tarreau wrote: On Thu, May 24, 2012 at 01:12:14AM -0700, Justin Karneges wrote: Well, the network could fail at anytime and have a similar effect. I'm not sure if killing all connections to the backup is really any worse than killing all connections to the non-backup (via on-marked-down). Either way a bunch of client errors may occur, but for a scenario that is hopefully rare. Killing connections when something fails is acceptable to many people, but killing connections when everything goes well is generally not accepted. This is certainly a sensible philosophy. I think what makes my scenario special is that the backup server is functionally worse than the non-backup. So even though we are discussing a destructive response to a positive event, it's the quickest way to get the service out of a degraded state. Maybe an on-marked-up shutdown-backup-sessions option would be good. I was thinking about something like this, but I still have doubts about its real usefulness. I don't know what others think here. If there is real demand for this and people think it serves a real purpose, I'm fine with accepting a patch to implement it. Thanks for being open. I'll mull this over some more and consider making a patch. Justin
patch: on-marked-up option
Hi,
This implements the feature discussed in the earlier thread of killing
connections on backup servers when a non-backup server comes back up. For
example, you can use this to route to a mysql master slave and ensure
clients don't stay on the slave after the master goes from down-up. I've done
some minimal testing and it seems to work.
Today is the first time I ever looked at haproxy's code but this feature seemed
straightforward enough to implement. I hope I've done it properly.
Justin
diff --git a/include/types/checks.h b/include/types/checks.h
index fd15c95..250a68f 100644
--- a/include/types/checks.h
+++ b/include/types/checks.h
@@ -80,6 +80,12 @@ enum {
};
enum {
+ HANA_ONMARKEDUP_NONE = 0,
+
+ HANA_ONMARKEDUP_SHUTDOWNBACKUPSESSIONS, /* Shutdown peer sessions */
+};
+
+enum {
HANA_OBS_NONE = 0,
HANA_OBS_LAYER4, /* Observe L4 - for example tcp */
diff --git a/include/types/server.h b/include/types/server.h
index aa2c4f8..1885eab 100644
--- a/include/types/server.h
+++ b/include/types/server.h
@@ -120,7 +120,8 @@ struct server {
int rise, fall;/* time in iterations */
int consecutive_errors_limit; /* number of consecutive errors that triggers an event */
short observe, onerror; /* observing mode: one of HANA_OBS_*; what to do on error: on of ANA_ONERR_* */
- short onmarkeddown; /* what to do when marked down: on of HANA_ONMARKEDDOWN_* */
+ short onmarkeddown; /* what to do when marked down: one of HANA_ONMARKEDDOWN_* */
+ short onmarkedup; /* what to do when marked up: one of HANA_ONMARKEDUP_* */
int inter, fastinter, downinter; /* checks: time in milliseconds */
int slowstart;/* slowstart time in seconds (ms in the conf) */
int result;/* health-check result : SRV_CHK_* */
diff --git a/include/types/session.h b/include/types/session.h
index f1b7451..a098002 100644
--- a/include/types/session.h
+++ b/include/types/session.h
@@ -67,6 +67,7 @@
#define SN_ERR_INTERNAL 0x7000 /* the proxy encountered an internal error */
#define SN_ERR_DOWN 0x8000 /* the proxy killed a session because the backend became unavailable */
#define SN_ERR_KILLED 0x9000 /* the proxy killed a session because it was asked to do so */
+#define SN_ERR_UP 0xa000 /* the proxy killed a session because a preferred backend became available */
#define SN_ERR_MASK 0xf000 /* mask to get only session error flags */
#define SN_ERR_SHIFT 12 /* bit shift */
diff --git a/src/cfgparse.c b/src/cfgparse.c
index 5bd2cfc..92dd094 100644
--- a/src/cfgparse.c
+++ b/src/cfgparse.c
@@ -4392,6 +4392,18 @@ stats_error_parsing:
cur_arg += 2;
}
+ else if (!strcmp(args[cur_arg], on-marked-up)) {
+if (!strcmp(args[cur_arg + 1], shutdown-backup-sessions))
+ newsrv-onmarkedup = HANA_ONMARKEDUP_SHUTDOWNBACKUPSESSIONS;
+else {
+ Alert(parsing [%s:%d]: '%s' expects 'shutdown-backup-sessions' but got '%s'\n,
+ file, linenum, args[cur_arg], args[cur_arg + 1]);
+ err_code |= ERR_ALERT | ERR_FATAL;
+ goto out;
+}
+
+cur_arg += 2;
+ }
else if (!strcmp(args[cur_arg], error-limit)) {
if (!*args[cur_arg + 1]) {
Alert(parsing [%s:%d]: '%s' expects an integer argument.\n,
diff --git a/src/checks.c b/src/checks.c
index febf77e..5299e98 100644
--- a/src/checks.c
+++ b/src/checks.c
@@ -358,15 +358,26 @@ static int check_for_pending(struct server *s)
return xferred;
}
-/* Shutdown connections when their server goes down.
+/* Shutdown all connections of a server
*/
-static void shutdown_sessions(struct server *srv)
+static void shutdown_sessions(struct server *srv, int why)
{
struct session *session, *session_bck;
list_for_each_entry_safe(session, session_bck, srv-actconns, by_srv)
if (session-srv_conn == srv)
- session_shutdown(session, SN_ERR_DOWN);
+ session_shutdown(session, why);
+}
+
+/* Shutdown all connections of all backup servers of a proxy
+ */
+static void shutdown_backup_sessions(struct proxy *px, int why)
+{
+ struct server *srv;
+
+ for (srv = px-srv; srv != NULL; srv = srv-next)
+ if (srv-state SRV_BACKUP)
+ shutdown_sessions(srv, why);
}
/* Sets server s down, notifies by all available means, recounts the
@@ -394,7 +405,7 @@ void set_server_down(struct server *s)
s-proxy-lbprm.set_server_status_down(s);
if (s-onmarkeddown HANA_ONMARKEDDOWN_SHUTDOWNSESSIONS)
- shutdown_sessions(s);
+ shutdown_sessions(s, SN_ERR_DOWN);
/* we might have sessions queued on this server and waiting for
* a connection. Those which are redispatchable will be queued
@@ -465,6 +476,9 @@ void set_server_up(struct server *s) {
s-state |= SRV_RUNNING;
s-state = ~SRV_MAINTAIN;
+ if (s-onmarkedup HANA_ONMARKEDUP_SHUTDOWNBACKUPSESSIONS)
+ shutdown_backup_sessions(s-proxy, SN_ERR_UP);
+
if (s-slowstart 0) {
s-state |= SRV_WARMINGUP;
if (s-proxy-lbprm.algo BE_LB_PROP_DYN) {
mysql failover and forcing disconnects
Hello list, I'm using haproxy to handle failover between a mysql master and slave. The slave replicates from master and is read-only. I specify both mysql servers in my haproxy configuration, and use the backup option on the slave. Applications connect to haproxy instead of mysql directly. Haproxy routes all connections to the master, unless the master is down in which case it routes them all to the slave. This actually works well enough, but a couple of peculiarities arise from the fact that haproxy doesn't disturb existing connections when servers go up and down: 1) Even if haproxy notices within seconds that the mysql master is down, existing connections remain pointed to the master. I set timeout server 5m so that within 5 minutes of inactivity, haproxy will eventually kill the connections, causing clients to reconnect and get routed to the slave. This means that in practice, the failover takes 5 minutes to fully complete. I could reduce this timeout value futher but this does not feel like the ideal solution. 2) If the master eventually comes back, all connections that ended up routing to the slave will stay on the slave indefinitely. The only solution I have for this is to restart mysql on the slave, which kicks everyone off causing them to reconnect and get routed back to the master. This is acceptable if restoring master required some kind of manual maintenance, since I'd already be getting my hands dirty anyway. However, if master disappears and comes back due to brief network outage that resolves itself automatically, it's unfortunate that I'd still have to manually react to this by kicking everyone off the slave. I wonder if both of these could be solved with some options that could make it so all clients are disconnected whenever master (not slave!) goes up or down. Or maybe there are some consequences to this approach that I'm not aware of. Thanks, Justin
mysql failover and forcing disconnects
(Apologies if this comes through twice. The first time I sent was before subscription approval, and I don't think it went through.) Hello list, I'm using haproxy to handle failover between a mysql master and slave. The slave replicates from master and is read-only. I specify both mysql servers in my haproxy configuration, and use the backup option on the slave. Applications connect to haproxy instead of mysql directly. Haproxy routes all connections to the master, unless the master is down in which case it routes them all to the slave. This actually works well enough, but a couple of peculiarities arise from the fact that haproxy doesn't disturb existing connections when servers go up and down: 1) Even if haproxy notices within seconds that the mysql master is down, existing connections remain pointed to the master. I set timeout server 5m so that within 5 minutes of inactivity, haproxy will eventually kill the connections, causing clients to reconnect and get routed to the slave. This means that in practice, the failover takes 5 minutes to fully complete. I could reduce this timeout value futher but this does not feel like the ideal solution. 2) If the master eventually comes back, all connections that ended up routing to the slave will stay on the slave indefinitely. The only solution I have for this is to restart mysql on the slave, which kicks everyone off causing them to reconnect and get routed back to the master. This is acceptable if restoring master required some kind of manual maintenance, since I'd already be getting my hands dirty anyway. However, if master disappears and comes back due to brief network outage that resolves itself automatically, it's unfortunate that I'd still have to manually react to this by kicking everyone off the slave. I wonder if both of these could be solved with some options that could make it so all clients are disconnected whenever master (not slave!) goes up or down. Or maybe there are some consequences to this approach that I'm not aware of. Thanks, Justin
