[PATCH] MINOR: cli: ability to change a server's name
I've been working on a change to Synapse[1] to allow it to use the "set server addr" directive to reserve a large(ish) pool of servers in a backend and dynamically set server host/ports into that pool as they come and go, which would significantly reduce the number of restarts we have to do during code deployments. While implementing this I came across a problem where I can't figure out how to dynamically update the server name (server->id internally), which is important because the only host/port information on the stats page is the name afaik. The stats page is frequently used in my org to debug which servers are up or down so having nice names that say where a server is (basically the address) would be useful. Also, historically Synapse and other tooling has used the server name to figure out which hosts and ports already have server lines but I can work around this by using "show stats" and actually using the addr field instead of the name field. But, having a nice human readable name is still good I think. Towards this end I've written up a patch (mostly just cribbed from set server fqdn) that updates the name for a server that enables: set server / name Would that patch be generally useful or is there a better way to update the server name? Or maybe this is already possible and I just don't know how? I'm happy to make any needed changes. Thanks! -Joey Lynch [1] https://github.com/airbnb/synapse 0001-MINOR-cli-ability-to-change-a-server-s-name.patch Description: Binary data
Re: HAProxy makes backend unresponsive when handling multiple thousand connections per second
netstat? Thread dumps? Test without SSL between haproxy and Tomcat. Monitor Tomcat via non haproxy path (direct curl?) *-Dsun.security.pkcs11.enable-solaris=false* On 22 Jun 2017 9:02 AM, "Igor Cicimov" wrote: > Hi Lukas, > > On 22 Jun 2017 3:02 am, "Lukas Tribus" wrote: > > Hello, > > > > Daniel, if using ssl to the backends shouldn't you use http mode? > > Per your config you are using tcp which is default one. Afaik tcp > > is for ssl passthrough. > > For the record, this is not true. Just because you need TCP mode > for TLS passthrough, doesn't mean you have to use HTTP mode when > terminating TLS. > > Actually, terminating TLS while using TCP mode is a quite common > configuration (for example with HTTP/2). > > > Thanks for clarifying this. > > > > > >> Try adding: > >> option httpclose > >> in the backend and see if that helps. > > > > Sorry, replace httpclose with http-server-close > > Actually, I would have suggested the opposite: making the whole > thing less expensive, by going full blown keep-alive with > http-reuse: > > option http-keep-alive > option prefer-last-server > timeout http-keep-alive 30s > http-reuse safe > > > Keep-alive is on by default hence my suggestion to use the opposite. Of > course keep-alive enabled is always better especially in case of ssl. > > > > > > global > > ulimit-n 2 > > Why specify ulimit? Haproxy will do this for you, you are just > asking for trouble. I suggest you remove this. > > > > Maybe something on your backend (conntrack or the application) > is rate-limiting per IP, or the aggressive client your are facing > is keep-aliving properly with the backend, while it doesn't when > using haproxy. > > > I would apply the keep-alive configurations above and I would > also suggest that you check the CPU load on your backend server > as connections through haproxy become unresponsive, because that > CPU can be saturated due to TLS negotiations as well. > > > That's what the haproxy log shows, the response time from the tomcat > backend is high suggesting something is wrong. Maybe something that you > mentioned above (which makes sesnse), some system settings or if we can see > the tomcat connector settings (and logs possibly) maybe something there is > causing issues. > > > > Regards, > Lukas > > >
Re: HAProxy makes backend unresponsive when handling multiple thousand connections per second
Hi everyone, thanks for your suggestions. Let me go through them step by step: Actually, I would have suggested the opposite: making the whole thing less expensive, by going full blown keep-alive with http-reuse: option http-keep-alive option prefer-last-server timeout http-keep-alive 30s http-reuse safe I will try these settings, thank you Lukas. If I understood the manual correctly when "prefer-last-server" is set, HAProxy tries to use an already established connection to a backend for an active session instead of rerouting it to another backend. Why specify ulimit? Haproxy will do this for you, you are just asking for trouble. I suggest you remove this. By default Solaris 11 has an ulimit of 256: -bash-4.4$ ulimit -n 256 If HAProxy can handle the ulimit beyond these 256 file descriptors this would be fine and the "ulimit"-Parameter isn't necessary indeed. Maybe something on your backend (conntrack or the application) is rate-limiting per IP, or the aggressive client your are facing is keep-aliving properly with the backend, while it doesn't when using haproxy. A rate-limit per IP is not active on any of our backends. I really suppose that our HAProxy config isn't sane and has some paradoxical parameters in it. A majority of the clients which access our application are using multiple backend which are NATed to the same IP on their side so on our side we just see one incoming IP which has several hundred to thousand of connections. if we can see the tomcat connector settings (and logs possibly) maybe something there is causing issues. Here are our Tomcat connector settings which are identical across our backends protocol="org.apache.coyote.http11.Http11NioProtocol" maxThreads="1024" enableLookups="false" acceptCount="500" compression="on" compressableMimeType="application/xml" clientAuth="false" URIEncoding="UTF-8" keystoreFile="/opt/tomcat/conf/.keystore" keystorePass="XXX" keyAlias="tomcat" SSLEnabled="true" scheme="https" secure="true" sslEnabledProtocols="TLSv1.2" ciphers="TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA" /> At first I will try the settings that Lukas suggested. This could take some time as we have to reproduce the problem in our test environment. I will get back to you once I got some results. Thank you very much and regards, Daniel
Re: haproxy does not capture the complete request header host sometimes
Hi! Phew, I was following this one with some concern, fearing it could be something more serious just waiting to hit us, too ;-) Great that the issue was found, thanks for that! There is just one thing I wanted to note regarding > […] It can be backported in 1.7, 1.6 and 1.5. I finally marked this patch as > a bug fix. If I read the patch correctly, even though it is classified as “MINOR” it will fail with an error on startup, when the configuration has a value outside the range. When backporting into the stable branches, this can lead to updates failing with existing configuration files — which may or may not be what you expect when doing minor upgrades. Granted, when you currently use an out-of-range value, you probably _want_ this fix, but still might hit you unexpectedly. It should be made very prominent in the release notes. Cheers, Daniel -- Daniel Schneller Principal Cloud Engineer CenterDevice GmbH | Hochstraße 11 | 42697 Solingen tel: +49 1754155711| Deutschland daniel.schnel...@centerdevice.de | www.centerdevice.de Geschäftsführung: Dr. Patrick Peschlow, Dr. Lukas Pustina, Michael Rosbach, Handelsregister-Nr.: HRB 18655, HR-Gericht: Bonn, USt-IdNr.: DE-815299431 > On 21. Jun. 2017, at 17:00, Christopher Faulet wrote: > > Le 13/06/2017 à 14:16, Christopher Faulet a écrit : >> Le 13/06/2017 à 10:31, siclesang a écrit : >>> haproxy balances by host,but often captures a part of request header >>> host or null, and requests balance to default server. >>> >>> how to debug it , >>> >> Hi, >> I'll try to help you. Can you share your configuration please ? It could >> help to find a potential bug. >> Could you also provide the tcpdump of a buggy request ? >> And finally, could you upgrade your HAProxy to the last 1.6 version >> (1.6.12) to be sure ? > > Hi, > > Just for the record. After some exchanges in private with siclesang, we found > the bug in the configuration parser, because of a too high value for > tune.http.maxhdr. Here is the explanation: > > Well, I think I found the problem. This is not a bug (not really). There > is something I missed in your configuration. You set tune.http.maxhdr to > 64000. I guess you keep this parameter during all your tests. This is an > invalid value. It needs to be in the range [0, 32767]. This is mandatory > to avoid integer overflow. the size of the array where headers offsets > are stored is a signed short. > > To be fair, there is no check on this value during the configuration > parsing. And the documentation does not specify any range for this > parameter. I will post a fix very quickly to avoid errors. > > BTW, this is a really huge value. The default one is 101. You can > legitimately increase this value. But there is no reason to have 64000 > headers in an HTTP message. IMHO, 1000/2000 is already an very huge limit. > > I attached a patch to improve the configuration parsing and to update the > documentation. It can be backported in 1.7, 1.6 and 1.5. I finally marked > this patch as a bug fix. > > Thanks siclesang for your help, > -- > Christopher Faulet > <0001-BUG-MINOR-cfgparse-Check-if-tune.http.maxhdr-is-in-t.patch>
Designing Your Commercial Website for More Visitors
Hello Haproxy.Org Team, I hope you are having a great day. We are Web Design company with a primary focus on SEO based Website Design & Development (ASP, ASP.Net, Java, Perl and PHP development). We have innovative web solutions for organizations including advanced database interactivity & database design, e-commerce solutions, Flash design with programming etc. We can provide you best solutions on given requirement:- Ø Website Design & Development Ø Graphic Design & Flash Ø Logo Design Ø Brochure Design Ø Website Templates Ø Maintenance Packages Please let us know in case you are interested. Thanks & Regard,** Carol Jones Sr. Manager-Sales and marketing [mn]* ** * Please let me know, if your want to make your existing website responsive for mobile devices. We will check and provide you an affordable cost along with time frame to make it mobile responsive.
Re: haproxy does not capture the complete request header host sometimes
Hi Daniel, On Thu, Jun 22, 2017 at 10:45:27AM +0200, Daniel Schneller wrote: > If I read the patch correctly, even though it is classified as "MINOR" it > will fail with an error on startup, when the configuration has a value > outside the range. > When backporting into the stable branches, this can lead to updates failing > with existing configuration files -- which may or may not be what you expect > when doing minor upgrades. > Granted, when you currently use an out-of-range value, you probably _want_ > this fix, but still might hit you unexpectedly. You're right, we should have raised it to MEDIUM at least but now it's merged. We can *try* to think about it when backporting it. Thanks for your feedback, Willy
Re: [PATCH] MINOR: cli: ability to change a server's name
Hi Joe,
On Thu, Jun 22, 2017 at 12:42:57AM -0700, Joseph Lynch wrote:
> I've been working on a change to Synapse[1] to allow it to use the
> "set server addr" directive to reserve a large(ish) pool of servers in
> a backend and dynamically set server host/ports into that pool as they
> come and go, which would significantly reduce the number of restarts
> we have to do during code deployments. While implementing this I came
> across a problem where I can't figure out how to dynamically update
> the server name (server->id internally), which is important because
> the only host/port information on the stats page is the name afaik.
If it's about the stats page, I think we'd rather emit the FQDN there,
what do you think ?
> The stats page is frequently used in my org to debug which servers are
> up or down so having nice names that say where a server is (basically
> the address) would be useful.
Everyone uses the stats page to debug ;-) That's why I'm fine with adding
further info there. We could even have a description field on servers and
listeners like we have for frontends and backends ("desc") that could be
used to pass extra info and which would not even require unicity (eg:
"3rd rack in DC2").
> Also, historically Synapse and other
> tooling has used the server name to figure out which hosts and ports
> already have server lines but I can work around this by using "show
> stats" and actually using the addr field instead of the name field.
> But, having a nice human readable name is still good I think.
I'm totally fine with adding a human readable field as it will really
help.
> Towards this end I've written up a patch (mostly just cribbed from set
> server fqdn) that updates the name for a server that enables:
>
> set server / name
>
> Would that patch be generally useful or is there a better way to
> update the server name? Or maybe this is already possible and I just
> don't know how? I'm happy to make any needed changes.
I'm having a few problems with changing the server's name :
- it's a unique identifier, so changing it after unicity has been
verified can be a problem and can become confusing in logs, stats
or anything else.
- the name is used for lookups at various places (eg: use_server)
and we really don't want to change it after the config is processed;
- you may even accidently set a name that you'll have a hard time
entering again just to change it, forcing you to restart after a
small mistake or an error in a script.
Given your intent is to provide human-readable information and that the
server name is a config primary key (not always the most readable one
by the way), better add a "desc " config option that you can change
at will on the CLI (set server / desc ), and add it to the
stats page (only when stats-show-desc is set), and let's add a sample
fetch function to return it (logs, headers, anything). What do you think ?
BTW, just a few minor comments on your patch below :
> From 2423ac26f56ecbf5dabe4624b124e84ccbe9f49a Mon Sep 17 00:00:00 2001
> From: Joseph Lynch
> Date: Wed, 21 Jun 2017 18:44:51 -0700
> Subject: [PATCH] MINOR: cli: ability to change a server's name
>
> This extends the "set server" socket command to also allow setting the
> name for a server. This allows tools to introspect those names and
> allows users of the stats page to have human readable names even when
> they are dynamically updating backends.
> ---
> doc/management.txt | 4
> include/proto/server.h | 1 +
> src/server.c | 47 ++-
> 3 files changed, 51 insertions(+), 1 deletion(-)
>
> diff --git a/src/server.c b/src/server.c
> index 008bafa..8ff804d 100644
> --- a/src/server.c
> +++ b/src/server.c
> @@ -4190,6 +4190,39 @@ int srv_init_addr(void)
> return return_code;
> }
>
> +const char *update_server_name(struct server *server, const char *name,
> const char *updater)
> +{
> + struct chunk *msg;
> +
> + msg = get_trash_chunk();
> + chunk_reset(msg);
> +
> + if (!strcmp(name, server->id)) {
> + chunk_appendf(msg, "no need to change the name");
> + goto out;
> + }
> +
> + char* old_name = server->id;
This will not build, please avoid inline declarations.
> + server->id = strdup(name);
> + if (!server->id) {
> + server->id = old_name;
> + chunk_reset(msg);
> + chunk_appendf(msg, "could not update %s/%s name",
> + server->proxy->id, server->id);
> + } else {
> + chunk_appendf(msg, "%s/%s changed its name from %s to %s",
> + server->proxy->id, old_name, old_name, name);
> + free(old_name);
> + }
I think you'll have less operations to perform if you first allocate the
new name then free the old one and switch the pointers in case of success.
It may also require less locking later for multithreading.
Thanks!
Willy
Re: Trouble getting rid of Connection Keep-Alive header
Title: Re: Trouble getting rid of Connection Keep-Alive header Hi Mats, Mats Eklund wrote on 22.06.2017: Hi Holger, Many thanks for this hint. I guess I will need to follow up with Openshift/RedHat. I have posted a question in the forum there, but so far no feedback: https://stackoverflow.com/questions/44134775/how-to-suppress-http-connection-keep-alive-header-in-response. Which openshift version do you use? I have also added this question to the stackoverflow question. Thanks also for the advice to upgrade. I'm not sure if that is easy or even possible in Openshift online. Maybe their next version of the platform (v3) will have a more recent version of HAProxy. Would be nice, because I am also missing some of the config options that were added in later versions. In openshift v3 is haproxy 1.5.18 (rh rpm) in place. The configfile for the openshift router can be found on github. https://github.com/openshift/origin/blob/master/images/router/haproxy/conf/haproxy-config.template If you want to use it on premise you can replace it with your own. https://docs.openshift.org/latest/install_config/router/customized_haproxy_router.html#using-configmap-replace-template As lukas written in this message below you will need to modify the haproxy config, which you can't on openshift online. https://www.mail-archive.com/haproxy@formilux.org/msg26530.html For the openshift router are some values which can be used to configure haproxy. https://docs.openshift.org/latest/install_config/router/customized_haproxy_router.html#using-env-var Side note: The *COMPRESSION* was my contribution ;-))) Kind regards, Mats Regards Aleks > > >Hi Mats, > >Mats Eklund wrote: >> I am running a load balanced Tomcat application on Openshift Online >>v2, with HAProxy ver. 1.4.22 as load balancer. > >With your current config, HAProxy will add a "Connection: close" header >to responses. However, since you mentioned you are running this in an >OpenShift environment, there might (and probably is) be another layer >of proxies involved between your HAProxy and your client. > >Since you are speaking plain HTTP here, this other proxy might chose to >use keep-alive connections towards the client, similar to how HAProxy's >option http-server-close works. In that case, you would have to change >the configuration of this other proxy too. > >Best, >Holger > >P.S. HAProxy 1.4 is OLD and receives only critical fixes now. You should >seriously consider upgrading to a newer version. The current stable >version if 1.7.26. > >At the very least, you should upgrade to the latest 1.4 version 1.4.27 >has fixed 83 known bugs since 1.4.22. See >https://www.haproxy.org/bugs/bugs-1.4.22.html for details.
Re: SD Termination state after upgrade from 1.5.12 to 1.7.3
Le 16/06/2017 à 16:19, Christopher Faulet a écrit :
Le 16/06/2017 à 13:29, Juan Pablo Mora a écrit :
Linux version:
Red Hat Enterprise Linux Server release 5.11 (Tikanga)
Linux dpoweb08 2.6.18-417.el5 #1 SMP Sat Nov 19 14:54:59 EST 2016 x86_64
x86_64 x86_64 GNU/Linux
HAProxy versión: 1.7.5
Summary:
After upgrading to HAProxy 1.7.5, requests with "SD" status in logs have
started to appear in one webservice. There have been no changes to the
webservice we invoke. The log shows:
Jun 16 12:41:06 localhost.lognet.pre.logalty.es haproxy[17315]:
172.31.2.70:59365 [16/Jun/2017:12:41:06.890] HTTP_INTERNO BUS/BUS1
1/0/0/65/75 200 225565 390 - - SD-- 12/2/0/0/0 0/0 {|Keep-Alive|174|}
{|close|} "POST
/lgt/lgtbus/rest/private/receiverServiceLight/getSignedBinaryContent
HTTP/1.1"
The same POST in HAProxy 1.5.12 ends with “ 200 “ and ““.
Hi,
Bernard McCormack has already reported the same issue. I'm on it. I
approximately identified the problem but I need more time to fully
understand it and to find the best way to fix it.
The bug occurs where we are waiting the end of the response to
synchronize the request and the response. The close on the server side
is detected as an error when there is no content-length nor
transfer-encoding header. But it is a bit tricky because it is partly a
timing issue. Depending when the connection close is catched, an error
is triggered or not.
So I'm on it. Stay tuned.
Hi guys,
I worked on this problem. Could you check if the attached patch fixes
your bug ?
Thanks,
--
Christopher Faulet
diff --git a/src/proto_http.c b/src/proto_http.c
index e5f67e5..521743a 100644
--- a/src/proto_http.c
+++ b/src/proto_http.c
@@ -6958,14 +6958,6 @@ int http_response_forward_body(struct stream *s, struct channel *res, int an_bit
if ((msg->flags & HTTP_MSGF_TE_CHNK) || (msg->flags & HTTP_MSGF_COMPRESSING))
res->flags |= CF_EXPECT_MORE;
- /* If there is neither content-length, nor transfer-encoding header
- * _AND_ there is no data filtering, we can safely forward all data
- * indefinitely. */
- if (!(msg->flags & HTTP_MSGF_XFER_LEN) && !HAS_DATA_FILTERS(s, res)) {
- buffer_flush(res->buf);
- channel_forward_forever(res);
- }
-
/* the stream handler will take care of timeouts and errors */
return 0;
@@ -7042,9 +7034,20 @@ http_msg_forward_body(struct stream *s, struct http_msg *msg)
goto missing_data_or_waiting;
}
- /* The server still sending data that should be filtered */
- if (!(msg->flags & HTTP_MSGF_XFER_LEN) && !(chn->flags & CF_SHUTR))
- goto missing_data_or_waiting;
+ /* This check can only be true for a response. HTTP_MSGF_XFER_LEN is
+ * always set for a request. */
+ if (!(msg->flags & HTTP_MSGF_XFER_LEN)) {
+ /* The server still sending data that should be filtered */
+ if (!(chn->flags & CF_SHUTR) && HAS_DATA_FILTERS(s, chn))
+ goto missing_data_or_waiting;
+ /* There is no data filtering (or no more data to filter) but at
+ * least one active filter on the stream. So force infinite
+ * forwarding here. */
+ else if (HAS_FILTERS(s)) {
+ buffer_flush(chn->buf);
+ channel_forward_forever(chn);
+ }
+ }
msg->msg_state = HTTP_MSG_ENDING;
Re: [PATCH] MINOR: cli: ability to change a server's name
> If it's about the stats page, I think we'd rather emit the FQDN there,
> what do you think ?
It is about the stats page, but also about our automation's ability to
take information from a service registry (which has host+port pairs +
health information) and identify which server in a HAProxy backend
maps to that service instance.
For example "enable server /" or "disable server /" has to
take the name, so if we can't rely on the server name being some
combination of host+port then for every up/down we'd have to "show
stat" and map from host+port to name for each update (as opposed to
issuing one socket "disable server". We also have lots of automation
(e.g. synapse, service monitoring scripts) which parse out the svname
to find the host+port, which while I can fix, is rather difficult.
Also there are a few stat commands (like "show servers state") which
don't show the host+port and only show the name.
> I'm having a few problems with changing the server's name :
> - it's a unique identifier, so changing it after unicity has been
> verified can be a problem and can become confusing in logs, stats
> or anything else.
I could verify that the name is unique perhaps? I don't feel like
logs/stats would be an issue because the only time you'd update the
name would be if the server is in MAINT (basically a free slot to
dynamically update to a new server)? Also logs usually have the actual
address information in addition to the name?
>
> - the name is used for lookups at various places (eg: use_server)
> and we really don't want to change it after the config is processed;
Is the concern that we might end up with a duplicate name and then the
lookup functions would return one or the other? Or is the concern that
changing it might cause bugs in those lookups?
>
> - you may even accidently set a name that you'll have a hard time
> entering again just to change it, forcing you to restart after a
> small mistake or an error in a script.
I don't really understand this, is the concern that you will enter a
name that conflicts so you then can't change it (because find_server
always returns the first server and not the one you set) or is it
something else?
> Given your intent is to provide human-readable information and that the
> server name is a config primary key (not always the most readable one
> by the way), better add a "desc " config option that you can change
> at will on the CLI (set server / desc ), and add it to the
> stats page (only when stats-show-desc is set), and let's add a sample
> fetch function to return it (logs, headers, anything). What do you think ?
I'll try it out, but that is a larger change that has backwards
compatibility concerns, and I'm still concerned about how does
automation go from address to name going forward. If I verified that
the name was unique would that be an ok solution?
>> +const char *update_server_name(struct server *server, const char *name,
>> const char *updater)
>> +{
>> + struct chunk *msg;
>> +
>> + msg = get_trash_chunk();
>> + chunk_reset(msg);
>> +
>> + if (!strcmp(name, server->id)) {
>> + chunk_appendf(msg, "no need to change the name");
>> + goto out;
>> + }
>> +
>> + char* old_name = server->id;
>
> This will not build, please avoid inline declarations.
I thought I declared it in /include/proto/server.h, oops I will double check!
>> + server->id = strdup(name);
>> + if (!server->id) {
>> + server->id = old_name;
>> + chunk_reset(msg);
>> + chunk_appendf(msg, "could not update %s/%s name",
>> + server->proxy->id, server->id);
>> + } else {
>> + chunk_appendf(msg, "%s/%s changed its name from %s to %s",
>> + server->proxy->id, old_name, old_name, name);
>> + free(old_name);
>> + }
>
> I think you'll have less operations to perform if you first allocate the
> new name then free the old one and switch the pointers in case of success.
> It may also require less locking later for multithreading.
Ok I will work on it!
Thanks for the reply!
-Joey
Re: [PATCH] MINOR: cli: ability to change a server's name
On Thu, Jun 22, 2017 at 08:55:03AM -0700, Joseph Lynch wrote: > > If it's about the stats page, I think we'd rather emit the FQDN there, > > what do you think ? > It is about the stats page, but also about our automation's ability to > take information from a service registry (which has host+port pairs + > health information) and identify which server in a HAProxy backend > maps to that service instance. > > For example "enable server /" or "disable server /" has to > take the name, so if we can't rely on the server name being some > combination of host+port then for every up/down we'd have to "show > stat" and map from host+port to name for each update (as opposed to > issuing one socket "disable server". We also have lots of automation > (e.g. synapse, service monitoring scripts) which parse out the svname > to find the host+port, which while I can fix, is rather difficult. > Also there are a few stat commands (like "show servers state") which > don't show the host+port and only show the name. The server state is a perfect example of huge breakage that would be caused by such a change. In short, you start with the names from the configuration file, which are looked up in the state file, then the names are changed, and once dumped, they are different. So upon next reload, nothing matches at all. > > I'm having a few problems with changing the server's name : > > - it's a unique identifier, so changing it after unicity has been > > verified can be a problem and can become confusing in logs, stats > > or anything else. > I could verify that the name is unique perhaps? I don't feel like > logs/stats would be an issue because the only time you'd update the > name would be if the server is in MAINT (basically a free slot to > dynamically update to a new server)? Also logs usually have the actual > address information in addition to the name? For me it's a more general problem of inconsistency between the configuration and what is running. For example if you have this : backend foo use-server srv1 if rule1 use-server srv2 if rule2 server srv1 1.1.1.1:80 server srv2 2.2.2.2:80 Do you know what happens once you'll switch the names over the CLI ? (hint: I know). And it will even be inconsistent with what use_backend does because use_backend *can* resolve at runtime instead of config time. The logs will return the current name depending on the last change, possibly conflicting with what is in the configuration. Another example : backend foo1 server srv1 1.1.1.1:80 track foo2/srv1 server srv2 2.2.2.2:80 track foo2/srv2 backend foo2 server srv1 1.1.1.1:80 check server srv2 2.2.2.2:80 check Here after a renaming of the tracked servers, it becomes a mess, you'll constantly have to know what's taken at startup time and what's taken at runtime. (...) > > Given your intent is to provide human-readable information and that the > > server name is a config primary key (not always the most readable one > > by the way), better add a "desc " config option that you can change > > at will on the CLI (set server / desc ), and add it to the > > stats page (only when stats-show-desc is set), and let's add a sample > > fetch function to return it (logs, headers, anything). What do you think ? > I'll try it out, but that is a larger change that has backwards > compatibility concerns, On the opposite, there's zero backwards compatibility concerns to have with adding a description field because we don't have it for servers for now, only for backends and frontends. > and I'm still concerned about how does > automation go from address to name going forward. Just a question, wouldn't your requirement in fact be to lookup a server by its address ? I mean, wouldn't it be easier for your use case ? If so I'm pretty sure that's something we could easily apply. Either we apply this to the first server that matches the address, or we apply it to all those which match, depending on the difficulty. For example wouldn't it be easier for you to be able to do : set server addr bk/@1.1.1.1:80 2.2.2.2:80 or : disable server bk/@1.1.1.1:80 ? > If I verified that > the name was unique would that be an ok solution? As explained above for me it's far from being a valid solution due to the major inconsistencies between the configuration, the state file, the logs, the stats etc... In the end everything suddenly becomes unobservable and undebuggable, and state file cannot be used anymore. I'm pretty sure you have a very valid use case that we probably need to discuss more to find the largest intersection with what can reasonably be done here without breaking existing setups. We know that the management socket needs to be significantly improved for some maintenance operations, so it's even more important to know what your ideal usage scenario would be to see how it could be mapped here. > >> +const char *update_server_name(struct server *server
Re: [PATCH] MINOR: cli: ability to change a server's name
Hm, I'm still struggling to understand why this is a problem as an option for operators, sorry if I'm being dense! > The server state is a perfect example of huge breakage that would be > caused by such a change. In short, you start with the names from the > configuration file, which are looked up in the state file, then the > names are changed, and once dumped, they are different. So upon next > reload, nothing matches at all. As long as the configuration file is updated before updating the name this should be fine, shouldn't it? I was under the impression that best practice for using dynamic HAProxy server features was always to write out the modified configuration file when sending socket commands. That way reloads are consistent, and if someone really needs to know what exactly is happening right now in the running process that's what show stat or show servers state is for? I'm also struggling to see why the state file is an issue, my understanding is that you save the server state file before a reload, so why would you send a name update after you save the server sate? Fwiw I've tested the patch and "show server state" correctly displays the updated name [1], but I don't think that's what you are concerned about... > For me it's a more general problem of inconsistency between the configuration > and what is running. For example if you have this :\ I very much agree, I just thought that we were already going down this path with dynamically updating addrs and using server templates (which introduce config-runtime drift as well). As long as whatever is doing the socket manipulation (in our case synapse), does file configuration changes first shouldn't this be reasonably ok? > > backend foo > use-server srv1 if rule1 > use-server srv2 if rule2 > server srv1 1.1.1.1:80 > server srv2 2.2.2.2:80 > > Do you know what happens once you'll switch the names over the CLI ? (hint: > I know). My understanding is that we'd continue sending to the original servers. I do see this would be bad, but isn't the requirement on the user to not do this, specifically if you have use-server rules then you have to reload the config to pick up changes in backends? Furthermore, rule1 or rule2 could refer to hard coded IP addresses which would also be wrong if we updated the addr of the server with "set server addr" as it is. Also thinking about this more, if use-server rules dynamically searched for the server instead of storing that might be a nifty way to do like primary replica failover, something like: backend foo use-server leader if write_operation server leader 1.1.1.1:1234 server follower1 2.2.2.2:1234 server follower2 3.3.3.3:1234 Then to do a leader failover you would just set the leader to a follower (during which time you 503 write requests), and promote a follower to a leader. I think named backends with a use_backend ACL is probably the _better_ way to do this, but it's an interesting thought. > And it will even be inconsistent with what use_backend does because > use_backend *can* resolve at runtime instead of config time. The logs will > return the current name depending on the last change, possibly conflicting > with what is in the configuration. Another example : > > backend foo1 > server srv1 1.1.1.1:80 track foo2/srv1 > server srv2 2.2.2.2:80 track foo2/srv2 > > backend foo2 > server srv1 1.1.1.1:80 check > server srv2 2.2.2.2:80 check > > Here after a renaming of the tracked servers, it becomes a mess, you'll > constantly have to know what's taken at startup time and what's taken at > runtime. I agree, but if an administrator is using features like track or use-server I think they should pick up a server change with a reload rather than with the dynamic socket. Similar to if they are using ip or port based ACLs they should probably reload instead of setting the address dynamically? > On the opposite, there's zero backwards compatibility concerns to have > with adding a description field because we don't have it for servers for > now, only for backends and frontends. I see, ok I will see if this can work. >> and I'm still concerned about how does >> automation go from address to name going forward. > > Just a question, wouldn't your requirement in fact be to lookup a server > by its address ? I mean, wouldn't it be easier for your use case ? If so > I'm pretty sure that's something we could easily apply. Either we apply > this to the first server that matches the address, or we apply it to all > those which match, depending on the difficulty. > > For example wouldn't it be easier for you to be able to do : > >set server addr bk/@1.1.1.1:80 2.2.2.2:80 > > or : > >disable server bk/@1.1.1.1:80 > > ? Interesting syntax, I think if we added that to all the set server commands it could work (we'd have to port from using enable/disable server to set server state, but that could work) for all but the
Re: Logging SSL pre-master-key
On 2017/6/19 13:54, Patrick Hemmer wrote:
>
>
> On 2017/6/17 00:00, Willy Tarreau wrote:
>> Hi Patrick,
>>
>> On Fri, Jun 16, 2017 at 09:36:30PM -0400, Patrick Hemmer wrote:
>>> The main reason I had for supporting the older code is that it seems
>>> many (most?) linux distros, such as the one we use (CentOS/7), still
>>> ship with 1.0.1 or 1.0.2. However since this is a minor change and a
>>> feature enhancement, I doubt this will get backported to 1.7, meaning
>>> we'll have to manually patch it into the version we use. And since we're
>>> doing that, we can just use the patch that supports older OpenSSL.
>> Yes in this case I think it makes sense. And given the proliferation
>> of *ssl implementations, I really prefer to avoid adding code touching
>> directly internal openssl stuff.
>>
>>> Anyway, here is the updated patch with the support for <1.1.0 dropped,
>>> as well as the BoringSSL support Emmanuel requested.
>> OK cool.
>>
>>> One other minor thing I was unsure about was the fetch name. Currently I
>>> have it as "ssl_fc_session_key", but "ssl_fc_session_master_key" might
>>> be more accurate. However this is rather verbose and would make it far
>>> longer than any other sample fetch name, and I was unsure if there were
>>> rules around the naming.
>> There are no particular rules but it's true that long names are painful,
>> especially for those with "fat fingers". I'd suggest that given that it
>> starts with "ssl_fc_" and "ssl_bc_", the context is already quite
>> restricted and you could probably call them "smk" or "smkey". One would
>> argue that looking at the doc will be necessary, but with long names you
>> also have to look at the doc to find how to spell them anyway, the
>> difference being that short names don't mangle your configs too much,
>> especially in logformat strings.
> Well my argument for keeping the name starting with `ssl_fc_session_`
> is that there is also `ssl_fc_session_id`. These 2 fetches pull their
> attribute from the same "session" structure. They are also closely
> related as using `ssl_fc_session_key` almost requires the
> `ssl_fc_session_id` value (you could technically get by without it,
> but it'll make using the key rather difficult unless you have some
> other way of correlating a key with a specific SSL session).
>>> +ssl_bc_session_key : binary
>>> + Returns the SSL session master key of the back connection when the
>>> outgoing
>>> + connection was made over an SSL/TLS transport layer. It is useful to
>>> decrypt
>>> + traffic sent using ephemeral ciphers.
>> Here it would be nice to add a short sentence like "Not available before
>> openssl 1.1.0" so that users don't waste too much time on something not
>> working.
>>
>>> static int
>>> +smp_fetch_ssl_fc_session_key(const struct arg *args, struct sample *smp,
>>> const char *kw, void *private)
>>> +{
>>> +#if OPENSSL_VERSION_NUMBER >= 0x1010L || defined(OPENSSL_IS_BORINGSSL)
>> Here I'd put the ifdef around the whole function, like we have for
>> ALPN for example, so that there's no risk it could be used in a non-working
>> config.
> My objection to this is that most of the other sample fetches don't do
> this, and so it makes the user experience inconsistent. For example
> the `ssl_fc_session_id` fetch, which `ssl_fc_session_key` is strongly
> related to, behaves the same way. The ALPN/NPN fetches are the only
> ones that make using the fetch a config error if the underlying
> support is missing.
>>> + struct connection *conn = objt_conn((kw[4] != 'b') ? smp->sess->origin :
>>> + smp->strm ? smp->strm->si[1].end :
>>> NULL);
>>> +
>>> + SSL_SESSION *ssl_sess;
>>> + int data_len;
>>> + struct chunk *data;
>>> +
>>> + if (!conn || !conn->xprt_ctx || conn->xprt != &ssl_sock)
>>> + return 0;
>>> +
>>> + ssl_sess = SSL_get_session(conn->xprt_ctx);
>>> + if (!ssl_sess)
>>> + return 0;
>>> +
>>> + data = get_trash_chunk();
>>> + data_len = SSL_SESSION_get_master_key(ssl_sess, (unsigned char
>>> *)data->str, data->size);
>>> + if (!data_len)
>>> + return 0;
>>> +
>>> + smp->flags = SMP_F_CONST;
>>> + smp->data.type = SMP_T_BIN;
>>> + data->len = data_len;
>> If you want, you can even get rid of your data_len variable by directly
>> assigning SSL_SESSION_get_master_key() to data->len.
>>
>>> +static int
>>> smp_fetch_ssl_fc_sni(const struct arg *args, struct sample *smp, const
>>> char *kw, void *private)
>>> {
>>> #ifdef SSL_CTRL_SET_TLSEXT_HOSTNAME
>>> @@ -7841,6 +7875,7 @@ static struct sample_fetch_kw_list
>>> sample_fetch_keywords = {ILH, {
>>> { "ssl_bc_unique_id", smp_fetch_ssl_fc_unique_id, 0,
>>>NULL,SMP_T_BIN, SMP_USE_L5SRV },
>>> { "ssl_bc_use_keysize", smp_fetch_ssl_fc_use_keysize, 0,
>>>NULL,SMP_T_SINT, SMP_USE_L5SRV },
>>> { "ssl_bc_session_id", smp_fetch_ssl_fc_session_id, 0,
>>>NULL,
Reverse Gateway Throught Security Zones
Hello Guys, Sorry to botter you with my specific questions :-) Let's imagine a paranoic security team who forbide http and tcp flows between the dmz zone and the green zone, they estimate that if an hacker can take control on the dmz zone server they the can access the green zone from that server, so flows going from the dmz zone to the green zone are forbidden and blocked by network firewalls, First idea : So what I need is to create something like a reverse tunnel between the green zone and HAProxy, Do you think it is possible, and if it's, what's the best way to do this, I would like to have Haproxy in the DMZ and Applications Servers connecting from the green zone to the HAProxy in the DMZ Zone and passing requests to the green zone (requests are going from the dmz zone to the green zone with a reverse connection) Do I need to code something for this, and if yes, what is the best way to do it ? Any idea ? Thank you really much to people interested on this exiting pattern ! Best regards, Himer --- Forbidden : Internet --> DMZ --> Green Zone Authorized : __ Internet --> DMZ <--- Green Zone
Re: HAProxy makes backend unresponsive when handling multiple thousand connections per second
Daniel, Here's a different approach to your problem. As someone who wears too many hats, I am often asked about reencrypting connections that are terminated by haproxy. Since this is a common occurrence between a small number of systems, it is much more efficient to create a small IPSec overlay network with the haproxy and target servers. The encryption costs are probably dropped by an order of magnitude or more for the connection. You do IKE renegotiation on the order of once an hour vs. TLS negotiations hundreds of times per second. The actual message encryption is quite fast on any modern server processor. You also don't need to manage certs on the inside machines. These load reductions are on both sides, so the tomcat machines will benefit as well. The down side is that you either need to mesh the IPSec connections or run a routing daemon to handle reachability on the IPSec overlay. One way or the other, you will probably need to do some automation to get this into production. (I build these in a somewhat star-like set of IPSec connections with the haproxy systems at the centers. Then I run a dirt simple routing protocol with bird.) At your traffic rate, I would do this as my default approach. jerry On 06/22/2017 01:21 AM, Daniel Heitepriem wrote: Hi everyone, thanks for your suggestions. Let me go through them step by step: Actually, I would have suggested the opposite: making the whole thing less expensive, by going full blown keep-alive with http-reuse: option http-keep-alive option prefer-last-server timeout http-keep-alive 30s http-reuse safe I will try these settings, thank you Lukas. If I understood the manual correctly when "prefer-last-server" is set, HAProxy tries to use an already established connection to a backend for an active session instead of rerouting it to another backend. Why specify ulimit? Haproxy will do this for you, you are just asking for trouble. I suggest you remove this. By default Solaris 11 has an ulimit of 256: -bash-4.4$ ulimit -n 256 If HAProxy can handle the ulimit beyond these 256 file descriptors this would be fine and the "ulimit"-Parameter isn't necessary indeed. Maybe something on your backend (conntrack or the application) is rate-limiting per IP, or the aggressive client your are facing is keep-aliving properly with the backend, while it doesn't when using haproxy. A rate-limit per IP is not active on any of our backends. I really suppose that our HAProxy config isn't sane and has some paradoxical parameters in it. A majority of the clients which access our application are using multiple backend which are NATed to the same IP on their side so on our side we just see one incoming IP which has several hundred to thousand of connections. if we can see the tomcat connector settings (and logs possibly) maybe something there is causing issues. Here are our Tomcat connector settings which are identical across our backends At first I will try the settings that Lukas suggested. This could take some time as we have to reproduce the problem in our test environment. I will get back to you once I got some results. Thank you very much and regards, Daniel -- Jerry Scharf, Soundhound DevOps "What could possibly go wrong?"
Re: [PATCH] MINOR: cli: ability to change a server's name
On Thu, Jun 22, 2017 at 11:46:12AM -0700, Joseph Lynch wrote: > Hm, I'm still struggling to understand why this is a problem as an > option for operators, sorry if I'm being dense! No it's not because you're dense, it's simply because you're using a very small subset of haproxy's features and in *your* environment, the name is just a random piece of information. In other places it's a primary key to a number of things. In fact the server's name should be seen as a technical internal identifier that happens to be user visible. It's very much like the numerical id and is even interchangeable with it. BTW as you noticed, it's stored in srv->id. There are places in the code just like external tools which heavily rely on this strong 1:1 unicity. I'm not judging whether or not it was a good or bad initial choice and whether or not it's still a good choice, it's just a fact that it exists, for SNMP monitoring, for log analysis, for the health checks when "http-check send-state" is used, in the agent, in the unique-id header, in http-send-name-header, maybe even for the peers protocol (I don't remember), for internal lookups. Changing this internal ID on the fly doesn't seem like a good idea to me just because we suddenly change this mapping *after* it was resolved and applied. The whole process relies on some consistency. If you dump the stats and you see for server srv1 "UP via bck2/srv1", this bck2/ srv1 does exist and is guaranteed to exist for the whole life of the process, which allows the stats to pause on full buffers and continue in the middle of a dump because the state doesn't change below it, which allows external tools too recursively check the bck2/srv1 stats in the same dump which is guaranteed to exist, etc. Or even worse, you click on the stats page to disable 4 servers, their names are sent, but due to the way the request arrives in partial segments, they're processed as data arrives, and in the middle, your service controller renames some servers so certain operations are applied to different servers, or multiple times to the same one and none to another one. It's hard for me to enumerate good examples of inconsistencies in a system where the consistency is guaranteed by design and has always been granted, because everything was built on top of such an assumption, so while some of the breakage is easy to predict, a lot more has to be discovered. > > The server state is a perfect example of huge breakage that would be > > caused by such a change. In short, you start with the names from the > > configuration file, which are looked up in the state file, then the > > names are changed, and once dumped, they are different. So upon next > > reload, nothing matches at all. > > As long as the configuration file is updated before updating the name > this should be fine, shouldn't it? That's the theory, probably that *you* are going to do it. Matching logs against configs is another game. Also when a log is emitted after a long session, what name do you dump, the one the session started on or the one the session ended on ? > I was under the impression that > best practice for using dynamic HAProxy server features was always to > write out the modified configuration file when sending socket > commands. It depends on the changes. For all changes that are made in order to avoid a reload, that's indeed the case. For regular operations, most often it's not (eg: change a server's weight during backups etc). > That way reloads are consistent, and if someone really needs > to know what exactly is happening right now in the running process > that's what show stat or show servers state is for? I'm also > struggling to see why the state file is an issue, my understanding is > that you save the server state file before a reload, so why would you > send a name update after you save the server sate? It could be the other way around in fact, it's people not dumping the server state as often as the name updates, for whatever reasons. And it could be seen differently as well, why would someone replace a server's internal name after everything was mapped on it ? :-) > > For me it's a more general problem of inconsistency between the > > configuration > > and what is running. For example if you have this :\ > > I very much agree, I just thought that we were already going down this > path with dynamically updating addrs and using server templates (which > introduce config-runtime drift as well). Server templates are a very good example : the server names are automatic numbers. If you happen to rename a server belonging to a template, there's no way you'll be able to find it again in the state file since you cannot change its name in the configuration. > As long as whatever is doing > the socket manipulation (in our case synapse), does file configuration > changes first shouldn't this be reasonably ok? I'm pretty fine with changing a lot of things in the config file and in the internal state at run time, as lon
Re: Reverse Gateway Throught Security Zones
Hello Himer, this is probably not the response you wanna hear ... Am 22.06.2017 um 22:47 schrieb Himer Martinez: > Hello Guys, > > Sorry to botter you with my specific questions :-) > > Let's imagine a paranoic security team who forbide http and tcp flows between > the dmz zone and the green zone, they estimate that if an hacker can take > control on the dmz zone server they the can access the green zone from that > server, so flows going from the dmz zone to the green zone are forbidden and > blocked by network firewalls, > > First idea : So what I need is to create something like a reverse tunnel > between the green zone and HAProxy, Clarification: what you or your security team is saying is: - a DMZ host establishing a TCP connection to the green zone is insecure (even if the only open port is HTTP) - a green zone host establishing whatever bidirectional connections to the DMZ servers is secure Is that a correct interpretation? > > (requests are going from the dmz zone to the green zone with a reverse > connection) So by reverse tunneling you basically circumvent your firewalls and any security policies that may be in place. You are opening the "DMZ --> Green Zone" path, just in a less direct way, and most likely without or with less considerations regarding security. > Forbidden : > Internet --> DMZ --> Green Zone > > Authorized : > Internet --> DMZ <--- Green Zone This is a ridiculous concept. DMZ needs Green Zone data, either move your Green Zone hosts into the DMZ or make the service you need reachable (considering security aspects, of course). By reverse-tunneling you don't gain any security advantage, instead, you are over complicating your setup, bypassing most likely restrictive firewalls, opening an attack surface you are not considering. > First idea : So what I need is to create something like a reverse tunnel > between the green zone and HAProxy, What you need to do is analyze your *REAL* requirements, from a security perspective and otherwise, and then build a concept around it. Instead you are slamming a 1999 "perimeter security" concept on your network which doesn't match your requirements and are now trying to circumvent the perimeters, because otherwise you are unable to provide whatever service you need to run. Now to the part that you do wanna hear: How you can one best bypass a perimeter firewall that is blocking one direction of traffic but not the other? Use any VPN that you are familiar with, as that is exactly what they are built for. OpenVPN, strongSwan, etc. cheers, lukas
Re: [PATCH] MINOR: cli: ability to change a server's name
Hi Willy, > OK so if we add a string as the external identifier (let's at least > give a name to this thing) and the ability to lookup by transport > address, you have everything : > - the mesos/kubernetes identifier directly accessible > - the IP:port directly accessible > > Did I miss anything ? An external identifier that socket commands can be passed as an acceptable would definitely solve the problem. I guess by default the external identifier can just be the name, and we'd prefer matching servers by name before external_identifier? An alternative we could add a lookup function that returns server names matching an address. This would turn current single call enable/disables into two socket commands (one to find the server ids and the next to send the command to the right server ids) but would be reasonably clean from the point of view of not having to modify so many CLI apis. Maybe something like: get servers / Automation could then lookup the server id from the address and then send appropriate commands by existing / CLI interfaces? Actually thinking about this more, why can't the service discovery system just maintain the map of identifier+host+port -> haproxy backend ids which are just numeric and ascending (e.g. s1-20)? The system would just have s1-20, and internally map "2.2.2.2:8080" to "s2". As the discovery system gets updates, it can change the mapping as it needs. It has to have some sort of understanding of which ids are available anyways, so this shouldn't be much extra complexity. Then the only thing we have to solve is showing the host+port in the stats UI, which is probably a pretty straightforward change. I sort of like "this is the service discovery system's job". It means the HAProxy patch is basically only to the stats UI, which is pretty easy to do presumably. -Joey
Re: [PATCH] MINOR: cli: ability to change a server's name
Hi Joey,
On Thu, Jun 22, 2017 at 04:20:36PM -0700, Joseph Lynch wrote:
> Hi Willy,
>
> > OK so if we add a string as the external identifier (let's at least
> > give a name to this thing) and the ability to lookup by transport
> > address, you have everything :
> > - the mesos/kubernetes identifier directly accessible
> > - the IP:port directly accessible
> >
> > Did I miss anything ?
>
> An external identifier that socket commands can be passed as an
> acceptable would definitely solve the problem. I guess by
> default the external identifier can just be the name, and we'd prefer
> matching servers by name before external_identifier?
That makes me think about the notion of alias I was thinking about and
I still don't feel much comfortable with it because that would cause
trouble if there are overlapping names. But we can simply have a way
to indicate that we're asking for the external name instead of the
internal one. It might be by forcing external names to start with a
specific character, it might be via the use of a different delimitor,
via a specific option, I think we have plenty of choice.
> An alternative we could add a lookup function that returns server
> names matching an address. This would turn current single call
> enable/disables into two socket commands (one to find the server ids
> and the next to send the command to the right server ids) but would be
> reasonably clean from the point of view of not having to modify so
> many CLI apis. Maybe something like:
>
> get servers /
>
> Automation could then lookup the server id from the address and then
> send appropriate commands by existing / CLI interfaces?
If that can be useful to you, that's trivial to do. In fact we could even
imagine "get servers {by-addr|by-name|by-id|by-state} [value]"
depending on the type of lookup.
> Actually thinking about this more, why can't the service discovery
> system just maintain the map of identifier+host+port -> haproxy
> backend ids which are just numeric and ascending (e.g. s1-20)?
That's the question I'd expect to ask you ;-)
> The
> system would just have s1-20, and internally map "2.2.2.2:8080" to
> "s2". As the discovery system gets updates, it can change the mapping
> as it needs. It has to have some sort of understanding of which ids
> are available anyways, so this shouldn't be much extra complexity.
> Then the only thing we have to solve is showing the host+port in the
> stats UI, which is probably a pretty straightforward change.
And on the CLI. We could imagine a set of extra commands to make life
easier for service discovery, like "show farm " listing the servers
in an easy to process format (internal id, internal name, external name,
address, enabled/disabled).
> I sort of like "this is the service discovery system's job". It means
> the HAProxy patch is basically only to the stats UI, which is pretty
> easy to do presumably.
Don't worry about the ease to do it in haproxy. Most of the bad designs
we've accumulated over time were caused by trying to do something quick
and dirty. Putting more thoughts on the change to ensure it's properly
done and perfectly matches the use case is much better, it will save us
from having to deal with non-natural uses causing side effects. The cost
of writing 50 more lines is nothing compared to the difficulty to solve
problems purposely created to work around certain limitations!
I was wondering about something else, do service discover tools really
have the notion of the backend/farm/whatever, or do they manipulate
globally unique IDs for servers ? I'm just trying to think whether we
need to support looking up a server via its public ID regardless of
the backend name or if it's better to do it within a backend as is
done today.
Willy
Re: 1.7.6 redirect regression (commit 73d071ecc84e0f26ebe1b9576fffc1ed0357ef32)
❦ 21 juin 2017 11:48 +0200, William Lallemand : >> > This bug was fixed in 1.8 (see commit >> > 9f724edbd8d1cf595d4177c3612607f395b4380e "BUG/MEDIUM: http: Drop the >> > connection establishment when a redirect is performed"). I attached >> > the patch. Could you quickly check if it fixes your bug (it should >> > do so) ? >> > >> > It was not backported in 1.7 because we thought it only affected the >> > 1.8. I will check with Willy. >> >> Thanks, patch fixes the problem (with test config (I'll try to >> test with prod. config later)). >> >> -Jarno >> > > Thanks for tests, I will backport it in the 1.7 branch. Is the patch important enough to warrant a 1.7.7 soon? Notably, should downstreams continue to push 1.7.6 to users? -- Parenthesise to avoid ambiguity. - The Elements of Programming Style (Kernighan & Plauger)
Irregular Set-Cookie behavior
Hi, I'm having another issue with HAProxy (v1.4.22). This time related to cookies that HAProxy sets for persistance. When I run a load test with JMeter I have noticed that some, but not all, new threads receive a Set-Cookie header. I'm thinking that this might be due to that I'm also using a sticky-table to maintain session stickiness on source IP address potentially rendering the cookie redundant, but I am not sure. (I have to stick on source IP since not all clients support cookies and session state needs to be maintained for each client). During the load tests with JMeter, I am relying on the set-cookie header to monitor and verify which thread gets routed to which server during the test. The relevant parts of the config file is included below. Thanks for any advice! / Mats Config: defaults modehttp log global option httplog option dontlognull option http-server-close #option forwardfor except 127.0.0.0/8 option redispatch retries 3 timeout http-request10s timeout queue 1m timeout connect 10s timeout client 1m timeout server 1m timeout http-keep-alive 10s timeout check 10s maxconn 128 listen express 127.:... cookie GEAR insert indirect nocache option httpchk GET / http-check expect rstatus 2..|3..|401 option httpclose # route all requests containing the word "foo" in the URL to the same server acl swi path_sub foo stick-table type ip size 200k expire 5m stick on src ipmask(0.0.0.0) if foo stick on src unless foo balance leastconn # more servers automatically added as load increases server local-gear 127...:... check fall 2 rise 3 inter 2000 cookie local-abc.
Re: Reverse Gateway Throught Security Zones
Lukas, Ha, I like the comment about DMZs being a concept from 1999 :-). Sorry if I'm going slightly off topic. We put a comic style picture at the bottom of this blog about, “Our DMZ is so secure we can’t even get into it!” https://www.loadbalancer.org/blog/what-exactly-is-a-reverse-proxy I find people are constantly trying to 'work around the DMZ' rather than just getting rid of them. And don't get me started on bridges: https://www.loadbalancer.org/blog/transparent-vs-explicit-proxy-which-method-should-i-use#bridge-mode Malcolm Turnbull Loadbalancer.org Ltd. www.loadbalancer.org +44 (0)330 380 1064 malc...@loadbalancer.org On 23 June 2017 at 00:05, Lukas Tribus wrote: > Hello Himer, > > > this is probably not the response you wanna hear ... > > > > Am 22.06.2017 um 22:47 schrieb Himer Martinez: >> Hello Guys, >> >> Sorry to botter you with my specific questions :-) >> >> Let's imagine a paranoic security team who forbide http and tcp flows >> between the dmz zone and the green zone, they estimate that if an hacker can >> take control on the dmz zone server they the can access the green zone from >> that server, so flows going from the dmz zone to the green zone are >> forbidden and blocked by network firewalls, >> >> First idea : So what I need is to create something like a reverse tunnel >> between the green zone and HAProxy, > > Clarification: what you or your security team is saying is: > > - a DMZ host establishing a TCP connection to the green zone > is insecure (even if the only open port is HTTP) > - a green zone host establishing whatever bidirectional connections > to the DMZ servers is secure > > Is that a correct interpretation? > > > >> >> (requests are going from the dmz zone to the green zone with a reverse >> connection) > > So by reverse tunneling you basically circumvent your > firewalls and any security policies that may be in place. > > You are opening the "DMZ --> Green Zone" path, just in a less > direct way, and most likely without or with less considerations > regarding security. > > > >> Forbidden : >> Internet --> DMZ --> Green Zone >> >> Authorized : >> Internet --> DMZ <--- Green Zone > > This is a ridiculous concept. DMZ needs Green Zone data, either move > your Green Zone hosts into the DMZ or make the service you need > reachable (considering security aspects, of course). > > By reverse-tunneling you don't gain any security advantage, instead, you > are over complicating your setup, bypassing most likely restrictive firewalls, > opening an attack surface you are not considering. > > > >> First idea : So what I need is to create something like a reverse tunnel >> between the green zone and HAProxy, > > What you need to do is analyze your *REAL* requirements, from a security > perspective > and otherwise, and then build a concept around it. > > Instead you are slamming a 1999 "perimeter security" concept on your network > which > doesn't match your requirements and are now trying to circumvent the > perimeters, > because otherwise you are unable to provide whatever service you need to run. > > > Now to the part that you do wanna hear: > > How you can one best bypass a perimeter firewall that is blocking one > direction > of traffic but not the other? Use any VPN that you are familiar with, as that > is > exactly what they are built for. OpenVPN, strongSwan, etc. > > > > cheers, > lukas > > > >
