RE: NOSRV/BADREQ from some Java based clients [SSL handshake issue]
On Tue, Feb 24, 2015 at 01:33:32PM -0700, NuSkooler wrote: Thanks, this has all been very helpful. Unfortunately it seems that some of the pieces to create a debuggable version of these old clients are currently missing here. If I can get that together I'll debug and hopefully find something. Until then, we'll be attempting to route their traffic around HAProxy leaving only newer clients to get the benefit. If I come up with something that works or otherwise useful I'll post it here. I have not yet checked your traces, but since it was mentionned that application protocol http was present in the traces, maybe the client actually wants an explicit support for http advertised by the server. If this is the case, you may want to try to add npn http alpn http (or just one of them) on your bind line so that haproxy advertises them. From the traces it looks like the client support neither NPN, nor ALPN. HAproxy linked against openssl 1.0.1 like in this case also only supports NPN. Another thing: don't trust Wireshark's decode of the Application Data Protocol in a TLSv1 record, it appears completely buggy to me. I also saw spdy on a HTTPS session to a server that doesn't support spdy. Lukas
acl + map
Hi, I have a list of valid cookies associated with client IP, that I try to make match in an acl. The map format is : cookie-value\tip-address\n This acl should do : if (client has cookie plop and plop value lookup in plop.map returns src); then the acl is valid endif I tried things like : acl valid_cookie src %[req.cook(plop),map_str_ip(plop.map)] or acl valid_cookie req.cook(plop),map_str_ip(plop.map) -m ip %[src] but it clearly don't works (error detected while parsing ACL 'valid_cookie' : '%[req.cook(plop),map_str_ip(plop.map)]' or %[src] is not a valid IPv4 or IPv6 address). I maybe misunderstand %[ substitution ? Does anyone here knows the right way to do that ? Maybe the -M switch ? Best regards Joris
MIB
Hi, I want to know if a MIB for HAProxy is available ? Regards, Mathieu
Re: timeout and acl (regression?)
On Wed, Feb 25, 2015 at 09:34:06AM +, Konstantin vz'One Enchant wrote: I have a config: backend some_backend acl is_copy_req method COPY hdr(Destination) acl is_copy_req method PUT hdr(X-Copy-From) timeout server 15m if is_copy_req It's worked in 1.5-dev21. But now, if I run last version (1.5.11) I get error on start: [ALERT] 055/123058 (29566) : parsing [/etc/haproxy/haproxy.cfg:76] : 'timeout server' : unexpected extra argument 'if' after value '15m'. [ALERT] 055/123058 (29566) : parsing [/etc/haproxy/haproxy.cfg:87] : 'timeout server' : unexpected extra argument 'if' after value '15m'. [ALERT] 055/123058 (29566) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg [ALERT] 055/123058 (29566) : Fatal errors found in configuration. ...fail! Is it regression? No, it is an improvement because previously the parser did not detect the extra words on the config line and now it correctly informs you that timeout doesn't take any such argument. Thus in the past you were believing that your config was working fine, while instead, it used to unconditionally apply the 15m timeout to all connections. Best regards, Willy
Re: Integrating a third party library
Hi Mike, On Wed, Feb 25, 2015 at 09:41:35AM +, Mike Zoom wrote: Hi everyone! I want to integrate a third party library, open source, under the Mozilla Public licence v2. The main role of this library is to process an HTTP request header and to return a string which describes the client. OK. Basically, we have the following questions: 1. how could we link our library in HAProxy at compilation time, You just want this lib to be optional, as we do with zlib, openssl, pcre etc. Just patch the makefile to add USE_XYZ=1 to enable it, and possibly pass XYZ_INC and XYZ_LIB to point to the libs' sources when they're not in the system's default path. and 2. how could we write a new function in HAProxy which takes a buffer of data in entry and can return a string (or buffer of data) I think that what you want to implement is a sample fetch function. For example, take a look at the recently introduced req.hdr_names function, which iterates over all request headers and produces a string that can be used to build a log line, another header or whatever. I think it will be straightforward enough for you to understand how to implement this with your lib. Best regards, Willy
problem using haproxy in backports
hi, i encountered problems upgrading from haproxy.debian.net to the backports repo with the following error: Setting up haproxy (1.5.8-2~bpo70+1) ... [] Starting haproxy: haproxy/usr/sbin/haproxy already running. failed! invoke-rc.d: initscript haproxy, action start failed. for this you nedd to: - stop haproxy - relaunch the apt-get upgrade Seems the script has some issue for upgrading existing installs as it try to start and start is designed to fail if the service is allready running :) best regards, Ghislain.
RE: [PATCH 2/2] DOC: Document the new tls-ticket-keys bind keyword
If a site has N haproxy hosts, how should new ticket-keys be distributed (and processes reloaded) and avoid the race condition of some hosts using the new keys before those keys are on all hosts? You distribute the new key to all instances for decryption, but use the penultimate key for encryption instead of the ultimate key: https://blog.cloudflare.com/tls-session-resumption-full-speed-and-secure/ Regards, Lukas
Re: [PATCH 2/2] DOC: Document the new tls-ticket-keys bind keyword
If a site has N haproxy hosts, how should new ticket-keys be distributed (and processes reloaded) and avoid the race condition of some hosts using the new keys before those keys are on all hosts? Seems that not all hosts would be updated at exactly the same time and that until all hosts are updated, that any requests with new ticket-keys that are routed to not-yet-updated hosts will force another full handshake. Seems like a use after time would be needed so that all hosts could start using the new ticket-keys only after some time when they all have all of the keys needed. -Bryan On Wed, Feb 25, 2015 at 10:49 AM, Pavlos Parissis pavlos.paris...@gmail.com wrote: On 25/02/2015 12:10 μμ, Lukas Tribus wrote: -- Use stats socket to update the list without reload -- Update Session state at disconnection log schema to include something useful in case server receives a ticket which was encrypted with key that is not anymore in the list. Debugging SSL problems is a nightmare by definition and having a lot of debug information is very much appreciated by sysadmins If the ticket is not in the list, it will simply fall back to a full handshake, not abort the handshake, so there is no error in that case. Generic SSL/TLS resumption counter should correctly account for those tings already. Error was the wrong word here as RFC 5077 clearly states it as a situation from which both ends can recovery without causing an error. But, you want to avoid the fall-back mechanism as much as possible as it defeats the purpose of TLS session resumption, which is a faster user experience over HTTPS. Thus, you need have a clear way to identify the volume of the traffic which is effected by this. I mentioned about session state at disconnection log schema as way to pass clear information to operator that your key rotation is degrading user experience I guess the generic counter you mentioned could do the trick here. -- Possible use peer logic to sync the list to others, tricky but it is required when you have several LBs, alternatively users can deploy the logic that twitter has used That doesn't make much sense for externally provided tls keys, you may as well use the external interface on all instances. Correct. I only mentioned as an easy way for users that don't have the external interface to facilitate this. This would make more sense for SSL session ids, they are currently shared between processes, but not between different haproxy instances (stud for example can do this iirc). Lukas Thanks getting back to me, Once again thanks to the people who work on this. I guess someone has to inform few bloggers about this in order to update their blog spot where they mention that you can't implement a proper TLS session resumption with HAProxy:-) Cheers, Pavlos
RE: [PATCH 2/2] DOC: Document the new tls-ticket-keys bind keyword
That is a nice solution. I didn't understand that was the behavior from reading the documentation patch from the OP. This makes it sound like the last key is used for encryption and not the next-to-last (penultimate). Correct. Currently there is no choice about which key to use, so maybe we should just default to the penultimate? I agree, we just need to make sure the reason for this is properly explained in the docs. Lukas
Re: [PATCH 2/2] DOC: Document the new tls-ticket-keys bind keyword
On Wed, Feb 25, 2015 at 12:09 PM, Lukas Tribus luky...@hotmail.com wrote: If a site has N haproxy hosts, how should new ticket-keys be distributed (and processes reloaded) and avoid the race condition of some hosts using the new keys before those keys are on all hosts? You distribute the new key to all instances for decryption, but use the penultimate key for encryption instead of the ultimate key: https://blog.cloudflare.com/tls-session-resumption-full-speed-and-secure/ That is a nice solution. I didn't understand that was the behavior from reading the documentation patch from the OP. This makes it sound like the last key is used for encryption and not the next-to-last (penultimate). +tls-ticket-keys keyfile + Sets the TLS ticket keys file to load the keys from. The keys need to be 48 + bytes long, encoded with base64 (ex. openssl rand -base64 48). Number of keys + is specified by the TLS_TICKETS_NO build option (default 3) and at least as + many keys need to be present in the file. Last TLS_TICKETS_NO keys will be + used for decryption and only the last one for encryption. This enables easy + key rotation by just appending new key to the file and reloading the process. -Bryan
Re: [PATCH 2/2] DOC: Document the new tls-ticket-keys bind keyword
Hello Lukas, On 2/25/2015 9:09 PM, Lukas Tribus wrote: If a site has N haproxy hosts, how should new ticket-keys be distributed (and processes reloaded) and avoid the race condition of some hosts using the new keys before those keys are on all hosts? You distribute the new key to all instances for decryption, but use the penultimate key for encryption instead of the ultimate key: https://blog.cloudflare.com/tls-session-resumption-full-speed-and-secure/ Currently there is no choice about which key to use, so maybe we should just default to the penultimate? Regards, Nenad
RE: [PATCH 2/2] DOC: Document the new tls-ticket-keys bind keyword
(sorry, again my mailer messed up ...) That is a nice solution. I didn't understand that was the behavior from reading the documentation patch from the OP. This makes it sound like the last key is used for encryption and not the next-to-last (penultimate). Correct. Currently there is no choice about which key to use, so maybe we should just default to the penultimate? I agree, we just need to make sure the reason for this is properly explained in the docs. Lukas
Re: [PATCH] BUG/MEDIUM: Do not consider an agent check as failed on L7 error
On Thu, Feb 26, 2015 at 07:09:25AM +0100, Willy Tarreau wrote: Hi Simon, On Thu, Feb 26, 2015 at 11:26:17AM +0900, Simon Horman wrote: As failure to connect to the agent check is not sufficient to mark it as failed it stands to reason that an L7 error shouldn't either. Without this fix if an L7 error occurs, for example of connectivity to the agent is lost immediately after establishing a connection to it, then the agent check will be considered to have failed and thus may end up with zero health. Once this has occurred if the primary health check also reaches zero health, which is likely if connectivity to the server is lost, then the server will be marked as down and not be marked as up again until a successful agent check occurs regardless of the success of any primary health checks. This behaviour is not correct as a failed agent check should never cause a server to be marked as down or by extension continue to be marked as down. Signed-off-by: Simon Horman ho...@verge.net.au Makes sense, thanks. Applied to both 1.6 and 1.5. Thanks. It seems that at the very least this problem was reproducible by rebooting backend servers.
Re: [PATCH] BUG/MEDIUM: Do not consider an agent check as failed on L7 error
Hi Simon, On Thu, Feb 26, 2015 at 11:26:17AM +0900, Simon Horman wrote: As failure to connect to the agent check is not sufficient to mark it as failed it stands to reason that an L7 error shouldn't either. Without this fix if an L7 error occurs, for example of connectivity to the agent is lost immediately after establishing a connection to it, then the agent check will be considered to have failed and thus may end up with zero health. Once this has occurred if the primary health check also reaches zero health, which is likely if connectivity to the server is lost, then the server will be marked as down and not be marked as up again until a successful agent check occurs regardless of the success of any primary health checks. This behaviour is not correct as a failed agent check should never cause a server to be marked as down or by extension continue to be marked as down. Signed-off-by: Simon Horman ho...@verge.net.au Makes sense, thanks. Applied to both 1.6 and 1.5. Best regards, Willy
Re: MIB
Hi Mathieu, There is no such MIB for HAProxy. Baptiste On Wed, Feb 25, 2015 at 4:17 PM, Mathieu Sergent mathieu.sergent...@gmail.com wrote: Hi, I want to know if a MIB for HAProxy is available ? Regards, Mathieu
Re: Integrating a third party library
and 2. how could we write a new function in HAProxy which takes a buffer of data in entry and can return a string (or buffer of data) I think that what you want to implement is a sample fetch function. For example, take a look at the recently introduced req.hdr_names function, which iterates over all request headers and produces a string that can be used to build a log line, another header or whatever. I think it will be straightforward enough for you to understand how to implement this with your lib. Best regards, Willy Hi Willy, I think a converter is more suited here. I mean, a fetch can't take a buffer issued from the result of an other fetch... The idea would to configure it something like http-request set-header Foobar req.hdr(HEADER),mikefunction(parameters if required) Mike, in such case, you want to have a look at this file: http://git.haproxy.org/?p=haproxy.git;a=blob_plain;f=src/sample.c;hb=HEAD and with the upper and lower and any other converter functions. Baptiste
Re: [PATCH 2/2] DOC: Document the new tls-ticket-keys bind keyword
On 24/02/2015 04:57 μμ, Nenad Merdanovic wrote: Hello Vincent, Lucas On 2/24/2015 4:56 PM, Lukas Tribus wrote: It would be nice to add a note that without proper rotation, PFS is compromised by the use of TLS tickets. People may not understand why they need to put 3 keys in this file and may never change them. Agreed, we have to clarify that a never changing tls-tickets-keys file is worse than no file at all. Done! I'll wait for more comments from ML before sending the updated patchset. -- Use stats socket to update the list without reload -- Update Session state at disconnection log schema to include something useful in case server receives a ticket which was encrypted with key that is not anymore in the list. Debugging SSL problems is a nightmare by definition and having a lot of debug information is very much appreciated by sysadmins -- Possible use peer logic to sync the list to others, tricky but it is required when you have several LBs, alternatively users can deploy the logic that twitter has used Excellent work guys, thank you. Pavlos signature.asc Description: OpenPGP digital signature
timeout and acl (regression?)
I have a config: backend some_backend acl is_copy_req method COPY hdr(Destination) acl is_copy_req method PUT hdr(X-Copy-From) timeout server 15m if is_copy_req It's worked in 1.5-dev21. But now, if I run last version (1.5.11) I get error on start: [ALERT] 055/123058 (29566) : parsing [/etc/haproxy/haproxy.cfg:76] : 'timeout server' : unexpected extra argument 'if' after value '15m'. [ALERT] 055/123058 (29566) : parsing [/etc/haproxy/haproxy.cfg:87] : 'timeout server' : unexpected extra argument 'if' after value '15m'. [ALERT] 055/123058 (29566) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg [ALERT] 055/123058 (29566) : Fatal errors found in configuration. ...fail! Is it regression?
Re: [PATCH 0/2] Add support for TLS ticket keys configuration
On 02/24/2015 04:42 PM, Nenad Merdanovic wrote: TLS_TICKETS_NO is a build time option, so you can set it to whatever you want. Ok, fair enough. The idea which I discussed with Willy is to build an interface to be able to update the keys via the socket so we don't even have to reload in the future. That would be awesome! Thanks again for this great feature! Best regards, -- Rémi signature.asc Description: OpenPGP digital signature
Integrating a third party library
Hi everyone! I want to integrate a third party library, open source, under the Mozilla Public licence v2. The main role of this library is to process an HTTP request header and to return a string which describes the client. Basically, we have the following questions: 1. how could we link our library in HAProxy at compilation time, and 2. how could we write a new function in HAProxy which takes a buffer of data in entry and can return a string (or buffer of data) Thanks a lot for your help, Mike
RE: [PATCH 2/2] DOC: Document the new tls-ticket-keys bind keyword
-- Use stats socket to update the list without reload -- Update Session state at disconnection log schema to include something useful in case server receives a ticket which was encrypted with key that is not anymore in the list. Debugging SSL problems is a nightmare by definition and having a lot of debug information is very much appreciated by sysadmins If the ticket is not in the list, it will simply fall back to a full handshake, not abort the handshake, so there is no error in that case. Generic SSL/TLS resumption counter should correctly account for those tings already. -- Possible use peer logic to sync the list to others, tricky but it is required when you have several LBs, alternatively users can deploy the logic that twitter has used That doesn't make much sense for externally provided tls keys, you may as well use the external interface on all instances. This would make more sense for SSL session ids, they are currently shared between processes, but not between different haproxy instances (stud for example can do this iirc). Lukas
