Re: A lot of 4xx request errors

[Willy Tarreau]

Hi,

On Fri, Apr 25, 2014 at 02:51:36AM +, Stefan wrote:
> Hello,
> 
> I'm stuck with one issue. Can you help me, please.
> 
> I have a service that gets about 1K connections/second 
> and 15K requests/sec in top. 
> And my service should response maximum in 120 ms.
> The client, that sends me these requests within keep-alive connection.
> 
> But i have a lot of 400 and 408 errors.
> Example:
> ...  -1/-1/-1/-1/50185 408 212 - - 
> cR-- 5236/1728/0/0/0 0/0 ""
> ...  -1/-1/-1/-1/13282 400 187 - - 
>CR-- 5169/3506/0/0/0 0/0 ""
> 
> >From the docs, i found the explanation of these errors:
> - for the first one:
>he client never completed its request, which was aborted by the
>time-out ("c---") after 50s, while the proxy was waiting for the 
> request
>headers ("-R--").  Nothing was sent to any server, but the proxy could
>send a 408 return code to the client.
> - for the second one: 
>the client never completed its request and aborted itself ("C---") 
> after
>8.5s, while the proxy was waiting for the request headers ("-R--").
>Nothing was sent to any server.
> 
> As i understand here, the client established connection 
> and in first case didn't send any request 
> during 50 sec (i'm closing the connection with 408 code). 
> And in second case the client aborting 
> connection during which no request was send.

Unfortunately, this has became the new norm for all of us after Google Chrome
invented the pre-connect feature. It decides to connect to web sites you
recently visited just in case you'd want to go there. And it litterally
kills the internet by establishing tons of useless connections to many
web sites, some of which cannot cope with the load and who have to shrink
their request timeouts. At least on haproxy, you can :

   1) lower the HTTP request timeout : "timeout http-request 5s"
   2) disable logging of empty requests : "option dontlognull"

It's a real disaster but they find it beneficial for end user's
experience. I believe Firefox does the same now :-(

> May be my settings (e.g. sysctl.conf or something else) 
> are not optimized for such load?

Your settings look really fine.

> And the main problem is that i'm not a system administrator, 
> i'm a developer ((( 
> But for the last couple of months i had to read so many docs )))

It seems you read the right docs :-)

> Another question:
> Is it possible to abort request with code 204 
> if it takes more than some time (e.g. 120 ms)?

It's 408 which is expected when the request does not come, not 204. If
for any reason you want to send a different code, you can rewrite the
error message using "errorfile 408". If you want to abort the request
processing on the server, you can simply use "timeout server 120" and
then an error 504 will be emitted. Similarly if you want to change that
error code, you can do it using "errorfile 504".

But I'd really advise you against changing the standard HTTP status
codes for error cases.  For example, 204 cannot report any message
since it forbids any contents.

Regards,
Willy

Re: WebSocket not working on haproxy-1.5-dev23

[Willy Tarreau]

Hi Stefan,

On Fri, Apr 25, 2014 at 03:05:56AM +, Stefan wrote:
> Hello,
> 
> After upgrading to dev23 version, WebSocket stopped working:
> 
> Error during WebSocket handshake: Unexpected response code: 301 

I think it's related to the bug I reported yesterday, causing sessions
to timeout. Please use latest snapshot (20140425) which fixes it.

Willy

WebSocket not working on haproxy-1.5-dev23

[Stefan]

Hello,

After upgrading to dev23 version, WebSocket stopped working:

Error during WebSocket handshake: Unexpected response code: 301 


With Regards, Stefan

A lot of 4xx request errors

[Stefan]

Hello,

I'm stuck with one issue. Can you help me, please.

I have a service that gets about 1K connections/second 
and 15K requests/sec in top. 
And my service should response maximum in 120 ms.
The client, that sends me these requests within keep-alive connection.

But i have a lot of 400 and 408 errors.
Example:
...  -1/-1/-1/-1/50185 408 212 - - 
cR-- 5236/1728/0/0/0 0/0 ""
...  -1/-1/-1/-1/13282 400 187 - - 
   CR-- 5169/3506/0/0/0 0/0 ""

>From the docs, i found the explanation of these errors:
- for the first one:
   he client never completed its request, which was aborted by the
   time-out ("c---") after 50s, while the proxy was waiting for the request
   headers ("-R--").  Nothing was sent to any server, but the proxy could
   send a 408 return code to the client.
- for the second one: 
   the client never completed its request and aborted itself ("C---") after
   8.5s, while the proxy was waiting for the request headers ("-R--").
   Nothing was sent to any server.

As i understand here, the client established connection 
and in first case didn't send any request 
during 50 sec (i'm closing the connection with 408 code). 
And in second case the client aborting 
connection during which no request was send.

May be my settings (e.g. sysctl.conf or something else) 
are not optimized for such load?
And the main problem is that i'm not a system administrator, 
i'm a developer ((( 
But for the last couple of months i had to read so many docs )))

Another question:
Is it possible to abort request with code 204 
if it takes more than some time (e.g. 120 ms)?

The configuration file is:

global
daemon
pidfile /var/run/haproxy-3.pid
maxconn 25
tune.bufsize8024
log 127.0.0.1 local0

defaults
log global
mode http
option httplog
#option dontlognull
option dontlog-normal

no option httpclose
option http-server-close
no option forceclose
option forwardfor
balance roundrobin
option redispatch
retries 3

timeout client 50s
timeout http-keep-alive 30s
timeout server 50s
timeout connect 10s

frontend http_front
maxconn 3
bindxxx.xxx.xxx.xxx:80
reqadd X-Scheme:\ http

acl is_value path_beg /some/path/
use_backend some_backend if is_value

backend some_backend
option http-server-close

server server1.1 xxx.xxx.xxx.xxx:8101 weight 1 
   maxconn 100 check port 8101
server server1.2 xxx.xxx.xxx.xxx:8102 weight 1 
   maxconn 100 check port 8102
i have about 32 app instances on 4 servers .



Here sysctl.conf:
# TCP tunning

# Do a 'modprobe tcp_cubic' first
net.ipv4.tcp_congestion_control = cubic

# Turn on the tcp_window_scaling
net.ipv4.tcp_window_scaling = 1

# Increase the maximum total buffer-space allocatable
# This is measured in units of pages (4096 bytes)
net.ipv4.tcp_mem = 65536 131072 262144
#net.ipv4.tcp_mem = 4096 1048576 16777216
net.ipv4.udp_mem = 65536 131072 262144

# Increase the read-buffer space allocatable
net.ipv4.tcp_rmem = 8192 87380 16777216
#net.ipv4.tcp_rmem = 4096 1048576 16777216
net.ipv4.udp_rmem_min = 16384
net.core.rmem_default = 131072
#net.core.rmem_default = 1048576
net.core.rmem_max = 16777216

# Increase the write-buffer-space allocatable
net.ipv4.tcp_wmem = 8192 65536 16777216
#net.ipv4.tcp_wmem = 4096 1048576 16777216
net.ipv4.udp_wmem_min = 16384
net.core.wmem_default = 131072
#net.core.wmem_default = 1048576
net.core.wmem_max = 16777216

# Increase number of incoming connections backlog
#net.core.netdev_max_backlog = 6
net.core.netdev_max_backlog = 4096
net.core.dev_weight = 64

# Increase number of incoming connections
net.core.somaxconn = 6

# Increase the maximum amount of option memory buffers
net.core.optmem_max = 65536 # 20480

# Increase the tcp-time-wait buckets pool size 
# to prevent simple DOS attacks
net.ipv4.tcp_max_tw_buckets = 144 # 131072
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_tw_reuse = 1

# Limit number of orphans, each orphan can eat up to 16M 
# (max wmem) of unswappable memory
net.ipv4.tcp_max_orphans = 16384
net.ipv4.tcp_orphan_retries = 0

# don't cache ssthresh from previous connection
net.ipv4.tcp_no_metrics_save = 1 # 0
net.ipv4.tcp_moderate_rcvbuf = 1

net.core.netdev_budget = 3
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_fack = 1
net.ipv4.tcp_fin_timeout = 10
net.ipv4.tcp_keepalive_intvl = 10
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_low_latency = 0

net.ipv4.tcp_max_syn_backlog = 65535
net.ipv4.tcp_slow_start_after_idle = 0
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_synack_retries = 3
net.ipv4.tcp_syncookies = 1

Re: HAproxy and Mysql

[Ben Timby]

My only feedback is that haproxy has a lot of features that make it useful
as a MySQL frontend. The stats are great for sizing and monitoring
purposes. Timeouts and queuing are also great for managing load etc. I used
to run haproxy in front of a single MySQL instance for those features alone
ala:

http://flavio.tordini.org/a-more-stable-mysql-with-haproxy

If you are looking to load balance multiple database servers, I think
haproxy is a good choice for doing that.

It will work great as long as everything is functioning normally, but you
will need to put a lot of work into handling failures and master migration
etc. These things haproxy has nothing directly to do with. Here is some
information on handling failure cases etc. using a simple agent along with
haproxy. It is old information, but should be useful.

http://www.alexwilliams.ca/blog/2009/08/10/using-haproxy-for-mysql-failover-and-redundancy/




On Thu, Apr 24, 2014 at 9:46 AM, Alexandre  wrote:

> Hello everyone,
>
> I'm looking for documentation to make a load balancer for mysql.
>
> I found this article :
> https://www.digitalocean.com/community/articles/how-to-use-
> haproxy-to-set-up-mysql-load-balancing--3
>
> What do you think?
>
> We also perform a test with LVS load balancing for mysql.
>
> Have you feedback of this load balancer.
>
>
> Thank you
>
> Alexandre
>
>

Financez vos obsèques à partir de 5 euros

[FamilyProtect par charles]

Title: Assurance Obsèques FamilyProtect - Société du groupe AXA
			Cliquez ici pour lire cet e-mail dans votre navigateur.	Saviez-vous que des obsèques coûtent en moyenne 3.800 € ?	Financez les vôtres à partir de 5 €/ mois !	Jusqu'à 10.000 €de montant versé *	Je demande + d'infossans engagement		Assistance		Administrative etpsychologique7 jours / 7	Coaching		Conseilssanté pourbien vieillir	Analyse		Analyse de vosdevis pompesfunèbres	Réduction		10% deremise pourvotre conjoint	Souscription		Simple etrapide enquelques clics	Capital		Versé sous48h et exonéréd'impôts	* Jusqu'à 10.000 € de capital versé à vos proches en cas de décès.Contrat d'assurance vie individuelle proposé par FamilyProtect, Entreprise régie par le Code des Assurances, siège social : 21 Avenue Matignon 75008 PARIS. SA au capital social de 89 037 000 € - 528 115 926 R.C.S. PARIS et contrôlée par l'Autorité de Contrôle Prudentiel - 61, rue Taitbout - 75436 Paris cedex 09. Application des garanties selon les modalités précisées dans les Conditions Générales. Offre réservée aux résidents français âgés d'au moins 40 ans et de moins de 86 ans. Possibilité de renoncer au contrat dans les 30 jours suivant la souscription. L'utilisation du capital décès est li

Re: haproxy-1.5-dev23 and ssl handshake failure

[Stefan]

Hello,

Here the configuration:

global
daemon
pidfile /var/run/haproxy-3.pid

maxconn 25

tune.bufsize8024

log 127.0.0.1 local0

defaults
log global

mode http
option httplog
#option dontlognull
option dontlog-normal

no option httpclose
option http-server-close
no option forceclose

option forwardfor
balance roundrobin

option redispatch
retries 3

timeout client 50s
timeout http-keep-alive 30s
timeout server 50s
timeout connect 10s

frontend https_rtb_lamoda25_ru
maxconn 3
bindxxx.xxx.xxx.xxx:443 ssl crt /opt/crt/domain.com.pem

reqadd X-Scheme:\ https

acl is_value path_beg  /some/path/
use_backend some_bacjend if is_value

acl is_value_2 path_beg  /some/path-2/
use_backend some_backend_2 if is_value_2

other backends 

backend some_backend
option http-server-close

server server1.1 xxx.xxx.xxx.xxx:8101 weight 1 
  maxconn 100 check port 8101
server server1.2 xxx.xxx.xxx.xxx:8102 weight 1 
  maxconn 100 check port 8102

I guess yes, i started see then in April. 
And ssl load balancing was working about for last 6 months.

With Regards, Stefan

You Click Domex Contributes!

[Domex]

If you're having trouble viewing this email, please 


		
			
   

 
You have received this mailer from us because you indicated that you would like to receive special offers.To unsubscribe from this offer, please click here to unsubscribe.

Choisissez la mutuelle qui vous correspond

[Mutuelle direct]

Cliquez ici pour lire cet e-mail dans votre navigateur.		Désinscrivez vous ici		

Regression with 1.5-dev23

[Willy Tarreau]

Hi all,

I found a nasty regression in dev23, it enforces the client/server timeout
during HTTP transfers and does not report any issue because it believes it
reached the end and closes the connection. This means that if you have a 30
second timeout and a visitor downloads a large file that needs more than 30
seconds to come, it will be truncated. This has been fixed in the git
repository and will be in the next snapshot in a few hours.

This bug has also caused some breakage for some downloads from the main site,
I could see in the logs that a number of 1.5-dev23 and 1.4.25 tarballs were
truncated (which can easily be detected by checking their md5sum).

I'll issue another version quickly after the snapshot, please test it as I
did other changes in it : I fixed the bug that Igor reported long ago with
the stats page in chunked mode, in fact it was not the stats page but a bug
at a much lower layer in the buffer management. It could have caused some
peers to occasionally disconnect/reconnect as well. And after fixing the
stats bug, I enabled keep-alive and compression on the page.

I already noticed that gitweb and the stats page compress at about 80%, this
already saves some bandwidth on my poor line :-)

Regards,
Willy

Re: Regarding Haproxy

[Willy Tarreau]

Hi,

On Thu, Apr 24, 2014 at 10:52:46PM +0530, ajay kumar wrote:
> Dear Sir,
> 
> I am getting syntex error in haproxy configuration after inserting below
> line :-
> 
> stick-table type ip size 500k expire 30s store
> conn_cur,conn_rate(10s),http_req_rate(10s),http_err_rate(10s)
> 
> as stick-table: unknown argument 'store'.
> 
> Please help me out in this error.
> Thanks in advance.

You're running too old a version for this. I suspect you found the
configuration on a blog and are using an old binary from a distro
or something like this. "store" is only support in 1.5-dev.

Regards,
Willy

Regarding Haproxy

[ajay kumar]

Dear Sir,

I am getting syntex error in haproxy configuration after inserting below
line :-

stick-table type ip size 500k expire 30s store
conn_cur,conn_rate(10s),http_req_rate(10s),http_err_rate(10s)

as stick-table: unknown argument 'store'.

Please help me out in this error.
Thanks in advance.

With Regards
Ajay Raj

Goodbye from our Newsletter

[Tu Informe]

  
  Goodbye from our Newsletter, sorry to see you go.

  You have been unsubscribed from our newsletters.

  This is the last email you will receive from us. Our newsletter system,
phpList,
  will refuse to send you any further messages, without manual intervention
by our administrator.

  If there is an error in this information, you can re-subscribe:
  please go to http://tuinforme.com.ar/lists/?p=subscribe and follow the
steps.

  Thank you
  
  

HAproxy and Mysql

[Alexandre]

Hello everyone,

I'm looking for documentation to make a load balancer for mysql.

I found this article :
https://www.digitalocean.com/community/articles/how-to-use-haproxy-to-set-up-mysql-load-balancing--3

What do you think?

We also perform a test with LVS load balancing for mysql.

Have you feedback of this load balancer.


Thank you

Alexandre

Re: Rule to serve /docs via. nginx not working with trailing slash

[Nenad Merdanovic]

Hello Nick,


I am pretty sure this is an nginx problem, but let's resolve it anyways :)

On 04/24/2014 12:54 PM, Nick Jennings wrote:
> ...
> 
> However when I try to use:
>   http://example.com/docs(note: no trailing slash)
> 
>  I get a long wait, nothing in the nginx logs, and then eventually the
> URL in the browser is rewritten to: example.com:8090/docs
>  and fails.

I guess you wanted to say: gets rewritten to: example.com:8090/docs/

You can prevent nginx from adding a port which breaks this by setting:
port_in_redirect off; [1]

> 
> Any ideas what I've got setup wrong here? Seems pretty straightforward
> to me and I'd like to avoid trying to hack things with redirecting to
> force a slash. Should just work. Initially I thought it was an nginx
> thing, but since I don't even get anything in the nginx logs on the
> second method (/docs) I assume the failure is happening within haproxy.

Other thing that might fix your issue is to include this in your nginx
configuration:
try_files $uri $uri/ =404;

This will essentially

> Thanks in advance for any help!
> -Nick
> 

[1] http://nginx.org/en/docs/http/ngx_http_core_module.html#port_in_redirect

Regards,
-- 
Nenad Merdanovic | PGP: 0x423edcb2 | Web: http://nimzo.info
Linkedin: http://www.linkedin.com/in/nenadmerdanovic

Rule to serve /docs via. nginx not working with trailing slash

[Nick Jennings]

Hi All,

 I've got a pretty simple setup at the moment. A node.js application (REST
API) which is listening on port localhost 8080, and a collection of static
html files which I'd like to be served from nginx listening on localhost
port 8090.

---

frontend public
bind *:80

acl is_api hdr_end(host) -i *api.example.com *
acl is_doc path_beg -i /docs

use_backend apidoc if is_api is_doc

default_backend api

backend api
timeout server 30s
option httpclose
option forwardfor
server api1 127.0.0.1:8080 #check

backend apidoc
timeout server 30s
option httpclose
option forwardfor
server api1 127.0.0.1:8090 #check

---

This setup works fine when I access:
   http://example.com/docs/

I see nginx logs, and the page loads (though it takes a couple seconds, not
sure if that is relevant).

However when I try to use:
  http://example.com/docs(note: no trailing slash)

 I get a long wait, nothing in the nginx logs, and then eventually the URL
in the browser is rewritten to: example.com:8090/docs and fails.

Any ideas what I've got setup wrong here? Seems pretty straightforward to
me and I'd like to avoid trying to hack things with redirecting to force a
slash. Should just work. Initially I thought it was an nginx thing, but
since I don't even get anything in the nginx logs on the second method
(/docs) I assume the failure is happening within haproxy.

Thanks in advance for any help!
-Nick

Re: haproxy-1.5-dev23 and ssl handshake failure

[Markus Rietzler]

>> my problem is, that i sometimes see an error message in my browser. i
>> also got one response from a user saying that he can't access our
>> ssl-pages and gets an error.
> 
> There are 2 issues here:
> - the fact that you sometimes (?) see this error in the browser
> - the fact that one user can't open the ssl-page at all (likely he has
>   a browser or SSL middlebox incompatible with your SSL settings)
> 
i try to confirm this (as it happens randomly its not that easy).

> 
> Markus, please follow Willy's advise and remove all force-* configurations
> from your bind line, you should use no-sslv3/no-tlsv1[0-2] keywords to
> configure specific TLS version, but in this case, as long as you
> troubleshooting this, I strongly suggest to not configure any specific TLS
> settings.
i have now removed them. my thought was to prevent use of "weaker" ssl-versions 
(like sslv2), but i found in the docs
that this is deactivated per default. so no real need to force "newer", as 
sslv3 and tlsv1x are used per default.
> 
> Also, we need the haproxy -vv output. You said you started running SSL
> on haproxy April, 8 th, but dev23 was only released these days. So what
> release did you run previsouly, and did you have the same problems (in
> the browsers, not the log)?
> 

i have activated ssl loadbalancing on 8th of april (not because of heartbleed).
so i have only numbers starting at 8th of april. while testing i used ssl 
loadbalancing before and saw a few errors,
that stopped me from activating ssl load balancing in haproxy in the first run.

i have used all versions starting from 1.5 dev19 to now dev23.


./haproxy -vv
HA-Proxy version 1.5-dev23-8317b28 2014/04/23
Copyright 2000-2014 Willy Tarreau 

Build options :
  TARGET  = linux2628
  CPU = generic
  CC  = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing
  OPTIONS = USE_OPENSSL=yes

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built without zlib support (USE_ZLIB not set)
Compression algorithms supported : identity
Built with OpenSSL version : OpenSSL 1.0.1 14 Mar 2012
Running on OpenSSL version : OpenSSL 1.0.1 14 Mar 2012
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built without PCRE support (using libc's regex instead)
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT 
IP_FREEBIND

Available polling systems :
  epoll : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.


> 
> [1] https://www.ssllabs.com/ssltest/
> 
everything is OK,
i see sslv2 is disabled ;-) just what i wanted when first using force-

RE: haproxy-1.5-dev23 and ssl handshake failure

[Lukas Tribus]

Hi,



> I've checked my own logs and found SSL handshake failures starting
> on April 8th, or the day after Heartbleed was disclosed, as can be
> seen below with the number of errors per day :

Yes, please everyone specify whether there are actually users reporting
this behavior, or if this is a log error only. We will see a lot of
automated Heartbleet exploiting the next months, I'm sure.

Check if a test @ssllabls [1] and others generates such an error.



> my problem is, that i sometimes see an error message in my browser. i
> also got one response from a user saying that he can't access our
> ssl-pages and gets an error.

There are 2 issues here:
- the fact that you sometimes (?) see this error in the browser
- the fact that one user can't open the ssl-page at all (likely he has
  a browser or SSL middlebox incompatible with your SSL settings)


Markus, please follow Willy's advise and remove all force-* configurations
from your bind line, you should use no-sslv3/no-tlsv1[0-2] keywords to
configure specific TLS version, but in this case, as long as you
troubleshooting this, I strongly suggest to not configure any specific TLS
settings.

Also, we need the haproxy -vv output. You said you started running SSL
on haproxy April, 8 th, but dev23 was only released these days. So what
release did you run previsouly, and did you have the same problems (in
the browsers, not the log)?


Exact browser and OS release informations are needed as well.



[1] https://www.ssllabs.com/ssltest/  

Re: haproxy-1.5-dev23 and ssl handshake failure

[Apollon Oikonomopoulos]

Hi all,

On 22:59 Wed 23 Apr , Willy Tarreau wrote:
> Hi again Markus,
> 
> I've checked my own logs and found SSL handshake failures starting
> on April 8th, or the day after Heartbleed was disclosed, as can be
> seen below with the number of errors per day :
> 
>   # err date
>   2 Mar 27
>   2 Mar 28
>   1 Mar 29
>   2 Mar 30
>   3 Mar 31
>   3 Apr  1
>   7 Apr  2
>   1 Apr  3
>   2 Apr  4
>   8 Apr  5
>  24 Apr  6
>   2 Apr  7
> 619 Apr  8
>   2 Apr  9
>   2 Apr 10
> 158 Apr 11
>   6 Apr 12
>   2 Apr 13
> 158 Apr 14
> 157 Apr 15
> 168 Apr 16
> 109 Apr 17
>   7 Apr 18
>   7 Apr 19
>   7 Apr 20
> 110 Apr 21
> 497 Apr 22
> 123 Apr 23
> 
> Interestingly, my version was neither upgraded nor restarted during this
> period, so it cannot be caused by a code change, and is very likely caused
> by bots trying the attack. So I think it's also possible that you're
> experiencing the same things and that you didn't notice them before
> upgrading and checking your logs.
> 
> Hoping this helps,
> Willy
> 
> 

We see similar results with -dev19:

 20140401378
 20140402922
 20140403346
 20140404370
 20140405807
 20140406501
 20140407445
 20140408   3509
 20140409360
 20140410   1143
 20140411   1525
 20140412989
 20140413991
 20140414   1217
 20140415   1139
 20140416   1141
 ...

Note the spike on the 8th of April, matching the Heartbleed hypothesis. 

These can be all sorts of failures occurring before the handshake is 
completed. I sampled a couple of requests using tcpdump: one of them was 
a plain HTTP request on the HTTPS port and in the other one the client 
sent a close-notify TLS alert, 250 ms after receiving the certificate 
(indicating perhaps a network issue).

To put things in perspective, on the 8th of April we had a total of 1.38 
million SSL connections¹ so these failures account for roughly 0.25%.  
Granted that on that day we were expecting a lot of unfinished 
handshakes probing the heartbeat vulnerability, I wouldn't worry much.

¹ actually unique source IP:source port entries

Regards,
Apollon

Re: haproxy-1.5-dev23 and ssl handshake failure

[Markus Rietzler]

Am 24.04.14 03:19, schrieb Stefan:
> We also have a lot of "SSL handshake failure" records in log file
> 
> Here some details on configs:
> 
> - haproxy -vv:
> HA-Proxy version 1.5-dev23-8317b28 2014/04/23
> Copyright 2000-2014 Willy Tarreau 
> 
> Build options :
>   TARGET  = linux2628
>   CPU = native
>   CC  = gcc
>   CFLAGS  = -m64 -march=x86-64 -O2 -march=native -g -fno-strict-aliasing
>   OPTIONS = USE_LINUX_SPLICE=1 USE_LINUX_TPROXY=1 USE_LIBCRYPT=1 USE_ZLIB=1 
> USE_OPENSSL=1 USE_STATIC_PCRE=1
> 
> Default settings :
>   maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200
> 
> Encrypted password support via crypt(3): yes
> Built with zlib version : 1.2.8
> Compression algorithms supported : identity, deflate, gzip
> Built with OpenSSL version : OpenSSL 1.0.1e 11 Feb 2013
> Running on OpenSSL version : OpenSSL 1.0.1e 11 Feb 2013
> OpenSSL library supports TLS extensions : yes
> OpenSSL library supports SNI : yes
> OpenSSL library supports prefer-server-ciphers : yes
> Built with PCRE version : 8.33 2013-05-28
> PCRE library supports JIT : no (USE_PCRE_JIT not set)
> Built with transparent proxy support using: 
> IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
> 
> Available polling systems :
>   epoll : pref=300,  test result OK
>poll : pref=200,  test result OK
>  select : pref=150,  test result OK
> Total: 3 (3 usable), will use epoll.
> 
> 
> 
could you send our ssl config in haproxy?
did you see those errors after 8th of april like willy and me (i have activated 
ssl loadbalancing on 8th of april, so i
can't compare before heartbleed)

markus

Re: haproxy-1.5-dev23 and ssl handshake failure

[Markus Rietzler]

Am 23.04.14 22:59, schrieb Willy Tarreau:
> Hi again Markus,
> 
> I've checked my own logs and found SSL handshake failures starting
> on April 8th, or the day after Heartbleed was disclosed, as can be
> seen below with the number of errors per day :
> 
>   # err date
>   2 Mar 27
>   2 Mar 28
>   1 Mar 29
>   2 Mar 30
>   3 Mar 31
>   3 Apr  1
>   7 Apr  2
>   1 Apr  3
>   2 Apr  4
>   8 Apr  5
>  24 Apr  6
>   2 Apr  7
> 619 Apr  8
>   2 Apr  9
>   2 Apr 10
> 158 Apr 11
>   6 Apr 12
>   2 Apr 13
> 158 Apr 14
> 157 Apr 15
> 168 Apr 16
> 109 Apr 17
>   7 Apr 18
>   7 Apr 19
>   7 Apr 20
> 110 Apr 21
> 497 Apr 22
> 123 Apr 23
> 
> Interestingly, my version was neither upgraded nor restarted during this
> period, so it cannot be caused by a code change, and is very likely caused
> by bots trying the attack. So I think it's also possible that you're
> experiencing the same things and that you didn't notice them before
> upgrading and checking your logs.
> 
> Hoping this helps,
> Willy
> 
> 
thats really interesting.
i can't compare with my numbers as i have activated ssl loadbalancing on 8th of 
april. i just checked all of my log
files and data, because i first doubt this. so i can't compare my "old" 
numbers. so heartbleed could really be the cause
of the high numbers.

my problem is, that i sometimes see an error message in my browser. i also got 
one response from a user saying that he
can't access our ssl-pages and gets an error.

markus