HAProxy 1.5

[Kobus Bensch]

Hi

Have you got a date for the final release of 1.5? There are a few 
features in 1.5 we badly need.


Thanks

Kobus

--


Trustpay Global Limited is an authorised Electronic Money Institution 
regulated by the Financial Conduct Authority registration number 900043. 
Company No 07427913 Registered in England and Wales with registered address 
130 Wood Street, London, EC2V 6DL, United Kingdom.


For further details please visit our website at www.trustpayglobal.com.

The information in this email and any attachments are confidential and 
remain the property of Trustpay Global Ltd unless agreed by contract. It is 
intended solely for the person to whom or the entity to which it is 
addressed. If you are not the intended recipient you may not use, disclose, 
copy, distribute, print or rely on the content of this email or its 
attachments. If this email has been received by you in error please advise 
the sender and delete the email from your system. Trustpay Global Ltd does 
not accept any liability for any personal view expressed in this message.

Re: HAProxy 1.5

[PiBa-NL]

Hi

If you need it badly then start using it. (after validating&testing with 
your configuration which you should do anyway.)


The name 'release' wont say there wont be any bugs left. As for the 
current 1.5devX releases lots of people use them in production 
environments and they are in general very stable.


As for testing with your current configuration you should actually start 
doing that a.s.a.p. so if you do find there are problems they can still 
be fixed before the release is called 'final'.


And for a date that would be "1.5 (ETA 2013/12/31)" (see roadmap in 
git.), besides that one and maybe some other estimations you probably 
wont get the actual date it until its really ready.. And it will be 
ready when its ready. As you might have read the 'release' is coming 
closer every day, as most major features are now implemented, and only a 
few development builds will probably be made for some final bug fix 
checks...


Greets PiBa-NL
Find below part of the 1.5dev20 release mail from Willy as the 
mailinglist archives are not containing this.. (followed a day later by 
dev21 to fix a small but annoying issue):

"""

I expect to release 1.5-final around January and mostly focus on chasing
bugs till there. So I'd like to set a feature freeze. I know it doesn't
mean much considering that we won't stop contribs. But I don't want to
merge another large patch set before the release. Ideally there will not
be any dev21 version. Reality probably is that we'll have to issue one
because people will inevitably report annoying bugs that were not reported
in snapshots.

"""

Kobus Bensch schreef op 9-1-2014 17:58:

Hi

Have you got a date for the final release of 1.5? There are a few 
features in 1.5 we badly need.


Thanks

Kobus

about HAProxy 1.5

[Xie Qingshan]

Hello Willy, 

Is there a timeline of v1.5 stable version release?
Is it CPU load high if v1.5 enables SSL termination?

Thanks, Q.Xie

HAProxy 1.5 release?

[Stephen Balukoff]

Hey Willy!

I'm involved in a group that is building a highly-scalable open source
virtual appliance-based load balancer for use with cloud operating systems
like OpenStack. We are planning on making haproxy the core component of the
solution we're building.

At my company we've actually been using haproxy 1.5 for a couple years now
in production to great effect, and absolutely love it. But I'm having
trouble getting the rest of the members of my team to go along with the
idea of using 1.5 in our solution simply because of its "official" status
as a development branch. There are just so many useful new features in 1.5
that I'd really rather not have to go back to 1.4 in our solution...

So! My question is: What can we do to help y'all bring the 1.5 branch far
enough along such that y'all are comfortable releasing it as the official
"stable" branch of haproxy? (Note we do have people in our group with
connections in some of the major linux distros who can help to fast-track
its adoption into "official" releases of said distros.)

Thanks,
Stephen

-- 
Stephen Balukoff
Blue Box Group, LLC
(800)613-4305 x807

question haproxy 1.5

[Labedan, Alain]

Hi,

I have HAPROXY in front of servers backend which are load balanced.

So,  in https, we have only one address where the front https haproxy listen  : 
 bind  :443.
And we have some clients for which, we only pass-through the traffic, so we use 
the mode tcp .

Frontend  https-tcp-in
Mode tcp
Option tcplog
Bind ip1:443
Tcp-request inspect-delay 5s
Tcp-request content accept if { req.ssl_hello_type 1 }
Acl  regle1 req.ssl_sni  -i  
Use_backend site1 if regle1

And we have also some clients for which in https, it is haproxy who have the 
certicate, so we use mode http ?
..
Mode http
Bind :443 ssl crt /etc/ssl/pem
Use_backend site1 if  { ssl_fc_sni  
..

Is it possible to manage both these two situations with only one socket for 
listen https (bind  :443.)  ?

Bests regards
Cordialement,

Alain Labedan
France GDC | CGI | Centre de Compétences IM France
17 place des Reflets, 92097 Paris la Défense cedex | France- 0157877340

alain.labe...@cgi.com |
 [Description : environment-logo-email]
[Description : CGI_Logo2012_color.png]
CGI France SAS
Capital : 12 913 933 euros | RCS Nanterre 702 042 755
Siège social : Immeuble CB16 | 17, place des Reflets | 92400 Courbevoie | France

AVIS DE CONFIDENTIALITÉ : Ce message peut contenir des renseignements 
confidentiels appartenant exclusivement au Groupe CGI inc. ou à ses filiales. 
Si vous n'êtes pas le destinataire indiqué ou prévu dans ce message (ou 
responsable de livrer ce message à la personne indiquée ou prévue) ou si vous 
pensez que ce message vous a été adressé par erreur, vous ne pouvez pas 
utiliser ou reproduire ce message, ni le livrer à quelqu'un d'autre. Dans ce 
cas, vous devez le détruire et vous êtes prié d'avertir l'expéditeur en 
répondant au courriel.​

question haproxy 1.5

[Labedan, Alain]

Hi,

For the front-end in https, we have several clients, so several certificates as 
in the exemple below.
..
Mode http
Bind :443 ssl crt /etc/ssl/certificate1.pem  crt 
/etc/ssl/certificate2.pem  crt ….

But we want to declare a folder for the certificates .

We try
Bind :443 ssl crt /etc/ssl  :   but without success .
Or
Bind :443 ssl crt /etc/ssl/*.pem:   but without success .

Bests regards

Cordialement,

Alain Labedan
France GDC | CGI | Centre de Compétences IM France
17 place des Reflets, 92097 Paris la Défense cedex | France- 0157877340

alain.labe...@cgi.com |
 [Description : environment-logo-email]
[Description : CGI_Logo2012_color.png]
CGI France SAS
Capital : 12 913 933 euros | RCS Nanterre 702 042 755
Siège social : Immeuble CB16 | 17, place des Reflets | 92400 Courbevoie | France

AVIS DE CONFIDENTIALITÉ : Ce message peut contenir des renseignements 
confidentiels appartenant exclusivement au Groupe CGI inc. ou à ses filiales. 
Si vous n'êtes pas le destinataire indiqué ou prévu dans ce message (ou 
responsable de livrer ce message à la personne indiquée ou prévue) ou si vous 
pensez que ce message vous a été adressé par erreur, vous ne pouvez pas 
utiliser ou reproduire ce message, ni le livrer à quelqu'un d'autre. Dans ce 
cas, vous devez le détruire et vous êtes prié d'avertir l'expéditeur en 
répondant au courriel.​

[ANNOUNCE] haproxy 1.5-dev1

[Willy Tarreau]

Hi !

Three months ago I was approached by the Stack Overflow Team team[1] who
needed to get some improvements in HAProxy. Overall, their needs would
have been addressed by the final release of version 1.5 scheduled around
the end of the year, but having to wait that long was not practical due
to some architectural constraints imposed by an intermediate solution.
They proposed that we find an agreement on which we could work together.
Since we were already having productive exchanges for some time, and I
knew they were good guys (after all they already donated to the project
last year), I accepted the deal.

Also, I must say that as a software engineer, it's always a lot better
to have someone explain their needs with high expectations than having
to guess how a feature will be used.

Geoff Dalgas and Jeff Atwood described to me in great details what they
needed to do : perform request throttling per IP address, possibly based
on various criteria, in order to limit risks of service abuse. That was
very interesting, because that feature was being thought about for about
4 years without enough time to completely develop it, and also the new
stickiness framework that was contributed by Exceliance and Loadbalancer.org
was making that really possible, although an important design rework had
to be operated first within the code.

During the tests with Geoff and Kyle Brandt, it appeared that some more
changes had to be operated to be able to store any criteria in the tables
(eg: bandwidth per IP address), and to be able to consult and change the
table contents at runtime, leading to a more and more generic code. Kyle
has been very patient and comprehensive, I think I have changed the
mechanisms and configuration syntax at least 5 or 6 times during the tests,
but he always took the time to understand the changes and adapt his
configurations. If I had been at his place, I would have got bored earlier,
so I owe him a big thanks for his patience !

Now the code has been running fine in production overthere, so it's time
to release it and merge it into the master branch. I won't extend further
on how it works, since Kyle has put an excellent explanation on his blog[2]
that is a lot more clear than the doc (that reminds me that the architecture
guide really needs some lifting).

Also, some of yours will like to get a quick status on the current code.
Some core changes that I wanted to do earlier will now start. But that means
that 1.5-dev1 should theorically be as stable as 1.4.8. I'm not saying that
I would suggest to anyone to push it into production, but it can clearly be
used to mitigate DDoS attacks as well as stop service abuses. I could get it
to stop connection floods slightly above 20 connections per second (yes,
two hundreds thousands) and let the good traffic pass through. So for this
reason, I think that people who are regularly exposed to such trouble may
find it useful to keep it handy.

Now what's next ? Right now the data in the tables is local to one process,
so it is not shared if you start multiple processes, nor it is across reloads.
The second step of the stickiness extensions developped by Exceliance and
Loadbalancer.org will include stickiness table synchronization between
multiple hosts. Some work will still be needed to be able to share counters,
but since this development is done in a flexible way, it should not be too
hard to adapt it later. BTW, I also owe a big kudos to the Git versionning
system, which has made it very easy to rework my patches after every change
and bugfix until they were looking good, through massive abuse of branching
and rebasing.

Too much talk. The code is available here :

   site index : http://haproxy.1wt.eu/
   sources: http://haproxy.1wt.eu/download/1.5/src/devel/
   changelog  : http://haproxy.1wt.eu/download/1.5/src/CHANGELOG
   binaries   : Since this is a development version, no binary is provided.

The last words naturally go to the really cool guys at Stack Overflow. It's
very nice to see some sites and companies involve time and money and take
risks to make Open Source products better. Of course they benefit from this
work, but at no point during the whole development did they try to reduce
the focus to their specific needs, quite the opposite. From the very first
exchanges, their goal clearly was to make the product better, and that must
be outlined. That's now achieved and I really appreciate their involvement.
Thank you guys !

Willy

[1] http://blog.serverfault.com/
[2] 
http://blog.serverfault.com/post/1016491873/better-rate-limiting-for-all-with-haproxy

[ANNOUNCE] haproxy 1.5-dev3

[Willy Tarreau]

Hi,

finally we managed to merge all the stuff ! Haproxy 1.5-dev3 was released
with everything that went into 1.4.9, plus some added bonus that were
mainly developped at Exceliance :

  - support for binding to UNIX socket on the accept side. Haproxy can
now receive connections over a UNIX socket. This is particularly
useful when combined with stunnel (we also have a patch for that
in the 'patches' directory).

  - support for a new "PROXY" protocol that was designed to forward
transport-level information between proxies. The idea is to permit a
component like stunnel to inform haproxy about the protocol, source
and destinations of an incoming connection, so that haproxy can make
use of that everywhere internally (acls, logs, transparent, ...)
instead of stunnel's address. The main advantage over the x-forwarded-for
patch is that it now supports keep-alive and is not limited to HTTP
anymore. When combined with the UNIX socket, it can make haproxy and
stunnel integrate seamlessly and reliably. Obviously, we have a patch
for stunnel ready too ;-)

  - tcp-response filtering : it's possible to wait for some ACLs to match in
the response before forwarding (or blocking).

  - stick-table learning from responses. It's now possible to learn some
patterns from responses and match them again in requests. Doing so
allows haproxy to learn SSL IDs in order to offer SSL-based stickiness
to SSL reverse-proxy farms.

  - stick-table synchronization : the stickiness information in stick-tables
can now be synchronized over the network between as many other haproxies
as you like in a multi-master fashion. Also, during soft-restarts, the
new process learns the table from the old one so that restarts do not
lose that precious information anymore. Designing this was quite a tough
work (Aleks might recall we started talking about such a protocol about
6 years ago now), and is the second half of the large work co-sponsored
by Exceliance[1] and LoadBalancer.org[2]. Now it's completely advisable
to simply rely on source IP for some protocols such as RDP in certain
environments, since restarts will not kill user connections.

For those interested in the last point, the protocol is very cheap over the
wire and is designed with a large window and ACKs, so that it can sync over
high latency networks and even recover from network outages. The sync is fast
enough so that even people using a round-robin L4 LB in front of two haproxies
should not experience any issues under moderate loads (thousands of new entries
per second).

A few typos, minor bugs and error reporting issues were fixed (including the
ones contributed by Cyril a few days ago).

Minor optimizations were performed in order to avoid a few useless operations
in process_session(). The acute observers may notice a tiny drop of CPU usage
(around 5% of user time) from previous versions.

Now you know where to get it :
   site index  : http://haproxy.1wt.eu/
   sources : http://haproxy.1wt.eu/download/1.5/src/devel/
   changelog   : http://haproxy.1wt.eu/download/1.5/src/CHANGELOG
   stunnel patches : http://haproxy.1wt.eu/download/patches/

For the next versions, I'd really like to be able to concentrate on the core
to try to finish the end-to-end keep-alive support. After that there are
less intrusive changes to work on. I'm still hoping for an 1.5 release by
the beginning of next year.

Stay tuned,
Willy
--
[1] http://www.exceliance.fr/
[2] http://www.loadbalancer.org/

[ANNOUNCE] haproxy 1.5-dev4

[Willy Tarreau]

Hi,

no less than 4 months have elapsed since 1.5-dev3. Most of the time was
spent trying to figure what needed to be changed in order to support
server-side keep-alive. After about 20 unpublished experimental branches
that were partly dropped, I finally reached a point where I'm confident
I'm in the right direction, so I decided to merge everything I had that
needed to be kept.

There are 94 patches in this version. Most of them are harmless and have
also been merged into 1.4.13. A few others imply more sensible changes
which should be OK but are still worth a few checks after upgrade :

  - frontend and backend stats are now split. That means that "listen"
instances involved in content switching rules will correctly report
the traffic that went through their frontend and backend parts. I
did that carefully and I think it's OK. But if you notice some stats
reporting oddities, please keep this change in mind and report the
issue ;

  - server-side IPv6 almost done. The work was done at Exceliance by
David du Colombier. We discovered that there were some issues in
some implementations of getaddrinfo() on various systems, so I
decided that for now I'd only merge his changes that were relevant
to the internals, and not the ones which touched the config parser.
That means that at this stage, server-side IPv6 is not supported
and that no existing config should be affected. The merge was not
easy due to massive changes I had performed in the same areas, so
if you notice an address or a port that is not correctly bound
anymore, that's probably a merge bug, so please report it too.

  - http-request rules are now evaluated *before* the stats rule. I
was quite surprized to discover that we could not block access to
stats using an "http-request deny" rule which was supposed to be
parsed earlier. While I don't expect any issue with this change,
maybe some setups were offering a wider access to stats than
expected and will exhibit the restriction.

There were very few user-visible improvements in this version. One of
them is an optimization which results in chunked-encoded data not
being sent one packet per chunk anymore. This can save some network
processing at places where contents are compressed. Another one is
that error captures and dumps are now more complete, they include an
identifier, they support wrapping messages, they can report incorrectly
chunked data. The rest is almost only internal redesign or bugfixes.

As usual, upgrade for 1.5-dev users is recommended with the level of
care due to a development version. I've upgraded the haproxy site to
run on it (demo.1wt.eu). In case important regressions would be reported,
I'd issue 1.5-dev5 with those fixes to make upgrades easier.

For the next step, I intend to merge the second half of the IPv6 work,
review and merge Simon's hot-reconfiguration stuff, then go back to work
on keep-alive.

Usual download URLs below :
   site index  : http://haproxy.1wt.eu/
   sources : http://haproxy.1wt.eu/download/1.5/src/devel/
   changelog   : http://haproxy.1wt.eu/download/1.5/src/CHANGELOG
   stunnel patches : http://haproxy.1wt.eu/download/patches/

Willy

[ANNOUNCE] haproxy 1.5-dev5

[Willy Tarreau]

Hi all,

There were a bunch of minor but annoying issues reported on 1.5-dev4,
it was enough to divert me from other activities, so it was time to
emit a fixed version (so that I can go back review Simon's work).

So I released 1.5-dev5 with several fixes, among which :
  - address parser bug was causing connections to loop on some setups
(David Du colombier)

  - src_conn_cur ACL returing src_conn_cnt instead (Cory Forsyth, Cyril Bonté)

  - stupid values reported in the retries field in logs upon request errors
(Johannes Smith)

  - stats accesses in logs reported as "NOSRV" instead of "STATS"

  - fix for 64-bit content-lengths on 32-bit platforms

  - fix some build issues (eg: CTTPROXY)

It also features a few addons that were pending :

  - full IPv6 with servers, loggers & stick-tables (David du Colombier)
=> some work remains to be done for monitor-net, proxy mode, etc...

  - table_cnt and table_avl ACLs to report usage and space available in
stick tables

  - "send-proxy" : support for the PROXY protocol with servers. It basically
allows haproxy to be daisy chained or installed in front of compatible
servers without losing the initial connection's information.

Users of 1.5-dev4 are strongly encouraged to upgrade.

Theorically (ie: if no big issue appears), 1.5-dev6 should come out with a
few more IPv6 patches and with Simon's work on seamless hot-reconfiguration.

In the mean time, please find the usual download URLs below :
   site index  : http://haproxy.1wt.eu/
   sources : http://haproxy.1wt.eu/download/1.5/src/devel/
   changelog   : http://haproxy.1wt.eu/download/1.5/src/CHANGELOG
   stunnel patches : http://haproxy.1wt.eu/download/patches/

Have fun,
Willy

[ANNOUNCE] haproxy 1.5-dev7

[Willy Tarreau]

Hi all,

Five months have elapsed since 1.5-dev6. A massive amount of changes was
merged since then. Most of them were cleanups and optimizations. A number
of changes were dedicated to making listeners more autonomous. The immediate
effect is a more robust handling of resource saturation, and the second
effect is the removal of the 10-years old maintain_proxies() function which
was harming performance and hard to get over.

Halog was improved too (faster with more filters). A significant number
of external contributions were merged, among them the stats socket updates
to clear session-table keys by values. There are too many changes to list,
but nothing too dangerous, so I'd say it's the 1.5-dev version I trust the
most today.

I'm planning on putting all the focus on server-side keep-alive again. Some
of the remaining issues have been overcome. Surely there are still a number,
but we can't know if we don't try :-)

Do not hesitate to give 1.5-dev7 a try. I'm currently updating all 1.5 I
have to it.

   site index  : http://haproxy.1wt.eu/
   sources : http://haproxy.1wt.eu/download/1.5/src/devel/
   changelog   : http://haproxy.1wt.eu/download/1.5/src/CHANGELOG

Cheers,
Willy

HAProxy 1.5 vs 1.6

[Himer Martinez]

Bonjour,

Je suis désolé de vous spammer, je sais que vous avez autre chose à faire
:-)

Je compte utiliser HAProxy pour un nouveau projet (très grosse
application), des experts me disent que la version stable/éprouvée est la
1.5 mais je vois que la 1.6 est aussi stable et disponible,

Du coup, je voulais savoir quelles étaient les différences exactes et avoir
un conseil sur la version à utiliser...

Vraiment ! Merci d'avance !

Et surtout, BRAVO, pour ce que vous faîtes !

Cordialement

Himer MARTINEZ

[ANNOUNCE] haproxy 1.5-dev8

[Willy Tarreau]

Hi all,

I didn't realize how old 1.5-dev7 was, so it was time to issue a new one,
especially given the number of fixes that went in since.

So 1.5-dev8 brings many fixes and minor optimizations. The ACL framework was
reworked to share the same pattern extraction methods as the stick-table, but
that work is not complete yet.

One major change concerns the log subsystem which was completely rewritten at
Exceliance so that you can now have you own log format. New information can
be logged sur as the outgoing source IP/port, etc... And the bonus is that
doing so improved logging performance :-)

We owe a big thanks to Sander Klein who was brave enough to take significant
risks in production for the sake of offering us an exploitable core caused
by a very recent regression in the logging subsystem. Without his help we
would have an unstable -dev8!

Olufemi Omojola was also kind enough to provide a core and config showing
that mixing captures and TCP mode was unsafe, so that has been fixed and
more controls were added in the config parser.

I invite you to report any issues with this version ASAP, because I'm going
to make important changes to buffers. The goal is to ease data manipulation
and particularly compression, which is being developped at Exceliance.

As usual, server-side keep-alive is not there yet, as I'm spending my time
doing other things, but in the end I'd possibly expect that it pays off if
the architecture changes I'm working on help getting compression, native
SSL and multi-process... More info on this later :-)

In the mean time, you can update your version to 1.5-dev8 which is available
below :

   site index  : http://haproxy.1wt.eu/
   sources : http://haproxy.1wt.eu/download/1.5/src/devel/
   changelog   : http://haproxy.1wt.eu/download/1.5/src/CHANGELOG

Cheers,
Willy

[ANNOUNCE] haproxy 1.5-dev9

[Willy Tarreau]

 is safe now.

A number of other minor issues were fixed :
  - balance source did not properly hash IPv6 addresses (Alex Markham)
  - logformat could sometimes segfault (William Lallemand)
  - req_ssl_sni would randomly fail if a session ID is present (Emmanuel Bégazu)
  - doc cleanups and fixes to support HTML converter (Cyril Bonté)

What's in the pipe now ? We're still working on getting SSL and compression to
work. I think that the deep changes are done for now. We found how to split
the socket and protocol handling and it looks promising. I hope that we'll get
something working soon so that we can work on the multi-process model which is
an absolute requirement if we want to get some performance on SSL. I already
have some ideas and I believe I found how we could share the stats and servers
states between all processes. But that's for another version :-)

I've just released haproxy 1.5-dev9 with all the changes above. Thanks to all
those who read till there.

   site index  : http://haproxy.1wt.eu/
   sources : http://haproxy.1wt.eu/download/1.5/src/devel/
   changelog   : http://haproxy.1wt.eu/download/1.5/src/CHANGELOG
 
Cheers,
Willy

[ANNOUNCE] haproxy 1.5-dev10

[Willy Tarreau]

Hi all,

as indicated last week, there were a number of bugs with -dev9 which were
quickly detected and fixed (thanks to Cyril BTW). I also noticed that during
the conversion I lost the getsockname/getdstname calls, resulting in ACLs
on dst never matching, and logs possibly being wrong.

So here is 1.5-dev10 with these issues fixed.

While I'm not fond of mixing fixes and new features, here we're still in
dev and I know that some people will like them, so I have continued the
work towards SSL integration with much more minor changes this time.

I had the time to add the "timeout tunnel" setting, which overrides
"timeout client" and "timeout server" when a connection switches to
tunnel mode (CONNECT forwarded to a proxy, or Upgrade returning 101).
This is mainly aimed at WebSocket setups. Right now people are using
either very long timeouts for all the traffic, or two different backends
with different timeouts in each, and none of these options is good since
all of them require a long client timeout.

With "timeout tunnel", you can keep low values for the client and server
timeouts, and have a large one for the tunnel. It will do the right thing
for you.

I also noticed during debugging sessions that the epoll_ctl() syscall was
making fun of me in the speculative poller. Basically, if a send() failed
after a pending connect(), we would then have the following scenario :

connect(fd) = EAGAIN
send(fd) = EAGAIN
epoll_ctl(ADD(fd:OUT))
epoll_wait() = fd:OUT
send(fd) > 0
epoll_ctl(DEL(fd))
recv(fd) = EAGAIN
epoll_ctl(ADD(fd:IN))
recv(fd) > 0

epoll_ctl() is expensive, we should never have the last two in the middle,
at most one. I realized that I poorly sequenced the sepoll loop. It was
calling speculative events, then updating poll lists, then calling
epoll_wait(), then sometimes calling some speculative events again and
adjusting poll lists again. The sequence above is the result of an already
identified sub-optimization that had to be worked around to speed up recv()
after an accept(). And the epoll_ctl(DEL) followed by an ADD are the result
of this loop...

So I reordered the loop to do this instead :

adjust poll events
epoll_wait()
speculative events

The results are a lot better :

connect(fd) = EAGAIN
send(fd) = EAGAIN
epoll_ctl(ADD(fd:OUT))
epoll_wait() = fd:OUT
send(fd) > 0
epoll_ctl(MOD(fd:IN))
recv(fd) > 0

A performance boost of up to 4.5% on connection rate was observed by just
fixing this !

A second improvement concerns the close() : when an HTTP server sends a close,
we have to respond with a close too. Till now it was done after a wake-up of
the main task, so we had :
recv() = 0
mark buffer as SHUTR
wake up the task
detect the close, decide to do so on the write side
re-scan analysers since the SHUTW flag appeared

In fact, this only happens in HTTP but it's the most important use of haproxy
where performance matters. So now there is a flag in the stream interface
which says that the upper layer is not interested in half-closed connections.
That way, when recv() receives zero, it closes the FD itself before waking up
the task. This has boosted connection rates by up to 7.5% !

So this version may bring 10% or more performance gain on workloads that are
composed of many short connections, and should improve configuration for
those with long connections, it should please everyone, that's why I thought
these changes were welcome with the fixes :-)

For those curious about SSL works in progress at Exceliance, last Friday
evening we managed to establish a backend connection between haproxy and
a server with this code and very little changes. The thing starts to become
a reality :-)

Here's the short changelog :

- BUG/MINOR: stats admin: "Unexpected result" was displayed unconditionally
- BUG/MAJOR: acl: http_auth_group() must not accept any user from the 
userlist
- CLEANUP: auth: make the code build again with DEBUG_AUTH
- BUG/MEDIUM: config: don't crash at config load time on invalid userlist 
names
- REORG: use the name sock_raw instead of stream_sock
- MINOR: stream_interface: add a client target : TARG_TYPE_CLIENT
- BUG/MEDIUM: stream_interface: restore get_src/get_dst
- CLEANUP: sock_raw: remove last references to stream_sock
- CLEANUP: stream_interface: stop exporting socket layer functions
- MINOR: stream_interface: add an init callback to sock_ops
- MEDIUM: stream_interface: derive the socket operations from the target
- MAJOR: fd: remove the need for the socket layer to recheck the connection
- MINOR: session: call the socket layer init function when a session 
establishes
- MEDIUM: session: add support for tunnel timeouts
- MINOR: standard: add a new debug macro : fddebug()
- CLEANUP: fd: remove unused cb->b pointers in the struct fdtab
- OPTIM: proto_http: don't enable quick-ack on empty buffers
- OPTIM/MAJOR: ev_sepoll: 

[ANNOUNCE] haproxy 1.5-dev11

[Willy Tarreau]

Hi,

there were still a significant number of issues in 1.5-dev10, to the point
that I'm ashamed of running it in prod! So I have released 1.5-dev11. Among
the worst things, we can count :

  - the trash size issue if tune.bufsize is increased past twice its
default value and reqrep rules are used (a crash may occur. I don't
think it's possible to make it execute code since it's in the BSS
which is not executable. Still it's possible to crash the process
and that's a real issue.
  - "option forwardfor if-none" stopped working on some configs
  - http-send-name-header stopped working, as well as any post-lb L7
processing (URL hashing, header hashing, bind to header, redirects, ...)
  - risk of crash when checking a server in a farm with "option transparent"
  - double free on exit when parsing some ACL args
  - fixed peer synchronisation which was broken.
  - fixed warnings emitted about log-format when in TCP mode or when
"option httplog" was specified before "mode http".

OK end of the thriller, now the good news :

  - added "on-marked-up" server option to kill sessions on backup servers
(Justin Karneges)
  - added "balance uri whole" to include query string (Oskar Stolc)
  - added "httponly" and "secure" cookie options
  - added a build target "linux2628" which is like linux26 but automatically
includes splicing and tproxy.
  - added "soft stop", "soft start" and "kill" on the stats page in admin mode.

Some changes were begun on connection handling since we discovered that
the way some things are done is a bit awkward when we merge SSL (eg:
incompatibilities with proxy protocol or TCP health checks). After
several days of scratching my head, I found that the changes will need
to be more important than what I expected, so I preferred to stop doing
them and issue 1.5-dev11 before, because it's very likely that there will
be some breakage again.

I'm appending the changelog, and you can and should download the code at
the usual place :

site index  : http://haproxy.1wt.eu/
sources : http://haproxy.1wt.eu/download/1.5/src/devel/
changelog   : http://haproxy.1wt.eu/download/1.5/src/CHANGELOG

Cheers,
Willy

---

2012/06/04 : 1.5-dev11
- BUG/MEDIUM: option forwardfor if-none doesn't work with some 
configurations
- BUG/MAJOR: trash must always be the size of a buffer
- DOC: fix minor regex example issue and improve doc on stats
- MINOR: stream_interface: add a pointer to the listener for 
TARG_TYPE_CLIENT
- MEDIUM: protocol: add a pointer to struct sock_ops to the listener struct
- MINOR: checks: add on-marked-up option
- MINOR: balance uri: added 'whole' parameter to include query string in 
hash calculation
- MEDIUM: stream_interface: remove the si->init
- MINOR: buffers: add a rewind function
- BUG/MAJOR: fix regression on content-based hashing and 
http-send-name-header
- MAJOR: http: stop using msg->sol outside the parsers
- CLEANUP: http: make it more obvious that msg->som is always null outside 
of chunks
- MEDIUM: http: get rid of msg->som which is not used anymore
- MEDIUM: http: msg->sov and msg->sol will never wrap
- BUG/MAJOR: checks: don't call set_server_status_* when no LB algo is set
- BUG/MINOR: stop connect timeout when connect succeeds
- REORG: move the send-proxy code to tcp_connect_write()
- REORG/MINOR: session: detect the TCP monitor checks at the protocol accept
- MINOR: stream_interface: introduce a new "struct connection" type
- REORG/MINOR: stream_interface: move si->fd to struct connection
- REORG/MEDIUM: stream_interface: move applet->state and private to 
connection
- MINOR: stream_interface: add a data channel close function
- MEDIUM: stream_interface: call si_data_close() before releasing the si
- MINOR: peers: use the socket layer operations from the peer instead of 
sock_raw
- BUG/MINOR: checks: expire on timeout.check if smaller than timeout.connect
- MINOR: add a new function call tracer for debugging purposes
- BUG/MINOR: perform_http_redirect also needs to rewind the buffer
- BUG/MAJOR: b_rew() must pass a signed offset to b_ptr()
- BUG/MEDIUM: register peer sync handler in the proper order
- BUG/MEDIUM: buffers: fix bi_putchr() to correctly advance the pointer
- BUG/MINOR: fix option httplog validation with TCP frontends
- BUG/MINOR: log: don't report logformat errors in backends
- REORG/MINOR: use dedicated proxy flags for the cookie handling
- BUG/MINOR: config: do not report twice the incompatibility between cookie 
and non-http
- MINOR: http: add support for "httponly" and "secure" cookie attributes
- BUG/MEDIUM: ensure that unresolved arguments are freed exactly once
- BUG/MINOR: commit 196729ef used wrong condition resulting in freeing 
constants
- MEDIUM: stats: add support for soft stop/soft start in the admin interface
- MEDIUM: stats: add the abili

[ANNOUNCE] haproxy 1.5-dev12

[Willy Tarreau]

Hi all,

So the long-awaited dev12 is here, with native SSL support on both
sides supporting SNI and wildcard certs, that was developped by the
Exceliance team.

We got many useful reports since the last post on the subject, thanks to
all those who contributed some feedback! All known build bugs were fixed.
I won't explain here again what changes were done, it's too long :-)

Since last post, we worked on integrating support for SNI because most of
the responders asked for it. So now it's possible on a "bind" line to load
as many certs as needed, and they'll be matched depending on the domains
they're valid for. Wildcards are supported too. And since certs are loaded
in trees, matching them is cheap even if you're dealing with tens of
thousands of virtual domains.

We have also added some ACLs to match the use of SSL for a connection and
to match presence/value of the SNI extension, as we think it will usually
be needed as well in virtual hosting environments.

Warning, we have changed the SSL config syntax since last version. Since
loading mutiple certs is possible, we now use the word "crt" before the
certs. So that now looks like this :

   bind :443 ssl crt default.pem crt /etc/haproxy/certs.d

SSL aside, there are some other new features such as IPv6 transparent mode,
"base" pattern/acl to match a concatenation of the Host header and the URI, 
"urlp_val" ACL to match a url parameter's value, support for the "nice"
keyword on "bind" lines to change the priority of sessions using this bind
line (useful to limit SSL CPU impact), the ability to clear/feed stick-table
entries on the stats CLI, and the usual set of halog features and optims.

Many bugs were fixes, and many were certainly introduced. If you observe any
bug, please report it, as I'd rather issue -dev13 quickly with many fixes.

I'm not appending the changelog, it's too large. The usual pointers follow :

Site index  : http://haproxy.1wt.eu/
Sources : http://haproxy.1wt.eu/download/1.5/src/devel/
Changelog   : http://haproxy.1wt.eu/download/1.5/src/CHANGELOG
Exceliance  : http://www.exceliance.fr/en/

Willy

[ANNOUNCE] haproxy 1.5-dev13

[Willy Tarreau]

Hi all,

This is the largest development version ever issued, 295 patches in 2 months!

We managed to keep the Exceliance team busy all the time, which means that
the code is becoming more modular with less cross-dependences, I really like
this !

First, we got an amazing amount of feedback from early adopters of dev12. It
seems like SSL was expected for too long a time. We really want to thank all
those who contributed patches, feedback, configs, cores (yes there were) and
even live gdb access, you know who you are and you deserve a big thanks for
this!

Git log says there were 55 bugs fixed since dev12 (a few of them might have
been introduced in between). Still, this means that dev12 should be avoided
as much as possible, which is why I redirected many of you to more recent
snapshots.

These bugs aside, I'm proud to say that the whole team did a really great
job which could be summarized like this :

1) SSL:
  - many more features ; client and server certificates supported on both
sides with CA and CRL checks. Most of the information available in SSL
can be used in ACLs for access control. Some information such as protocol
and ciphers can be reported in the logs. These information are still not
added to HTTP logs though, a lot of config work is still needed.

  - cache life time and maximum concurrent SSL connections can be set.
Unfortunately OpenSSL happily dereferences NULL malloc returns and
causes the process to die if memory becomes scarce. So we can only
limit its maximum amount of connections if we want to limit the
memory it uses.

  - TLS NPN was implemented with the help from Simone Bordet from Jetty,
and can be used to offload SSL/TLS for SPDY and to direct to a
different server depending on the protocol chosen by the client.

  - Ivan Ristic from ssllabs and Andy Humphreys from Robinson-way provided
very valuable help in diagnosing and fixing some nasty issues with
aborts on failed handshakes and improve from an E-grade to an A-grade :
   https://www.ssllabs.com/ssltest/analyze.html?d=demo.1wt.eu

2) HTTP Compression
  - HTTP payload compression was implemented at Exceliance to achieve
bandwidth usage reduction and reduce page load time on congested or
small links. Compression is extremely CPU and memory intensive, so we
spent most of the time developing dynamic adaptations. It is possible
to limit the maximum RAM dedicated to compression, the CPU usage
threshold and bandwidth thresholds above which compression is disabled.
It is even possible to adjust some of these settings from the stats
socket and to monitor bandwidth savings in real time. Proceeding like
this ensures a high reliability at low cost and with little added
latency. I've put it on the haproxy web site with nice bandwidth savings
(72% avg on compressible objects, 50% on average, considering that most
downloads are compressed sources). I must say I'm very happy of this new
feature which will reduce bandwidth costs in hosted infrastructures ! And
it goes back to the origins of haproxy in zprox 14 years ago :-)

3) Health checks
  - SSL is now usable with health checks. By default it is enabled if the
server has the "ssl" keyword and no "port" nor "addr" setting. It
can be forced using "check-ssl" otherwise. So now running an HTTPS
health check simply consists in using "option httpchk" with "ssl" on
the server.

  - send-proxy is also usable with health checks, with the same rules as
above, and the "check-send-proxy" directive to force it. The checks
also respect the updated spec which suggests sending real addresses
with health checks instead of sending unknown addresses. This makes
it compatible with some products such as postfix 2.10 for example.

4) Polling
  - speculative polling was generalized to all pollers, and sepoll
disappeared as it was superseded by epoll. The main reason for this
important change is the way OpenSSL works and the fact that it can
easily get stuck with some data in buffers with no I/O event to
unblock them. So we needed to engage into this difficult change. I'd
have preferred to delay it to 1.6 if I was offered the choice ! But
in the end this is good because it's done and it improves both
performance and reliability. Even select() and poll() are now fast.

  - the maxaccept setting was too low on some platforms to achieve the
highest possible performance, so it was doubled to 64 and is now per
listener so that it automatically adjusts to the number of processes
the listener is bound to. This ensures both best performance in single
process mode, and quite good fairness in multi-process mode.

5) Platform improvements
  - Linux 3.6 TCP Fast Open is supported on listeners ("tfo" bind keyword).
This is used to allow compatible clients to re-establish a TCP connection
in a single packet and save one round-trip. The kern

[ANNOUNCE] haproxy 1.5-dev14

[Willy Tarreau]

Hi,

after the nice failure of dev13 which worked well before being deployed
in field, here comes dev14.

The main pain was fixing health checks. Their port to use connections
was minimalist but incomplete. And each time I fixed something, something
else broke in various areas, explaining that about half of the commits
concern health checks and polling. I'm more confident in this new version
now, because there were fixes in many areas :
  - Cyril and William found and fixed a compression breakage with chunked
encoded data ;
  - Baptiste and Falco Schmutz helped collecting data to debug spinning
processes
  - Hervé reported that an old process would not leave due to some peers
breakage (maxaccept = 0)
  - Dmitry Sivachenko provided lots of information to debug a crash of
the old process upon replacement on FreeBSD, which in turn revealed
horrible things in ACL argument management
  - Ben Timby reported that SMTP checks were broken with send-proxy

The good news is that we could add minor features such as the "v6only"
and "v4v6" bind options that were discussed on the list this week-end.
Some other very minor features were added mainly for debugging,  such
as "show sess all" on the CLI to dump all session states, because it was
a bit frustrating to try to get relevant information during bugs.

The health-checks have experienced a major lifting. All reported and
detected issues were fixed, and the behaviour could be improved a bit
in that health-checks won't leave any more TIME_WAIT sockets on the
local machine. This was causing some trouble at places where source
ports are heavily used, because many local source ports became unusable
for any outgoing connection for 2 minutes.

The usual links follow, and I'm appending the changelog (short this time)
at the end of this e-mail.

Site index   : http://haproxy.1wt.eu/
Sources  : http://haproxy.1wt.eu/download/1.5/src/devel/
Changelog: http://haproxy.1wt.eu/download/1.5/src/CHANGELOG
Cyril's HTML doc : 
http://cbonte.github.com/haproxy-dconv/configuration-1.5.html

Thanks to all those who contributed bug reports and fixes, and I hope
this time everyone will have a better experience.

Willy

-- Changelog from 1.5-dev13 to 1.5-dev14 :

2012/11/26 : 1.5-dev14
- DOC: fix minor typos
- BUG/MEDIUM: compression: does not forward trailers
- MINOR: buffer_dump with ASCII
- BUG/MEDIUM: checks: mark the check as stopped after a connect error
- BUG/MEDIUM: checks: ensure we completely disable polling upon success
- BUG/MINOR: checks: don't mark the FD as closed before transport close
- MEDIUM: checks: avoid accumulating TIME_WAITs during checks
- MINOR: cli: report the msg state in full text in "show sess $PTR"
- CLEANUP: checks: rename some server check flags
- MAJOR: checks: rework completely bogus state machine
- BUG/MINOR: checks: slightly clean the state machine up
- MEDIUM: checks: avoid waking the application up for pure TCP checks
- MEDIUM: checks: close the socket as soon as we have a response
- BUG/MAJOR: checks: close FD on all timeouts
- MINOR: checks: fix recv polling after connect()
- MEDIUM: connection: provide a common conn_full_close() function
- BUG/MEDIUM: checks: prevent TIME_WAITs from appearing also on timeouts
- BUG/MAJOR: peers: the listener's maxaccept was not set and caused loops
- MINOR: listeners: make the accept loop more robust when maxaccept==0
- BUG/MEDIUM: acl: correctly resolve all args, not just the first one
- BUG/MEDIUM: acl: make prue_acl_expr() correctly free ACL expressions upon 
exit
- BUG/MINOR: stats: fix inversion of the report of a check in progress
- MEDIUM: tcp: add explicit support for delayed ACK in connect()
- BUG/MEDIUM: connection: always disable polling upon error
- MINOR: connection: abort earlier when errors are detected
- BUG/MEDIUM: checks: report handshake failures
- BUG/MEDIUM: connection: local_send_proxy must wait for connection to 
establish
- MINOR: tcp: add support for the "v6only" bind option
- MINOR: stats: also report the computed compression savings in html stats
- MINOR: stats: report the total number of compressed responses per 
front/back
- MINOR: tcp: add support for the "v4v6" bind option
- DOC: stats: document the comp_rsp stats column
- BUILD: buffer: fix another isprint() warning on solaris
- MINOR: cli: add support for the "show sess all" command
- BUG/MAJOR: cli: show sess  may randomly corrupt the back-ref list
- MINOR: cli: improve output format for show sess $ptr
--

[ANNOUNCE] haproxy 1.5-dev15

[Willy Tarreau]

Hi,

dev14 was quite calm but we got exactly 3 reports of users experiencing
some high CPU usage patterns caused by the poll loop spinning with an
unprocessed Jvent. This bug was very timing dependant and mostly affects
TCP-based proxies, though in theory it could also affect HTTP with an even
lower probability (it took days to find the proper machine combination to
reproduce it).

With the continued help from Bryan Berry and Baptiste Assmann, we could
finally isolate the bug, understand it and fix it, which was confirmed
yesterday by Bryan.

Another annoying bug that was fixed was a file descriptor leak when
logging SSL protocol and/or cipher (%sslv/%sslc) since dev14.

Among the other interesting bugs fixed, most of us have been experiencing
for years some occasional "SD" flags in their logs and "resp errors" in
the stats page that were never explained. This was in fact an error on the
client propagated to the server that was diagnosed as a server fault, but
the fault really is the client's. This has been fixed and now the haproxy
demo page does not report any server error at all anymore. This bug also
affects 1.4 and maybe 1.3, I'll have to check.

The gzip compression is not applied anymore to responses with a status
different from 200 nor to multipart responses. Byte-ranges are not
compatible with compression, and 200 is where almost all the compressible
traffic is, so let's stay on the safe side of affairs! Some stats were
added on the compression in the stats page.

SSL to the server was failing the first connection when presenting a
certificate. This has been fixed. SSL and PROXY protocol handshake
errors are now logged. The logs are still not very detailed but they
clearly help spotting what is not working during debug. The SSL cache
was also improved to take less space by default and to support larger
entries for large client certs. This has been done by managing fragmented
blocks.

Some build errors on Solaris were fixed thanks to the tests and reports
from Benjamin Polidore.

Pattern conversion in stick-tables was broken in dev13 and was fixed now.

A much requested feature is the support for L7 information in track-counters.
In practice, this only works for content tracking and is a bit tricky, but
with some efforts it is possible to stick on an IP address that was extracted
from the X-Forwarded-For header passed with the request for example. Some
"base32" and "base32+src" fetch methods were added to provide a 32-bit hash
of the base URL (host+uri) and to concatenate it with the source address to
have per-url and per-ip short tracking keys.

That's about all for the main changes, for more details, please check the
changelog.

The usual links follow, and I'm appending the small changelog at the end
of this e-mail.

Site index   : http://haproxy.1wt.eu/
Sources  : http://haproxy.1wt.eu/download/1.5/src/devel/
Changelog: http://haproxy.1wt.eu/download/1.5/src/CHANGELOG
Cyril's HTML doc : 
http://cbonte.github.com/haproxy-dconv/configuration-1.5.html

I know I have already said this twice, but given the small number of
changes and fixes, I do trust this version even more than previous ones.

Thanks again to all participants.

Willy

-- Changelog from 1.5-dev14 to 1.5-dev15 :

- DOC: add a few precisions on compression
- BUG/MEDIUM: ssl: Fix handshake failure on session resumption with client 
cert.
- BUG/MINOR: ssl: One free session in cache remains unused.
- BUG/MEDIUM: ssl: first outgoing connection would fail with 
{ca,crt}-ignore-err
- MEDIUM: ssl: manage shared cache by blocks for huge sessions.
- MINOR: acl: add fetch for server session rate
- BUG/MINOR: compression: Content-Type is case insensitive
- MINOR: compression: disable on multipart or status != 200
- BUG/MINOR: http: don't report client aborts as server errors
- MINOR: stats: compute the ratio of compressed response based on 2xx 
responses
- MINOR: http: factor out the content-type checks
- BUG/MAJOR: stats: correctly check for a possible divide error when 
showing compression ratios
- BUILD: ssl: OpenSSL 0.9.6 has no renegociation
- BUG/MINOR: http: disable compression when message has no body
- MINOR: compression: make the stats a bit more robust
- BUG/MEDIUM: comp: DEFAULT_MAXZLIBMEM was expressed in bytes and not 
megabytes
- MINOR: connection: don't remove failed handshake flags
- MEDIUM: connection: add an error code in connections
- MEDIUM: connection: add minimal error reporting in logs for incomplete 
connections
- MEDIUM: connection: add error reporting for the PROXY protocol header
- MEDIUM: connection: add error reporting for the SSL
- DOC: document the connection error format in logs
- BUG/MINOR: http: don't log a 503 on client errors while waiting for 
requests
- BUILD: stdbool is not portable
- BUILD: ssl: NAME_MAX is not portable, use MAXPATHLEN instead
- BUG/MAJOR: raw_sock

[ANNOUNCE] haproxy-1.5-dev16

[Willy Tarreau]

Hi all,

Here comes 1.5-dev16. Thanks to the amazing work Sander Klein and John
Rood have done at Picturae ICT ( http://picturae.com/ ) we could finally
spot the freeze bug after one week of restless digging ! This bug was
amazingly hard to reproduce in general and would only affect POST requests
under certain circumstances that I never could reproduce despite many
efforts. It is likely that other users were affected too but did not
notice it because end users did not complain (I'm thinking about webmail
and file sharing environments for example).

During this week of code review and testing, around 10 other minor to medium
bugs related to the polling changes could be fixed.

Another nasty bug was fixed on SSL. It happens that OpenSSL maintains a
global error stack that must constantly be flushed (surely they never heard
how errno works). The result is that some SSL errors could cause another SSL
session to break as a side effect of this error. This issue was reported by
J. Maurice (wiz technologies) who first encountered it when playing with the
tests on ssllabs.com.

Another bug present since 1.4 concerns the premature close of the response
when the server responds before the end of a POST upload. This happens when
the server responds with a redirect or with a 401, sometimes the client would
not get the response. This has been fixed.

Krzysztof Rutecki reported some issues on client certificate checks, because
the check for the presence of the certificate applies to the connection and
not just to the session. So this does not match upon session resumption. Thus
another ssl_c_used ACL was added to check for such sessions.

Among the other nice additions, it is now possible to log the result of any
sample fetch method using %[]. This allows to log SSL certificates for example.
And similarly, passing such information to HTTP headers was implemented too,
as "http-request add-header" and "http-request set-header", using the same
format as the logs. This also becomes useful for combining headers !

Some people have been asking for logging the amount of uploaded data from the
client to the server, so this is now available as the %U log-format tag.
Some other log-format tags were deprecated and replaced with easier to remind
ones. The old ones still work but emit a warning suggesting the replacement.

And last, the stats HTML version was improved to present detailed information
using hover tips instead of title attributes, allowing multi-line details on
the page. The result is nicer, more readable and more complete.

The changelog is short enough to append it here after the usual links :

Site index   : http://haproxy.1wt.eu/
Sources  : http://haproxy.1wt.eu/download/1.5/src/devel/
Changelog: http://haproxy.1wt.eu/download/1.5/src/CHANGELOG
Cyril's HTML doc : 
http://cbonte.github.com/haproxy-dconv/configuration-1.5.html

At the moment, nobody broke the latest snapshots, so I think we're getting
closer to something stable to base future work on.

Thanks!
Willy

--
Changelog from 1.5-dev15 to 1.5-dev16:
  - BUG/MEDIUM: ssl: Prevent ssl error from affecting other connections.
  - BUG/MINOR: ssl: error is not reported if it occurs simultaneously with peer 
close detection.
  - MINOR: ssl: add fetch and acl "ssl_c_used" to check if current SSL session 
uses a client certificate.
  - MINOR: contrib: make the iprange tool grep for addresses
  - CLEANUP: polling: gcc doesn't always optimize constants away
  - OPTIM: poll: optimize fd management functions for low register count CPUs
  - CLEANUP: poll: remove a useless double-check on fdtab[fd].owner
  - OPTIM: epoll: use a temp variable for intermediary flag computations
  - OPTIM: epoll: current fd does not count as a new one
  - BUG/MINOR: poll: the I/O handler was called twice for polled I/Os
  - MINOR: http: make resp_ver and status ACLs check for the presence of a 
response
  - BUG/MEDIUM: stream-interface: fix possible stalls during transfers
  - BUG/MINOR: stream_interface: don't return when the fd is already set
  - BUG/MEDIUM: connection: always update connection flags prior to computing 
polling
  - CLEANUP: buffer: use buffer_empty() instead of buffer_len()==0
  - BUG/MAJOR: stream_interface: fix occasional data transfer freezes
  - BUG/MEDIUM: stream_interface: fix another case where the reader might not 
be woken up
  - BUG/MINOR: http: don't abort client connection on premature responses
  - BUILD: no need to clean up when making git-tar
  - MINOR: log: add a tag for amount of bytes uploaded from client to server
  - BUG/MEDIUM: log: fix possible segfault during config parsing
  - MEDIUM: log: change a few log tokens to make them easier to remember
  - BUG/MINOR: log: add_to_logformat_list() used the wrong constants
  - MEDIUM: log-format: make the format parser more robust and more extensible
  - MINOR: sample: support cast from bool to string
  - MINOR: samples: add a function to fetch and convert any sample to

[ANNOUNCE] haproxy-1.5-dev17

[Willy Tarreau]

Hi,

here's a new version with very few changes. It fixes the latest issues
reported in dev16 :
  - broken stats page
  - add-header emitting a trailing \0
  - POST on the stats page segfaulting the process
  - interferences between log-format, unique-id and add-header

It also adds a few minor features that were pending for a long time and
some that were missing :
  - ability to disable the SSL session cache by setting its size to zero
  - support for "http-request redirect" and "http-request tarpit" rules.

I hope it's the last of the year and that it gets rid of all the issues
we introduced since dev12.

Usual links and changelog below :

 Site index   : http://haproxy.1wt.eu/
 Sources  : http://haproxy.1wt.eu/download/1.5/src/devel/
 Changelog: http://haproxy.1wt.eu/download/1.5/src/CHANGELOG
 Cyril's HTML doc : 
http://cbonte.github.com/haproxy-dconv/configuration-1.5.html

Changelog from 1.5-dev16 to 1.5-dev17:

- MINOR: ssl: Setting global tune.ssl.cachesize value to 0 disables SSL 
session cache.
- BUG/MEDIUM: stats: fix stats page regression introduced by commit 20b0de5
- BUG/MINOR: stats: last fix was still wrong
- BUG/MINOR: stats: http-request rules still don't cope with stats
- BUG/MINOR: http: http-request add-header emits a corrupted header
- BUG/MEDIUM: stats: disable request analyser when processing POST or HEAD
- BUG/MINOR: log: make log-format, unique-id-format and add-header more 
independant
- BUILD: log: unused variable svid
- CLEANUP: http: rename the misleading http_check_access_rule
- MINOR: http: move redirect rule processing to its own function
- REORG: config: move the http redirect rule parser to proto_http.c
- MEDIUM: http: add support for "http-request redirect" rules
- MEDIUM: http: add support for "http-request tarpit" rule

Willy

[ANNOUNCE] haproxy-1.5-dev18

[Willy Tarreau]

Hi,

here's an update for 1.5. It contains a security fix, please read, as
you'll want either to update or to apply the fix on top of your version
if you are running it.

Configurations at risk are those which combine use of HTTP keywords in
TCP content inspection rules, client-side keep-alive, header rewriting
rules and which receive pipelined requests. These configurations may be
remotely crashed when run with haproxy 1.4 up to and including 1.4.22
or development versions up to and including 1.5-dev17. Versions 1.4.23
and 1.5-dev18 are safe. This issue was reported and troubleshooted by
Yves Lafon from the W3C. Thanks Yves for the time you spent on this and
all the efforts you made to get this core!

For those who want to quickly deploy a fix, please use this patch for
1.5 : http://git.1wt.eu/web?p=haproxy.git;a=commitdiff;h=aae75e3279

Many things have changed since 1.5-dev17. Many bugs were fixed, and bunch
of regressions were introduced during the chanegs, as was seen today. But
we're improving the code modularity and maintenability, this is important.
The most important changes (not counting the bugs) are listed below :

  - systemd support was brought by Marc-Antoine Perennou

  - agent-based health check was implemented by Simon Hormans based
on specs provided by Loadbalancer.org. This is a first step towards
a more complete health check system. In this version, health checks
can be switched to use a special agent running on the system, which
can tell haproxy what weight it wants, if it is up/down/ill/in
maintenance, etc...

  - all address parsers were merged intoo a single one. This means
that "bind", "log", "source", "server", ... now all use the same
address parser. It has been improved to support forcing the
address family by the use of the "@" prefix, and environment
variables. It is now possible to build configurations which use
the local address without using sed. It is also possible to listen
to a pre-bound file descriptor passed by the parent.

  - ability to limit the maximum SSL/TLS record size (suggested by
Ilya Grigorik, and earlier by Mike Belshe). In short, when you
send a 16kB ciphered block to a browser, it has to receive it
completely before deciphering it and starting to render the
contents. By reducing the size, we slightly reduce the performance
but we allow the browser to start rendering the page and fetch new
objects much earlier.

  - HTTP redirect can now emit statuses 307 and 308 (Yves Lafon).

  - PCRE JIT support was contributed by Hiroaki Nakamura, it should
provide much faster regex processing.

  - poll() is now enabled by default in the makefile even for unknown
targets.

  - ACL and sample fetch converged to something much better : everything
that could be extracted from a request or response is now accessible
via a sample fetch (so no more lamenting about whether it's availble
as one or the other), and ACLs do not have a fetch function anymore,
they use the sample's fetch methods. There are several benefits of
doing this. The first one is that we don't duplicate code anymore and
we'll avoid many bugs (many were found during this work). The second
benefit is that ACLs can now select the matching method to be used
with the new -m parameter, so many more combinations are possible
and won't require duplicating code anymore. The third benefit is that
error reporting was greatly improved, as each fetch argument knows
its exact position in the expression, and each fetch method knows
it compatibility matrix. Thus, no more silent uses of an L6 fetch
with an L4 or things like this. Some further changes are expected
on this point (maps, variables, concatenation, conversions).

  - TLS ALPN was implemented similarly as NPN was made. It is supposed
to replace NPN.

All the rest are bugs. I counted 43...

Usual links below :

 Site index   : http://haproxy.1wt.eu/
 Sources  : http://haproxy.1wt.eu/download/1.5/src/devel/
 Changelog: http://haproxy.1wt.eu/download/1.5/src/CHANGELOG
 Cyril's HTML doc : 
http://cbonte.github.com/haproxy-dconv/configuration-1.5.html

I'm not appending the changelog, it's too large.

And remember, if you use "tcp-request content" with any HTTP keyword,
you need either a config change or an update.

Willy

problem with haproxy 1.5

[Alejandro Perretta]

I have a problem with haproxy v1.5. If i request 10.000 request with 500 
concurres balanced  with two nodes the 50% of the request is failed. But 
if i request the same to haproxy with only one node i have no failed  
request's.



The conf


global
log /dev/loglocal0
log /dev/loglocal1 debug
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
user haproxy
group haproxy
daemon

maxconn 200
# Default SSL material locations

# Default ciphers to use on SSL-enabled listening sockets.
# For more information, see ciphers(1SSL). This list is from:
# https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
# An alternative list with additional directives can be obtained from
# 
https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy

 tune.ssl.cachesize 100
 tune.ssl.default-dh-param 1024
ssl-default-bind-ciphers 
ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:DES-CBC3-SHA:HIGH:SEED:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!RSAPSK:!aDH:!aECDH:!EDH-DSS-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA:!SRP

ssl-default-bind-options  no-tls-tickets
ssl-default-server-ciphers 
ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:DES-CBC3-SHA:HIGH:SEED:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!RSAPSK:!aDH:!aECDH:!EDH-DSS-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA:!SRP

ssl-default-server-options  no-tls-tickets
nbproc 4
cpu-map 1 0
cpu-map 2 1
cpu-map 3 2
cpu-map 4 3
stats bind-process 4




listen statistics
   bind *:8080
   mode http
   stats enable
   stats hide-version
   stats realm Haproxy\ Statistics
   stats uri /stats
   stats auth admin:xxx
   stats admin if TRUE
   stats refresh 5s

defaults
logglobal
modehttp
optionhttplog
option http-keep-alive
optiondontlognull
timeout connect 5000
   option forwardfor
   option http-server-close
timeout client  5
timeout server  5
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http




frontend www.bancocredicoop.coop-ssl
option http-server-close
  reqadd X-Forwarded-Proto:\ https
bind-process 1 2 3
bind :443 ssl crt /etc/ssl/private/certificados.pem
   bind:80
redirect scheme https code 301 if !{ ssl_fc }
  default_backend 

backend 
mode http
balance roundrobin
server sweb7101x 10.9.1.180:80 check
   # server sweb7401x 10.9.1.181:80 check


--
Alejandro Perretta
Plataforma Linux
Soporte Servicios Centrales de Plataforma Abierta
Banco Credicoop C.L.
TE +54.11.4590.1115 | FAX +54.11.4590.1165
aperre...@bancocredicoop.coop

"Este mensaje y sus anexos son confidenciales y para uso exclusivo del titular 
de la dirección de correo electrónico a que está dirigido. Puede contener 
información amparada por el secreto bancario cuyo uso inadecuado puede derivar en 
responsabilidad civil y/o penal. Por lo expuesto, su contenido no debe ser copiado, 
enviado, revelado o utilizado por personas distintas de su destinatario. Si Ud. no 
fuera el destinatario especificado en el mensaje o persona autorizada por éste, 
informe de tal situación reenviando el mensaje al emisor e inmediatamente proceda a 
borrar el mensaje recibido y sus adjuntos. No se garantiza la seguridad o la 
exactitud de lo transmitido, ya que el correo electrónico remitido pudo haber sido 
interceptado, modificado 

[ANNOUNCE] haproxy-1.5-dev20

[Willy Tarreau]

Hi all,

here is probably the largest update we ever had, it's composed of 345
patches!

Some very difficult changes had to be made and as usual when such changes
happen, they take a lot of time due to the multiple attempts at getting
them right, and as time goes, people submit features :-)

After two weeks spent doing only fixes, I thought it was time to issue dev20.
I'm sure I'll forget a large number of things, but the main features of this
version include the following points (in merge order) :

  - optimizations (splicing, polling, etc...) : a few percent CPU could be
saved ;

  - memory : the connections and applets are now allocated only when needed.
Additionally, some structures were reorganized to avoid fragmentation on
64-bit systems. In practice, an idle session size has dropped from 1936
bytes to 1296 bytes (-640 bytes, or -33%).

  - samples : all sample fetch expressions now support a comma-delimited
list of converters. This is also true in ACLs, so that it becomes
possible to do things like :

# convert to lower case and use fast tree indexing
acl known_domain hdr(host),lower -f huge-domain-list.lst

  - a lot of code has been deduplicated in the tracked counters, it's now
possible to use sc_foo_bar(1, args) instead of sc1_foo_bar(args). Doing
so has simplified the code and makes life of APIs easier.

  - it's now possible to look up a tracked key from another table. This allows
to retrieve multiple counters for the same key.

  - several hash algorithms are provided, and it is possible to select them
per backend. This high quality work was done at Tumblr by Bhaskar Maddala.

  - agent-checks: this new feature was merged and replaced the lb-agent-chk.
Some changes are still planned but feedback is welcome. The goal of this
agent is to retrieve soem weight information from a server independantly
of the service health. A typical usage would consist in reporting the
server's idle percentage as an estimate of the possible weight. This work
was done by Simon Horman for Loadbalancer.org.

  - samples : more automatic conversions between types are supported, making
it easier to stick to any parameter. The types are much more dynamic now.
Some improvements are still pending. This work was done by Thierry Fournier
at Exceliance.

  - map : a new type of converter appeared : maps. A map matches a key from
a file just like ACLs do, and replaces this value with the value associated
with the key on the same line of the file. As it is a converter, it can be
used in any sample expression. The first usage consists in geolocation,
where networks are associated with country codes. Maps may be consulted,
deleted, updated and filled from the CLI. Some will probably use this to
program actions or emulate ACLs without even reloading a config. This
work was also achieved by Thierry Fournier, and reviewed by Cyril Bonté
who developped the original Geoip patchset for 1.4 and 1.5.

  - http-request redirect now supports log-format like expressions, just like
http-request add-header. This allows to emit strings extracted from the
request (host header, country code from a map, ...). Thierry again here.

  - checks: tcp-check supports send/expect sequences with strings/regex/binary.
Thus it now becomes possible to check unsupported protocols, even binary.
This work is from Baptiste Assmann.

  - keep-alive: the dynamic allocation of the connection and applet in the
session now allows to reuse or kill a connection that was previously
associated with the session. Thus we now have a very basic support for
keep-alive to the servers. There is even an option to relax the load
balancing to try to keep the same connection. Right now we don't do
any connection sharing so the main use is for static servers and for
far remote servers or those which require the broken NTLM auth. That
said, the performance tests I have run show an increase from 71000
connections per second to 15 keep-alive requests per second running
on one core of a Xeon E5 3.6 GHz. This doubled to 300k requests per
second with two cores. I didn't test above, I lacked injection tools :-)
One good point is that it will help people assemble haproxy and varnish
together with haproxy doing the consistent hash and varnish caching after
it.

As most of you know, server-side keep-alive is the condition to release 1.5.
Now we have it, we'll be able to improve on it but it's basically working.

I expect to release 1.5-final around January and mostly focus on chasing
bugs till there. So I'd like to set a feature freeze. I know it doesn't
mean much considering that we won't stop contribs. But I don't want to
merge another large patch set before the release. Ideally there will not
be any dev21 version. Reality probably is that we'll have to issue one
because people will inevitably report 

[ANNOUNCE] haproxy-1.5-dev21

[Willy Tarreau]

Hi all,

I've released 1.5-dev21 which fixes all the issue reported on dev20 :

- MINOR: stats: don't use a monospace font to report numbers
- MINOR: session: remove debugging code
- BUG/MAJOR: patterns: fix double free caused by loading strings from files
- MEDIUM: http: make option http_proxy automatically rewrite the URL
- BUG/MEDIUM: http: cook_cnt() forgets to set its output type
- BUG/MINOR: stats: correctly report throttle rate of low weight servers
- BUG/MEDIUM: checks: servers must not start in slowstart mode
- BUG/MINOR: acl: parser must also stop at comma on ACL-only keywords
- MEDIUM: stream-int: implement a very simplistic idle connection manager
- DOC: update the ROADMAP file

I'd suggest using this one instead of dev20 considering the numerous
annoying bugs that were present there.

Usual links below :

 Site index   : http://haproxy.1wt.eu/
 Sources  : http://haproxy.1wt.eu/download/1.5/src/devel/
 Changelog: http://haproxy.1wt.eu/download/1.5/src/CHANGELOG
 Cyril's HTML doc : 
http://cbonte.github.com/haproxy-dconv/configuration-1.5.html

Thanks to all the early testers who reported issues, configs etc.

Willy

[ANNOUNCE] haproxy-1.5-dev22

[Willy Tarreau]

Hi all,

after 1.5 months of head scratching and hair pulling leading to many
bugs being fixed, here comes 1.5-dev22.

This release comes with two important changes :

  - rework of the whole polling system, which is the lower layer of
haproxy ; This was needed to definitely get rid of the frequent
regressions that were caused each time we did a small change
more or less related to this area. The "speculative I/O" mechanism
designed 7 years ago was totally reworked to become a complete
event cache which remembers what direction a file descriptor is
ready in even after being temporarily disabled. This was necessary
because the previous model didn't work well with SSL. Or in fact,
it used to work well enough to hide the fact that the SSL API is
not compatible at all with polled I/O due to its internal buffers.
This part was really difficult to get right, but the code is much
less tricky and much safer, and despite the important change, I
already trust it much more than I did for the previous one.

  - switch to HTTP keep-alive mode by default. This is a major step
forwards since 1.1 where we used to run in tunnel mode by default.
The reason is that tunnel mode was the only way to have something
close to keep-alive for many years. Now that we have end-to-end
keep-alive, we have no reason for keeping tunnel mode as the
default. It causes all the trouble everyone has faced at least
once ("my rule randomly matches") which everyone now is used to
respond to with "your config is missing http-server-close". So
now a config without any close directive is not tunnel anymore
but end-to-end keep-alive. I know there are corner cases where
people want the tunnel mode. There's now a new option "tunnel"
exactly for this. It will be needed to have it in both the
frontend and the backend, just as before when it was needed to
have none of them there.

Eventhough I took extreme care on these changes and did many many
tests (I individually tested the 25 combinations of the 5 HTTP
modes), it is still possible that I didn't notice something, despite
this version currently being run in production on the main site. So
reports are welcome (success, doubts or failures).

I won't enumerate all of the 32 bugs that were fixed since dev21
(some of them introduced there) thanks to all the feedback we got
here on the list and to the detailed information some participants
provided.

The main interesting features that were included are :
  - optimization of the SSL buffer sizes during a handshake to
reduce the number of round trips, as suggested by Ilya Grigorik.
Tests run by Ilya show that the handshake time can be reduced by
3! Work done by Emeric.

  - addition of more debugging information on the stats socket in
"show info" such as SSL connections etc, and memory pools usage
using "show pools".

  - added the ability to set a hard limit on the SSL session rate
(maxsslrate) in order to protect the SSL stack against incoming
connection rushes which can happen during a restart, a config
change (eg: different algos) or an attack. It works exactly
like the "rate-limit sessions" except that it applies to SSL
only.

  - new "capture.req.hdr()" and "capture.res.hdr()" sample fetches
are used to include contents of selected captured headers in logs
or other headers (William).

  - keep-alive: stick to the same server if possible after receiving
a 401 or 407 from the server, so that the user has a chance to
complete an authentication handshake (eg: NTLM). This avoids the
need for "option prefer-last-server" for such situations.

  - tcp-check: new "tcp-check connect" directive to establish a
connection to a specific port. This allows multi-port checks
(Baptiste).

Some code is still pending for a next version. Thierry has finished
the map+acl merge which will allow to manipulate ACLs on the fly just
like maps today, the code is still under review (massive changes),
and is so often requested that we'd better merge it before 1.5-final.

Another SSL optim is currently under test.

All the easy things that were pending have been merged. This leaves
us with only the bind-process fixes, buffer management to fix
compression on chunks, and the agent-checks modifications. We'll see
how all this goes and if some parts are too difficult to fix before
the release.

In the mean time, please test and report. Testers have been amazingly
helpful and determined these last months, and that's what makes the
quality in the end. So please continue like this!

Last point, I've been backporting all relevant fixes to 1.4 and am
planning to issue 1.4.25 once I have finished validating them all.

And for those who ask "when will 1.5-final be released ?", let's say
"when it's ready, probably soon".

Willy

---
Usual links below :

 Site index   : http://haproxy.1wt.eu/
 Sources  : http://h

HAProxy 1.5 possible bug

[Fredz Fredz]

Hello,

Is this a known bug in HAProxy 1.5?
When I use 0.0.0.0 or * as server address for a certain host, HAProxy
crashes with a oom_killer log. This is what is in the man page:

server
...

Address “0.0.0.0″ or “*” has a special meaning.
It indicates that the connection will be forwarded to the same IP
address as the one from the client connection. This is useful in
transparent proxy architectures where the client’s connection is
intercepted and haproxy must forward to the original destination
address.

eg in the backend:

use-server www.speedtest.net if { hdr_sub(host) speedtest.net }
server www.speedtest.net *

so what should happen is that the alias 'www.speedtest.net' should be
equal to the same IP address as was transmitted.

Or am I doing something wrong?

Thanks
Fred

Re: about HAProxy 1.5

[Willy Tarreau]

Hello,

On Mon, Dec 02, 2013 at 04:05:40PM -0700, Xie Qingshan wrote:


Please could you fix your mailer so that it sends a correct date
with your e-mails ? And please don't reuse old posts to send a
new question, it makes it much harder to follow threads.

Thanks,
Willy

[ANNOUNCE] haproxy-1.5-dev23

[Willy Tarreau]

Hi,

here comes dev23, addressing half of what was planned before 1.5-final.

Among the user-visible changes, these ones come to mind :

  - use_backend now support dynamic backend names built using the log-format
string like %[expression], and the condition is now optional ;

  - SSL supports ALPN as available in OpenSSL 1.0.2 and not as provided in
early patches ;

  - samples/converters/patterns : the maps' and acls' usage of patterns were
merged so that they can now be updated dynamically from the CLI. Very
large ACLs could possibly use slightly more memory, at the benefit of not
having to reload the configuration to update them. IPv6 addresses are now
indexed using trees.

  - new "language" converter to extract the preferred language among a list,
respecting the q-values expressed by the client.

  - SSL now support dynamic record size adjustments to send small objects with
a low latency and large objects with an optimal bandwidth. This can save a
few hundreds of ms on page load time for ssl-only sites.

  - use of tree-based backend name lookups should improve the startup time
and runtime speed for large configurations with many backends.

  - fixed the request body parser to wait for either the first chunk of data
to be complete or the buffer to be full. This removes the need for the
the max_wait argument to "balance url_param" which is now silently ignored.

  - fixed compression of chunked HTTP responses which are now enabled again.

  - emit warnings on misplaced http-request and use-server rules

As usual, a number of bugs were fixed (35, but about 1/4 of them were introduced
after dev22).

Normally we just have to perform the agent-check modifications and to
prepare the bind-process changes for a post-release fix and we should be
OK for a release.

I tried to re-enable the chunked mode on the stats page that I disabled
when Igor Chan sent me a screenshot of a broken page. I did it on the
demo web site only, because it allows me to compress the output and
load faster :-)

I never understood what could cause the bug Igor faced, it's possible
that it was related to something else that was fixed since. I'll probably
re-introduce that patch early in the next snapshot so that we get more
coverage. If it breaks again, we'll fix or revert as it's not really
important anyway.

Willy
---
Usual links below :

 Site index   : http://haproxy.1wt.eu/
 Sources  : http://haproxy.1wt.eu/download/1.5/src/devel/
 Changelog: http://haproxy.1wt.eu/download/1.5/src/CHANGELOG
 Cyril's HTML doc : 
http://cbonte.github.com/haproxy-dconv/configuration-1.5.html

And the changelog :

2014/04/23 : 1.5-dev23
- BUG/MINOR: reject malformed HTTP/0.9 requests
- MINOR: systemd wrapper: re-execute on SIGUSR2
- MINOR: systemd wrapper: improve logging
- MINOR: systemd wrapper: propagate exit status
- BUG/MINOR: tcpcheck connect wrong behavior
- MEDIUM: proxy: support use_backend with dynamic names
- MINOR: stats: Enhancement to stats page to provide information of last 
session time.
- BUG/MEDIUM: peers: fix key consistency for integer stick tables
- DOC: fix a typo on http-server-close and encapsulate options with 
double-quotes
- DOC: fix fetching samples syntax
- MINOR: ssl: add ssl_fc_unique_id to fetch TLS Unique ID
- MEDIUM: ssl: Use ALPN support as it will be available in OpenSSL 1.0.2
- DOC: fix typo
- CLEANUP: code style: use tabs to indent codes instead of spaces
- DOC: fix a few config typos.
- BUG/MINOR: raw_sock: also consider ENOTCONN in addition to EAGAIN for 
recv()
- DOC: lowercase format string in unique-id
- MINOR: set IP_FREEBIND on IPv6 sockets in transparent mode
- BUG/MINOR: acl: req_ssl_sni fails with SSLv3 record version
- BUG/MINOR: build: add missing objects in osx and bsd Makefiles
- BUG/MINOR: build: handle whitespaces in wc -l output
- BUG/MINOR: Fix name lookup ordering when compiled with USE_GETADDRINFO
- MEDIUM: ssl: Add standardized DH parameters >= 1024 bits
- BUG/MEDIUM: map: The map parser includes blank lines.
- BUG/MINOR: log: The log of quotted capture header has been terminated by 
2 quotes.
- MINOR: standard: add function "encode_chunk"
- BUG/MINOR: http: fix encoding of samples used in http headers
- MINOR: sample: add hex converter
- MEDIUM: sample: change the behavior of the bin2str cast
- MAJOR: auth: Change the internal authentication system.
- MEDIUM: acl/pattern: standardisation "of pat_parse_int()" and 
"pat_parse_dotted_ver()"
- MEDIUM: pattern: The pattern parser no more uses  and just takes 
one string.
- MEDIUM: pattern: Change the prototype of the function pattern_register().
- CONTRIB: ip6range: add a network IPv6 range to mask converter
- MINOR: pattern: separe list element from the data part.
- MEDIUM: pattern: add indexation function.
- MEDIUM: 

[ANNOUNCE] haproxy-1.5-dev24

[Willy Tarreau]

OK, so as usual when I'm sending a version with many patches, a second
one is not that far away...

1.5-dev23 was a mess. I broke a few things very quickly in the last
changes made to fix the chunked compression. On the positive side of
things, I start to have a nose for this, the faulty commit was properly
tagged "MAJOR" :-)

So three things broke :
  - forwarding of a message body (request or response) would automatically
stop after the transfer timeout strikes, and with no error. This is the
reason some people retrieved truncated tarballs from the main site.

  - redirects failed to update the msg->next offset after consuming the
request, so if they were made with keep-alive enabled and starting with
a slash (relative location), then the buffer was shifted by a negative
amount of data, causing a crash.

That's for the regressions. Now the other issues fixed :

  - the bug that was randomly breaking the stats page in chunked mode was
fixed, and it could likely have affected peers synchronization as well,
with some rare but possible connection breakages.

  - 100-continue responses could be blocked if the final response came in
the same packet.

  - fixed reporting of some server errors and client errors in the stats
and logs.

  - stats activation in frontend/backend/defaults was still a mess start
started in 1.3.4 and which did not match the doc since then. Now it's
supported in front/back/defaults and we don't emit a warning anymore
when it's specified in a frontend.

  - the code to standardize DH parameters caused an important performance
regression for Sander Klein, so it was temporarily reverted for the
time needed to understand the cause and to fix it.

And some improvements :

  - the stats page now finally supports keep-alive and compression. The
compression ratio is between 75 and 85% depending on the number of
servers, that's quite appreciable.

  - some people were complaining about the time it took to start checks
with long intervals, so now there's a global max-spread-checks setting
to limit the delay between the first and the last check.

  - now we have the discussed max-keep-alive-queue setting to avoid reusing
a connection if a server already has a certain amount of queuing.

  - http-request/response support set-map/del-map, add-acl/del-acl to add
or remove patterns to maps and acls on the fly by extracting them from
various sources. The lookups are still linear since they search in the
pattern database (text version), but that can already be very useful
as a complement to stick tables.

  - we now detect TLSv1 handshakes containing a heartbeat record, and
among them, specifically the Heartbleed attack which we can block. It
will only work with a heartbeat-enabled openssl though. The detected
heartbeat type is reported. At least it will tell us whether we're
facing real handshake failures or just noise from random attackers
and bots. It can also save ones' butts when accidentely deploying
using a fresh non-upgraded install of a system shipped with the
vulnerability.

And that's all, it's enough for a new release. Please report any bugs you
find. I hope none, of course. I was surprized to see that little bug reports
on dev23 despite the two I faced, but maybe we're starting to hit the rare
ones already.

Willy
---
Usual links below :

 Site index   : http://haproxy.1wt.eu/
 Sources  : http://haproxy.1wt.eu/download/1.5/src/devel/
 Changelog: http://haproxy.1wt.eu/download/1.5/src/CHANGELOG
 Cyril's HTML doc : 
http://cbonte.github.com/haproxy-dconv/configuration-1.5.html

And the changelog :

2014/04/26 : 1.5-dev24
- MINOR: pattern: find element in a reference
- MEDIUM: http: ACL and MAP updates through http-(request|response) rules
- MEDIUM: ssl: explicitly log failed handshakes after a heartbeat
- DOC: Full section dedicated to the converters
- MEDIUM: http: register http-request and http-response keywords
- BUG/MINOR: compression: correctly report incoming byte count
- BUG/MINOR: http: don't report server aborts as client aborts
- BUG/MEDIUM: channel: bi_putblk() must not wrap before the end of buffer
- CLEANUP: buffers: remove unused function buffer_contig_space_with_res()
- MEDIUM: stats: reimplement HTTP keep-alive on the stats page
- BUG/MAJOR: http: fix timeouts during data forwarding
- BUG/MEDIUM: http: 100-continue responses must process the next part 
immediately
- MEDIUM: http: move skipping of 100-continue earlier
- BUILD: stats: let gcc know that last_fwd cannot be used uninitialized...
- CLEANUP: general: get rid of all old occurrences of "session *t"
- CLEANUP: http: remove the useless "if (1)" inherited from version 1.4
- BUG/MEDIUM: stats: mismatch between behaviour and doc about front/back
- MEDIUM: http: enable analysers to have keep-alive on 

[ANNOUNCE] haproxy-1.5-dev25

[Willy Tarreau]

Hi all,

we're almost done!

Now the bind-process mess is fixed so that we now support per-listener
process binding using the "process" bind keyword, which ensures that
we won't need to change the config format during the stable release if
we want to slightly improve it. And that allows us to have one stats
socket per process, finally!

As most of you might have been following recently, four important issues
were fixed since dev24. One could cause crashes on out-of-memory. Another
one concerns FreeBSD where the shared session cache could have been used
without locking, causing random crashes as well. The recent fixes for HTTP
request body forwarding randomly caused pauses when using balance url_param.
Last, arguments "-i" and "-n" were ignored on ACLs since dev23.

Some low hanging fruits from the roadmap were done as well. Half-closed
timeouts are now supported, so it will be possible to quickly get rid of
dead connections even with long tunnel timeouts. Unix sockets are now
supported on the server side, as well as abstract namespace sockets on
Linux. This allows backends and frontend to connect together without
consuming TCP ports. The old unmaintained BSD and OSX Makefiles were
removed, so BSD users will have to use GNU make, which most of them
already use anyway since the BSD makefile did not implement half of the
1.5 features. Version 2 of the PROXY protocol was implemented on the
server side. A few other minor improvements were made, as seen in the
changelog below.

What's missing before 1.5-final ? Only the started updates to the agent-
check. Maybe by now we'll have fixed peers to support running with nbproc
greater than one, but I don't fix this as a top priority considering that
the real priority is to stabilize the API (and agent-check is part of the
API).

As usual, bug reports were of high quality and very useful, so I'm
addressing a big thanks to all those involved!

Ah, I have set up "git.haproxy.org" on the formilux.org server and updated
the README to reference it instead of git.1wt.eu, hoping it will avoid
some of the disappointment some people have experienced with the slow
master server.

On a side note, Patrick Hemmer informed me that spamcop is currently
blacklisting the mailing list server as spam sender. Since there's no
spontaneous e-mails sent from there, I suspect that someone subscribed
one of their traps. I contacted them just in case we'd have a response.
So if you're having difficulties receiving some of the mails from the
ML, you'll probably need to whitelist the server's address (88.191.124.161)
or disable spamcop as I did for dnsbl which was abusively blocking gmail.

Have a nice week-end,
Willy

---
Usual links below :

 Site index   : http://haproxy.1wt.eu/
 Sources  : http://haproxy.1wt.eu/download/1.5/src/devel/
 Changelog: http://haproxy.1wt.eu/download/1.5/src/CHANGELOG
 Cyril's HTML doc : 
http://cbonte.github.com/haproxy-dconv/configuration-1.5.html

And the changelog :

2014/05/10 : 1.5-dev25
- MEDIUM: connection: Implement and extented PROXY Protocol V2
- MINOR: ssl: clean unused ACLs declarations
- MINOR: ssl: adds fetchs and ACLs for ssl back connection.
- MINOR: ssl: merge client's and frontend's certificate functions.
- MINOR: ssl: adds ssl_f_sha1 fetch to return frontend's certificate 
fingerprint
- MINOR: ssl: adds sample converter base64 for binary type.
- MINOR: ssl: convert to binary ssl_fc_unique_id and ssl_bc_unique_id.
- BUG/MAJOR: ssl: Fallback to private session cache if current lock mode is 
not supported.
- MAJOR: ssl: Change default locks on ssl session cache.
- BUG/MINOR: chunk: Fix function chunk_strcmp and chunk_strcasecmp match a 
substring.
- MINOR: ssl: add global statement tune.ssl.force-private-cache.
- MINOR: ssl: remove fallback to SSL session private cache if lock init 
fails.
- BUG/MEDIUM: patterns: last fix was still not enough
- MINOR: http: export the smp_fetch_cookie function
- MINOR: http: generic pointer to rule argument
- BUG/MEDIUM: pattern: a typo breaks automatic acl/map numbering
- BUG/MAJOR: patterns: -i and -n are ignored for inlined patterns
- BUG/MINOR: proxy: unsafe initialization of HTTP transaction when 
switching from TCP frontend
- BUG/MINOR: http: log 407 in case of proxy auth
- MINOR: http: rely on the message body parser to send 100-continue
- MEDIUM: http: move reqadd after execution of http_request redirect
- MEDIUM: http: jump to dedicated labels after http-request processing
- BUG/MINOR: http: block rules forgot to increment the denied_req counter
- BUG/MINOR: http: block rules forgot to increment the session's request 
counter
- MEDIUM: http: move Connection header processing earlier
- MEDIUM: http: remove even more of the spaghetti in the request path
- MINOR: http: silently support the "block" action for http-request
- CLEANUP: proxy: rename "block_cond" to "b

Re: HAProxy 1.5 release?

[Patrick Hemmer]

Haproxy 1.6 is very close to release.
See http://marc.info/?l=haproxy&m=140129354705695 and
http://marc.info/?l=haproxy&m=140085816115800

-Patrick


*From: *Stephen Balukoff 
*Sent: * 2014-06-18 08:40:55 EDT
*To: *haproxy@formilux.org
*Subject: *HAProxy 1.5 release?

> Hey Willy!
>
> I'm involved in a group that is building a highly-scalable open source
> virtual appliance-based load balancer for use with cloud operating
> systems like OpenStack. We are planning on making haproxy the core
> component of the solution we're building.
>
> At my company we've actually been using haproxy 1.5 for a couple years
> now in production to great effect, and absolutely love it. But I'm
> having trouble getting the rest of the members of my team to go along
> with the idea of using 1.5 in our solution simply because of its
> "official" status as a development branch. There are just so many
> useful new features in 1.5 that I'd really rather not have to go back
> to 1.4 in our solution...
>
> So! My question is: What can we do to help y'all bring the 1.5 branch
> far enough along such that y'all are comfortable releasing it as the
> official "stable" branch of haproxy? (Note we do have people in our
> group with connections in some of the major linux distros who can help
> to fast-track its adoption into "official" releases of said distros.)
>
> Thanks,
> Stephen
>
> -- 
> Stephen Balukoff
> Blue Box Group, LLC
> (800)613-4305 x807

Re: HAProxy 1.5 release?

[Patrick Hemmer]

Err, pardon the typo, 1.5 :-)

-Patrick


*From: *Patrick Hemmer 
*Sent: * 2014-06-18 08:49:27 EDT
*To: *Stephen Balukoff , haproxy@formilux.org
*Subject: *Re: HAProxy 1.5 release?

> Haproxy 1.6 is very close to release.
> See http://marc.info/?l=haproxy&m=140129354705695 and
> http://marc.info/?l=haproxy&m=140085816115800
>
> -Patrick
>
> 
> *From: *Stephen Balukoff 
> *Sent: * 2014-06-18 08:40:55 EDT
> *To: *haproxy@formilux.org
> *Subject: *HAProxy 1.5 release?
>
>> Hey Willy!
>>
>> I'm involved in a group that is building a highly-scalable open
>> source virtual appliance-based load balancer for use with cloud
>> operating systems like OpenStack. We are planning on making haproxy
>> the core component of the solution we're building.
>>
>> At my company we've actually been using haproxy 1.5 for a couple
>> years now in production to great effect, and absolutely love it. But
>> I'm having trouble getting the rest of the members of my team to go
>> along with the idea of using 1.5 in our solution simply because of
>> its "official" status as a development branch. There are just so many
>> useful new features in 1.5 that I'd really rather not have to go back
>> to 1.4 in our solution...
>>
>> So! My question is: What can we do to help y'all bring the 1.5 branch
>> far enough along such that y'all are comfortable releasing it as the
>> official "stable" branch of haproxy? (Note we do have people in our
>> group with connections in some of the major linux distros who can
>> help to fast-track its adoption into "official" releases of said
>> distros.)
>>
>> Thanks,
>> Stephen
>>
>> -- 
>> Stephen Balukoff
>> Blue Box Group, LLC
>> (800)613-4305 x807
>

Re: HAProxy 1.5 release?

[Willy Tarreau]

On Wed, Jun 18, 2014 at 08:52:01AM -0400, Patrick Hemmer wrote:
> Err, pardon the typo, 1.5 :-)

This typo is a proof that we're close :-)

I merged today what I think might be the last commit. I'm working on
a human-readable changelog right now (ie: something that will also
enlight people who are not power users about the changes).

I realized that the web site is significantly outdated, I'll have to
update it a little bit so that newcomers are not too much confused
(ie: remove old stuff and performance reports).

Willy

Re: HAProxy 1.5 release?

[Stephen Balukoff]

Hi y'all!

Thanks for the responses and pointers. Willy-- if there's anything we can
to do help with this, please let us know! You have no idea how much we're
looking forward to this release!

Thanks,
Stephen


On Wed, Jun 18, 2014 at 1:09 PM, Willy Tarreau  wrote:

> On Wed, Jun 18, 2014 at 08:52:01AM -0400, Patrick Hemmer wrote:
> > Err, pardon the typo, 1.5 :-)
>
> This typo is a proof that we're close :-)
>
> I merged today what I think might be the last commit. I'm working on
> a human-readable changelog right now (ie: something that will also
> enlight people who are not power users about the changes).
>
> I realized that the web site is significantly outdated, I'll have to
> update it a little bit so that newcomers are not too much confused
> (ie: remove old stuff and performance reports).
>
> Willy
>
>
>


-- 
Stephen Balukoff
Blue Box Group, LLC
(800)613-4305 x807

Haproxy 1.5 ssl redirect

[Sean Patronis]

I seem to be having an interesting issue with forced ssl redirects in 
Haproxy 1.5.11


Sorry if this sounds clear as mud, but here goes:

When I load a domain that is served by haproxy that is supposed to force 
https, it initially forces the connection to be https (if you attempt to 
connect over http), but I get a Mixed content warning when it attempts 
to load another url that is based on the same domain.  If I allow the 
mixed content through the browser, it does not get redirected to https.  
I am sure I just have something configured incorrectly, but I am not 
sure where.


I go to URL:
https://localcaleb.test123.com/apps/test123/test.html

inside the test123 app it makes a protocol-less request to another app 
which ends up using http (since the backend is http balanced) using this 
URL:

http://localcaleb.test23.com/apps/testgw/login.jsp

Since the we have a redirect for ssl in place, shouldn't the request get 
forced to https?  It worked this way when apache was acting as our SSL 
reverse proxy.  What am I doing incorrectly?  If I allow mixed content 
in the browser, the haproxy logs show that it indeed connects over port 
80 without getting redirected to 443.



here is the fontend:

frontend localcaleb.test123.com ## local Backends
bind 10.0.60.5:80
bind 10.0.60.5:443  ssl crt /etc/certs/test.bundle no-sslv3 ciphers 
ECDHE-RSA-AES256-SHA384:AES256-SHA256:!RC4:HIGH:!MD5:!aNULL:!EDH:!AESGCM:!SSLV3:!eNULL

redirect scheme https if !{ ssl_fc }
option http-server-close
acl is_apps_match url_beg /apps/
use_backend caleblocal.test123.com if is_apps_match
default_backend caleb.test123.com



here are the relevant backends:

backend caleblocal.test123.com
reqrep ^([^\ ]*)\ /apps/(.*) \1\ /\2
server caleb-pc.staff.test123.com 192.168.166.182:8080 weight 50 check
server maint maint.test123.com:81 check backup
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { ssl_fc }


backend caleb.test123.com
reqrep ^([^\ ]*)\ /apps/(.*) \1\ /\2
server caleb 10.0.3.216:80 weight 50 check
server maint maint.test123.com:81 check backup
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { ssl_fc }


Thanks.

--
--Sean Patronis
Auto Data Direct Inc.
850.877.8804

questions for haproxy 1.5

[Labedan, Alain]

Hi,

I have HAPROXY in front of servers backend which are load balanced.


-  For terminated SSL haproxy, I want HAproxy give the good certificate 
to the client associated with the good domain .

I’ve not found how to configure HA for that:  I ‘ve 4 domains associated with 
one public IP in front . So how declare and use the 4  certificates SSL for the 
4 domains ?



-  How use affinity session ? is it SERVERID insert ?





Thanks for your answer .

Bests regards .

Alain Labedan

AVIS DE CONFIDENTIALITÉ : Ce message peut contenir des renseignements 
confidentiels appartenant exclusivement au Groupe CGI inc. ou à ses filiales. 
Si vous n'êtes pas le destinataire indiqué ou prévu dans ce message (ou 
responsable de livrer ce message à la personne indiquée ou prévue) ou si vous 
pensez que ce message vous a été adressé par erreur, vous ne pouvez pas 
utiliser ou reproduire ce message, ni le livrer à quelqu'un d'autre. Dans ce 
cas, vous devez le détruire et vous êtes prié d'avertir l'expéditeur en 
répondant au courriel.​

Re: question haproxy 1.5

[Willy Tarreau]

Hi Alain,

On Thu, Dec 03, 2015 at 12:14:20PM +, Labedan, Alain wrote:
> 
> Hi,
> 
> I have HAPROXY in front of servers backend which are load balanced.
> 
> So,  in https, we have only one address where the front https haproxy listen  
> :  bind  :443.
> And we have some clients for which, we only pass-through the traffic, so we 
> use the mode tcp .
> 
> Frontend  https-tcp-in
> Mode tcp
> Option tcplog
> Bind ip1:443
> Tcp-request inspect-delay 5s
> Tcp-request content accept if { req.ssl_hello_type 1 }
> Acl  regle1 req.ssl_sni  -i  
> Use_backend site1 if regle1
> 
> And we have also some clients for which in https, it is haproxy who have the 
> certicate, so we use mode http ?
> ..
> Mode http
> Bind :443 ssl crt /etc/ssl/pem
> Use_backend site1 if  { ssl_fc_sni  
> ..
> 
> Is it possible to manage both these two situations with only one socket for 
> listen https (bind  :443.)  ?

1.7 might make this possible but for now the only solution consists
in having the first layer fall back to the second one, so in short
you need a backend "default_site" which connects to the other frontend.

Best regards,
Willy

RE: question haproxy 1.5

[Lukas Tribus]

Hi Alain,


> We try 
> Bind :443 ssl crt /etc/ssl : but without success . 
> 
> Or 
> Bind :443 ssl crt /etc/ssl/*.pem : but without success 

Try:
bind :443 ssl crt /etc/ssl/

with the slash at the end.



Regards,

Lukas

  

RE: question haproxy 1.5

[Labedan, Alain]

Hi Lukas,

We try : 
bind :443 ssl crt /etc/ssl/   :   it's doesn't work 

Regards

Alain Labedan 
France GDC | CGI | Centre de Compétences IM France
17 place des Reflets, 92097 Paris la Défense cedex | France- 0157877340

alain.labe...@cgi.com |
 

CGI France SAS 
Capital : 12 913 933 euros | RCS Nanterre 702 042 755 
Siège social : Immeuble CB16 | 17, place des Reflets | 92400 Courbevoie | France

AVIS DE CONFIDENTIALITÉ : Ce message peut contenir des renseignements 
confidentiels appartenant exclusivement au Groupe CGI inc. ou à ses filiales. 
Si vous n'êtes pas le destinataire indiqué ou prévu dans ce message (ou 
responsable de livrer ce message à la personne indiquée ou prévue) ou si vous 
pensez que ce message vous a été adressé par erreur, vous ne pouvez pas 
utiliser ou reproduire ce message, ni le livrer à quelqu'un d'autre. Dans ce 
cas, vous devez le détruire et vous êtes prié d'avertir l'expéditeur en 
répondant au courriel.​

-Message d'origine-
De : Lukas Tribus [mailto:luky...@hotmail.com] 
Envoyé : lundi 14 décembre 2015 15:16
À : Labedan, Alain; haproxy@formilux.org
Objet : RE: question haproxy 1.5

Hi Alain,


> We try 
> Bind :443 ssl crt /etc/ssl : but without success . 
> 
> Or 
> Bind :443 ssl crt /etc/ssl/*.pem : but without success 

Try:
bind :443 ssl crt /etc/ssl/

with the slash at the end.



Regards,

Lukas

  

RE: question haproxy 1.5

[Lukas Tribus]

> Hi Lukas,
>
> We try :
> bind :443 ssl crt /etc/ssl/ : it's doesn't work

Can you elaborate what it doesn't work mean? Does haproxy
refuse to start with an error? If so, please post the exact error
message.

Also post the output of haproxy -vv and confirm that you are
starting haproxy with root privileges.



Thanks,

Lukas

  

Re: [ANNOUNCE] haproxy 1.5-dev1

[Andrew Azarov]

 Hi Willy!

This is a very great feature!
I've already set-up to tcp request acl's one with inspect delay other 
with rejecting. Seems to work!

Thank you very much, will donate soon :D

BRG,
Andrew

On 27.08.2010 1:59, Willy Tarreau wrote:

Hi !

Three months ago I was approached by the Stack Overflow Team team[1] who
needed to get some improvements in HAProxy. Overall, their needs would
have been addressed by the final release of version 1.5 scheduled around
the end of the year, but having to wait that long was not practical due
to some architectural constraints imposed by an intermediate solution.
They proposed that we find an agreement on which we could work together.
Since we were already having productive exchanges for some time, and I
knew they were good guys (after all they already donated to the project
last year), I accepted the deal.

Also, I must say that as a software engineer, it's always a lot better
to have someone explain their needs with high expectations than having
to guess how a feature will be used.

Geoff Dalgas and Jeff Atwood described to me in great details what they
needed to do : perform request throttling per IP address, possibly based
on various criteria, in order to limit risks of service abuse. That was
very interesting, because that feature was being thought about for about
4 years without enough time to completely develop it, and also the new
stickiness framework that was contributed by Exceliance and Loadbalancer.org
was making that really possible, although an important design rework had
to be operated first within the code.

During the tests with Geoff and Kyle Brandt, it appeared that some more
changes had to be operated to be able to store any criteria in the tables
(eg: bandwidth per IP address), and to be able to consult and change the
table contents at runtime, leading to a more and more generic code. Kyle
has been very patient and comprehensive, I think I have changed the
mechanisms and configuration syntax at least 5 or 6 times during the tests,
but he always took the time to understand the changes and adapt his
configurations. If I had been at his place, I would have got bored earlier,
so I owe him a big thanks for his patience !

Now the code has been running fine in production overthere, so it's time
to release it and merge it into the master branch. I won't extend further
on how it works, since Kyle has put an excellent explanation on his blog[2]
that is a lot more clear than the doc (that reminds me that the architecture
guide really needs some lifting).

Also, some of yours will like to get a quick status on the current code.
Some core changes that I wanted to do earlier will now start. But that means
that 1.5-dev1 should theorically be as stable as 1.4.8. I'm not saying that
I would suggest to anyone to push it into production, but it can clearly be
used to mitigate DDoS attacks as well as stop service abuses. I could get it
to stop connection floods slightly above 20 connections per second (yes,
two hundreds thousands) and let the good traffic pass through. So for this
reason, I think that people who are regularly exposed to such trouble may
find it useful to keep it handy.

Now what's next ? Right now the data in the tables is local to one process,
so it is not shared if you start multiple processes, nor it is across reloads.
The second step of the stickiness extensions developped by Exceliance and
Loadbalancer.org will include stickiness table synchronization between
multiple hosts. Some work will still be needed to be able to share counters,
but since this development is done in a flexible way, it should not be too
hard to adapt it later. BTW, I also owe a big kudos to the Git versionning
system, which has made it very easy to rework my patches after every change
and bugfix until they were looking good, through massive abuse of branching
and rebasing.

Too much talk. The code is available here :

site index : http://haproxy.1wt.eu/
sources: http://haproxy.1wt.eu/download/1.5/src/devel/
changelog  : http://haproxy.1wt.eu/download/1.5/src/CHANGELOG
binaries   : Since this is a development version, no binary is provided.

The last words naturally go to the really cool guys at Stack Overflow. It's
very nice to see some sites and companies involve time and money and take
risks to make Open Source products better. Of course they benefit from this
work, but at no point during the whole development did they try to reduce
the focus to their specific needs, quite the opposite. From the very first
exchanges, their goal clearly was to make the product better, and that must
be outlined. That's now achieved and I really appreciate their involvement.
Thank you guys !

Willy

[1] http://blog.serverfault.com/
[2] 
http://blog.serverfault.com/post/1016491873/better-rate-limiting-for-all-with-haproxy

[ANNOUNCE] haproxy 1.5-dev2 (fixup)

[Willy Tarreau]

Hi,

30 minutes after I installed 1.5-dev1 in front of my gitweb server,
Cyril Bonté contacted me to report a breakage. Diagnostics showed
that chunked encoding did not always work as expected, and that it
was sometimes possible to truncate the response after the last
chunk that was found in the recv() preceeding the one which reports
the close. In fact it's all a matter of timing, so many people will
never be hit and other ones will always (in my case, strace was enough
to make it disappear).

The bug has been fixed. While fixing it I realized that the bug could
also reduce the keep-alive rate on responses, because some responses
would still have the close forwarded to the client.

So I have issued 1.5-dev2 with this fix, so that testers don't waste
their time on something we know has a bug.

As usual, the code is available here :
 
   site index : http://haproxy.1wt.eu/
   sources: http://haproxy.1wt.eu/download/1.5/src/devel/
   changelog  : http://haproxy.1wt.eu/download/1.5/src/CHANGELOG
   binaries   : Since this is a development version, no binary is provided.
 
Now back to the kernel ;-)

Regards,
Willy

Re: [ANNOUNCE] haproxy 1.5-dev1

[Matt]

Hi Willy,

I was looking for something similar and couldn't believe it when I
came across the stack overflow post, because we use HA Proxy in
production, I thought to myself how come I don't know about that
feature!  And then I saw 1.5-dev1 :-D

Before I delve into trying it out, my requirements are slightly
different.  Rather than build the acl from the src IP, i'd like to
build it from the URI and HTTP method.  Is this currently possible?

So only allow 200 requests per minute for a URI match /john/index.htm
and HTTP method PUT? If this gets exceeded i'd like to reply with a
HTTP 503.

If it is i'll grab 1.5 and start playing.  If it isn't, then i'll be
happy to test any future builds.

Thanks,

Matt

On 27 August 2010 00:59, Willy Tarreau  wrote:
> Hi !
>
> Three months ago I was approached by the Stack Overflow Team team[1] who
> needed to get some improvements in HAProxy. Overall, their needs would
> have been addressed by the final release of version 1.5 scheduled around
> the end of the year, but having to wait that long was not practical due
> to some architectural constraints imposed by an intermediate solution.
> They proposed that we find an agreement on which we could work together.
> Since we were already having productive exchanges for some time, and I
> knew they were good guys (after all they already donated to the project
> last year), I accepted the deal.
>
> Also, I must say that as a software engineer, it's always a lot better
> to have someone explain their needs with high expectations than having
> to guess how a feature will be used.
>
> Geoff Dalgas and Jeff Atwood described to me in great details what they
> needed to do : perform request throttling per IP address, possibly based
> on various criteria, in order to limit risks of service abuse. That was
> very interesting, because that feature was being thought about for about
> 4 years without enough time to completely develop it, and also the new
> stickiness framework that was contributed by Exceliance and Loadbalancer.org
> was making that really possible, although an important design rework had
> to be operated first within the code.
>
> During the tests with Geoff and Kyle Brandt, it appeared that some more
> changes had to be operated to be able to store any criteria in the tables
> (eg: bandwidth per IP address), and to be able to consult and change the
> table contents at runtime, leading to a more and more generic code. Kyle
> has been very patient and comprehensive, I think I have changed the
> mechanisms and configuration syntax at least 5 or 6 times during the tests,
> but he always took the time to understand the changes and adapt his
> configurations. If I had been at his place, I would have got bored earlier,
> so I owe him a big thanks for his patience !
>
> Now the code has been running fine in production overthere, so it's time
> to release it and merge it into the master branch. I won't extend further
> on how it works, since Kyle has put an excellent explanation on his blog[2]
> that is a lot more clear than the doc (that reminds me that the architecture
> guide really needs some lifting).
>
> Also, some of yours will like to get a quick status on the current code.
> Some core changes that I wanted to do earlier will now start. But that means
> that 1.5-dev1 should theorically be as stable as 1.4.8. I'm not saying that
> I would suggest to anyone to push it into production, but it can clearly be
> used to mitigate DDoS attacks as well as stop service abuses. I could get it
> to stop connection floods slightly above 20 connections per second (yes,
> two hundreds thousands) and let the good traffic pass through. So for this
> reason, I think that people who are regularly exposed to such trouble may
> find it useful to keep it handy.
>
> Now what's next ? Right now the data in the tables is local to one process,
> so it is not shared if you start multiple processes, nor it is across reloads.
> The second step of the stickiness extensions developped by Exceliance and
> Loadbalancer.org will include stickiness table synchronization between
> multiple hosts. Some work will still be needed to be able to share counters,
> but since this development is done in a flexible way, it should not be too
> hard to adapt it later. BTW, I also owe a big kudos to the Git versionning
> system, which has made it very easy to rework my patches after every change
> and bugfix until they were looking good, through massive abuse of branching
> and rebasing.
>
> Too much talk. The code is available here :
>
>   site index : http://haproxy.1wt.eu/
>   sources    : http://haproxy.1wt.eu/download/1.5/src/devel/
>   changelog  : http://haproxy.1wt.eu/download/1.5/src/CHANGELOG
>   binaries   : Since this is a development version, no binary is provided.
>
> The last words naturally go to the really cool guys at Stack Overflow. It's
> very nice to see some sites and companies involve time and money and take
> risks to make Open Source produ

Re: [ANNOUNCE] haproxy 1.5-dev3

[Cyril Bonté]

Hi Willy,

Le vendredi 12 novembre 2010 00:34:29, Willy Tarreau a écrit :
> Hi,
> 
> finally we managed to merge all the stuff ! Haproxy 1.5-dev3 was released
> with everything that went into 1.4.9, plus some added bonus that were
> mainly developped at Exceliance :

I've quicky tested these 2 first features :

>   - support for binding to UNIX socket on the accept side. Haproxy can
> now receive connections over a UNIX socket. This is particularly
> useful when combined with stunnel (we also have a patch for that
> in the 'patches' directory).

First of all, it works :-) But using ab to stress stunnel+haproxy, I got some 
"SSL read failed" errors (with at least 10 concurrent connections on a 
laptop). I suspect it comes from ab and not from stunnel or haproxy, but as 
soon as I go back to TCP instead of a UNIX socket, I don't have these errors. 
I also tested stunnel+nginx with UNIX sockets, still no error.
And replacing ab with httperf, it always works.

>   - support for a new "PROXY" protocol that was designed to forward
> transport-level information between proxies. The idea is to permit a
> component like stunnel to inform haproxy about the protocol, source
> and destinations of an incoming connection, so that haproxy can make
> use of that everywhere internally (acls, logs, transparent, ...)
> instead of stunnel's address. The main advantage over the
> x-forwarded-for patch is that it now supports keep-alive and is not
> limited to HTTP anymore. When combined with the UNIX socket, it can make
> haproxy and stunnel integrate seamlessly and reliably. Obviously, we have
> a patch for stunnel ready too ;-)

It didn't work with "option http-server-close". My guess is that the 
AN_REQ_DECODE_PROXY bit is re-enabled after the first transaction.
I don't provide a full patch because I don't know if it's the better solution, 
but applying this fixes the issue :
--- haproxy-1.5-dev3/src/proto_http.c   2010-11-11 23:29:35.0 +0100
+++ /home/cbonte/Public/haproxy/haproxy-1.5-dev3/src/proto_http.c   
2010-11-12 
13:53:14.154398641 +0100
@@ -3949,6 +3949,7 @@
s->rep->lr -= s->req->size;
 
s->req->analysers |= s->listener->analysers;
+   s->req->analysers &= ~AN_REQ_DECODE_PROXY;
s->rep->analysers = 0;
 
http_silent_debug(__LINE__, s);

I'll make some tests on the other features soon.

-- 
Cyril Bonté

Re: [ANNOUNCE] haproxy 1.5-dev3

[Willy Tarreau]

Hi Cyril,

On Fri, Nov 12, 2010 at 02:07:22PM +0100, Cyril Bonté wrote:
> >   - support for binding to UNIX socket on the accept side. Haproxy can
> > now receive connections over a UNIX socket. This is particularly
> > useful when combined with stunnel (we also have a patch for that
> > in the 'patches' directory).
> 
> First of all, it works :-) But using ab to stress stunnel+haproxy, I got some 
> "SSL read failed" errors (with at least 10 concurrent connections on a 
> laptop). I suspect it comes from ab and not from stunnel or haproxy, but as 
> soon as I go back to TCP instead of a UNIX socket, I don't have these errors. 
> I also tested stunnel+nginx with UNIX sockets, still no error.
> And replacing ab with httperf, it always works.

Do you know if keep-alive was involved in any of these tests ?

> >   - support for a new "PROXY" protocol that was designed to forward
> > transport-level information between proxies. The idea is to permit a
> > component like stunnel to inform haproxy about the protocol, source
> > and destinations of an incoming connection, so that haproxy can make
> > use of that everywhere internally (acls, logs, transparent, ...)
> > instead of stunnel's address. The main advantage over the
> > x-forwarded-for patch is that it now supports keep-alive and is not
> > limited to HTTP anymore. When combined with the UNIX socket, it can make
> > haproxy and stunnel integrate seamlessly and reliably. Obviously, we have
> > a patch for stunnel ready too ;-)
> 
> It didn't work with "option http-server-close". My guess is that the 
> AN_REQ_DECODE_PROXY bit is re-enabled after the first transaction.
> I don't provide a full patch because I don't know if it's the better 
> solution, 
> but applying this fixes the issue :
> --- haproxy-1.5-dev3/src/proto_http.c 2010-11-11 23:29:35.0 +0100
> +++ /home/cbonte/Public/haproxy/haproxy-1.5-dev3/src/proto_http.c 
> 2010-11-12 
> 13:53:14.154398641 +0100
> @@ -3949,6 +3949,7 @@
>   s->rep->lr -= s->req->size;
>  
>   s->req->analysers |= s->listener->analysers;
> + s->req->analysers &= ~AN_REQ_DECODE_PROXY;
>   s->rep->analysers = 0;
>  
>   http_silent_debug(__LINE__, s);
> 

Good catch, you're perfectly right, I did not think about this case !
Right now we should apply your fix as-is. Later we'd probably try to
split analysers between connection-based and transaction-based.

> I'll make some tests on the other features soon.

Thanks !
Willy

Re: [ANNOUNCE] haproxy 1.5-dev3

[Cyril Bonté]

Le vendredi 12 novembre 2010 15:05:40, Willy Tarreau a écrit :
> On Fri, Nov 12, 2010 at 02:07:22PM +0100, Cyril Bonté wrote:
> > >   - support for binding to UNIX socket on the accept side. Haproxy can
> > >   
> > > now receive connections over a UNIX socket. This is particularly
> > > useful when combined with stunnel (we also have a patch for that
> > > in the 'patches' directory).
> > 
> > First of all, it works :-) But using ab to stress stunnel+haproxy, I got
> > some "SSL read failed" errors (with at least 10 concurrent connections
> > on a laptop). I suspect it comes from ab and not from stunnel or
> > haproxy, but as soon as I go back to TCP instead of a UNIX socket, I
> > don't have these errors. I also tested stunnel+nginx with UNIX sockets,
> > still no error.
> > And replacing ab with httperf, it always works.
> 
> Do you know if keep-alive was involved in any of these tests ?

I tried both, It's easier to reproduce without keep-alive.
Actually, I also met the issue with httperf.

My configuration files :
# stunnel.conf
socket = l:TCP_NODELAY=1
socket = r:TCP_NODELAY=1
foreground=yes
pid=/var/run/stunnel.pid
unix-sockets-dir=/var/run
debug=3

[localhost-uxst]
cert = /home/cbonte/tmp/server.crt
key  = /home/cbonte/tmp/server.key
accept=0.0.0.0:8443
connect=/ssl.sock
TIMEOUTclose = 0
;sendproxy=yes

[localhost-tcp]
cert = /home/cbonte/tmp/server.crt
key  = /home/cbonte/tmp/server.key
accept=0.0.0.0:8444
connect=127.0.0.1:8080
TIMEOUTclose = 1
;sendproxy=yes

# haproxy.conf
global
stats socket /var/run/haproxy.sock

defaults
timeout server 60s
timeout client 60s
timeout connect 10s

listen https-in
bind /var/run/ssl.sock user root mode 600 # accept-proxy
bind :8080

mode http
#option http-server-close
option httpclose
stats enable

server local localhost:80

Using the UNIX Socket 
httperf --server localhost --port 8443 --uri / --rate 100 --num-conn 1000 \
--ssl --num-call 1
=> I see a lot of "readsocket: Invalid argument (22)" in stunnel

Using the TCP Socket 
httperf --server localhost --port 8444 --uri / --rate 100 --num-conn 1000 \
--ssl --num-call 1
=> no error message

-- 
Cyril Bonté

Re: [ANNOUNCE] haproxy 1.5-dev3

[Willy Tarreau]

On Fri, Nov 12, 2010 at 03:51:11PM +0100, Cyril Bonté wrote:
(...)
> > Do you know if keep-alive was involved in any of these tests ?
> 
> I tried both, It's easier to reproduce without keep-alive.
> Actually, I also met the issue with httperf.
> 
> My configuration files :
> # stunnel.conf
> socket = l:TCP_NODELAY=1
> socket = r:TCP_NODELAY=1
> foreground=yes
> pid=/var/run/stunnel.pid
> unix-sockets-dir=/var/run
> debug=3
> 
> [localhost-uxst]
> cert = /home/cbonte/tmp/server.crt
> key  = /home/cbonte/tmp/server.key
> accept=0.0.0.0:8443
> connect=/ssl.sock
> TIMEOUTclose = 0
> ;sendproxy=yes
> 
> [localhost-tcp]
> cert = /home/cbonte/tmp/server.crt
> key  = /home/cbonte/tmp/server.key
> accept=0.0.0.0:8444
> connect=127.0.0.1:8080
> TIMEOUTclose = 1
> ;sendproxy=yes
> 
> # haproxy.conf
> global
>   stats socket /var/run/haproxy.sock
> 
> defaults
>   timeout server 60s
>   timeout client 60s
>   timeout connect 10s
> 
> listen https-in
>   bind /var/run/ssl.sock user root mode 600 # accept-proxy
>   bind :8080
> 
>   mode http
>   #option http-server-close
>   option httpclose
>   stats enable
> 
>   server local localhost:80
> 
> Using the UNIX Socket 
> httperf --server localhost --port 8443 --uri / --rate 100 --num-conn 1000 \
> --ssl --num-call 1
> => I see a lot of "readsocket: Invalid argument (22)" in stunnel
> 
> Using the TCP Socket 
> httperf --server localhost --port 8444 --uri / --rate 100 --num-conn 1000 \
> --ssl --num-call 1
> => no error message

Thank you Cyril, I'll forward all that material to Emeric in case
he finds a clue about that. I hope we're not hitting buffer size
limits or things like this on the unix sockets :-/

TCP_NODELAY should not be set because it does not exist on the UNIX
sockets, but I don't think there is any relation. More likely it's
a matter of a connection limit or too fast reuse somewhere, and I'm
not used to tune for that !

Thanks !
Willy

Re: [ANNOUNCE] haproxy 1.5-dev3

[Guillaume Bourque]

You guy's simply Rock !!!

keep up the very good work.

Bye


Willy Tarreau a écrit :

On Fri, Nov 12, 2010 at 03:51:11PM +0100, Cyril Bonté wrote:
(...)
  

Do you know if keep-alive was involved in any of these tests ?
  

I tried both, It's easier to reproduce without keep-alive.
Actually, I also met the issue with httperf.

My configuration files :
# stunnel.conf
socket = l:TCP_NODELAY=1
socket = r:TCP_NODELAY=1
foreground=yes
pid=/var/run/stunnel.pid
unix-sockets-dir=/var/run
debug=3

[localhost-uxst]
cert = /home/cbonte/tmp/server.crt
key  = /home/cbonte/tmp/server.key
accept=0.0.0.0:8443
connect=/ssl.sock
TIMEOUTclose = 0
;sendproxy=yes

[localhost-tcp]
cert = /home/cbonte/tmp/server.crt
key  = /home/cbonte/tmp/server.key
accept=0.0.0.0:8444
connect=127.0.0.1:8080
TIMEOUTclose = 1
;sendproxy=yes

# haproxy.conf
global
stats socket /var/run/haproxy.sock

defaults
timeout server 60s
timeout client 60s
timeout connect 10s

listen https-in
bind /var/run/ssl.sock user root mode 600 # accept-proxy
bind :8080

mode http
#option http-server-close
option httpclose
stats enable

server local localhost:80

Using the UNIX Socket 
httperf --server localhost --port 8443 --uri / --rate 100 --num-conn 1000 \

--ssl --num-call 1
=> I see a lot of "readsocket: Invalid argument (22)" in stunnel

Using the TCP Socket 
httperf --server localhost --port 8444 --uri / --rate 100 --num-conn 1000 \

--ssl --num-call 1
=> no error message



Thank you Cyril, I'll forward all that material to Emeric in case
he finds a clue about that. I hope we're not hitting buffer size
limits or things like this on the unix sockets :-/

TCP_NODELAY should not be set because it does not exist on the UNIX
sockets, but I don't think there is any relation. More likely it's
a matter of a connection limit or too fast reuse somewhere, and I'm
not used to tune for that !

Thanks !
Willy


  



--
Guillaume Bourque, B.Sc.,
consultant, infrastructures technologiques libres !
Logisoft Technologies inc.  http://www.logisoftech.com
514 576-7638, http://ca.linkedin.com/in/GuillaumeBourque/fr

Re: [ANNOUNCE] haproxy 1.5-dev3

[Ross West]

Hi Willy,

I got a compile error on Freebsd while doing up the new port for
1.5.dev3

This machine is running Freebsd-8.1 Release, amd64 arch.

-= start
... much successful compile output clipped...
gcc -Iinclude -Iebtree -Wall  -O2 -g   -DFREEBSD_PORTS-DTPROXY 
-DCONFIG_HAP_CRYPT -DENABLE_POLL -DENABLE_KQUEUE -DUSE_PCRE 
-I/usr/local/include  -DCONFIG_HAPROXY_VERSION=\"1.5-dev3\" 
-DCONFIG_HAPROXY_DATE=\"2010/11/11\" -c -o src/signal.o src/signal.c
gcc -Iinclude -Iebtree -Wall  -O2 -g   -DFREEBSD_PORTS-DTPROXY 
-DCONFIG_HAP_CRYPT -DENABLE_POLL -DENABLE_KQUEUE -DUSE_PCRE 
-I/usr/local/include  -DCONFIG_HAPROXY_VERSION=\"1.5-dev3\" 
-DCONFIG_HAPROXY_DATE=\"2010/11/11\" -c -o src/acl.o src/acl.c
gcc -Iinclude -Iebtree -Wall  -O2 -g   -DFREEBSD_PORTS-DTPROXY 
-DCONFIG_HAP_CRYPT -DENABLE_POLL -DENABLE_KQUEUE -DUSE_PCRE 
-I/usr/local/include  -DCONFIG_HAPROXY_VERSION=\"1.5-dev3\" 
-DCONFIG_HAPROXY_DATE=\"2010/11/11\" -c -o src/pattern.o src/pattern.c
In file included from include/proto/pattern.h:25,
 from src/pattern.c:16:
include/types/pattern.h:57: error: expected specifier-qualifier-list before 
'int32_t'
src/pattern.c: In function 'pattern_arg_str':
src/pattern.c:484: error: 'union pattern_arg_data' has no member named 'str'
src/pattern.c:485: error: 'union pattern_arg_data' has no member named 'str'
gmake: *** [src/pattern.o] Error 1
*** Error code 1

Stop in /usr/ports/net/haproxy-devel.
*** Error code 1

Stop in /usr/ports/net/haproxy-devel.
-=

If there's anything specific you'd want me to check into, let me know.

Cheers,
  Ross.


-- 

Re: [ANNOUNCE] haproxy 1.5-dev3

[Willy Tarreau]

Hi Ross,

On Fri, Nov 12, 2010 at 01:21:11PM -0500, Ross West wrote:
> Hi Willy,
> 
> I got a compile error on Freebsd while doing up the new port for
> 1.5.dev3
> 
> This machine is running Freebsd-8.1 Release, amd64 arch.
> 
> -= start
> ... much successful compile output clipped...
> gcc -Iinclude -Iebtree -Wall  -O2 -g   -DFREEBSD_PORTS-DTPROXY 
> -DCONFIG_HAP_CRYPT -DENABLE_POLL -DENABLE_KQUEUE -DUSE_PCRE 
> -I/usr/local/include  -DCONFIG_HAPROXY_VERSION=\"1.5-dev3\" 
> -DCONFIG_HAPROXY_DATE=\"2010/11/11\" -c -o src/signal.o src/signal.c
> gcc -Iinclude -Iebtree -Wall  -O2 -g   -DFREEBSD_PORTS-DTPROXY 
> -DCONFIG_HAP_CRYPT -DENABLE_POLL -DENABLE_KQUEUE -DUSE_PCRE 
> -I/usr/local/include  -DCONFIG_HAPROXY_VERSION=\"1.5-dev3\" 
> -DCONFIG_HAPROXY_DATE=\"2010/11/11\" -c -o src/acl.o src/acl.c
> gcc -Iinclude -Iebtree -Wall  -O2 -g   -DFREEBSD_PORTS-DTPROXY 
> -DCONFIG_HAP_CRYPT -DENABLE_POLL -DENABLE_KQUEUE -DUSE_PCRE 
> -I/usr/local/include  -DCONFIG_HAPROXY_VERSION=\"1.5-dev3\" 
> -DCONFIG_HAPROXY_DATE=\"2010/11/11\" -c -o src/pattern.o src/pattern.c
> In file included from include/proto/pattern.h:25,
>  from src/pattern.c:16:
> include/types/pattern.h:57: error: expected specifier-qualifier-list before 
> 'int32_t'

I see, it's probably just a matter of including . However, this one
is not present on all machines. It's interesting to note that the uint32_t at
the line just before did not cause any trouble. Since those types are rarely
used in haproxy, I'd rather replace "uint32_t" with "unsigned int" and "int32_t"
with "int".

Could you please try with this minuscule non-invasive patch ? If it works, I'll
merge it as-is for now.

Thanks!
Willy

diff --git a/include/types/pattern.h b/include/types/pattern.h
index 78614bc..a3d5c36 100644
--- a/include/types/pattern.h
+++ b/include/types/pattern.h
@@ -54,7 +54,7 @@ enum {
 union pattern_arg_data {
struct in_addr ip;/* used for ipv4 type */
uint32_t integer; /* used for unsigned 32bits integer type */
-   int32_t sinteger; /* used for signed 32bits integer type */
+   int sinteger; /* used for signed 32bits integer type */
struct chunk str;
 };
 

RE: [ANNOUNCE] haproxy 1.5-dev3

[Jeff Buchbinder]

On 11/11/2010 06:34 PM, Willy Tarreau wrote:
> Now you know where to get it :
>site index  : http://haproxy.1wt.eu/
>sources : http://haproxy.1wt.eu/download/1.5/src/devel/
>changelog   : http://haproxy.1wt.eu/download/1.5/src/CHANGELOG
>stunnel patches : http://haproxy.1wt.eu/download/patches/
>

OpenBSD 4.3 builds for this release (and hopefully future releases) are here:

http://www.mediafire.com/?u1m9uwiuo7af5

-- 
Jeff Buchbinder
Principal Engineer / Interim Director of Infrastructure
Rave Mobile Safety, Inc
jbuchbin...@ravemobilesafety.com 

Re: [ANNOUNCE] haproxy 1.5-dev3

[Cyril Bonté]

Hi Willy,

Le vendredi 12 novembre 2010 16:45:54, Willy Tarreau a écrit :
> Thank you Cyril, I'll forward all that material to Emeric in case
> he finds a clue about that. I hope we're not hitting buffer size
> limits or things like this on the unix sockets :-/

OK, it took me some times and a lot of tests/modifications in stunnel and 
haproxy but I've found the limit.
In proto_uxst.c there's a call to listen(sock, 0)

I tried with listen(sock, 2000) and could run
 $ ab -n1 -c500  https://localhost:8443/
 $ httperf --server localhost --port 8443 --uri / --rate 200 \
   --num-conn 1--ssl --num-call 1
without any problem, which was not the case before.

Now, as it's shared with the stats, I don't know what to do.
Should we use the listener backlog value for both or should we keep 0 for the 
stats ?

-- 
Cyril Bonté

Re: [ANNOUNCE] haproxy 1.5-dev3

[Willy Tarreau]

Hi Cyril,

On Sun, Nov 14, 2010 at 02:57:42PM +0100, Cyril Bonté wrote:
> Hi Willy,
> 
> Le vendredi 12 novembre 2010 16:45:54, Willy Tarreau a écrit :
> > Thank you Cyril, I'll forward all that material to Emeric in case
> > he finds a clue about that. I hope we're not hitting buffer size
> > limits or things like this on the unix sockets :-/
> 
> OK, it took me some times and a lot of tests/modifications in stunnel and 
> haproxy but I've found the limit.
> In proto_uxst.c there's a call to listen(sock, 0)

Ah cool, thank you for chasing this one down !

> I tried with listen(sock, 2000) and could run
>  $ ab -n1 -c500  https://localhost:8443/
>  $ httperf --server localhost --port 8443 --uri / --rate 200 \
>--num-conn 1--ssl --num-call 1
> without any problem, which was not the case before.
> 
> Now, as it's shared with the stats, I don't know what to do.
> Should we use the listener backlog value for both or should we keep 0 for the 
> stats ?

In my opinion, we should use the listener's backlog. This will require
some code changes in order to be able to pass the backlog's size to
create_uxst_socket(). On the other hand, this function is quite old now
and is only used by uxst_bind_listener(). Probably that it will be easier
to move its code there and get rid of the function.

Do you want to send a patch with that ?

Cheers,
Willy

Re: [ANNOUNCE] haproxy 1.5-dev3

[Cyril Bonté]

Le dimanche 14 novembre 2010 15:06:50, Willy Tarreau a écrit :
> In my opinion, we should use the listener's backlog. This will require
> some code changes in order to be able to pass the backlog's size to
> create_uxst_socket(). On the other hand, this function is quite old now
> and is only used by uxst_bind_listener(). Probably that it will be easier
> to move its code there and get rid of the function.
> 
> Do you want to send a patch with that ?

OK to send a patch, just the time to merge create_uxst_socket() in 
uxst_bind_listener(), then, and doing some tests ;-)

-- 
Cyril Bonté

[ANNOUNCE] haproxy 1.5-dev6 (IMPORTANT)

[Willy Tarreau]

Hi again,

This version is an update to the development branch which aims at
fixing the cookie parsing bug described in the 1.4 announce, as well
as the few issues reported these last two days, in relation with the
IPv6 changes.

There are very few patches for a development release, but I know that
a sensible number of sites are running on 1.5-dev[3..5] and I found
it important to offer a version that fixes all currently known issues.

Please find the code at the usual place below :

   site index  : http://haproxy.1wt.eu/
   sources : http://haproxy.1wt.eu/download/1.5/src/devel/
   changelog   : http://haproxy.1wt.eu/download/1.5/src/CHANGELOG

Regards,
Willy

Re: [ANNOUNCE] haproxy 1.5-dev7

[Willy Tarreau]

I forgot to add something : I have updated the README file to request
a change in the format of the subject in patches. To put it short,
I'd like that we avoid the square brackets for the parts we want to
keep in the commit message. The reason is that Git either removes all
words enclosed within brackets, or keeps them all. When a patch series
is submitted, the [PATCH] prefix is added and I have to remove it by
hand for each patch. Also, right now it's not easy to indicate the
criticity of a bug, so it as time to suggest a change. Anyway it's
not critical if you forget, it's just something I'd appreciate.

All the details are in the README.

I'll try not to forget to do it myself ;-)

Thanks,
Willy

Migrating haproxy 1.5 monitor-uri

[CJ Ess]

I need to migrate to a different URL for our haproxy health checks, and it
would be really helpful if I could respond to multiple URLs as part of the
transition, or could create an empty 200 response.

Re: HAProxy 1.5 vs 1.6

[Steven Le Roux]

Hi a first good coverage for a comparison between 1.5 and 1.6 would be
http://blog.haproxy.com/2015/10/14/whats-new-in-haproxy-1-6/

1.6 is perfectly considered stable and hasn't seen any maintenance
release for more than 2 months. It's being widely used so I would be
confident with it. It brings many improvements and features (libslz,
lua, server states checkpointing,...) over 1.5

For many "not so much critical" project I use to build the last dev
snapshot (from 1.7) since it helps testing the next release and it's
pretty stable too.



2016-11-09 15:38 GMT+01:00 Himer Martinez :
> Bonjour,
>
> Je suis désolé de vous spammer, je sais que vous avez autre chose à faire
> :-)
>
> Je compte utiliser HAProxy pour un nouveau projet (très grosse application),
> des experts me disent que la version stable/éprouvée est la 1.5 mais je vois
> que la 1.6 est aussi stable et disponible,
>
> Du coup, je voulais savoir quelles étaient les différences exactes et avoir
> un conseil sur la version à utiliser...
>
> Vraiment ! Merci d'avance !
>
> Et surtout, BRAVO, pour ce que vous faîtes !
>
> Cordialement
>
> Himer MARTINEZ



-- 
Steven Le Roux
Jabber-ID : ste...@jabber.fr
0x39494CCB 
2FF7 226B 552E 4709 03F0  6281 72D7 A010 3949 4CCB

Re: HAProxy 1.5 vs 1.6

[Willy Tarreau]

Hi Steven,

On Wed, Nov 09, 2016 at 09:20:10PM +0100, Steven Le Roux wrote:
> Hi a first good coverage for a comparison between 1.5 and 1.6 would be
> http://blog.haproxy.com/2015/10/14/whats-new-in-haproxy-1-6/
> 
> 1.6 is perfectly considered stable and hasn't seen any maintenance
> release for more than 2 months. It's being widely used so I would be
> confident with it. It brings many improvements and features (libslz,
> lua, server states checkpointing,...) over 1.5

I need to emit 1.6.10 since we finally tackled the last two peers bugs
and the last systemd bugs, but that's all. I consider that 1.6 really
reached its stable status with 1.6.9 which was the first one to kill
all *known* bugs. And interestingly, the bugs we see now on 1.6 often
also affect 1.5 so we've reached the regular maintenance stage where
bugs hit randomly without a marked preference for the last version.

I don't use 1.5 anymore either.

> For many "not so much critical" project I use to build the last dev
> snapshot (from 1.7) since it helps testing the next release and it's
> pretty stable too.

In fact development on 1.7 has been relatively quiet in part due to
the numerous bugs we faced on early 1.6, and the fact that it got
some internal improvements like filters and various keyword registration
mechanisms making it easier to "plug" code into standard places using
standardized hooks, limiting the need to touch sensitive areas. I tend
to consider that even with the very latest changes, 1.7 should be almost
as stable as 1.6. By the way when you look at the commit tags between
1.6 and 1.7, it's obvious that bug fixes represent a huge part of it,
thus 1.7 is 1.6++ :-)

191 BUG
175 MINOR
 70 MEDIUM
 56 DOC
 40 CLEANUP
 28 BUILD
  8 MAJOR
  7 REORG
  5 OPTIM
  3 SCRIPTS
  3 DEBUG
  1 TESTS
  1 FIX
  1 CONTRIB

Cheers,
Willy

Re: HAProxy 1.5 vs 1.6

[Pavlos Parissis]

On 09/11/2016 09:20 μμ, Steven Le Roux wrote:
> Hi a first good coverage for a comparison between 1.5 and 1.6 would be
> http://blog.haproxy.com/2015/10/14/whats-new-in-haproxy-1-6/
> 
> 1.6 is perfectly considered stable and hasn't seen any maintenance
> release for more than 2 months. It's being widely used so I would be
> confident with it. It brings many improvements and features (libslz,
> lua, server states checkpointing,...) over 1.5
> 

Same story here. 1.6 is a rock solid release and works fine.

Cheers,
Pavlos




signature.asc
Description: OpenPGP digital signature

Re: HAProxy 1.5 vs 1.6

[Markus Rietzler]

Am 10.11.16 um 10:24 schrieb Pavlos Parissis:
> On 09/11/2016 09:20 μμ, Steven Le Roux wrote:
>> Hi a first good coverage for a comparison between 1.5 and 1.6 would be
>> http://blog.haproxy.com/2015/10/14/whats-new-in-haproxy-1-6/
>>
>> 1.6 is perfectly considered stable and hasn't seen any maintenance
>> release for more than 2 months. It's being widely used so I would be
>> confident with it. It brings many improvements and features (libslz,
>> lua, server states checkpointing,...) over 1.5
>>
> 
> Same story here. 1.6 is a rock solid release and works fine.
> 
> Cheers,
> Pavlos
> 
> 
We are using 1.6.x for quite a long time and it runs perfectly!

Markus

Re: [ANNOUNCE] haproxy 1.5-dev7

[Baptiste]

On Sun, Sep 11, 2011 at 12:23 AM, Willy Tarreau  wrote:
> Hi all,
>
> Five months have elapsed since 1.5-dev6. A massive amount of changes was
> merged since then. Most of them were cleanups and optimizations. A number
> of changes were dedicated to making listeners more autonomous. The immediate
> effect is a more robust handling of resource saturation, and the second
> effect is the removal of the 10-years old maintain_proxies() function which
> was harming performance and hard to get over.
>
> Halog was improved too (faster with more filters). A significant number
> of external contributions were merged, among them the stats socket updates
> to clear session-table keys by values. There are too many changes to list,
> but nothing too dangerous, so I'd say it's the 1.5-dev version I trust the
> most today.
>
> I'm planning on putting all the focus on server-side keep-alive again. Some
> of the remaining issues have been overcome. Surely there are still a number,
> but we can't know if we don't try :-)
>
> Do not hesitate to give 1.5-dev7 a try. I'm currently updating all 1.5 I
> have to it.
>
>   site index      : http://haproxy.1wt.eu/
>   sources         : http://haproxy.1wt.eu/download/1.5/src/devel/
>   changelog       : http://haproxy.1wt.eu/download/1.5/src/CHANGELOG
>
> Cheers,
> Willy
>
>
>


Hi all,

I wrote up a small post entry on our blog which summarizes the main
new features:

http://blog.exceliance.fr/2011/10/03/whats-new-in-haproxy-1-5-dev7/

cheers

Re: [ANNOUNCE] haproxy 1.5-dev9

[Baptiste]

> I thought I could make the track-sc1 and track-sc2 actions track headers but
> some more changes were needed that were out of the scope of all these changes,
> so I left them for later.
>

That is really sad :)
Hopefully you'll be able to add string tracking to track-sc[12] soon,
cause we'll be able to do great things :)

cheers

Re: [ANNOUNCE] haproxy 1.5-dev9

[Willy Tarreau]

On Tue, May 08, 2012 at 11:38:53PM +0200, Baptiste wrote:
> > I thought I could make the track-sc1 and track-sc2 actions track headers but
> > some more changes were needed that were out of the scope of all these 
> > changes,
> > so I left them for later.
> >
> 
> That is really sad :)

No it's not sad because the code is really taking shape and such features
become easier to add day after day.

> Hopefully you'll be able to add string tracking to track-sc[12] soon,
> cause we'll be able to do great things :)

I know but as you're well aware, the most important for me is to ensure
that we can concurrently work on this code. So I sometimes prefer delay
minor features to focus on architectural changes which allow multiple
persons to develop in parallel. This is the most important as I'm still
too much the bottleneck.

Willy

Re: [ANNOUNCE] haproxy 1.5-dev9

[Baptiste]

> I know but as you're well aware, the most important for me is to ensure
> that we can concurrently work on this code. So I sometimes prefer delay
> minor features to focus on architectural changes which allow multiple
> persons to develop in parallel. This is the most important as I'm still
> too much the bottleneck.
>
> Willy
>

I know, but I was expecting that we could play with strings in stick
tables with this release and so I'm just a bit disappointed :)
Well, I'll wait a bit more time for it.

cheers

Re: [ANNOUNCE] haproxy 1.5-dev9

[Aleksandar Lazic]

Hi,

On 08-05-2012 22:33, Willy Tarreau wrote:

Hi all,

I found some time to work on haproxy last weeks and to perform a 
number of

fundamental changes that have been needed for a long time.


[snipp]

Some of these changes conflicted with the ACL and pattern frameworks, 
so it

was the right moment to merge them together. We now have a single
sample fetch
function for each type of data we want to extract, and both ACLs and 
patterns
rely on this. The first user-visible benefit from this is that ACLs 
can now

match cookies, URL parameters and arbitrary payloads. In practice,
the current
code is almost ready to enable session tracking on any input 
criteria. I
thought I could make the track-sc1 and track-sc2 actions track 
headers but

some more changes were needed that were out of the scope of all these
changes,
so I left them for later.

Since some ACLs and pattern fetch methods supported an argument, a
new argument
management framework was implemented, making it very easy to declare 
variable
number of typed arguments for new keywords. Thanks to this extension, 
I could
bring new optional arguments to hdr() and cook() fetch methods to 
specify an
occurrence number. This allows stick-tables to extract an IP address 
from a
precise occurrence of the X-Forwarded-For header for instance, and to 
write

ACLs which match such headers against networks found in files.


[snipp]

After all this changes is it still necessary to have the appsession 
directive in haproxy?


Could it not be removed to avoid confusions and future question what 
should be used appsession

or *cook*?

Cheers
Aleks

Re: [ANNOUNCE] haproxy 1.5-dev9

[Willy Tarreau]

Hi Aleks,

On Wed, May 09, 2012 at 12:26:38AM +0200, Aleksandar Lazic wrote:
> After all this changes is it still necessary to have the appsession 
> directive in haproxy?
> 
> Could it not be removed to avoid confusions and future question what 
> should be used appsession
> or *cook*?

I remember that there were a few special tricks in appsession, such
as request-learn and things like this. I think that right now we have
everything ready to emulate appsession, so maybe we could replace the
whole thing with stick-tables and have the appsession keyword just
alias the required configuration statements with a warning. This would
enable appsession sharing between LBs, the thing we wanted to implement
back in 2005 :-)

Are you interested in looking at this ?

Cheers,
Willy

Re: [ANNOUNCE] haproxy 1.5-dev9

[Baptiste]

Hi,

Yes, appsession has been obsoleted by "cookie" and "set-cookie" stick
tables pattern extraction (in HAProxy 1.5-dev7 as far as I remember).
As an example:

 stick-table type string len 32 size 10K
 stick store-response set-cookie(PHPSESSID)
 stick on cookie(PHPSESSID)

or, better, if your cookie is presented on the query string by the key
"session_id", then this would do the persistence as well:
 stick on url_param(session_id)

You can use "peers" section and to share the content of the table
between two LBs and to recover your table after a reload of haproxy.

regards

Re: [ANNOUNCE] haproxy 1.5-dev9

[Cyril Bonté]

Hi,

Le 09/05/2012 07:04, Baptiste a écrit :

Hi,

Yes, appsession has been obsoleted by "cookie" and "set-cookie" stick
tables pattern extraction (in HAProxy 1.5-dev7 as far as I remember).
As an example:

  stick-table type string len 32 size 10K
  stick store-response set-cookie(PHPSESSID)
  stick on cookie(PHPSESSID)

or, better, if your cookie is presented on the query string by the key
"session_id", then this would do the persistence as well:
  stick on url_param(session_id)


I may be wrong but I think there are still some points where 
"appsession" can't be replaced by stick tables.


- for example the use of path parameters :
http://example.com/url;jsessionid=XXX

- the use of cookie prefixes :
I don't see how to match cookies (or path parameters/query string) like 
ASPSESSIONIDXXX= (where XXX is dynamic).


Maybe it still requires some development if we want to replace it by an 
equivalent with stick tables.


--
Cyril Bonté

Re: [ANNOUNCE] haproxy 1.5-dev9

[Willy Tarreau]

Hi Cyril,

On Wed, May 09, 2012 at 08:55:45AM +0200, Cyril Bonté wrote:
> Hi,
> 
> Le 09/05/2012 07:04, Baptiste a écrit :
> >Hi,
> >
> >Yes, appsession has been obsoleted by "cookie" and "set-cookie" stick
> >tables pattern extraction (in HAProxy 1.5-dev7 as far as I remember).
> >As an example:
> >
> >  stick-table type string len 32 size 10K
> >  stick store-response set-cookie(PHPSESSID)
> >  stick on cookie(PHPSESSID)
> >
> >or, better, if your cookie is presented on the query string by the key
> >"session_id", then this would do the persistence as well:
> >  stick on url_param(session_id)
> 
> I may be wrong but I think there are still some points where 
> "appsession" can't be replaced by stick tables.
> 
> - for example the use of path parameters :
> http://example.com/url;jsessionid=XXX
> 
> - the use of cookie prefixes :
> I don't see how to match cookies (or path parameters/query string) like 
> ASPSESSIONIDXXX= (where XXX is dynamic).
> 
> Maybe it still requires some development if we want to replace it by an 
> equivalent with stick tables.

You're perfectly right.

Also the logic in appsession is somewhat different to the logic in
stick-tables. For instance, if a client comes with an inexistant cookie
and the server replaces it, then the client's cookie won't be learned.

With the "request-learn" option, it's even trickier. The client's cookie
will be learned from the URL parameter unless the server overwrites it
in the response.

I think it is fairly acceptable to change this behaviour a bit provided
that what currently works still works. After all it's not really a problem
if we store old expired entries in the table once in a while. But I want
to insist (like you did) on the fact that there is not a 1:1 match right
now.

Regards,
Willy

Re: [ANNOUNCE] haproxy 1.5-dev9

[Aleksandar Lazic]

Hi Willy,

On 09-05-2012 05:47, Willy Tarreau wrote:

Hi Aleks,

On Wed, May 09, 2012 at 12:26:38AM +0200, Aleksandar Lazic wrote:

After all this changes is it still necessary to have the appsession
directive in haproxy?

Could it not be removed to avoid confusions and future question what
should be used appsession
or *cook*?


I remember that there were a few special tricks in appsession, such
as request-learn and things like this. I think that right now we have
everything ready to emulate appsession, so maybe we could replace the
whole thing with stick-tables and have the appsession keyword just
alias the required configuration statements with a warning. This 
would
enable appsession sharing between LBs, the thing we wanted to 
implement

back in 2005 :-)


Yes with the peers we are now in a much better setup ;-)

Still I think a plugin witch is able to get/set data to memcached or 
Kyoto Tycoon

would be a hugh benefit ;-)


Are you interested in looking at this ?


Yes, I will think about it an keep you informed ;-).

Cheers

Re: [ANNOUNCE] haproxy 1.5-dev10

[Brane F. Gračnar]

10x for great progress!!!

I have a question regarding IP based stick tables. Currently i have the
following setup:

backend some_backend
 BEGIN: Session stickyness
stick on src table STICK_some_backend
stick on src6 table STICK6_some_backend

backend STICK_some_backend
stick-table type ip size 200k expire 2h

backend STICK6_some_backend
stick-table type ipv6 size 200k expire 2h

This works great for IPv4 and IPv6 clients. Since -dev9 src6 is gone,
but stick type "ipv6" is still there; i'd like to upgrade to -dev10 but
i'm confused about stick tables.

Are two backends for session stickyness still necessary or can i have
only one for both address families?

Best regards, Brane

Re: [ANNOUNCE] haproxy 1.5-dev10

[Willy Tarreau]

Hi Brane,

On Mon, May 14, 2012 at 11:02:13AM +0200, "Brane F. Gra??nar" wrote:
> 10x for great progress!!!
> 
> I have a question regarding IP based stick tables. Currently i have the
> following setup:
> 
> backend some_backend
>  BEGIN: Session stickyness
> stick on src table STICK_some_backend
> stick on src6 table STICK6_some_backend
> 
> backend STICK_some_backend
>   stick-table type ip size 200k expire 2h
> 
> backend STICK6_some_backend
>   stick-table type ipv6 size 200k expire 2h
> 
> This works great for IPv4 and IPv6 clients. Since -dev9 src6 is gone,
> but stick type "ipv6" is still there; i'd like to upgrade to -dev10 but
> i'm confused about stick tables.

IPv4 addresses can be cast to IPv6 addresses, so if you need to store
both IPv4 and IPv6 addresses, just use the IPv6 one and your IPv4 entries
will automatically be converted when stored :

backend some_backend
  BEGIN: Session stickyness
 stick on src table STICK_some_backend
 
backend STICK_some_backend
stick-table type ipv6 size 200k expire 2h

Regards,
Willy

Re: [ANNOUNCE] haproxy 1.5-dev10

[Brane F. Gračnar]

On 05/14/2012 11:15 AM, Willy Tarreau wrote:
> IPv4 addresses can be cast to IPv6 addresses, so if you need to store
> both IPv4 and IPv6 addresses, just use the IPv6 one and your IPv4 entries
> will automatically be converted when stored :
> 
> backend some_backend
>   BEGIN: Session stickyness
>  stick on src table STICK_some_backend
>  
> backend STICK_some_backend
>   stick-table type ipv6 size 200k expire 2h

Thanks alot, tried and works! Smooth upgrade!

Best regards, Brane

Re: [ANNOUNCE] haproxy 1.5-dev12

[Duncan Hall]

On 10/09/12 18:10, Willy Tarreau wrote:


Many bugs were fixes, and many were certainly introduced. If you observe any
bug, please report it, as I'd rather issue -dev13 quickly with many fixes.


Great work, very much appreciated.

I have rolled 1.5-dev12 into a test environment and noticed that the 
statistics page is now much slower to load. Previously (on 1.5-dev 11 
built 2012/06/04) the time between requesting the stats page and it 
displaying was about 2.5 seconds, this has now jumped to 10 seconds. It 
is not a big problem for me but I thought it was worth mentioning.


I am measuring this with Lori. 
https://addons.mozilla.org/en-US/firefox/addon/lori-life-of-request-info/


Thanks again,

Duncan

Re: [ANNOUNCE] haproxy 1.5-dev12

[Willy Tarreau]

Hi Duncan,

On Mon, Sep 10, 2012 at 07:16:30PM +1000, Duncan Hall wrote:
> On 10/09/12 18:10, Willy Tarreau wrote:
> >
> >Many bugs were fixes, and many were certainly introduced. If you observe 
> >any
> >bug, please report it, as I'd rather issue -dev13 quickly with many fixes.
> 
> Great work, very much appreciated.
> 
> I have rolled 1.5-dev12 into a test environment and noticed that the 
> statistics page is now much slower to load. Previously (on 1.5-dev 11 
> built 2012/06/04) the time between requesting the stats page and it 
> displaying was about 2.5 seconds, this has now jumped to 10 seconds. It 
> is not a big problem for me but I thought it was worth mentioning.
> 
> I am measuring this with Lori. 
> https://addons.mozilla.org/en-US/firefox/addon/lori-life-of-request-info/

If you were already experiencing 2.5 seconds to load the stats page, then
definitely you have a problem somewhere either in your environment or in
your browser. The load page should be in the order of one millisecond or
less, even for large configs.

Maybe you should take a network capture so that we can check where the
time is lost ?

Regards,
Willy

Re: [ANNOUNCE] haproxy 1.5-dev12

[Guillaume Castagnino]

Nice !

Just set up on my personnal server with 2 wildcard certificates. It 
seems to work like a charm :)

I use this, TLSv1.2 enabled (so using openssl 1.0.1):
bind :::443 ssl crt /etc/ssl/startssl/haproxy/xwing.info.pem crt 
/etc/ssl/startssl/haproxy/ ciphers ECDHE-RSA-AES128-SHA256:AES128-GCM-
SHA256:RC4:HIGH:!MD5:!aNULL:!EDH prefer-server-ciphers


Thanks, great job !

-- 
Guillaume Castagnino
ca...@xwing.info / guilla...@castagnino.org

Re: [ANNOUNCE] haproxy 1.5-dev12

[Willy Tarreau]

Hi Guillaume,

On Mon, Sep 10, 2012 at 03:46:26PM +0200, Guillaume Castagnino wrote:
> Nice !
> 
> Just set up on my personnal server with 2 wildcard certificates. It 
> seems to work like a charm :)
> 
> I use this, TLSv1.2 enabled (so using openssl 1.0.1):
>   bind :::443 ssl crt /etc/ssl/startssl/haproxy/xwing.info.pem crt 
> /etc/ssl/startssl/haproxy/ ciphers ECDHE-RSA-AES128-SHA256:AES128-GCM-
> SHA256:RC4:HIGH:!MD5:!aNULL:!EDH prefer-server-ciphers

Nice, thank you for the feedback !

Willy

Re: [ANNOUNCE] haproxy 1.5-dev12

[Baptiste]

And of course, the article on Exceliance blog:
http://blog.exceliance.fr/2012/09/10/how-to-get-ssl-with-haproxy-getting-rid-of-stunnel-stud-nginx-or-pound/

have fun

Re: [ANNOUNCE] haproxy 1.5-dev12

[Guillaume Castagnino]

Le lundi 10 septembre 2012 15:52:23 Willy Tarreau a écrit :
> Hi Guillaume,
> 
> On Mon, Sep 10, 2012 at 03:46:26PM +0200, Guillaume Castagnino wrote:
> > Nice !
> > 
> > Just set up on my personnal server with 2 wildcard certificates. It
> > seems to work like a charm :)
> > 
> > I use this, TLSv1.2 enabled (so using openssl 1.0.1):
> > bind :::443 ssl crt /etc/ssl/startssl/haproxy/xwing.info.pem crt
> > 
> > /etc/ssl/startssl/haproxy/ ciphers
> > ECDHE-RSA-AES128-SHA256:AES128-GCM-
> > SHA256:RC4:HIGH:!MD5:!aNULL:!EDH prefer-server-ciphers
> 
> Nice, thank you for the feedback !

Just one precision on the cert.pem content, to achieve the best 
compliance: it seems that haproxy is fine when feeding the full 
certificate chain in the .pem file instead of only the the 
certificate/private key pair (as suggest on the first SSL announce from 
last week). This make clients that do certificate chain verification 
happy:

So cert.pem contains:
- Server certificate
- Intermediate CA 1 certificate
- Intermediate CA 2 certificate
...
- Intermediate CA n certificate
- Root CA certificate
- Private key

Of course, the number of intermediate CA may change depending on the 
certificate chain of the SSL provider (usually, there is just one 
intermediate CA).


And this is just working flawlessly, making SSL nazis happy ;).

Thanks !

-- 
Guillaume Castagnino
ca...@xwing.info / guilla...@castagnino.org

Re: [ANNOUNCE] haproxy 1.5-dev12

[Duncan Hall]

On 10/09/12 19:23, Willy Tarreau wrote:

Hi Duncan,

On Mon, Sep 10, 2012 at 07:16:30PM +1000, Duncan Hall wrote:

On 10/09/12 18:10, Willy Tarreau wrote:

Many bugs were fixes, and many were certainly introduced. If you observe
any
bug, please report it, as I'd rather issue -dev13 quickly with many fixes.

Great work, very much appreciated.

I have rolled 1.5-dev12 into a test environment and noticed that the
statistics page is now much slower to load. Previously (on 1.5-dev 11
built 2012/06/04) the time between requesting the stats page and it
displaying was about 2.5 seconds, this has now jumped to 10 seconds. It
is not a big problem for me but I thought it was worth mentioning.

I am measuring this with Lori.
https://addons.mozilla.org/en-US/firefox/addon/lori-life-of-request-info/

If you were already experiencing 2.5 seconds to load the stats page, then
definitely you have a problem somewhere either in your environment or in
your browser. The load page should be in the order of one millisecond or
less, even for large configs.

Maybe you should take a network capture so that we can check where the
time is lost ?

Regards,
Willy


Willy,

Thanks for your help on this, there was a network issue in the VPN I use 
to access my dev environment that was causing the delays.


One thing I did notice on CentOS 5.8 and 6.3 is that at compile time I 
now need to use USE_STATIC_PCRE=1 instead of USE_PCRE=1. If I use 
USE_PCRE=1 it will compile and run but if the conf file references an 
ssl cert it cannot read the key in the pem file.


Regards,

Duncan

Re: [ANNOUNCE] haproxy 1.5-dev12

[Willy Tarreau]

Hi Duncan,

On Fri, Sep 14, 2012 at 01:34:04PM +1000, Duncan Hall wrote:
> Thanks for your help on this, there was a network issue in the VPN I use 
> to access my dev environment that was causing the delays.

OK, thanks for the feedback.

> One thing I did notice on CentOS 5.8 and 6.3 is that at compile time I 
> now need to use USE_STATIC_PCRE=1 instead of USE_PCRE=1. If I use 
> USE_PCRE=1 it will compile and run but if the conf file references an 
> ssl cert it cannot read the key in the pem file.

That's amazing, it should be totally unrelated. I just checked my libssl
and libcrypto here and none of them makes use of any regex call. This
looks like a nasty side effect.

If you could post a minimal config which reliably reproduces the issue,
including one such pem file (a test one, not yours of course) and report
your build options, we could try on different platforms and try to find
a workaround. USE_STATIC_PCRE is not always an option for everyone.

Regards,
Willy

Re: [ANNOUNCE] haproxy 1.5-dev12

[Duncan Hall]

On 14/09/12 15:07, Willy Tarreau wrote:



One thing I did notice on CentOS 5.8 and 6.3 is that at compile time I
now need to use USE_STATIC_PCRE=1 instead of USE_PCRE=1. If I use
USE_PCRE=1 it will compile and run but if the conf file references an
ssl cert it cannot read the key in the pem file.

That's amazing, it should be totally unrelated. I just checked my libssl
and libcrypto here and none of them makes use of any regex call. This
looks like a nasty side effect.

If you could post a minimal config which reliably reproduces the issue,
including one such pem file (a test one, not yours of course) and report
your build options, we could try on different platforms and try to find
a workaround. USE_STATIC_PCRE is not always an option for everyone.

Regards,
Willy


Willy,

I can't reproduce it again, lets put it down to user error (it has been 
a very very long week).


If I come across it again I'll repost to the list.

Regards,

Duncan

Re: [ANNOUNCE] haproxy 1.5-dev12

[Willy Tarreau]

Hi Duncan,

On Fri, Sep 14, 2012 at 04:49:58PM +1000, Duncan Hall wrote:
> I can't reproduce it again, lets put it down to user error (it has been 
> a very very long week).

OK, thanks for checking and reporting.

> If I come across it again I'll repost to the list.

You're welcome.

Cheers,
Willy

Re: [ANNOUNCE] haproxy 1.5-dev13

[Willy Tarreau]

Hi all,

in two days we got several reports of nasty issues among which :
  - randomly failing servers when checked in pure TCP
  - truncated stats page (caused by extra close of failed health check)
  - failure to sync between peers
  - old process staying alive for too long or crashing during reloads
  - high CPU usage during failed health checks
  - randomly truncated chunked-encoded gzipped objects

All these issues have been addressed now and the fixes are already
committed. They will appear in the nighly snapshot 20121124 which
will be generated in about 4 hours.

So all those who are trying dev13, don't waste your time, grab the
snapshot or wait for dev14 which will replace it shortly.

If you think that you're facing an issue which was not mentionned
here or that is still appearing with the latest snapshot, please
report it quickly before we do dev14 !

Thanks,
Willy

Re: [ANNOUNCE] haproxy 1.5-dev15

[Baptiste]

Hi Willy,

Thanks for the bugfixes and the great job on:
MEDIUM: proto_tcp: add support for tracking L7 information

We'll now have more fun fighting DDOS with HAProxy :)

cheers

Segfault executing haproxy-1.5-dev15

[Javi Legido]

Hi there.

Brief history:

1. I compiled haproxy-1.5-dev15

sudo aptitude update; sudo aptitude install build-essential
libpcre3-dev libssl-dev
cd /usr/local; sudo wget -c --tries=0
http://haproxy.1wt.eu/download/1.5/src/devel/haproxy-1.5-dev15.tar.gz
sudo tar xvfz haproxy-1.5-dev15.tar.gz; sudo ln -s haproxy-1.5-dev15 haproxy
make TARGET=linux2628 USE_STATIC_PCRE=1 USE_OPENSSL=1
sudo make install

2. I copied a working config  from another haproxy server (version
1.4.8-1) for testing purposes. If you think the config is relevant
I'll post it

3. I tried to start haproxy:

sudo /usr/local/sbin/haproxy -c -f /etc/haproxy/haproxy.cfg

No output, no process running. I got this in the logs:

==> /var/log/messages <==
Dec 18 11:48:31 balancer-2 kernel: [ 3732.683419] haproxy[8652]:
segfault at 0 ip 00443dfa sp 7fffb1c0 error 6 in
haproxy[40+a8000]

Should I blame the compile process? Should I try a different version?

My target is to test SSL offload.

Thanks.

Javier

RE: [ANNOUNCE] haproxy-1.5-dev16

[seri0...@naver.com]

 
stats http-request configuration is broken!!


 

-Original Message-
From: "Willy Tarreau"<w...@1wt.eu> 
To: <haproxy@formilux.org>; 
Cc: 
Sent: 2012-12-25 (화) 00:51:37
Subject: [ANNOUNCE] haproxy-1.5-dev16

Hi all,

Here comes 1.5-dev16. Thanks to the amazing work Sander Klein and John
Rood have done at Picturae ICT ( http://picturae.com/ ) we could finally
spot the freeze bug after one week of restless digging ! This bug was
amazingly hard to reproduce in general and would only affect POST requests
under certain circumstances that I never could reproduce despite many
efforts. It is likely that other users were affected too but did not
notice it because end users did not complain (I'm thinking about webmail
and file sharing environments for example).

During this week of code review and testing, around 10 other minor to medium
bugs related to the polling changes could be fixed.

Another nasty bug was fixed on SSL. It happens that OpenSSL maintains a
global error stack that must constantly be flushed (surely they never heard
how errno works). The result is that some SSL errors could cause another SSL
session to break as a side effect of this error. This issue was reported by
J. Maurice (wiz technologies) who first encountered it when playing with the
tests on ssllabs.com.

Another bug present since 1.4 concerns the premature close of the response
when the server responds before the end of a POST upload. This happens when
the server responds with a redirect or with a 401, sometimes the client would
not get the response. This has been fixed.

Krzysztof Rutecki reported some issues on client certificate checks, because
the check for the presence of the certificate applies to the connection and
not just to the session. So this does not match upon session resumption. Thus
another ssl_c_used ACL was added to check for such sessions.

Among the other nice additions, it is now possible to log the result of any
sample fetch method using %[]. This allows to log SSL certificates for example.
And similarly, passing such information to HTTP headers was implemented too,
as "http-request add-header" and "http-request set-header", using the same
format as the logs. This also becomes useful for combining headers !

Some people have been asking for logging the amount of uploaded data from the
client to the server, so this is now available as the %U log-format tag.
Some other log-format tags were deprecated and replaced with easier to remind
ones. The old ones still work but emit a warning suggesting the replacement.

And last, the stats HTML version was improved to present detailed information
using hover tips instead of title attributes, allowing multi-line details on
the page. The result is nicer, more readable and more complete.

The changelog is short enough to append it here after the usual links :

Site index : http://haproxy.1wt.eu/
Sources : http://haproxy.1wt.eu/download/1.5/src/devel/
Changelog : http://haproxy.1wt.eu/download/1.5/src/CHANGELOG
Cyril's HTML doc : http://cbonte.github.com/haproxy-dconv/configuration-1.5.html

At the moment, nobody broke the latest snapshots, so I think we're getting
closer to something stable to base future work on.

Thanks!
Willy

--
Changelog from 1.5-dev15 to 1.5-dev16:
- BUG/MEDIUM: ssl: Prevent ssl error from affecting other connections.
- BUG/MINOR: ssl: error is not reported if it occurs simultaneously with peer 
close detection.
- MINOR: ssl: add fetch and acl "ssl_c_used" to check if current SSL session 
uses a client certificate.
- MINOR: contrib: make the iprange tool grep for addresses
- CLEANUP: polling: gcc doesn't always optimize constants away
- OPTIM: poll: optimize fd management functions for low register count CPUs
- CLEANUP: poll: remove a useless double-check on fdtab[fd].owner
- OPTIM: epoll: use a temp variable for intermediary flag computations
- OPTIM: epoll: current fd does not count as a new one
- BUG/MINOR: poll: the I/O handler was called twice for polled I/Os
- MINOR: http: make resp_ver and status ACLs check for the presence of a 
response
- BUG/MEDIUM: stream-interface: fix possible stalls during transfers
- BUG/MINOR: stream_interface: don't return when the fd is already set
- BUG/MEDIUM: connection: always update connection flags prior to computing 
polling
- CLEANUP: buffer: use buffer_empty() instead of buffer_len()==0
- BUG/MAJOR: stream_interface: fix occasional data transfer freezes
- BUG/MEDIUM: stream_interface: fix another case where the reader might not be 
woken up
- BUG/MINOR: http: don't abort client connection on premature responses
- BUILD: no need to clean up when making git-tar
- MINOR: log: add a tag for amount of bytes uploaded from client to server
- BUG/MEDIUM: log: fix possible segfault during config parsing
- MEDIUM: log: change a few log tokens to make them easier to remember
- BUG/MINOR: log: add_to_logformat_list() used the

Re: [ANNOUNCE] haproxy-1.5-dev16

[Dmitry Sivachenko]

Hello!

After update from -dev15, the following stats listener:

listen stats9 :30009
mode http
stats enable
stats uri /
stats show-node
stats show-legends

returns 503/Service unavailable.

With -dev15 it shows statistics page.


On 24.12.2012, at 19:51, Willy Tarreau  wrote:

> Hi all,
> 
> Here comes 1.5-dev16. Thanks to the amazing work Sander Klein and John
> Rood have done at Picturae ICT ( http://picturae.com/ ) we could finally
> spot the freeze bug after one week of restless digging ! This bug was
> amazingly hard to reproduce in general and would only affect POST requests
> under certain circumstances that I never could reproduce despite many
> efforts. It is likely that other users were affected too but did not
> notice it because end users did not complain (I'm thinking about webmail
> and file sharing environments for example).
> 
> During this week of code review and testing, around 10 other minor to medium
> bugs related to the polling changes could be fixed.
> 
> Another nasty bug was fixed on SSL. It happens that OpenSSL maintains a
> global error stack that must constantly be flushed (surely they never heard
> how errno works). The result is that some SSL errors could cause another SSL
> session to break as a side effect of this error. This issue was reported by
> J. Maurice (wiz technologies) who first encountered it when playing with the
> tests on ssllabs.com.
> 
> Another bug present since 1.4 concerns the premature close of the response
> when the server responds before the end of a POST upload. This happens when
> the server responds with a redirect or with a 401, sometimes the client would
> not get the response. This has been fixed.
> 
> Krzysztof Rutecki reported some issues on client certificate checks, because
> the check for the presence of the certificate applies to the connection and
> not just to the session. So this does not match upon session resumption. Thus
> another ssl_c_used ACL was added to check for such sessions.
> 
> Among the other nice additions, it is now possible to log the result of any
> sample fetch method using %[]. This allows to log SSL certificates for 
> example.
> And similarly, passing such information to HTTP headers was implemented too,
> as "http-request add-header" and "http-request set-header", using the same
> format as the logs. This also becomes useful for combining headers !
> 
> Some people have been asking for logging the amount of uploaded data from the
> client to the server, so this is now available as the %U log-format tag.
> Some other log-format tags were deprecated and replaced with easier to remind
> ones. The old ones still work but emit a warning suggesting the replacement.
> 
> And last, the stats HTML version was improved to present detailed information
> using hover tips instead of title attributes, allowing multi-line details on
> the page. The result is nicer, more readable and more complete.
> 
> The changelog is short enough to append it here after the usual links :
> 
>Site index   : http://haproxy.1wt.eu/
>Sources  : http://haproxy.1wt.eu/download/1.5/src/devel/
>Changelog: http://haproxy.1wt.eu/download/1.5/src/CHANGELOG
>Cyril's HTML doc : 
> http://cbonte.github.com/haproxy-dconv/configuration-1.5.html
> 
> At the moment, nobody broke the latest snapshots, so I think we're getting
> closer to something stable to base future work on.
> 
> Thanks!
> Willy
> 
> --
> Changelog from 1.5-dev15 to 1.5-dev16:
>  - BUG/MEDIUM: ssl: Prevent ssl error from affecting other connections.
>  - BUG/MINOR: ssl: error is not reported if it occurs simultaneously with 
> peer close detection.
>  - MINOR: ssl: add fetch and acl "ssl_c_used" to check if current SSL session 
> uses a client certificate.
>  - MINOR: contrib: make the iprange tool grep for addresses
>  - CLEANUP: polling: gcc doesn't always optimize constants away
>  - OPTIM: poll: optimize fd management functions for low register count CPUs
>  - CLEANUP: poll: remove a useless double-check on fdtab[fd].owner
>  - OPTIM: epoll: use a temp variable for intermediary flag computations
>  - OPTIM: epoll: current fd does not count as a new one
>  - BUG/MINOR: poll: the I/O handler was called twice for polled I/Os
>  - MINOR: http: make resp_ver and status ACLs check for the presence of a 
> response
>  - BUG/MEDIUM: stream-interface: fix possible stalls during transfers
>  - BUG/MINOR: stream_interface: don't return when the fd is already set
>  - BUG/MEDIUM: connection: always update connection flags prior to computing 
> polling
>  - CLEANUP: buffer: use buffer_empty() instead of buffer_len()==0
>  - BUG/MAJOR: stream_interface: fix occasional data transfer freezes
>  - BUG/MEDIUM: stream_interface: fix another case where the reader might not 
> be woken up
>  - BUG/MINOR: http: don't abort client connection on premature responses
>  - BUILD: no need to clean up when making git-tar
>  - MINOR: log: add a tag for am

RE: [ANNOUNCE] haproxy-1.5-dev16

[kai]

On Tue, 25 Dec 2012 02:40:39 +0900 (KST), seri0...@naver.com wrote:

stats http-request configuration is broken!!


I confirm this, stats seem broken. this code is working in dev15:

[code]
listen stats 123.45.67.89:12345
mode http
option httpclose
stats enable
stats uri /
stats realm   restricted
stats auth1234:5678
[/code]

but returns "503 Service Unavailable" in dev16


Cheers,

Kai

RE: [ANNOUNCE] haproxy-1.5-dev16

[seri0...@naver.com]

 
I've patched dev16 as belows:

# diff -Nu haproxy-1.5-dev16/src/proto_http.c 
haproxy-1.5-dev16-new/src/proto_http.c 
--- haproxy-1.5-dev16/src/proto_http.c  2012-12-25 00:48:14.0 +0900
+++ haproxy-1.5-dev16-new/src/proto_http.c  2012-12-25 09:26:27.545049268 
+0900
@@ -3117,7 +3117,7 @@
}
}
}
-   return rule;
+   return NULL;
 }
 
 /* This stream analyser runs all HTTP request processing which is common to

 

 

Then, stat page works well!!


 

-Original Message-
From: <k...@rhynn.net> 
To: <haproxy@formilux.org>; 
Cc: 
Sent: 2012-12-25 (화) 04:32:55
Subject: RE: [ANNOUNCE] haproxy-1.5-dev16

On Tue, 25 Dec 2012 02:40:39 +0900 (KST), seri0...@naver.com wrote:
> stats http-request configuration is broken!!

I confirm this, stats seem broken. this code is working in dev15:

[code]
listen stats 123.45.67.89:12345
mode http
option httpclose
stats enable
stats uri /
stats realm restricted
stats auth 1234:5678
[/code]

but returns "503 Service Unavailable" in dev16


Cheers,

Kai

Re: [ANNOUNCE] haproxy-1.5-dev16

[Willy Tarreau]

Hello Dmitry,

On Mon, Dec 24, 2012 at 11:31:47PM +0400, Dmitry Sivachenko wrote:
> Hello!
> 
> After update from -dev15, the following stats listener:
> 
> listen stats9 :30009
> mode http
> stats enable
> stats uri /
> stats show-node
> stats show-legends
> 
> returns 503/Service unavailable.

Oh dear! I'm really angry because this is a beginner's bug and because
I did all the tests for this last commit using the config I've been using
for the stats page, except that I did not request the stats page during
the tests. Pfff...

Here's the fix, it will show up in snapshot 20121226 tomorrow.

Thanks for reporting this, Dmitry.

Willy

>From 418c1a0a9563f90a665c6434f39d2576af48cdc1 Mon Sep 17 00:00:00 2001
From: Willy Tarreau 
Date: Tue, 25 Dec 2012 20:52:58 +0100
Subject: [PATCH] BUG/MEDIUM: stats: fix stats page regression introduced by
 commit 20b0de5

This commit adding http-request add-header/set-header unfortunately introduced
a regression to the handling of the stats page which is not matched anymore.

Thanks to Dmitry Sivachenko for reporting this.
---
 src/proto_http.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/proto_http.c b/src/proto_http.c
index c715828..deed52e 100644
--- a/src/proto_http.c
+++ b/src/proto_http.c
@@ -3113,11 +3113,11 @@ http_check_access_rule(struct proxy *px, struct list 
*rules, struct session *s,
trash.str[trash.len++] = ' ';
trash.len += build_logline(s, trash.str + 
trash.len, trash.size - trash.len, &rule->arg.hdr_add.fmt);
http_header_add_tail2(&txn->req, &txn->hdr_idx, 
trash.str, trash.len);
-   break;
+   return rule;
}
}
}
-   return rule;
+   return NULL;
 }
 
 /* This stream analyser runs all HTTP request processing which is common to
-- 
1.7.12.4.dirty

Re: [ANNOUNCE] haproxy-1.5-dev16

[Willy Tarreau]

On Tue, Dec 25, 2012 at 08:57:28PM +0100, Willy Tarreau wrote:
> Hello Dmitry,
> 
> On Mon, Dec 24, 2012 at 11:31:47PM +0400, Dmitry Sivachenko wrote:
> > Hello!
> > 
> > After update from -dev15, the following stats listener:
> > 
> > listen stats9 :30009
> > mode http
> > stats enable
> > stats uri /
> > stats show-node
> > stats show-legends
> > 
> > returns 503/Service unavailable.
> 
> Oh dear! I'm really angry because this is a beginner's bug and because
> I did all the tests for this last commit using the config I've been using
> for the stats page, except that I did not request the stats page during
> the tests. Pfff...
> 
> Here's the fix, it will show up in snapshot 20121226 tomorrow.

This fix is still wrong, as it only accepts one add-header rule, so
please use the other fix posted in this thread by "seri0528" instead.

Regards,
Willy

Re: [ANNOUNCE] haproxy-1.5-dev16

[Dmitry Sivachenko]

On 26.12.2012, at 1:03, Willy Tarreau  wrote:
>> 
> 
> This fix is still wrong, as it only accepts one add-header rule, so
> please use the other fix posted in this thread by "seri0528" instead.
> 


Thanks a lot! Works now.

Tilde in haproxy 1.5 log

[Jeremy Wilson]

I'm implementing the 1.5 version of haproxy to get SSL termination, and I've 
noticed that the https log entries have a "~" character after the frontend 
name.  I was wondering what the significance of that was and how I might get 
rid of it, because it breaks some of our analytics scripting.

RE: [ANNOUNCE] haproxy-1.5-dev18

[Lukas Tribus]

Thank you, you did an amazing job here again!


> TLS ALPN was implemented similarly as NPN was made. It is supposed
> to replace NPN.

Jesus Christ, this was fast. It was only discussed at IETF 86 mid-march
(and in 2 drafts this year), and 15 days later its already in
haproxy-1.5.

I think HAProxy is the first project to integrate ALPN. In fact afaik,
ALPN is not even in openssl upstream yet (there are just patches).


> PCRE JIT support was contributed by Hiroaki Nakamura, it should
> provide much faster regex processing.

This one is great as well! Looking forward to give it a try.


[1] http://www.ietf.org/mail-archive/web/tls/current/msg09419.html  
  

Re: [ANNOUNCE] haproxy-1.5-dev18

[Willy Tarreau]

On Wed, Apr 03, 2013 at 06:01:31PM +0200, Lukas Tribus wrote:
> Thank you, you did an amazing job here again!

Thanks! You know, people like you who qualify bug reports on the
ML help save a lot of time and shorten the time to produce a fix.

> > TLS ALPN was implemented similarly as NPN was made. It is supposed
> > to replace NPN.
> 
> Jesus Christ, this was fast. It was only discussed at IETF 86 mid-march
> (and in 2 drafts this year), and 15 days later its already in
> haproxy-1.5.
> 
> I think HAProxy is the first project to integrate ALPN. In fact afaik,
> ALPN is not even in openssl upstream yet (there are just patches).

I know there is some demand for this already, and it was easy to merge it so
it was worth trying. I haven't even built it, it's just a copy/paste/rename.
But if something breaks it will be easy to fix. In the worst case it's a good
starting point for experimentation.

> > PCRE JIT support was contributed by Hiroaki Nakamura, it should
> > provide much faster regex processing.
> 
> This one is great as well! Looking forward to give it a try.

We got one failure report, but I don't have a recent pcre to test.
This should still be considered experimental in my opinion.

Regards,
Willy

Re: [ANNOUNCE] haproxy-1.5-dev20

[Baptiste]

Hi Willy,

This is a great news 
Thanks a lot for this wonderful release.

Baptiste


On Mon, Dec 16, 2013 at 3:41 AM, Willy Tarreau  wrote:
> Hi all,
>
> here is probably the largest update we ever had, it's composed of 345
> patches!
>
> Some very difficult changes had to be made and as usual when such changes
> happen, they take a lot of time due to the multiple attempts at getting
> them right, and as time goes, people submit features :-)
>
> After two weeks spent doing only fixes, I thought it was time to issue dev20.
> I'm sure I'll forget a large number of things, but the main features of this
> version include the following points (in merge order) :
>
>   - optimizations (splicing, polling, etc...) : a few percent CPU could be
> saved ;
>
>   - memory : the connections and applets are now allocated only when needed.
> Additionally, some structures were reorganized to avoid fragmentation on
> 64-bit systems. In practice, an idle session size has dropped from 1936
> bytes to 1296 bytes (-640 bytes, or -33%).
>
>   - samples : all sample fetch expressions now support a comma-delimited
> list of converters. This is also true in ACLs, so that it becomes
> possible to do things like :
>
> # convert to lower case and use fast tree indexing
> acl known_domain hdr(host),lower -f huge-domain-list.lst
>
>   - a lot of code has been deduplicated in the tracked counters, it's now
> possible to use sc_foo_bar(1, args) instead of sc1_foo_bar(args). Doing
> so has simplified the code and makes life of APIs easier.
>
>   - it's now possible to look up a tracked key from another table. This allows
> to retrieve multiple counters for the same key.
>
>   - several hash algorithms are provided, and it is possible to select them
> per backend. This high quality work was done at Tumblr by Bhaskar Maddala.
>
>   - agent-checks: this new feature was merged and replaced the lb-agent-chk.
> Some changes are still planned but feedback is welcome. The goal of this
> agent is to retrieve soem weight information from a server independantly
> of the service health. A typical usage would consist in reporting the
> server's idle percentage as an estimate of the possible weight. This work
> was done by Simon Horman for Loadbalancer.org.
>
>   - samples : more automatic conversions between types are supported, making
> it easier to stick to any parameter. The types are much more dynamic now.
> Some improvements are still pending. This work was done by Thierry 
> Fournier
> at Exceliance.
>
>   - map : a new type of converter appeared : maps. A map matches a key from
> a file just like ACLs do, and replaces this value with the value 
> associated
> with the key on the same line of the file. As it is a converter, it can be
> used in any sample expression. The first usage consists in geolocation,
> where networks are associated with country codes. Maps may be consulted,
> deleted, updated and filled from the CLI. Some will probably use this to
> program actions or emulate ACLs without even reloading a config. This
> work was also achieved by Thierry Fournier, and reviewed by Cyril Bonté
> who developped the original Geoip patchset for 1.4 and 1.5.
>
>   - http-request redirect now supports log-format like expressions, just like
> http-request add-header. This allows to emit strings extracted from the
> request (host header, country code from a map, ...). Thierry again here.
>
>   - checks: tcp-check supports send/expect sequences with 
> strings/regex/binary.
> Thus it now becomes possible to check unsupported protocols, even binary.
> This work is from Baptiste Assmann.
>
>   - keep-alive: the dynamic allocation of the connection and applet in the
> session now allows to reuse or kill a connection that was previously
> associated with the session. Thus we now have a very basic support for
> keep-alive to the servers. There is even an option to relax the load
> balancing to try to keep the same connection. Right now we don't do
> any connection sharing so the main use is for static servers and for
> far remote servers or those which require the broken NTLM auth. That
> said, the performance tests I have run show an increase from 71000
> connections per second to 15 keep-alive requests per second running
> on one core of a Xeon E5 3.6 GHz. This doubled to 300k requests per
> second with two cores. I didn't test above, I lacked injection tools :-)
> One good point is that it will help people assemble haproxy and varnish
> together with haproxy doing the consistent hash and varnish caching after
> it.
>
> As most of you know, server-side keep-alive is the condition to release 1.5.
> Now we have it, we'll be able to improve on it but it's basically working.
>
> I expect to release 1.5-final around January and mostly focus on chasing
> bugs till the