Hi Willy.
Thank you as always for the detailed answer.
Am 10-11-2016 06:51, schrieb Willy Tarreau:
Hi Aleks,
On Thu, Nov 10, 2016 at 12:52:22AM +0100, Aleksandar Lazic wrote:
> http://www.haproxy.org/download/1.7/doc/SPOE.txt
I have read the doc. very interesting.
When I understand this sentence right currently it is only possible to
check
some headers right?
###
Actually, for now, the SPOE can offload the processing before
"tcp-request
content",
"tcp-response content", "http-request" and "http-response" rules.
###
In theory since you pass sample fetch results as arguments, it should
be possible to pass anything. For example you can already parse the
beginning of a body in http-request if you have enabled the correct
option to wait for the body (http-buffer-request or something like
this). So in theory you could even pass a part of it even right now.
Interesting Idea.
So a header only WAF is now "easily" possible instead of the full
stack with
mod_security.
http://blog.haproxy.com/2012/10/12/scalable-waf-protection-with-haproxy-and-apache-with-modsecurity/
In theory yes. And that's one of the goals. My initial intent regarding
this protocol was to be able to delegate some heavy processing outside
of the process, to do it for blocking stuff (eg: ldap libs are always
at list a little bit blocking), as well as for anything requiring
threads.
Then I realized that it would solve other problems. For example we have
3 device detection engines, none of them is ever built in by default
because they have external dependencies, so users who want to use them
have to rebuild haproxy and will not be able to use their distro
packages
anymore. Such components could possibly be moved to external agents.
Another point is WAF. People have a love and hate relation with their
WAF, whatever it is. When you deploy your first WAF, you start by
loving
it because you see in the logs that it blocks a lot of stuff. Then your
customers complain about breakage and you have to tune it and find a
tradeoff between protection and compatibility. And one day they get
hacked and they declare this WAF useless and you want to change
everything. Having the WAF built into haproxy would mean that users
would have to switch to another LB just to use a different WAF! With
SPOP we can imagine having various WAF implementations in external
processes that users can chose from.
Well, nothing to add!
A last motive is stability. At haptech we have implemented support for
loadable modules (and you know how much I don't want to see this in our
version here). Developing these modules require extreme care and lots
of skills regarding haproxy's internals. We currently have a few such
modules, providing nice improvements but whose usage will be debatable
depending on users. Thus supporting modules is interesting because not
everyone is forced to load some code they don't necessarily need or
want, and it saves us from having to maintain patches. However we have
to enforce a very tight check on the internal API to ensure a module is
not loaded on a different version, which means that users have to
update
their modules at the same time they update the haproxy executable. But
despite this there's always the risk that a bug in some experimental
code we put there corrupts the whole process and does nasty stuff
(random crashes, corrupted responses, never-ending connections, etc).
With an external process it's much easier for anyone to develop
experimental code without taking any risk for the main process. And if
something crashes, you know on which side it crashes thus you can guess
why. And typically a WAF is not something I would like to see
implemented
as a module, I would fear support escalations for crashed processes!
So you see, there are plenty of good reasons for being able to move
some
content processing outside of haproxy, and these reasons have driven
the
protocol design. The first implementation focuses on having something
usable even if not optimal first (eg: we didn't implement pipelining of
requests yet but given there are connection pools it's not a big deal).
Some attacks are also in the post body, I assume this will come in the
future after some good tests.
Yes that's planned. You should already be able to pass a full buffer of
data using req.body (not tested). This is even why the protocol
supports
fragmented payload. It's more complicated to implement though. We could
even imagine doing some compression outside (eg: sdch, snappy, or
whatever). In 1.7, the compression was moved to filters so it's pretty
possible to move it to an external process as well.
We'll be very interested in getting feedback such as "I tried to
implement
this and failed". The protocol currently looks nice and evolutive. But
I
know by experience that you can plan everything and the first feature
someone requests cannot be fulfilled and will require a protocol update
:-)
I will start to create a Dockerfile for 1.7 and share the experience ;-)
I will try to replace the haproxy 1.5 from openshift with the 1.7.
This will be a challenge but I curious how it performs ;-)
1.7 have so much interesting features like http-reuse, resolver,
agent-check, ...
which helps a PaaS to have dynamic service configuration.
There are some new products in his area like https://traefik.io/ on
which I keep
an eye.
The dynamic configuration part over etcd, consul or any other Key-Value
store is
one of the benefits of this new SW components.
I hope I find the time to use haproxy as a kubernetes ingress controller
https://github.com/kubernetes/contrib/tree/master/ingress/controllers
but this is a future project.
As I interpret this right the HTTP/2 will be on the roadmap of 1.8 or
2.0?
Yes definitely. But I know too well that a community-driven project
cannot
have a roadmap, just hints for contributors. Also I am a bottleneck
because
while I review and integrate code I cannot develop. I thought I would
have
H2 in 1.6 if you remember :-) So let's say that we'll put a lot of
efforts
into getting H2 in 1.8. I also count a lot on SPOP to help move some
contribs outside of the core and to release some of my time :-)
@H2: I think it's the same as we have before with the ssl stuff.
Everyone want's it but it takes time to implement and finally it's here.
Some of our customers want to use http2_push.
I think this requires that also the HTTP/2 client (Backend) need to be
implemented right?
In theory yes. I don't think we'll have the backend side in 1.8 though
but we'll see. However there is currently a discussion on the HTTP WG
to use 1xx status codes to let an HTTP/1 server pass hints to a client
to retrieve extra objects. These may be usable as well by gateways to
produce H2 push responses to H2 clients. So maybe by then we'd have
support for something like this, I don't know either. Time will tell.
?!
This sounds weird to me.
But time will show how it will work
BR Aleks
Cheers,
Willy
