Hi Aleks, On Thu, Nov 10, 2016 at 12:52:22AM +0100, Aleksandar Lazic wrote: > > http://www.haproxy.org/download/1.7/doc/SPOE.txt > > I have read the doc. very interesting. > > When I understand this sentence right currently it is only possible to check > some headers right?
> ### > Actually, for now, the SPOE can offload the processing before "tcp-request > content", > "tcp-response content", "http-request" and "http-response" rules. > ### In theory since you pass sample fetch results as arguments, it should be possible to pass anything. For example you can already parse the beginning of a body in http-request if you have enabled the correct option to wait for the body (http-buffer-request or something like this). So in theory you could even pass a part of it even right now. > So a header only WAF is now "easily" possible instead of the full stack with > mod_security. > http://blog.haproxy.com/2012/10/12/scalable-waf-protection-with-haproxy-and-apache-with-modsecurity/ In theory yes. And that's one of the goals. My initial intent regarding this protocol was to be able to delegate some heavy processing outside of the process, to do it for blocking stuff (eg: ldap libs are always at list a little bit blocking), as well as for anything requiring threads. Then I realized that it would solve other problems. For example we have 3 device detection engines, none of them is ever built in by default because they have external dependencies, so users who want to use them have to rebuild haproxy and will not be able to use their distro packages anymore. Such components could possibly be moved to external agents. Another point is WAF. People have a love and hate relation with their WAF, whatever it is. When you deploy your first WAF, you start by loving it because you see in the logs that it blocks a lot of stuff. Then your customers complain about breakage and you have to tune it and find a tradeoff between protection and compatibility. And one day they get hacked and they declare this WAF useless and you want to change everything. Having the WAF built into haproxy would mean that users would have to switch to another LB just to use a different WAF! With SPOP we can imagine having various WAF implementations in external processes that users can chose from. A last motive is stability. At haptech we have implemented support for loadable modules (and you know how much I don't want to see this in our version here). Developing these modules require extreme care and lots of skills regarding haproxy's internals. We currently have a few such modules, providing nice improvements but whose usage will be debatable depending on users. Thus supporting modules is interesting because not everyone is forced to load some code they don't necessarily need or want, and it saves us from having to maintain patches. However we have to enforce a very tight check on the internal API to ensure a module is not loaded on a different version, which means that users have to update their modules at the same time they update the haproxy executable. But despite this there's always the risk that a bug in some experimental code we put there corrupts the whole process and does nasty stuff (random crashes, corrupted responses, never-ending connections, etc). With an external process it's much easier for anyone to develop experimental code without taking any risk for the main process. And if something crashes, you know on which side it crashes thus you can guess why. And typically a WAF is not something I would like to see implemented as a module, I would fear support escalations for crashed processes! So you see, there are plenty of good reasons for being able to move some content processing outside of haproxy, and these reasons have driven the protocol design. The first implementation focuses on having something usable even if not optimal first (eg: we didn't implement pipelining of requests yet but given there are connection pools it's not a big deal). > Some attacks are also in the post body, I assume this will come in the > future after some good tests. Yes that's planned. You should already be able to pass a full buffer of data using req.body (not tested). This is even why the protocol supports fragmented payload. It's more complicated to implement though. We could even imagine doing some compression outside (eg: sdch, snappy, or whatever). In 1.7, the compression was moved to filters so it's pretty possible to move it to an external process as well. We'll be very interested in getting feedback such as "I tried to implement this and failed". The protocol currently looks nice and evolutive. But I know by experience that you can plan everything and the first feature someone requests cannot be fulfilled and will require a protocol update :-) > > Finally some minor performance improvements were brought to the HTTP > > parser > > for large requests or responses (eg: long URLs, huge cookies). I've > > observed > > up to 10% increase in request rate with 1kB cookies and 100-char URIs. > > For me very impressive, wow respect. Well for me it's not impressive, these were low hanging fruits that I identified a long time ago in the parser and which I thought "if I don't do these micro-optimizations now I'll never do them". (...) > As I interpret this right the HTTP/2 will be on the roadmap of 1.8 or 2.0? Yes definitely. But I know too well that a community-driven project cannot have a roadmap, just hints for contributors. Also I am a bottleneck because while I review and integrate code I cannot develop. I thought I would have H2 in 1.6 if you remember :-) So let's say that we'll put a lot of efforts into getting H2 in 1.8. I also count a lot on SPOP to help move some contribs outside of the core and to release some of my time :-) > Some of our customers want to use http2_push. > I think this requires that also the HTTP/2 client (Backend) need to be > implemented right? In theory yes. I don't think we'll have the backend side in 1.8 though but we'll see. However there is currently a discussion on the HTTP WG to use 1xx status codes to let an HTTP/1 server pass hints to a client to retrieve extra objects. These may be usable as well by gateways to produce H2 push responses to H2 clients. So maybe by then we'd have support for something like this, I don't know either. Time will tell. Cheers, Willy
