Re: Content inspection using tcp-request/tcp-response content send-spoa-group

[Stanislav Pavlíček]

út 24. 11. 2020 v 14:29 odesílatel Stanislav Pavlíček <
stanislav.pavli...@gmail.com> napsal:

> tcp rulesets are only evaluated once. But, you may wait to eval a rule
> using an
>
>> ACL. For instance "if { req.len gt 0 }".
>>
>
I tried to follow your example and with acl and req.body as argument I can
at least get the body.

But I cannot get it working on the tcp-response side. No matter what I try,
my
tcp-response content send-spoe-group contentdebug contentdebug-res-payload
if { res.len gt 0 }
rule never sends anything to SPOA.

Can you please spot any mistakes in my configuration regarding tcp-response
content?

I thought that:

backend api
  http-request set-log-level silent

  tcp-response inspect-delay 60s
  tcp-response content send-spoe-group contentdebug
contentdebug-res-payload if { res.len gt 0 }

should suffice.

SP

Re: Content inspection using tcp-request/tcp-response content send-spoa-group

[Stanislav Pavlíček]

út 24. 11. 2020 v 13:57 odesílatel Christopher Faulet 
napsal:

> > The issue is that although I declared tcp-request/tcp-reponse content
> > send-spoa-group rules, my SPOA agent is called only once with request
> length 0
> > and no payload.
> >
>
> tcp rulesets are only evaluated once. But, you may wait to eval a rule
> using an
> ACL. For instance "if { req.len gt 0 }".
>

>From my experiments I started growing the suspicion that it is the case,
thanks for clarification.


> > My goal is to send every chunk of data read/written on given proxy to
> SPOA
> > agent. Ideally I would like to avoid any buffering, which I thought I
> could
> > achieve using
> >
> https://www.arpalert.org/src/haproxy-lua-api/2.2/index.html#Channel.forward
> > <
> https://www.arpalert.org/src/haproxy-lua-api/2.2/index.html#Channel.forward>
>
> > (not used in my example).
> >
> > Is it feasible? Or do I need to implement my own filter?
>
> With the current SPOE design, it is not possible. But the filters API is
> able to
> do that. Thus with a SPOE refactoring is could be possible too. For now,
> the
> only way to achieve that is to write your own filter. The trace filter is
> a good
> example.
>
> But before, you must eval the requests and responses size you expect.
> Because,
> if smaller than a buffer, including the headers, it is already possible.
>

Unfortunately the limits for request/response sizes capture will be at
least hundreds of kB or more.
So it seems that it's time to dig into the depths of HAProxy filters API :)

Btw are you aware whether HAPEE offers such functionality what I am looking
for (full traffic/content inspection/mirroring)?

SP

Re: Content inspection using tcp-request/tcp-response content send-spoa-group

[Christopher Faulet]

Le 24/11/2020 à 11:48, Stanislav Pavlíček a écrit :

Hello,

I'm trying to implement content inspection using haproxy/SPOE and SPOA agent.

I created basic sample configuration to demonstrate my issue:

https://github.com/haproxy/haproxy/issues/956#issuecomment-732806414 



To reproduce locally, just download contentdebug.zip archive from link above, 
run it using docker-compose up and hit it with curl (e.g. curl -d '{}' 
http://localhost ).


The issue is that although I declared tcp-request/tcp-reponse content 
send-spoa-group rules, my SPOA agent is called only once with request length 0 
and no payload.




tcp rulesets are only evaluated once. But, you may wait to eval a rule using an 
ACL. For instance "if { req.len gt 0 }".


I suspect I don't fully understand processing of tcp-request/tcp-response rules, 
acls and accept/reject criteria. I tried to add various acls mainly based on 
req.len/res.len, which I thought could be used to detect end of payload (The 
documentation says that req.len/res.len returns false when no more data is 
available), but still no luck.


req.len/res.len return the current number of bytes in the buffer. That may be 0 
if nothing is received yet. they only return false if you wait for more data but 
haproxy knows no more are expected. But len/payload/payload_lv must not be used 
on an HTTP message because data are structured and not raw. The documentation 
must warn against this use and a warning should probably be added in the code.




My goal is to send every chunk of data read/written on given proxy to SPOA 
agent. Ideally I would like to avoid any buffering, which I thought I could 
achieve using 
https://www.arpalert.org/src/haproxy-lua-api/2.2/index.html#Channel.forward 
 
(not used in my example).


Is it feasible? Or do I need to implement my own filter?


With the current SPOE design, it is not possible. But the filters API is able to 
do that. Thus with a SPOE refactoring is could be possible too. For now, the 
only way to achieve that is to write your own filter. The trace filter is a good 
example.


But before, you must eval the requests and responses size you expect. Because, 
if smaller than a buffer, including the headers, it is already possible.


--
Christopher Faulet

Re: Content inspection using tcp-request/tcp-response content send-spoa-group

[Stanislav Pavlíček]

út 24. 11. 2020 v 12:28 odesílatel Aleksandar Lazic 
napsal:

> I have downloaded the zip and see that you use the "contrib/spoa_server"
> which have some issues which have Christopher Faulet explained in this post
> https://www.mail-archive.com/haproxy@formilux.org/msg38484.html
>
> As far as I know there is no other scriptable spoa solution for now.
> You can try to fix the issues for spoa_server or build your solution based
> on
> contrib/spoa_example for example.
>

You are right and I am aware of this issue, but I am not using the
spoa_server in debug mode and for non-production workloads I am able to
match my load requirements with sufficient number of spoa_server threads.

Regards,
Stanislav Pavlicek

Re: Content inspection using tcp-request/tcp-response content send-spoa-group

[Aleksandar Lazic]

Hi.

On 24.11.20 11:48, Stanislav Pavlíček wrote:

Hello,

I'm trying to implement content inspection using haproxy/SPOE and SPOA agent.

I created basic sample configuration to demonstrate my issue:

https://github.com/haproxy/haproxy/issues/956#issuecomment-732806414 

To reproduce locally, just download contentdebug.zip archive from link above, 
run it using docker-compose up and hit it with curl (e.g. curl -d '{}' http://localhost ).


The issue is that although I declared tcp-request/tcp-reponse content 
send-spoa-group rules, my SPOA agent is called only once with request length 0 and no payload.


I have downloaded the zip and see that you use the "contrib/spoa_server"
which have some issues which have Christopher Faulet explained in this post
https://www.mail-archive.com/haproxy@formilux.org/msg38484.html

I suspect I don't fully understand processing of tcp-request/tcp-response 
rules, acls and accept/reject criteria. I tried to add various acls mainly 
based on req.len/res.len, which I thought could be used to detect end of payload 
(The documentation says that req.len/res.len returns false when no more data is 
available), but still no luck.


My goal is to send every chunk of data read/written on given proxy to SPOA agent. 
Ideally I would like to avoid any buffering, which I thought I could achieve using 
https://www.arpalert.org/src/haproxy-lua-api/2.2/index.html#Channel.forward  (not used in my example).


Is it feasible? Or do I need to implement my own filter?


As far as I know there is no other scriptable spoa solution for now.
You can try to fix the issues for spoa_server or build your solution based on
contrib/spoa_example for example.

contrib/modsecurity looks like that is based on the spoa_example ;-)



This is really important for the project I am working on.

Thanks for any help.

Regards,
Stanislav Pavlicek


Regards
Aleks