Re: req.body_param([])

2018-05-20 Thread Simon Schabel

Dear Jarno,

Thanks for the feedback on this.

I investigated further to find the solution to this issue. It turned out 
that not all PUT requests had a problem with filtering out the 
customerId parameter we're looking for.


Some requests seem to encode the HTTP PUT body somehow or have set some 
HTTP header which prevents haproxy from extracting the parameter.


I'm still trying to find out what exactly is different.

Best
Simon

Am 14.05.2018 um 14:40 schrieb Jarno Huuskonen:

Hi Simon,

On Mon, May 14, Simon Schabel wrote:

HA-Proxy version 1.7.5-2~bpo8+1 2017/05/27

The setting for the logging was done in the /default /section as:

    log-format %Ci:%Cp\ [%t]\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %st\
%B\ %cc\ %cs\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %hr\ %hs\
%[capture.req.hdr(0)]\ %{+Q}r
     option log-separate-errors
     option log-health-checks

and in the /http /and /https /section the body parameter capturing
is activated as:

     # enable HTTP body logging
     option http-buffer-request
     declare capture request len 4
     http-request capture req.body_param(customerId) id 0

As my haproxy version differs from yours I'm unsure where I might
made a configuration error.

I tested with 1.8.8 and 1.7.5 and with both versions I managed to
log customerId (with simple curl -X PUT/POST).

Are the POST/PUT requests large, is it possible that the customerId doesn't
fit in haproxy buffer (default 16k (I think)) ?

Can you test with curl to see if customerId is logged then:
curl -v -X PUT -d'customerId=911' http://yourhost.yourdomain/yourpath

# bigfile is some random file much larger than 16k
and curl -v -X PUT -d@bigfile -d'customerId=912' 
http://yourhost.yourdomain/yourpath

-Jarno



--
SEMKNOX GmbH
Simon Schabel
Webergasse 1, Haus B/1 · 01067 Dresden
simon.scha...@semknox.com · +49-351-32 123 102 · www.semknox.com




Re: req.body_param([])

2018-05-14 Thread Simon Schabel

Dear Jarno,

Thanks for your message.

Ok, my /haproxy -vv/ output is as follows:

HA-Proxy version 1.7.5-2~bpo8+1 2017/05/27
Copyright 2000-2017 Willy Tarreau 

Build options :
  TARGET  = linux2628
  CPU = generic
  CC  = gcc
  CFLAGS  = -g -O2 -fPIE -fstack-protector-strong -Wformat 
-Werror=format-security -D_FORTIFY_SOURCE=2
  OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 
USE_LUA=1 USE_PCRE=1 USE_NS=1


Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity("identity"), 
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")

Built with OpenSSL version : OpenSSL 1.0.2k  26 Jan 2017
Running on OpenSSL version : OpenSSL 1.0.2l  25 May 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.35 2014-04-04
Running on PCRE version : 8.35 2014-04-04
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with Lua version : Lua 5.3.1
Built with transparent proxy support using: IP_TRANSPARENT 
IPV6_TRANSPARENT IP_FREEBIND

Built with network namespace support

Available polling systems :
  epoll : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
    [COMP] compression
    [TRACE] trace
    [SPOE] spoe



The setting for the logging was done in the /default /section as:

   log-format %Ci:%Cp\ [%t]\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %st\ %B\ 
%cc\ %cs\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %hr\ %hs\ 
%[capture.req.hdr(0)]\ %{+Q}r

    option log-separate-errors
    option log-health-checks

and in the /http /and /https /section the body parameter capturing is 
activated as:


    # enable HTTP body logging
    option http-buffer-request
    declare capture request len 4
    http-request capture req.body_param(customerId) id 0

As my haproxy version differs from yours I'm unsure where I might made a 
configuration error.


Thanks for your help,
best
Simon


Am 11.05.2018 um 12:43 schrieb Jarno Huuskonen:

Hi,

On Wed, May 09, Simon Schabel wrote:

We use the req.body_param([]) setting to retrieve body
parameter from the incoming HTTP queries and place them into the
logs.

Unfortunately this only works with HTTP POST requests. In our case
we need to extract the parameter from PUT requests as well.

Would it be an option to use req.body_param([]) on any HTTP
method type instead of restricting it to HTTP POST?

Can you share your haproxy -vv and the logging config ?

I just tested with haproxy-ss-20180507 and this minimal test
seems to get req.body_param(log) to stick table on both POST/PUT requests
(tested w/curl -X PUT|POST):
...
frontend test
 option  http-buffer-request
 bind ipv4@127.0.0.1:8080
 http-request track-sc2 req.body_param(log),lower table test_be if 
METH_POST || METH_PUT

 default_backend test_be

backend test_be
 stick-table type string len 48 size 64 expire 240s store http_req_cnt
 server wp2 some.ip.add.ress:80 id 2
...

curl -X PUT -d'log=user1' http://127.0.0.1:8080/
curl -X POST -d'log=user2' http://127.0.0.1:8080/

-Jarno



--
SEMKNOX GmbH
Simon Schabel
Webergasse 1, Haus B/1 · 01067 Dresden
simon.scha...@semknox.com · +49-351-32 123 102 · www.semknox.com



req.body_param([])

2018-05-09 Thread Simon Schabel

Hello,

We use the req.body_param([]) setting to retrieve body parameter 
from the incoming HTTP queries and place them into the logs.


Unfortunately this only works with HTTP POST requests. In our case we 
need to extract the parameter from PUT requests as well.


Would it be an option to use req.body_param([]) on any HTTP method 
type instead of restricting it to HTTP POST?


Best
Simon

--
SEMKNOX GmbH
Simon Schabel
Webergasse 1, Haus B/1 · 01067 Dresden
simon.scha...@semknox.com · +49-351-32 123 102 · www.semknox.com