Re: [PATCH] BUILD: ssl: include buffer common headers for ssl_sock_ctx

[William Lallemand]

On Sun, May 17, 2020 at 01:41:53PM +0200, William Dauchy wrote:
> since commit c0cdaffaa338 ("REORG: ssl: move ssl_sock_ctx and fix
> cross-dependencies issues"), `struct ssl_sock_ctx` was moved in
> ssl_sock.h. As it contains a `struct buffer`, including
> `common/buffer.h` is now mandatory. I encountered an issue while
> including ssl_sock.h on another patch:
> 
> include/types/ssl_sock.h:240:16: error: field ‘early_buf’ has incomplete type
>   240 |  struct buffer early_buf;  /* buffer to store the early data 
> received */
> 
> no backport needed.
> 
> Fixes: c0cdaffaa338 ("REORG: ssl: move ssl_sock_ctx and fix
> cross-dependencies issues")
> Signed-off-by: William Dauchy 

Thanks, merged!

-- 
William Lallemand

Re: decode key created with url32+src

[Aleksandar Lazic]

Tim.

Thank you for your prompt answer.

Regards

Aleks

On 18.05.20 01:30, Tim Düsterhus wrote:
> Aleks,
>
> Am 18.05.20 um 00:48 schrieb Aleksandar Lazic:
>> Is there a easy way to know which URL+src the key is?
>> [...]
>>   http-request track-sc1 url32+src table per_ip_and_url_rates unless { 
>> path_end .css .js .png .gif }
> No, as per the documentation:
>
>> url32+src : binary
>> This returns the concatenation of the "url32" fetch and the "src" fetch. The
>> resulting type is of type binary, with a size of 8 or 20 bytes depending on
>> the source address family. This can be used to track per-IP, per-URL 
>> counters.
> and
>
>> url32 : integer
>> This returns a 32-bit hash of the value obtained by concatenating the first
>> Host header and the whole URL including parameters (not only the path part of
>> the request, as in the "base32" fetch above). This is useful to track per-URL
>> activity. A shorter hash is stored, saving a lot of memory. The output type
>> is an unsigned integer.
> Thus you only have a hash value of the URL in question. However the IP
> address is stored in clear at the end of the resulting key. You might
> need to hex decode it.
>
> Best regards
> Tim Düsterhus
>

Re: decode key created with url32+src

[Tim Düsterhus]

Aleks,

Am 18.05.20 um 00:48 schrieb Aleksandar Lazic:
> Is there a easy way to know which URL+src the key is?
> [...]
>   http-request track-sc1 url32+src table per_ip_and_url_rates unless { 
> path_end .css .js .png .gif }

No, as per the documentation:

> url32+src : binary
> This returns the concatenation of the "url32" fetch and the "src" fetch. The
> resulting type is of type binary, with a size of 8 or 20 bytes depending on
> the source address family. This can be used to track per-IP, per-URL counters.

and

> url32 : integer
> This returns a 32-bit hash of the value obtained by concatenating the first
> Host header and the whole URL including parameters (not only the path part of
> the request, as in the "base32" fetch above). This is useful to track per-URL
> activity. A shorter hash is stored, saving a lot of memory. The output type
> is an unsigned integer.

Thus you only have a hash value of the URL in question. However the IP
address is stored in clear at the end of the resulting key. You might
need to hex decode it.

Best regards
Tim Düsterhus

decode key created with url32+src

[Aleksandar Lazic]

Hi.

I have this lines in the Table per_ip_and_url_rates.
Is there a easy way to know which URL+src the key is?

# table: per_ip_and_url_rates, type: binary, size:1048576, used:56781
0x559813fc9200: key=xxx use=0 exp=85821390 http_req_rate(8640)=27
0x7fef40373630: key= use=0 exp=86380499 http_req_rate(8640)=4494

I used this blog post as base for the table.

https://www.haproxy.com/blog/bot-protection-with-haproxy/

That's the backend definition with HA-Proxy version 2.1.4-1ppa1~bionic

```
frontend https-in

  bind :::443 v4v6 alpn h2,http/1.1 ssl ca-file {{ ansible_nodename 
}}/fullchain.pem crt /etc/ssl/haproxy/

  tcp-request inspect-delay 5s
  tcp-request content accept if { req_ssl_hello_type 1 }

  # DNS labels are case insensitive (RFC 4343), we need to convert the hostname 
into lowercase
  # before matching, or any requests containing uppercase characters will never 
match.
  # http-request set-header Host %[req.hdr(Host),lower]

  # 
https://www.haproxy.com/blog/application-layer-ddos-attack-protection-with-haproxy/
  http-request track-sc0 src table per_ip_rates
 
  # https://www.haproxy.com/blog/bot-protection-with-haproxy/
  # track client's source IP + URL accessed in
  # per_ip_and_url_rates stick table
  http-request track-sc1 url32+src table per_ip_and_url_rates unless { path_end 
.css .js .png .gif }
 
  # Set the threshold to 15 within the time period
  acl exceeds_limit sc_gpc0_rate(0) gt 20

  # Increase the new-page count if this is the first time
  # they've accessed this page, unless they've already
  # exceeded the limit
  #http-request sc-inc-gpc0(0) if { sc_http_req_rate(1) eq 1 } !exceeds_limit

  # Deny requests if over the limit
  #http-request deny deny_status 429 if exceeds_limit

  # 10 requests per second
  #http-request deny deny_status 429 if { sc_http_req_rate(0) gt 200 }

  # Strip off Proxy headers to prevent HTTpoxy (https://httpoxy.org/)
  http-request del-header Proxy


  declare capture request len 128
  declare capture request len 148
  declare capture request len 148

  http-request capture req.hdr(host) len 148

  # Add CORS response header
  acl is_cors_preflight method OPTIONS
  http-response add-header Access-Control-Allow-Origin "*" if is_cors_preflight
  http-response add-header Access-Control-Allow-Methods "GET,POST" if 
is_cors_preflight
  http-response add-header Access-Control-Allow-Credentials "true" if 
is_cors_preflight
  http-response add-header Access-Control-Max-Age "600" if is_cors_preflight

  use_backend be_nginx if { path_beg /.well-known/acme-challenge/ }
  use_backend 
%[req.hdr(host),lower,map(/etc/haproxy/haproxy_backend.map,be_default)]
```

Thanks for help.

Cheers

Aleks

Re: [External] Re: [RFC PATCH] Improved moving averages

[Marcin Deranek]

Great, thanks WIlly. Enjoy! :-)

Marcin Deranek

On Sat, May 16, 2020 at 10:40 PM Willy Tarreau  wrote:

> Hi Marcin,
>
> On Sat, May 16, 2020 at 04:31:30PM +0200, Marcin Deranek wrote:
> > Actually I split this into 3 (see attachments):
> > 1. Adds improved swrate_rate_dynamic function
> > 2. Adds proper req_tot metric for a server (as we have for
> frontend/backend)
> > 3. Updates the code to use the new function for existing *time metrics.
> > They should behave the same except they should be much more accurate at
> the
> > very beginning. This one depends on 1 & 2.
> > Regards,
>
> Nothing else to say but "looks perfect, now merged" :-)
>
> Thanks!
> Willy
>


-- 
Marcin Deranek
Senior System Administrator

Booking.com B.V.
Singel 540 Amsterdam 1017AZ Netherlands
Direct +31207243362
[image: Booking.com] 
Making it easier for everyone to experience the world since 1996
43 languages, 214+ offices worldwide, 141,000+ global destinations, 29
million reported listings
Subsidiary of Booking Holdings Inc. (NASDAQ: BKNG)

Re: [PATCH] enable arm64 builds in travis-ci

[Martin Grigorov]

Hi Willy,

On Fri, May 15, 2020 at 6:07 PM Willy Tarreau  wrote:

> Ilya,
>
> > also, I'd suggest to purge travis-ci cache (if you are build in your own
> > fork).
> > some travis related issue might be related when something is took from
> > cache (which was not supposed to happen)
>
> Could you please handle Martin's patch, possibly cut it into several
> pieces if relevant and add a commit message indicating what it does
> (and why) ? Martin is not at ease with Git (which is not a problem),
> and it seems only him and you understand how the reasons of the changes
> in his patch. At least it's totally unclear to me why there's a new
> install target for arm64 and why there's a special "make" invocation
> there.
>

Let me explain the change.
At
https://github.com/haproxy/haproxy/blob/a8dbdf3c4b463a3f3e018f0cd02fa0d8d179bc07/.travis.yml#L113-L117
you
may see the default 'install' phase.
At
https://github.com/haproxy/haproxy/blob/a8dbdf3c4b463a3f3e018f0cd02fa0d8d179bc07/.travis.yml#L12-L19
is
the default environment.
They are used by every job from the matrix (
https://github.com/haproxy/haproxy/blob/a8dbdf3c4b463a3f3e018f0cd02fa0d8d179bc07/.travis.yml#L35
).
But each job can override the default environment and any of the phases
(before_install, install, after_install, script).
For the ARM64 build I overwrote the 'install' phase by copying the default
one and removing the execution of the build_ssl() function (the one that
builds OpenSSL from source) and I also overwrote the environment to update
the values of SSL_INC and SSL_LIB variables.
'openssl' and 'libssl-dev' packages are already installed in the Ubuntu
image used by TravisCI so there is nothing to install manually.. I've added
a comment (
https://github.com/haproxy/haproxy/blob/a8dbdf3c4b463a3f3e018f0cd02fa0d8d179bc07/.travis.yml#L47)
to remind us how it works.


> Feel free to add your "purge cache" change as an extra patch if needed.
> But in any case, please make sure it's still possible to follow the
> impact of each change, because we've touched many things blindly for
> a while on this arm64 issue and most of the changes were basically
> "let's see if this helps", which is a real mess :-/
>
> Thanks!
> Willy
>

Re: [PATCH] enable arm64 builds in travis-ci

[Martin Grigorov]

Thank you for applying the patch!

The ARM64 build passed at
https://travis-ci.com/github/haproxy/haproxy/jobs/335338296 !
But it passes only for the builds which have 'haproxy-mirror' :
https://travis-ci.com/github/haproxy/haproxy/builds
I am not sure what exactly "haproxy-mirror" is in the TravisCI config.
The builds which do not have "haproxy-mirror" next to the branch name also
do not have ARM64 build at all, e.g.
https://travis-ci.com/github/haproxy/haproxy/builds/166573019.

One thing that I notice is that the successful link above has *jobs *and
the failing one: *builds *before the job/build id.

On Fri, May 15, 2020 at 9:53 PM Willy Tarreau  wrote:

> On Fri, May 15, 2020 at 11:44:48PM +0500,  ??? wrote:
> > commit message adjusted
>
> Many thanks for this, Ilya, now pushed!
>
> Willy
>

[PATCH] BUILD: ssl: include buffer common headers for ssl_sock_ctx

[William Dauchy]

since commit c0cdaffaa338 ("REORG: ssl: move ssl_sock_ctx and fix
cross-dependencies issues"), `struct ssl_sock_ctx` was moved in
ssl_sock.h. As it contains a `struct buffer`, including
`common/buffer.h` is now mandatory. I encountered an issue while
including ssl_sock.h on another patch:

include/types/ssl_sock.h:240:16: error: field ‘early_buf’ has incomplete type
  240 |  struct buffer early_buf;  /* buffer to store the early data 
received */

no backport needed.

Fixes: c0cdaffaa338 ("REORG: ssl: move ssl_sock_ctx and fix
cross-dependencies issues")
Signed-off-by: William Dauchy 
---
 include/types/ssl_sock.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/include/types/ssl_sock.h b/include/types/ssl_sock.h
index 99c964d6e..78639ac10 100644
--- a/include/types/ssl_sock.h
+++ b/include/types/ssl_sock.h
@@ -31,6 +31,7 @@
 #include 
 #include 
 
+#include 
 #include 
 #include 
 #include 
-- 
2.26.2