Re: [PATCH 0/2] CI Updates
How it works? I mean if matrix is built on the fly, schedule is inside matrix, right? How github can have prior knowledge when to run scheduled jobs then On Fri, Nov 20, 2020, 4:16 AM Tim Duesterhus wrote: > Willy, > Ilya, > > find two patches improving the CI. > > 1) This lays the foundation to check the workflow trigger in the matrix >generator Python script, allowing us to move less important jobs into >a daily / weekly schedule. > > 2) This cleans up the Windows CI. I've made sure to re-use as much as > possible >from the VTest workflow, like I did for the compliance one. > >You can find the example output here: > > https://github.com/TimWolla/haproxy/runs/1427467730?check_suite_focus=true > > Best regards > > Tim Düsterhus (2): > CI: Pass the github.event_name to matrix.py > CI: Clean up Windows CI > > .github/matrix.py| 9 + > .github/workflows/vtest.yml | 2 +- > .github/workflows/windows-latest.yml | 20 -- > .github/workflows/windows.yml| 60 > 4 files changed, 70 insertions(+), 21 deletions(-) > delete mode 100644 .github/workflows/windows-latest.yml > create mode 100644 .github/workflows/windows.yml > > -- > 2.29.0 > >
[PATCH 0/2] CI Updates
Willy, Ilya, find two patches improving the CI. 1) This lays the foundation to check the workflow trigger in the matrix generator Python script, allowing us to move less important jobs into a daily / weekly schedule. 2) This cleans up the Windows CI. I've made sure to re-use as much as possible from the VTest workflow, like I did for the compliance one. You can find the example output here: https://github.com/TimWolla/haproxy/runs/1427467730?check_suite_focus=true Best regards Tim Düsterhus (2): CI: Pass the github.event_name to matrix.py CI: Clean up Windows CI .github/matrix.py| 9 + .github/workflows/vtest.yml | 2 +- .github/workflows/windows-latest.yml | 20 -- .github/workflows/windows.yml| 60 4 files changed, 70 insertions(+), 21 deletions(-) delete mode 100644 .github/workflows/windows-latest.yml create mode 100644 .github/workflows/windows.yml -- 2.29.0
[PATCH 1/2] CI: Pass the github.event_name to matrix.py
This is a preparation to later run some matrix entries on schedule only.
Within the matrix.py script it can now be detected whether the workflow is
running on schedule by using:
if build_type == "schedule":
matrix.append(...)
---
.github/matrix.py | 9 +
.github/workflows/vtest.yml | 2 +-
2 files changed, 10 insertions(+), 1 deletion(-)
diff --git a/.github/matrix.py b/.github/matrix.py
index 6e5b9247e..ab27ee1d9 100644
--- a/.github/matrix.py
+++ b/.github/matrix.py
@@ -7,6 +7,15 @@
# 2 of the License, or (at your option) any later version.
import json
+import sys
+
+if len(sys.argv) == 2:
+build_type = sys.argv[1]
+else:
+print("Usage: {} ".format(sys.argv[0]), file=sys.stderr)
+sys.exit(1)
+
+print("Generating matrix for type '{}'.".format(build_type))
def clean_os(os):
diff --git a/.github/workflows/vtest.yml b/.github/workflows/vtest.yml
index 28e814153..442c2d082 100644
--- a/.github/workflows/vtest.yml
+++ b/.github/workflows/vtest.yml
@@ -23,7 +23,7 @@ jobs:
- uses: actions/checkout@v2
- name: Generate Build Matrix
id: set-matrix
-run: python3 .github/matrix.py
+run: python3 .github/matrix.py "${{ github.event_name }}"
# The Test job actually runs the tests.
Test:
--
2.29.0
[PATCH 2/2] CI: Clean up Windows CI
This patch cleans up the Windows CI to look more similar to the refactored
Linux CI on GitHub Actions.
It switches the environment set-up from some manual cygwin setup via choco to
the msys2/setup-msys2@v2 action which just works and allows later steps to look
like any others without need to manually specify the shell.
This new setup is much faster than before where a single Windows build required
more than 10 minutes with more than 5 minutes just spent setting up the
environment and more than 6 minutes compiling HAProxy.
With this patch the setting of of the environment is done in less than a minute
and HAProxy is compiled in less than 2 minutes.
The only drawback is that Lua does not appear to be readily available. I expect
this to be acceptable and that the benefits far outweight this small drawback.
---
.github/workflows/windows-latest.yml | 20 --
.github/workflows/windows.yml| 60
2 files changed, 60 insertions(+), 20 deletions(-)
delete mode 100644 .github/workflows/windows-latest.yml
create mode 100644 .github/workflows/windows.yml
diff --git a/.github/workflows/windows-latest.yml
b/.github/workflows/windows-latest.yml
deleted file mode 100644
index 445e021a4..0
--- a/.github/workflows/windows-latest.yml
+++ /dev/null
@@ -1,20 +0,0 @@
-# build status appears on https://github.com/haproxy/haproxy/actions
-
-name: windows-latest
-
-on: [push]
-
-jobs:
- cygwin:
-
-runs-on: windows-latest
-
-steps:
-- uses: actions/checkout@v1
-- name: install prerequisites
- run: choco install bash make libssl-devel cygwin-devel gcc-core libgcc1
binutils lua-devel libpcre-devel zlib-devel --source cygwin
-- name: fake step
- run: C:\\tools\\cygwin\\bin\\bash -lc 'pwd'
-- name: build
- run: C:\\tools\\cygwin\\bin\\bash -lc 'cd $OLDPWD && make TARGET=cygwin
ERR=1 USE_ZLIB=1 USE_PCRE=1 USE_PCRE_JIT=1 USE_LUA=1 USE_OPENSSL=1 USE_THREAD=1
&& ./haproxy -vv'
-
diff --git a/.github/workflows/windows.yml b/.github/workflows/windows.yml
new file mode 100644
index 0..5393a4c3a
--- /dev/null
+++ b/.github/workflows/windows.yml
@@ -0,0 +1,60 @@
+# Copyright 2019 Ilya Shipitsin
+# Copyright 2020 Tim Duesterhus
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version
+# 2 of the License, or (at your option) any later version.
+
+name: Windows
+
+on:
+ push:
+
+jobs:
+ msys2:
+name: ${{ matrix.name }}
+runs-on: ${{ matrix.os }}
+defaults:
+ run:
+shell: msys2 {0}
+strategy:
+ matrix:
+include:
+- name: "Windows, gcc, all features"
+ TARGET: cygwin
+ CC: gcc
+ os: windows-latest
+ FLAGS:
+ - USE_OPENSSL=1
+ - USE_PCRE=1
+ - USE_PCRE_JIT=1
+ - USE_THREAD=1
+ - USE_ZLIB=1
+steps:
+- uses: actions/checkout@v2
+- uses: msys2/setup-msys2@v2
+ with:
+install: >-
+ coreutils
+ curl
+ diffutils
+ gawk
+ gcc
+ make
+ tar
+ openssl-devel
+ pcre-devel
+ zlib-devel
+- name: Compile HAProxy with ${{ matrix.CC }}
+ run: |
+make -j$(nproc) all \
+ ERR=1 \
+ TARGET=${{ matrix.TARGET }} \
+ CC=${{ matrix.CC }} \
+ ${{ join(matrix.FLAGS, ' ') }}
+- name: Show HAProxy version
+ id: show-version
+ run: |
+./haproxy -vv
+echo "::set-output name=version::$(./haproxy -v |awk 'NR==1{print
$3}')"
--
2.29.0
discuss, how better guard SSL_CTX_set_ciphersuites ?
I'd like to get rid of OPENSSL_VERSION as much as possible. what would be better for guarding TLS13 ciphers manipulation ? approach 1 (macro defined in openssl-compat.h) #if ((OPENSSL_VERSION_NUMBER >= 0x10101000L) && !defined(LIBRESSL_VERSION_NUMBER) && !defined(OPENSSL_IS_BORINGSSL)) #define HAVE_SSL_CTX_SET_CIPHERSUITES #endif approach 2 (macro TLS13_NUM_CIPHERS) #ifdef TLS13_NUM_CIPHERS conf_ciphersuites = (ssl_conf && ssl_conf->ciphersuites) ? ssl_conf->ciphersuites : bind_conf->ssl_conf.ciphersuites; ... #endif Ilya
Re: [PATCH] enable prometheus exporter in GH actions, adjust SSL matrix. cleanup travis-ci builds
Ilya, Am 19.11.20 um 21:00 schrieb Илья Шипицин: > next CI additions. > Thanks, the results of the changes look good to me. Can you please split the first patch into 2 or 3 different patches, though? It makes the commit history a bit cleaner and if they are all pushed together it will be effectively the same from resource usage. I'm thinking about: Patch 1. Add Prometheus. Patch 2. Remove LibreSSL 3.0.2. Patch 3. Add BoringSSL. Best regards Tim Düsterhus
[PATCH] enable prometheus exporter in GH actions, adjust SSL matrix. cleanup travis-ci builds
Hello, next CI additions. Ilya From bdb90411323c5d87dca8a10bc105ede3e46ab20e Mon Sep 17 00:00:00 2001 From: Ilya Shipitsin Date: Fri, 20 Nov 2020 00:58:46 +0500 Subject: [PATCH 2/2] CI: travis-ci: remove builds migrated to GH actions --- .travis.yml | 20 1 file changed, 20 deletions(-) diff --git a/.travis.yml b/.travis.yml index 1057ccc37..bb0dfeb8f 100644 --- a/.travis.yml +++ b/.travis.yml @@ -62,32 +62,12 @@ matrix: compiler: clang env: TARGET=linux-glibc OPENSSL_VERSION=1.1.0l FIFTYONEDEGREES_SRC="contrib/51d/src/trie" CC=clang-9 name: openssl-1.1.1 | 51d trie - - os: linux -if: type == push -compiler: clang -env: TARGET=linux-glibc LIBRESSL_VERSION=3.1.1 CC=clang-9 -name: libressl-3.1.1 - os: linux env: DEBUG_OPTIONS="" if: type == cron compiler: clang env: TARGET=linux-glibc LIBRESSL_VERSION=3.0.2 CC=clang-9 name: libressl-3.0.2 | ERR= - - os: linux -if: type == cron -compiler: clang -env: TARGET=linux-glibc LIBRESSL_VERSION=2.9.2 EXTRA_OBJS="contrib/prometheus-exporter/service-prometheus.o" CC=clang-9 -name: libressl-2.9.2 | prometheus-exporter - - os: linux -if: type == cron -compiler: clang -env: TARGET=linux-glibc BORINGSSL=yes -name: boringssl - - os: linux -if: type == push -compiler: clang -env: TARGET=linux-glibc FLAGS= CC=clang-9 -name: FLAGS= - os: linux if: type == cron compiler: clang -- 2.28.0 From da22093905a66ec3720b67c3dc21760b72b94944 Mon Sep 17 00:00:00 2001 From: Ilya Shipitsin Date: Fri, 20 Nov 2020 00:55:49 +0500 Subject: [PATCH 1/2] CI: Github Actions: enable prometheus exporter, adjust SSL variants Let us remove LibreSSL-3.0.2 (we do not need so many LibreSSL variants), enable BoringSSL --- .github/matrix.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/.github/matrix.py b/.github/matrix.py index 6e5b9247e..c08f85040 100644 --- a/.github/matrix.py +++ b/.github/matrix.py @@ -72,6 +72,7 @@ for CC in ["gcc", "clang"]: "WURFL_LIB=contrib/wurfl", "USE_DEVICEATLAS=1", "DEVICEATLAS_SRC=contrib/deviceatlas", +"EXTRA_OBJS=contrib/prometheus-exporter/service-prometheus.o", # "USE_51DEGREES=1", # "FIFTYONEDEGREES_SRC=contrib/51d/src/pattern", ], @@ -95,8 +96,8 @@ for CC in ["gcc", "clang"]: "stock", "OPENSSL_VERSION=1.0.2u", "LIBRESSL_VERSION=2.9.2", -"LIBRESSL_VERSION=3.0.2", "LIBRESSL_VERSION=3.1.1", +"BORINGSSL=yes", ]: flags = ["USE_OPENSSL=1"] if ssl != "stock": @@ -137,6 +138,7 @@ matrix.append( "WURFL_LIB=contrib/wurfl", "USE_DEVICEATLAS=1", "DEVICEATLAS_SRC=contrib/deviceatlas", +"EXTRA_OBJS=contrib/prometheus-exporter/service-prometheus.o", # "USE_51DEGREES=1", # "FIFTYONEDEGREES_SRC=contrib/51d/src/pattern", ], -- 2.28.0
Re: [PATCH] simplify openssl async detection
Thanks :) On Fri, Nov 20, 2020, 12:01 AM William Lallemand wrote: > On Thu, Nov 19, 2020 at 12:58:06AM +0500, Илья Шипицин wrote: > > ping :) ? > > > > сб, 14 нояб. 2020 г. в 02:04, Илья Шипицин : > > > > > Hi. > > > > > > next define improvement. > > > > > > Ilya > > > > > Thanks, merged. > > -- > William Lallemand >
Re: [PATCH] simplify openssl async detection
On Thu, Nov 19, 2020 at 12:58:06AM +0500, Илья Шипицин wrote: > ping :) ? > > сб, 14 нояб. 2020 г. в 02:04, Илья Шипицин : > > > Hi. > > > > next define improvement. > > > > Ilya > > Thanks, merged. -- William Lallemand
Re: [2.2.5] High cpu usage after switch to threads
Tim. Cool big thank to clarify that for me. Regards Aleks On 19.11.20 17:03, Tim Düsterhus wrote: Aleks, Am 19.11.20 um 16:53 schrieb Aleksandar Lazic: When a H2 client send the header in lowercase then and h1 in mixed-case could the "del-header" line not match when it's only written in lowercase or mixed-case . HTTP headers are defined to be case-insensitive. You quoted it yourself: Just as in HTTP/1.x, header field names are strings of ASCII characters that are compared in a case-insensitive fashion. "as in HTTP/1.x" Being able to request "case insensitive matching" with -i is redundant and confusing for the administrator. Personally I define all my headers within ACLs in a lowercase fashion and that works for whatever casing the client wants to use today. I think that was also one of the reason why the h1-case-adjust* feature exists ;-) The "feature" exists to support broken software that pretends to speak HTTP when it in fact does not. Best regards Tim Düsterhus
Re: [2.2.5] High cpu usage after switch to threads
Aleks, Am 19.11.20 um 16:53 schrieb Aleksandar Lazic: > When a H2 client send the header in lowercase then and h1 in mixed-case > could the "del-header" > line not match when it's only written in lowercase or mixed-case . HTTP headers are defined to be case-insensitive. You quoted it yourself: >Just as in HTTP/1.x, header field names are strings of ASCII >characters that are compared in a case-insensitive fashion. "as in HTTP/1.x" Being able to request "case insensitive matching" with -i is redundant and confusing for the administrator. Personally I define all my headers within ACLs in a lowercase fashion and that works for whatever casing the client wants to use today. > I think that was also one of the reason why the h1-case-adjust* feature > exists ;-) The "feature" exists to support broken software that pretends to speak HTTP when it in fact does not. Best regards Tim Düsterhus
Re: [2.2.5] High cpu usage after switch to threads
Hi. On 19.11.20 16:16, Maciej Zdeb wrote: Hi, Alaksandar I've looked into code and... :) Great ;-) śr., 18 lis 2020 o 15:30 Aleksandar Lazic mailto:al-hapr...@none.at>> napisał(a): Can you think to respectthe '-i'. http://git.haproxy.org/?p=haproxy.git=search=HEAD=grep=PAT_MF_IGNORE_CASE I'm not sure if I understand you correctly, but in case of http-request del-header the "case insensitivity" must be always enabled, because header names should be case insensitive according to RFC. So we should not implement "-i" flag in this scenario. Well in H2 are the headers in lowercase as far as I understand this part of H2 RFC in a proper way. But I'm not the expert in H2 so please correct me if I'm wrong. Hypertext Transfer Protocol Version 2 (HTTP/2) https://tools.ietf.org/html/rfc7540#section-8.1.2 ``` 8.1.2. HTTP Header Fields ... Just as in HTTP/1.x, header field names are strings of ASCII characters that are compared in a case-insensitive fashion. However, header field names MUST be converted to lowercase prior to their encoding in HTTP/2. ... ``` When a H2 client send the header in lowercase then and h1 in mixed-case could the "del-header" line not match when it's only written in lowercase or mixed-case . I think that was also one of the reason why the h1-case-adjust* feature exists ;-) Additional Info. What I have see in the the checking of '-i' (PAT_MF_IGNORE_CASE), the '-m reg' functions have not the PAT_MF_IGNORE_CASE check. I think you're looking at the regex execution, but flags are considered during regex compile. If you look at regex_comp function http://git.haproxy.org/?p=haproxy.git;a=blob;f=src/regex.c;h=45a7e9004e8e4f6ad9604ed9a858aba0060b6204;hb=0217b7b24bb33d746d2bf625f5e894007517d1b0#l312 you'll notice cs param. Function is called in pattern.c and couple other places. In my opinion -i with -m reg is perfectly valid and should work. Cool great, I thought I missed something. Thanks for the clarification. Regards Aleks
Re: [2.2.5] High cpu usage after switch to threads
Hi, Alaksandar I've looked into code and... :) śr., 18 lis 2020 o 15:30 Aleksandar Lazic napisał(a): > Can you think to respectthe '-i'. > > http://git.haproxy.org/?p=haproxy.git=search=HEAD=grep=PAT_MF_IGNORE_CASE > I'm not sure if I understand you correctly, but in case of http-request del-header the "case insensitivity" must be always enabled, because header names should be case insensitive according to RFC. So we should not implement "-i" flag in this scenario. > Additional Info. > > What I have see in the the checking of '-i' (PAT_MF_IGNORE_CASE), the '-m > reg' functions > have not the PAT_MF_IGNORE_CASE check. > I think you're looking at the regex execution, but flags are considered during regex compile. If you look at regex_comp function http://git.haproxy.org/?p=haproxy.git;a=blob;f=src/regex.c;h=45a7e9004e8e4f6ad9604ed9a858aba0060b6204;hb=0217b7b24bb33d746d2bf625f5e894007517d1b0#l312 you'll notice cs param. Function is called in pattern.c and couple other places. In my opinion -i with -m reg is perfectly valid and should work.
University study about the use of web technologies on websites
Dear web professional, We invite you to participate in a research study conducted at Ruhr University Bochum (Germany) in cooperation with Leibniz University Hannover (Germany) and the University of Michigan (USA). We found your email address (haproxy@formilux.org) on the website www.haproxy.com. We are interested in understanding how web technologies are integrated into and used by websites. We are looking for perspectives from a wide variety of people working on or with websites, such as developers, content creators, marketing, customer service, legal, and others. Our research relies on the generous help of interested web professionals like you! This online survey will take about 20-25 minutes to complete. You are eligible to participate in the study if you (1) are at least 18 years old, (2) are involved with websites in some capacity (e. g., their design, development, deployment, maintenance, and / or management), and (3) are comfortable completing the survey in English. In case you do not meet these requirements, you can still help us by forwarding this email to someone involved with the website www.haproxy.com. Participate! To participate, please use this link to access our survey: https://survey.syssec.ruhr-uni-bochum.de/index.php?r=survey/index=218135=xf4EoHGUR5cwI5X=en Other Options: * If you do not want to participate in this survey and do not want us to send you any further emails, please click the following link: https://survey.syssec.ruhr-uni-bochum.de/index.php?r=optout/tokens=218135=en=xf4EoHGUR5cwI5X * If you would like to reverse your opt-out decision, please click the following link: https://survey.syssec.ruhr-uni-bochum.de/index.php?r=optin/tokens=218135=en=xf4EoHGUR5cwI5X More details about our study and the recruiting process are available at https://www.syssec.ruhr-uni-bochum.de/research/webtechsurvey/. If you have any further questions, please contact us at survey+websi...@syssec-survey.de. Thank you for helping our research! Sincerely, Prof. Thorsten Holz (Ruhr University Bochum) Prof. Sascha Fahl (Leibniz University Hannover) Prof. Florian Schaub (University of Michigan)
Re: Disable client keep-alive using ACL
Christopher, Am 19.11.20 um 11:47 schrieb Christopher Faulet: > Le 19/11/2020 à 10:49, Tim Düsterhus, WoltLab GmbH a écrit : >> John, >> >> Am 19.11.20 um 06:57 schrieb John Lauro: >>> A couple of possible options... >>> You could use tcp-request inspect-delay to delay the response a >>> number of >>> seconds (and accept it quick if legitimate traffic). >> >> I believe the tcp-(request|response) rules only apply to the very first >> buffer of a single TCP connection and thus do not apply here. Let me >> give an example of what I want to do. >> > > Tim, > > tcp-(request|response) content rules are evaluated on all HTTP > transactions. tcp-request connection/session rules are only evaluated on > the client connection establishment. Oh, indeed and it's even documented. I just applied my knowledge about `mode tcp` without checking. I never had a use for tcp-request content rules for HTTP traffic, the http rules are simply much better. > However, this does not help you, because in H2, adding a delay on a > stream will not stop the traffic for other streams on the same connection. And even for H1 it would not help much for my use case, because having a connection sit idle for a few seconds still allows a rogue client to pay the TLS cost only once per connection, just increasing the number of connections required by a constant factor instead of linearly increasing the number of connections. > Thanks for the feature request. > Do you have a first estimation whether this would require intimate knowledge of the architecture and larger changes to implement? If it's a matter of adding a flag somewhere and a few lines checking that flag after handling the response then I might attempt it myself. Best regards Tim Düsterhus Developer WoltLab GmbH -- WoltLab GmbH Nedlitzer Str. 27B 14469 Potsdam Tel.: +49 331 96784338 duester...@woltlab.com www.woltlab.com Managing director: Marcel Werk AG Potsdam HRB 26795 P
Re: Disable client keep-alive using ACL
Le 19/11/2020 à 10:49, Tim Düsterhus, WoltLab GmbH a écrit : John, Am 19.11.20 um 06:57 schrieb John Lauro: A couple of possible options... You could use tcp-request inspect-delay to delay the response a number of seconds (and accept it quick if legitimate traffic). I believe the tcp-(request|response) rules only apply to the very first buffer of a single TCP connection and thus do not apply here. Let me give an example of what I want to do. Tim, tcp-(request|response) content rules are evaluated on all HTTP transactions. tcp-request connection/session rules are only evaluated on the client connection establishment. However, this does not help you, because in H2, adding a delay on a stream will not stop the traffic for other streams on the same connection. Thanks for the feature request. -- Christopher Faulet
Re: Disable client keep-alive using ACL
John, Am 19.11.20 um 06:57 schrieb John Lauro: > A couple of possible options... > You could use tcp-request inspect-delay to delay the response a number of > seconds (and accept it quick if legitimate traffic). I believe the tcp-(request|response) rules only apply to the very first buffer of a single TCP connection and thus do not apply here. Let me give an example of what I want to do. 1) Client connects 2) Client performs TLS handshake 3) Client sends HTTP request 4) Server sends response 5) Client sends HTTP request over the same connection (H1 keep-alive / H2). tcp-request / tcp-response does not apply here. 6) Server sends response 7) Repeat steps 4 and 5. If I start detecting unusual request rates for the client I would like to piggy back a clean (!) close onto step 6, forcing the client to perform steps 1 and 2 again. > You could use redirects which will have the clients do more requests > (Possibly with the inspect delays). Redirects do not play well with POST requests and they are very cheap on keep-alive connections. The costly part is the TLS handshake. > That said, it would be useful to force a client connection closed at times, > but there are ways to protect the backends and slow some clients without > completely blocking them. I went ahead and filed a feature request in the tracker now: https://github.com/haproxy/haproxy/issues/969 Thanks for the suggestions to both of you :-) Best regards Tim Düsterhus Developer WoltLab GmbH -- WoltLab GmbH Nedlitzer Str. 27B 14469 Potsdam Tel.: +49 331 96784338 duester...@woltlab.com www.woltlab.com Managing director: Marcel Werk AG Potsdam HRB 26795 P
