Re: "Success" logs in HTTP frontends

[Christian Ruppert]

On 2022-08-19 11:50, Christian Ruppert wrote:

On 2022-08-01 09:45, Christian Ruppert wrote:

On 2022-07-29 13:59, William Lallemand wrote:

On Fri, Jul 29, 2022 at 11:10:32AM +0200, Christopher Faulet wrote:

Le 7/29/22 à 10:13, Christian Ruppert a écrit :
> Hi list,
>
> so I noticed on my private HAProxy I have 2 of those logs within the
> past ~1-2 months:
> haproxy[28669]: 1.2.3.4:48596 [17/Jun/2022:13:55:18.530] public/HTTPSv4:
> Success
>
> So that's nothing so far but still no idea what that means.
> At work, of 250 mio log entries per day, there are about 600k of those
> "Success" ones.
> haproxy[27892]: 192.168.70.102:7904 [29/May/2022:00:13:37.316]
> genfrontend_35310-foobar/3: Success
>
> I'm not sure what it means by "3". Is it the third bind?
>
> I couldn't trigger those "Success" logs by either restarting or
> reloading. What is it for / where does it come from?
>

Hi Christian,

What is your version ? At first glance, I can't find such log 
message in the

code. It could come from a lua module.

In fact, I found something. It is probably because an "embryonic" 
session is
killed with no connection/ssl error. For instance, an SSL connection 
rejected
because of a "tcp-request session" rule (so after the SSL 
handshake). The same

may happen with a listener using the PROXY protocol.

Regards,



Could be something like that indeed, the "Success" message is the 
string

for CO_ER_NONE in the fc_err_str fetch. (The default error string)

Maybe we lack some intermediate state, or we could just change the
string ?

It is only the string for the handshake status so this is confusing 
when

used as an error.


Since it's that much every day I'd agree to change/improve it.
If it's the connection one then I only see it in combination with
SOCKS. There is no SOCKS in my config though, unless that also
triggers if something does a SOCKS request on that bind anyway.
I wasn't able to reproduce/trigger it that way yet.



Does anybody know how to trigger that on purpose? Would be really 
interesting.


So we have one system that does some DNAT stuff, well kind of at least, 
and triggers around ~700k of those "Success" logs per day. I still 
couldn't figure out the exact reason. The only hint I have is that it's 
really mostly that "DNAT" host. Some others also have some but by far 
not that much. On the same day, the others are all between 0 and 50.


--
Regards,
Christian Ruppert

[PATCH 1/2] CI: Replace the deprecated `::set-output` command by writing to $GITHUB_OUTPUT in matrix.py

[Tim Duesterhus]

As announced in

https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/

the `::set-output` command is deprecated, because processes during the workflow
execution might output untrusted information that might include the
`::set-output` command, thus allowing these untrusted information to hijack the
build.

The replacement is writing to the file indicated by the `$GITHUB_OUTPUT`
environment variable.
---
 .github/matrix.py | 5 -
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/.github/matrix.py b/.github/matrix.py
index 19ae87d79c..76e0893a7b 100755
--- a/.github/matrix.py
+++ b/.github/matrix.py
@@ -12,6 +12,7 @@ import json
 import sys
 import urllib.request
 import re
+from os import environ
 
 if len(sys.argv) == 2:
 build_type = sys.argv[1]
@@ -208,4 +209,6 @@ for CC in ["clang"]:
 
 print(json.dumps(matrix, indent=4, sort_keys=True))
 
-print("::set-output name=matrix::{}".format(json.dumps({"include": matrix})))
+if environ.get('GITHUB_OUTPUT') is not None:
+with open(environ.get('GITHUB_OUTPUT'), 'a') as f:
+print("matrix={}".format(json.dumps({"include": matrix})), file=f)
-- 
2.38.0

[PATCH 2/2] CI: Replace the deprecated `::set-output` command by writing to $GITHUB_OUTPUT in workflow definition

[Tim Duesterhus]

See "CI: Replace the deprecated `::set-output` command by writing to
$GITHUB_OUTPUT in matrix.py" for the reasoning behind this commit.
---
 .github/workflows/compliance.yml | 4 ++--
 .github/workflows/vtest.yml  | 4 ++--
 .github/workflows/windows.yml| 2 +-
 3 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/.github/workflows/compliance.yml b/.github/workflows/compliance.yml
index 9d14ff57d5..509eaf8411 100644
--- a/.github/workflows/compliance.yml
+++ b/.github/workflows/compliance.yml
@@ -27,7 +27,7 @@ jobs:
 curl -fsSL 
https://github.com/summerwind/h2spec/releases/download/${H2SPEC_VERSION}/h2spec_linux_amd64.tar.gz
 -o h2spec.tar.gz
 tar xvf h2spec.tar.gz
 sudo install -m755 h2spec /usr/local/bin/h2spec
-echo "::set-output name=version::${H2SPEC_VERSION}"
+echo "version=${H2SPEC_VERSION}" >> $GITHUB_OUTPUT
 - name: Compile HAProxy with ${{ matrix.CC }}
   run: |
 make -j$(nproc) all \
@@ -50,7 +50,7 @@ jobs:
 fi
 echo "::endgroup::"
 haproxy -vv
-echo "::set-output name=version::$(haproxy -v |awk 'NR==1{print $3}')"
+echo "version=$(haproxy -v |awk 'NR==1{print $3}')" >> $GITHUB_OUTPUT
 - name: Launch HAProxy ${{ steps.show-version.outputs.version }}
   run: haproxy -f .github/h2spec.config -D
 - name: Run h2spec ${{ steps.install-h2spec.outputs.version }}
diff --git a/.github/workflows/vtest.yml b/.github/workflows/vtest.yml
index 7a1c1ef62a..75c79c25af 100644
--- a/.github/workflows/vtest.yml
+++ b/.github/workflows/vtest.yml
@@ -53,7 +53,7 @@ jobs:
 - name: Generate cache key
   id: generate-cache-key
   run: |
-echo "::set-output name=key::$(echo ${{ matrix.name }} | sha256sum | 
awk '{print $1}')"
+echo "key=$(echo ${{ matrix.name }} | sha256sum | awk '{print $1}')" 
>> $GITHUB_OUTPUT
 
 - name: Cache SSL libs
   if: ${{ matrix.ssl && matrix.ssl != 'stock' && matrix.ssl != 
'BORINGSSL=yes' && matrix.ssl != 'QUICTLS=yes' }}
@@ -124,7 +124,7 @@ jobs:
 fi
 echo "::endgroup::"
 haproxy -vv
-echo "::set-output name=version::$(haproxy -v |awk 'NR==1{print $3}')"
+echo "version=$(haproxy -v |awk 'NR==1{print $3}')" >> $GITHUB_OUTPUT
 - name: Install problem matcher for VTest
   # This allows one to more easily see which tests fail.
   run: echo "::add-matcher::.github/vtest.json"
diff --git a/.github/workflows/windows.yml b/.github/workflows/windows.yml
index 3d034617fe..3030908623 100644
--- a/.github/workflows/windows.yml
+++ b/.github/workflows/windows.yml
@@ -64,4 +64,4 @@ jobs:
   id: show-version
   run: |
 ./haproxy -vv
-echo "::set-output name=version::$(./haproxy -v |awk 'NR==1{print 
$3}')"
+echo "version=$(./haproxy -v |awk 'NR==1{print $3}')" >> $GITHUB_OUTPUT
-- 
2.38.0

Re: [PATCH 2/2] CI: Replace the deprecated `::set-output` command by writing to $GITHUB_OUTPUT in workflow definition

[Willy Tarreau]

On Fri, Oct 14, 2022 at 07:46:07PM +0200, Tim Duesterhus wrote:
> See "CI: Replace the deprecated `::set-output` command by writing to
> $GITHUB_OUTPUT in matrix.py" for the reasoning behind this commit.
(...)

both patches applied, thanks Tim!
Willy

[ANNOUNCE] haproxy-2.7-dev8

[Willy Tarreau]

Hi,

HAProxy 2.7-dev8 was released on 2022/10/14. It added 81 new commits
after version 2.7-dev7.

Not much changed since dev7, we're seeing a few fixes and twice as many
cleanup and doc patches. That's nice. Other improvements are roughly
spread like this:

  - master-cli: the "reload" command now reports the operation status and
dumps the startup logs. This is particularly useful in environments
like systemd where there's normally no feedback reload on actions.
The feature requires shm_open() and is currently only enabled on the
linux-glibc target, though I could verify that it does also work on
freebsd. Testers are welcome, just pass USE_SHM_OPEN=1 to the make
command for this.

  - health checks: the checks were being scheduled on any thread, which
caused all threads to periodically wake up and compete on the shared
wait queue lock. I noticed that on a machine with 48 threads, only 40
servers were enough to cause fast wakeups and 2% CPU! And as a side
effect, threads were picking many checks at once, causing a significant
imbalance that was directly responsible for some of the higher latency
tails in tests. This was modified so that a check is now rescheduled
on a random thread if that other thread is les loaded. This resulted
in significantly smoothing the load over just a few seconds, and
eliminating the latency spikes and the CPU load since there's no more
contention on the shared lock. However, checks are now run by more
threads, which may cause more CPU usage to be reported in a very
specific case (many servers with very short check interval, ~10ms),
because threads do not have the time to sleep a full time slice
anymore. Such setups are clearly unlikely (and even less likely to
run idle with no load at all) so that's normally not a problem,
considering how it improves a number of shortcomings. But I'm
interested in any reports of increased idle CPU usage if that would
happen.

  - quic: the "thread" keyword on bind lines was not respected, all
threads were always used. This was now changed so that it becomes
possible to dedicate only certain threads to QUIC, or possibly to
start several listeners on different thread groups.

  - quic: the frontend's destination address was not retrieved previously
due to limitations of the BSD socket API. One of the limitations is
that having a listener on 0.0.0.0 doesn't work if several addresses
are present on the machine, as the response will take the source
address that can reach the target from the routing table. Switching
to recvmsg() allows on certain systems to receive packet info with
datagrams, and among these info is the packet's destination address.
This is automatically detected at build time, and we could verify
that Linux and FreeBSD both work (IPv4 + IPv6). Maybe we should add
an entry for this in "haproxy -vv" to help diagnose incompatible
setups. There were also various other low-level improvements.

  - stick-tables: since 1.8, all the locking needed to protect against
concurrent accesses from different threads was being performed using
spinlocks. This doesn't scale with many threads, as can be seen when
using track-sc on a large-scale machine. The code was adapted to
support R/W locks and that completely unleashed the unused performance.
A 48-thread machine jumped from 180k to 2.1M requests per second with
3 track-sc, that's a 11x improvement!

  - init: we've seen a few reports of strange errors in certain docker
setups roughly saying "impossible to set the FD limit to 8k, leaving
it to 1 billion". This sounds absurd, but might be caused by certain
syscall restrictions in such environments. There's no point reducing
the number of FDs, it's just a limit. So the code was adjusted so
that we only try to raise the limit but do nothing in case where we
need to lower it.

  - lua: some fixes to better match what the doc says, and to avoid
a risk of crash when passing wrong arguments to certain functions
related to stick-tables.

I'm also aware of an upcoming improvement in the Lua area to speed up
startup of large configs (Lua even if unused builds a table of all
backends and servers, and that takes time). And some updates to QUIC
should happen soon to better deal with packet drops, polling and reloads.

With the haproxyconf in less than one month and the amazing amount of time
it takes to the speakers to prepare their talks, I'm anticipating a bit of
slowdown on new patches and bug fixes for the forthcoming weeks (it has
already started for me), so we need to keep reasonable expectations and
goals for the upcoming release to keep a target on end of November, and
at the same time test reports are more than ever well appreciated.

Please find the usual URLs below :
   Site index   : https://www.haproxy.org/
   Documentation: https://docs.hap

[PATCH] improve quictls build time

[Илья Шипицин]

Hello,

currently QuicTLS takes 3m40s
disabling "tests" saves 40sec per build.


cheers,
Ilya
From ab0b26d299c143a69ca32834f5044b498cc602d0 Mon Sep 17 00:00:00 2001
From: Ilya Shipitsin 
Date: Sat, 15 Oct 2022 09:55:49 +0500
Subject: [PATCH] BUILD: scripts: disable tests build on QuicTLS build

during CI builds QuicTLS is not cached, let us speed it up by
disabling tests build
---
 scripts/build-ssl.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/build-ssl.sh b/scripts/build-ssl.sh
index 42eef1902..4360adf34 100755
--- a/scripts/build-ssl.sh
+++ b/scripts/build-ssl.sh
@@ -137,7 +137,7 @@ if [ ! -z ${QUICTLS+x} ]; then
 download_quictls
 cd download-cache/quictls
 
-./config shared  ${QUICTLS_EXTRA_ARGS:-} --prefix="${HOME}/opt" --openssldir="${HOME}/opt" --libdir=lib -DPURIFY
+./config shared no-tests ${QUICTLS_EXTRA_ARGS:-} --prefix="${HOME}/opt" --openssldir="${HOME}/opt" --libdir=lib -DPURIFY
 make -j$(nproc) build_sw
 make install_sw
 
-- 
2.37.3