Re: Thoughts on QUIC/HTTP3

[Shawn Heisey]

On 7/9/22 18:08, William Lallemand wrote:

But is there any certificates in the /opt/quictls/ssl/certs/ directory ?


No, it is empty.  I didn't think to actually look inside it because it 
didn't occur to me that it would be empty.  I just checked an install of 
stock openssl 3 and it also has no certs.


I see that /usr/lib/ssl/certs is a symlink to /etc/ssl/certs ... so I 
have one more build step to put in my script -- make that symlink for 
quictls.  Once I did that manually, no more warning.


Thanks,
Shawn

Re: Thoughts on QUIC/HTTP3

[William Lallemand]

On Fri, Jul 08, 2022 at 09:11:02AM -0600, Shawn Heisey wrote:
> 
> The openssl that haproxy is compiled against is in /opt/quictls/ssl ... 
> but there is a distribution-provided openssl package in /usr/lib/ssl as 
> well.  Both locations contain "certs".
> 

But is there any certificates in the /opt/quictls/ssl/certs/ directory ?

> Setting either environment variable that you have mentioned does not 
> eliminate the warning.

It should already be set to /opt/quictls/ssl/certs/, if you specified
the openssldir at /opt/quictls/ssl/ during the build of your library.

> root@bilbo:~# SSL_CERT_DIR=/opt/quictls/ssl/certs haproxy -c -f 
> /etc/haproxy/haproxy.cfg
> [NOTICE]   (2379692) : haproxy version is 2.6.1
> [NOTICE]   (2379692) : path to executable is /usr/local/sbin/haproxy
> [WARNING]  (2379692) : config : ca-file: 0 CA were loaded from '@system-ca'
> Warnings were found.
> Configuration file is valid
> root@bilbo:~# OPENSSLDIR=/opt/quictls/ssl haproxy -c -f 
> /etc/haproxy/haproxy.cfg
> [NOTICE]   (2379701) : haproxy version is 2.6.1
> [NOTICE]   (2379701) : path to executable is /usr/local/sbin/haproxy
> [WARNING]  (2379701) : config : ca-file: 0 CA were loaded from '@system-ca'
> Warnings were found.
> Configuration file is valid
> 
> My setup has no need to verify certificates, so the warning doesn't 
> actually matter for me.  But it could be a problem for someone else.
> 

In fact there is a warning because you might want to use the httpclient
at runtime, and the httpclient is using the CAs, so they are loaded at
startup.

I supposed you don't have anything in this directory, or it failed to
load for some reason and this could be a bug we need to fix.

The message should have been about the httpclient, I'll look into this
to clarify it.

> I did figure out the correct way to run the "version -d" command you 
> mentioned on the quictls install:
> 
> elyograg@smeagol:~$ LD_LIBRARY_PATH=/opt/quictls/lib64 
> /opt/quictls/bin/openssl version -d
> OPENSSLDIR: "/opt/quictls/ssl"
 
You had to use LD_LIBRARY_PATH because you didn't use the rpath when
compiling, this is necessary if you don't install the library in
/usr/lib/.
You only need to add -Wl,-rpath=/opt/quictls/lib64 to your ./config line.
https://wiki.openssl.org/index.php/Compilation_and_Installation#Using_RPATHs


> My install does quic/http3 correctly, so I know it is finding and using 
> quictls.
> 

Ok, you can always check with ldd if you have some doubts.

-- 
William Lallemand

Re: Thoughts on QUIC/HTTP3

[Shawn Heisey]

On 7/8/22 03:30, William Lallemand wrote:

HAProxy uses the ca-certificates provided by OpenSSL.
The SSL_CERT_DIR by default is set to the "certs" directory inside your
openssldir. You can check your openssldir by using the "openssl" binary
you compiled with your library (not the one of your distribution).

   $ openssl version -d
   OPENSSLDIR: "/usr/lib/ssl"

So you might want to set the SSL_CERT_DIR environment variable before
starting HAProxy or doing a symlink from your openssldir to the real
path of your ca-certificates ( /etc/ssl/certs ? )

This warning is emitted when trying to load the ca-certificates into the
httpclient at startup with an empty directory. (Which is not supposed to
happen on the openssl build of your distribution)



The openssl that haproxy is compiled against is in /opt/quictls/ssl ... 
but there is a distribution-provided openssl package in /usr/lib/ssl as 
well.  Both locations contain "certs".


Setting either environment variable that you have mentioned does not 
eliminate the warning.


root@bilbo:~# SSL_CERT_DIR=/opt/quictls/ssl/certs haproxy -c -f 
/etc/haproxy/haproxy.cfg

[NOTICE]   (2379692) : haproxy version is 2.6.1
[NOTICE]   (2379692) : path to executable is /usr/local/sbin/haproxy
[WARNING]  (2379692) : config : ca-file: 0 CA were loaded from '@system-ca'
Warnings were found.
Configuration file is valid
root@bilbo:~# OPENSSLDIR=/opt/quictls/ssl haproxy -c -f 
/etc/haproxy/haproxy.cfg

[NOTICE]   (2379701) : haproxy version is 2.6.1
[NOTICE]   (2379701) : path to executable is /usr/local/sbin/haproxy
[WARNING]  (2379701) : config : ca-file: 0 CA were loaded from '@system-ca'
Warnings were found.
Configuration file is valid

My setup has no need to verify certificates, so the warning doesn't 
actually matter for me.  But it could be a problem for someone else.


I did figure out the correct way to run the "version -d" command you 
mentioned on the quictls install:


elyograg@smeagol:~$ LD_LIBRARY_PATH=/opt/quictls/lib64 
/opt/quictls/bin/openssl version -d

OPENSSLDIR: "/opt/quictls/ssl"

My install does quic/http3 correctly, so I know it is finding and using 
quictls.


Thanks,
Shawn

Re: Thoughts on QUIC/HTTP3

[William Lallemand]

On Thu, Jul 07, 2022 at 07:53:24AM -0600, Shawn Heisey wrote:
> On 7/6/22 09:50, Илья Шипицин wrote:
> > haproxy is built in CI against latest quictls, for example quictls-3.0.5
> >
> > https://github.com/haproxy/haproxy/runs/721404?check_suite_focus=true
> >
> > please open an issue on github with failure details, no known build 
> > failures so far
> 
> Shortly after I saw this message, I tried the build again.  My script 
> does "git pull" on the repo.  There were a bunch of updates to the 
> quictls repo, and now haproxy compiles and runs.
> 
> I am getting a new config warning, though:
> 
> elyograg@bilbo:/usr/local/src$ sudo haproxy -c -f /etc/haproxy/haproxy.cfg
> [NOTICE]   (2080586) : haproxy version is 2.6.1
> [NOTICE]   (2080586) : path to executable is /usr/local/sbin/haproxy
> [WARNING]  (2080586) : config : ca-file: 0 CA were loaded from '@system-ca'
> Warnings were found.
> Configuration file is valid
> 
 
HAProxy uses the ca-certificates provided by OpenSSL.
The SSL_CERT_DIR by default is set to the "certs" directory inside your
openssldir. You can check your openssldir by using the "openssl" binary
you compiled with your library (not the one of your distribution).

  $ openssl version -d
  OPENSSLDIR: "/usr/lib/ssl"

So you might want to set the SSL_CERT_DIR environment variable before
starting HAProxy or doing a symlink from your openssldir to the real
path of your ca-certificates ( /etc/ssl/certs ? )

This warning is emitted when trying to load the ca-certificates into the
httpclient at startup with an empty directory. (Which is not supposed to
happen on the openssl build of your distribution)

-- 
William Lallemand

Re: Thoughts on QUIC/HTTP3

[Shawn Heisey]

On 7/6/22 09:50, Илья Шипицин wrote:

haproxy is built in CI against latest quictls, for example quictls-3.0.5

https://github.com/haproxy/haproxy/runs/721404?check_suite_focus=true

please open an issue on github with failure details, no known build 
failures so far


Shortly after I saw this message, I tried the build again.  My script 
does "git pull" on the repo.  There were a bunch of updates to the 
quictls repo, and now haproxy compiles and runs.


I am getting a new config warning, though:

elyograg@bilbo:/usr/local/src$ sudo haproxy -c -f /etc/haproxy/haproxy.cfg
[NOTICE]   (2080586) : haproxy version is 2.6.1
[NOTICE]   (2080586) : path to executable is /usr/local/sbin/haproxy
[WARNING]  (2080586) : config : ca-file: 0 CA were loaded from '@system-ca'
Warnings were found.
Configuration file is valid

This is happening on Ubuntu 20, Ubuntu 22, and two different OSes for 
arm64 hardware.  I have the latest from the haproxy 2.6 git repo and the 
quictls 3.0.5 branch.


elyograg@bilbo:/usr/local/src$ haproxy -vv
HAProxy version 2.6.1 2022/06/21 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 
2027.

Known bugs: http://www.haproxy.org/bugs/bugs-2.6.1.html
Running on: Linux 5.15.0-1014-aws #18~20.04.1-Ubuntu SMP Wed Jun 15 
21:28:54 UTC 2022 x86_64

Build options :
  TARGET  = linux-glibc
  CPU = native
  CC  = cc
  CFLAGS  = -O2 -march=native -g -Wall -Wextra -Wundef 
-Wdeclaration-after-statement -Wfatal-errors -Wtype-limits 
-Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond 
-Wnull-dereference -fwrapv -Wno-address-of-packed-member 
-Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered 
-Wno-missing-field-initializers -Wno-cast-function-type 
-Wno-string-plus-int -Wno-atomic-alignment
  OPTIONS = USE_PCRE2_JIT=1 USE_OPENSSL=1 USE_ZLIB=1 USE_SYSTEMD=1 
USE_QUIC=1

  DEBUG   = -DDEBUG_H3 -DDEBUG_QPACK

Feature list : +EPOLL -KQUEUE +NETFILTER -PCRE -PCRE_JIT -PCRE2 
+PCRE2_JIT +POLL +THREAD +BACKTRACE -STATIC_PCRE -STATIC_PCRE2 +TPROXY 
+LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H -ENGINE +GETADDRINFO 
+OPENSSL -LUA +ACCEPT4 -CLOSEFROM +ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL 
+RT -DEVICEATLAS -51DEGREES -WURFL +SYSTEMD -OBSOLETE_LINKER +PRCTL 
-PROCCTL +THREAD_DUMP -EVPORTS -OT +QUIC -PROMEX -MEMORY_PROFILING


Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=2).
Built with OpenSSL version : OpenSSL 3.0.5+quic 5 Jul 2022
Running on OpenSSL version : OpenSSL 3.0.5+quic 5 Jul 2022
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
OpenSSL providers loaded : default
Built with network namespace support.
Support for malloc_trim() is enabled.
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), 
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT 
IPV6_TRANSPARENT IP_FREEBIND

Built with PCRE2 version : 10.34 2019-11-21
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 9.4.0

Available polling systems :
  epoll : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as  cannot be specified using 'proto' keyword)
   quic : mode=HTTP  side=FE mux=QUIC flags=HTX|NO_UPG|FRAMED
 h2 : mode=HTTP  side=FE|BE  mux=H2 flags=HTX|HOL_RISK|NO_UPG
   fcgi : mode=HTTP  side=BE mux=FCGI flags=HTX|HOL_RISK|NO_UPG
   : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
 h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
   : mode=TCP   side=FE|BE  mux=PASS  flags=
   none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG

Available services : none

Available filters :
    [CACHE] cache
    [COMP] compression
    [FCGI] fcgi-app
    [SPOE] spoe
    [TRACE] trace

Re: Thoughts on QUIC/HTTP3

[Илья Шипицин]

ср, 6 июл. 2022 г. в 19:11, Shawn Heisey :

> On 5/31/22 08:16, Amaury Denoyelle wrote:
> > Thanks for your continuing your journey on HTTP/3 :)
>
> Yesterday I pulled down the haproxy 2.6 and quictls git repos. The
> branch for quictls was openssl-3.0.4+quic.  I built and installed
> quictls and then haproxy.
>
> This combination is working better than previous combinations.  It does
> look like some of my sites are still having issues on http3, but
> roundcube webmail is working very well, and that was one of the sites
> where I had a LOT of trouble before.
>
> A note for you, which you may already know:  If the openssl-3.0.5+quic
> branch of quictls is installed instead of earlier 3.0.x releases, then
> haproxy will not compile.
>

haproxy is built in CI against latest quictls, for example quictls-3.0.5

https://github.com/haproxy/haproxy/runs/721404?check_suite_focus=true

please open an issue on github with failure details, no known build
failures so far



>
> Thanks,
> Shawn
>
>
>

Re: Thoughts on QUIC/HTTP3

[Shawn Heisey]

On 5/31/22 08:16, Amaury Denoyelle wrote:
Thanks for your continuing your journey on HTTP/3 :) 


Yesterday I pulled down the haproxy 2.6 and quictls git repos. The 
branch for quictls was openssl-3.0.4+quic.  I built and installed 
quictls and then haproxy.


This combination is working better than previous combinations.  It does 
look like some of my sites are still having issues on http3, but 
roundcube webmail is working very well, and that was one of the sites 
where I had a LOT of trouble before.


A note for you, which you may already know:  If the openssl-3.0.5+quic 
branch of quictls is installed instead of earlier 3.0.x releases, then 
haproxy will not compile.


Thanks,
Shawn

Re: Thoughts on QUIC/HTTP3

[Amaury Denoyelle]

On Sun, May 29, 2022 at 12:37:14PM -0600, Shawn Heisey wrote:
> On 4/29/2022 10:10 AM, Shawn Heisey wrote:
> > I did a build and install this morning, a bunch of quic-related changes
> > in that.  Now everything seems to be working on my paste site.  Large
> > pastes work, and I can reload the page a ton of times without it hanging
> > until browser restart.
> I have found that the roundcube package doesn't work completely right with
> quic/http3. It has proven to be an excellent way of revealing problems.  The
> roundcube package is a php webmail app that utilizes a database and an imap
> server.  It's using many of the features that are possible with modern
> browsers.
> When I first started this http3 journey, I couldn't even log in to roundcube
> because any POST request was failing ... but that problem got fixed.  Now I
> find that when quickly jumping between folders in my big mailbox, the site
> stops responding.  If I switch back to http/2, everything works.  If it
> would help, I can give one of the devs full access to use my roundcube
> install and experiment with the config, see if maybe I am doing something
> wrong.

Hi Shawn,

Thanks for your continuing your journey on HTTP/3 :)

On our side we finish our work on improving POST, it should be fairly
complete now. We also improve other aspects and complete the
documentation for new users. Now, we are using nextcloud, which is
another big webapp, through haproxy/QUIC as a test. We have already fix
some bugs which may also help with your roundcube installation.

However, despite our efforts we know there is still a grey area in our
QUIC stack. We do not properly manage the connection closing, and it's
not completely clear how we should handle all the cases. Sadly, this
requires some reflexion time and it won't be solve before the imminent
2.6 release. For the moment, you can try on your side to
increase/decrease the haproxy client timeout to see if this change your
situation.

Regards,

-- 
Amaury Denoyelle

Re: Thoughts on QUIC/HTTP3

[Илья Шипицин]

пн, 30 мая 2022 г. в 00:56, Shawn Heisey :

> On 5/29/2022 12:49 PM, Илья Шипицин wrote:
> > roundcube runs automatic browser tests
> >
> >
> https://github.com/roundcube/roundcubemail/runs/6642129873?check_suite_focus=true
> >
> > I think we can try to run those tests with haproxy between browser and
> > roundcube
>
> That looks cool.  Are there instructions somewhere for setting up and
> configuring the test suite to run it against my install?  I didn't see
> any with a quick look, but I will keep on the search for a little while.
>

usually I take script written in action definition

roundcubemail/browser_tests.yml at master · roundcube/roundcubemail
(github.com)


and modify it accordingly.

there's a little chance that someone has done that already, I'm afraid it
is about zero.



> Thanks,
> Shawn
>
>
>

Re: Thoughts on QUIC/HTTP3

[Shawn Heisey]

On 5/29/2022 12:49 PM, Илья Шипицин wrote:

roundcube runs automatic browser tests

https://github.com/roundcube/roundcubemail/runs/6642129873?check_suite_focus=true

I think we can try to run those tests with haproxy between browser and 
roundcube


That looks cool.  Are there instructions somewhere for setting up and 
configuring the test suite to run it against my install?  I didn't see 
any with a quick look, but I will keep on the search for a little while.


Thanks,
Shawn

Re: Thoughts on QUIC/HTTP3

[Илья Шипицин]

вс, 29 мая 2022 г. в 23:40, Shawn Heisey :

> On 4/29/2022 10:10 AM, Shawn Heisey wrote:
> > I did a build and install this morning, a bunch of quic-related
> > changes in that.  Now everything seems to be working on my paste
> > site.  Large pastes work, and I can reload the page a ton of times
> > without it hanging until browser restart.
>
> I have found that the roundcube package doesn't work completely right
> with quic/http3. It has proven to be an excellent way of revealing
> problems.  The roundcube package is a php webmail app that utilizes a
> database and an imap server.  It's using many of the features that are
> possible with modern browsers.
>

roundcube runs automatic browser tests

https://github.com/roundcube/roundcubemail/runs/6642129873?check_suite_focus=true

I think we can try to run those tests with haproxy between browser and
roundcube


>
> When I first started this http3 journey, I couldn't even log in to
> roundcube because any POST request was failing ... but that problem got
> fixed.  Now I find that when quickly jumping between folders in my big
> mailbox, the site stops responding.  If I switch back to http/2,
> everything works.  If it would help, I can give one of the devs full
> access to use my roundcube install and experiment with the config, see
> if maybe I am doing something wrong.
>
> Here's my haproxy -vv output.  I built it from the master branch right
> after I saw the dev12 release announcement:
>
> HAProxy version 2.6-dev12 2022/05/27 - https://haproxy.org/
> Status: development branch - not safe for use in production.
> Known bugs: https://github.com/haproxy/haproxy/issues?q=is:issue+is:open
> Running on: Linux 5.13.0-1023-aws #25~20.04.1-Ubuntu SMP Mon Apr 25
> 19:28:27 UTC 2022 x86_64
> Build options :
>TARGET  = linux-glibc
>CPU = native
>CC  = cc
>CFLAGS  = -O2 -march=native -g -Wall -Wextra -Wundef
> -Wdeclaration-after-statement -Wfatal-errors -Wtype-limits
> -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond
> -Wnull-dereference -fwrapv -Wno-address-of-packed-member
> -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered
> -Wno-missing-field-initializers -Wno-cast-function-type
> -Wno-string-plus-int -Wno-atomic-alignment
>OPTIONS = USE_PCRE2_JIT=1 USE_OPENSSL=1 USE_ZLIB=1 USE_SYSTEMD=1
> USE_QUIC=1
>DEBUG   = -DDEBUG_H3 -DDEBUG_QPACK
>
> Feature list : +EPOLL -KQUEUE +NETFILTER -PCRE -PCRE_JIT -PCRE2
> +PCRE2_JIT +POLL +THREAD +BACKTRACE -STATIC_PCRE -STATIC_PCRE2 +TPROXY
> +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H -ENGINE +GETADDRINFO
> +OPENSSL -LUA +ACCEPT4 -CLOSEFROM +ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL
> +RT -DEVICEATLAS -51DEGREES -WURFL +SYSTEMD -OBSOLETE_LINKER +PRCTL
> -PROCCTL +THREAD_DUMP -EVPORTS -OT +QUIC -PROMEX -MEMORY_PROFILING
>
> Default settings :
>bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
>
> Built with multi-threading support (MAX_THREADS=64, default=2).
> Built with OpenSSL version : OpenSSL 3.0.3+quic 3 May 2022
> Running on OpenSSL version : OpenSSL 3.0.3+quic 3 May 2022
> OpenSSL library supports TLS extensions : yes
> OpenSSL library supports SNI : yes
> OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
> OpenSSL providers loaded : default
> Built with network namespace support.
> Built with zlib version : 1.2.11
> Running on zlib version : 1.2.11
> Compression algorithms supported : identity("identity"),
> deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
> Support for malloc_trim() is enabled.
> Built with transparent proxy support using: IP_TRANSPARENT
> IPV6_TRANSPARENT IP_FREEBIND
> Built with PCRE2 version : 10.34 2019-11-21
> PCRE2 library supports JIT : yes
> Encrypted password support via crypt(3): yes
> Built with gcc compiler version 9.4.0
>
> Available polling systems :
>epoll : pref=300,  test result OK
> poll : pref=200,  test result OK
>   select : pref=150,  test result OK
> Total: 3 (3 usable), will use epoll.
>
> Available multiplexer protocols :
> (protocols marked as  cannot be specified using 'proto' keyword)
> quic : mode=HTTP  side=FE mux=QUIC flags=HTX|NO_UPG|FRAMED
>   h2 : mode=HTTP  side=FE|BE  mux=H2 flags=HTX|HOL_RISK|NO_UPG
> fcgi : mode=HTTP  side=BE mux=FCGI flags=HTX|HOL_RISK|NO_UPG
> : mode=HTTP  side=FE|BE  mux=H1flags=HTX
>   h1 : mode=HTTP  side=FE|BE  mux=H1flags=HTX|NO_UPG
> : mode=TCP   side=FE|BE  mux=PASS  flags=
> none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG
>
> Available services : none
>
> Available filters :
>  [CACHE] cache
>  [COMP] compression
>  [FCGI] fcgi-app
>  [SPOE] spoe
>  [TRACE] trace
>
>
>

Re: Thoughts on QUIC/HTTP3

[Shawn Heisey]

On 4/29/2022 10:10 AM, Shawn Heisey wrote:
I did a build and install this morning, a bunch of quic-related 
changes in that.  Now everything seems to be working on my paste 
site.  Large pastes work, and I can reload the page a ton of times 
without it hanging until browser restart.


I have found that the roundcube package doesn't work completely right 
with quic/http3. It has proven to be an excellent way of revealing 
problems.  The roundcube package is a php webmail app that utilizes a 
database and an imap server.  It's using many of the features that are 
possible with modern browsers.


When I first started this http3 journey, I couldn't even log in to 
roundcube because any POST request was failing ... but that problem got 
fixed.  Now I find that when quickly jumping between folders in my big 
mailbox, the site stops responding.  If I switch back to http/2, 
everything works.  If it would help, I can give one of the devs full 
access to use my roundcube install and experiment with the config, see 
if maybe I am doing something wrong.


Here's my haproxy -vv output.  I built it from the master branch right 
after I saw the dev12 release announcement:


HAProxy version 2.6-dev12 2022/05/27 - https://haproxy.org/
Status: development branch - not safe for use in production.
Known bugs: https://github.com/haproxy/haproxy/issues?q=is:issue+is:open
Running on: Linux 5.13.0-1023-aws #25~20.04.1-Ubuntu SMP Mon Apr 25 
19:28:27 UTC 2022 x86_64

Build options :
  TARGET  = linux-glibc
  CPU = native
  CC  = cc
  CFLAGS  = -O2 -march=native -g -Wall -Wextra -Wundef 
-Wdeclaration-after-statement -Wfatal-errors -Wtype-limits 
-Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond 
-Wnull-dereference -fwrapv -Wno-address-of-packed-member 
-Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered 
-Wno-missing-field-initializers -Wno-cast-function-type 
-Wno-string-plus-int -Wno-atomic-alignment
  OPTIONS = USE_PCRE2_JIT=1 USE_OPENSSL=1 USE_ZLIB=1 USE_SYSTEMD=1 
USE_QUIC=1

  DEBUG   = -DDEBUG_H3 -DDEBUG_QPACK

Feature list : +EPOLL -KQUEUE +NETFILTER -PCRE -PCRE_JIT -PCRE2 
+PCRE2_JIT +POLL +THREAD +BACKTRACE -STATIC_PCRE -STATIC_PCRE2 +TPROXY 
+LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H -ENGINE +GETADDRINFO 
+OPENSSL -LUA +ACCEPT4 -CLOSEFROM +ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL 
+RT -DEVICEATLAS -51DEGREES -WURFL +SYSTEMD -OBSOLETE_LINKER +PRCTL 
-PROCCTL +THREAD_DUMP -EVPORTS -OT +QUIC -PROMEX -MEMORY_PROFILING


Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=2).
Built with OpenSSL version : OpenSSL 3.0.3+quic 3 May 2022
Running on OpenSSL version : OpenSSL 3.0.3+quic 3 May 2022
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
OpenSSL providers loaded : default
Built with network namespace support.
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), 
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")

Support for malloc_trim() is enabled.
Built with transparent proxy support using: IP_TRANSPARENT 
IPV6_TRANSPARENT IP_FREEBIND

Built with PCRE2 version : 10.34 2019-11-21
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 9.4.0

Available polling systems :
  epoll : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as  cannot be specified using 'proto' keyword)
   quic : mode=HTTP  side=FE mux=QUIC flags=HTX|NO_UPG|FRAMED
 h2 : mode=HTTP  side=FE|BE  mux=H2 flags=HTX|HOL_RISK|NO_UPG
   fcgi : mode=HTTP  side=BE mux=FCGI flags=HTX|HOL_RISK|NO_UPG
   : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
 h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
   : mode=TCP   side=FE|BE  mux=PASS  flags=
   none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG

Available services : none

Available filters :
    [CACHE] cache
    [COMP] compression
    [FCGI] fcgi-app
    [SPOE] spoe
    [TRACE] trace

Re: Thoughts on QUIC/HTTP3

[Shawn Heisey]

On 4/25/22 10:55, Shawn Heisey wrote:
I was testing with the master branch from 
https://github.com/haproxy/haproxy.git. Just pulled down the latest 
changes, built it, and installed it.  Now I am sometimes seeing 
different behavior on the large POST.  It will load a page quickly 
sometimes, returning to the same page with blank fields, just as it 
would when first going there.  Another time, apache returned a 504 
error, which is very weird.  When haproxy got the 504, it sent its own 
error page. 


I did a build and install this morning, a bunch of quic-related changes 
in that.  Now everything seems to be working on my paste site.  Large 
pastes work, and I can reload the page a ton of times without it hanging 
until browser restart.


I changed the URL of my paste website, and now that everything seems to 
be working with http3, it's still using http3:


https://stikked.elyograg.org/

Thanks,
Shawn

Re: Thoughts on QUIC/HTTP3

[Shawn Heisey]

On 4/25/22 08:13, Amaury Denoyelle wrote:

I would not put too much faith in it for the near future. The OpenSSL
team seems to have put aside a simple QUIC API integration in favor of a
brand new full QUIC stack, which should take quite some time. So for
now, manually rebuilding your SSL library seems the only way to go


Ah.  Not Invented Here syndrome?


We already experienced this kind of problems but we though we have fixed
them. It seems the connection closure it not always properly handle on
haproxy side, which left the client with no notification that it should
open a new connection. It may help to increase the timeout client to be
greater than the default QUIC idle timeout, which is hardcoded to 10s on
haproxy side. For haproxy.org, we use a value of 1m and it seems to be
working. Please tell me if this makes your problem disappears or not.


My client timeout is set to 15 seconds in the "defaults" section, which 
is larger than that hardcoded default.  Should I go with something higher?



We did not have enough time to work on POST so there is still bugs in
this part. I just recently fixed two bugs which may be enough to fix
your situattion with the latest 2.6-dev7. However, I still have issues
when I use large payloads.

Thanks for your kind compliment for haproxy reliability. We hope one day
we will reach this level for QUIC but for now this objectif is still
far.


I was testing with the master branch from 
https://github.com/haproxy/haproxy.git.  Just pulled down the latest 
changes, built it, and installed it.  Now I am sometimes seeing 
different behavior on the large POST.  It will load a page quickly 
sometimes, returning to the same page with blank fields, just as it 
would when first going there.  Another time, apache returned a 504 
error, which is very weird.  When haproxy got the 504, it sent its own 
error page.


If there is anything I can do to help troubleshoot, give me instructions 
on how to do it.


Thanks,
Shawn

Re: Thoughts on QUIC/HTTP3

[Amaury Denoyelle]

Hi,

Sorry for not having answers your last mails, I was busy with some
new improvment for QUIC on haproxy side. Most notably, we improved the
transfer performance slightly by being able to use multiple buffers per
streams. Please find my answers to your remarks below,

On Sat, Apr 23, 2022 at 11:13:01PM -0600, Shawn Heisey wrote:
> After seeing http/3 (orange lightning bolt with the HTTP Version Indicator
> extension) talking to a lot of websites, I had thought the standard was
> further along than it is.  I see that the openssl team is discussing it, and
> plans to fully embrace it, but hasn't actually started putting QUIC code in
> openssl yet, and it may be quite some time before something usable shows up
> even in their master branch.

I would not put too much faith in it for the near future. The OpenSSL
team seems to have put aside a simple QUIC API integration in favor of a
brand new full QUIC stack, which should take quite some time. So for
now, manually rebuilding your SSL library seems the only way to go.

> It's been fun fiddling with it using haproxy with quictls, and I hope I can
> provide useful information to stabilize the implementation.
> I'd like to say thank you to Willy and all the other people who make haproxy
> one of the best things in my problem-solving arsenal.  It handles the
> internet side of all my web deployments.  I haven't yet put other services
> behind it.  At a previous $DAYJOB I had been testing FTP load-balancing,
> which I did get working, but didn't actually get to the deployment stage.
> At the moment, I am experiencing two problems with http3.  The second
> problem might actually just be another instance of the first problem.
> First problem:  If I do enough fiddling with an HTTP3 page, in either
> Firefox or Chrome, eventually that page will stop loading and the only way
> I've found to fix it is to completely close the browser and reopen it. 
> Restarting haproxy or Apache doesn't seem to help.

We already experienced this kind of problems but we though we have fixed
them. It seems the connection closure it not always properly handle on
haproxy side, which left the client with no notification that it should
open a new connection. It may help to increase the timeout client to be
greater than the default QUIC idle timeout, which is hardcoded to 10s on
haproxy side. For haproxy.org, we use a value of 1m and it seems to be
working. Please tell me if this makes your problem disappears or not.

> Second problem:  If I try pasting a REALLY large block of text into my paste
> website at the following URL while I have it configured to use HTTP/3, it
> won't work.  The page never loads. I can't tell if this is a separate
> problem from the one above, or just another occurrence of it that triggers
> more readily because there is more data transferred.  The reason I think it
> might be actually the first problem is that if I open another tab, I can't
> get to the website ... but if I close the browser and reopen it, then I can
> get to the website again.
> https://paste.elyograg.org/
> If I remove the paste website from the http3 ACL so it doesn't send the
> alt-svc header, then everything works once I can convince the browser to
> stop using HTTP/3.

We did not have enough time to work on POST so there is still bugs in
this part. I just recently fixed two bugs which may be enough to fix
your situattion with the latest 2.6-dev7. However, I still have issues
when I use large payloads.

Thanks for your kind compliment for haproxy reliability. We hope one day
we will reach this level for QUIC but for now this objectif is still
far.

-- 
Amaury Denoyelle