Re: running h2spec in CI ?
On Wed, Mar 25, 2020 at 12:49:29AM +0500, ??? wrote: > Hello, > > here's patch. Applied now, thanks Ilya! Willy
Re: running h2spec in CI ?
Hello, here's patch. вт, 24 мар. 2020 г. в 23:51, Willy Tarreau : > On Tue, Mar 24, 2020 at 11:49:11PM +0500, ??? wrote: > > > In fact it's very difficult to make such a file load and work > properly, it > > > depends on the headers appended etc. > > > > > > > initially, I expected that after WARNING (and truncation), file will be > > loaded :) > > but I was wrong. > > I could be wrong but my suspicion is that it was indeed truncated, but > the truncation was still too large and made it fail later. But that's > pure speculation. > > > ok, so shall we add it as weekly Github Action job ? I do not think we > have > > to run it on every push > > That's something we can do, probably, yes. > > Willy > From 96e726d4e0c3c1bb6281b9fe1b85049c5d878cb6 Mon Sep 17 00:00:00 2001 From: Ilya Shipitsin Date: Wed, 25 Mar 2020 00:46:14 +0500 Subject: [PATCH] CI: github actions: add weekly h2spec test ML link: https://www.mail-archive.com/haproxy@formilux.org/msg36753.html this commit adds scheduled run of h2spec tool to test http2 and HPACK compliance. h2spec might be found at: https://github.com/summerwind/h2spec --- .github/errorfile| 209 +++ .github/h2spec.config| 30 + .github/workflows/h2spec.yml | 27 + 3 files changed, 266 insertions(+) create mode 100644 .github/errorfile create mode 100644 .github/h2spec.config create mode 100644 .github/workflows/h2spec.yml diff --git a/.github/errorfile b/.github/errorfile new file mode 100644 index 0..f15d8e0d0 --- /dev/null +++ b/.github/errorfile @@ -0,0 +1,209 @@ +HTTP/1.0 200 OK +Cache-Control: no-cache +Connection: close +Content-Type: text/html + + + +b1z6lx9BLl3rSuonLqkIJAn9k9Hsah5qGfx9aq3qWSw6Nn37AZBJ1lxI0UyI7zvjXIEjSEVdCS4U +k6rTW/LndurrbPieC6OcEBPMjzGtsfpR9IsZ3QH6/mtBGnvAtbhxAfhMZ/QQXkqfv0JPjuLdXBdM +Z9cInHOr4ykoETgRbHaNt9ykBv32nIKt81YxLtTOMyyFAzH/AVSHUs6PanUKhxG11Csqn5RnlvSj +PBCaF0lJAGvndF/1PTSIhzjEtXR3ZUzCfO03/j0q0/4+cduV5jf3XNFeICjY19OHKSMIWVN0XVht +bY2eSMG0LoL8TqWyv6VSnclsVM5S5LJe7prJtWFobEpU3AMZzzjMPsxDiyMGJhjbJa0TnsDGAwln +IVO5n56gtdUdhwWUnVn8ZYGZlFVjOQt++q6XL/Vhm+DFCArXZ6Xz8+mz1o109JpM28jHhxg6e7A1 +CF08n0mwN+adNFTi3Wg8D4RJOOQ90Q1bS/gmW7LtPjYxGuu8k27MjUspEHeeEr5rdAcBJbKiG8C9 +191DBDLxlJv/V4ZYG/FdGIqX/a2F7x03Uj7rsVnBPmOz7U0EbcHGcEEpZlSN9YLuUKvPXeZ8HWa9 +fbaGO39Yt+9DByPWC1Xyr65sPBH8eDURPdSDMh3Sr+16HH46anVI40tjK8NZC6jjFBQPfJBP6MVW +HZF1F48rZXxesnHLHaMESCvwTruBf5R4JjYB1gt1Vv76e4Pew1MTK3/1ooNPV5kvoBV5PSLkMDqx +XO6dxSC9Y3HzxhzkoRK56h7SWDbwxd5OmUHNvjm3k0QtVTAWsAEbJ5q/gp65ikG3+hGvp9xF80pU +C1dAxK9+MHg7Ya3UiV6G/dB9prc3v92lEVqtK5CNKzlFiWQHSF3H+sz4qPGlB2Ub8H0T59TSZcyu +oTFKi802BYc8UnPdUX+mf4Uda4Vad4dPE409UQ1XIEqI+2pCTgOCUm80xM2Hgyjpp8bi1mnv6rI1 +8jafv0e6S23Meb9d93E/MLm82KWfXHIjPkDFaTouGS78IJie3giG68U270AL1+gpUwNunW+Ez0Ch +AwKUOM5BUL9pFfRMeDshy8KfiDGr7enLupqa2xe65Hbo47eioZZfIb0AD3P7yzlciIUXsy5JAwCF +B+L0T+XuRUJuXCaJ+ioDmgW0PenJ6xfL/+BuJ9yVrMGYb0paL/cD7VSVDda1L7+VSbLW7sQ6BOHM +ZgFV6+O81p48hGDquMb9+eURGrFKhQFipUPEi5sTQ7ocmyRZIAI3VeEOdBsX6zuwR9a2L5aV4yZc +HoiQikOgAeF1O8FNoVBIKh6TvFIzG+JFxb64pfWiwku+njgQu/xXkhuDSnYh/tzzqwghzmKHpQQl +7fbxF7jBihOr4qTcD/fUNPNGZYAZnZk/+wuA/6NOqwJl7nV2E7Ht7N13E6RZqGzfcL8KWldZbuFL +cFUVZ7sxogftmAWhSrQ1Io4IcAqt19XL5uGFlAiELphh5v+3mWKVNh5kOaCIDcoOggaerDZMl05F +C/d5veJxVLFBsSKFfdADmGh/8g85JDQC1UJuHYXbmPKuQ3pzUZRg6a3JYoi9ssLz7GijrSmmgkXs +71+8TsvHFP193euCd9N981+Gp4NMxpVvWkrYFsjkdtkzSQBda4dUlJ/QhbbHoRAuzNl2zDkUU7SA +P7LCNw3SKJlCQlwDEtqNYuO6jiQBZcUnajdk/UKwVwiH/p6q8rg8bh+NPV04hTZraUoaKumsPG+5 +tCBRj/WxPDKWfLjpgLx2gJPU4SKJrKwbfSot+VVO0Tc9viV8Jl0PPOkcW3ixx3Hc9WEqj0QZYHsB +kUk4E/q8N5WDnvmCzp3t6tlpeNqqkXhvNOAg0XxVXmKE3pWEytX1iMdggdjnoo9dLLGcNwW5+tnw +XdAlEVuqcvTeKJfYT/RMxpfdB7bXsg6OGEokMEkZHltLPmlYkB9i+J6zpWgo0WCwjawocVc7Y9Lj +FfSezs/Fs2s8OJhFlrHQzh3SwZoyAXHgOPC1wJDanMZWjLASi6W7ds2H2FHuyKYfx/gJb02+d199 +1ac7QG3Vgi0QiNB+6D8vuj4r05jQgHj7REIvFwbvJX/eVCY2kle+nXjzOiTL9M4AoDdaW9Hfzoao +YnwcKslhmHhRl/Q+9jA0YX7TCHh/VcKg6+lao3ScQ5F+MjwZewK0lwOlE9Z7Oz5rDNwTdBe6LhwA +tTkeItrhm45IPdipFKRRcqY7OPV0GgHeqIg704hGnpzws0cJi++lpLi2c+387h28ymvUXArndPtF +at7mboqJ7XCi2mYBOa73e7Q58R5UBhNzZB+M+SbNM3Xi5hcbXMH1UtnHGx8E8uNS5oXQvm4Cm4k0 +v1g0jU5xxU3m8j6ze99Z6sFZ3EJ7IrIdIkKHl0jkr+WZww64BEKmNsJfh0nWO+5Bm50ZK+sNkvDg +PtjkehxsWaaERJ8aQeqzIVQTK81m6FqHYdcSsuxMiY2ZAQSnVRarmSJqyd2oPy5vEkCnS9yd2ha9 +bYqAREVHUEy//dw/XJAtttnZSgwAKdn+SRSQuDiWZ9GPs0k/zKuohKkSXkHPlhIDuer+lJ1Hs17m +r0JTCt2LQVXLdCbKScAHOm4wdGeeIyMsV8MJv/PIWoySW8PIm3IjjzFinphnj6COvvVJUYg6zvPb +1WN7ZU0UyI0nFklUVguc6RF2ByO9ZzZA7nmVXlFawnDc5UkotXSGYZJaiV9c44Mvqg8CgxbfLLk+ +OCOJgQF9xIEk0bUp/QAfKj6o5aP2qHr+YvKxxTxlEthlxdGyM+F8YX4WpR6wf2mbFjc6jWC67xk3 +F7zfTdXtmezimB6vUkRAf4P5yS8J6Q7m7JCE/V0CN0Z6fUG4Z/8cGxpVQwqD44McT257MqSdiwrp +C9NiXxqWiLXcj0NbUI0rxAlzSwzuFAjMdLpPOm3zjm6I3SltMKn9BPSOyz5Q4wclInCx6yvZAqK3 +r95cvb72qVEt7YJKaM3b6Vb0oQRMpSWyoYHZQ75WTwcwd8PRecAXGPqgn0e0GYUSfRqiBi5Z3tUp +eljOJvV9ujIs2rFeySLKVfkCfHvcCRVyFZwsiUO0W9NvvIy40lkHtFshFANYlDkJOznhLVSlUGNW +lNwSTjEuG/bcGiiAm4NogFSmu6ijWrrJZbFAjH2CSKkKJUxCpAeasm5nDBqXY6fmS56WLv+mZmlJ
Re: running h2spec in CI ?
On Tue, Mar 24, 2020 at 11:49:11PM +0500, ??? wrote: > > In fact it's very difficult to make such a file load and work properly, it > > depends on the headers appended etc. > > > > initially, I expected that after WARNING (and truncation), file will be > loaded :) > but I was wrong. I could be wrong but my suspicion is that it was indeed truncated, but the truncation was still too large and made it fail later. But that's pure speculation. > ok, so shall we add it as weekly Github Action job ? I do not think we have > to run it on every push That's something we can do, probably, yes. Willy
Re: running h2spec in CI ?
вт, 24 мар. 2020 г. в 23:38, Willy Tarreau : > On Tue, Mar 24, 2020 at 11:01:13PM +0500, ??? wrote: > > I created 16kb response file, and found something that looks like an > error > > > > Run ./haproxy -f .github/h2spec.config -D > > [WARNING] 083/174625 (3194) : custom error message file > '.github/errorfile' > > larger than 16384 bytes. Truncating. > > [ALERT] 083/174625 (3194) : parsing [.github/h2spec.config:29] : > errorfile > > : unable to convert custom error message file '.github/errorfile' in HTX. > > [ALERT] 083/174625 (3194) : Error(s) found in configuration file : > > .github/h2spec.config > > [ALERT] 083/174625 (3194) : Fatal errors found in configuration. > > It's normal if the file is too large. > > > since, I see "WARNING", i.e. "no worry, I'll truncate file". but file is > > not loaded :-) > > In fact it's very difficult to make such a file load and work properly, it > depends on the headers appended etc. > initially, I expected that after WARNING (and truncation), file will be loaded :) but I was wrong. > > > I reduced file size a bit, 146/146 tests > > Excellent! > ok, so shall we add it as weekly Github Action job ? I do not think we have to run it on every push > > Willy >
Re: running h2spec in CI ?
On Tue, Mar 24, 2020 at 11:01:13PM +0500, ??? wrote: > I created 16kb response file, and found something that looks like an error > > Run ./haproxy -f .github/h2spec.config -D > [WARNING] 083/174625 (3194) : custom error message file '.github/errorfile' > larger than 16384 bytes. Truncating. > [ALERT] 083/174625 (3194) : parsing [.github/h2spec.config:29] : errorfile > : unable to convert custom error message file '.github/errorfile' in HTX. > [ALERT] 083/174625 (3194) : Error(s) found in configuration file : > .github/h2spec.config > [ALERT] 083/174625 (3194) : Fatal errors found in configuration. It's normal if the file is too large. > since, I see "WARNING", i.e. "no worry, I'll truncate file". but file is > not loaded :-) In fact it's very difficult to make such a file load and work properly, it depends on the headers appended etc. > I reduced file size a bit, 146/146 tests Excellent! Willy
Re: running h2spec in CI ?
сб, 21 мар. 2020 г. в 15:23, Willy Tarreau : > On Sat, Mar 21, 2020 at 03:13:07PM +0500, ??? wrote: > > > Really you *need* to have some response data, not just an empty 200. > > > Probably that you can do it with something like this to build 10kB of > > > garbage: > > > > > > > am I correct, dealing with large blocks is something HPACK related ? > > No it's unrelated. HPACK only deals with headers encoding. Here it's more > about making sure there are still bytes on the wire to be sent when a > window > is reopening after a WINDOW_UPDATE etc. > I created 16kb response file, and found something that looks like an error Run ./haproxy -f .github/h2spec.config -D [WARNING] 083/174625 (3194) : custom error message file '.github/errorfile' larger than 16384 bytes. Truncating. [ALERT] 083/174625 (3194) : parsing [.github/h2spec.config:29] : errorfile : unable to convert custom error message file '.github/errorfile' in HTX. [ALERT] 083/174625 (3194) : Error(s) found in configuration file : .github/h2spec.config [ALERT] 083/174625 (3194) : Fatal errors found in configuration. since, I see "WARNING", i.e. "no worry, I'll truncate file". but file is not loaded :-) I reduced file size a bit, 146/146 tests https://github.com/chipitsine/haproxy/runs/531318490 no valgrind yet, but we can add it later. > > > so, for example, if we decide, we can split into several steps, like > http2, > > HPACK, ... > > Quite frankly given that a generic config works fine there's no point in > having distinct setups for all the tests, it would be a burden to maintain > and would cause extra confusion. > > > h2spec can report in junit format. but no CI can import it (well, > gitlab-ci > > can, but I did not try). > > I'd actually prefer to see history test by test (and for reg-tests as > well) > > By default in verbose more you have all the output and a summary of failed > tests at the end, which is very easy to read. > > > the most we can do (and it is relatively cheap) is to define several > steps > > in github actions. > > like, I already done for "build" and "download h2spec", we can define > > several steps for h2spec itself. > > I think it's asking for more difficulties and pain than really needed. > The normal case is that h2spec should work fine so we don't need to have > details about what fails, if it fails we can check the output and see what > test failed. You'll never know why anyway, it will always require manual > attempts to reproduce. > > Just my two cents, > Willy >
Re: running h2spec in CI ?
On Sat, Mar 21, 2020 at 03:13:07PM +0500, ??? wrote: > > Really you *need* to have some response data, not just an empty 200. > > Probably that you can do it with something like this to build 10kB of > > garbage: > > > > am I correct, dealing with large blocks is something HPACK related ? No it's unrelated. HPACK only deals with headers encoding. Here it's more about making sure there are still bytes on the wire to be sent when a window is reopening after a WINDOW_UPDATE etc. > so, for example, if we decide, we can split into several steps, like http2, > HPACK, ... Quite frankly given that a generic config works fine there's no point in having distinct setups for all the tests, it would be a burden to maintain and would cause extra confusion. > h2spec can report in junit format. but no CI can import it (well, gitlab-ci > can, but I did not try). > I'd actually prefer to see history test by test (and for reg-tests as well) By default in verbose more you have all the output and a summary of failed tests at the end, which is very easy to read. > the most we can do (and it is relatively cheap) is to define several steps > in github actions. > like, I already done for "build" and "download h2spec", we can define > several steps for h2spec itself. I think it's asking for more difficulties and pain than really needed. The normal case is that h2spec should work fine so we don't need to have details about what fails, if it fails we can check the output and see what test failed. You'll never know why anyway, it will always require manual attempts to reproduce. Just my two cents, Willy
Re: running h2spec in CI ?
сб, 21 мар. 2020 г. в 14:59, Willy Tarreau : > On Sat, Mar 21, 2020 at 02:51:16PM +0500, ??? wrote: > > ??, 21 ???. 2020 ?. ? 14:29, Willy Tarreau : > > > > > On Sat, Mar 21, 2020 at 02:04:04PM +0500, ??? wrote: > > > > how do you call h2spec ? > > > > > > Like this (for example): > > > > > > $ h2spec -Svtk -h 127.0.0.1 -p 4443 > > > > > > > indeed. 146/146 > > > > https://github.com/chipitsine/haproxy/runs/523821049 > > > > ok, I'll try to run it multiple times to figure out how stable it is > > Really you *need* to have some response data, not just an empty 200. > Probably that you can do it with something like this to build 10kB of > garbage: > am I correct, dealing with large blocks is something HPACK related ? so, for example, if we decide, we can split into several steps, like http2, HPACK, ... h2spec can report in junit format. but no CI can import it (well, gitlab-ci can, but I did not try). I'd actually prefer to see history test by test (and for reg-tests as well) the most we can do (and it is relatively cheap) is to define several steps in github actions. like, I already done for "build" and "download h2spec", we can define several steps for h2spec itself. ok, I'll try it. > ># 40B >http-request set-var(req.v) > str(0123456789.123456789.123456789.123456789) ># 160B >http-request set-var(req.v) > var(req.v),concat(,req.v),concat(,req.v),concat(,req.v) ># 640B >http-request set-var(req.v) > var(req.v),concat(,req.v),concat(,req.v),concat(,req.v) ># 2560B >http-request set-var(req.v) > var(req.v),concat(,req.v),concat(,req.v),concat(,req.v) ># 10240B >http-request set-var(req.v) > var(req.v),concat(,req.v),concat(,req.v),concat(,req.v) >http-request return status 200 content-type text/plain lf-string > %[var(req.v)] > > Note that I didn't test it, and might have written a bit of crap, but you > get the idea. > > Willy >
Re: running h2spec in CI ?
On Sat, Mar 21, 2020 at 02:51:16PM +0500, ??? wrote: > ??, 21 ???. 2020 ?. ? 14:29, Willy Tarreau : > > > On Sat, Mar 21, 2020 at 02:04:04PM +0500, ??? wrote: > > > how do you call h2spec ? > > > > Like this (for example): > > > > $ h2spec -Svtk -h 127.0.0.1 -p 4443 > > > > indeed. 146/146 > > https://github.com/chipitsine/haproxy/runs/523821049 > > ok, I'll try to run it multiple times to figure out how stable it is Really you *need* to have some response data, not just an empty 200. Probably that you can do it with something like this to build 10kB of garbage: # 40B http-request set-var(req.v) str(0123456789.123456789.123456789.123456789) # 160B http-request set-var(req.v) var(req.v),concat(,req.v),concat(,req.v),concat(,req.v) # 640B http-request set-var(req.v) var(req.v),concat(,req.v),concat(,req.v),concat(,req.v) # 2560B http-request set-var(req.v) var(req.v),concat(,req.v),concat(,req.v),concat(,req.v) # 10240B http-request set-var(req.v) var(req.v),concat(,req.v),concat(,req.v),concat(,req.v) http-request return status 200 content-type text/plain lf-string %[var(req.v)] Note that I didn't test it, and might have written a bit of crap, but you get the idea. Willy
Re: running h2spec in CI ?
сб, 21 мар. 2020 г. в 14:29, Willy Tarreau : > On Sat, Mar 21, 2020 at 02:04:04PM +0500, ??? wrote: > > how do you call h2spec ? > > Like this (for example): > > $ h2spec -Svtk -h 127.0.0.1 -p 4443 > indeed. 146/146 https://github.com/chipitsine/haproxy/runs/523821049 ok, I'll try to run it multiple times to figure out how stable it is > > > in my case, I'm getting 95/95 tests. > > OK you only have the H2 tests, it doesn't test HPACK nor the generic > part, you just have not to pass "http2" and it will run everything. I > don't exactly remember the difference between http2 and generic though, > I think only one of them checks the framing. > > > of course we do not have to run all tests. we can run just specific > > We could indeed drop unreliable ones but I'd rather make sure that we're > in good condition to trust them. > > Willy >
Re: running h2spec in CI ?
On Sat, Mar 21, 2020 at 02:04:04PM +0500, ??? wrote: > how do you call h2spec ? Like this (for example): $ h2spec -Svtk -h 127.0.0.1 -p 4443 > in my case, I'm getting 95/95 tests. OK you only have the H2 tests, it doesn't test HPACK nor the generic part, you just have not to pass "http2" and it will run everything. I don't exactly remember the difference between http2 and generic though, I think only one of them checks the framing. > of course we do not have to run all tests. we can run just specific We could indeed drop unreliable ones but I'd rather make sure that we're in good condition to trust them. Willy
Re: running h2spec in CI ?
сб, 21 мар. 2020 г. в 14:00, Willy Tarreau : > Hi Ilya, > > On Sat, Mar 21, 2020 at 01:44:40AM +0500, ??? wrote: > > Hello, > > > > I played with "special purpose" job, which runs h2spec > > > > here's code: > > > > > https://github.com/chipitsine/haproxy/commit/8c90ea82fd32c0ca9bd3df0ae7d9361525eda590 > > > > output: > > > > https://github.com/chipitsine/haproxy/runs/522959386 > > > > > > I think such jobs might be run on schedule, for example weekly ? > > I'm hesitating. While h2spec is a fantastic tool to detect some breakage, > it also relies on extremely precise behaviors and timing. Typically I > never managed to get it to work reproducibly by sending less than 8kB or > so of response data. This is related to the fact that it will for example > send an RST_STREAM after a request and will check if some data flow back, > which will essentially depend on the bytes in flight (hence bandwidth times > latency) between h2spec, haproxy and the server. That's typically what > makes > the success rate vary from 141 to 146 tests for me. > how do you call h2spec ? in my case, I'm getting 95/95 tests. of course we do not have to run all tests. we can run just specific > > Now that we have the return directive we could imagine creating a second > layer and returning a large response there. But as you see I'm slightly > worried of having to deal with even more false positives while we haven't > still completely addressed the abns+reload randomness :-/ > > What's others' opinion on this ? > > Thanks, > Willy >
Re: running h2spec in CI ?
Hi Ilya, On Sat, Mar 21, 2020 at 01:44:40AM +0500, ??? wrote: > Hello, > > I played with "special purpose" job, which runs h2spec > > here's code: > > https://github.com/chipitsine/haproxy/commit/8c90ea82fd32c0ca9bd3df0ae7d9361525eda590 > > output: > > https://github.com/chipitsine/haproxy/runs/522959386 > > > I think such jobs might be run on schedule, for example weekly ? I'm hesitating. While h2spec is a fantastic tool to detect some breakage, it also relies on extremely precise behaviors and timing. Typically I never managed to get it to work reproducibly by sending less than 8kB or so of response data. This is related to the fact that it will for example send an RST_STREAM after a request and will check if some data flow back, which will essentially depend on the bytes in flight (hence bandwidth times latency) between h2spec, haproxy and the server. That's typically what makes the success rate vary from 141 to 146 tests for me. Now that we have the return directive we could imagine creating a second layer and returning a large response there. But as you see I'm slightly worried of having to deal with even more false positives while we haven't still completely addressed the abns+reload randomness :-/ What's others' opinion on this ? Thanks, Willy
running h2spec in CI ?
Hello, I played with "special purpose" job, which runs h2spec here's code: https://github.com/chipitsine/haproxy/commit/8c90ea82fd32c0ca9bd3df0ae7d9361525eda590 output: https://github.com/chipitsine/haproxy/runs/522959386 I think such jobs might be run on schedule, for example weekly ? cheers, Ilya Shipitcin
