Hi ML,
Here are two patches for haproxy reg testing.
Note that we have recently added an new feature to varnishtest so that
to send commands to the CLI without running a shell, socat etc
(https://varnish-cache.org/docs/trunk/reference/vtc.html#haproxy). This
breaks reg-tests/spoe/h00000.vtc reg test case (fixed by the first patch).
We can send commands to the CLI as follows:
haproxy h1 -conf {...}
haproxy h1 -cli {
send "show info"
expect ~ "something to expect"
} -wait
With the 2nd patch we changed the prefix of the VTC files from 'h'
letter to 'b' (as "bug") and added a new LEVEL 4 to run all the VTC
files prefixed with 'b' letter. This VTC files are in relation with bugs
they help to reproduce (and prevent from coming back).
So, for now on, if you run reg-tests, as there is no more LEVEL 1
(default LEVEL) VTC files, no VTC file will be run.
To run the LEVEL 4 (VTC files for already fixed bugs) you will have to
use such a command:
$
VARNISHTEST_PROGRAM=~/src/varnish-cache-trunk/bin/varnishtest/varnishtest
LEVEL=4 make reg-tests
Fred.
>From 35486065bf63a47d17ba65b94d7b42b08d958abb Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Fr=C3=A9d=C3=A9ric=20L=C3=A9caille?= <flecai...@haproxy.com>
Date: Wed, 22 Aug 2018 10:41:33 +0200
Subject: [PATCH 2/2] REGTEST/MINOR: Add a new class of regression testing
files.
Add LEVEL #4 regression testing files which is dedicated to
VTC files in relation with bugs they help to reproduce.
At the date of this commit, all VTC files are LEVEL 4 VTC files.
---
Makefile | 7 +++++++
reg-tests/log/{h00000.vtc => b00000.vtc} | 0
reg-tests/lua/{h00000.lua => b00000.lua} | 0
reg-tests/lua/{h00000.vtc => b00000.vtc} | 2 +-
reg-tests/seamless-reload/{h00000.vtc => b00000.vtc} | 0
reg-tests/spoe/{h00000.vtc => b00000.vtc} | 0
reg-tests/ssl/{h00000.vtc => b00000.vtc} | 0
reg-tests/stick-table/{h00000.vtc => b00000.vtc} | 0
8 files changed, 8 insertions(+), 1 deletion(-)
rename reg-tests/log/{h00000.vtc => b00000.vtc} (100%)
rename reg-tests/lua/{h00000.lua => b00000.lua} (100%)
rename reg-tests/lua/{h00000.vtc => b00000.vtc} (97%)
rename reg-tests/seamless-reload/{h00000.vtc => b00000.vtc} (100%)
rename reg-tests/spoe/{h00000.vtc => b00000.vtc} (100%)
rename reg-tests/ssl/{h00000.vtc => b00000.vtc} (100%)
rename reg-tests/stick-table/{h00000.vtc => b00000.vtc} (100%)
diff --git a/Makefile b/Makefile
index 817161f7..c4923995 100644
--- a/Makefile
+++ b/Makefile
@@ -999,6 +999,11 @@ update-version:
echo "$(SUBVERS)" > SUBVERS
echo "$(VERDATE)" > VERDATE
+# Target to run the regression testing script files.
+# LEVEL 1 scripts are dedicated to pure haproxy compliance tests (prefixed with 'h' letter).
+# LEVEL 2 scripts are slow scripts (prefixed with 's' letter).
+# LEVEL 3 scripts are low interest scripts (prefixed with 'l' letter).
+# LEVEL 4 scripts are in relation with bugs they help to reproduce (prefixed with 'b' letter).
reg-tests:
@if [ ! -x "$(VARNISHTEST_PROGRAM)" ]; then \
echo "Please make the VARNISHTEST_PROGRAM variable point to the location of the varnishtest program."; \
@@ -1011,6 +1016,8 @@ reg-tests:
EXPR='s*.vtc'; \
elif [ $$LEVEL = 3 ] ; then \
EXPR='l*.vtc'; \
+ elif [ $$LEVEL = 4 ] ; then \
+ EXPR='b*.vtc'; \
fi ; \
if [ -n "$$EXPR" ] ; then \
find reg-tests -type f -name "$$EXPR" -print0 | \
diff --git a/reg-tests/log/h00000.vtc b/reg-tests/log/b00000.vtc
similarity index 100%
rename from reg-tests/log/h00000.vtc
rename to reg-tests/log/b00000.vtc
diff --git a/reg-tests/lua/h00000.lua b/reg-tests/lua/b00000.lua
similarity index 100%
rename from reg-tests/lua/h00000.lua
rename to reg-tests/lua/b00000.lua
diff --git a/reg-tests/lua/h00000.vtc b/reg-tests/lua/b00000.vtc
similarity index 97%
rename from reg-tests/lua/h00000.vtc
rename to reg-tests/lua/b00000.vtc
index 2b2ffb0e..4229eeb0 100644
--- a/reg-tests/lua/h00000.vtc
+++ b/reg-tests/lua/b00000.vtc
@@ -40,7 +40,7 @@ server s1 -repeat 2 {
haproxy h1 -conf {
global
- lua-load ${testdir}/h00000.lua
+ lua-load ${testdir}/b00000.lua
frontend fe1
mode http
diff --git a/reg-tests/seamless-reload/h00000.vtc b/reg-tests/seamless-reload/b00000.vtc
similarity index 100%
rename from reg-tests/seamless-reload/h00000.vtc
rename to reg-tests/seamless-reload/b00000.vtc
diff --git a/reg-tests/spoe/h00000.vtc b/reg-tests/spoe/b00000.vtc
similarity index 100%
rename from reg-tests/spoe/h00000.vtc
rename to reg-tests/spoe/b00000.vtc
diff --git a/reg-tests/ssl/h00000.vtc b/reg-tests/ssl/b00000.vtc
similarity index 100%
rename from reg-tests/ssl/h00000.vtc
rename to reg-tests/ssl/b00000.vtc
diff --git a/reg-tests/stick-table/h00000.vtc b/reg-tests/stick-table/b00000.vtc
similarity index 100%
rename from reg-tests/stick-table/h00000.vtc
rename to reg-tests/stick-table/b00000.vtc
--
2.11.0
>From 971de0c29c683d629eae0f3a2e52715933f18463 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Fr=C3=A9d=C3=A9ric=20L=C3=A9caille?= <flecai...@haproxy.com>
Date: Tue, 21 Aug 2018 15:29:24 +0200
Subject: [PATCH 1/2] REGTEST/MINOR: Missing mandatory "ignore_unknown_macro".
Since bbc34e2 varnish commit (for varnishtest), a new "cli"
macro is automatically created for each VTC script to dialog with
the CLI. Consequently, as this macro is unknown from higher level
code for varnishtest, it makes the scripts fail if we
we do not ask varnishtest to disregard the unknown macros.
To prevent this, from now on, for each VTC file for haproxy we MUST add
"feature ignore_unknown_macro" line to do so. This is mandatory
---
doc/regression-testing.txt | 4 +++-
reg-tests/spoe/h00000.vtc | 2 ++
2 files changed, 5 insertions(+), 1 deletion(-)
diff --git a/doc/regression-testing.txt b/doc/regression-testing.txt
index 68845997..1b8cdb22 100644
--- a/doc/regression-testing.txt
+++ b/doc/regression-testing.txt
@@ -64,7 +64,9 @@ The VTC files for haproxy must also contain a "feature ignore_unknown_macro" lin
if any macro is used for haproxy in this file. This is due to the fact that
varnishtest parser code for haproxy commands generates macros the varnishtest
parser code for varnish has no knowledge of. This line prevents varnishtest from
-failing in such cases.
+failing in such cases. Since bbc34e2 varnish commit, as a "cli" macro is
+automatically generated, this "feature ignore_unknown_macro" is mandatory
+for each VTC file for haproxy.
To make varnishtest capable of testing haproxy, two new VTC commands have been
implemented: "haproxy" and "syslog". "haproxy" is used to start haproxy processus.
diff --git a/reg-tests/spoe/h00000.vtc b/reg-tests/spoe/h00000.vtc
index b4ac8774..76b41d60 100644
--- a/reg-tests/spoe/h00000.vtc
+++ b/reg-tests/spoe/h00000.vtc
@@ -7,6 +7,8 @@
varnishtest "SPOE bug: missing configuration file"
+feature ignore_unknown_macro
+
haproxy h1 -conf-BAD {} {
defaults
timeout connect 5000ms
--
2.11.0
