Re: [ovs-dev] [PATCH ovn v2] ci: Remove '--recheck' in CI.
On 7/11/23 11:59, Dumitru Ceara wrote:
> On 7/11/23 10:30, Ales Musil wrote:
>> On Tue, Jul 11, 2023 at 7:14 AM Ales Musil wrote:
>>
>>>
>>>
>>> On Mon, Jul 10, 2023 at 5:26 PM Dumitru Ceara wrote:
>>>
If we want to catch new failures faster we have a better chance if CI
doesn't auto-retry (once).
There are some tests that are still "unstable" and fail every now and
then. In order to reduce the number of false negatives keep the
--recheck for them. To achieve that we use a new macro, TAG_UNSTABLE,
to tag all these tests. The list of "unstable" tests is compiled based
on the following discussion:
https://mail.openvswitch.org/pipermail/ovs-dev/2023-June/405465.html
In order to avoid new GitHub actions jobs, we re-purpose the last job of
each target type to also run the unstable tests. These jobs were
already running less tests than others so the additional run time should
not be an issue.
Signed-off-by: Dumitru Ceara
---
V2:
- Addressed Ales' comments:
- always run stable and unstable tests before declaring pass/fail
Changes in v1 (since RFC):
- kept recheck for unstable tests
- introduced TAG_UNSTABLE
- changed test.yml to run unstable tests in the last batch of every
test target type.
---
.ci/ci.sh | 2 +-
.ci/linux-build.sh | 76 ++
.github/workflows/test.yml | 15
tests/ovn-ic.at| 1 +
tests/ovn-ipsec.at | 1 +
tests/ovn-macros.at| 5 +++
tests/ovn-northd.at| 1 +
tests/ovn-performance.at | 1 +
tests/ovn.at | 13 +++
9 files changed, 92 insertions(+), 23 deletions(-)
diff --git a/.ci/ci.sh b/.ci/ci.sh
index 10f11939c5..a500aba764 100755
--- a/.ci/ci.sh
+++ b/.ci/ci.sh
@@ -101,7 +101,7 @@ function run_tests() {
&& \
ARCH=$ARCH CC=$CC LIBS=$LIBS OPTS=$OPTS TESTSUITE=$TESTSUITE \
TEST_RANGE=$TEST_RANGE SANITIZERS=$SANITIZERS DPDK=$DPDK \
-./.ci/linux-build.sh
+UNSTABLE=$UNSTABLE ./.ci/linux-build.sh
>>>
>> I've missed one thing, please add the recheck here as well.
>>
>>
>
> Good catch! I'll send a v3. I think I also need to add at least one
> more test to the list of unstable ones: "system-ovn.at:11332 Tiered ACLs
> -- ovn-northd -- parallelization=yes -- ovn_monitor_all=yes"
>
I posted v3:
https://patchwork.ozlabs.org/project/ovn/list/?series=363690
In the end I didn't add the "Tiered ACLs" test case to the list of
unstable ones because Xavier informed me privately that he's working on
a fix.
Regards,
Dumitru
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [PATCH ovn v2] ci: Remove '--recheck' in CI.
On 7/11/23 10:30, Ales Musil wrote:
> On Tue, Jul 11, 2023 at 7:14 AM Ales Musil wrote:
>
>>
>>
>> On Mon, Jul 10, 2023 at 5:26 PM Dumitru Ceara wrote:
>>
>>> If we want to catch new failures faster we have a better chance if CI
>>> doesn't auto-retry (once).
>>>
>>> There are some tests that are still "unstable" and fail every now and
>>> then. In order to reduce the number of false negatives keep the
>>> --recheck for them. To achieve that we use a new macro, TAG_UNSTABLE,
>>> to tag all these tests. The list of "unstable" tests is compiled based
>>> on the following discussion:
>>> https://mail.openvswitch.org/pipermail/ovs-dev/2023-June/405465.html
>>>
>>> In order to avoid new GitHub actions jobs, we re-purpose the last job of
>>> each target type to also run the unstable tests. These jobs were
>>> already running less tests than others so the additional run time should
>>> not be an issue.
>>>
>>> Signed-off-by: Dumitru Ceara
>>> ---
>>> V2:
>>> - Addressed Ales' comments:
>>> - always run stable and unstable tests before declaring pass/fail
>>> Changes in v1 (since RFC):
>>> - kept recheck for unstable tests
>>> - introduced TAG_UNSTABLE
>>> - changed test.yml to run unstable tests in the last batch of every
>>> test target type.
>>> ---
>>> .ci/ci.sh | 2 +-
>>> .ci/linux-build.sh | 76 ++
>>> .github/workflows/test.yml | 15
>>> tests/ovn-ic.at| 1 +
>>> tests/ovn-ipsec.at | 1 +
>>> tests/ovn-macros.at| 5 +++
>>> tests/ovn-northd.at| 1 +
>>> tests/ovn-performance.at | 1 +
>>> tests/ovn.at | 13 +++
>>> 9 files changed, 92 insertions(+), 23 deletions(-)
>>>
>>> diff --git a/.ci/ci.sh b/.ci/ci.sh
>>> index 10f11939c5..a500aba764 100755
>>> --- a/.ci/ci.sh
>>> +++ b/.ci/ci.sh
>>> @@ -101,7 +101,7 @@ function run_tests() {
>>> && \
>>> ARCH=$ARCH CC=$CC LIBS=$LIBS OPTS=$OPTS TESTSUITE=$TESTSUITE \
>>> TEST_RANGE=$TEST_RANGE SANITIZERS=$SANITIZERS DPDK=$DPDK \
>>> -./.ci/linux-build.sh
>>> +UNSTABLE=$UNSTABLE ./.ci/linux-build.sh
>>>
>>
> I've missed one thing, please add the recheck here as well.
>
>
Good catch! I'll send a v3. I think I also need to add at least one
more test to the list of unstable ones: "system-ovn.at:11332 Tiered ACLs
-- ovn-northd -- parallelization=yes -- ovn_monitor_all=yes"
Regards,
Dumitru
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [PATCH ovn v2] ci: Remove '--recheck' in CI.
On Tue, Jul 11, 2023 at 7:14 AM Ales Musil wrote:
>
>
> On Mon, Jul 10, 2023 at 5:26 PM Dumitru Ceara wrote:
>
>> If we want to catch new failures faster we have a better chance if CI
>> doesn't auto-retry (once).
>>
>> There are some tests that are still "unstable" and fail every now and
>> then. In order to reduce the number of false negatives keep the
>> --recheck for them. To achieve that we use a new macro, TAG_UNSTABLE,
>> to tag all these tests. The list of "unstable" tests is compiled based
>> on the following discussion:
>> https://mail.openvswitch.org/pipermail/ovs-dev/2023-June/405465.html
>>
>> In order to avoid new GitHub actions jobs, we re-purpose the last job of
>> each target type to also run the unstable tests. These jobs were
>> already running less tests than others so the additional run time should
>> not be an issue.
>>
>> Signed-off-by: Dumitru Ceara
>> ---
>> V2:
>> - Addressed Ales' comments:
>> - always run stable and unstable tests before declaring pass/fail
>> Changes in v1 (since RFC):
>> - kept recheck for unstable tests
>> - introduced TAG_UNSTABLE
>> - changed test.yml to run unstable tests in the last batch of every
>> test target type.
>> ---
>> .ci/ci.sh | 2 +-
>> .ci/linux-build.sh | 76 ++
>> .github/workflows/test.yml | 15
>> tests/ovn-ic.at| 1 +
>> tests/ovn-ipsec.at | 1 +
>> tests/ovn-macros.at| 5 +++
>> tests/ovn-northd.at| 1 +
>> tests/ovn-performance.at | 1 +
>> tests/ovn.at | 13 +++
>> 9 files changed, 92 insertions(+), 23 deletions(-)
>>
>> diff --git a/.ci/ci.sh b/.ci/ci.sh
>> index 10f11939c5..a500aba764 100755
>> --- a/.ci/ci.sh
>> +++ b/.ci/ci.sh
>> @@ -101,7 +101,7 @@ function run_tests() {
>> && \
>> ARCH=$ARCH CC=$CC LIBS=$LIBS OPTS=$OPTS TESTSUITE=$TESTSUITE \
>> TEST_RANGE=$TEST_RANGE SANITIZERS=$SANITIZERS DPDK=$DPDK \
>> -./.ci/linux-build.sh
>> +UNSTABLE=$UNSTABLE ./.ci/linux-build.sh
>>
>
I've missed one thing, please add the recheck here as well.
> "
>> }
>>
>> diff --git a/.ci/linux-build.sh b/.ci/linux-build.sh
>> index 5a79a52daf..4c5361f3ca 100755
>> --- a/.ci/linux-build.sh
>> +++ b/.ci/linux-build.sh
>> @@ -9,6 +9,7 @@ COMMON_CFLAGS=""
>> OVN_CFLAGS=""
>> OPTS="$OPTS --enable-Werror"
>> JOBS=${JOBS:-"-j4"}
>> +RECHECK=${RECHECK:-"no"}
>>
>> function install_dpdk()
>> {
>> @@ -99,6 +100,17 @@ function configure_clang()
>> COMMON_CFLAGS="${COMMON_CFLAGS}
>> -Wno-error=unused-command-line-argument"
>> }
>>
>> +function run_tests()
>> +{
>> +if ! make distcheck CFLAGS="${COMMON_CFLAGS} ${OVN_CFLAGS}" $JOBS \
>> +TESTSUITEFLAGS="$JOBS $TEST_RANGE" RECHECK=$RECHECK
>> +then
>> +# testsuite.log is necessary for debugging.
>> +cat */_build/sub/tests/testsuite.log
>> +return 1
>> +fi
>> +}
>> +
>> function execute_tests()
>> {
>> # 'distcheck' will reconfigure with required options.
>> @@ -106,27 +118,61 @@ function execute_tests()
>> configure_ovn
>>
>> export DISTCHECK_CONFIGURE_FLAGS="$OPTS"
>> -if ! make distcheck CFLAGS="${COMMON_CFLAGS} ${OVN_CFLAGS}" $JOBS \
>> -TESTSUITEFLAGS="$JOBS $TEST_RANGE" RECHECK=yes
>> -then
>> -# testsuite.log is necessary for debugging.
>> -cat */_build/sub/tests/testsuite.log
>> +
>> +local stable_rc=0
>> +local unstable_rc=0
>> +
>> +if ! SKIP_UNSTABLE=yes run_tests; then
>> +stable_rc=1
>> +fi
>> +
>> +if [ "$UNSTABLE" ]; then
>> +if ! SKIP_UNSTABLE=no TEST_RANGE="-k unstable" RECHECK=yes \
>> +run_tests; then
>> +unstable_rc=1
>> +fi
>> +fi
>> +
>> +if [[ $stable_rc -ne 0 ]] || [[ $unstable_rc -ne 0 ]]; then
>> exit 1
>> fi
>> }
>>
>> +function run_system_tests()
>> +{
>> +local type=$1
>> +local log_file=$2
>> +
>> +if ! sudo make $JOBS $type TESTSUITEFLAGS="$TEST_RANGE" \
>> +RECHECK=$RECHECK; then
>> +# $log_file is necessary for debugging.
>> +cat tests/$log_file
>> +return 1
>> +fi
>> +}
>> +
>> function execute_system_tests()
>> {
>> - type=$1
>> - log_file=$2
>> -
>> - configure_ovn $OPTS
>> - make $JOBS || { cat config.log; exit 1; }
>> - if ! sudo make $JOBS $type TESTSUITEFLAGS="$TEST_RANGE"
>> RECHECK=yes; then
>> - # $log_file is necessary for debugging.
>> - cat tests/$log_file
>> - exit 1
>> - fi
>> +configure_ovn $OPTS
>> +make $JOBS || { cat config.log; exit 1; }
>> +
>> +local stable_rc=0
>> +local unstable_rc=0
>> +
>> +if ! SKIP_UNSTABLE=yes run_system_tests $@; then
>> +stable_rc=1
>> +fi
>> +
>> +if [ "$UNSTABLE" ]; then
>> +if ! SKIP_UNSTABLE=no TEST_RANGE="-k unstable" RECHECK=yes \
>> +run_system_tests $@; then
>> +
Re: [ovs-dev] [PATCH ovn v2] ci: Remove '--recheck' in CI.
On Mon, Jul 10, 2023 at 5:26 PM Dumitru Ceara wrote:
> If we want to catch new failures faster we have a better chance if CI
> doesn't auto-retry (once).
>
> There are some tests that are still "unstable" and fail every now and
> then. In order to reduce the number of false negatives keep the
> --recheck for them. To achieve that we use a new macro, TAG_UNSTABLE,
> to tag all these tests. The list of "unstable" tests is compiled based
> on the following discussion:
> https://mail.openvswitch.org/pipermail/ovs-dev/2023-June/405465.html
>
> In order to avoid new GitHub actions jobs, we re-purpose the last job of
> each target type to also run the unstable tests. These jobs were
> already running less tests than others so the additional run time should
> not be an issue.
>
> Signed-off-by: Dumitru Ceara
> ---
> V2:
> - Addressed Ales' comments:
> - always run stable and unstable tests before declaring pass/fail
> Changes in v1 (since RFC):
> - kept recheck for unstable tests
> - introduced TAG_UNSTABLE
> - changed test.yml to run unstable tests in the last batch of every
> test target type.
> ---
> .ci/ci.sh | 2 +-
> .ci/linux-build.sh | 76 ++
> .github/workflows/test.yml | 15
> tests/ovn-ic.at| 1 +
> tests/ovn-ipsec.at | 1 +
> tests/ovn-macros.at| 5 +++
> tests/ovn-northd.at| 1 +
> tests/ovn-performance.at | 1 +
> tests/ovn.at | 13 +++
> 9 files changed, 92 insertions(+), 23 deletions(-)
>
> diff --git a/.ci/ci.sh b/.ci/ci.sh
> index 10f11939c5..a500aba764 100755
> --- a/.ci/ci.sh
> +++ b/.ci/ci.sh
> @@ -101,7 +101,7 @@ function run_tests() {
> && \
> ARCH=$ARCH CC=$CC LIBS=$LIBS OPTS=$OPTS TESTSUITE=$TESTSUITE \
> TEST_RANGE=$TEST_RANGE SANITIZERS=$SANITIZERS DPDK=$DPDK \
> -./.ci/linux-build.sh
> +UNSTABLE=$UNSTABLE ./.ci/linux-build.sh
> "
> }
>
> diff --git a/.ci/linux-build.sh b/.ci/linux-build.sh
> index 5a79a52daf..4c5361f3ca 100755
> --- a/.ci/linux-build.sh
> +++ b/.ci/linux-build.sh
> @@ -9,6 +9,7 @@ COMMON_CFLAGS=""
> OVN_CFLAGS=""
> OPTS="$OPTS --enable-Werror"
> JOBS=${JOBS:-"-j4"}
> +RECHECK=${RECHECK:-"no"}
>
> function install_dpdk()
> {
> @@ -99,6 +100,17 @@ function configure_clang()
> COMMON_CFLAGS="${COMMON_CFLAGS}
> -Wno-error=unused-command-line-argument"
> }
>
> +function run_tests()
> +{
> +if ! make distcheck CFLAGS="${COMMON_CFLAGS} ${OVN_CFLAGS}" $JOBS \
> +TESTSUITEFLAGS="$JOBS $TEST_RANGE" RECHECK=$RECHECK
> +then
> +# testsuite.log is necessary for debugging.
> +cat */_build/sub/tests/testsuite.log
> +return 1
> +fi
> +}
> +
> function execute_tests()
> {
> # 'distcheck' will reconfigure with required options.
> @@ -106,27 +118,61 @@ function execute_tests()
> configure_ovn
>
> export DISTCHECK_CONFIGURE_FLAGS="$OPTS"
> -if ! make distcheck CFLAGS="${COMMON_CFLAGS} ${OVN_CFLAGS}" $JOBS \
> -TESTSUITEFLAGS="$JOBS $TEST_RANGE" RECHECK=yes
> -then
> -# testsuite.log is necessary for debugging.
> -cat */_build/sub/tests/testsuite.log
> +
> +local stable_rc=0
> +local unstable_rc=0
> +
> +if ! SKIP_UNSTABLE=yes run_tests; then
> +stable_rc=1
> +fi
> +
> +if [ "$UNSTABLE" ]; then
> +if ! SKIP_UNSTABLE=no TEST_RANGE="-k unstable" RECHECK=yes \
> +run_tests; then
> +unstable_rc=1
> +fi
> +fi
> +
> +if [[ $stable_rc -ne 0 ]] || [[ $unstable_rc -ne 0 ]]; then
> exit 1
> fi
> }
>
> +function run_system_tests()
> +{
> +local type=$1
> +local log_file=$2
> +
> +if ! sudo make $JOBS $type TESTSUITEFLAGS="$TEST_RANGE" \
> +RECHECK=$RECHECK; then
> +# $log_file is necessary for debugging.
> +cat tests/$log_file
> +return 1
> +fi
> +}
> +
> function execute_system_tests()
> {
> - type=$1
> - log_file=$2
> -
> - configure_ovn $OPTS
> - make $JOBS || { cat config.log; exit 1; }
> - if ! sudo make $JOBS $type TESTSUITEFLAGS="$TEST_RANGE"
> RECHECK=yes; then
> - # $log_file is necessary for debugging.
> - cat tests/$log_file
> - exit 1
> - fi
> +configure_ovn $OPTS
> +make $JOBS || { cat config.log; exit 1; }
> +
> +local stable_rc=0
> +local unstable_rc=0
> +
> +if ! SKIP_UNSTABLE=yes run_system_tests $@; then
> +stable_rc=1
> +fi
> +
> +if [ "$UNSTABLE" ]; then
> +if ! SKIP_UNSTABLE=no TEST_RANGE="-k unstable" RECHECK=yes \
> +run_system_tests $@; then
> +unstable_rc=1
> +fi
> +fi
> +
> +if [[ $stable_rc -ne 0 ]] || [[ $unstable_rc -ne 0 ]]; then
> +exit 1
> +fi
> }
>
> configure_$CC
> diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
> index fe2a14c401..7d40251003 100644
>
Re: [ovs-dev] [PATCH ovn v2] ci: Remove '--recheck' in CI.
Bleep bloop. Greetings Dumitru Ceara, I am a robot and I have tried out your
patch.
Thanks for your contribution.
I encountered some error that I wasn't expecting. See the details below.
checkpatch:
WARNING: Line is 85 characters long (recommended limit is 79)
#171 FILE: .github/workflows/test.yml:108:
- { compiler: gcc, testsuite: test, test_range: "1001-", unstable:
unstable }
WARNING: Line is 111 characters long (recommended limit is 79)
#177 FILE: .github/workflows/test.yml:113:
- { compiler: clang, testsuite: test, sanitizers: sanitizers,
test_range: "1201-", unstable: unstable }
WARNING: Line is 103 characters long (recommended limit is 79)
#181 FILE: .github/workflows/test.yml:116:
- { compiler: gcc, testsuite: test, libs: -ljemalloc, test_range:
"1001-", unstable: unstable }
WARNING: Line is 108 characters long (recommended limit is 79)
#185 FILE: .github/workflows/test.yml:119:
- { compiler: gcc, testsuite: system-test-dpdk, dpdk: dpdk, test_range:
"201-", unstable: unstable }
WARNING: Line is 101 characters long (recommended limit is 79)
#189 FILE: .github/workflows/test.yml:122:
- { compiler: gcc, testsuite: system-test-userspace, test_range:
"201-", unstable: unstable }
WARNING: Line is 91 characters long (recommended limit is 79)
#193 FILE: .github/workflows/test.yml:125:
- { compiler: gcc, testsuite: system-test, test_range: "201-",
unstable: unstable }
WARNING: Line is 117 characters long (recommended limit is 79)
#197 FILE: .github/workflows/test.yml:128:
- { compiler: clang, testsuite: system-test, sanitizers: sanitizers,
test_range: "201-", unstable: unstable }
Lines checked: 373, Warnings: 7, Errors: 0
Please check this out. If you feel there has been an error, please email
acon...@redhat.com
Thanks,
0-day Robot
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
[ovs-dev] [PATCH ovn v2] ci: Remove '--recheck' in CI.
If we want to catch new failures faster we have a better chance if CI
doesn't auto-retry (once).
There are some tests that are still "unstable" and fail every now and
then. In order to reduce the number of false negatives keep the
--recheck for them. To achieve that we use a new macro, TAG_UNSTABLE,
to tag all these tests. The list of "unstable" tests is compiled based
on the following discussion:
https://mail.openvswitch.org/pipermail/ovs-dev/2023-June/405465.html
In order to avoid new GitHub actions jobs, we re-purpose the last job of
each target type to also run the unstable tests. These jobs were
already running less tests than others so the additional run time should
not be an issue.
Signed-off-by: Dumitru Ceara
---
V2:
- Addressed Ales' comments:
- always run stable and unstable tests before declaring pass/fail
Changes in v1 (since RFC):
- kept recheck for unstable tests
- introduced TAG_UNSTABLE
- changed test.yml to run unstable tests in the last batch of every
test target type.
---
.ci/ci.sh | 2 +-
.ci/linux-build.sh | 76 ++
.github/workflows/test.yml | 15
tests/ovn-ic.at| 1 +
tests/ovn-ipsec.at | 1 +
tests/ovn-macros.at| 5 +++
tests/ovn-northd.at| 1 +
tests/ovn-performance.at | 1 +
tests/ovn.at | 13 +++
9 files changed, 92 insertions(+), 23 deletions(-)
diff --git a/.ci/ci.sh b/.ci/ci.sh
index 10f11939c5..a500aba764 100755
--- a/.ci/ci.sh
+++ b/.ci/ci.sh
@@ -101,7 +101,7 @@ function run_tests() {
&& \
ARCH=$ARCH CC=$CC LIBS=$LIBS OPTS=$OPTS TESTSUITE=$TESTSUITE \
TEST_RANGE=$TEST_RANGE SANITIZERS=$SANITIZERS DPDK=$DPDK \
-./.ci/linux-build.sh
+UNSTABLE=$UNSTABLE ./.ci/linux-build.sh
"
}
diff --git a/.ci/linux-build.sh b/.ci/linux-build.sh
index 5a79a52daf..4c5361f3ca 100755
--- a/.ci/linux-build.sh
+++ b/.ci/linux-build.sh
@@ -9,6 +9,7 @@ COMMON_CFLAGS=""
OVN_CFLAGS=""
OPTS="$OPTS --enable-Werror"
JOBS=${JOBS:-"-j4"}
+RECHECK=${RECHECK:-"no"}
function install_dpdk()
{
@@ -99,6 +100,17 @@ function configure_clang()
COMMON_CFLAGS="${COMMON_CFLAGS} -Wno-error=unused-command-line-argument"
}
+function run_tests()
+{
+if ! make distcheck CFLAGS="${COMMON_CFLAGS} ${OVN_CFLAGS}" $JOBS \
+TESTSUITEFLAGS="$JOBS $TEST_RANGE" RECHECK=$RECHECK
+then
+# testsuite.log is necessary for debugging.
+cat */_build/sub/tests/testsuite.log
+return 1
+fi
+}
+
function execute_tests()
{
# 'distcheck' will reconfigure with required options.
@@ -106,27 +118,61 @@ function execute_tests()
configure_ovn
export DISTCHECK_CONFIGURE_FLAGS="$OPTS"
-if ! make distcheck CFLAGS="${COMMON_CFLAGS} ${OVN_CFLAGS}" $JOBS \
-TESTSUITEFLAGS="$JOBS $TEST_RANGE" RECHECK=yes
-then
-# testsuite.log is necessary for debugging.
-cat */_build/sub/tests/testsuite.log
+
+local stable_rc=0
+local unstable_rc=0
+
+if ! SKIP_UNSTABLE=yes run_tests; then
+stable_rc=1
+fi
+
+if [ "$UNSTABLE" ]; then
+if ! SKIP_UNSTABLE=no TEST_RANGE="-k unstable" RECHECK=yes \
+run_tests; then
+unstable_rc=1
+fi
+fi
+
+if [[ $stable_rc -ne 0 ]] || [[ $unstable_rc -ne 0 ]]; then
exit 1
fi
}
+function run_system_tests()
+{
+local type=$1
+local log_file=$2
+
+if ! sudo make $JOBS $type TESTSUITEFLAGS="$TEST_RANGE" \
+RECHECK=$RECHECK; then
+# $log_file is necessary for debugging.
+cat tests/$log_file
+return 1
+fi
+}
+
function execute_system_tests()
{
- type=$1
- log_file=$2
-
- configure_ovn $OPTS
- make $JOBS || { cat config.log; exit 1; }
- if ! sudo make $JOBS $type TESTSUITEFLAGS="$TEST_RANGE" RECHECK=yes; then
- # $log_file is necessary for debugging.
- cat tests/$log_file
- exit 1
- fi
+configure_ovn $OPTS
+make $JOBS || { cat config.log; exit 1; }
+
+local stable_rc=0
+local unstable_rc=0
+
+if ! SKIP_UNSTABLE=yes run_system_tests $@; then
+stable_rc=1
+fi
+
+if [ "$UNSTABLE" ]; then
+if ! SKIP_UNSTABLE=no TEST_RANGE="-k unstable" RECHECK=yes \
+run_system_tests $@; then
+unstable_rc=1
+fi
+fi
+
+if [[ $stable_rc -ne 0 ]] || [[ $unstable_rc -ne 0 ]]; then
+exit 1
+fi
}
configure_$CC
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
index fe2a14c401..7d40251003 100644
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -92,6 +92,7 @@ jobs:
TESTSUITE: ${{ matrix.cfg.testsuite }}
TEST_RANGE: ${{ matrix.cfg.test_range }}
SANITIZERS: ${{ matrix.cfg.sanitizers }}
+ UNSTABLE:${{ matrix.cfg.unstable }}
name: linux ${{ join(matrix.cfg.*, ' ') }}
runs-on: ubuntu-latest
@@
