Re: [ovs-dev] [PATCH v2 2/2] ci: Run tc offload tests in GitHub Actions.
On Tue, Feb 28, 2023 at 04:52:15PM +0100, Eelco Chaudron wrote: > Run "make check-offloads" as part of the GitHub actions tests. > > This test was run 25 times using GitHub actions, and the > failing rerun test cases where excluded. There are quite some > first-run failures, but unfortunately, there is no other > more stable kernel available as a GitHub-hosted runner. > > Did not yet include sanitizers in the run, as it's causing > the test to run too long >30min and there seems to be (timing) > issues with some of the tests. > > Signed-off-by: Eelco Chaudron Hi Eelco, perhaps I am missing something obvious, but I seem to be getting failures with; 9: offloads - check_pkt_len action - offloads disabled FAILED (system-offloads-traffic.at:342) Also, there does not seem to be enough in the logs to determine why. https://github.com/horms/ovs/actions/runs/4323873948/jobs/7549439777#step:12:2847 ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [PATCH v2 2/2] ci: Run tc offload tests in GitHub Actions.
On 3 Mar 2023, at 16:15, Simon Horman wrote: > On Tue, Feb 28, 2023 at 04:52:15PM +0100, Eelco Chaudron wrote: >> Run "make check-offloads" as part of the GitHub actions tests. >> >> This test was run 25 times using GitHub actions, and the >> failing rerun test cases where excluded. There are quite some >> first-run failures, but unfortunately, there is no other >> more stable kernel available as a GitHub-hosted runner. >> >> Did not yet include sanitizers in the run, as it's causing >> the test to run too long >30min and there seems to be (timing) >> issues with some of the tests. >> >> Signed-off-by: Eelco Chaudron > > Hi Eelco, > > perhaps I am missing something obvious, > but I seem to be getting failures with; > > 9: offloads - check_pkt_len action - offloads disabled FAILED > (system-offloads-traffic.at:342) > > Also, there does not seem to be enough in the logs to determine why. > > https://github.com/horms/ovs/actions/runs/4323873948/jobs/7549439777#step:12:2847 Looked at the logs captures and they only show that not all packets passed trough: --- - 2023-03-03 14:59:53.139186466 + +++ /home/runner/work/ovs/ovs/tests/system-offloads-testsuite.dir/at-groups/9/stdout 2023-03-03 14:59:53.135502663 + @@ -1,2 +1,2 @@ -10 1032 +2 1032 Interesting as it was running fine in my setup; https://github.com/chaudron/ovs/actions/runs/4294641204/jobs/7483953067 Can you do a re-run and see if it’s fine a second run? I’ll do another couple of re-runs and see if it fails in any of them. //Eelco ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [PATCH v2 2/2] ci: Run tc offload tests in GitHub Actions.
On 7 Mar 2023, at 16:32, Eelco Chaudron wrote: > On 3 Mar 2023, at 16:15, Simon Horman wrote: > >> On Tue, Feb 28, 2023 at 04:52:15PM +0100, Eelco Chaudron wrote: >>> Run "make check-offloads" as part of the GitHub actions tests. >>> >>> This test was run 25 times using GitHub actions, and the >>> failing rerun test cases where excluded. There are quite some >>> first-run failures, but unfortunately, there is no other >>> more stable kernel available as a GitHub-hosted runner. >>> >>> Did not yet include sanitizers in the run, as it's causing >>> the test to run too long >30min and there seems to be (timing) >>> issues with some of the tests. >>> >>> Signed-off-by: Eelco Chaudron >> >> Hi Eelco, >> >> perhaps I am missing something obvious, >> but I seem to be getting failures with; >> >> 9: offloads - check_pkt_len action - offloads disabled FAILED >> (system-offloads-traffic.at:342) >> >> Also, there does not seem to be enough in the logs to determine why. >> >> https://github.com/horms/ovs/actions/runs/4323873948/jobs/7549439777#step:12:2847 Looking at your GitHub actions for this test (I think I was looking at the right commit :) It seems you only applied the second patch of the series, but the first patch updates the ubuntu release, this might be the problem. > > Looked at the logs captures and they only show that not all packets passed > trough: > > --- - 2023-03-03 14:59:53.139186466 + > +++ > /home/runner/work/ovs/ovs/tests/system-offloads-testsuite.dir/at-groups/9/stdout > 2023-03-03 14:59:53.135502663 + > @@ -1,2 +1,2 @@ > -10 1032 > +2 1032 > > > Interesting as it was running fine in my setup; > > https://github.com/chaudron/ovs/actions/runs/4294641204/jobs/7483953067 > > > Can you do a re-run and see if it’s fine a second run? I’ll do another couple > of re-runs and see if it fails in any of them. > ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [PATCH v2 2/2] ci: Run tc offload tests in GitHub Actions.
On Tue, Mar 07, 2023 at 04:58:47PM +0100, Eelco Chaudron wrote: > > > On 7 Mar 2023, at 16:32, Eelco Chaudron wrote: > > > On 3 Mar 2023, at 16:15, Simon Horman wrote: > > > >> On Tue, Feb 28, 2023 at 04:52:15PM +0100, Eelco Chaudron wrote: > >>> Run "make check-offloads" as part of the GitHub actions tests. > >>> > >>> This test was run 25 times using GitHub actions, and the > >>> failing rerun test cases where excluded. There are quite some > >>> first-run failures, but unfortunately, there is no other > >>> more stable kernel available as a GitHub-hosted runner. > >>> > >>> Did not yet include sanitizers in the run, as it's causing > >>> the test to run too long >30min and there seems to be (timing) > >>> issues with some of the tests. > >>> > >>> Signed-off-by: Eelco Chaudron > >> > >> Hi Eelco, > >> > >> perhaps I am missing something obvious, > >> but I seem to be getting failures with; > >> > >> 9: offloads - check_pkt_len action - offloads disabled FAILED > >> (system-offloads-traffic.at:342) > >> > >> Also, there does not seem to be enough in the logs to determine why. > >> > >> https://github.com/horms/ovs/actions/runs/4323873948/jobs/7549439777#step:12:2847 > > > Looking at your GitHub actions for this test (I think I was looking at the > right commit :) It seems you only applied the second patch of the series, but > the first patch updates the ubuntu release, this might be the problem. Yikes! Sorry about that. I've fixed up my branch and am running the workflow again. https://github.com/horms/ovs/actions/runs/4356340517 ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [PATCH v2 2/2] ci: Run tc offload tests in GitHub Actions.
On 3/7/23 17:31, Simon Horman wrote: > On Tue, Mar 07, 2023 at 04:58:47PM +0100, Eelco Chaudron wrote: >> >> >> On 7 Mar 2023, at 16:32, Eelco Chaudron wrote: >> >>> On 3 Mar 2023, at 16:15, Simon Horman wrote: >>> On Tue, Feb 28, 2023 at 04:52:15PM +0100, Eelco Chaudron wrote: > Run "make check-offloads" as part of the GitHub actions tests. > > This test was run 25 times using GitHub actions, and the > failing rerun test cases where excluded. There are quite some > first-run failures, but unfortunately, there is no other > more stable kernel available as a GitHub-hosted runner. > > Did not yet include sanitizers in the run, as it's causing > the test to run too long >30min and there seems to be (timing) > issues with some of the tests. > > Signed-off-by: Eelco Chaudron Hi Eelco, perhaps I am missing something obvious, but I seem to be getting failures with; 9: offloads - check_pkt_len action - offloads disabled FAILED (system-offloads-traffic.at:342) Also, there does not seem to be enough in the logs to determine why. https://github.com/horms/ovs/actions/runs/4323873948/jobs/7549439777#step:12:2847 >> >> >> Looking at your GitHub actions for this test (I think I was looking at the >> right commit :) It seems you only applied the second patch of the series, >> but the first patch updates the ubuntu release, this might be the problem. FWIW, this change belongs to the second patch. It's not justified in the first one. > > Yikes! > Sorry about that. > I've fixed up my branch and am running the workflow again. > > https://github.com/horms/ovs/actions/runs/4356340517 ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [PATCH v2 2/2] ci: Run tc offload tests in GitHub Actions.
On 2/28/23 16:52, Eelco Chaudron wrote:
> Run "make check-offloads" as part of the GitHub actions tests.
>
> This test was run 25 times using GitHub actions, and the
> failing rerun test cases where excluded. There are quite some
> first-run failures, but unfortunately, there is no other
> more stable kernel available as a GitHub-hosted runner.
>
> Did not yet include sanitizers in the run, as it's causing
> the test to run too long >30min and there seems to be (timing)
> issues with some of the tests.
>
> Signed-off-by: Eelco Chaudron
> ---
> v2: Added a new test keyword to exclude the failing tests.
> Added some documentation around the keyword usage.
>
> .ci/linux-build.sh |6 +-
> .github/workflows/build-and-test.yml | 10 +-
> Documentation/topics/testing.rst | 16
> tests/system-offloads-traffic.at |3 +++
> tests/system-traffic.at |2 ++
> 5 files changed, 35 insertions(+), 2 deletions(-)
>
> diff --git a/.ci/linux-build.sh b/.ci/linux-build.sh
> index 6394a8137..19ed9796d 100755
> --- a/.ci/linux-build.sh
> +++ b/.ci/linux-build.sh
> @@ -159,7 +159,7 @@ fi
>
> OPTS="${EXTRA_OPTS} ${OPTS} $*"
>
> -if [ "$TESTSUITE" ]; then
> +if [ "$TESTSUITE" = 'test' ]; then
> # 'distcheck' will reconfigure with required options.
> # Now we only need to prepare the Makefile without sparse-wrapped CC.
> configure_ovs
> @@ -169,6 +169,10 @@ if [ "$TESTSUITE" ]; then
> TESTSUITEFLAGS=-j4 RECHECK=yes
> else
> build_ovs
> +if [ -n "$TESTSUITE" ]; then
> +sudo -E PATH="$PATH" make "$TESTSUITE" TESTSUITEFLAGS="$TEST_OPTS" \
> +RECHECK=yes
> +fi
> fi
>
> exit 0
> diff --git a/.github/workflows/build-and-test.yml
> b/.github/workflows/build-and-test.yml
> index 86e594bf3..45326b659 100644
> --- a/.github/workflows/build-and-test.yml
> +++ b/.github/workflows/build-and-test.yml
> @@ -17,6 +17,7 @@ jobs:
>OPTS:${{ matrix.opts }}
>SANITIZERS: ${{ matrix.sanitizers }}
>TESTSUITE: ${{ matrix.testsuite }}
> + TEST_OPTS: ${{ matrix.test_opts }}
>
> name: linux ${{ join(matrix.*, ' ') }}
> runs-on: ubuntu-22.04
> @@ -86,6 +87,10 @@ jobs:
> m32: m32
> opts: --disable-ssl
>
> + - compiler: gcc
> +testsuite:check-offloads
> +test_opts:"-k !github_offloads_skip"
> +
> steps:
> - name: checkout
>uses: actions/checkout@v3
> @@ -147,7 +152,10 @@ jobs:
> # So, we're just archiving everything here to avoid any issues.
> mkdir logs
> cp config.log ./logs/
> -cp -r ./*/_build/sub/tests/testsuite.* ./logs/ || true
> +sudo chown -R $USER ./tests/testsuite.* \
> +./tests/system-offloads-testsuite.* || true
> +cp -r ./tests/testsuite.* ./logs/ || true
This seems suspicious because distcheck builds and runs tests in a different
directory.
> +cp -r ./tests/system-offloads-testsuite.* ./logs/ || true
> tar -czvf logs.tgz logs/
>
> - name: upload logs on failure
> diff --git a/Documentation/topics/testing.rst
> b/Documentation/topics/testing.rst
> index 5f6940b84..d06167ee3 100644
> --- a/Documentation/topics/testing.rst
> +++ b/Documentation/topics/testing.rst
> @@ -114,6 +114,22 @@ And from another window, one can execute ovs-xxx
> commands like::
>
> Once done with investigation, press ENTER to perform cleanup operation.
>
> +GitHub actions
> +++
Empty line should be here.
> +The OVS GitHub repository also runs some of these unit tests through GitHub
> +actions. These tests are defined in the
> +``ovs/.github/workflows/build-and-test.yml`` file.
> +
> +Based on the GitHub runners available, not all tests will work. In these
> cases,
> +the AT_KEYWORDS() macro can be used. For example, to skip a
> +``make check-offloads`` test, use the ``github_offloads_skip`` keyword.
> +
> +Only use these keywords if no other way to skip the test is available.
s/skip/fix or skip/ maybe?
> +
> +To see a list of currently skipped tests, you can do something like::
> +
> +$ make check-offloads TESTSUITEFLAGS="-l" | grep -B 1
> github_offloads_skip
> +
> .. _testing-coverage:
>
> Coverage
> diff --git a/tests/system-offloads-traffic.at
> b/tests/system-offloads-traffic.at
> index 7558812eb..6ee92746b 100644
> --- a/tests/system-offloads-traffic.at
> +++ b/tests/system-offloads-traffic.at
> @@ -191,6 +191,7 @@ AT_CLEANUP
>
> AT_SETUP([offloads - check interface meter offloading - offloads disabled])
> AT_KEYWORDS([dp-meter])
> +AT_KEYWORDS([github_offloads_skip])
> AT_SKIP_IF([test $HAVE_NC = "no"])
> OVS_TRAFFIC_VSWITCHD_START()
>
> @@ -240,6 +241,7 @@ AT_CLEANUP
>
> AT_SETUP([offloads - check interface meter offloading - offloads enabled])
> AT_KEYWORDS([offload-meter])
> +AT_KEYWORDS([github_of
Re: [ovs-dev] [PATCH v2 2/2] ci: Run tc offload tests in GitHub Actions.
On 7 Mar 2023, at 18:01, Ilya Maximets wrote:
> On 2/28/23 16:52, Eelco Chaudron wrote:
>> Run "make check-offloads" as part of the GitHub actions tests.
>>
>> This test was run 25 times using GitHub actions, and the
>> failing rerun test cases where excluded. There are quite some
>> first-run failures, but unfortunately, there is no other
>> more stable kernel available as a GitHub-hosted runner.
>>
>> Did not yet include sanitizers in the run, as it's causing
>> the test to run too long >30min and there seems to be (timing)
>> issues with some of the tests.
>>
>> Signed-off-by: Eelco Chaudron
>> ---
>> v2: Added a new test keyword to exclude the failing tests.
>> Added some documentation around the keyword usage.
>>
>> .ci/linux-build.sh |6 +-
>> .github/workflows/build-and-test.yml | 10 +-
>> Documentation/topics/testing.rst | 16
>> tests/system-offloads-traffic.at |3 +++
>> tests/system-traffic.at |2 ++
>> 5 files changed, 35 insertions(+), 2 deletions(-)
>>
>> diff --git a/.ci/linux-build.sh b/.ci/linux-build.sh
>> index 6394a8137..19ed9796d 100755
>> --- a/.ci/linux-build.sh
>> +++ b/.ci/linux-build.sh
>> @@ -159,7 +159,7 @@ fi
>>
>> OPTS="${EXTRA_OPTS} ${OPTS} $*"
>>
>> -if [ "$TESTSUITE" ]; then
>> +if [ "$TESTSUITE" = 'test' ]; then
>> # 'distcheck' will reconfigure with required options.
>> # Now we only need to prepare the Makefile without sparse-wrapped CC.
>> configure_ovs
>> @@ -169,6 +169,10 @@ if [ "$TESTSUITE" ]; then
>> TESTSUITEFLAGS=-j4 RECHECK=yes
>> else
>> build_ovs
>> +if [ -n "$TESTSUITE" ]; then
>> +sudo -E PATH="$PATH" make "$TESTSUITE" TESTSUITEFLAGS="$TEST_OPTS" \
>> +RECHECK=yes
>> +fi
>> fi
>>
>> exit 0
>> diff --git a/.github/workflows/build-and-test.yml
>> b/.github/workflows/build-and-test.yml
>> index 86e594bf3..45326b659 100644
>> --- a/.github/workflows/build-and-test.yml
>> +++ b/.github/workflows/build-and-test.yml
>> @@ -17,6 +17,7 @@ jobs:
>>OPTS:${{ matrix.opts }}
>>SANITIZERS: ${{ matrix.sanitizers }}
>>TESTSUITE: ${{ matrix.testsuite }}
>> + TEST_OPTS: ${{ matrix.test_opts }}
>>
>> name: linux ${{ join(matrix.*, ' ') }}
>> runs-on: ubuntu-22.04
>> @@ -86,6 +87,10 @@ jobs:
>> m32: m32
>> opts: --disable-ssl
>>
>> + - compiler: gcc
>> +testsuite:check-offloads
>> +test_opts:"-k !github_offloads_skip"
>> +
>> steps:
>> - name: checkout
>>uses: actions/checkout@v3
>> @@ -147,7 +152,10 @@ jobs:
>> # So, we're just archiving everything here to avoid any issues.
>> mkdir logs
>> cp config.log ./logs/
>> -cp -r ./*/_build/sub/tests/testsuite.* ./logs/ || true
>> +sudo chown -R $USER ./tests/testsuite.* \
>> +./tests/system-offloads-testsuite.* || true
>> +cp -r ./tests/testsuite.* ./logs/ || true
>
> This seems suspicious because distcheck builds and runs tests in a different
> directory.
You are right, I messed this up :( Will fix it in the next version, currently
running some “failures” to make sure it works for both type of tests.
>> +cp -r ./tests/system-offloads-testsuite.* ./logs/ || true
>> tar -czvf logs.tgz logs/
>>
>> - name: upload logs on failure
>> diff --git a/Documentation/topics/testing.rst
>> b/Documentation/topics/testing.rst
>> index 5f6940b84..d06167ee3 100644
>> --- a/Documentation/topics/testing.rst
>> +++ b/Documentation/topics/testing.rst
>> @@ -114,6 +114,22 @@ And from another window, one can execute ovs-xxx
>> commands like::
>>
>> Once done with investigation, press ENTER to perform cleanup operation.
>>
>> +GitHub actions
>> +++
>
> Empty line should be here.
Added in v3.
>> +The OVS GitHub repository also runs some of these unit tests through GitHub
>> +actions. These tests are defined in the
>> +``ovs/.github/workflows/build-and-test.yml`` file.
>> +
>> +Based on the GitHub runners available, not all tests will work. In these
>> cases,
>> +the AT_KEYWORDS() macro can be used. For example, to skip a
>> +``make check-offloads`` test, use the ``github_offloads_skip`` keyword.
>> +
>> +Only use these keywords if no other way to skip the test is available.
>
> s/skip/fix or skip/ maybe?
Added in v3.
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
