Re: [ovs-dev] [PATCH v2 2/2] ci: Run tc offload tests in GitHub Actions.
On Tue, Feb 28, 2023 at 04:52:15PM +0100, Eelco Chaudron wrote: > Run "make check-offloads" as part of the GitHub actions tests. > > This test was run 25 times using GitHub actions, and the > failing rerun test cases where excluded. There are quite some > first-run failures, but unfortunately, there is no other > more stable kernel available as a GitHub-hosted runner. > > Did not yet include sanitizers in the run, as it's causing > the test to run too long >30min and there seems to be (timing) > issues with some of the tests. > > Signed-off-by: Eelco Chaudron Hi Eelco, perhaps I am missing something obvious, but I seem to be getting failures with; 9: offloads - check_pkt_len action - offloads disabled FAILED (system-offloads-traffic.at:342) Also, there does not seem to be enough in the logs to determine why. https://github.com/horms/ovs/actions/runs/4323873948/jobs/7549439777#step:12:2847 ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [PATCH v2 2/2] ci: Run tc offload tests in GitHub Actions.
On 3 Mar 2023, at 16:15, Simon Horman wrote: > On Tue, Feb 28, 2023 at 04:52:15PM +0100, Eelco Chaudron wrote: >> Run "make check-offloads" as part of the GitHub actions tests. >> >> This test was run 25 times using GitHub actions, and the >> failing rerun test cases where excluded. There are quite some >> first-run failures, but unfortunately, there is no other >> more stable kernel available as a GitHub-hosted runner. >> >> Did not yet include sanitizers in the run, as it's causing >> the test to run too long >30min and there seems to be (timing) >> issues with some of the tests. >> >> Signed-off-by: Eelco Chaudron > > Hi Eelco, > > perhaps I am missing something obvious, > but I seem to be getting failures with; > > 9: offloads - check_pkt_len action - offloads disabled FAILED > (system-offloads-traffic.at:342) > > Also, there does not seem to be enough in the logs to determine why. > > https://github.com/horms/ovs/actions/runs/4323873948/jobs/7549439777#step:12:2847 Looked at the logs captures and they only show that not all packets passed trough: --- - 2023-03-03 14:59:53.139186466 + +++ /home/runner/work/ovs/ovs/tests/system-offloads-testsuite.dir/at-groups/9/stdout 2023-03-03 14:59:53.135502663 + @@ -1,2 +1,2 @@ -10 1032 +2 1032 Interesting as it was running fine in my setup; https://github.com/chaudron/ovs/actions/runs/4294641204/jobs/7483953067 Can you do a re-run and see if it’s fine a second run? I’ll do another couple of re-runs and see if it fails in any of them. //Eelco ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [PATCH v2 2/2] ci: Run tc offload tests in GitHub Actions.
On 7 Mar 2023, at 16:32, Eelco Chaudron wrote: > On 3 Mar 2023, at 16:15, Simon Horman wrote: > >> On Tue, Feb 28, 2023 at 04:52:15PM +0100, Eelco Chaudron wrote: >>> Run "make check-offloads" as part of the GitHub actions tests. >>> >>> This test was run 25 times using GitHub actions, and the >>> failing rerun test cases where excluded. There are quite some >>> first-run failures, but unfortunately, there is no other >>> more stable kernel available as a GitHub-hosted runner. >>> >>> Did not yet include sanitizers in the run, as it's causing >>> the test to run too long >30min and there seems to be (timing) >>> issues with some of the tests. >>> >>> Signed-off-by: Eelco Chaudron >> >> Hi Eelco, >> >> perhaps I am missing something obvious, >> but I seem to be getting failures with; >> >> 9: offloads - check_pkt_len action - offloads disabled FAILED >> (system-offloads-traffic.at:342) >> >> Also, there does not seem to be enough in the logs to determine why. >> >> https://github.com/horms/ovs/actions/runs/4323873948/jobs/7549439777#step:12:2847 Looking at your GitHub actions for this test (I think I was looking at the right commit :) It seems you only applied the second patch of the series, but the first patch updates the ubuntu release, this might be the problem. > > Looked at the logs captures and they only show that not all packets passed > trough: > > --- - 2023-03-03 14:59:53.139186466 + > +++ > /home/runner/work/ovs/ovs/tests/system-offloads-testsuite.dir/at-groups/9/stdout > 2023-03-03 14:59:53.135502663 + > @@ -1,2 +1,2 @@ > -10 1032 > +2 1032 > > > Interesting as it was running fine in my setup; > > https://github.com/chaudron/ovs/actions/runs/4294641204/jobs/7483953067 > > > Can you do a re-run and see if it’s fine a second run? I’ll do another couple > of re-runs and see if it fails in any of them. > ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [PATCH v2 2/2] ci: Run tc offload tests in GitHub Actions.
On Tue, Mar 07, 2023 at 04:58:47PM +0100, Eelco Chaudron wrote: > > > On 7 Mar 2023, at 16:32, Eelco Chaudron wrote: > > > On 3 Mar 2023, at 16:15, Simon Horman wrote: > > > >> On Tue, Feb 28, 2023 at 04:52:15PM +0100, Eelco Chaudron wrote: > >>> Run "make check-offloads" as part of the GitHub actions tests. > >>> > >>> This test was run 25 times using GitHub actions, and the > >>> failing rerun test cases where excluded. There are quite some > >>> first-run failures, but unfortunately, there is no other > >>> more stable kernel available as a GitHub-hosted runner. > >>> > >>> Did not yet include sanitizers in the run, as it's causing > >>> the test to run too long >30min and there seems to be (timing) > >>> issues with some of the tests. > >>> > >>> Signed-off-by: Eelco Chaudron > >> > >> Hi Eelco, > >> > >> perhaps I am missing something obvious, > >> but I seem to be getting failures with; > >> > >> 9: offloads - check_pkt_len action - offloads disabled FAILED > >> (system-offloads-traffic.at:342) > >> > >> Also, there does not seem to be enough in the logs to determine why. > >> > >> https://github.com/horms/ovs/actions/runs/4323873948/jobs/7549439777#step:12:2847 > > > Looking at your GitHub actions for this test (I think I was looking at the > right commit :) It seems you only applied the second patch of the series, but > the first patch updates the ubuntu release, this might be the problem. Yikes! Sorry about that. I've fixed up my branch and am running the workflow again. https://github.com/horms/ovs/actions/runs/4356340517 ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [PATCH v2 2/2] ci: Run tc offload tests in GitHub Actions.
On 3/7/23 17:31, Simon Horman wrote: > On Tue, Mar 07, 2023 at 04:58:47PM +0100, Eelco Chaudron wrote: >> >> >> On 7 Mar 2023, at 16:32, Eelco Chaudron wrote: >> >>> On 3 Mar 2023, at 16:15, Simon Horman wrote: >>> On Tue, Feb 28, 2023 at 04:52:15PM +0100, Eelco Chaudron wrote: > Run "make check-offloads" as part of the GitHub actions tests. > > This test was run 25 times using GitHub actions, and the > failing rerun test cases where excluded. There are quite some > first-run failures, but unfortunately, there is no other > more stable kernel available as a GitHub-hosted runner. > > Did not yet include sanitizers in the run, as it's causing > the test to run too long >30min and there seems to be (timing) > issues with some of the tests. > > Signed-off-by: Eelco Chaudron Hi Eelco, perhaps I am missing something obvious, but I seem to be getting failures with; 9: offloads - check_pkt_len action - offloads disabled FAILED (system-offloads-traffic.at:342) Also, there does not seem to be enough in the logs to determine why. https://github.com/horms/ovs/actions/runs/4323873948/jobs/7549439777#step:12:2847 >> >> >> Looking at your GitHub actions for this test (I think I was looking at the >> right commit :) It seems you only applied the second patch of the series, >> but the first patch updates the ubuntu release, this might be the problem. FWIW, this change belongs to the second patch. It's not justified in the first one. > > Yikes! > Sorry about that. > I've fixed up my branch and am running the workflow again. > > https://github.com/horms/ovs/actions/runs/4356340517 ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [PATCH v2 2/2] ci: Run tc offload tests in GitHub Actions.
On 2/28/23 16:52, Eelco Chaudron wrote: > Run "make check-offloads" as part of the GitHub actions tests. > > This test was run 25 times using GitHub actions, and the > failing rerun test cases where excluded. There are quite some > first-run failures, but unfortunately, there is no other > more stable kernel available as a GitHub-hosted runner. > > Did not yet include sanitizers in the run, as it's causing > the test to run too long >30min and there seems to be (timing) > issues with some of the tests. > > Signed-off-by: Eelco Chaudron > --- > v2: Added a new test keyword to exclude the failing tests. > Added some documentation around the keyword usage. > > .ci/linux-build.sh |6 +- > .github/workflows/build-and-test.yml | 10 +- > Documentation/topics/testing.rst | 16 > tests/system-offloads-traffic.at |3 +++ > tests/system-traffic.at |2 ++ > 5 files changed, 35 insertions(+), 2 deletions(-) > > diff --git a/.ci/linux-build.sh b/.ci/linux-build.sh > index 6394a8137..19ed9796d 100755 > --- a/.ci/linux-build.sh > +++ b/.ci/linux-build.sh > @@ -159,7 +159,7 @@ fi > > OPTS="${EXTRA_OPTS} ${OPTS} $*" > > -if [ "$TESTSUITE" ]; then > +if [ "$TESTSUITE" = 'test' ]; then > # 'distcheck' will reconfigure with required options. > # Now we only need to prepare the Makefile without sparse-wrapped CC. > configure_ovs > @@ -169,6 +169,10 @@ if [ "$TESTSUITE" ]; then > TESTSUITEFLAGS=-j4 RECHECK=yes > else > build_ovs > +if [ -n "$TESTSUITE" ]; then > +sudo -E PATH="$PATH" make "$TESTSUITE" TESTSUITEFLAGS="$TEST_OPTS" \ > +RECHECK=yes > +fi > fi > > exit 0 > diff --git a/.github/workflows/build-and-test.yml > b/.github/workflows/build-and-test.yml > index 86e594bf3..45326b659 100644 > --- a/.github/workflows/build-and-test.yml > +++ b/.github/workflows/build-and-test.yml > @@ -17,6 +17,7 @@ jobs: >OPTS:${{ matrix.opts }} >SANITIZERS: ${{ matrix.sanitizers }} >TESTSUITE: ${{ matrix.testsuite }} > + TEST_OPTS: ${{ matrix.test_opts }} > > name: linux ${{ join(matrix.*, ' ') }} > runs-on: ubuntu-22.04 > @@ -86,6 +87,10 @@ jobs: > m32: m32 > opts: --disable-ssl > > + - compiler: gcc > +testsuite:check-offloads > +test_opts:"-k !github_offloads_skip" > + > steps: > - name: checkout >uses: actions/checkout@v3 > @@ -147,7 +152,10 @@ jobs: > # So, we're just archiving everything here to avoid any issues. > mkdir logs > cp config.log ./logs/ > -cp -r ./*/_build/sub/tests/testsuite.* ./logs/ || true > +sudo chown -R $USER ./tests/testsuite.* \ > +./tests/system-offloads-testsuite.* || true > +cp -r ./tests/testsuite.* ./logs/ || true This seems suspicious because distcheck builds and runs tests in a different directory. > +cp -r ./tests/system-offloads-testsuite.* ./logs/ || true > tar -czvf logs.tgz logs/ > > - name: upload logs on failure > diff --git a/Documentation/topics/testing.rst > b/Documentation/topics/testing.rst > index 5f6940b84..d06167ee3 100644 > --- a/Documentation/topics/testing.rst > +++ b/Documentation/topics/testing.rst > @@ -114,6 +114,22 @@ And from another window, one can execute ovs-xxx > commands like:: > > Once done with investigation, press ENTER to perform cleanup operation. > > +GitHub actions > +++ Empty line should be here. > +The OVS GitHub repository also runs some of these unit tests through GitHub > +actions. These tests are defined in the > +``ovs/.github/workflows/build-and-test.yml`` file. > + > +Based on the GitHub runners available, not all tests will work. In these > cases, > +the AT_KEYWORDS() macro can be used. For example, to skip a > +``make check-offloads`` test, use the ``github_offloads_skip`` keyword. > + > +Only use these keywords if no other way to skip the test is available. s/skip/fix or skip/ maybe? > + > +To see a list of currently skipped tests, you can do something like:: > + > +$ make check-offloads TESTSUITEFLAGS="-l" | grep -B 1 > github_offloads_skip > + > .. _testing-coverage: > > Coverage > diff --git a/tests/system-offloads-traffic.at > b/tests/system-offloads-traffic.at > index 7558812eb..6ee92746b 100644 > --- a/tests/system-offloads-traffic.at > +++ b/tests/system-offloads-traffic.at > @@ -191,6 +191,7 @@ AT_CLEANUP > > AT_SETUP([offloads - check interface meter offloading - offloads disabled]) > AT_KEYWORDS([dp-meter]) > +AT_KEYWORDS([github_offloads_skip]) > AT_SKIP_IF([test $HAVE_NC = "no"]) > OVS_TRAFFIC_VSWITCHD_START() > > @@ -240,6 +241,7 @@ AT_CLEANUP > > AT_SETUP([offloads - check interface meter offloading - offloads enabled]) > AT_KEYWORDS([offload-meter]) > +AT_KEYWORDS([github_of
Re: [ovs-dev] [PATCH v2 2/2] ci: Run tc offload tests in GitHub Actions.
On 7 Mar 2023, at 18:01, Ilya Maximets wrote: > On 2/28/23 16:52, Eelco Chaudron wrote: >> Run "make check-offloads" as part of the GitHub actions tests. >> >> This test was run 25 times using GitHub actions, and the >> failing rerun test cases where excluded. There are quite some >> first-run failures, but unfortunately, there is no other >> more stable kernel available as a GitHub-hosted runner. >> >> Did not yet include sanitizers in the run, as it's causing >> the test to run too long >30min and there seems to be (timing) >> issues with some of the tests. >> >> Signed-off-by: Eelco Chaudron >> --- >> v2: Added a new test keyword to exclude the failing tests. >> Added some documentation around the keyword usage. >> >> .ci/linux-build.sh |6 +- >> .github/workflows/build-and-test.yml | 10 +- >> Documentation/topics/testing.rst | 16 >> tests/system-offloads-traffic.at |3 +++ >> tests/system-traffic.at |2 ++ >> 5 files changed, 35 insertions(+), 2 deletions(-) >> >> diff --git a/.ci/linux-build.sh b/.ci/linux-build.sh >> index 6394a8137..19ed9796d 100755 >> --- a/.ci/linux-build.sh >> +++ b/.ci/linux-build.sh >> @@ -159,7 +159,7 @@ fi >> >> OPTS="${EXTRA_OPTS} ${OPTS} $*" >> >> -if [ "$TESTSUITE" ]; then >> +if [ "$TESTSUITE" = 'test' ]; then >> # 'distcheck' will reconfigure with required options. >> # Now we only need to prepare the Makefile without sparse-wrapped CC. >> configure_ovs >> @@ -169,6 +169,10 @@ if [ "$TESTSUITE" ]; then >> TESTSUITEFLAGS=-j4 RECHECK=yes >> else >> build_ovs >> +if [ -n "$TESTSUITE" ]; then >> +sudo -E PATH="$PATH" make "$TESTSUITE" TESTSUITEFLAGS="$TEST_OPTS" \ >> +RECHECK=yes >> +fi >> fi >> >> exit 0 >> diff --git a/.github/workflows/build-and-test.yml >> b/.github/workflows/build-and-test.yml >> index 86e594bf3..45326b659 100644 >> --- a/.github/workflows/build-and-test.yml >> +++ b/.github/workflows/build-and-test.yml >> @@ -17,6 +17,7 @@ jobs: >>OPTS:${{ matrix.opts }} >>SANITIZERS: ${{ matrix.sanitizers }} >>TESTSUITE: ${{ matrix.testsuite }} >> + TEST_OPTS: ${{ matrix.test_opts }} >> >> name: linux ${{ join(matrix.*, ' ') }} >> runs-on: ubuntu-22.04 >> @@ -86,6 +87,10 @@ jobs: >> m32: m32 >> opts: --disable-ssl >> >> + - compiler: gcc >> +testsuite:check-offloads >> +test_opts:"-k !github_offloads_skip" >> + >> steps: >> - name: checkout >>uses: actions/checkout@v3 >> @@ -147,7 +152,10 @@ jobs: >> # So, we're just archiving everything here to avoid any issues. >> mkdir logs >> cp config.log ./logs/ >> -cp -r ./*/_build/sub/tests/testsuite.* ./logs/ || true >> +sudo chown -R $USER ./tests/testsuite.* \ >> +./tests/system-offloads-testsuite.* || true >> +cp -r ./tests/testsuite.* ./logs/ || true > > This seems suspicious because distcheck builds and runs tests in a different > directory. You are right, I messed this up :( Will fix it in the next version, currently running some “failures” to make sure it works for both type of tests. >> +cp -r ./tests/system-offloads-testsuite.* ./logs/ || true >> tar -czvf logs.tgz logs/ >> >> - name: upload logs on failure >> diff --git a/Documentation/topics/testing.rst >> b/Documentation/topics/testing.rst >> index 5f6940b84..d06167ee3 100644 >> --- a/Documentation/topics/testing.rst >> +++ b/Documentation/topics/testing.rst >> @@ -114,6 +114,22 @@ And from another window, one can execute ovs-xxx >> commands like:: >> >> Once done with investigation, press ENTER to perform cleanup operation. >> >> +GitHub actions >> +++ > > Empty line should be here. Added in v3. >> +The OVS GitHub repository also runs some of these unit tests through GitHub >> +actions. These tests are defined in the >> +``ovs/.github/workflows/build-and-test.yml`` file. >> + >> +Based on the GitHub runners available, not all tests will work. In these >> cases, >> +the AT_KEYWORDS() macro can be used. For example, to skip a >> +``make check-offloads`` test, use the ``github_offloads_skip`` keyword. >> + >> +Only use these keywords if no other way to skip the test is available. > > s/skip/fix or skip/ maybe? Added in v3. ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev