On 1/27/21 2:59 PM, Michal Privoznik wrote:
Since we've switched to meson our tests run with a timeout (meson
uses 30 seconds as the default). However, not every machine that
builds libvirt is fast enough to run every test under 30 seconds
(each test binary has its own timeout, but still). For instance
when building a package for distro on a farm that's under load.
Or on a generally slow ARM hardware. While each developer can
tune their command line for building by adding
--timeout-multiplier=10, this is hard to do for aforementioned
build farms.

It's time to admit that not everybody has the latest, top shelf
CPU and increase the timeout.

Signed-off-by: Michal Privoznik <mpriv...@redhat.com>
---

This sure will help these build farms environments, but what about the cases
where an actual timeout means that there is something wrong with the code?
E.g. commits 46d88d8dba56 and 2ba0b7497ce7 were only possible because I was
seeing tests timing out in Power hosts when the 30 sec timeout was being
enforced.

A 120 second default timeout for the majority of the test cases is a long time.
virschematest in this laptop I use takes 2.5 sec to complete. If I do something
wrong in the code and now the test is now 4 times slower (10 sec) I will not be
able to detect it (I'll need to start keeping track or something). You'll have 
to
run the test suit on your RasPi 2B to see that something went wrong because the
timeout is better tuned to your RasPI than this laptop, but then the code is 
already
upstream.

And the tests will get more complex and will naturally take longer to complete.
Eventually this timeout might no be enough. Increase the timeout again?

Meson 0.57 has support for disabling test timeout entirely with 
--timeout-multiplier=0,
disabling test timeout entirely. Can't we bump the meson version to 0.57 and
then tell the distros to use timeout-multiplier=0? That will solve the problems
for them, I keep the short timeouts for development, and we won't need to keep
bumping the test timeout once every 2 years or something because the s390x
TCG enviroment in Fedora COPR is timing out again.



Thanks,


DHB




Successfully tested on my RasPi 2B, where the top three longest test
were:

   qemuxml2argvtest              OK  142.43s
   virschematest                 OK  40.82s
   virnettlssessiontest          OK  38.64s

and syntax-check:

  sc_spacing-check               OK  50.26
  sc_avoid_if_before_free        OK  42.30s
  sc_prohibit_cross_inclusion    OK  40.51s

  build-aux/meson.build |  1 +
  tests/meson.build     | 14 ++++++--------
  2 files changed, 7 insertions(+), 8 deletions(-)

diff --git a/build-aux/meson.build b/build-aux/meson.build
index c506feefd2..e7b2db8759 100644
--- a/build-aux/meson.build
+++ b/build-aux/meson.build
@@ -44,6 +44,7 @@ if git
          potfiles_dep,
        ],
        suite: 'syntax-check',
+      timeout: 120,
      )
    endforeach
  endif
diff --git a/tests/meson.build b/tests/meson.build
index 0de0783839..702090d594 100644
--- a/tests/meson.build
+++ b/tests/meson.build
@@ -546,6 +546,10 @@ if conf.has('WITH_YAJL')
    ]
  endif
+# Increase the default timeout because some tests may run longer,
+# esp. on slower systems.
+timeout = 180
+
  foreach data : tests
    test_sources = '@0@.c'.format(data['name'])
    test_bin = executable(
@@ -577,12 +581,6 @@ foreach data : tests
      ],
      export_dynamic: true,
    )
-  if data['name'] == 'qemuxml2argvtest'
-    timeout = 90
-  else
-    # default meson timeout
-    timeout = 30
-  endif
    test(data['name'], test_bin, env: tests_env, timeout: timeout)
  endforeach
@@ -683,7 +681,7 @@ endif foreach name : test_scripts
    script = find_program(name)
-  test(name, script, env: tests_env)
+  test(name, script, env: tests_env, timeout: timeout)
  endforeach
add_test_setup(
@@ -703,6 +701,6 @@ add_test_setup(
      '--suppressions=@0@'.format(meson.current_source_dir() / 
'.valgrind.supp'),
      '--error-exitcode=1',
    ],
-  # default timeout in meson is 30s
+  # Tests take a lot longer when run under Valgrind
    timeout_multiplier: 4,
  )


Reply via email to