Valid point, thank you. I can see three options what to do: 1) Ignore older version of make and assume that 4.2 or newer is used (4.1 is in Ubuntu 18.04 that is no longer supported as a build platform as I was told; 20.04 has 4.2). In this case make provides number of jobs in $MAKEFLAGS and this makes getting it trivial. In case of an older make only two options will be available: (default) single-threaded, and using all cores. 2) Use some environment variable to provide number of jobs instead of -j. For example NRUNNER_MAX_JOBS=4 make check-avocado. I don't like this option because it is not transparent to the user who doesn't know about this env_var. Maybe a message should be printed when no environment variable is provided (e.g. make check-avocado -> 1 worker is used for check-avocado testing. You can speed up the testing by providing desired number of workers with environment variable like this: NRUNNER_MAX_JOBS=16 make check-avocado) 3) A harder approach: rewrite python code from scratch, as you have suggested.
Which one should I choose? November 16, 2022 6:06 PM, "Daniel P. Berrangé" <berra...@redhat.com> wrote: > On Wed, Nov 16, 2022 at 04:48:10PM +0300, Pavel Dovgalyuk wrote: > >> From: bakulinm <bakul...@gmail.com> >> >> make check-avocado takes a lot of time, and avocado since version 91 has >> multithreaded mode for running several tests simultaneously. >> This patch allows to run "make check-avocado -j" to use all cores or, >> for example, "make check-avocado -j4" to select number of workers to use. >> By default ("make check-avocado") only one worker is used. >> >> Changes: >> 1) Version of avocado in requirements.txt upgraded from 88.1 to <93 >> (LTS version is used, as mentioned here >> https://avocado-framework.readthedocs.io/en/latest/releases/lts/92_0.html ) >> 2) Makefile 4.1 (used in e.g. Ubuntu 18.04) doesn't provide number of jobs >> in $MAKEFLAGS, so python script from here >> https://stackoverflow.com/a/67247743/5936122 is used. >> >> diff --git a/tests/jobs.py b/tests/jobs.py >> new file mode 100644 >> index 0000000000..a339192d97 >> --- /dev/null >> +++ b/tests/jobs.py >> @@ -0,0 +1,42 @@ > > No license information or attribution put on this code that > you've said was directly copied from stackoverflow. AFAICT, > all content on stackoverflow is placed under the creative > commons license. This is not something we would generally > want to be applied to code in QEMU as that's generally > considered as a content license. > > Unless the copied code is trivial (this case is not), then > stackoverflow should really only be used a learning resource, > and then code written from scratch without copying, so it > can be placed under the project's usual license. > >> +import argparse, os >> +import sys >> + >> +def safe_int(s): >> + try: >> + return int(s) >> + except: >> + return -1 >> + >> +class JobserverArgs: >> + known_names = ["jobserver-fds","jobserver-auth"] >> + def __init__(self): >> + self.fds = "-1,-1" >> + >> + @staticmethod >> + def from_argv(): >> + ja = JobserverArgs() >> + parser = argparse.ArgumentParser() >> + for name in JobserverArgs.known_names: >> + parser.add_argument('--'+name, dest="fds") >> + parser.parse_known_args(namespace=ja) >> + return ja >> + >> + def get_fds(self): >> + return tuple([safe_int(fd) for fd in (self.fds+",").split(",")][:2]) >> + >> +fd_in, fd_out = JobserverArgs.from_argv().get_fds() >> + >> +if fd_in == -1 or fd_out == -1: >> +# if no jobserver is used, but -j is present, use total number of cpu cores >> + if '-j' in sys.argv: >> + print(os.cpu_count()) >> +# use single thread >> + else: >> + print(1) >> +else: >> + os.set_blocking(fd_in, False) >> + >> + tokens = os.read(fd_in, 1024) >> + os.write(fd_out, tokens) >> + >> + print(len(tokens)+1) >> \ No newline at end of file >> diff --git a/tests/requirements.txt b/tests/requirements.txt >> index 0ba561b6bd..3b8c4d4706 100644 >> --- a/tests/requirements.txt >> +++ b/tests/requirements.txt >> @@ -2,5 +2,5 @@ >> # in the tests/venv Python virtual environment. For more info, >> # refer to: https://pip.pypa.io/en/stable/user_guide/#id1 >> # Note that qemu.git/python/ is always implicitly installed. >> -avocado-framework==88.1 >> +avocado-framework<93 >> pycdlib==1.11.0 > > With regards, > Daniel > -- > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|