On 9/22/25 10:08, Daniel P. Berrangé wrote:
On Mon, Sep 22, 2025 at 10:59:11AM +0400, Marc-André Lureau wrote:
Hi
On Fri, Sep 19, 2025 at 5:50 PM Daniel P. Berrangé <[email protected]>
wrote:
On Fri, Sep 19, 2025 at 05:32:56PM +0400, [email protected]
wrote:
From: Marc-André Lureau <[email protected]>
Running meson on each subproject is quite slow.
Why is this faster ? Does it make meson do parallel downloads
if you list then all at once ?
It doesn't look like it does it in paralllel, yet it is x times faster...
time sh -c 'for s in keycodemapdb libvfio-user berkeley-softfloat-3
berkeley-testfloat-3 anyhow-1-rs arbitrary-int-1-rs bilge-0.2-rs
bilge-impl-0.2-rs either-1-rs foreign-0.3-rs itertools-0.11-rs libc-0.2-rs
proc-macro2-1-rs proc-macro-error-1-rs proc-macro-error-attr-1-rs
quote-1-rs syn-2-rs unicode-ident-1-rs; do
meson subprojects download $s >/dev/null
done'
sh -c 5.27s user 0.33s system 99% cpu 5.613 total
time sh -c 'meson subprojects download keycodemapdb libvfio-user
berkeley-softfloat-3 berkeley-testfloat-3 anyhow-1-rs arbitrary-int-1-rs
bilge-0.2-rs bilge-impl-0.2-rs either-1-rs foreign-0.3-rs itertools-0.11-rs
libc-0.2-rs proc-macro2-1-rs proc-macro-error-1-rs
proc-macro-error-attr-1-rs quote-1-rs syn-2-rs unicode-ident-1-rs'
sh -c 0.32s user 0.01s system 99% cpu 0.339 total
Oh I bet that 0.32secs is the overhead of python importing all modules
that meson uses. Running python 18 times, gives the 5 second overhead.
Oh, it actually is parallel!
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
executor = ThreadPoolExecutor(options.num_processes)
...
for wrap in wraps:
dirname = Path(source_dir, subproject_dir,
wrap.directory).as_posix()
runner = Runner(logger, r, wrap, dirname, options)
task = loop.run_in_executor(executor, runner.run)
tasks.append(task)
task_names.append(wrap.name)
results = loop.run_until_complete(asyncio.gather(*tasks))
and then Runner.run -> Runner.run_method -> Runner.download
