I turned off all output to check. It was still slower with
multiple threads. That was the only "weird" thing I was doing I
could think of as the cause. Otherwise it's just a foreach(test;
tests.parallel) { test(); }.
Atila
On Saturday, 3 May 2014 at 11:54:55 UTC, Atila Neves wrote:
So I tried using unit-threaded to run Phobos unit tests again
and had problems (which I'll look into later) with its
compile-time reflection. Then I realised I was an idiot since I
don't need to reflect on anything: all Phobos tests are in
unittest blocks so all I need to do is include them in the
build and unit-threaded will run them for me.
I tried a basic sanity check by running them in one thread only
with the -s option and got a segfault, and a failing test
before that. None of this should happen, and I'll be taking a
look at that as well.
But I carried on by removing the troublesome modules from the
build. These turned out to be:
std.datetime (fails)
std.process (fails and causes the segfault)
std.stdio (fails)
All the others pass in single threaded mode. After this I tried
using threads and std.parallelism failed, so I took that away
from the build as well.
Another thing to mention is that although the tests are running
in threads, since when I wrote the library the getUnitTests
__traits wasn't available (and since then I wasn't interested
in using it), each module's unit tests run as one test. So they
only interleave with other modules, not with each other.
Running in one thread took 39 +/- 1 seconds.
Running in 8 threads took... ~41 seconds.
Oops. I noticed some tests take a lot longer so I tried
removing those. They were:
std.file
std.conv
std.regex
std.random
std.container
std.xml
std.utf
std.numeric
std.uuid
std.exception
I also removed any modules that were likely to be problematic
like std.concurrency and std.socket. With the reduced sample
size the results were:
1 thread: ~1.9s
8 threads: 4.1s +/- 0.2
So the whole threading thing isn't looking so great. Or at
least not how I implemented it. This got me thinking about my
own projects. The tests run so fast I never really paid
attention to how fast they were running. I compared running the
unit tests in Cerealed in one or more threads and got the same
result: running in one thread was faster.
I have to look to be sure but maybe the bottleneck is output.
As in actually printing the results to the screen. I had to
jump through a few hoops to make sure the output wasn't
interleaved, and in the end decided to have one thread be
responsible for that, with the tests sending it output messages.
For reference, I copied all of the std/*.d modules into a local
std directory, compiled all of them with dmd -unittest -c, then
used this as the build command:
dmd -unittest -I~/coding/d/unit-threaded/source ut.d
std/algorithm.o std/array.o std/ascii.o std/base64.o
std/bigint.o std/bitmanip.o std/compiler.o std/complex.o
std/container.o std/cstream.o std/csv.o std/demangle.o
std/encoding.o std/format.o std/functional.o std/getopt.o
std/json.o std/math.o std/mathspecial.o std/metastrings.o
std/mmfile.o std/numeric.o std/outbuffer.o std/range.o
std/signals.o std/stdint.o std/stdiobase.o std/stream.o
std/string.o std/syserror.o std/system.o std/traits.o
std/typecons.o std/typelist.o std/typetuple.o std/uri.o
std/variant.o std/zip.o std/zlib.o libunit-threaded.a
-ofphobos_ut
I got libunit-threaded.a by running "dub build" in the root
directory of unit-threaded.
I might just implement a random order option now. Hmm.
Atila