When using the -profile flag is it known behaviour that phobos unit tests fail?
When using the -profile flag is it known behaviour that phobos unit tests fail? (Ubuntu 16.04 - DMD64 D Compiler v2.071.0) For example, when following these steps I get a failed unit test: $ cd /usr/include/dmd/phobos/std/ $ rdmd -I/usr/include/dmd/phobos/std -I/usr/include/dmd/phobos/core -main -unittest -profile format.d
Re: phobos unit tests
On Saturday, 3 September 2016 at 17:16:24 UTC, Jonathan M Davis wrote: On Saturday, September 03, 2016 16:56:08 ZombineDev via Digitalmars-d wrote: You can just enable the unittests for a single instance that you know for sure that it will be used. For example: 1) https://github.com/dlang/phobos/blob/v2.071.2-b3/std/experimental/ndslice/sl ice.d#L808 2) https://github.com/dlang/phobos/blob/v2.071.2-b3/std/experimental/ndslice/sl ice.d#L947 So, in order to avoid having the unit tests compiled into the code of anyone who uses your template, you have to create a special version identifier that you use with your unit test build and your documentation build, and then you version the tests within the template with that version. And in order to avoid having the unit test compiled into every instantiation of that template during your test, you have to also put it in a static if for a specific instantiation? Sure, that's feasible, but that's getting _ugly_. - Jonathan M Davis I like and support your DIP about static unittests, I was just pointing out that there are workarounds, mainly for people like Manu, who need ways to get their job done now. Personally, I think that putting a single `static if (doUnittests)` before each unittest is not too much, considering how flexible the doUnittests condition can be (e.g. check if ddoc or unittest build, check template parameters, etc.).
Re: phobos unit tests
On Saturday, September 03, 2016 16:56:08 ZombineDev via Digitalmars-d wrote: > You can just enable the unittests for a single instance that you > know for sure that it will be used. For example: > 1) > https://github.com/dlang/phobos/blob/v2.071.2-b3/std/experimental/ndslice/sl > ice.d#L808 > > 2) > https://github.com/dlang/phobos/blob/v2.071.2-b3/std/experimental/ndslice/sl > ice.d#L947 So, in order to avoid having the unit tests compiled into the code of anyone who uses your template, you have to create a special version identifier that you use with your unit test build and your documentation build, and then you version the tests within the template with that version. And in order to avoid having the unit test compiled into every instantiation of that template during your test, you have to also put it in a static if for a specific instantiation? Sure, that's feasible, but that's getting _ugly_. - Jonathan M Davis
Re: phobos unit tests
On Saturday, 3 September 2016 at 15:58:51 UTC, Jonathan M Davis wrote: On Saturday, September 03, 2016 07:48:14 H. S. Teoh via Digitalmars-d wrote: > In any case, for now, I never put non-generic unit tests in > templates, and I reject PRs that have them. Sure, having to > copy-paste your examples sucks, but it doesn't affect the > code of everyone who uses the template, whereas ddoc-ed unit > tests do. [...] Actually you don't need to copy-paste your examples, which IMO is a bad idea to begin with. Just version out the non-generic unittests when compiling user code, and you can have the best of both worlds. And you still have the problem that all of those unit tests will be compiled into every instantiation of the template that's part of your unittest build. So, you end up with longer build times and longer run times for your unit tests. I agree that copy-pasting sucks, but it's what we had to do before with had ddoc-ed unit tests, and I still think that it's better than resulting in all of the additional copies of the tests being compiled in and run - especially if the project isn't small. - Jonathan M Davis No need for that - see my other post: http://forum.dlang.org/post/psrgjdlvsiukkuhre...@forum.dlang.org
Re: phobos unit tests
On Saturday, 3 September 2016 at 15:54:31 UTC, Jonathan M Davis wrote: On Saturday, September 03, 2016 15:06:33 Andrei Alexandrescu via Digitalmars-d wrote: On 9/3/16 5:36 AM, Manu via Digitalmars-d wrote: > This document: https://wiki.dlang.org/Contributing_to_Phobos > > States: "Avoid unittest in templates (it will generate a new > unittest for each instance) - put your tests outside" Actually that's a good thing, sometimes you do want to run a unittest for each instantiation. Sometimes, sure. But usually not. > Sounds reasonable, but then I realised that most of my unit > tests are documenting unittests... this recommendation is in > conflict with the documentation standards... who wins? Just version it only for the documentation. Then the tests won't be run unless your documentation build is also your unittest build, and they frequently can't be the same - e.g. when something differs between operating systems, and you have a separate D_Ddoc or StdDdoc block so that you can have proper documentation without requiring that the documentation be built on all systems and potentially being able to document what exists on each system. For instance, the supported clock types vary quite a bit across operating systems, so the core.time.ClockType enum has different values on different systems and version(CoreDdoc) is used so that the documentation can list them all and list the consistently across systems. You could add some sort of version identifier that's specific to your build that you use for your unittest builds (similar to how Phobos has StdDdoc separate from D_Ddoc), but even if you do that, you still have the problem of the non-generic unit tests within a template being compiled into every instantiation of the template that's part of your unittest build, and all of those tests will get run, increasing both the build times and the time it takes to run the unit tests. - Jonathan M Davis You can just enable the unittests for a single instance that you know for sure that it will be used. For example: 1) https://github.com/dlang/phobos/blob/v2.071.2-b3/std/experimental/ndslice/slice.d#L808 2) https://github.com/dlang/phobos/blob/v2.071.2-b3/std/experimental/ndslice/slice.d#L947
Re: phobos unit tests
On Saturday, September 03, 2016 07:48:14 H. S. Teoh via Digitalmars-d wrote: > > In any case, for now, I never put non-generic unit tests in templates, > > and I reject PRs that have them. Sure, having to copy-paste your > > examples sucks, but it doesn't affect the code of everyone who uses > > the template, whereas ddoc-ed unit tests do. > > [...] > > Actually you don't need to copy-paste your examples, which IMO is a bad > idea to begin with. Just version out the non-generic unittests when > compiling user code, and you can have the best of both worlds. And you still have the problem that all of those unit tests will be compiled into every instantiation of the template that's part of your unittest build. So, you end up with longer build times and longer run times for your unit tests. I agree that copy-pasting sucks, but it's what we had to do before with had ddoc-ed unit tests, and I still think that it's better than resulting in all of the additional copies of the tests being compiled in and run - especially if the project isn't small. - Jonathan M Davis
Re: phobos unit tests
On Saturday, September 03, 2016 15:06:33 Andrei Alexandrescu via Digitalmars-d wrote: > On 9/3/16 5:36 AM, Manu via Digitalmars-d wrote: > > This document: https://wiki.dlang.org/Contributing_to_Phobos > > > > States: "Avoid unittest in templates (it will generate a new unittest > > for each instance) - put your tests outside" > > Actually that's a good thing, sometimes you do want to run a unittest > for each instantiation. Sometimes, sure. But usually not. > > Sounds reasonable, but then I realised that most of my unit tests are > > documenting unittests... this recommendation is in conflict with the > > documentation standards... who wins? > > Just version it only for the documentation. Then the tests won't be run unless your documentation build is also your unittest build, and they frequently can't be the same - e.g. when something differs between operating systems, and you have a separate D_Ddoc or StdDdoc block so that you can have proper documentation without requiring that the documentation be built on all systems and potentially being able to document what exists on each system. For instance, the supported clock types vary quite a bit across operating systems, so the core.time.ClockType enum has different values on different systems and version(CoreDdoc) is used so that the documentation can list them all and list the consistently across systems. You could add some sort of version identifier that's specific to your build that you use for your unittest builds (similar to how Phobos has StdDdoc separate from D_Ddoc), but even if you do that, you still have the problem of the non-generic unit tests within a template being compiled into every instantiation of the template that's part of your unittest build, and all of those tests will get run, increasing both the build times and the time it takes to run the unit tests. - Jonathan M Davis
Re: phobos unit tests
On Sat, Sep 03, 2016 at 05:46:23AM -0700, Jonathan M Davis via Digitalmars-d
wrote:
> On Saturday, September 03, 2016 13:36:06 Manu via Digitalmars-d wrote:
> > This document: https://wiki.dlang.org/Contributing_to_Phobos
> >
> > States: "Avoid unittest in templates (it will generate a new
> > unittest for each instance) - put your tests outside"
> >
> > Sounds reasonable, but then I realised that most of my unit tests
> > are documenting unittests... this recommendation is in conflict with
> > the documentation standards... who wins?
>
> Well, do you want everyone who ever uses your template to have your
> unit tests in their code? Yes, having ddoc unit tests is great, but I
> for one definitely don't think that it's worth having the unit tests
> inside of the templates, since they end up in everyone's code, and I'd
> strongly argue that no unit tests that aren't meant to be generic
> should ever be in templates so long as the language works this way.
version(libraryBuild) { version = do_unittest; }
version(libraryDocs) { version = do_unittest; }
///
struct S(T) {
///
void method() { ... }
///
unittest { /* generic test here */ }
version(do_unittest)
///
unittest { /* non-generic test here */ }
}
> That's why I created DIP 82:
>
> http://wiki.dlang.org/DIP82
>
> Unfortunately, it didn't generate much discussion and hasn't been
> implemented yet (which is often the case with DIPs), and now I need to
> figure out how to migrate it to the new DIP stuff on github and do
> that, or it's definitely not going anywhere. But I think that this is
> definitely a case where a language change is needed.
It would certainly be nice, I agree. But there *are* ways of working
around this, even if they are a bit ugly (like above).
> In any case, for now, I never put non-generic unit tests in templates,
> and I reject PRs that have them. Sure, having to copy-paste your
> examples sucks, but it doesn't affect the code of everyone who uses
> the template, whereas ddoc-ed unit tests do.
[...]
Actually you don't need to copy-paste your examples, which IMO is a bad
idea to begin with. Just version out the non-generic unittests when
compiling user code, and you can have the best of both worlds.
In fact, versioning the unittests will also solve another problem: that
of not compiling library unittests when user code is being compiled with
-unittest (because the end user shouldn't need to bear the burden of
running unittests for the library just because they want to unittest
their own code).
T
--
Life is too short to run proprietary software. -- Bdale Garbee
Re: phobos unit tests
On Saturday, 3 September 2016 at 12:46:23 UTC, Jonathan M Davis wrote: so long as the language works this way. That's why I created DIP 82: http://wiki.dlang.org/DIP82 Unfortunately, it didn't generate much discussion and hasn't been implemented yet (which is often the case with DIPs), and now I need to figure out how to migrate it to the new DIP stuff on github and do that, or it's definitely not going anywhere. But I think that this is definitely a case where a language change is needed. There's a migration tool that can help you to convert from the D Wiki to Markdown: https://github.com/dlang/DIPs/tree/master/tools/dwikiquery run it like dub -- fetch --id 82 in case you don't have Pandoc installed, I have run a similar tool on most DIPs a couple of months ago, maybe that helps you: https://github.com/wilzbach/d-dip/tree/gh-pages/md Of course the conversion to Markdown isn't perfect and most probably requires a bit of manual tweaking. Also the header format between my crawl (yaml) and the new DIP repo (table) is different.
Re: phobos unit tests
On 9/3/16 5:36 AM, Manu via Digitalmars-d wrote: This document: https://wiki.dlang.org/Contributing_to_Phobos States: "Avoid unittest in templates (it will generate a new unittest for each instance) - put your tests outside" Actually that's a good thing, sometimes you do want to run a unittest for each instantiation. Sounds reasonable, but then I realised that most of my unit tests are documenting unittests... this recommendation is in conflict with the documentation standards... who wins? Just version it only for the documentation. Andrei
Re: phobos unit tests
On Saturday, September 03, 2016 13:36:06 Manu via Digitalmars-d wrote: > This document: https://wiki.dlang.org/Contributing_to_Phobos > > States: "Avoid unittest in templates (it will generate a new unittest > for each instance) - put your tests outside" > > Sounds reasonable, but then I realised that most of my unit tests are > documenting unittests... this recommendation is in conflict with the > documentation standards... who wins? Well, do you want everyone who ever uses your template to have your unit tests in their code? Yes, having ddoc unit tests is great, but I for one definitely don't think that it's worth having the unit tests inside of the templates, since they end up in everyone's code, and I'd strongly argue that no unit tests that aren't meant to be generic should ever be in templates so long as the language works this way. That's why I created DIP 82: http://wiki.dlang.org/DIP82 Unfortunately, it didn't generate much discussion and hasn't been implemented yet (which is often the case with DIPs), and now I need to figure out how to migrate it to the new DIP stuff on github and do that, or it's definitely not going anywhere. But I think that this is definitely a case where a language change is needed. In any case, for now, I never put non-generic unit tests in templates, and I reject PRs that have them. Sure, having to copy-paste your examples sucks, but it doesn't affect the code of everyone who uses the template, whereas ddoc-ed unit tests do. - Jonathan M Davis
Re: phobos unit tests
On Saturday, 3 September 2016 at 03:36:06 UTC, Manu wrote:
This document: https://wiki.dlang.org/Contributing_to_Phobos
States: "Avoid unittest in templates (it will generate a new
unittest for each instance) - put your tests outside"
Sounds reasonable, but then I realised that most of my unit
tests are documenting unittests... this recommendation is in
conflict with the documentation standards... who wins?
For the rationale see e.g. this thread:
https://forum.dlang.org/post/wnydmelyxprvhsxew...@forum.dlang.org
A trick around this is sth. like:
version(D_Ddoc)
{
enum doUnittest = true;
}
else
{
version(unittest)
{
enum doUnittest = true;
}
else
{
enum doUnittest = false;
}
}
and then `static if (doUnittest)` the tests inside. For example
ndslice uses a similar pattern.
phobos unit tests
This document: https://wiki.dlang.org/Contributing_to_Phobos States: "Avoid unittest in templates (it will generate a new unittest for each instance) - put your tests outside" Sounds reasonable, but then I realised that most of my unit tests are documenting unittests... this recommendation is in conflict with the documentation standards... who wins?
Re: When using the -profile flag is it known behaviour that phobos unit tests fail?
On 6/25/2016 12:52 AM, Gary Willoughby wrote: Done. Good!
Re: When using the -profile flag is it known behaviour that phobos unit tests fail?
On Friday, 24 June 2016 at 22:24:09 UTC, Walter Bright wrote: Please post bug reports to bugzilla. They'll get lost in the n.g. Done. https://issues.dlang.org/show_bug.cgi?id=16204
Re: When using the -profile flag is it known behaviour that phobos unit tests fail?
On 6/24/2016 10:50 AM, Gary Willoughby wrote: On Friday, 24 June 2016 at 17:36:49 UTC, Gary Willoughby wrote: When using the -profile flag is it known behaviour that phobos unit tests fail? (Ubuntu 16.04 - DMD64 D Compiler v2.071.0) For example, when following these steps I get a failed unit test: $ cd /usr/include/dmd/phobos/std/ $ rdmd -I/usr/include/dmd/phobos/std -I/usr/include/dmd/phobos/core -main -unittest -profile format.d Without the -profile flag it works. You may need to add the --force option to rdmd. $ rdmd --force -I/usr/include/dmd/phobos/std -I/usr/include/dmd/phobos/core -main -unittest -profile format.d Please post bug reports to bugzilla. They'll get lost in the n.g.
Re: When using the -profile flag is it known behaviour that phobos unit tests fail?
On Friday, 24 June 2016 at 17:36:49 UTC, Gary Willoughby wrote: When using the -profile flag is it known behaviour that phobos unit tests fail? (Ubuntu 16.04 - DMD64 D Compiler v2.071.0) For example, when following these steps I get a failed unit test: $ cd /usr/include/dmd/phobos/std/ $ rdmd -I/usr/include/dmd/phobos/std -I/usr/include/dmd/phobos/core -main -unittest -profile format.d Without the -profile flag it works. You may need to add the --force option to rdmd. $ rdmd --force -I/usr/include/dmd/phobos/std -I/usr/include/dmd/phobos/core -main -unittest -profile format.d
When using the -profile flag is it known behaviour that phobos unit tests fail?
When using the -profile flag is it known behaviour that phobos unit tests fail? (Ubuntu 16.04 - DMD64 D Compiler v2.071.0) For example, when following these steps I get a failed unit test: $ cd /usr/include/dmd/phobos/std/ $ rdmd -I/usr/include/dmd/phobos/std -I/usr/include/dmd/phobos/core -main -unittest -profile format.d Without the -profile flag it works.
Re: Running Phobos unit tests in threads: I have data
https://issues.dlang.org/show_bug.cgi?id=12708 On Sunday, 4 May 2014 at 16:07:30 UTC, Andrei Alexandrescu wrote: On 5/4/14, 1:44 AM, Atila Neves wrote: On Saturday, 3 May 2014 at 22:46:03 UTC, Andrei Alexandrescu wrote: On 5/3/14, 2:42 PM, Atila Neves wrote: gdc gave _very_ different results. I had to use different modules because at some point tests started failing, but with gdc the threaded version runs ~3x faster. On my own unit-threaded benchmarks, running the UTs for Cerealed over and over again was only slightly slower with threads than without. With dmd the threaded version was nearly 3x slower. Sounds like a severe bug in dmd or dependents. -- Andrei Seems like it. Just to be sure I swapped ld.gold for ld.bfd and the problem was still there. I'm not entirely sure how to file this bug: with just my simple example above? The simpler the better. -- Andrei
Re: Running Phobos unit tests in threads: I have data
On Monday, 5 May 2014 at 17:56:11 UTC, Dicebot wrote: On Saturday, 3 May 2014 at 12:26:13 UTC, Rikki Cattermole wrote: On Saturday, 3 May 2014 at 12:24:59 UTC, Atila Neves wrote: Out of curiosity are you on Windows? No, Arch Linux 64-bit. I also just noticed a glaring threading bug in my code as well that somehow's never turned up. This is not a good day. Atila I'm surprised. Threads should be cheap on Linux. Something funky is definitely going on I bet. Threads are never cheap. Regarding this, I found this talk interesting: https://www.youtube.com/watch?v=KXuZi9aeGTw
Re: Running Phobos unit tests in threads: I have data
On 5 May 2014 19:07, Orvid King via Digitalmars-d wrote: > Going to take a wild guess, but as core.atomic.casImpl will never be > inlined anywhere with DMD, due to it's inline assembly, you have the > cost of building and destroying a stack frame, the cost of passing the > args in, moving them into registers, saving potentially trashed > registers, etc. every time it even attempts to acquire a lock, and the > GC uses a single global lock for just about everything. As you can > imagine, I suspect this is far from optimal, and, if I remember right, > GDC uses intrinsics for the atomic operations. > Aye, and atomic intrinsics though they may be, it could even be improved by switching over to C++ atomic intrinsics, which map directly to core.atomics. :)
Re: Running Phobos unit tests in threads: I have data
Going to take a wild guess, but as core.atomic.casImpl will never be inlined anywhere with DMD, due to it's inline assembly, you have the cost of building and destroying a stack frame, the cost of passing the args in, moving them into registers, saving potentially trashed registers, etc. every time it even attempts to acquire a lock, and the GC uses a single global lock for just about everything. As you can imagine, I suspect this is far from optimal, and, if I remember right, GDC uses intrinsics for the atomic operations. On 5/5/14, Atila Neves via Digitalmars-d wrote: > On Sunday, 4 May 2014 at 17:01:23 UTC, safety0ff wrote: >> On Saturday, 3 May 2014 at 22:46:03 UTC, Andrei Alexandrescu >> wrote: >>> On 5/3/14, 2:42 PM, Atila Neves wrote: gdc gave _very_ different results. I had to use different modules because at some point tests started failing, but with gdc the threaded version runs ~3x faster. On my own unit-threaded benchmarks, running the UTs for Cerealed over and over again was only slightly slower with threads than without. With dmd the threaded version was nearly 3x slower. >>> >>> Sounds like a severe bug in dmd or dependents. -- Andrei >> >> This reminds me of when I was parallelizing a project euler >> solution: atomic access was so much slower on DMD that it made >> performance worse than the single threaded version for one >> stage of the program. >> >> I know that std.parallelism does make use of core.atomic under >> the hood, so this may be a factor when using DMD. > > Funny you should say that, a friend of mine tried porting a > lock-free algorithm of his from Java to D a few weeks ago. The D > version ran 3 orders of magnitude slower. Then I tried gdc and > ldc on his code. ldc produced code running at around 80% of the > speed of the Java version, fdc was around 30%. But dmd... >
Re: Running Phobos unit tests in threads: I have data
On Saturday, 3 May 2014 at 12:26:13 UTC, Rikki Cattermole wrote: On Saturday, 3 May 2014 at 12:24:59 UTC, Atila Neves wrote: Out of curiosity are you on Windows? No, Arch Linux 64-bit. I also just noticed a glaring threading bug in my code as well that somehow's never turned up. This is not a good day. Atila I'm surprised. Threads should be cheap on Linux. Something funky is definitely going on I bet. Threads are never cheap.
Re: Running Phobos unit tests in threads: I have data
On Sunday, 4 May 2014 at 17:01:23 UTC, safety0ff wrote: On Saturday, 3 May 2014 at 22:46:03 UTC, Andrei Alexandrescu wrote: On 5/3/14, 2:42 PM, Atila Neves wrote: gdc gave _very_ different results. I had to use different modules because at some point tests started failing, but with gdc the threaded version runs ~3x faster. On my own unit-threaded benchmarks, running the UTs for Cerealed over and over again was only slightly slower with threads than without. With dmd the threaded version was nearly 3x slower. Sounds like a severe bug in dmd or dependents. -- Andrei This reminds me of when I was parallelizing a project euler solution: atomic access was so much slower on DMD that it made performance worse than the single threaded version for one stage of the program. I know that std.parallelism does make use of core.atomic under the hood, so this may be a factor when using DMD. Funny you should say that, a friend of mine tried porting a lock-free algorithm of his from Java to D a few weeks ago. The D version ran 3 orders of magnitude slower. Then I tried gdc and ldc on his code. ldc produced code running at around 80% of the speed of the Java version, fdc was around 30%. But dmd...
Re: Running Phobos unit tests in threads: I have data
On Saturday, 3 May 2014 at 22:46:03 UTC, Andrei Alexandrescu wrote: On 5/3/14, 2:42 PM, Atila Neves wrote: gdc gave _very_ different results. I had to use different modules because at some point tests started failing, but with gdc the threaded version runs ~3x faster. On my own unit-threaded benchmarks, running the UTs for Cerealed over and over again was only slightly slower with threads than without. With dmd the threaded version was nearly 3x slower. Sounds like a severe bug in dmd or dependents. -- Andrei This reminds me of when I was parallelizing a project euler solution: atomic access was so much slower on DMD that it made performance worse than the single threaded version for one stage of the program. I know that std.parallelism does make use of core.atomic under the hood, so this may be a factor when using DMD.
Re: Running Phobos unit tests in threads: I have data
On 04/05/14 09:49, Russel Winder via Digitalmars-d wrote: (*) Physical cores are not necessarily the number reported by the OS due to core hyperthreads. Quad core no hyperthreads, and dual core, two hyperthreads per core, both get reported as four processor systems. However if you benchmark them you get very, very different performance characteristics. Yup. That bit me with a new laptop the first time I tried parallel programming with D :-)
Re: Running Phobos unit tests in threads: I have data
On 5/4/14, 3:06 AM, Russel Winder via Digitalmars-d wrote: On Sun, 2014-05-04 at 08:47 +, Atila Neves via Digitalmars-d wrote: Like I mentioned afterwards, I tried a different number of threads. On my machine, at least, std.parallelism.totalCPUs returns 8, the number of virtual cores. As it should. If you can create a small example of the problem, and I can remember how to run std.parallelism as a separate module, I can try and take a look at this later next week. This is an awesome offer, Russel. Thanks! -- Andrei
Re: Running Phobos unit tests in threads: I have data
On 5/4/14, 1:44 AM, Atila Neves wrote: On Saturday, 3 May 2014 at 22:46:03 UTC, Andrei Alexandrescu wrote: On 5/3/14, 2:42 PM, Atila Neves wrote: gdc gave _very_ different results. I had to use different modules because at some point tests started failing, but with gdc the threaded version runs ~3x faster. On my own unit-threaded benchmarks, running the UTs for Cerealed over and over again was only slightly slower with threads than without. With dmd the threaded version was nearly 3x slower. Sounds like a severe bug in dmd or dependents. -- Andrei Seems like it. Just to be sure I swapped ld.gold for ld.bfd and the problem was still there. I'm not entirely sure how to file this bug: with just my simple example above? The simpler the better. -- Andrei
Re: Running Phobos unit tests in threads: I have data
On Sun, 2014-05-04 at 08:47 +, Atila Neves via Digitalmars-d wrote: > Like I mentioned afterwards, I tried a different number of > threads. On my machine, at least, std.parallelism.totalCPUs > returns 8, the number of virtual cores. As it should. If you can create a small example of the problem, and I can remember how to run std.parallelism as a separate module, I can try and take a look at this later next week. -- Russel. = Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.win...@ekiga.net 41 Buckmaster Roadm: +44 7770 465 077 xmpp: rus...@winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Re: Running Phobos unit tests in threads: I have data
Like I mentioned afterwards, I tried a different number of threads. On my machine, at least, std.parallelism.totalCPUs returns 8, the number of virtual cores. As it should. Atila On Sunday, 4 May 2014 at 07:49:51 UTC, Russel Winder via Digitalmars-d wrote: On Sat, 2014-05-03 at 19:37 +, Atila Neves via Digitalmars-d wrote: […] I'm using parallel and taskPool from std.parallelism. I was under the impression it gave me a ready-to-use pool with as many threads as I have cores. There is a default, related to the number of cores the OS thinks there is (*), but you can also set the number manually. std.parallelism could do with some work to make it better than it already is. (*) Physical cores are not necessarily the number reported by the OS due to core hyperthreads. Quad core no hyperthreads, and dual core, two hyperthreads per core, both get reported as four processor systems. However if you benchmark them you get very, very different performance characteristics.
Re: Running Phobos unit tests in threads: I have data
On Saturday, 3 May 2014 at 22:46:03 UTC, Andrei Alexandrescu wrote: On 5/3/14, 2:42 PM, Atila Neves wrote: gdc gave _very_ different results. I had to use different modules because at some point tests started failing, but with gdc the threaded version runs ~3x faster. On my own unit-threaded benchmarks, running the UTs for Cerealed over and over again was only slightly slower with threads than without. With dmd the threaded version was nearly 3x slower. Sounds like a severe bug in dmd or dependents. -- Andrei Seems like it. Just to be sure I swapped ld.gold for ld.bfd and the problem was still there. I'm not entirely sure how to file this bug: with just my simple example above? Atila
Re: Running Phobos unit tests in threads: I have data
On Sat, 2014-05-03 at 19:37 +, Atila Neves via Digitalmars-d wrote: […] > I'm using parallel and taskPool from std.parallelism. I was under > the impression it gave me a ready-to-use pool with as many > threads as I have cores. There is a default, related to the number of cores the OS thinks there is (*), but you can also set the number manually. std.parallelism could do with some work to make it better than it already is. (*) Physical cores are not necessarily the number reported by the OS due to core hyperthreads. Quad core no hyperthreads, and dual core, two hyperthreads per core, both get reported as four processor systems. However if you benchmark them you get very, very different performance characteristics. -- Russel. = Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.win...@ekiga.net 41 Buckmaster Roadm: +44 7770 465 077 xmpp: rus...@winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Re: Running Phobos unit tests in threads: I have data
On 5/3/14, 2:42 PM, Atila Neves wrote: gdc gave _very_ different results. I had to use different modules because at some point tests started failing, but with gdc the threaded version runs ~3x faster. On my own unit-threaded benchmarks, running the UTs for Cerealed over and over again was only slightly slower with threads than without. With dmd the threaded version was nearly 3x slower. Sounds like a severe bug in dmd or dependents. -- Andrei
Re: Running Phobos unit tests in threads: I have data
Same thing with unit_threaded on Phobos, 3x faster even without
repeating the modules (0.1s vs 0.3s). Since the example is
shorter than the other one, I'll post it here in case anyone else
wants to try:
import unit_threaded.runner;
int main(string[] args) {
return args.runTests!(
"ustd.array",
"ustd.ascii",
"ustd.base64",
"ustd.bigint",
"ustd.bitmanip",
"ustd.concurrency",
"ustd.container",
"ustd.cstream",
);
}
On Saturday, 3 May 2014 at 21:42:13 UTC, Atila Neves wrote:
gdc gave _very_ different results. I had to use different
modules because at some point tests started failing, but with
gdc the threaded version runs ~3x faster.
On my own unit-threaded benchmarks, running the UTs for
Cerealed over and over again was only slightly slower with
threads than without. With dmd the threaded version was nearly
3x slower.
Atila
On Saturday, 3 May 2014 at 21:14:29 UTC, Atila Neves wrote:
if(single) {
foreach(test; tests) {
test();
}
} else {
foreach(test; tests.parallel) {
Try different batch size:
test.parallel(1), test.parallel(2) etc.
So as to not have thread creation be disproportionately
represented, I repeated the module list over and over again,
making the number of tests run equal to 9990. This takes 5s on
my machine to run in on thread and 12s in multiple. Here are
the things I tried:
1. Created my own TaskPool so I could decide how many threads
to use
2. Changed the batch size in parallel from 1 to 10 to 100 to
1000
3. Explicitly spawn two threads and tell each to do a foreach
on half of the tests
None of them made it go any faster. I had similar results
using unit-threaded on my own projects. This is weird.
Atila
Re: Running Phobos unit tests in threads: I have data
gdc gave _very_ different results. I had to use different modules
because at some point tests started failing, but with gdc the
threaded version runs ~3x faster.
On my own unit-threaded benchmarks, running the UTs for Cerealed
over and over again was only slightly slower with threads than
without. With dmd the threaded version was nearly 3x slower.
Atila
On Saturday, 3 May 2014 at 21:14:29 UTC, Atila Neves wrote:
if(single) {
foreach(test; tests) {
test();
}
} else {
foreach(test; tests.parallel) {
Try different batch size:
test.parallel(1), test.parallel(2) etc.
So as to not have thread creation be disproportionately
represented, I repeated the module list over and over again,
making the number of tests run equal to 9990. This takes 5s on
my machine to run in on thread and 12s in multiple. Here are
the things I tried:
1. Created my own TaskPool so I could decide how many threads
to use
2. Changed the batch size in parallel from 1 to 10 to 100 to
1000
3. Explicitly spawn two threads and tell each to do a foreach
on half of the tests
None of them made it go any faster. I had similar results using
unit-threaded on my own projects. This is weird.
Atila
Re: Running Phobos unit tests in threads: I have data
if(single) {
foreach(test; tests) {
test();
}
} else {
foreach(test; tests.parallel) {
Try different batch size:
test.parallel(1), test.parallel(2) etc.
So as to not have thread creation be disproportionately
represented, I repeated the module list over and over again,
making the number of tests run equal to 9990. This takes 5s on my
machine to run in on thread and 12s in multiple. Here are the
things I tried:
1. Created my own TaskPool so I could decide how many threads to
use
2. Changed the batch size in parallel from 1 to 10 to 100 to 1000
3. Explicitly spawn two threads and tell each to do a foreach on
half of the tests
None of them made it go any faster. I had similar results using
unit-threaded on my own projects. This is weird.
Atila
Re: Running Phobos unit tests in threads: I have data
03-May-2014 21:22, Atila Neves пишет:
I can reproduce the slower-with-threads issue without using my library.
I've included the source file below and would like to know if other
people see the same thing.
The Phobos modules are all called "ustd" because I couldn't/didn't know
how to get this to work otherwise. So I copied the std/*.d files to a
directory called ustd and changed their module declarations. Silly but
it works. I'd love to know how to do this properly.
[snip]
if(single) {
foreach(test; tests) {
test();
}
} else {
foreach(test; tests.parallel) {
Try different batch size:
test.parallel(1), test.parallel(2) etc.
--
Dmitry Olshansky
Re: Running Phobos unit tests in threads: I have data
On Saturday, 3 May 2014 at 18:16:52 UTC, Andrei Alexandrescu wrote: On 5/3/14, 4:54 AM, Atila Neves wrote: So I tried using unit-threaded to run Phobos unit tests [snip] Thanks. Are you using thread pooling (a limited number of threads e.g. 1.5 * cores running all unittests)? -- Andrei I'm using parallel and taskPool from std.parallelism. I was under the impression it gave me a ready-to-use pool with as many threads as I have cores.
Re: Running Phobos unit tests in threads: I have data
On Saturday, 3 May 2014 at 18:26:37 UTC, Walter Bright wrote: On 5/3/2014 10:22 AM, Atila Neves wrote: I can reproduce the slower-with-threads issue without using my library. I've included the source file below and would like to know if other people see the same thing. I haven't investigated this, but my suspicions are: 1. thread creation/destruction is dominating the times. In the current measurements probably since the whole run takes less than a second. But the first ones I did were dozens of seconds long, so I don't think so. 2. since very few of the unittests block, there is no speed advantage from having more threads than cores. If you limit the number of threads to the number of cores on your machine, you might see a speedup. Like I mentioned above, unless I'm mistaken taskPool should be using a correct number of threads for my machine already.
Re: Running Phobos unit tests in threads: I have data
On 5/3/2014 10:22 AM, Atila Neves wrote: I can reproduce the slower-with-threads issue without using my library. I've included the source file below and would like to know if other people see the same thing. I haven't investigated this, but my suspicions are: 1. thread creation/destruction is dominating the times. 2. since very few of the unittests block, there is no speed advantage from having more threads than cores. If you limit the number of threads to the number of cores on your machine, you might see a speedup.
Re: Running Phobos unit tests in threads: I have data
On 5/3/2014 5:26 AM, Rikki Cattermole wrote: Something funky is definitely going on I bet. No doubt: http://www.youtube.com/watch?v=aZcbDESaxhY
Re: Running Phobos unit tests in threads: I have data
On 5/3/14, 4:54 AM, Atila Neves wrote: So I tried using unit-threaded to run Phobos unit tests [snip] Thanks. Are you using thread pooling (a limited number of threads e.g. 1.5 * cores running all unittests)? -- Andrei
Re: Running Phobos unit tests in threads: I have data
I can reproduce the slower-with-threads issue without using my
library. I've included the source file below and would like to
know if other people see the same thing.
The Phobos modules are all called "ustd" because I
couldn't/didn't know how to get this to work otherwise. So I
copied the std/*.d files to a directory called ustd and changed
their module declarations. Silly but it works. I'd love to know
how to do this properly.
With this file, I consistenly get faster times with "-s" (for
single-threaded) than without (multi-threaded):
import std.parallelism;
import std.getopt;
import ustd.array;
import ustd.ascii;
import ustd.base64;
import ustd.bigint;
import ustd.bitmanip;
import ustd.concurrency;
import ustd.container;
import ustd.cstream;
alias TestFunction = void function();
auto getTests(Modules...)() {
TestFunction[] tests;
foreach(mod; Modules) {
foreach(test; __traits(getUnitTests, mod)) {
tests ~= &test;
}
}
return tests;
}
void main(string[] args) {
bool single;
getopt(args,
"single|s", &single
);
enum tests = getTests!(
ustd.array,
ustd.ascii,
ustd.base64,
ustd.bigint,
ustd.bitmanip,
ustd.concurrency,
ustd.container,
ustd.cstream,
);
if(single) {
foreach(test; tests) {
test();
}
} else {
foreach(test; tests.parallel) {
test();
}
}
}
Re: Running Phobos unit tests in threads: I have data
Ok, so I went and added __traits(getUnitTests) to unit-threaded. That way each unittest block is its own test case. I registered these modules in std to run: array, ascii, base64, bigint, bitmanip, concurrency, container, cstream. On the good news front, they all passed even though they were running concurrently. On the bad news front, single-threaded operation was still faster (0.22s vs 0.28s). I still don't know why. I fixed my concurrency bug, now I'm using taskPool.amap. Atila
Re: Running Phobos unit tests in threads: I have data
On Saturday, 3 May 2014 at 12:24:59 UTC, Atila Neves wrote: Out of curiosity are you on Windows? No, Arch Linux 64-bit. I also just noticed a glaring threading bug in my code as well that somehow's never turned up. This is not a good day. Atila I'm surprised. Threads should be cheap on Linux. Something funky is definitely going on I bet.
Re: Running Phobos unit tests in threads: I have data
On Saturday, 3 May 2014 at 12:08:56 UTC, Atila Neves wrote:
I turned off all output to check. It was still slower with
multiple threads. That was the only "weird" thing I was doing I
could think of as the cause. Otherwise it's just a
foreach(test; tests.parallel) { test(); }.
Atila
On Saturday, 3 May 2014 at 11:54:55 UTC, Atila Neves wrote:
So I tried using unit-threaded to run Phobos unit tests again
and had problems (which I'll look into later) with its
compile-time reflection. Then I realised I was an idiot since
I don't need to reflect on anything: all Phobos tests are in
unittest blocks so all I need to do is include them in the
build and unit-threaded will run them for me.
I tried a basic sanity check by running them in one thread
only with the -s option and got a segfault, and a failing test
before that. None of this should happen, and I'll be taking a
look at that as well.
But I carried on by removing the troublesome modules from the
build. These turned out to be:
std.datetime (fails)
std.process (fails and causes the segfault)
std.stdio (fails)
All the others pass in single threaded mode. After this I
tried using threads and std.parallelism failed, so I took that
away from the build as well.
Another thing to mention is that although the tests are
running in threads, since when I wrote the library the
getUnitTests __traits wasn't available (and since then I
wasn't interested in using it), each module's unit tests run
as one test. So they only interleave with other modules, not
with each other.
Running in one thread took 39 +/- 1 seconds.
Running in 8 threads took... ~41 seconds.
Oops. I noticed some tests take a lot longer so I tried
removing those. They were:
std.file
std.conv
std.regex
std.random
std.container
std.xml
std.utf
std.numeric
std.uuid
std.exception
I also removed any modules that were likely to be problematic
like std.concurrency and std.socket. With the reduced sample
size the results were:
1 thread: ~1.9s
8 threads: 4.1s +/- 0.2
So the whole threading thing isn't looking so great. Or at
least not how I implemented it. This got me thinking about my
own projects. The tests run so fast I never really paid
attention to how fast they were running. I compared running
the unit tests in Cerealed in one or more threads and got the
same result: running in one thread was faster.
I have to look to be sure but maybe the bottleneck is output.
As in actually printing the results to the screen. I had to
jump through a few hoops to make sure the output wasn't
interleaved, and in the end decided to have one thread be
responsible for that, with the tests sending it output
messages.
For reference, I copied all of the std/*.d modules into a
local std directory, compiled all of them with dmd -unittest
-c, then used this as the build command:
dmd -unittest -I~/coding/d/unit-threaded/source ut.d
std/algorithm.o std/array.o std/ascii.o std/base64.o
std/bigint.o std/bitmanip.o std/compiler.o std/complex.o
std/container.o std/cstream.o std/csv.o std/demangle.o
std/encoding.o std/format.o std/functional.o std/getopt.o
std/json.o std/math.o std/mathspecial.o std/metastrings.o
std/mmfile.o std/numeric.o std/outbuffer.o std/range.o
std/signals.o std/stdint.o std/stdiobase.o std/stream.o
std/string.o std/syserror.o std/system.o std/traits.o
std/typecons.o std/typelist.o std/typetuple.o std/uri.o
std/variant.o std/zip.o std/zlib.o libunit-threaded.a
-ofphobos_ut
I got libunit-threaded.a by running "dub build" in the root
directory of unit-threaded.
I might just implement a random order option now. Hmm.
Atila
Out of curiosity are you on Windows?
Re: Running Phobos unit tests in threads: I have data
Out of curiosity are you on Windows? No, Arch Linux 64-bit. I also just noticed a glaring threading bug in my code as well that somehow's never turned up. This is not a good day. Atila
Re: Running Phobos unit tests in threads: I have data
I turned off all output to check. It was still slower with
multiple threads. That was the only "weird" thing I was doing I
could think of as the cause. Otherwise it's just a foreach(test;
tests.parallel) { test(); }.
Atila
On Saturday, 3 May 2014 at 11:54:55 UTC, Atila Neves wrote:
So I tried using unit-threaded to run Phobos unit tests again
and had problems (which I'll look into later) with its
compile-time reflection. Then I realised I was an idiot since I
don't need to reflect on anything: all Phobos tests are in
unittest blocks so all I need to do is include them in the
build and unit-threaded will run them for me.
I tried a basic sanity check by running them in one thread only
with the -s option and got a segfault, and a failing test
before that. None of this should happen, and I'll be taking a
look at that as well.
But I carried on by removing the troublesome modules from the
build. These turned out to be:
std.datetime (fails)
std.process (fails and causes the segfault)
std.stdio (fails)
All the others pass in single threaded mode. After this I tried
using threads and std.parallelism failed, so I took that away
from the build as well.
Another thing to mention is that although the tests are running
in threads, since when I wrote the library the getUnitTests
__traits wasn't available (and since then I wasn't interested
in using it), each module's unit tests run as one test. So they
only interleave with other modules, not with each other.
Running in one thread took 39 +/- 1 seconds.
Running in 8 threads took... ~41 seconds.
Oops. I noticed some tests take a lot longer so I tried
removing those. They were:
std.file
std.conv
std.regex
std.random
std.container
std.xml
std.utf
std.numeric
std.uuid
std.exception
I also removed any modules that were likely to be problematic
like std.concurrency and std.socket. With the reduced sample
size the results were:
1 thread: ~1.9s
8 threads: 4.1s +/- 0.2
So the whole threading thing isn't looking so great. Or at
least not how I implemented it. This got me thinking about my
own projects. The tests run so fast I never really paid
attention to how fast they were running. I compared running the
unit tests in Cerealed in one or more threads and got the same
result: running in one thread was faster.
I have to look to be sure but maybe the bottleneck is output.
As in actually printing the results to the screen. I had to
jump through a few hoops to make sure the output wasn't
interleaved, and in the end decided to have one thread be
responsible for that, with the tests sending it output messages.
For reference, I copied all of the std/*.d modules into a local
std directory, compiled all of them with dmd -unittest -c, then
used this as the build command:
dmd -unittest -I~/coding/d/unit-threaded/source ut.d
std/algorithm.o std/array.o std/ascii.o std/base64.o
std/bigint.o std/bitmanip.o std/compiler.o std/complex.o
std/container.o std/cstream.o std/csv.o std/demangle.o
std/encoding.o std/format.o std/functional.o std/getopt.o
std/json.o std/math.o std/mathspecial.o std/metastrings.o
std/mmfile.o std/numeric.o std/outbuffer.o std/range.o
std/signals.o std/stdint.o std/stdiobase.o std/stream.o
std/string.o std/syserror.o std/system.o std/traits.o
std/typecons.o std/typelist.o std/typetuple.o std/uri.o
std/variant.o std/zip.o std/zlib.o libunit-threaded.a
-ofphobos_ut
I got libunit-threaded.a by running "dub build" in the root
directory of unit-threaded.
I might just implement a random order option now. Hmm.
Atila
Running Phobos unit tests in threads: I have data
So I tried using unit-threaded to run Phobos unit tests again and had problems (which I'll look into later) with its compile-time reflection. Then I realised I was an idiot since I don't need to reflect on anything: all Phobos tests are in unittest blocks so all I need to do is include them in the build and unit-threaded will run them for me. I tried a basic sanity check by running them in one thread only with the -s option and got a segfault, and a failing test before that. None of this should happen, and I'll be taking a look at that as well. But I carried on by removing the troublesome modules from the build. These turned out to be: std.datetime (fails) std.process (fails and causes the segfault) std.stdio (fails) All the others pass in single threaded mode. After this I tried using threads and std.parallelism failed, so I took that away from the build as well. Another thing to mention is that although the tests are running in threads, since when I wrote the library the getUnitTests __traits wasn't available (and since then I wasn't interested in using it), each module's unit tests run as one test. So they only interleave with other modules, not with each other. Running in one thread took 39 +/- 1 seconds. Running in 8 threads took... ~41 seconds. Oops. I noticed some tests take a lot longer so I tried removing those. They were: std.file std.conv std.regex std.random std.container std.xml std.utf std.numeric std.uuid std.exception I also removed any modules that were likely to be problematic like std.concurrency and std.socket. With the reduced sample size the results were: 1 thread: ~1.9s 8 threads: 4.1s +/- 0.2 So the whole threading thing isn't looking so great. Or at least not how I implemented it. This got me thinking about my own projects. The tests run so fast I never really paid attention to how fast they were running. I compared running the unit tests in Cerealed in one or more threads and got the same result: running in one thread was faster. I have to look to be sure but maybe the bottleneck is output. As in actually printing the results to the screen. I had to jump through a few hoops to make sure the output wasn't interleaved, and in the end decided to have one thread be responsible for that, with the tests sending it output messages. For reference, I copied all of the std/*.d modules into a local std directory, compiled all of them with dmd -unittest -c, then used this as the build command: dmd -unittest -I~/coding/d/unit-threaded/source ut.d std/algorithm.o std/array.o std/ascii.o std/base64.o std/bigint.o std/bitmanip.o std/compiler.o std/complex.o std/container.o std/cstream.o std/csv.o std/demangle.o std/encoding.o std/format.o std/functional.o std/getopt.o std/json.o std/math.o std/mathspecial.o std/metastrings.o std/mmfile.o std/numeric.o std/outbuffer.o std/range.o std/signals.o std/stdint.o std/stdiobase.o std/stream.o std/string.o std/syserror.o std/system.o std/traits.o std/typecons.o std/typelist.o std/typetuple.o std/uri.o std/variant.o std/zip.o std/zlib.o libunit-threaded.a -ofphobos_ut I got libunit-threaded.a by running "dub build" in the root directory of unit-threaded. I might just implement a random order option now. Hmm. Atila
Re: Phobos unit tests, unreadable code
On 2013-06-12 10:55, monarch_dodra wrote: Do you have a link to the actual problem? This, for example: https://github.com/jacob-carlborg/orange/blob/master/orange/serialization/Serializer.d#L1217 I could probably add an else. -- /Jacob Carlborg
Re: Phobos unit tests, unreadable code
On Wednesday, 12 June 2013 at 08:49:36 UTC, Jacob Carlborg wrote:
I'm merging the orange (std.serialization) unit tests with the
test of Phobos. I hit a problem. I'm getting a warning about
unreadable code but it's clear the code is reachable in some
cases because there's a static-if involved. The unit tests
won't run if there's a warning. How can we solve this?
The problem is that there must be a static if branch that is
shortcicuiting the code, right? Eg:
unittest
{
static if (somecondition) return;
unreachable_code
}
The compiler error (AFAIK) for "unreachable code" is not generic
to the code, but only appears for the specific types that makes
the unreachable_code actually unreachable. I've hit this a couple
of times in algorithm's unittests.
The workaround is usally to use static else and/or assert(0), and
to make sure that regardless of types, there is never any code
inserted that could be un-used.
Do you have a link to the actual problem?
Phobos unit tests, unreadable code
I'm merging the orange (std.serialization) unit tests with the test of Phobos. I hit a problem. I'm getting a warning about unreadable code but it's clear the code is reachable in some cases because there's a static-if involved. The unit tests won't run if there's a warning. How can we solve this? -- /Jacob Carlborg
