Re: Quit running foreign unittests _
On Thursday, 30 April 2015 at 02:26:19 UTC, Steven Schveighoffer wrote: On 4/29/15 7:43 AM, Dicebot wrote: On Wednesday, 29 April 2015 at 04:53:47 UTC, Steven Schveighoffer wrote: or rdmd -main -unittest - fail to build because the templated unit test doesn't work on your code. Good luck with that. I will create an upstream PR to fix it, problem solved. Have never had a need to do so though, not even a single time. Also : can you please point me again what part of RBT causes compilation slowdowns with version(unittest)? I looked through and found only runtime checks. It's runtime checks that slow down, the unit test compilation is not slow (although it was at one point, but that was for the proper full unit tests). Then test separation + custom runtime filter fixes the issue without language changes. And for that move out of the aggregate + runtime test filtering does what you want. Move out of the aggregate makes it oh so ugly. I want my unit tests right where they belong. Runtime test filtering seems like it adds more complexity to the system unnecessarily. All I need is a way to say yes compiler, I want to run those templated unit tests I imported or no compiler, I don't want to do that. What the default is really isn't important. It is exactly the other way around. Implementing such compile-time filtering would mean adding totally new feature to the language, considerably complicating version system which is very simple right now. And remembering all the issues we had with -allinst it will take a while to figure out. You need something better than ugly to justify such language change. Especially when there is a working solution within existing rules - even if it seems ugly, it is achievable without language changes and addresses the issue.
Re: Quit running foreign unittests _
On Wednesday, 29 April 2015 at 15:00:53 UTC, Dicebot wrote: The core of Nick proposal (and what he does in his own libraries) is to not compile tests of dependnecies even when those are all compiled at once in -unittest mode. If they are compiled in unittest mode, all their unittests are compiled and run. What's problem? He uses `version(MyLibUnittest)` to disable all those completely and to run all tests you need to provide full set of such version flags for each dependency (transitively). This is what affects me and what I don't see implemented as a default. If the enhancement is implemented, that hack with versions won't be needed, all tests will be plain unittests, which will suit both you and Nick.
Re: Quit running foreign unittests _
On 04/28/2015 07:04 PM, Dicebot wrote: I do want do test everything as part of my app tests, including all possible dependencies, transitively. Even Phobos? This is awesome default. With a simple `rdmd -main -unittest` call I can ensure that certain app/module works correctly without having to trust maintainers of dependencies to run tests regularly and without even knowing what those dependencies are. It is beautiful in its simplicity which makes it good default.
Re: Quit running foreign unittests _
On Wednesday, 29 April 2015 at 04:53:47 UTC, Steven Schveighoffer wrote: or rdmd -main -unittest - fail to build because the templated unit test doesn't work on your code. Good luck with that. I will create an upstream PR to fix it, problem solved. Have never had a need to do so though, not even a single time. Also : can you please point me again what part of RBT causes compilation slowdowns with version(unittest)? I looked through and found only runtime checks. And for that move out of the aggregate + runtime test filtering does what you want. Again, there are so many reasons I should not have to worry about unit tests in my library being run with your code. That's on you. I didn't write it for your code to run, if you want to run it, run my unit test script. If you don't wan't to run it, filter it out in the test runner. I assure you, there are at least as much reasons why I shouldn't worry if you actually run tests for your library and how those need to be run. Both defaults can be circumvented.
Re: Quit running foreign unittests _
On Wednesday, 29 April 2015 at 07:44:17 UTC, Kagamin wrote: On Tuesday, 28 April 2015 at 16:40:05 UTC, Dicebot wrote: Those are two points I fundamentally disagree with. It doesn't matter where the code comes from - in the end only thing that matters is correctness of your application as a whole. 3rd party libraries are supposed to be tested already, if you want to test it, you should go and properly run its test suite. Whatever template unittests you accidentally instantiated in your code mean nothing with respect to overall correctness. And software is supposed to not have bugs, right. I can put some trust in regular testing of Phobos but it ends there. If accidental template tests from user code break any 3d party library and its author refuses to accept PR with the fix I will simply call it broken and fork. And if tests are written in a weird way that they can only be ran within that library test step, those are not really unittests. The library can be tested only when it's compiled in unittest mode as a whole. When you link with its release version its unittests are not even compiled at all. Which is exactly why all source builds with same flags is the only reasonable compilation model for D. Providing release static libraries causes only problems and never benefits. Even for Phobos it feels like a mistake in the long run.
Re: Quit running foreign unittests _
On 2014-09-10 04:13, Nick Sabalausky wrote: This is getting to be (or rather, *continuing* to be) a royal PITA: https://github.com/rejectedsoftware/vibe.d/issues/673 I don't mean to pick on Vibe.d in particular, but can we have a solution (that doesn't involve obscure corners of druntime, or demanding everyone use the same build system) for running our unittests *without* triggering a cascading avalanche of unittests from ALL third party libs that don't cooperate with the [frankly] quite clumsy version(mylib_unittests) hack?! The most simple solution that ever body will hate (except me), put the unit tests in a separate directory. -- /Jacob Carlborg
Re: Quit running foreign unittests _
On Tuesday, 28 April 2015 at 16:40:05 UTC, Dicebot wrote: Those are two points I fundamentally disagree with. It doesn't matter where the code comes from - in the end only thing that matters is correctness of your application as a whole. 3rd party libraries are supposed to be tested already, if you want to test it, you should go and properly run its test suite. Whatever template unittests you accidentally instantiated in your code mean nothing with respect to overall correctness. And if tests are written in a weird way that they can only be ran within that library test step, those are not really unittests. The library can be tested only when it's compiled in unittest mode as a whole. When you link with its release version its unittests are not even compiled at all.
Re: Quit running foreign unittests _
On Wednesday, 29 April 2015 at 11:46:57 UTC, Dicebot wrote: Which is exactly why all source builds with same flags is the only reasonable compilation model for D. If you compile all libraries in unittest mode, you get full set of unit tests from those libraries and the proposed change doesn't affect you in the least, it only affects linking with a library compiled in release mode.
Re: Quit running foreign unittests _
On 4/29/15 7:43 AM, Dicebot wrote: On Wednesday, 29 April 2015 at 04:53:47 UTC, Steven Schveighoffer wrote: or rdmd -main -unittest - fail to build because the templated unit test doesn't work on your code. Good luck with that. I will create an upstream PR to fix it, problem solved. Have never had a need to do so though, not even a single time. Also : can you please point me again what part of RBT causes compilation slowdowns with version(unittest)? I looked through and found only runtime checks. It's runtime checks that slow down, the unit test compilation is not slow (although it was at one point, but that was for the proper full unit tests). And for that move out of the aggregate + runtime test filtering does what you want. Move out of the aggregate makes it oh so ugly. I want my unit tests right where they belong. Runtime test filtering seems like it adds more complexity to the system unnecessarily. All I need is a way to say yes compiler, I want to run those templated unit tests I imported or no compiler, I don't want to do that. What the default is really isn't important. Again, there are so many reasons I should not have to worry about unit tests in my library being run with your code. That's on you. I didn't write it for your code to run, if you want to run it, run my unit test script. If you don't wan't to run it, filter it out in the test runner. I assure you, there are at least as much reasons why I shouldn't worry if you actually run tests for your library and how those need to be run. Both defaults can be circumvented. I don't *care* if you run it. But the compiler shouldn't force you to. The funny thing is, you DON'T run imported non-template unit tests, because the compiler doesn't make you (and actually doesn't allow you to). So why should this be any different? A unit test is supposed to test the unit, not the world. -Steve
Re: Quit running foreign unittests _
On Wednesday, 29 April 2015 at 14:10:32 UTC, Kagamin wrote: On Wednesday, 29 April 2015 at 11:46:57 UTC, Dicebot wrote: Which is exactly why all source builds with same flags is the only reasonable compilation model for D. If you compile all libraries in unittest mode, you get full set of unit tests from those libraries and the proposed change doesn't affect you in the least, it only affects linking with a library compiled in release mode. The core of Nick proposal (and what he does in his own libraries) is to not compile tests of dependnecies even when those are all compiled at once in -unittest mode. He uses `version(MyLibUnittest)` to disable all those completely and to run all tests you need to provide full set of such version flags for each dependency (transitively). This is what affects me and what I don't see implemented as a default. (P.S. release and -unittest are not exclusive in D, I presume you have meant release non-test mode :P)
Re: Quit running foreign unittests _
On Tuesday, 28 April 2015 at 21:28:09 UTC, Steven Schveighoffer
wrote:
I think by default, nobody wants to test already-tested code in
their application. That's not the point of unit tests.
How many more times should I repeat that I am exactly that nobody?
I do want do test everything as part of my app tests, including
all possible dependencies, transitively. This is awesome default.
With a simple `rdmd -main -unittest` call I can ensure that
certain app/module works correctly without having to trust
maintainers of dependencies to run tests regularly and without
even knowing what those dependencies are. It is beautiful in its
simplicity which makes it good default.
struct S(T)
{
unittest {...}
}
unittest
{
S!int; // run unit tests for int
}
If this a very slow test (and there are expected many S) simply
put the test blocks out of the aggregate.
Now, if I want to run unit tests for S!MyCustomType, that makes
sense. The library didn't test that. Which is why there should
be a way to do it (if it's valid!).
There is something fundamentally broken with a template that
needs to be tested for each new user type argument. Built-in
tests must cover all possible type classes or they are incomplete.
Unless compiling some specific tests causes some proven
_compilation_
slowdown (I have yet to see that) those all must be compiled
and
filtered by runtime test runner optionally.
For example, RedBlackTree when running unit tests does a sanity
check of the tree for every operation. If you then use a
RedBlackTree in your unit tests, you are doing that again.
Maybe it's worth it to you, but it can increase the runtime of
your test by orders of magnitude. Last time I checked,
performance was high on people's priority lists.
It must be very slow sanity checks :X Sounds weird but I will
accept it as given. Move the tests out of the RBL definition then.
Also performance has nothing in common with test performance. I
can't comment about rest because from the very beginning you seem
to make statements about testing in general which do not match my
vision at all.
Re: Quit running foreign unittests _
On Tuesday, 28 April 2015 at 20:57:01 UTC, Ivan Kazmenko wrote: Then how do you propose to approach the containers problem? On one hand, a unittest on containers themselves involves testing the container for integrity after each operation. On the other hand, a unittest on another module may involve heavy use of containers (say, N operations with a container), and if integrity checks are enabled at this time, it totals to N^2 trivial operations which may not be feasible. If it is that slow I tend to put such tests outside of the tested symbol so that it won't be repeated over and over again for templates. There is still full access to private symbols so it is always possible. Though that is very exceptional case reserved only for most critical cases (everything below 30-60 seconds total is ok to me)
Re: Quit running foreign unittests _
On 4/28/15 7:04 PM, Dicebot wrote: On Tuesday, 28 April 2015 at 21:28:09 UTC, Steven Schveighoffer wrote: I think by default, nobody wants to test already-tested code in their application. That's not the point of unit tests. How many more times should I repeat that I am exactly that nobody? OK, sorry. nodbody-1 :) I do want do test everything as part of my app tests, including all possible dependencies, transitively. This is awesome default. With a simple `rdmd -main -unittest` call I can ensure that certain app/module works correctly without having to trust maintainers of dependencies to run tests regularly and without even knowing what those dependencies are. It is beautiful in its simplicity which makes it good default. or rdmd -main -unittest - fail to build because the templated unit test doesn't work on your code. Good luck with that. Again, there are so many reasons I should not have to worry about unit tests in my library being run with your code. That's on you. I didn't write it for your code to run, if you want to run it, run my unit test script. -Steve
Re: Quit running foreign unittests _
On Monday, 27 April 2015 at 11:30:04 UTC, Steven Schveighoffer wrote: On 4/27/15 6:20 AM, Dicebot wrote: On Monday, 27 April 2015 at 10:15:20 UTC, Kagamin wrote: On Monday, 27 April 2015 at 09:22:48 UTC, Dicebot wrote: Compiling tests of dependencies pretty much never causes any notable slowdown. This thread doesn't support that view, see the first post. Which part exactly? I only see comparisons for compiling AND running tests for dependencies. And it is usually running which causes the slowdown. The problem is as follows: 1. Unit tests for some library are written for that library. They are written to run tests during unit tests of that library only (possibly with certain requirements of environment, including build lines, or expectations of system resource availability). 2. People who import that library's modules are not trying to test the library, they are trying to test their code. Those are two points I fundamentally disagree with. It doesn't matter where the code comes from - in the end only thing that matters is correctness of your application as a whole. And considering tests are not necessarily pure the results may very well differ between running those tests spearately and as part of application test suite a whole. Unless compiling some specific tests causes some proven _compilation_ slowdown (I have yet to see that) those all must be compiled and filtered by runtime test runner optionally. And if tests are written in a weird way that they can only be ran within that library test step, those are not really unittests. Usage of version(MyLibTests) in Nick SDL library annoyed me so much that I forked it to never deal with those pesky versions again. Don't want to do that with Phobos too.
Re: Quit running foreign unittests _
On 4/28/15 12:40 PM, Dicebot wrote:
On Monday, 27 April 2015 at 11:30:04 UTC, Steven Schveighoffer wrote:
On 4/27/15 6:20 AM, Dicebot wrote:
On Monday, 27 April 2015 at 10:15:20 UTC, Kagamin wrote:
On Monday, 27 April 2015 at 09:22:48 UTC, Dicebot wrote:
Compiling tests of dependencies pretty much never causes any notable
slowdown.
This thread doesn't support that view, see the first post.
Which part exactly? I only see comparisons for compiling AND running
tests for dependencies. And it is usually running which causes the
slowdown.
The problem is as follows:
1. Unit tests for some library are written for that library. They are
written to run tests during unit tests of that library only (possibly
with certain requirements of environment, including build lines, or
expectations of system resource availability).
2. People who import that library's modules are not trying to test the
library, they are trying to test their code.
Those are two points I fundamentally disagree with.
I think by default, nobody wants to test already-tested code in their
application. That's not the point of unit tests.
For example, if there's a module that has:
struct S(T)
{
unittest {...}
}
unittest
{
S!int; // run unit tests for int
}
Then I can assume that unit test was run and passed. If I want to test
it to be sure, I'll run that library's unit tests!
But I don't want my unit test that uses S!int to also test S's unit
tests for S!int. That doesn't make sense. I'm wasting cycles on
something that is already proven.
Now, if I want to run unit tests for S!MyCustomType, that makes sense.
The library didn't test that. Which is why there should be a way to do
it (if it's valid!).
Right now, there isn't a way to control what runs and what doesn't. I
don't care where it's decided or how it's decided, but somehow I should
be able to NOT run S!int tests again.
Unless compiling some specific tests causes some proven _compilation_
slowdown (I have yet to see that) those all must be compiled and
filtered by runtime test runner optionally.
For example, RedBlackTree when running unit tests does a sanity check of
the tree for every operation. If you then use a RedBlackTree in your
unit tests, you are doing that again. Maybe it's worth it to you, but it
can increase the runtime of your test by orders of magnitude. Last time
I checked, performance was high on people's priority lists.
Or, potentially you could be running INVALID TESTS, because the tests
weren't written for your specific usages. I ran into this when others
complained that they couldn't test their code that uses
RedBlackTree!string, because all the unit tests instantiated with int
literals. This is simply a does not work thing. You can't turn them
off, and you can't test your code. Is it worth it to you for a library
to try to compile tests that prevent your test build from happening? Do
you enjoy waiting on others to fix their problems so you can test your
code? This is a very less than ideal situation. And it's not something
the compiler tells you will happen.
And if tests are written in a weird way that they can only be ran within
that library test step, those are not really unittests.
This means you should never ever write unit tests into a template.
Because it's impossible to create tests that will work for every single
instantiation. See RedBlackTree for all the crud you have to put in for
this to be viable today. I'd love to get rid of all that.
In which case, let's disallow that capability. I'm fine with that too.
This simply means your all encompassing tests will not be all
encompassing any more, they will only test a select few instantiations,
and you can't control that either from your application code.
I want a way to control it as a template designer and as a template
instantiator. Flexibility is king.
-Steve
Re: Quit running foreign unittests _
On Tuesday, 28 April 2015 at 16:40:05 UTC, Dicebot wrote: On Monday, 27 April 2015 at 11:30:04 UTC, Steven Schveighoffer wrote: On 4/27/15 6:20 AM, Dicebot wrote: On Monday, 27 April 2015 at 10:15:20 UTC, Kagamin wrote: On Monday, 27 April 2015 at 09:22:48 UTC, Dicebot wrote: Compiling tests of dependencies pretty much never causes any notable slowdown. This thread doesn't support that view, see the first post. Which part exactly? I only see comparisons for compiling AND running tests for dependencies. And it is usually running which causes the slowdown. The problem is as follows: 1. Unit tests for some library are written for that library. They are written to run tests during unit tests of that library only (possibly with certain requirements of environment, including build lines, or expectations of system resource availability). 2. People who import that library's modules are not trying to test the library, they are trying to test their code. Those are two points I fundamentally disagree with. It doesn't matter where the code comes from - in the end only thing that matters is correctness of your application as a whole. And considering tests are not necessarily pure the results may very well differ between running those tests spearately and as part of application test suite a whole. Unless compiling some specific tests causes some proven _compilation_ slowdown (I have yet to see that) those all must be compiled and filtered by runtime test runner optionally. And if tests are written in a weird way that they can only be ran within that library test step, those are not really unittests. Usage of version(MyLibTests) in Nick SDL library annoyed me so much that I forked it to never deal with those pesky versions again. Don't want to do that with Phobos too. Then how do you propose to approach the containers problem? On one hand, a unittest on containers themselves involves testing the container for integrity after each operation. On the other hand, a unittest on another module may involve heavy use of containers (say, N operations with a container), and if integrity checks are enabled at this time, it totals to N^2 trivial operations which may not be feasible. Ivan Kazmenko.
Re: Quit running foreign unittests _
https://issues.dlang.org/show_bug.cgi?id=13454
Re: Quit running foreign unittests _
A fairly simple solution to this would be to compile unittests only for root modules (the ones that are part of the compilation), not for imported modules. Then everyone can decide for which modules to use -unittest.
Re: Quit running foreign unittests _
On Monday, 27 April 2015 at 07:28:28 UTC, Martin Nowak wrote: https://issues.dlang.org/show_bug.cgi?id=13454 Great, lets compromise both unittest correctness and convienience at the same time. Can we please just close the issue as `RESOLVED WONTFIX`? Compiling tests of dependencies pretty much never causes any notable slowdown. Trying to be smart about it does harm the users and contributors of the library though.
Re: Quit running foreign unittests _
On Monday, 27 April 2015 at 10:15:20 UTC, Kagamin wrote: On Monday, 27 April 2015 at 09:22:48 UTC, Dicebot wrote: Compiling tests of dependencies pretty much never causes any notable slowdown. This thread doesn't support that view, see the first post. Which part exactly? I only see comparisons for compiling AND running tests for dependencies. And it is usually running which causes the slowdown.
Re: Quit running foreign unittests _
On Monday, 27 April 2015 at 09:22:48 UTC, Dicebot wrote: Compiling tests of dependencies pretty much never causes any notable slowdown. This thread doesn't support that view, see the first post.
Re: Quit running foreign unittests _
On 4/27/15 6:20 AM, Dicebot wrote:
On Monday, 27 April 2015 at 10:15:20 UTC, Kagamin wrote:
On Monday, 27 April 2015 at 09:22:48 UTC, Dicebot wrote:
Compiling tests of dependencies pretty much never causes any notable
slowdown.
This thread doesn't support that view, see the first post.
Which part exactly? I only see comparisons for compiling AND running
tests for dependencies. And it is usually running which causes the
slowdown.
The problem is as follows:
1. Unit tests for some library are written for that library. They are
written to run tests during unit tests of that library only (possibly
with certain requirements of environment, including build lines, or
expectations of system resource availability).
2. People who import that library's modules are not trying to test the
library, they are trying to test their code.
3. The library runs its unit tests for templates in this case, not the
other unit tests in the module (those would be run in that module's unit
tests, which means you'd have to recompile the library to run unit
tests). Most often, unit tests in templates are only expected to be
defined by the library.
I think ketmar had the most reasonable request here -- only compile/run
unit tests in templates for the module that defines the template.
If you want to create a unit test that runs for your specialized
template version, there should be a way to do that, but it should be
opt-in. I envision something like this:
unittest
{
import somelib.somemod : SomeModTemplate, unittest
SomeModTemplate!MyLocalType t; // runs templated unit tests
}
If you look in RedBlackTree, I use templated unit tests to great effect
(found about 4-5 compiler bugs that way). But I have a lot of machinery
around the unit tests to try and only test unit tests on integral types.
It would be nice to not have to write all that shit.
And BTW, I didn't even get it right, I forgot about other parameters to
the RBTree, necessitating a PR later.
-Steve
Re: Quit running foreign unittests _
On Monday, 27 April 2015 at 11:30:04 UTC, Steven Schveighoffer
wrote:
The problem is as follows:
1. Unit tests for some library are written for that library.
They are written to run tests during unit tests of that library
only (possibly with certain requirements of environment,
including build lines, or expectations of system resource
availability).
By the way, a unittest-related issue still stands in DMD 2.067.1
for RedBlackTree:
https://issues.dlang.org/show_bug.cgi?id=12246
A similar matter got resolved quickly for BinaryHeap:
https://issues.dlang.org/show_bug.cgi?id=12245
BinaryHeap's case was handled by putting the container's
unittests into debug(BinaryHeap).
RedBlackTree's case is controlled via debug(RBDoChecks)
(formerly version(RBDoChecks)). The difference is the presence
of a version(unittest) debug = RBDoChecks; line. This looks
inconsistent.
For RedBlackTree, compile the following with or without -unittest
option and run for a visible difference in speed (runs
momentarily or for a few seconds):
-
import std.container;
void main() {
auto t = redBlackTree!int;
foreach (i; 0..3000) t.insert(i);
}
-
Ivan Kazmenko.
Re: Quit running foreign unittests _
On Monday, 27 April 2015 at 15:29:05 UTC, Steven Schveighoffer
wrote:
On 4/27/15 10:30 AM, Ivan Kazmenko wrote:
On Monday, 27 April 2015 at 11:30:04 UTC, Steven Schveighoffer
wrote:
The problem is as follows:
1. Unit tests for some library are written for that library.
They are
written to run tests during unit tests of that library only
(possibly
with certain requirements of environment, including build
lines, or
expectations of system resource availability).
By the way, a unittest-related issue still stands in DMD
2.067.1 for
RedBlackTree:
https://issues.dlang.org/show_bug.cgi?id=12246
A similar matter got resolved quickly for BinaryHeap:
https://issues.dlang.org/show_bug.cgi?id=12245
BinaryHeap's case was handled by putting the container's
unittests into
debug(BinaryHeap).
RedBlackTree's case is controlled via debug(RBDoChecks)
(formerly
version(RBDoChecks)). The difference is the presence of a
version(unittest) debug = RBDoChecks; line. This looks
inconsistent.
For RedBlackTree, compile the following with or without
-unittest option
and run for a visible difference in speed (runs momentarily or
for a few
seconds):
-
import std.container;
void main() {
auto t = redBlackTree!int;
foreach (i; 0..3000) t.insert(i);
}
-
Ivan Kazmenko.
It's an anecdotal fix. I remember arguing over the change to
debug, that was done for purity (pure functions need debug mode
to print out something, which rbdochecks will do if there is an
issue), but I can't find the conversation.
But someone else will complain that when they try to debug
their code, adding -debug to the command line debugs RBTree's
algorithm (similarly to how they complained BinaryHeap was
doing this).
I am doing just that, right now =) . But it's -unittest, not
-debug, which triggers RBTree's expensive checks.
This really should only EVER run during phobos unit tests.
A choice would be even better. When a user's code is wrong, for
example, a comparison function does not define a comparison in
mathematical sense, the user might benefit from container's
sanity checks. From personal experience, when I implement my own
sorted container and some complex logic using it, I do benefit
from the container's checks sometimes.
I don't know how to fix this properly without something like I
outlined above, or without doing some global
version(PhobosUnitTests) hack.
In fact, I don't agree with the BinaryHeap change. At this
point, phobos unit tests are NOT testing the binary heap
structure. Sure that makes user code run faster, but at the
cost of never testing it even when it should be tested.
-Steve
Right now, it is possible to test binary heap integrity with an
explicit -debug=BinaryHeap, and it is not tested by default.
And it is mandatory to test red-black tree integrity when running
any unittests involving a red-black tree, as it is tested by
default.
So, from a user's point of view, I like the binary heap situation
better, since it gives me a choice with a reasonable default.
How Phobos' unittests should be handled is another matter which,
in my opinion, should not take away choices, and reasonable
defaults too, from the user.
To me, the proposed version(PhobosUnitTests) hack looks good for
the particular case. But I don't see whether such trick scales
well for the whole ecosystem of libraries.
Ivan Kazmenko.
Re: Quit running foreign unittests _
On 4/27/15 10:30 AM, Ivan Kazmenko wrote:
On Monday, 27 April 2015 at 11:30:04 UTC, Steven Schveighoffer wrote:
The problem is as follows:
1. Unit tests for some library are written for that library. They are
written to run tests during unit tests of that library only (possibly
with certain requirements of environment, including build lines, or
expectations of system resource availability).
By the way, a unittest-related issue still stands in DMD 2.067.1 for
RedBlackTree:
https://issues.dlang.org/show_bug.cgi?id=12246
A similar matter got resolved quickly for BinaryHeap:
https://issues.dlang.org/show_bug.cgi?id=12245
BinaryHeap's case was handled by putting the container's unittests into
debug(BinaryHeap).
RedBlackTree's case is controlled via debug(RBDoChecks) (formerly
version(RBDoChecks)). The difference is the presence of a
version(unittest) debug = RBDoChecks; line. This looks inconsistent.
For RedBlackTree, compile the following with or without -unittest option
and run for a visible difference in speed (runs momentarily or for a few
seconds):
-
import std.container;
void main() {
auto t = redBlackTree!int;
foreach (i; 0..3000) t.insert(i);
}
-
Ivan Kazmenko.
It's an anecdotal fix. I remember arguing over the change to debug, that
was done for purity (pure functions need debug mode to print out
something, which rbdochecks will do if there is an issue), but I can't
find the conversation.
But someone else will complain that when they try to debug their code,
adding -debug to the command line debugs RBTree's algorithm (similarly
to how they complained BinaryHeap was doing this). This really should
only EVER run during phobos unit tests.
I don't know how to fix this properly without something like I outlined
above, or without doing some global version(PhobosUnitTests) hack.
In fact, I don't agree with the BinaryHeap change. At this point, phobos
unit tests are NOT testing the binary heap structure. Sure that makes
user code run faster, but at the cost of never testing it even when it
should be tested.
-Steve
Re: Quit running foreign unittests _
To me, the proposed version(PhobosUnitTests) hack looks good for the particular case. But I don't see whether such trick scales well for the whole ecosystem of libraries. Another approach that might work specifically for containers is to pass as an additional template parameter whether the particular RedBlackTree must continuously check its integrity or not. That way, the parameter can be explicitly set to doCheck from the unittest instantiations, default to doNotCheck for other code, and be enabled on a case-by-case basis when debugging. Again, I don't see whether it fits the larger picture, but still think it's worth to consider. Ivan Kazmenko.
Re: Quit running foreign unittests _
On 4/27/15 12:00 PM, Ivan Kazmenko wrote: Right now, it is possible to test binary heap integrity with an explicit -debug=BinaryHeap, and it is not tested by default. Right, but phobos unit tests are not doing that. If we added a version(unittest) debug = BinaryHeap; to fix that problem there, then you would still be complaining ;) -Steve
Re: Quit running foreign unittests _
On Tue, 09 Sep 2014 22:13:57 -0400, Nick Sabalausky seewebsitetocontac...@semitwist.com wrote: This is getting to be (or rather, *continuing* to be) a royal PITA: https://github.com/rejectedsoftware/vibe.d/issues/673 I don't mean to pick on Vibe.d in particular, but can we have a solution (that doesn't involve obscure corners of druntime, or demanding everyone use the same build system) for running our unittests *without* triggering a cascading avalanche of unittests from ALL third party libs that don't cooperate with the [frankly] quite clumsy version(mylib_unittests) hack?! Hm... would it be acceptable if unit tests were compiled, but not run? Because the unit tester runtime just foreach's over every module and runs the tests contained in that module. Now, I don't know how it works if you compile a template unit test -- does it get deposited into the module that instantiated or into the module that defined it? Worth looking into I think. -Steve
Re: Quit running foreign unittests _
On 9/10/2014 1:01 AM, Kagamin wrote: It's questionable that testing every template instantiation is intended: you can't do the same for templated functions. Regardless of the (de)merits of deliberately putting unittests in a template, the important thing is this: Whether a person intentionally *or inadvertently* winds up with a unittest inside a template, the whole causes some of my unittests run when my lib's *users* run *their* project's unittests is a highly unintuitive non-localized consequence.
Re: Quit running foreign unittests _
On Wednesday, 10 September 2014 at 02:14:39 UTC, Nick Sabalausky wrote: This is getting to be (or rather, *continuing* to be) a royal PITA: https://github.com/rejectedsoftware/vibe.d/issues/673 I don't mean to pick on Vibe.d in particular, but can we have a solution (that doesn't involve obscure corners of druntime, or demanding everyone use the same build system) for running our unittests *without* triggering a cascading avalanche of unittests from ALL third party libs that don't cooperate with the [frankly] quite clumsy version(mylib_unittests) hack?! My opinion stays the same - it is good reliable default and should stay that way. And yes, I absolutely hate when libraries do version(mylib_unittest) hacks forcing me to bother with millions of versions to enable them back. Simple solutions is to provide alternative unit test runner module as part of Phobos which does the module/package filtering.
Re: Quit running foreign unittests _
On Tue, 09 Sep 2014 22:37:23 -0400 Nick Sabalausky via Digitalmars-d digitalmars-d@puremagic.com wrote: If you think that's reasonable when it's the COMPILER that inserts all those calls (or druntime, whatever), then go ahead with it yourself. sure it's a compiler. that's why we have -unittest arg. and separate compilation rocks (why, it's cheap! and windows... ah, i don't care). yes, templates still sux (considering built-in unittests). this is where compiler should be fixed: it should compile unittest invocations only for templates that comes from the module it compiles, i think. signature.asc Description: PGP signature
Re: Quit running foreign unittests _
On Tue, 09 Sep 2014 23:00:06 -0400 Nick Sabalausky via Digitalmars-d digitalmars-d@puremagic.com wrote: Well, ok, I guess it could be done with separate invocations of DMD, but that has non-trivial implications for build performance we can cache analyzed ASTs somewhere. this can speedup separate compilations a little. but i don't see any problems with separate compilation: it's fast enough even with alot of modules. signature.asc Description: PGP signature
Re: Quit running foreign unittests _
On 9/10/2014 3:57 AM, Dicebot wrote: My opinion stays the same - it is good reliable default and should stay that way. And yes, I absolutely hate when libraries do version(mylib_unittest) hacks forcing me to bother with millions of versions to enable them back. Being default is one thing, I really don't care what the default is - but for now it's not just the default, it's the only realistic option without getting into PITA workarounds. Some people like including 3rd party library unittests with their own (apparently), some people like *not* including them. So we have some projects using version(foo_unittest) and other projects not using it. The current result is: *Nobody* can easily get what they want: Not people who like it my way, and not people who like it your way. Not easily enough anyway. So it's a global ecosystem problem. It needs fixed. (And to start with, general agreeance that it should be fixed.) Simple solutions is to provide alternative unit test runner module as part of Phobos which does the module/package filtering. It needs to be *crystal* clear and trivially simple: A. Knowing that it exists. B. Sussing out how to use it. C. Actually using it. Otherwise, version(foo_unittest) will continue to linger out of an individual's convenience, and so once again, not everyone gets to have things their way. A related concern I have about a Phobos-only solution: When/where do you configure it? Obviously it can't happen anywhere in the scope of main. Are static ctors sufficient? I would *think* so, but I'm not certain.
Re: Quit running foreign unittests _
On Wednesday, 10 September 2014 at 08:21:34 UTC, Nick Sabalausky wrote: Simple solutions is to provide alternative unit test runner module as part of Phobos which does the module/package filtering. It needs to be *crystal* clear and trivially simple: A. Knowing that it exists. B. Sussing out how to use it. C. Actually using it. Otherwise, version(foo_unittest) will continue to linger out of an individual's convenience, and so once again, not everyone gets to have things their way. A related concern I have about a Phobos-only solution: When/where do you configure it? Obviously it can't happen anywhere in the scope of main. Are static ctors sufficient? I would *think* so, but I'm not certain. Leandro Lucarella has written similar solution for our internal testing purposes. It provides a test runner that is configured by CLI arguments and allows filtering of modules and non-fatal test failures. It is for D1/tango but I am pretty sure it can be ported to D2 runtime with close to 0 changes. I have not managed to convince him it is worth making public though :(
Re: Quit running foreign unittests _
On Wednesday, 10 September 2014 at 08:05:28 UTC, ketmar via
Digitalmars-d wrote:
yes, templates still sux (considering built-in unittests). this
is
where compiler should be fixed: it should compile unittest
invocations
only for templates that comes from the module it compiles, i
think.
That way it will be clear.
By the way, will this link?
---
template A(T)
{
int a;
unittest{ f(); }
}
version(unittest) void f(){}
---
If this module is compiled without unittests, `f` function will
not be compiled, then if another module is compiled with
unittests and instantiates the template, the template's unittest
is compiled and it will link with `f` function, which wasn't
compiled, so linking will fail?
Re: Quit running foreign unittests _
On 09/10/2014 07:01 AM, Kagamin wrote: It's questionable that testing every template instantiation is intended: you can't do the same for templated functions. Yes you can.
Quit running foreign unittests _
This is getting to be (or rather, *continuing* to be) a royal PITA: https://github.com/rejectedsoftware/vibe.d/issues/673 I don't mean to pick on Vibe.d in particular, but can we have a solution (that doesn't involve obscure corners of druntime, or demanding everyone use the same build system) for running our unittests *without* triggering a cascading avalanche of unittests from ALL third party libs that don't cooperate with the [frankly] quite clumsy version(mylib_unittests) hack?!
Re: Quit running foreign unittests _
On Tue, 09 Sep 2014 22:13:57 -0400 Nick Sabalausky via Digitalmars-d digitalmars-d@puremagic.com wrote: write build tool that is capable of doing that... signature.asc Description: PGP signature
Re: Quit running foreign unittests _
On Wednesday, 10 September 2014 at 02:14:39 UTC, Nick Sabalausky wrote: This is getting to be (or rather, *continuing* to be) a royal PITA: https://github.com/rejectedsoftware/vibe.d/issues/673 I don't mean to pick on Vibe.d in particular, but can we have a solution (that doesn't involve obscure corners of druntime, or demanding everyone use the same build system) for running our unittests *without* triggering a cascading avalanche of unittests from ALL third party libs that don't cooperate with the [frankly] quite clumsy version(mylib_unittests) hack?! A while ago, I though about adding an optional module/package whitelist as a compiler flag, something like -unittest=myapp,mylib.important. Never really got to implementing it, or even thinking about the implications in any details. David
Re: Quit running foreign unittests _
On Wednesday, 10 September 2014 at 02:29:41 UTC, David Nadlinger wrote: A while ago, I though about adding an optional module/package whitelist as a compiler flag, something like -unittest=myapp,mylib.important. Never really got to implementing it, or even thinking about the implications in any details. s/compiler/rdmd/
Re: Quit running foreign unittests _
On 9/9/2014 10:21 PM, ketmar via Digitalmars-d wrote: On Tue, 09 Sep 2014 22:13:57 -0400 Nick Sabalausky via Digitalmars-d digitalmars-d@puremagic.com wrote: write build tool that is capable of doing that... ??? If you think that's reasonable when it's the COMPILER that inserts all those calls (or druntime, whatever), then go ahead with it yourself. Seriously, DMD needs a --omit-unittest=some.package.*. I'd try a PR myself (I'd have *already* tried), but last time I brought it up the whole idea was completely shot down in favor of both the version(mylib_unittests) hack (which people don't fucking use half the time, sometimes deliberately) and some goofy druntime trickery (ie, writing your own test runner and getting it to druntime before druntime gets to the run unittests phase. Seriously? That's *preferable* to a --omit-unittest=some.package.*??).
Re: Quit running foreign unittests _
On 9/9/2014 10:37 PM, Nick Sabalausky wrote: On 9/9/2014 10:21 PM, ketmar via Digitalmars-d wrote: On Tue, 09 Sep 2014 22:13:57 -0400 Nick Sabalausky via Digitalmars-d digitalmars-d@puremagic.com wrote: write build tool that is capable of doing that... ??? If you think that's reasonable when it's the COMPILER that inserts all those calls (or druntime, whatever), then go ahead with it yourself. Well, ok, I guess it could be done with separate invocations of DMD, but that has non-trivial implications for build performance, and templates probably could still fuck it all up anyway.
Re: Quit running foreign unittests _
On Wednesday, 10 September 2014 at 03:00:48 UTC, Nick Sabalausky wrote: separate invocations […] and templates probably could still fuck it all up anyway. If so, then this is either a critical bug in DMD, or an issue with your code (e.g. due to version(unittest) use). I'm aware of the issues that affect some ways of doing incremental compilation, but that's mostly a different story. David
Re: Quit running foreign unittests _
On 9/9/2014 11:07 PM, David Nadlinger wrote:
On Wednesday, 10 September 2014 at 03:00:48 UTC, Nick Sabalausky wrote:
separate invocations […] and templates probably could still fuck it
all up anyway.
If so, then this is either a critical bug in DMD, or an issue with your
code (e.g. due to version(unittest) use).
I'm aware of the issues that affect some ways of doing incremental
compilation, but that's mostly a different story.
David
I was thinking about something like this:
//a.d
import b.d;
alias x = Foo!whatever;
//b.d
struct Foo(Arg) {
void foo() {...}
unittest { /+...unittest foo()...+/ }
}
// Attempt to exclude module b's unittests, except...aren't
// they compiled when module a's is compiled? The first
// dmd invocation here has no idea module b will be compiled
// without the -unittest flag.
dmd -c -unittest a.d
dmd -c b.d // Duplicate a whole buttload of processing here
dmd a.o b.o
Re: Quit running foreign unittests _
I think, libraries should support compilation without unittests, i.e. write unittests outside of templates. But then one may want to test every template instantiation, if it makes sense and intended.
Re: Quit running foreign unittests _
It's questionable that testing every template instantiation is intended: you can't do the same for templated functions.
