Re: Named unittests
On 2015-04-02 22:31, Dicebot wrote: I don't think anyone is going to put those in a same inline style as unittest blocks, so this is not truly relevant. At least I hope so. You mean inline with the code it tests? No, I hope so too. I put my unit tests in separate files as well, but that's just the way I prefer it. -- /Jacob Carlborg
Re: Named unittests
On Thursday, 2 April 2015 at 06:21:53 UTC, Jacob Carlborg wrote: On 2015-04-01 21:16, Dicebot wrote: This is fixed by having smaller modules. If test for a single module takes more than few blinks of an eye, something is wrong already. For unit tests, yes. But there are other kinds of tests as well. Integration, functional, user acceptance test and so on. I don't think anyone is going to put those in a same inline style as unittest blocks, so this is not truly relevant. At least I hope so.
Re: Named unittests
On Tuesday, 31 March 2015 at 21:24:20 UTC, Dicebot wrote:
I see no value in test names limited to valid identifiers. It
is only tiny bit more informative than `unittestXXX` we have
already. If we add names, please, let them be proper names that
are easy to read.
I'd rather have the name obey the same restrictions as normal
function names and keep additional information in optional
@description("...")
Re: Named unittests
On 2015-04-01 21:16, Dicebot wrote: This is fixed by having smaller modules. If test for a single module takes more than few blinks of an eye, something is wrong already. For unit tests, yes. But there are other kinds of tests as well. Integration, functional, user acceptance test and so on. -- /Jacob Carlborg
Re: Named unittests
On 2015-04-01 20:49, Idan Arye wrote: So now we just need a way to keep a line number unmodified when you add lines above it... I'm not sure what kind of tools you have in mind. * Print a link which points back to a failing test, requires file and line information * Running a specific test, requires file and line information, name or a tag What else do you have in mind? -- /Jacob Carlborg
Re: Named unittests
On Wednesday, 1 April 2015 at 18:35:14 UTC, Idan Arye wrote: The problem is not with running the tests, it's with building them. In heavily templated libraries(like, for example Phobos), building without unittests takes seconds and building with unitetests takes minutes - mainly because the tests need to to many template instantiations. If we could tell the compiler to only build a single, specific test the development cycle can become orders of magnitude faster. This is fixed by having smaller modules. If test for a single module takes more than few blinks of an eye, something is wrong already.
Re: Named unittests
On 2015-04-01 20:35, Idan Arye wrote: If we could tell the compiler to only build a single, specific test the development cycle can become orders of magnitude faster. There should be a lot of option to run tests: * Base on a file * Line number * Name * Tag And so on. -- /Jacob Carlborg
Re: Named unittests
On Wednesday, 1 April 2015 at 14:07:14 UTC, Jacob Carlborg wrote: On 2015-04-01 00:02, Idan Arye wrote: I think you and I work under different assumptions of the goals for this feature. If we only want unittest names to be something that can be printed when the unittest runner runs the unittests, than a UDA with a string is indeed preferable. If we want something that tools can actually use to refer to a specific unittest, we need a proper identifier(yes, even though it can be implemented in library code because D is Turing-complete...) The tools should, mostly, use the file and line information. So now we just need a way to keep a line number unmodified when you add lines above it...
Re: Named unittests
On Wednesday, 1 April 2015 at 14:05:46 UTC, Jacob Carlborg wrote:
On 2015-03-31 23:14, Idan Arye wrote:
Building by unittest name! Imagine - instead of placing
temporary code
in `main` to develop a new feature or fix a bug, you put in a
named
unittest and tell your IDE/build-system to only build that
unittest(and
whatever code needed for it to run). You `writeln` stuff to
the console,
and when you get some changes to output what you want you
change the
`writeln`s to `assert`s and proceed to work on the next step.
When you
are done, all you have to do is tidy it up a bit and BAM -
without any
special effort you get a unittest that tests that feature/bug
you just
worked on.
You just reinvented test driven development ;). It's perfectly
possible to do this with a UDA an a text string as well. UDA's
also allows you to tag the tests. Basically a short name you
put on multiple tests, then tell the test runner to run only
those tests, or ignore those.
@tag("foo") @name("this is my test name") unittest {}
$ run-tests -t foo
The problem is not with running the tests, it's with building
them. In heavily templated libraries(like, for example Phobos),
building without unittests takes seconds and building with
unitetests takes minutes - mainly because the tests need to to
many template instantiations. If we could tell the compiler to
only build a single, specific test the development cycle can
become orders of magnitude faster.
Re: Named unittests
On 4/1/15 12:46 AM, Atila Neves wrote: On Tuesday, 31 March 2015 at 23:53:29 UTC, Andrei Alexandrescu wrote: On 3/31/15 4:38 PM, Martin Nowak wrote: On 04/01/2015 01:11 AM, Andrei Alexandrescu wrote: The reasonable course is to see how far we can get with a library-only solution. Amaury, want to work on that? -- Andrei In any case you should talk to Atila Neves who wrote a really good unittest library. http://code.dlang.org/packages/unit-threaded http://code.dlang.org/packages/dtest It's also used by quite a lot of people already. http://code.dlang.org/api/packages/unit-threaded/stats Atila, is there interest in moving your library to std.experimental? -- Andrei Sure, I can have the PR ready by next week. As mentioned earlier, a UDA to attach a name to a unittest block and then run with that name in the output is 30min work. Making the whole thing std.experimental ready will take a few days. TBH, if I'd known it had a chance of making it in there, I'd've done it ages ago! Atila Sounds like a plan. Thanks in advance! -- Andrei
Re: Named unittests
On 2015-03-31 15:31, Dicebot wrote:
Most powerful solution would be to simply put attributes for unittest
blocks in runtime information for tests (using RTTI it should be
possible to define such variadic structure in similar manner as D-style
variadic function arguments).
I think the most powerful and most generic solution would be to:
1. Remove the unittest keyword
2. Make code executable at module scope
3. Add support for trailing delegate syntax
module foo;
unittest("foobar")
{
}
Would be lowered to
unittest("foobar", {
});
Works with benchmark and other things as well.
--
/Jacob Carlborg
Re: Named unittests
On 2015-04-01 00:02, Idan Arye wrote: I think you and I work under different assumptions of the goals for this feature. If we only want unittest names to be something that can be printed when the unittest runner runs the unittests, than a UDA with a string is indeed preferable. If we want something that tools can actually use to refer to a specific unittest, we need a proper identifier(yes, even though it can be implemented in library code because D is Turing-complete...) The tools should, mostly, use the file and line information. -- /Jacob Carlborg
Re: Named unittests
On 2015-03-31 23:14, Idan Arye wrote:
Building by unittest name! Imagine - instead of placing temporary code
in `main` to develop a new feature or fix a bug, you put in a named
unittest and tell your IDE/build-system to only build that unittest(and
whatever code needed for it to run). You `writeln` stuff to the console,
and when you get some changes to output what you want you change the
`writeln`s to `assert`s and proceed to work on the next step. When you
are done, all you have to do is tidy it up a bit and BAM - without any
special effort you get a unittest that tests that feature/bug you just
worked on.
You just reinvented test driven development ;). It's perfectly possible
to do this with a UDA an a text string as well. UDA's also allows you to
tag the tests. Basically a short name you put on multiple tests, then
tell the test runner to run only those tests, or ignore those.
@tag("foo") @name("this is my test name") unittest {}
$ run-tests -t foo
--
/Jacob Carlborg
Re: Named unittests
On 2015-03-31 11:08, Johannes Pfau wrote: But here's the problem: 1) The compile time approach requires some kind of explicit registration of the unittests. At least one mixin per module. 2) This mixin will usually provide a module constructor. But using module constructors will cause issues with cycle detection. When we get RTInfo for modules [1] there shouldn't be any problems. A template will be instantiated once for each module the compiler sees. With that module __traits(getUnitTests) and __traits(allMembers) can be used to access the unit tests, benchmarks, or whatever. Not need for any registration or module constructors. [1] https://github.com/D-Programming-Language/dmd/pull/2271 -- /Jacob Carlborg
Re: Named unittests
On 2015-03-31 23:08, Idan Arye wrote: Limiting unittest names to legal identifiers will save a lot of headache when we set our tools to actually use these names. Matching a legal identifier in a text stream is much easier than matching an arbitrary unicode string, even if that string is escaped. I don't see any problem with arbitrary strings. It works perfectly fine in RSpec, which has much better tooling than D. -- /Jacob Carlborg
Re: Named unittests
I have PR https://github.com/D-Programming-Language/phobos/pull/2995 open since October 2014. it includes: * extensible haskell like quickcheck * benchmarking with "names" and record keeping to see progress * offline tool to plot benchmark results with gnuplot * most std.string functions already have benchmarks * makefiles got a new BUILD=benchmark IMO continuous benchmarking of phobos is a must-have feature and a nice marketing tool ( benchmark graphs on dlang.org). Something similar to quickcheck is equally important, to get the test input you didn't expect. I got one and a half comments so far.
Re: Named unittests
On Wednesday, 1 April 2015 at 04:42:33 UTC, Shammah Chancellor wrote: I can possibly help with a DIP, but it seems like a very straightforward request? -Shammah Considering the heated debate going on here, I'd say it's not as straightforward as it seems...
Re: Named unittests
On Tuesday, 31 March 2015 at 23:53:29 UTC, Andrei Alexandrescu wrote: On 3/31/15 4:38 PM, Martin Nowak wrote: On 04/01/2015 01:11 AM, Andrei Alexandrescu wrote: The reasonable course is to see how far we can get with a library-only solution. Amaury, want to work on that? -- Andrei In any case you should talk to Atila Neves who wrote a really good unittest library. http://code.dlang.org/packages/unit-threaded http://code.dlang.org/packages/dtest It's also used by quite a lot of people already. http://code.dlang.org/api/packages/unit-threaded/stats Atila, is there interest in moving your library to std.experimental? -- Andrei Sure, I can have the PR ready by next week. As mentioned earlier, a UDA to attach a name to a unittest block and then run with that name in the output is 30min work. Making the whole thing std.experimental ready will take a few days. TBH, if I'd known it had a chance of making it in there, I'd've done it ages ago! Atila
Re: Named unittests
On Tuesday, March 31, 2015 14:45:49 Idan Arye via Digitalmars-d wrote:
> But unittests already have
> names(http://dpaste.dzfl.pl/b15e94000f15), so the only required
> change is to allow the user to specify that name. This should be
> much simpler than adding entirely new fields.
Exactly what I was thinking. If we just made
unittest myTest
{
}
or
unittest(myTest)
{
}
legal, then the unit test function would then be named myTest instead of
whatever it's already named right now. It's a lot less ugly than using
attirbutes, and it fits really well into what we already have. Certainly,
it seems like a very straightforward and simple solution to the problem.
- Jonathan M Davis
Re: Named unittests
We need template to use compile time reflection. Please don't make it all compile time. It is useful to register tests at runtime as well. Like one tests per SAT instance in a given SAT benchmark. Practically only thing we need is a standard way to register a test function as a unittest and associate a key-value store with it.
Re: Named unittests
On 2015-03-30 21:52:36 +, Andrei Alexandrescu said: We're having a strong need for named unittests at Facebook for multiple reasons. 1. We have sophisticated tooling that verifies whether unittests are flaky. The automated monitor (for e.g. C++) figures whether a given unittest fails several times across several commits. Unittests are identified by name; relying on file/line is impossible because the line of a failure is not stable across changes. 2. Again for efficient automated testing and flakiness detection, one should be able to run only a subset of unittests by mentioning them by line in the command line. Note that this implies there's no interdependency between distinct unittests, which is fine because the new ability is opt-on; I'd say is pure style anyway. 3. Mentioning unittest names in failure messages helps human communication (e.g. "AddPeer is failing after your change"). This is impossible with file and line numbers. I'd like to make a DIP for named unittests. Who can help me with that? Andrei This is a fantastic idea -- and something I was concerned about lately. +1 I can possibly help with a DIP, but it seems like a very straightforward request? -Shammah
Re: Named unittests
On Tuesday, 31 March 2015 at 23:29:28 UTC, Martin Nowak wrote: On 04/01/2015 12:58 AM, deadalnix wrote: So now we are going to change the language for this ? There is a natural name for unitests, the name of the module. We have way to break module into pieces in a backward compatible manner now, so it's all good. Are you saying one should split off unittests into submodules? No necessarily. Typically, in most test framework, you have a class, and various method, each of them testing various aspect of what you want to test. I see the class here as the module, and each of the method as a unitest block. Or are you saying https://github.com/D-Programming-Language/druntime/blob/3656ba9469a60b4b23bb4a3cd95812b4f410b8a1/src/test_runner.d is enough for your needs? I think having multiple tests per module makes sense, e.g. because it's better to parallelize the tests. See my previous comment, that is not needed. In fact i also think having several unitest blocks is useful. We may want to add various annotation to a test, and we have UDA for that. Sure use a UDA, I don't care. It seems important though, that the compiler recognizes it and emits it as metadata to the ModuleInfo. And if you need some compiler magic anyhow, a UDA just looks redundant. How I see it, it should be possible for a user to specify a test runner template. This test runner template can do compile time reflection and format its output the way to wants. We must not bake all required attribute we need to associate a test in the language. What if we want to associate the test with a team name, or give it a name, or give an id that my tools uses to check for tests, or whatever. Every user will want to add various metadata, and these will vary. Every users will want various output, and these will vary. It should be possible to have a test runner that use template and compile time reflection to make sense of the unitest block. Yes, there is a bit of compiler magic involved, but nothing terribly fancy. You mean this? http://dlang.org/phobos/core_runtime.html#.Runtime.moduleUnitTester We need template to use compile time reflection.
Re: Named unittests
On 3/31/15 5:58 PM, deadalnix wrote: On Tuesday, 31 March 2015 at 23:11:46 UTC, Andrei Alexandrescu wrote: The reasonable course is to see how far we can get with a library-only solution. Amaury, want to work on that? -- Andrei If nobody picks it up, I can, but I have very little bandwidth right now with the work involved on SDC. I'll quote that when you post a long rant here :o). -- Andrei
Re: Named unittests
On Tuesday, 31 March 2015 at 23:11:46 UTC, Andrei Alexandrescu wrote: The reasonable course is to see how far we can get with a library-only solution. Amaury, want to work on that? -- Andrei If nobody picks it up, I can, but I have very little bandwidth right now with the work involved on SDC.
Re: Named unittests
On 3/31/15 4:38 PM, Martin Nowak wrote: On 04/01/2015 01:11 AM, Andrei Alexandrescu wrote: The reasonable course is to see how far we can get with a library-only solution. Amaury, want to work on that? -- Andrei In any case you should talk to Atila Neves who wrote a really good unittest library. http://code.dlang.org/packages/unit-threaded http://code.dlang.org/packages/dtest It's also used by quite a lot of people already. http://code.dlang.org/api/packages/unit-threaded/stats Atila, is there interest in moving your library to std.experimental? -- Andrei
Re: Named unittests
On 04/01/2015 01:11 AM, Andrei Alexandrescu wrote: > The reasonable course is to see how far we can get with a library-only > solution. Amaury, want to work on that? -- Andrei In any case you should talk to Atila Neves who wrote a really good unittest library. http://code.dlang.org/packages/unit-threaded http://code.dlang.org/packages/dtest It's also used by quite a lot of people already. http://code.dlang.org/api/packages/unit-threaded/stats
Re: Named unittests
On 03/31/2015 11:01 PM, Atila Neves wrote: > This is true, but importing modules by name can be code that is > generated. Which is exactly what I do with dtest [1] and unit-threaded > [2]. It's not that big of a deal when it's part of the build system. Dub does it as well (dub test), because if you don't import a module, it might not get linked too.
Re: Named unittests
On 04/01/2015 12:58 AM, deadalnix wrote:
> So now we are going to change the language for this ?
>
> There is a natural name for unitests, the name of the module. We have
> way to break module into pieces in a backward compatible manner now, so
> it's all good.
Are you saying one should split off unittests into submodules?
Or are you saying
https://github.com/D-Programming-Language/druntime/blob/3656ba9469a60b4b23bb4a3cd95812b4f410b8a1/src/test_runner.d
is enough for your needs?
I think having multiple tests per module makes sense, e.g. because it's
better to parallelize the tests.
> We may want to add various annotation to a test, and we have UDA for that.
Sure use a UDA, I don't care. It seems important though, that the
compiler recognizes it and emits it as metadata to the ModuleInfo.
And if you need some compiler magic anyhow, a UDA just looks redundant.
@unittest("my test") unittest
unittest("my test")
> So the only things that is really needed is a way to customize the test
> runner from client code to output whatever everybody needs.
You mean this?
http://dlang.org/phobos/core_runtime.html#.Runtime.moduleUnitTester
Re: Named unittests
On 03/31/2015 04:45 PM, Idan Arye wrote: > > But unittests already have names(http://dpaste.dzfl.pl/b15e94000f15), so > the only required change is to allow the user to specify that name. This > should be much simpler than adding entirely new fields. That's the line number, which can't be used as the OP pointed out.
Re: Named unittests
On 3/31/15 3:58 PM, deadalnix wrote:
On Tuesday, 31 March 2015 at 21:50:56 UTC, Martin Nowak wrote:
On 03/30/2015 11:52 PM, Andrei Alexandrescu wrote:
I'd like to make a DIP for named unittests. Who can help me with that?
Why a DIP, isn't the only question what syntax to use.
unittest (myname) vs. unittest ("mynameexpression")
Andrei
There is an ER and an half of an implementation.
https://issues.dlang.org/show_bug.cgi?id=4747
https://github.com/D-Programming-Language/dmd/pull/1131
https://github.com/D-Programming-Language/druntime/pull/308
So now we are going to change the language for this ?
There is a natural name for unitests, the name of the module. We have
way to break module into pieces in a backward compatible manner now, so
it's all good.
We may want to add various annotation to a test, and we have UDA for that.
So the only things that is really needed is a way to customize the test
runner from client code to output whatever everybody needs.
The reasonable course is to see how far we can get with a library-only
solution. Amaury, want to work on that? -- Andrei
Re: Named unittests
On Tuesday, 31 March 2015 at 21:50:56 UTC, Martin Nowak wrote:
On 03/30/2015 11:52 PM, Andrei Alexandrescu wrote:
I'd like to make a DIP for named unittests. Who can help me
with that?
Why a DIP, isn't the only question what syntax to use.
unittest (myname) vs. unittest ("mynameexpression")
Andrei
There is an ER and an half of an implementation.
https://issues.dlang.org/show_bug.cgi?id=4747
https://github.com/D-Programming-Language/dmd/pull/1131
https://github.com/D-Programming-Language/druntime/pull/308
So now we are going to change the language for this ?
There is a natural name for unitests, the name of the module. We
have way to break module into pieces in a backward compatible
manner now, so it's all good.
We may want to add various annotation to a test, and we have UDA
for that.
So the only things that is really needed is a way to customize
the test runner from client code to output whatever everybody
needs.
Re: Named unittests
On 3/31/15 3:28 PM, Andrei Alexandrescu wrote:
On 3/31/15 2:50 PM, Martin Nowak wrote:
On 03/30/2015 11:52 PM, Andrei Alexandrescu wrote:
I'd like to make a DIP for named unittests. Who can help me with that?
Why a DIP, isn't the only question what syntax to use.
unittest (myname) vs. unittest ("mynameexpression")
Also unittest myname and unittest "mynameexpression". There will be no
shortage of folks willing to debate this to smithereens.
One argument for myname (using standard identifier naming rules) is that
it's command-line friendly: it won't require quoting when invoking a
single unit test, and tends toward shorter names. It may also be more
obvious that standard identifier names would use the existing lookup
rules for guaranteeing uniqueness among unittest names.
One argument for "mynameexpression" is that it allows for nice
descriptions (e.g. unittest "count should accept a custom comparator
function").
One compromise might be something like:
@description("count should accept a custom comparator function")
unittest countCustomComparator
Under this example the unittest could be invoked from the command line
by the countCustomComparator name or possibly by the description string
as an alternative.
For the very little it's worth my bikeshed is painted without
parentheses, assuming that's easily implemented.
Re: Named unittests
On 3/31/15 2:50 PM, Martin Nowak wrote:
On 03/30/2015 11:52 PM, Andrei Alexandrescu wrote:
I'd like to make a DIP for named unittests. Who can help me with that?
Why a DIP, isn't the only question what syntax to use.
unittest (myname) vs. unittest ("mynameexpression")
Also unittest myname and unittest "mynameexpression". There will be no
shortage of folks willing to debate this to smithereens.
Andrei
There is an ER and an half of an implementation.
https://issues.dlang.org/show_bug.cgi?id=4747
https://github.com/D-Programming-Language/dmd/pull/1131
https://github.com/D-Programming-Language/druntime/pull/308
Nice, I'll check that out.
Andrei
Re: Named unittests
On Tuesday, 31 March 2015 at 21:22:09 UTC, Dicebot wrote: On Tuesday, 31 March 2015 at 14:45:50 UTC, Idan Arye wrote: But unittests already have names(http://dpaste.dzfl.pl/b15e94000f15), so the only required change is to allow the user to specify that name. This should be much simpler than adding entirely new fields. And does not really help. Of all metadata that one may want to attach to a test block, names are actually _least_ important in practice. It will be matter of time until Andrei creates new topic about adding descriptions to tests or way to mark flakey ones. Better to invest into something that scales at least a bit. I think you and I work under different assumptions of the goals for this feature. If we only want unittest names to be something that can be printed when the unittest runner runs the unittests, than a UDA with a string is indeed preferable. If we want something that tools can actually use to refer to a specific unittest, we need a proper identifier(yes, even though it can be implemented in library code because D is Turing-complete...)
Re: Named unittests
On 03/30/2015 11:52 PM, Andrei Alexandrescu wrote:
> I'd like to make a DIP for named unittests. Who can help me with that?
Why a DIP, isn't the only question what syntax to use.
unittest (myname) vs. unittest ("mynameexpression")
> Andrei
There is an ER and an half of an implementation.
https://issues.dlang.org/show_bug.cgi?id=4747
https://github.com/D-Programming-Language/dmd/pull/1131
https://github.com/D-Programming-Language/druntime/pull/308
Re: Named unittests
On Tuesday, 31 March 2015 at 20:18:41 UTC, Andrei Alexandrescu
wrote:
On 3/31/15 1:04 PM, Jacob Carlborg wrote:
On 2015-03-31 16:55, Meta wrote:
Aren't unittest blocks just special functions? If that's the
case, there
should be no problem being able to give them names. It seems
to me that
it would entail the lifting of a restriction rather than a
real language
change.
Before:
unittest
{
assert(1 == 1);
}
After:
unittest checkBasicLaw
{
assert(1 == 1);
}
I prefer a UDA accepting a string, this can contain spaces and
it's not
limited to identifier names.
I used to think the same, but then I figured a bit of structure
might be preferable. -- Andrei
I see no value in test names limited to valid identifiers. It is
only tiny bit more informative than `unittestXXX` we have
already. If we add names, please, let them be proper names that
are easy to read.
Re: Named unittests
On Tuesday, 31 March 2015 at 14:45:50 UTC, Idan Arye wrote: But unittests already have names(http://dpaste.dzfl.pl/b15e94000f15), so the only required change is to allow the user to specify that name. This should be much simpler than adding entirely new fields. And does not really help. Of all metadata that one may want to attach to a test block, names are actually _least_ important in practice. It will be matter of time until Andrei creates new topic about adding descriptions to tests or way to mark flakey ones. Better to invest into something that scales at least a bit.
Re: Named unittests
I already have a library that finds all unittest blocks (as well
as its own) and names them by appending a monotonically
increasing integer to mymodulename.unittest. I've been thinking
of using a UDA to decorate unittest blocks so that they show up
with that instead but never got around to it because I don't
really use unittest blocks and nobody is asking for the feature.
It'd take 30min to add though.
Obviously 1st class support would be preferable, I like the
unittest nameGoesHere {
//...
}
that was mentioned later in the thread.
Atila
On Monday, 30 March 2015 at 21:52:35 UTC, Andrei Alexandrescu
wrote:
We're having a strong need for named unittests at Facebook for
multiple reasons.
1. We have sophisticated tooling that verifies whether
unittests are flaky. The automated monitor (for e.g. C++)
figures whether a given unittest fails several times across
several commits. Unittests are identified by name; relying on
file/line is impossible because the line of a failure is not
stable across changes.
2. Again for efficient automated testing and flakiness
detection, one should be able to run only a subset of unittests
by mentioning them by line in the command line. Note that this
implies there's no interdependency between distinct unittests,
which is fine because the new ability is opt-on; I'd say is
pure style anyway.
3. Mentioning unittest names in failure messages helps human
communication (e.g. "AddPeer is failing after your change").
This is impossible with file and line numbers.
I'd like to make a DIP for named unittests. Who can help me
with that?
Andrei
Re: Named unittests
On Tuesday, 31 March 2015 at 21:05:40 UTC, Xavier Bigand wrote: Le 31/03/2015 20:21, Andrei Alexandrescu a écrit : On 3/31/15 7:45 AM, Idan Arye wrote: On Tuesday, 31 March 2015 at 13:34:24 UTC, Dicebot wrote: On Tuesday, 31 March 2015 at 10:25:57 UTC, Idan Arye wrote: I understand the preference to librarize as much as possible, but I don't think the desire to sacrifice every possible bit of convenience to avoid the tiniest changes to the language is always beneficial. I don't say that implementing everything inside the compiler is good either though, but in many cases some slight changes to the language can make the library solution so much more simple and elegant. In this case, allowing to name a unittest should be a very simple language change that'll make any library implementation of the rest of the feature more elegant to use, simpler to implement, and more consistent with alternative library implementations. It isn't simple at all. Name is just one of many meta-values you commonly want to attach to unittest block. Some others: description, dependency, parallelization, benchmark tag, I/O indicator. It is simply impossible to foresee it all in a language feature - but it is exactly kind of data UDA are designed for. All we need is to enhance/fix the language to actually make using of that information convenient. But unittests already have names(http://dpaste.dzfl.pl/b15e94000f15), so the only required change is to allow the user to specify that name. This should be much simpler than adding entirely new fields. Interesting point! -- Andrei It will be nice to have named unittest. And better if IDEs will be able to retrieve those names, to allow replay by name, displaying results by names,... Building by unittest name! Imagine - instead of placing temporary code in `main` to develop a new feature or fix a bug, you put in a named unittest and tell your IDE/build-system to only build that unittest(and whatever code needed for it to run). You `writeln` stuff to the console, and when you get some changes to output what you want you change the `writeln`s to `assert`s and proceed to work on the next step. When you are done, all you have to do is tidy it up a bit and BAM - without any special effort you get a unittest that tests that feature/bug you just worked on.
Re: Named unittests
On Tuesday, 31 March 2015 at 20:04:01 UTC, Jacob Carlborg wrote:
On 2015-03-31 16:55, Meta wrote:
Aren't unittest blocks just special functions? If that's the
case, there
should be no problem being able to give them names. It seems
to me that
it would entail the lifting of a restriction rather than a
real language
change.
Before:
unittest
{
assert(1 == 1);
}
After:
unittest checkBasicLaw
{
assert(1 == 1);
}
I prefer a UDA accepting a string, this can contain spaces and
it's not limited to identifier names.
Limiting unittest names to legal identifiers will save a lot of
headache when we set our tools to actually use these names.
Matching a legal identifier in a text stream is much easier than
matching an arbitrary unicode string, even if that string is
escaped.
Re: Named unittests
Le 31/03/2015 20:21, Andrei Alexandrescu a écrit : On 3/31/15 7:45 AM, Idan Arye wrote: On Tuesday, 31 March 2015 at 13:34:24 UTC, Dicebot wrote: On Tuesday, 31 March 2015 at 10:25:57 UTC, Idan Arye wrote: I understand the preference to librarize as much as possible, but I don't think the desire to sacrifice every possible bit of convenience to avoid the tiniest changes to the language is always beneficial. I don't say that implementing everything inside the compiler is good either though, but in many cases some slight changes to the language can make the library solution so much more simple and elegant. In this case, allowing to name a unittest should be a very simple language change that'll make any library implementation of the rest of the feature more elegant to use, simpler to implement, and more consistent with alternative library implementations. It isn't simple at all. Name is just one of many meta-values you commonly want to attach to unittest block. Some others: description, dependency, parallelization, benchmark tag, I/O indicator. It is simply impossible to foresee it all in a language feature - but it is exactly kind of data UDA are designed for. All we need is to enhance/fix the language to actually make using of that information convenient. But unittests already have names(http://dpaste.dzfl.pl/b15e94000f15), so the only required change is to allow the user to specify that name. This should be much simpler than adding entirely new fields. Interesting point! -- Andrei It will be nice to have named unittest. And better if IDEs will be able to retrieve those names, to allow replay by name, displaying results by names,...
Re: Named unittests
On Tuesday, 31 March 2015 at 14:13:29 UTC, Johannes Pfau wrote: Am Tue, 31 Mar 2015 13:31:58 + schrieb "Dicebot" : > But here's the problem: > > 1) The compile time approach requires some kind >of explicit registration of the unittests. At least one > mixin per >module. > 2) This mixin will usually provide a module constructor. But >using module constructors will cause issues with cycle > detection. This is not really true. You only need one mixin in the root module(s), rest can be iterated recursively by test runner itself using __traits(allMembers) reflection. Only issue with that approach is that last time I checked there was a DMD bug which prevented from getting complete list of imported modules via allMembers. Should be fixable. But then you still have to explicitly import (or at least name) all modules that should be tested. This is a drawback compared to the current builtin-unittests where you do not have to explicitly import to-be-tested modules. This is true, but importing modules by name can be code that is generated. Which is exactly what I do with dtest [1] and unit-threaded [2]. It's not that big of a deal when it's part of the build system. Atila [1] http://code.dlang.org/my_packages/dtest [2] http://code.dlang.org/my_packages/unit-threaded I was thinking of a fully-automated way where you only have to import a module (could even be object-d => no explicit import required) and have all tests run. This can be done by mixing in a module constructor in every module. From that constructor you'd call std.unittest.registerTest(...) and in the main function you could then call std.unittest.runTests(). This way you never need to name or know the tested modules. IIRC std.benchmark used that approach, with the drawback of manual mixins and module constructor cycle issues. And module constructors are not needed at all for this, there is http://dlang.org/traits.html#getUnitTests Sure, but I was thinking about a runtime registration scheme as explained above.
Re: Named unittests
On 3/31/15 1:04 PM, Jacob Carlborg wrote:
On 2015-03-31 16:55, Meta wrote:
Aren't unittest blocks just special functions? If that's the case, there
should be no problem being able to give them names. It seems to me that
it would entail the lifting of a restriction rather than a real language
change.
Before:
unittest
{
assert(1 == 1);
}
After:
unittest checkBasicLaw
{
assert(1 == 1);
}
I prefer a UDA accepting a string, this can contain spaces and it's not
limited to identifier names.
I used to think the same, but then I figured a bit of structure might be
preferable. -- Andrei
Re: Named unittests
On 2015-03-31 16:55, Meta wrote:
Aren't unittest blocks just special functions? If that's the case, there
should be no problem being able to give them names. It seems to me that
it would entail the lifting of a restriction rather than a real language
change.
Before:
unittest
{
assert(1 == 1);
}
After:
unittest checkBasicLaw
{
assert(1 == 1);
}
I prefer a UDA accepting a string, this can contain spaces and it's not
limited to identifier names.
--
/Jacob Carlborg
Re: Named unittests
On 3/31/15 7:45 AM, Idan Arye wrote: On Tuesday, 31 March 2015 at 13:34:24 UTC, Dicebot wrote: On Tuesday, 31 March 2015 at 10:25:57 UTC, Idan Arye wrote: I understand the preference to librarize as much as possible, but I don't think the desire to sacrifice every possible bit of convenience to avoid the tiniest changes to the language is always beneficial. I don't say that implementing everything inside the compiler is good either though, but in many cases some slight changes to the language can make the library solution so much more simple and elegant. In this case, allowing to name a unittest should be a very simple language change that'll make any library implementation of the rest of the feature more elegant to use, simpler to implement, and more consistent with alternative library implementations. It isn't simple at all. Name is just one of many meta-values you commonly want to attach to unittest block. Some others: description, dependency, parallelization, benchmark tag, I/O indicator. It is simply impossible to foresee it all in a language feature - but it is exactly kind of data UDA are designed for. All we need is to enhance/fix the language to actually make using of that information convenient. But unittests already have names(http://dpaste.dzfl.pl/b15e94000f15), so the only required change is to allow the user to specify that name. This should be much simpler than adding entirely new fields. Interesting point! -- Andrei
Re: Named unittests
On 3/31/15 7:55 AM, Meta wrote:
On Monday, 30 March 2015 at 21:58:13 UTC, Dicebot wrote:
I'd prefer putting alternative test runner into Phobos instead which
will support `@name("Something") unittest { }`
Aren't unittest blocks just special functions? If that's the case, there
should be no problem being able to give them names. It seems to me that
it would entail the lifting of a restriction rather than a real language
change.
Before:
unittest
{
assert(1 == 1);
}
After:
unittest checkBasicLaw
{
assert(1 == 1);
}
That's what I had in mind, too. -- Andrei
Re: Named unittests
On Monday, 30 March 2015 at 21:58:13 UTC, Dicebot wrote:
I'd prefer putting alternative test runner into Phobos instead
which will support `@name("Something") unittest { }`
Aren't unittest blocks just special functions? If that's the
case, there should be no problem being able to give them names.
It seems to me that it would entail the lifting of a restriction
rather than a real language change.
Before:
unittest
{
assert(1 == 1);
}
After:
unittest checkBasicLaw
{
assert(1 == 1);
}
Re: Named unittests
On Tuesday, 31 March 2015 at 13:34:24 UTC, Dicebot wrote: On Tuesday, 31 March 2015 at 10:25:57 UTC, Idan Arye wrote: I understand the preference to librarize as much as possible, but I don't think the desire to sacrifice every possible bit of convenience to avoid the tiniest changes to the language is always beneficial. I don't say that implementing everything inside the compiler is good either though, but in many cases some slight changes to the language can make the library solution so much more simple and elegant. In this case, allowing to name a unittest should be a very simple language change that'll make any library implementation of the rest of the feature more elegant to use, simpler to implement, and more consistent with alternative library implementations. It isn't simple at all. Name is just one of many meta-values you commonly want to attach to unittest block. Some others: description, dependency, parallelization, benchmark tag, I/O indicator. It is simply impossible to foresee it all in a language feature - but it is exactly kind of data UDA are designed for. All we need is to enhance/fix the language to actually make using of that information convenient. But unittests already have names(http://dpaste.dzfl.pl/b15e94000f15), so the only required change is to allow the user to specify that name. This should be much simpler than adding entirely new fields.
Re: Named unittests
On Tuesday, 31 March 2015 at 14:27:45 UTC, Johannes Pfau wrote:
Am Tue, 31 Mar 2015 13:31:58 +
schrieb "Dicebot" :
On Tuesday, 31 March 2015 at 09:08:47 UTC, Johannes Pfau wrote:
> Am Mon, 30 Mar 2015 14:52:36 -0700
> schrieb Andrei Alexandrescu :
>
>> We're having a strong need for named unittests at Facebook
>> for
>> multiple reasons.
> Right now the default implementation works by putting
> pointers to a
> test function into ModuleInfo. We could instead add arrays
> of some
> 'unittest information' struct to ModuleInfo to support names
> etc. But
> we can't make this as extensible and powerful as it should
> be: In order
> to support arbitrary UDAs we'd always need some kind of
> UDA=>runtime
> serialization.
Most powerful solution would be to simply put attributes for
unittest blocks in runtime information for tests (using RTTI
it should be possible to define such variadic structure in
similar manner as D-style variadic function arguments).
Yes, one array of TestAttribute[]
struct TestAttribute
{
TypeInfo ti;
void* value;
}
per unittest and an array of unittests in ModuleInfo would
likely work.
But this works only for unittests and it's still restricted*.
I'd prefer
a more general solution which also works for benchmarks and
similar
usecases.
*For example you can extract file/line
information (useful for IDEs) but you'll have to add explicit
@fileline
UDAs to the tests. With compile-time reflection you can gather
this
information without additional UDAs.
Problem with more generic solution is that it is also note
demanding and potentially more intrusive - right now UDA don't
affect actual types at all. I'd prefer something simple and
practical that can work for texts right now - those are special
enough case to justify dedicated approach (even if generic rtinfo
gets added later)
Re: Named unittests
Am Tue, 31 Mar 2015 13:31:58 +
schrieb "Dicebot" :
> On Tuesday, 31 March 2015 at 09:08:47 UTC, Johannes Pfau wrote:
> > Am Mon, 30 Mar 2015 14:52:36 -0700
> > schrieb Andrei Alexandrescu :
> >
> >> We're having a strong need for named unittests at Facebook for
> >> multiple reasons.
>
> > Right now the default implementation works by putting pointers
> > to a
> > test function into ModuleInfo. We could instead add arrays of
> > some
> > 'unittest information' struct to ModuleInfo to support names
> > etc. But
> > we can't make this as extensible and powerful as it should be:
> > In order
> > to support arbitrary UDAs we'd always need some kind of
> > UDA=>runtime
> > serialization.
>
> Most powerful solution would be to simply put attributes for
> unittest blocks in runtime information for tests (using RTTI it
> should be possible to define such variadic structure in similar
> manner as D-style variadic function arguments).
Yes, one array of TestAttribute[]
struct TestAttribute
{
TypeInfo ti;
void* value;
}
per unittest and an array of unittests in ModuleInfo would likely work.
But this works only for unittests and it's still restricted*. I'd prefer
a more general solution which also works for benchmarks and similar
usecases.
*For example you can extract file/line
information (useful for IDEs) but you'll have to add explicit @fileline
UDAs to the tests. With compile-time reflection you can gather this
information without additional UDAs.
Re: Named unittests
Am Tue, 31 Mar 2015 13:31:58 + schrieb "Dicebot" : > > But here's the problem: > > > > 1) The compile time approach requires some kind > >of explicit registration of the unittests. At least one > > mixin per > >module. > > 2) This mixin will usually provide a module constructor. But > >using module constructors will cause issues with cycle > > detection. > > This is not really true. You only need one mixin in the root > module(s), rest can be iterated recursively by test runner itself > using __traits(allMembers) reflection. Only issue with that > approach is that last time I checked there was a DMD bug which > prevented from getting complete list of imported modules via > allMembers. Should be fixable. But then you still have to explicitly import (or at least name) all modules that should be tested. This is a drawback compared to the current builtin-unittests where you do not have to explicitly import to-be-tested modules. I was thinking of a fully-automated way where you only have to import a module (could even be object-d => no explicit import required) and have all tests run. This can be done by mixing in a module constructor in every module. From that constructor you'd call std.unittest.registerTest(...) and in the main function you could then call std.unittest.runTests(). This way you never need to name or know the tested modules. IIRC std.benchmark used that approach, with the drawback of manual mixins and module constructor cycle issues. > > And module constructors are not needed at all for this, there is > http://dlang.org/traits.html#getUnitTests Sure, but I was thinking about a runtime registration scheme as explained above.
Re: Named unittests
On 3/31/15 9:28 AM, Steven Schveighoffer wrote:
On 3/31/15 9:05 AM, w0rp wrote:
On Tuesday, 31 March 2015 at 12:33:31 UTC, Steven Schveighoffer wrote:
On 3/30/15 5:58 PM, Dicebot wrote:
I'd prefer putting alternative test runner into Phobos instead which
will support `@name("Something") unittest { }`
Yes, this is one of the benefits I touted 2 years ago when I asked for
module RTInfo -- we can use this information in the runtime to
instrument how we run unit tests.
We still don't have module RTInfo.
And yes, then it can be a library solution. unittests are a language
feature, but only in how they are compiled and linked. The runtime is
fully responsible for how they are run. All we need is a way to tell
the compiler how to describe them to the runtime.
ModuleInfo does actually exist, but it's not documented. I'm not sure if
it's usable for this purpose though. Maybe?
No, I mean this: https://github.com/D-Programming-Language/dmd/pull/2271
Essentially, you have user-defined generation of runtime info stored
inside the ModuleInfo. When this is working, we can do whatever we want
for unit tests via attributes.
See here too:
https://issues.dlang.org/show_bug.cgi?id=10023
-Steve
Re: Named unittests
On Tuesday, 31 March 2015 at 09:08:47 UTC, Johannes Pfau wrote: Am Mon, 30 Mar 2015 14:52:36 -0700 schrieb Andrei Alexandrescu : We're having a strong need for named unittests at Facebook for multiple reasons. Right now the default implementation works by putting pointers to a test function into ModuleInfo. We could instead add arrays of some 'unittest information' struct to ModuleInfo to support names etc. But we can't make this as extensible and powerful as it should be: In order to support arbitrary UDAs we'd always need some kind of UDA=>runtime serialization. Most powerful solution would be to simply put attributes for unittest blocks in runtime information for tests (using RTTI it should be possible to define such variadic structure in similar manner as D-style variadic function arguments). The other option is getting a list of unittests at compile time. (__traits allMEmbers, etc). AFAIK all unittest frameworks supporting UDA use this approach. This is much more powerful and extensible. It might make sense to switch the default implementation. But here's the problem: 1) The compile time approach requires some kind of explicit registration of the unittests. At least one mixin per module. 2) This mixin will usually provide a module constructor. But using module constructors will cause issues with cycle detection. This is not really true. You only need one mixin in the root module(s), rest can be iterated recursively by test runner itself using __traits(allMembers) reflection. Only issue with that approach is that last time I checked there was a DMD bug which prevented from getting complete list of imported modules via allMembers. Should be fixable. And module constructors are not needed at all for this, there is http://dlang.org/traits.html#getUnitTests
Re: Named unittests
On Tuesday, 31 March 2015 at 10:25:57 UTC, Idan Arye wrote: I understand the preference to librarize as much as possible, but I don't think the desire to sacrifice every possible bit of convenience to avoid the tiniest changes to the language is always beneficial. I don't say that implementing everything inside the compiler is good either though, but in many cases some slight changes to the language can make the library solution so much more simple and elegant. In this case, allowing to name a unittest should be a very simple language change that'll make any library implementation of the rest of the feature more elegant to use, simpler to implement, and more consistent with alternative library implementations. It isn't simple at all. Name is just one of many meta-values you commonly want to attach to unittest block. Some others: description, dependency, parallelization, benchmark tag, I/O indicator. It is simply impossible to foresee it all in a language feature - but it is exactly kind of data UDA are designed for. All we need is to enhance/fix the language to actually make using of that information convenient.
Re: Named unittests
On 3/31/15 9:05 AM, w0rp wrote:
On Tuesday, 31 March 2015 at 12:33:31 UTC, Steven Schveighoffer wrote:
On 3/30/15 5:58 PM, Dicebot wrote:
I'd prefer putting alternative test runner into Phobos instead which
will support `@name("Something") unittest { }`
Yes, this is one of the benefits I touted 2 years ago when I asked for
module RTInfo -- we can use this information in the runtime to
instrument how we run unit tests.
We still don't have module RTInfo.
And yes, then it can be a library solution. unittests are a language
feature, but only in how they are compiled and linked. The runtime is
fully responsible for how they are run. All we need is a way to tell
the compiler how to describe them to the runtime.
ModuleInfo does actually exist, but it's not documented. I'm not sure if
it's usable for this purpose though. Maybe?
No, I mean this: https://github.com/D-Programming-Language/dmd/pull/2271
Essentially, you have user-defined generation of runtime info stored
inside the ModuleInfo. When this is working, we can do whatever we want
for unit tests via attributes.
-Steve
Re: Named unittests
On Tuesday, 31 March 2015 at 12:33:31 UTC, Steven Schveighoffer
wrote:
On 3/30/15 5:58 PM, Dicebot wrote:
I'd prefer putting alternative test runner into Phobos instead
which
will support `@name("Something") unittest { }`
Yes, this is one of the benefits I touted 2 years ago when I
asked for module RTInfo -- we can use this information in the
runtime to instrument how we run unit tests.
We still don't have module RTInfo.
And yes, then it can be a library solution. unittests are a
language feature, but only in how they are compiled and linked.
The runtime is fully responsible for how they are run. All we
need is a way to tell the compiler how to describe them to the
runtime.
-Steve
ModuleInfo does actually exist, but it's not documented. I'm not
sure if it's usable for this purpose though. Maybe?
Re: Named unittests
On 3/30/15 5:58 PM, Dicebot wrote:
I'd prefer putting alternative test runner into Phobos instead which
will support `@name("Something") unittest { }`
Yes, this is one of the benefits I touted 2 years ago when I asked for
module RTInfo -- we can use this information in the runtime to
instrument how we run unit tests.
We still don't have module RTInfo.
And yes, then it can be a library solution. unittests are a language
feature, but only in how they are compiled and linked. The runtime is
fully responsible for how they are run. All we need is a way to tell the
compiler how to describe them to the runtime.
-Steve
Re: Named unittests
On Tuesday, 31 March 2015 at 12:10:15 UTC, w0rp wrote:
On Tuesday, 31 March 2015 at 11:39:02 UTC, tcak wrote:
On Monday, 30 March 2015 at 21:57:33 UTC, Andrei Alexandrescu
wrote:
On 3/30/15 2:55 PM, Panke wrote:
I've implemented this in a library and I'm sure others have
as well. Are
you sure, you want a language solution?
With attributes? That might be palatable but only as a
standard solution. I'd want to add that to Phobos. -- Andrei
The "unittest" is a language thing. Why would name support be
put into library I don't get it.
unittest{} => Unnamed
unittest!"Testing new classes" {} => Named (Explained)
Becaused adding language features takes longer than using a
library, and every single feature, no matter how seemingly
simple, will increase the number of langauge bugs and lead to
more odd things happening.
I can understand this, and I agree with you. But this is no
different then ordinary businesses those do not care about
quality of their products or customer service as long as they
make profit. And as you know, we mostly swear at them due to this.
Re: Named unittests
On Tuesday, 31 March 2015 at 11:39:02 UTC, tcak wrote:
On Monday, 30 March 2015 at 21:57:33 UTC, Andrei Alexandrescu
wrote:
On 3/30/15 2:55 PM, Panke wrote:
I've implemented this in a library and I'm sure others have
as well. Are
you sure, you want a language solution?
With attributes? That might be palatable but only as a
standard solution. I'd want to add that to Phobos. -- Andrei
The "unittest" is a language thing. Why would name support be
put into library I don't get it.
unittest{} => Unnamed
unittest!"Testing new classes" {} => Named (Explained)
Becaused adding language features takes longer than using a
library, and every single feature, no matter how seemingly
simple, will increase the number of langauge bugs and lead to
more odd things happening.
Re: Named unittests
On Monday, 30 March 2015 at 21:57:33 UTC, Andrei Alexandrescu
wrote:
On 3/30/15 2:55 PM, Panke wrote:
I've implemented this in a library and I'm sure others have as
well. Are
you sure, you want a language solution?
With attributes? That might be palatable but only as a standard
solution. I'd want to add that to Phobos. -- Andrei
The "unittest" is a language thing. Why would name support be put
into library I don't get it.
unittest{} => Unnamed
unittest!"Testing new classes" {} => Named (Explained)
Re: Named unittests
I understand the preference to librarize as much as possible, but I don't think the desire to sacrifice every possible bit of convenience to avoid the tiniest changes to the language is always beneficial. I don't say that implementing everything inside the compiler is good either though, but in many cases some slight changes to the language can make the library solution so much more simple and elegant. In this case, allowing to name a unittest should be a very simple language change that'll make any library implementation of the rest of the feature more elegant to use, simpler to implement, and more consistent with alternative library implementations. Another argument in favor of language solution - sometimes the unittests take the better part of the compilation process, especially when the project is a heavily-templated library(Phobos is a prime example). When working on a bug discovered by a unittest, that means you either have to either build all unittests every time you want to check your changes, or copy the unittest code a `main` function and work on it there. If unittest names are part of the language, it could be possible to instruct the compiler to only build a single unittest, which will make the iterations much faster.
Re: Named unittests
Am Mon, 30 Mar 2015 14:52:36 -0700
schrieb Andrei Alexandrescu :
> We're having a strong need for named unittests at Facebook for
> multiple reasons.
>
As there have already been suggestions to use UDAs I think we should
discuss the fundamental difficulty with unittests:
Right now the default implementation works by putting pointers to a
test function into ModuleInfo. We could instead add arrays of some
'unittest information' struct to ModuleInfo to support names etc. But
we can't make this as extensible and powerful as it should be: In order
to support arbitrary UDAs we'd always need some kind of UDA=>runtime
serialization.
The other option is getting a list of unittests at compile time.
(__traits allMEmbers, etc). AFAIK all unittest frameworks
supporting UDA use this approach. This is much more powerful and
extensible. It might make sense to switch the default implementation.
But here's the problem:
1) The compile time approach requires some kind
of explicit registration of the unittests. At least one mixin per
module.
2) This mixin will usually provide a module constructor. But
using module constructors will cause issues with cycle detection.
There are some similar usecases with exactly the same issues (benchmark
functions, ...). So we'd need a general solution for this.
One possible solution for problem 1 would be to have mixins that
automatically get mixed in once at module scope in imported modules (a
little bit like RTInfo):
---
module benchmark;
@automodule mixin template Foo()
{
shared static this() @nocycle
{
//traits allmembers, ...
}
}
---
module test;
import benchmark; //=> automatically inserts mixin Foo;
---
To solve problem problem 2 a nocycle UDA could be used to ignore
cycle checking (or @depends("benchmark") to only list explicit
dependencies).
Re: Named unittests
On Monday, 30 March 2015 at 21:52:35 UTC, Andrei Alexandrescu
wrote:
We're having a strong need for named unittests at Facebook for
multiple reasons.
1. We have sophisticated tooling that verifies whether
unittests are flaky. The automated monitor (for e.g. C++)
figures whether a given unittest fails several times across
several commits. Unittests are identified by name; relying on
file/line is impossible because the line of a failure is not
stable across changes.
2. Again for efficient automated testing and flakiness
detection, one should be able to run only a subset of unittests
by mentioning them by line in the command line. Note that this
implies there's no interdependency between distinct unittests,
which is fine because the new ability is opt-on; I'd say is
pure style anyway.
3. Mentioning unittest names in failure messages helps human
communication (e.g. "AddPeer is failing after your change").
This is impossible with file and line numbers.
I'd like to make a DIP for named unittests. Who can help me
with that?
Andrei
Would it be doable to make something like
unittest("Say my name") {
// tests
}
and still be backward compatible ?
In the other hand, other proposed solutions seem to be good imho.
Re: Named unittests
On 2015-03-30 23:52, Andrei Alexandrescu wrote:
We're having a strong need for named unittests at Facebook for multiple
reasons.
1. We have sophisticated tooling that verifies whether unittests are
flaky. The automated monitor (for e.g. C++) figures whether a given
unittest fails several times across several commits. Unittests are
identified by name; relying on file/line is impossible because the line
of a failure is not stable across changes.
2. Again for efficient automated testing and flakiness detection, one
should be able to run only a subset of unittests by mentioning them by
line in the command line. Note that this implies there's no
interdependency between distinct unittests, which is fine because the
new ability is opt-on; I'd say is pure style anyway.
3. Mentioning unittest names in failure messages helps human
communication (e.g. "AddPeer is failing after your change"). This is
impossible with file and line numbers.
I'd like to make a DIP for named unittests. Who can help me with that?
I completely agree. I always thought that the built-in unit test support
wasn't sufficient. All of the above should be possible to implement in
library code without needing to change the language.
Now, it depends on how far you want to go. We could do something simple
as adding a UDA, which has already been suggested by others:
@name("this is a test") unittest
{
assert(false);
}
Personally I would like a complete testing framework like RSpec. I have
a very simple implementation [1] of this:
unittest
{
describe("std.uni", {
describe("toUpper", {
it("converts a string to uppercase", {
"foo".toUpper.should.eq("FOO")
});
});
describe("toLower", {
it("converts a string to lowercase", {
"Foo".toLower.should.eq("foo")
});
});
});
}
With RSpec, which supports multiple formatters, it can look like this:
---
Randomized with seed 28149
std.uni
toUpper
converts a string to uppercase
toLower
converts a string to lowercase
Finished in 0.00078 seconds (files took 0.0931 seconds to load)
2 examples, 0 failures
Randomized with seed 28149
---
This shows a failing test:
---
Randomized with seed 57730
std.uni
toLower
converts a string to lowercase
toUpper
converts a string to uppercase (FAILED - 1)
Failures:
1) std.uni toUpper converts a string to uppercase
Failure/Error: 'foo'.should == 'FOO'
expected: "FOO"
got: "foo" (using ==)
# ./spec/foo_spec.rb:6:in `block (3 levels) in '
Finished in 0.00298 seconds (files took 0.08553 seconds to load)
2 examples, 1 failure
Failed examples:
rspec ./spec/foo_spec.rb:5 # std.uni toUpper converts a string to uppercase
Randomized with seed 57730
---
Or with the TextMate formatter [2]. It shows both passing and failing
test. When a test fails to get a link pointing back to the editor and a
syntax highlighted snippet of the failing source code.
With RSpec it's also possible to specify the --line-number flag which
will only run the tests matching a given line number. Or the --example
flag which will run all examples (tests) matching the given string.
Is any of this interesting to have in Phobos? Otherwise I'll continue
working on my own framework.
[1] https://github.com/jacob-carlborg/dspec
[2] http://thejqr.com/2009/02/06/textmate-rspec-and-dot-spec-party.html
--
/Jacob Carlborg
Re: Named unittests
On 31/03/2015 10:52 a.m., Andrei Alexandrescu wrote:
We're having a strong need for named unittests at Facebook for multiple
reasons.
1. We have sophisticated tooling that verifies whether unittests are
flaky. The automated monitor (for e.g. C++) figures whether a given
unittest fails several times across several commits. Unittests are
identified by name; relying on file/line is impossible because the line
of a failure is not stable across changes.
2. Again for efficient automated testing and flakiness detection, one
should be able to run only a subset of unittests by mentioning them by
line in the command line. Note that this implies there's no
interdependency between distinct unittests, which is fine because the
new ability is opt-on; I'd say is pure style anyway.
3. Mentioning unittest names in failure messages helps human
communication (e.g. "AddPeer is failing after your change"). This is
impossible with file and line numbers.
I'd like to make a DIP for named unittests. Who can help me with that?
Andrei
/**
* Does something funny
*
* Uses writeln, to tell the user off.
* And perhaps something else too.
*/
unittest {
import std.stdio;
import std.ddoc;
writeln("BAD USER, now say \"NO\"");
assert(readln() == "NO");
assert(ddocParse(__DDOC__).summary == "Does something funny");
}
Only one addition required. __DDOC__ to the compiler. The rest is all
library solution. Oh and assert should probably be aware of it.
void assert(bool value, string text="", string mod = __MODULE__, uint
line = __LINE__, string ddoc = __DDOC__);
Well along those lines for a prototype.
Re: Named unittests
On Monday, 30 March 2015 at 23:02:51 UTC, Mathias Lang wrote:
We do have an `@name` as UDA in Vibe.d, so that'll be a
breaking change
(But `@NamedUnittest("name")` will do).
I also think it should be a library solution.
I can't afford to care about used names for attributes, same as
we still add new symbols to Phobos (though those do break user
code for same reason). They can be disambugated using module
system.
(and yes, this is HUGE drawback of D module system)
Re: Named unittests
We do have an `@name` as UDA in Vibe.d, so that'll be a breaking change
(But `@NamedUnittest("name")` will do).
I also think it should be a library solution.
2015-03-31 0:21 GMT+02:00 Kapps via Digitalmars-d <
digitalmars-d@puremagic.com>:
> On Monday, 30 March 2015 at 21:52:35 UTC, Andrei Alexandrescu wrote:
>
>>
>> I'd like to make a DIP for named unittests. Who can help me with that?
>>
>>
>> Andrei
>>
>
> I agree that using library-defined annotations would be a better approach
> than language changes. Currently things like tested use the form
> @name("AddPeer") unittest { /* ... */ }
> Which is nice, because then you can extend it as desired, such as
> @parallel @name("AddPeer") unittest
>
> The main issue is that at this point, practically every single person has
> defined:
> struct Name {
> string val;
> }
> string name(string val) { return Name(val); }
>
> Adding common attributes such as this to Phobos or druntime and
> potentially have the default unittest runner include them would be good.
>
Re: Named unittests
On Monday, 30 March 2015 at 21:52:35 UTC, Andrei Alexandrescu
wrote:
I'd like to make a DIP for named unittests. Who can help me
with that?
Andrei
I agree that using library-defined annotations would be a better
approach than language changes. Currently things like tested use
the form
@name("AddPeer") unittest { /* ... */ }
Which is nice, because then you can extend it as desired, such as
@parallel @name("AddPeer") unittest
The main issue is that at this point, practically every single
person has defined:
struct Name {
string val;
}
string name(string val) { return Name(val); }
Adding common attributes such as this to Phobos or druntime and
potentially have the default unittest runner include them would
be good.
Re: Named unittests
On Monday, 30 March 2015 at 21:58:13 UTC, Dicebot wrote:
I'd prefer putting alternative test runner into Phobos instead
which will support `@name("Something") unittest { }`
I agree that an annotation (Probably defined in object.d) is the
best way to handle this.
Here's something I don't think many people know: You can't use
`~` in deprecated() attributes. You can only put a string literal
in the parenthesis. If you want to break messages over multiple
lines you need to rely on implicit string concatenation.
Re: Named unittests
Backward compatibility is not an issue seeing this is a pure addition. Andrei I mean that it lives peacefully alongside normal unittest blocks.
Re: Named unittests
On 3/30/15 2:59 PM, Panke wrote:
I've implemented this in a library and I'm sure others have as well.
Are you sure, you want a language solution?
Basic principle:
---
unittest {
testCase("a testcase",
{
// actual test code here
});
}
---
Looks a bit too much work.
Testcase registers the structure and the runtime just runs the unittest
as is does today and than calls all closures registered during the
unittest. Makes this fully backward compatible.
Backward compatibility is not an issue seeing this is a pure addition.
Andrei
Re: Named unittests
On Monday, 30 March 2015 at 21:57:33 UTC, Andrei Alexandrescu wrote: On 3/30/15 2:55 PM, Panke wrote: I've implemented this in a library and I'm sure others have as well. Are you sure, you want a language solution? With attributes? That might be palatable but only as a standard solution. I'd want to add that to Phobos. -- Andrei In my case: no. I wrote it before UDA and wanted to be able to generate testcases at runtime. Used this to register a test case for a SAT solver for every instance in a subdirectory etc.
Re: Named unittests
Sounds like something that should be added as a library rather than the core language.
Re: Named unittests
On Monday, 30 March 2015 at 21:52:35 UTC, Andrei Alexandrescu wrote: We're having a strong need for named unittests at Facebook for multiple reasons. 1. We have sophisticated tooling that verifies whether unittests are flaky. The automated monitor (for e.g. C++) figures whether a given unittest fails several times across several commits. Unittests are identified by name; relying on file/line is impossible because the line of a failure is not stable across changes. 2. Again for efficient automated testing and flakiness detection, one should be able to run only a subset of unittests by mentioning them by line in the command line. Note that this implies there's no interdependency between distinct unittests, which is fine because the new ability is opt-on; I'd say is pure style anyway. 3. Mentioning unittest names in failure messages helps human communication (e.g. "AddPeer is failing after your change"). This is impossible with file and line numbers. I'd like to make a DIP for named unittests. Who can help me with that? Andrei I've implemented this in a library and I'm sure others have as well. Are you sure, you want a language solution?
Re: Named unittests
I'd prefer putting alternative test runner into Phobos instead
which will support `@name("Something") unittest { }`
Re: Named unittests
I've implemented this in a library and I'm sure others have as
well. Are you sure, you want a language solution?
Basic principle:
---
unittest {
testCase("a testcase",
{
// actual test code here
});
}
---
Testcase registers the structure and the runtime just runs the
unittest as is does today and than calls all closures registered
during the unittest. Makes this fully backward compatible.
Re: Named unittests
On 3/30/15 2:55 PM, Panke wrote: I've implemented this in a library and I'm sure others have as well. Are you sure, you want a language solution? With attributes? That might be palatable but only as a standard solution. I'd want to add that to Phobos. -- Andrei
Named unittests
We're having a strong need for named unittests at Facebook for multiple reasons. 1. We have sophisticated tooling that verifies whether unittests are flaky. The automated monitor (for e.g. C++) figures whether a given unittest fails several times across several commits. Unittests are identified by name; relying on file/line is impossible because the line of a failure is not stable across changes. 2. Again for efficient automated testing and flakiness detection, one should be able to run only a subset of unittests by mentioning them by line in the command line. Note that this implies there's no interdependency between distinct unittests, which is fine because the new ability is opt-on; I'd say is pure style anyway. 3. Mentioning unittest names in failure messages helps human communication (e.g. "AddPeer is failing after your change"). This is impossible with file and line numbers. I'd like to make a DIP for named unittests. Who can help me with that? Andrei
