Re: Mono-D 2.0 - XamarinStudio 5.0 support, completion improvements
You are my personal hero!
Re: Announcing TitaniumD - A D Binding for the Botan Cryptography Library
On 4/30/14, 6:28 PM, Adam Wilson wrote: [snip] http://www.reddit.com/r/programming/comments/24gpb9/titaniumd_a_d_binding_for_the_botan_cryptography/ https://www.facebook.com/dlang.org/posts/839257452754604?stream_ref=10 https://hn.algolia.com/#!/story/forever/0/titaniumd https://twitter.com/D_Programming/status/461897947270369281 Andrei
Re: Announcing TitaniumD - A D Binding for the Botan Cryptography Library
Adam, this is very cool! do you have any examples showing its use? In particular, examples highlighting D code using AES and SHA libs thanks! brad
Re: Announcing TitaniumD - A D Binding for the Botan Cryptography Library
On Thu, 01 May 2014 14:55:05 -0700, brad clawsie b...@b7j0c.org wrote:
Adam, this is very cool!
do you have any examples showing its use? In particular, examples
highlighting D code using AES and SHA libs
thanks!
brad
You're welcome. There are no example yet. And the build script only
supports Windows. Pull requests using *make tools most welcome.
A little walkthrough:
Build TitaniumCore, TitaniumD and TDI as static libraries. TitaniumCore
and TitaniumD can be compiled with any C++ compiler and the availability
of Boost Filesystem and ASIO. Although you'll have to specify the paths to
Boost.
Once those libraries are built you'll need to pass all three of them to
DMD along with your project.
Then the following code should work:
module test;
import std.stdio;
import Titanium.TDI; //Make sure to specify the correct search path with
-I on the DMD commandline.
void main(string[] args)
{
auto t = new TitaniumLibrary();
//Hash Test Vectors
string vec1 = The quick brown fox jumps over the lazy dog;
string vec2 = The quick brown fox jumps over the lazy dog.;
//Hash Testing
writeln(HashSHA2(HashSize.Size256, vec1));
SHA2 h = new SHA2(HashSize.Size256);
writeln(h.Process(vec2));
AES c = new AES(CipherKeySize.Key128, CipherMode.GCM, 16);
ubyte[] enc1 = c.Encrypt(cast(ubyte[])vec1.dup);
string dec1 = cast(string)c.Decrypt(enc1);
ubyte[] enc2 = c.Encrypt(cast(ubyte[])vec2.dup);
string dec2 = cast(string)c.Decrypt(enc2);
writeln(enc1);
writeln(dec1);
writeln(enc2);
writeln(dec2);
}
The above sample shows SHA2 and AES-128/GCM.
Please submit bug reports and pull requests on GitHub if you run into any
problems!
--
Adam Wilson
IRC: LightBender
Project Coordinator
The Aurora Project
Re: Mono-D 2.0 - XamarinStudio 5.0 support, completion improvements
Great work, thank you !
Re: python vs d
On Wed, 2014-04-30 at 15:21 -0400, Nick Sabalausky via Digitalmars-d wrote: […] I've heard this a lot, but I've yet to hear anyone explain concretely how this dynamic mindset causes the lack of things like static code checks and low-overhead primitives from being actual drawbacks. Maybe it really is a case of me not getting it, but it always sounds to me like this dynamic mindset gets around these issues simply by ignoring them. Since I don't personally do heavy development in dynamic languages, I'd be interested in a strong rebuttal to this. The best way of approaching this is not to try and attempt a long theoretical essay, but to try (a possibly lengthy exchange of question and answer accompanied by actually writing code in both Python and D. Also looking at things like Rosettacode and all Bearophile and others' work there. There are two questions above to kick start things, let's take the second first: Low-overhead primitives: This immediately implies you are looking for raw CPU-bound performance or you are undertaking premature optimization. Python has no direct play in this game. If a Python code requires out and out CPU-bound performance, you profile to find the small bit of the total code that is performance critical. If it is not already a function you extract it as a function, put it in it's own module and Cythonize it (*). Still Python code but with a few judicious extra annotations so as to enable Cython to generate C which is then compiled to native code. Static code checks: The object (aka data) model, and variable model of Python (and indeed other dynamic languages) means there cannot be any static code checks. At this point, I have to yield (in this co-routine exchange :-) with a question: what is it about static code checks that you feel you have to have in order to be able to program? (*) There are other techniques, but this is the lowest overhead in terms of programmer time, and it gives excellent runtime speed-up. -- Russel. = Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.win...@ekiga.net 41 Buckmaster Roadm: +44 7770 465 077 xmpp: rus...@winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Re: D For A Web Developer
Am 01.05.2014 01:05, schrieb Ary Borenszweig: On 4/30/14, 10:38 AM, Adam D. Ruppe wrote: On Wednesday, 30 April 2014 at 04:19:15 UTC, Russel Winder via Digitalmars-d wrote: Of course, I doubt the gap will ever be closed, since Ruby's awfulness isn't dependent on my experience level. It's not like it will ever get static typing even if I used it all the time. Nah, that will never happen. But you can get very close ( http://crystal-lang.org/ ), although you loose some dynamism at runtime... Nice to see Crystal still alive.
Re: Parallel execution of unittests
On Wed, 30 Apr 2014 22:32:33 -0700 Andrei Alexandrescu via Digitalmars-d digitalmars-d@puremagic.com wrote: On 4/30/14, 10:01 PM, Jonathan M Davis via Digitalmars-d wrote: I'm all for parallelizing all unittest blocks that are pure, as doing so would be safe, but I think that we're making a big mistake if we try and insist that all unittest blocks be able to be run in parallel. Any that aren't pure are not guaranteed to be parallelizable, and any which access system resources or other global, mutable state stand a good chance of breaking. There are a number of assumptions here: (a) most unittests that can be effectively parallelized can be actually inferred (or declared) as pure; (b) most unittests that cannot be inferred as pure are likely to break; (c) it's a big deal if unittests break. I question all of these assumptions. In particular I consider unittests that depend on one another an effective antipattern that needs to be eradicated. Even if they don't depend on each other, they can depend on the system. std.file's unit tests will break if we parallelize them, because it operates on files and directories, and many of those tests operate on the same temp directories. That can be fixed by changing the tests, but it will break the tests. Other tests _can't_ be fixed if we force them to run in parallel. For instance, some of std.datetime's unit tests set the local time zone of the system in order to test that LocalTime works correctly. That sets it for the whole program, so all threads will be affected even if they're running other tests. Right now, this isn't a problem, because those tests set the timezone at their start and reset it at their end. But if they were made to run in parallel with any other tests involving LocalTime, there's a good chance that those tests would have random test failures. They simply can't be run in parallel due to a system resource that we can't make thread-local. So, regardless of how we want to mark up unittest blocks as parallelizable or not parallelizable (be it explicit, implict, using pure, or using something else), we do need a way to make it so that a unittest block is not run in parallel with any other unittest block. We can guarantee that pure functions can safely be run in parallel. We _cannot_ guarantee that impure functions can safely be run in parallel. I'm sure that many impure unittest functions could be safely run in parallel, but it would require that the programmer verify that if we don't want undefined behavior - just like programmers have to verify that @system code is actually @safe. Simply running all unittest blocks in parallel is akin to considering @system code @safe in a particular piece of code simply because by convention that code should be @safe. pure allows us to detect guaranteed, safe parallelizability. If we want to define some other way to make it so a unittest block can be marked as parallelizable regardless of purity, then fine. But automatically parallelizing impure functions means that we're going to have undefined behavior for those unittest functions, and I really think that that is a bad idea - in addition to the fact that some unittest blocks legitimately cannot be run in parallel due to the use of system resources, so parallelizing them _will_ not only break them but make them impossible to write in a way that's not broken without adding mutexes to the unittest blocks to stop the test runner from running them in parallel. And IMHO, if we end up having to do that anywhere, we've done something very wrong with how unit tests work. - Jonathan M Davis
Re: More radical ideas about gc and reference counting
Walter Bright: Actually, I think inference of the ability to implicitly and safely convert from mutable to immutable or shared to be very powerful. I also think it is superior than requiring the user to add ever more annotations, which some research (and my experience) shows that users are reluctant to do. I agree the people are reluctant to add annotations, but I think it's also depends on the return of investment of the specific annotation. I am annotating all my code carefully with pure, but the ROI of all those purity annotations is not great. And I agree the piling of special cases is a not clean design. It's now simpler to ask the compiler if a certain assignment to immutable is allowed or not. Because I can't remember the supported cases. So I suggest a more principled and visible approach at the problem of ownership and uniqueness. Bye, bearophile
Re: Parallel execution of unittests
On 4/30/14, 11:31 PM, Jonathan M Davis via Digitalmars-d wrote: On Wed, 30 Apr 2014 22:32:33 -0700 Andrei Alexandrescu via Digitalmars-d digitalmars-d@puremagic.com wrote: There are a number of assumptions here: (a) most unittests that can be effectively parallelized can be actually inferred (or declared) as pure; (b) most unittests that cannot be inferred as pure are likely to break; (c) it's a big deal if unittests break. I question all of these assumptions. In particular I consider unittests that depend on one another an effective antipattern that needs to be eradicated. Even if they don't depend on each other, they can depend on the system. Understood, no need to repeat, thanks. std.file's unit tests will break if we parallelize them, because it operates on files and directories, and many of those tests operate on the same temp directories. Yah, I remember even times when make unittest -j broke unittests because I've used the file name deleteme in multiple places. We need to fix those. That can be fixed by changing the tests, but it will break the tests. I'm not too worried about breaking tests. I have in mind that we'll display a banner at the beginning of unittesting explaining that tests are ran in parallel and to force serial execution they'd need to set this thing or that. In a way I don't see it as breakage in the traditional tests. Unittests are in a way supposed to break :o). Other tests _can't_ be fixed if we force them to run in parallel. For instance, some of std.datetime's unit tests set the local time zone of the system in order to test that LocalTime works correctly. Sure. We could specify that tests are to be run serially within one specific module, or to use classic interlocking in the unittest code. I see it as a problem relatively easy to address. We can guarantee that pure functions can safely be run in parallel. We _cannot_ guarantee that impure functions can safely be run in parallel. I'm sure that many impure unittest functions could be safely run in parallel, but it would require that the programmer verify that if we don't want undefined behavior - just like programmers have to verify that @system code is actually @safe. Simply running all unittest blocks in parallel is akin to considering @system code @safe in a particular piece of code simply because by convention that code should be @safe. I don't think undefined behavior is at stake here, and I find the simile invalid. Thread isolation is a done deal in D and we may as well take advantage of it. Worse that could happen is that a unittest sets a global and surprisingly the next one doesn't see it. At any rate I think it's pointless to insist on limiting parallel running to pure - let me just say I understood the point (thanks) so there is no need to restate it, and that I think it doesn't take us a good place. Andrei
Re: More radical ideas about gc and reference counting
Andrei Alexandrescu: We're considering deprecating ~this() for classes in the future. Such changes could happen in parallel to the (planned?) removal of some of the methods of Object. Bye, bearophile
Re: D For A Web Developer
On Wednesday, 30 April 2014 at 20:36:15 UTC, Andrei Alexandrescu wrote: On 4/30/14, 12:25 PM, Dicebot wrote: On Wednesday, 30 April 2014 at 19:08:15 UTC, Jacob Carlborg wrote: On 2014-04-30 11:43, Dicebot wrote: This is common complaint I still fail to understand. I have never ever wanted to run a single unit test, why would one need it? If running all module tests at once creates problems than either module is too big or unit tests are not really unit tests. Why would I run more tests than I have to? Because you hardly notice difference between 0.1 and 0.5 seconds *cough* std.datetime *cough* :o) Pretty much everyone agrees that std.datetime needs to be split into smaller module which was one of my original points. One good example is networking tests - if I worked on an airplane I'd love to not test tests that need connectivity with a simple regex. Again, networking (as well as any other I/O) has no place in unit tests. Never. Supporting such kind of tests natively means designing completely new system not changing existing one. For most simple example, you can't run non-unit tests in parallel without explicit annotations from programmer (your other thread).
Re: Parallel execution of unittests
On Wednesday, 30 April 2014 at 20:20:26 UTC, Jacob Carlborg wrote: On 2014-04-30 19:30, Dicebot wrote: I believe only missing step right now is propagation of UDA's to RTInfo when demanded. Everything else can be done as Phobos solution. I don't see why this is necessary for this case. It is not strictly necessary but you can't reliably get all unit test blocks during compile-time (must be transitively imported) and current run-time reflection for tests is missing any data but actual function pointers. I am personally perfectly satisfied with single root module imports all approach but it is likely to be complained if proposed as standard way.
Re: Parallel execution of unittests
On Wednesday, 30 April 2014 at 21:49:06 UTC, Jonathan M Davis via Digitalmars-d wrote: On Wed, 30 Apr 2014 21:09:14 +0100 Russel Winder via Digitalmars-d digitalmars-d@puremagic.com wrote: On Wed, 2014-04-30 at 11:19 -0700, Jonathan M Davis via Digitalmars-d wrote: unittest blocks just like any other unit test. I would very much consider std.file's tests to be unit tests. But even if you don't want to call them unit tests, because they access the file system, the reality of the matter is that tests like them are going to be run in unittest blocks, and we have to take that into account when we decide how we want unittest blocks to be run (e.g. whether they're parallelizable or not). In which case D is wrong to allow them in the unittest blocks and should introduce a new way of handling these tests. And even then all tests can and should be parallelized. If they cannot be then there is an inappropriate dependency. Why? Because Andrei suddenly proposed that we parallelize unittest blocks? If I want to test a function, I'm going to put a unittest block after it to test it. If that means accessing I/O, then it means accessing I/O. If that means messing with mutable, global variables, then that means messing with mutable, global variables. Why should I have to put the tests elsewhere or make is that they don't run whenthe -unttest flag is used just because they don't fall under your definition of unit test? You do this because unit tests must be fast. You do this because unit tests must be naively parallel. You do this because unit tests verify basic application / library sanity and expected to be quickly run after every build in deterministic way (contrary to full test suite which can take hours). Also you do that because doing _reliably_ correct tests with I/O is relatively complicated and one does not want to pollute actual source modules with all environment checks. In the end it is all about supporting quick edit-compile-test development cycle.
Re: Parallel execution of unittests
On Thursday, 1 May 2014 at 01:45:21 UTC, Xavier Bigand wrote: Splitting all features at an absolute atomic level can be achieve for open-source libraries, but it's pretty much impossible for an industrial software. Why being so restrictive when it's possible to support both vision by extending a little the language by something already logical? You are pretty much saying here writing good code is possible for open-source libraries but not for industrial software.
Re: Parallel execution of unittests
On Wed, 30 Apr 2014 23:56:53 -0700 Andrei Alexandrescu via Digitalmars-d digitalmars-d@puremagic.com wrote: I don't think undefined behavior is at stake here, and I find the simile invalid. Thread isolation is a done deal in D and we may as well take advantage of it. Worse that could happen is that a unittest sets a global and surprisingly the next one doesn't see it. At any rate I think it's pointless to insist on limiting parallel running to pure - let me just say I understood the point (thanks) so there is no need to restate it, and that I think it doesn't take us a good place. I'm only arguing for using pure on the grounds that it _guarantees_ that the unittest block is safely parallelizable. If we decide that that guarantee isn't necessary, then we decide that it isn't necessary, though I definitely worry that not having that guarantee will be problematic. I do agree though that D's thread-local by default helps quite a bit in ensuring that most tests will be runnable in parallel. However, if we went with purity to indicate parallelizability, I could easily see doing it implicitly based on purity and allowing for a UDA or somesuch which marked a unittest block as trusted pure so that it could be run in parallel. So, I don't think that going with pure would necessarily be too restrictive. It just would require that the programmer do some extra work to be able to treat a unittest block as safely parallelizable when the compiler couldn't guarantee that it was. Ultimately, my biggest concern here is that it be possible to guarantee that a unittest block is not run in parallel with any other unittest block if that particular unittest requires it for any reason, and some folks seem to be arguing that such tests are always invalid, and I want to make sure that we don't ever consider that to be the case for unittest blocks in D. If we do parallel by default and allow for some kind of markup to make a unittest block serial, then that can work. I fully expect that switching to parallel by default would break a number of tests, which I do think is a problem (particularly since a number of those tests will be completely valid), but it could also be an acceptable one - especially if for the most part, the code that it breaks is badly written code. Regardless, we will need to make sure that we message the change clearly in order to ensure that a minimal number of people end up with random test failures due to the change. On a side note, regardless of whether we want to use purity to infer paralellizability, I think that it's very cool that we have the capability to do so if we so choose, whereas most other languages have no way of even coming close to being able to tell whether a function can be safely parallelized or not. The combination of attributes such as pure and compile-time inference is very cool indeed. - Jonathan M Davis
Re: Parallel execution of unittests
On Wednesday, 30 April 2014 at 21:09:51 UTC, Átila Neves wrote: I don't know about anyone else, but I make my tests fail a lot. I think this is key difference. For me failing unit test is always exceptional situation. I TDD a lot. Tests failing are normal. Not only that, I refactor a lot as well. Which causes tests to fail. Fortunately I have tests failing to tell me I screwed up. I dream of a day when TDD crap will be finally discarded and fade into oblivion. Even if failing tests were exceptional, I still want everything I just mentioned. Probably. But will you still _need_ it? ;) And if test group is complex enough to require categorization then either my code is not procedural enough or module is just too big and needs to be split. And when I split them I put them into a subcategory. This is somewhat redundant as they are already categorized by module / package.
Re: More radical ideas about gc and reference counting
On 4/30/2014 11:57 PM, bearophile wrote: I agree the people are reluctant to add annotations, but I think it's also depends on the return of investment of the specific annotation. I am annotating all my code carefully with pure, but the ROI of all those purity annotations is not great. That's why I want to move more towards attribute inference. And I agree the piling of special cases is a not clean design. It's now simpler to ask the compiler if a certain assignment to immutable is allowed or not. Because I can't remember the supported cases. So I suggest a more principled and visible approach at the problem of ownership and uniqueness. The supported cases are not random special case hacks. They are based on a solid principle - if the expression represents a unique reference to an object, then it is implicitly convertible to immutable or shared. For example, new int; Formerly, that created a pointer to a mutable int object. It could not be implicitly cast to a pointer to an immutable int. But we know that operator new returns the only pointer to what it created, so it should be convertible. This knowledge was simply added to the compiler. I don't see anything unprincipled about that, or hard to understand, or hackish, or whatever.
Re: Parallel execution of unittests
On Thursday, 1 May 2014 at 04:50:30 UTC, Jonathan M Davis via Digitalmars-d wrote: std.file's unit tests would break immediately. It wouldn't surprise me if std.socket's unit tests broke. std.datetime's unit tests would probably break on Posix systems, because some of them temporarily set the local time zone - which sets it for the whole program, not just the current thread (those tests aren't done on Windows, because Windows only lets you set it for the whole OS, not just the program). Any tests which aren't pure risk breakage due to changes in whatever global, mutable state they're accessing. We really should think about separating Phobos tests into unit tests and higher level ones (in separate top-level source folder). The fact that importing std.file in my code with `rdmd -unittest` may trigger file I/O makes me _extremely_ disgusted. How did we even get here? _
Re: Parallel execution of unittests
Walter Bright:
On 4/30/2014 8:54 AM, bearophile wrote:
I'd also like some built-in way (or partially built-in) to use
a module only as
main module (to run its demos) or as module to be imported.
This problem is
solved in Python with the if __name__ == __main__: idiom.
dmd foo.d -unittest -main
I think you are greatly missing the point. The unittests are
mainly for the developer, while the demo is for the user of the
demo. The unittests validate the code, while the demo shows how
to use the module (and surely the demo is not meant to run in
parallel with the other unittests). Sometimes the demo part is
more than a demo, it contains usable and useful code, with a
command-line interface, to allow stand alone usage of the module
functionality. Currently this is how I do it:
module foobar;
// Here functions and
// classes with their unittests
version (foobar_main) {
void main() {
// foobar module demo code here.
}
}
I can't use just version(main), because the module could import
other modules with their own demo sections. So I need something
to tell them apart from each other.
This could be simplified with a solution similar to the Python
one:
version (__is_main) {
void main() {
// foobar module demo code here.
}
}
Now if I have module A and B, where B imports A, and both have
the demo main, I can use this to compile A stand-alone with its
demo:
dmd A.d
And I can use this to not compile the demo of A and compile the
demo section of B:
dmd A.d B.d
This is just the basic idea, and perhaps people suggested
something better than this.
Bye,
bearophile
Re: std.allocator: primitives for helping GC
01-May-2014 08:05, Andrei Alexandrescu пишет: So far allocators have been quite GC-agnostic; deallocation would be needed for each allocation, and there was little structure to help tracing. The primitive resolveInternalPointer has made a step toward more precise structuring of heaps, and now it's time to look at some real GC helpers. These are markAllAsUnused, markAsUsed, and doneMarking. Find their description here: http://erdani.com/d/phobos-prerelease/std_allocator.html There are a few proof of concept implementations here: https://github.com/andralex/phobos/blob/allocator/std/allocator.d The basic idea is that the GC would initiate a collection by calling markAllAsUnused. That would cause the underlying untyped allocator to softly mark all memory as free, but without actually messing it up. (Many allocators are capable of implementing such a primitive cheaply.) It would be interesting to define where GC sits and how it integrates with allocators to begin with. Given that currently GCAllocator is mentioned somewhere at the bottom of heap layers I'm really at loss about how these helpers fit into the picture. Then, the GC would trace objects starting from roots. Whenever it finds a used pointer, it would mark the corresponding memory as allocated by using markAsUsed, effectively undoing the soft freeing for it. How GC determines which allocator an object belongs to ? -- Dmitry Olshansky
Re: Parallel execution of unittests
On Thu, 01 May 2014 07:26:59 + Dicebot via Digitalmars-d digitalmars-d@puremagic.com wrote: On Thursday, 1 May 2014 at 04:50:30 UTC, Jonathan M Davis via Digitalmars-d wrote: std.file's unit tests would break immediately. It wouldn't surprise me if std.socket's unit tests broke. std.datetime's unit tests would probably break on Posix systems, because some of them temporarily set the local time zone - which sets it for the whole program, not just the current thread (those tests aren't done on Windows, because Windows only lets you set it for the whole OS, not just the program). Any tests which aren't pure risk breakage due to changes in whatever global, mutable state they're accessing. We really should think about separating Phobos tests into unit tests and higher level ones (in separate top-level source folder). The fact that importing std.file in my code with `rdmd -unittest` may trigger file I/O makes me _extremely_ disgusted. How did we even get here? _ Honestly, I see no problem with std.file's unit tests triggering I/O. That's what the module _does_. And it specifically uses the system's temp directory so that it doesn't screw with anything else on the system. Separating the tests out into some other set of tests wouldn't buy us anything IMHO. The tests need to be run regardless, and they need to be run with the same frequency regardless. Splitting those tests out would just make them harder for developers to run, because now they'd have to worry about running two sets of tests instead of just one. As far as I can see, splitting out tests that do I/O would be purely for ideological reasons and would be of no practical benefit. In fact, it would be _less_ practical if we were to do so. - Jonathan M Davis
Re: More radical ideas about gc and reference counting
On 05/01/2014 12:48 AM, deadalnix wrote: On Wednesday, 30 April 2014 at 22:24:29 UTC, Timon Gehr wrote: However, then, whether to do const(S!T) = S!(const(T)) or const(S!T) = S!(TailConst!T) should maybe be specified on a per-parameter basis, because this is in general not easy to figure out for the compiler. That is the whole problem :D Well, actually, just check whether there is a field that would need to change type, of the exact type of some parameter. If so, try to do the second thing for that parameter, otherwise try to do the first.
Re: More radical ideas about gc and reference counting
On Wed, 30 Apr 2014 13:21:33 -0700 Andrei Alexandrescu via Digitalmars-d digitalmars-d@puremagic.com wrote: Walter and I have had a long chat in which we figured our current offering of abstractions could be improved. Here are some thoughts. There's a lot of work ahead of us on that and I wanted to make sure we're getting full community buy-in and backup. First off, we're considering eliminating destructor calls from within the GC entirely. It makes for a faster and better GC, but the real reason here is that destructors are philosophically bankrupt in a GC environment. I think there's no need to argue that in this community. The GC never guarantees calling destructors even today, so this decision would be just a point in the definition space (albeit an extreme one). I really don't like the fact that struct destructors are not called by the GC, and if anything, I'd be inclined to argue for finding a way to guarantee that they get run rather than guaranteeing that they never get run. It's just far too easy to have a struct expect that its destructor will be run and then have issues when it's not run. But it would be better to define struct destructors as never getting run rather than having it be undefined as it is now. We're considering deprecating ~this() for classes in the future. While it's not good to rely on finalizers, they're good to have as backup if the appropriate cleanup function doesn't get called like it's supposed to. They're not as critical as they'd be in Java, since we have structs, but I'd be disinclined to remove finalizers from D without a really good reason. Also, we're considering a revamp of built-in slices, as follows. Slices of types without destructors stay as they are. Slices T[] of structs with destructors shall be silently lowered into RCSlice!T, defined inside object.d. That type would occupy THREE words, one of which being a pointer to a reference count. That type would redefine all slice primitives to update the reference count accordingly. RCSlice!T will not convert implicitly to void[]. Explicit cast(void[]) will be allowed, and will ignore the reference count (so if a void[] extracted from a T[] via a cast outlives all slices, dangling pointers will ensue). I foresee any number of theoretical and practical issues with this approach. Let's discuss some of them here. I'm really going to have to think about this one. It's such a radical change that I really don't know what to think about it. It will be interesting to see what others have to say about. - Jonathan M Davis
Re: Parallel execution of unittests
On Thursday, 1 May 2014 at 07:32:43 UTC, bearophile wrote: This is just the basic idea, and perhaps people suggested something better than this. Bye, bearophile Yeah I sometimes have commented out main() for that purpose. Sounds like a useful generic addition.
Re: static unittest
On Wednesday, 30 April 2014 at 22:14:44 UTC, Walter Bright wrote:
On 4/30/2014 2:34 PM, Meta wrote:
On Wednesday, 30 April 2014 at 21:04:19 UTC, Walter Bright
wrote:
On 4/30/2014 1:38 PM, Meta wrote:
Also, while we're thinking about static unittest, what about
contracts? I've
seen Bearophile suggest it quite a few times, and I agree
that it'd be very
useful to have contracts that are able to check a subset of
function
contracts/object invariants at compile time.
Already have them - template constraints.
Your function needs to be a template for that.
Adding () turns a function into a function template. Also,
static asserts.
Also, object invariants.
Easily handled with static asserts.
Just a thought, this might be a good idea for the problem of not
knowing which condition fails when a template instantiation
fails, i.e.:
auto reduce(alias fun, Args...)(Args args)
//We don't know which condition fails when this template fails to
instantiate
if (Args.length 0 Args.length = 2 isIterable!(Args[$ -
1]))
{
//...
}
Instead we could do:
auto reduce(alias fun, Args...)(Args args)
static in
{
assert(Args.length 0);
assert(Args.length = 2);
assert(isIterable!(Args[$ - 1]));
}
body
{
//...
}
And the error message which show you exactly which condition
failed. You could also just use static asserts in the contract,
but the topic of this thread is about `static unittest`, which is
more or less redundant as well.
Re: Parallel execution of unittests
On Thursday, 1 May 2014 at 07:47:27 UTC, Jonathan M Davis via Digitalmars-d wrote: Honestly, I see no problem with std.file's unit tests triggering I/O. That's what the module _does_. And it specifically uses the system's temp directory so that it doesn't screw with anything else on the system. Separating the tests out into some other set of tests wouldn't buy us anything IMHO. The tests need to be run regardless, and they need to be run with the same frequency regardless. Splitting those tests out would just make them harder for developers to run, because now they'd have to worry about running two sets of tests instead of just one. As far as I can see, splitting out tests that do I/O would be purely for ideological reasons and would be of no practical benefit. In fact, it would be _less_ practical if we were to do so. - Jonathan M Davis I have just recently went through some of out internal projects removing all accidental I/O tests for the very reason that /tmp was full and those tests were randomly failing when testing my own program that have used the library. This _sucks_. You can't do any test with I/O without verifying the environment (free space, concurrent access from other processes, file system access etc). And once you do it properly such beast has no longer fits into the same module because of sheer size. There is a very practical reason to separate tests - to become sure that you always can run -unittest build to verify basic sanity of your program and it will never spuriously fail or take eternity to complete.
Re: Parallel execution of unittests
Am Wed, 30 Apr 2014 13:31:30 -0700
schrieb Andrei Alexandrescu seewebsiteforem...@erdani.org:
I'd argue for regular identifiers instead of strings - they can be
seen in stack traces, accessed with __FUNCTION__ etc. -- Andrei
If we actually want to make unittests work just like functions
(__FUNCTION__, identifier which are visible in stacktraces) then we
could also simply declare functions and mark them as (@)unittest:
unittest void myUnittest()
{
}
This then allows for further improvements in the future, for example a
unit test can then return a result value (skipped, error, ..) or it
could optionally receive parameters (like some kind of state from a
unittest framework)
Re: More radical ideas about gc and reference counting
On Wed, 30 Apr 2014 14:00:31 -0700 Andrei Alexandrescu via Digitalmars-d digitalmars-d@puremagic.com wrote: On 4/30/14, 1:57 PM, Timon Gehr wrote: On 04/30/2014 10:45 PM, Andrei Alexandrescu wrote: An extreme one indeed, it would break a lot of my code. Every D project I wrote that does networking manages memory using a class that resides on the managed heap, but holds the actual wrapped data in the unmanaged heap. So should I take it those classes all have destructors? -- Andrei (Yes, those destructors free the unmanaged memory.) Thanks... that would need to change :o). -- Andrei And it doesn't even work now, because it's not guaranteed that finalizers get run. And IIRC, based on some of the discussions at dconf last year, dealing with the GC and unloading shared libraries would probably make the situation even worse. But not being able to rely on finalizers running does put us in a bit of a pickle, because it basically means that any case where you need a finalizer, you should probably be using reference counting rather than the GC. That would tend to mean that either classes are going to need to be wrapped in a struct that reference-counts them and/or they're going to need to be allocated with a custom allocator rather than the GC. This problem makes me think of C#'s using blocks where the object is created at the beginning of the using block, and it's dispose method is called when that block is exited (it may also be collected then, but I don't remember). I don't think that that's quite what we want, since there are plenty of cases where you want to pass a class around, so some kind of reference counting would be better, but we probably should consider having some kind of standard function similar to dispose so that there's a standard method to call when a class needs to be cleaned up. And that could tie into a struct in Phobos that we would have to do the reference counting by wrapping the class. - Jonathan M Davis
Re: Parallel execution of unittests
Am Wed, 30 Apr 2014 22:19:24 +0200 schrieb Jacob Carlborg d...@me.com: __traits(getUnitTests) I've always wondered, but never asked: Doesn't __traits(getUnitTests) usage suffer from the same problem std.benchmark had? That in order to make it work for all modules you a) have to explicitly mention every module b) use module constructors, leading to module constructor dependency hell (afaik the main reason we don't have a std.benchmark now) ?
Re: python vs d
On Thursday, 1 May 2014 at 06:04:57 UTC, Russel Winder via Digitalmars-d wrote: On Wed, 2014-04-30 at 15:21 -0400, Nick Sabalausky via Digitalmars-d wrote: […] I've heard this a lot, but I've yet to hear anyone explain concretely how this dynamic mindset causes the lack of things like static code checks and low-overhead primitives from being actual drawbacks. Maybe it really is a case of me not getting it, but it always sounds to me like this dynamic mindset gets around these issues simply by ignoring them. Since I don't personally do heavy development in dynamic languages, I'd be interested in a strong rebuttal to this. The best way of approaching this is not to try and attempt a long theoretical essay, but to try (a possibly lengthy exchange of question and answer accompanied by actually writing code in both Python and D. Also looking at things like Rosettacode and all Bearophile and others' work there. There are two questions above to kick start things, let's take the second first: Low-overhead primitives: This immediately implies you are looking for raw CPU-bound performance or you are undertaking premature optimization. Python has no direct play in this game. If a Python code requires out and out CPU-bound performance, you profile to find the small bit of the total code that is performance critical. If it is not already a function you extract it as a function, put it in it's own module and Cythonize it (*). Still Python code but with a few judicious extra annotations so as to enable Cython to generate C which is then compiled to native code. In my experience the problems with this approach are as follows: 1. The workflow: you write Python code, then profile it, extract the bottlenecks and then compile them to native code. A lot of overhead. 2. It can soon become messy when the code is changed, and there's a mess of different technologies applied over the years (Swig, Cython and whatnot). In my opinion it's cleaner to write a project in D (C/C++). Code native, in one go. I always feel uneasy, when I have to bring in third party software (i.e. a blackbox) to boost the performance of my code. I accept that, if you already have a substantial code base in Python, technologies like Cython are the way to go. But when I start a new project, why on earth should I go down that rocky path again? Static code checks: The object (aka data) model, and variable model of Python (and indeed other dynamic languages) means there cannot be any static code checks. At this point, I have to yield (in this co-routine exchange :-) with a question: what is it about static code checks that you feel you have to have in order to be able to program? (*) There are other techniques, but this is the lowest overhead in terms of programmer time, and it gives excellent runtime speed-up.
Re: More radical ideas about gc and reference counting
On Wednesday, 30 April 2014 at 20:21:33 UTC, Andrei Alexandrescu wrote: First off, we're considering eliminating destructor calls from within the GC entirely. It makes for a faster and better GC, but the real reason here is that destructors are philosophically bankrupt in a GC environment. I think there's no need to argue that in this community. The GC never guarantees calling destructors even today, so this decision would be just a point in the definition space (albeit an extreme one). That means classes that need cleanup (either directly or by having fields that are structs with destructors) would need to garner that by other means, such as reference counting or manual. We're considering deprecating ~this() for classes in the future. I'm all for it, this will break code that was not correct in the first place, and was working by accident. Also, being called by the GC from any thread prevent many kinds of cleanup, such as those with a thread-wise bound context (CUDA, OpenGL, etc...).
Re: Parallel execution of unittests
On Thursday, 1 May 2014 at 08:51:47 UTC, Johannes Pfau wrote: Am Wed, 30 Apr 2014 22:19:24 +0200 schrieb Jacob Carlborg d...@me.com: __traits(getUnitTests) I've always wondered, but never asked: Doesn't __traits(getUnitTests) usage suffer from the same problem std.benchmark had? That in order to make it work for all modules you a) have to explicitly mention every module b) use module constructors, leading to module constructor dependency hell (afaik the main reason we don't have a std.benchmark now) ? You only need to make sure all modules are transitively imported from initial one. Everything else can be done via recursive reflection with __traits(allMembers), module constructors are not needed either. One can also generate special test entry module that imports all project modules using build system help.
Re: Parallel execution of unittests
On Thursday, 1 May 2014 at 08:51:47 UTC, Johannes Pfau wrote: Am Wed, 30 Apr 2014 22:19:24 +0200 schrieb Jacob Carlborg d...@me.com: __traits(getUnitTests) I've always wondered, but never asked: Doesn't __traits(getUnitTests) usage suffer from the same problem std.benchmark had? That in order to make it work for all modules you a) have to explicitly mention every module b) use module constructors, leading to module constructor dependency hell (afaik the main reason we don't have a std.benchmark now) ? You only need to make sure all modules are transitively imported from initial one. Everything else can be done via recursive reflection with __traits(allMembers), module constructors are not needed either.
Re: A few considerations on garbage collection
On Wed, 30 Apr 2014 20:08:03 -0400 Steven Schveighoffer via Digitalmars-d digitalmars-d@puremagic.com wrote: On Wed, 30 Apr 2014 14:15:03 -0400, Dmitry Olshansky dmitry.o...@gmail.com wrote: IIRC they do, it's only arrays of such that doesn't. Anyhow having such a dangerous construct built-in (new = resource leak) in the language becomes questionable. No, they don't. Only objects are marked as having a finalizer. The finalize flag in the GC assumes that the first size_t in the block is a pointer to a vtable. A struct cannot have this. We need to fundamentally modify how this works if we want finalizers for structs to be called, but I think it's worth doing. IIRC, Rainer's precise GC does this. It would be _very_ cool if we could make it so that struct destructors get run when they're on the GC heap. I'm sure that the fact that they're not called currently creates a number of subtle bugs when structs are used which assume that they're destructor is going to be called when they're destroyed/freed. - Jonathan M Davis
Re: More radical ideas about gc and reference counting
On Wednesday, 30 April 2014 at 22:48:28 UTC, Andrei Alexandrescu wrote: On 4/30/14, 3:08 PM, John Colvin wrote: On Wednesday, 30 April 2014 at 21:51:17 UTC, Andrei Alexandrescu wrote: On 4/30/14, 2:47 PM, John Colvin wrote: On Wednesday, 30 April 2014 at 20:57:26 UTC, Andrei Alexandrescu wrote: Finally, immutable is sharable accross thread. That mean, even if we bypass the type system, that RC must be atomic for immutable. As they convert automatically for co,st, that mean that all const code will be full of atomic increment/decrement. They are bad for the CPU, and cannot be optimized away. Good point. I see that as a problem, albeit a solvable one. How? Having lock; instructions implicitly appearing in normal looking slice code is unacceptable. I'm thinking e.g. non-interlocked refcounts go like 1, 3, 5, ... and interlocked refcounts go like 2, 4, 6, ... Then you do an unprotected read of the refcount. If it's odd, then it's impossible to having originated as an interlocked one. So proceed with simple increment. If it's even, do an interlocked increment. Andrei I don't think I fully understand. Either all RC changes for a given type need to be atomic or none do, and that information is given by the type (everything that is immutable/const/shared). I don't see any feasible way of escaping this, or any advantage to a runtime convention like the odd/even trick above. An object starting as shared or immutable would always need to be atomically refcounted. That information is statically known. For those we'd initialize the refcount to 2. An object starting as regular mutable would always be refcounted non-atomically. That's also known statically. So that constructor initializes the refcount to 1. Then a const object would dynamically choose the approach to refcounting depending on the counter's evenness. Andrei Ah ok, that makes sense.
Re: D For A Web Developer
On 2014-04-30 22:11, Russel Winder via Digitalmars-d wrote: This cannot be a good idea. If the block says unittest then it contains unit tests, not integration tests or system tests, just unit tests. Then we need to come up with a separate framework for doing all other kinds of tests. -- /Jacob Carlborg
Re: DIP61: redone to do extern(C++,N) syntax
On Wed, 30 Apr 2014 20:56:15 +0100, Timon Gehr timon.g...@gmx.ch wrote:
If this is a problem, I guess the most obvious alternatives are to:
1. Get rid of namespace scopes. Require workarounds in the case of
conflicting definitions in different namespaces in the same file. (Eg.
use a mixin template.) I'd presume this would not happen often.
2. Give the global C++ namespace a distinctive name and put all other
C++ namespaces below it. This way fully qualified name lookup will be
reliable.
3. Use the C++ namespace for mangling, but not lookup. C++ symbols will
belong in the module they are imported into, and be treated exactly the
same as a D symbol, e.g.
module a;
extern(C++, std) ..string..
module b;
extern(C++, std) ..string..
module c;
import a;
import b;
void main() { .. string .. } // error could be a.string or b.string
void main() { .. a.string .. } // resolved
R
--
Using Opera's revolutionary email client: http://www.opera.com/mail/
Re: More radical ideas about gc and reference counting
On Wednesday, 30 April 2014 at 23:19:18 UTC, H. S. Teoh via Digitalmars-d wrote: On Wed, Apr 30, 2014 at 03:55:38PM -0700, Andrei Alexandrescu via Digitalmars-d wrote: On 4/30/14, 3:47 PM, H. S. Teoh via Digitalmars-d wrote: [...] I don't like the sound of that. I haven't found myself in a place where I needed to do something like this, but if I had to, I'd be very unhappy if struct dtors only work when they're not class members. Can we make them always work, and if necessary prohibit using them as class members? Then we're back to effectively class destructors. I think we've gathered quite a bit of evidence there are pernicious issues associated with them. -- Andrei [...] How so? If we prohibit structs with dtors from being class members, then it could work. Why would we prohibit structs with destructors from being class members? That don't make sense to me. A class is allowed to have a destructor = A class can have members with destructors. So we either kill off class destructor entirelly (which would mean no members with destructors) (But that seems like a bad idea), or we keep both. Or did I miss something in the argument?
Re: More radical ideas about gc and reference counting
On Thursday, 1 May 2014 at 01:04:08 UTC, Steven Schveighoffer wrote: That means classes that need cleanup (either directly or by having fields that are structs with destructors) would need to garner that by other means, such as reference counting or manual. We're considering deprecating ~this() for classes in the future. So essentially, any class with a dtor needs reference counting? How does one clean up a cycle of them? Yeah, what he said. This has me very worried. Making cycles is actually incredibly easy. Any inteligently implemented node-based data structure, implemented with classes, such as a linked list, a tree or a graph, is virtually guaranteed to have a cycle somewhere... *Or* is the idea that reference counting only works for finalization, but the memory is still managed by the GC? In that case, the destructors would leak, but the memory still be released? That's still bad, of course, but not as bad...
Re: DIP61: redone to do extern(C++,N) syntax
On Thu, 01 May 2014 11:03:21 +0100, Regan Heath re...@netmail.co.nz
wrote:
On Wed, 30 Apr 2014 20:56:15 +0100, Timon Gehr timon.g...@gmx.ch wrote:
If this is a problem, I guess the most obvious alternatives are to:
1. Get rid of namespace scopes. Require workarounds in the case of
conflicting definitions in different namespaces in the same file. (Eg.
use a mixin template.) I'd presume this would not happen often.
2. Give the global C++ namespace a distinctive name and put all other
C++ namespaces below it. This way fully qualified name lookup will be
reliable.
3. Use the C++ namespace for mangling, but not lookup. C++ symbols will
belong in the module they are imported into, and be treated exactly the
same as a D symbol, e.g.
module a;
extern(C++, std) ..string..
module b;
extern(C++, std) ..string..
module c;
import a;
import b;
void main() { .. string .. } // error could be a.string or b.string
void main() { .. a.string .. } // resolved
Sorry, #1 is the same suggestion :)
R
--
Using Opera's revolutionary email client: http://www.opera.com/mail/
Re: FYI - mo' work on std.allocator
Hi Andrey. Have you even test your allocator on different arch(32/64) and/or with different compiler flags ? On Windows, 32 bit. dmd allocator.d -main allocator.d(1492): Error: cannot implicitly convert expression (x / 64LU) of typ e ulong to immutable(uint) allocator.d(1494): Error: cannot implicitly convert expression (y / 64LU) of typ e ulong to immutable(uint) allocator.d(1518): Error: cannot implicitly convert expression (x / 64LU) of typ e ulong to uint allocator.d(1524): Error: cannot implicitly convert expression (i) of type ulong to uint allocator.d(1525): Error: cannot implicitly convert expression (i) of type ulong to uint allocator.d(1542): Error: cannot implicitly convert expression (w) of type ulong to uint allocator.d(1551): Error: cannot implicitly convert expression (w) of type ulong to uint allocator.d(1570): Error: cannot implicitly convert expression (w) of type ulong to uint allocator.d(1580): Error: cannot implicitly convert expression (w) of type ulong to uint allocator.d(1596): Error: cannot implicitly convert expression (x) of type ulong to uint allocator.d(1596): Error: cannot implicitly convert expression (y) of type ulong to uint If i fix these errors it's ok, but not ok with unittests. dmd allocator.d -main -debug -unittest allocator.d(4042): Error: function std.allocator.roundUpToMultipleOf (uint s, ui nt base) is not callable using argument types (ulong, uint) allocator.d(4082): Error: function std.allocator.roundUpToMultipleOf (uint s, ui nt base) is not callable using argument types (ulong, uint) allocator.d(2123): Error: template instance std.allocator.InSituRegion!(10240, 6 4) error instantiating allocator.d(2391): Error: cannot implicitly convert expression (4096LU * i) of t ype ulong to uint allocator.d(2391): Error: cannot implicitly convert expression (4096LU * j) of t ype ulong to uint allocator.d(6513): Error: template instance std.traits.hasMember!(HeapBlockWithI nternalPointers!4096, resolveInternalPointer) error instantiating allocator.d(2266):instantiated from here: testAllocator!(delegate () = (HeapBlockWithInternalPointers!4096 __ctmp2115 = 0; , __ctmp2115).this(m)) allocator.d(2439):instantiated from here: HeapBlockWithInternalPointers! 4096 allocator.d(4042): Error: function std.allocator.roundUpToMultipleOf (uint s, ui nt base) is not callable using argument types (ulong, uint) allocator.d(4082): Error: function std.allocator.roundUpToMultipleOf (uint s, ui nt base) is not callable using argument types (ulong, uint) allocator.d(2740): Error: template instance std.allocator.InSituRegion!16384 err or instantiating allocator.d(3838): Error: function std.allocator.roundUpToMultipleOf (uint s, ui nt base) is not callable using argument types (ulong, uint) allocator.d(3882): Error: function std.allocator.roundUpToMultipleOf (uint s, ui nt base) is not callable using argument types (ulong, uint) allocator.d(3993): Error: template instance std.allocator.Region!() error instan tiating allocator.d(4042): Error: function std.allocator.roundUpToMultipleOf (uint s, ui nt base) is not callable using argument types (ulong, uint) allocator.d(4082): Error: function std.allocator.roundUpToMultipleOf (uint s, ui nt base) is not callable using argument types (ulong, uint) allocator.d(4135): Error: template instance std.allocator.InSituRegion!(131072, 64) error instantiating allocator.d(4042): Error: function std.allocator.roundUpToMultipleOf (uint s, ui nt base) is not callable using argument types (ulong, uint) allocator.d(4082): Error: function std.allocator.roundUpToMultipleOf (uint s, ui nt base) is not callable using argument types (ulong, uint) allocator.d(4140): Error: template instance std.allocator.InSituRegion!131072 er ror instantiating allocator.d(4042): Error: function std.allocator.roundUpToMultipleOf (uint s, ui nt base) is not callable using argument types (ulong, uint) allocator.d(4082): Error: function std.allocator.roundUpToMultipleOf (uint s, ui nt base) is not callable using argument types (ulong, uint) allocator.d(4161): Error: template instance std.allocator.InSituRegion!4096 erro r instantiating
Re: D For A Web Developer
On 2014-04-30 22:38, Adam D. Ruppe wrote:
Yeah, foreign keys are really an absolute must. So are uniqueness
constraints. Rails can kinda do these, but in its own reinvented ways
that don't actually hit the DB (at least not without add on gems)
Rails unique constraints do hit the DB, but they can be painfully slow.
It's usually faster to let the DB handle it and handle the exception in
Ruby.
I also find myself really missing outer joins and views.
For outer joins:
1. You can always use raw SQL, also in combination with ActiveRecord
Post.joins(:comments).joins(outer join foos on foos.post_id =
posts.id).where(title: bar)
2. My preferred choice, using Squeel [1] (another gem). With Squeel the
above would look like this:
Post.joins(:comments).joins{ foos.outer }.where(title: bar)
It also supports queries like this:
Post.where{ |q| q.id = 3 }
For views you can use raw SQL in the migration to create the view. Then
it behaves just like a regular table. ActiveRecord won't know the
difference.
At my previous work we used the SchemaPlus [3] gem. It supports foreign
keys, views and indexes.
That doesn't change the race condition because the database is still
shared across those processes. With a regular SQL query, the database
guarantees consistent, atomic operations in the DB engine itself, but
with active record pattern, you give that up... while still having the
shared data.
I'm not sure I understand. ActiveRecord does regular SQL queries. The
example you wrote:
account = BankAccount.find(1)
account.balance -= 10
account.save!
Are you referring to if one process executes line 1 and another line 2?
It also hides all the beauty of it :(
Properly escape values
If you're escaping values, unless you're writing some low level database
library, you're almost certainly doing it wrong.
I don't know how other libraries do it these days but last time I didn't
use ActiveRecord it looked something like this:
query(select * from foo where name = ?, foobar);
BTW, this is still how you need to do it in ActiveRecord when needing
something more than the equal operator. It was quite similar in Rails 2
as well. Rails 3 and later makes it a bit better. Very ugly. All values
are far to the right and the query is to the left. It's get a quick
overview of which values go where.
Therefore I prefer Squeel as soon I need to do something more advanced
than the equal operator:
Foo.where{ |q| q.size = 4 }
You can also use SQL functions:
Foo.where{ |q| q.created_at == q.getdate() }
CSS is the view in that scenario. The HTML the object returns is just
its fields wrapped up in tags and classes. Works really well, for my
last big job, the designer worked almost exclusively in CSS and we got a
lot of stuff done.
I never get that working. The HTML always need to change when the design
changes.
It calls functions on the variable. So like {$foo|capitalize} would be
kinda like %= capitalize(foo) %. The plural example uses arguments to
the function too so we can customize the words if it is plural or no.
Ah, I see.
html% some ruby here %/html
I hate that stuff, especially when the view starts doing implicit
database queries! Defeats the whole point of MVC separation if you ask me.
That's useful for looping and if-statements. I also sometimes but a
variable there at the top. In Haml:
- @posts.each do |post|
%h3= post.title
.body= post.body
- if current_url == post_url(post)
= post.title
- else
= link_to post.title, post
This would of course be better handled with a partial and a view helper.
Yeah, I agree that it's ugly with database queries in the view. But it's
very easy to end up with something like this, instead of the above:
- Post.all.each do |post|
Queries in the view can be good, it depends on how you look at it. If
you write the query in the controller and then access the result in the
view, the query will be executed in the view because since Rails 3
queries are lazy. The advantage of this is the you can stream out the
response [2].
meh, with my libs, they might not be accessible to others, being poorly
documented and all, but I know them and have used them for almost
everything that's come up over the years. So for me, getting work done
means spending an hour using D instead of days searching the web for
some gem that won't even work right anyway.
For me it's the same with Rails. I know Rails and know a lot of useful gems.
[1] https://github.com/activerecord-hackery/squeel
[2] http://asciicasts.com/episodes/266-http-streaming
[3] https://github.com/lomba/schema_plus
--
/Jacob Carlborg
Re: D For A Web Developer
On 2014-04-30 22:36, Nick Sabalausky wrote: Aren't you talking about databases? Single-threading won't save you from races there unless the DBMS itself is single-threaded (which would be a pretty undesirable DBMS). Are you referring to if one process executes line 1 while another executes line 2? I don't see how ActiveRecord make this any worse. Or does the ORM above automatically lock the row/table behind-the-scenes? It does saves in transactions but I don't think it locks. -- /Jacob Carlborg
Re: D For A Web Developer
On Wednesday, 30 April 2014 at 19:25:40 UTC, Dicebot wrote: On Wednesday, 30 April 2014 at 19:08:15 UTC, Jacob Carlborg wrote: On 2014-04-30 11:43, Dicebot wrote: This is common complaint I still fail to understand. I have never ever wanted to run a single unit test, why would one need it? If running all module tests at once creates problems than either module is too big or unit tests are not really unit tests. Why would I run more tests than I have to? Because you hardly notice difference between 0.1 and 0.5 seconds The compilation time is often more of a problem than the runtime.
Re: Parallel execution of unittests
On 2014-04-30 23:25, Jonathan M Davis via Digitalmars-d wrote: Sure, that helps, but it's trivial to write a unittest block which depends on a previous unittest block There for the tests should be run in random order. -- /Jacob Carlborg
Re: D For A Web Developer
On Wednesday, 30 April 2014 at 15:04:53 UTC, Adam D. Ruppe wrote: On Wednesday, 30 April 2014 at 07:14:34 UTC, Jacob Carlborg wrote: I think one of the great things about Rails and Ruby is all the libraries and plugins that are available. If I want to do something, in RoR there's a big chance there's already a library for that. In D, there's a big chance I need to implement it myself. I like implementing things myself :P That's the question I dread most at meetings now: is there a gem for this? idk, in the time it takes to search for and evaluate third party code, I could have just written it myself. Especially since libraries almost always need some kind of customization for our specific case anyway! There's a few exceptions where something is hard to write, but most things just aren't that hard. I agree with this, but with a caveat: The valuable work in a 3rd party lib is more often the design than the body of the implementation. Designing a good API and general design for a library requires experience and perspective that I don't have in most problem spaces, but a quick bit of reading and the internals are often trivial to reproduce. I think this is particularly relevant in D; Implementation is made easy, but the flexibility available makes the design stage especially important. That is not to say that it's harder to make good designs with D, more that it's feasible to make *even better* designs if you have the expertise and time.
Re: Parallel execution of unittests
On 2014-04-30 23:35, Andrei Alexandrescu wrote: Agreed. I think we should look into parallelizing all unittests. -- Andrei I recommend running the tests in random order as well. -- /Jacob Carlborg
DIP(?) Warning to facilitate porting to other archs
Hi everyone. I think it's need to have -w64(or other name, offers ?) flag that warns if code may not compile on other archs. Example: size_t a; uint b = a; // ok on 32 without a warning but fail on 64 with error And on 32 with -w64 it'll be : Warning : size_t.sizeof may be greater than 32 bit What you thinks ? Should i create proposal or nobody cares about porting and it's useless ? Any ideas are welcome.
Re: Parallel execution of unittests
On 2014-04-30 21:12, bearophile wrote:
Are UDAs enough?
@uname(foo) unittest {}
What I'd like is to tie one or more unittests to other entities, like
all the unittests of a specific function.
Something similar is done in RSpec. A BDD framework in Ruby. In D it
might look like this:
@describe(foo)
{
@it(does something useful) unittest
{
}
@it(also does some other stuff) unittest
{
}
}
--
/Jacob Carlborg
Re: Parallel execution of unittests
On 2014-05-01 11:37, Dicebot wrote: You only need to make sure all modules are transitively imported from initial one. The solution for that would be RMInfo [1], like RTInfo but for modules instead of types. [1] https://github.com/D-Programming-Language/dmd/pull/2271 -- /Jacob Carlborg
Re: Parallel execution of unittests
On 2014-05-01 09:10, Dicebot wrote: It is not strictly necessary but you can't reliably get all unit test blocks during compile-time (must be transitively imported) and current run-time reflection for tests is missing any data but actual function pointers. I am personally perfectly satisfied with single root module imports all approach but it is likely to be complained if proposed as standard way. The current runner wouldn't work. It needs to be replace with one that uses __traits(getUnitTests). -- /Jacob Carlborg
Re: Parallel execution of unittests
Am Thu, 01 May 2014 09:38:51 +
schrieb Dicebot pub...@dicebot.lv:
On Thursday, 1 May 2014 at 08:51:47 UTC, Johannes Pfau wrote:
You only need to make sure all modules are transitively imported
from initial one. Everything else can be done via recursive
reflection with __traits(allMembers), module constructors are not
needed either.
One can also generate special test entry module that imports all
project modules using build system help.
Is there some boost licensed reflection code somewhere which shows how
to do recursive reflection correctly? I know the basic idea but stuff
like dealing with protection of members would take some time to
figure that out.
Maybe someone familiar with the details could sketch up a quick
std.test proof-of-concept which parses the following (with all
necessary protection checks + detecting unittest blocks in
classes/structs):
---
unittest
{
}
@_unittest void namedTest()
{
}
--
@Andrei do you think having to explicitly import modules to be tested
is an issue? Apart from this limitation such an approach sounds great
to me. There are quite some possibilities how user frameworks could
extend std.test so it sounds like an interesting idea. (And
std.benchmark could work in the same way)
Re: Parallel execution of unittests
Am Wed, 30 Apr 2014 09:02:54 -0700 schrieb Andrei Alexandrescu seewebsiteforem...@erdani.org: To summarize: It provides a function pointer for every unit test to druntime or user code. This is actually easy to do. Naming tests requires changes in the parser, but I guess that shouldn't be difficult either. That's fantastic, would you be willing to reconsider that work? Are you still interested in this? I guess we could build a std.test phobos module to completely replace the current unittest implementation, see: http://forum.dlang.org/post/ljtbch$lg6$1...@digitalmars.com This seems to be a better solution, especially considering extension possibilities.
Re: Parallel execution of unittests
On 2014-04-30 22:41, Andrei Alexandrescu wrote:
Yah I think that's possible but I'd like the name to be part of the
function name as well e.g. unittest__%s.
Why is that necessary? To have the correct symbol name when debugging?
I'm using something quite similar to RSpec from the Ruby world:
describe! toMsec in {
it! returns the time in milliseconds in {
assert(true);
}
}
This uses the old syntax, with UDA's it becomes something like this:
@describe(toMsec)
{
@it(returns the time in milliseconds) unittest
{
assert(true);
}
}
That looks... interesting.
The Ruby syntax looks like this:
describe toMsec do
it reruns the time in milliseconds do
assert true
end
context when the time parameter is nil do
it returns nil do
assert true
end
end
end
The interesting part about the Ruby implementation is that each it-block
(the code between do/end) is executed in the context of an anonymous
class instance. Each describe/context-block is turned in to a class,
nested blocks inherit from the outer block. In D the implementation
would look like this:
class __toMsec
{
void __someUniqueName123 ()
{
}
class __SomeUniqueName456 : __toMsec
{
void __someUniqueName789 ()
{
}
}
}
Each it-block (unit test) will be executed in a new instance of the
closest surrounding class. This means you can have helper methods and
instance variables shared across multiple test, but they will each get a
fresh copy of the data.
Since the describe-blocks are implemented with classes that inherit you
can override helper methods in subclasses.
The unit test runner can also print out a documentation, basically all
text in the it and describe parameters. Something like this:
https://coderwall-assets-0.s3.amazonaws.com/uploads/picture/file/1949/rspec_html_screen.png
--
/Jacob Carlborg
Re: Parallel execution of unittests
On Thursday, 1 May 2014 at 11:05:55 UTC, Jacob Carlborg wrote: On 2014-04-30 23:35, Andrei Alexandrescu wrote: Agreed. I think we should look into parallelizing all unittests. -- Andrei I recommend running the tests in random order as well. This is a bad idea. Tests could fail only some of the time. Even if bugs are missed, I would prefer it if tests did exactly the same thing every time.
Re: Parallel execution of unittests
Am Thu, 01 May 2014 13:24:07 +0200 schrieb Jacob Carlborg d...@me.com: On 2014-05-01 11:37, Dicebot wrote: You only need to make sure all modules are transitively imported from initial one. The solution for that would be RMInfo [1], like RTInfo but for modules instead of types. [1] https://github.com/D-Programming-Language/dmd/pull/2271 But on a quick look I don't understand how this (or DRT #775) completely solves the issue. Now you don't have to import the modules anymore, which is a step forward, but let's say I want to used this to find all SubClasses of a class. Now I can inspect that code in CTFE, but how do I build a list of all subclasses? In the most generic way (with DLLs/shared libraries) this can only work if code can be executed at runtime, so we'd need a way to emit module constructors AFAICS. These should be able to avoid the usual constructor dependency rules though.
Re: Parallel execution of unittests
Le 01/05/2014 09:23, Dicebot a écrit : On Thursday, 1 May 2014 at 01:45:21 UTC, Xavier Bigand wrote: Splitting all features at an absolute atomic level can be achieve for open-source libraries, but it's pretty much impossible for an industrial software. Why being so restrictive when it's possible to support both vision by extending a little the language by something already logical? You are pretty much saying here writing good code is possible for open-source libraries but not for industrial software. It's just a lot harder when you are under pressure. I am working for a very small company and our dead lines clearly doesn't help us with that, and because I am in the video game industry it's not really critical to have small bugs. Not every body have the capacity or resources (essentially time) to design his code in the pure conformance of unittests definition, and IMO isn't not an excuse to avoid tests completely. If a language/standard library can help democratization of tests it's a good thing, so maybe writing tests have to stay relatively simple and straightforward. My point is just when you are doing things only for you it's often simpler to them like they must be.
Re: More radical ideas about gc and reference counting
Removing class destructors would break my DQt library as it currently stands, which wraps C++ classes with D classes, Qt uses classes for polymorphism, for protected overrides and slots. I would have to consider some other means of wrapping C++. It might also be confusing for a few people. So, having said that... I am completely in favour of removing class destructors. It's a simple principle. When you have to do less work, things can run faster. So I can see removing class destructors as a feature leading to reduced pause times. Destructors exist only for resource management. Doing resource management through GC... is a bad idea.
Re: D For A Web Developer
On 5/1/14, 6:58 AM, Jacob Carlborg wrote: On 2014-04-30 22:11, Russel Winder via Digitalmars-d wrote: This cannot be a good idea. If the block says unittest then it contains unit tests, not integration tests or system tests, just unit tests. Then we need to come up with a separate framework for doing all other kinds of tests. Yes. And then you need two different commands to check if the system works. And if you want, for example, coverage analysis I wonder how you'd do that... That can't be good.
Re: Parallel execution of unittests
On Thu, 01 May 2014 11:44:11 +, w0rp wrote: On Thursday, 1 May 2014 at 11:05:55 UTC, Jacob Carlborg wrote: On 2014-04-30 23:35, Andrei Alexandrescu wrote: Agreed. I think we should look into parallelizing all unittests. -- Andrei I recommend running the tests in random order as well. This is a bad idea. Tests could fail only some of the time. Even if bugs are missed, I would prefer it if tests did exactly the same thing every time. Running tests in random order helps find hidden dependencies, but I wouldn't want it as a default. I lot of unittesting libraries offer this. If you don't run tests often it doesn't help much, but if you do TDD it can help.
Re: D For A Web Developer
On Wednesday, 30 April 2014 at 17:17:02 UTC, Ola Fosheim Grøstad wrote: On Wednesday, 30 April 2014 at 16:56:11 UTC, Nick Sabalausky wrote: Sounds pretty much exactly what I'd expect from just about any PHP-based application. :/ Modern PHP isn't so bad. I can write acceptable code in PHP. Though, I only do so when there is no other option, since it is the least desirable option next to Perl. The good thing about PHP is that default installs tend to have good C libraries. I think it would have died without that. So, if PHP is ok then it must be the PHP programmers that are to blame. I shudder to think what happens with a niche community if they pick it as the next fad… It could destroy any upcoming programming community with spaghetti-hell. Are you sure you want to market D as a web platform? This hasn't happened to Ruby (or Rails for that matter), which is definitely marketed as a web platform. Most of the Ruby code that I've seen is of exceptionally high quality. That's true not only regarding the code itself, but also for the interfaces it provides (in the case of libraries), which are usually well thought out. For PHP, on the other hand... I believe this has a lot to do with the language. Apparently it's harder to write bad code in some languages than in others. familiarity with the other stuff than I do. Ugh, but SQL can be such a pain, especially with all the vendor differences, and when compared to accomplishing something in whatever language I'm invoking SQL from. You can implement it in the ORB or wherever unit that provides transactions. I was more pointing to what I find useful conceptually in terms of layers: 1. user input on the client 2. validate on client 3. post using ajax 4. server unwraps the data and blindly inserts it into the database 5. if transaction fails, notify client, goto 1 6. done. IMO the client shouldn't do any validation, unless you can really, really trust it. That's why I like to do things the following way: 1. user input on the client 2. post using ajax 3. server validates and stores the data 4a. if transaction or data is invalid fails, send errors to the client 4b. if everything's ok, tell the client to redirect to the next page 5. on the client, add error CSS classes to the relevant fields, or execute the redirect I've created a gem that does almost all of that transparently. I just need to mark the form with remote: true, and adhere to a few simple naming conventions (which Rails' form generators do automatically).
Re: D For A Web Developer
On Thursday, 1 May 2014 at 10:44:42 UTC, Jacob Carlborg wrote:
On 2014-04-30 22:38, Adam D. Ruppe wrote:
I also find myself really missing outer joins and views.
For outer joins:
1. You can always use raw SQL, also in combination with
ActiveRecord
Post.joins(:comments).joins(outer join foos on foos.post_id =
posts.id).where(title: bar)
2. My preferred choice, using Squeel [1] (another gem). With
Squeel the above would look like this:
Post.joins(:comments).joins{ foos.outer }.where(title: bar)
You can also use the built-in `includes()`, which does a LEFT
OUTER JOIN:
Post.includes(:comments).where(comments: {title: bar})
(It also eager-loads the comments, but this is usually desired
anyway, because an OUTER JOIN doesn't make sense if you're not
going to use the joined records.)
Re: More radical ideas about gc and reference counting
On Wed, Apr 30, 2014 at 06:30:16PM -0700, Walter Bright via Digitalmars-d wrote:
On 4/30/2014 4:17 PM, H. S. Teoh via Digitalmars-d wrote:
If we're going
to have dtors at all, let's do it *right*. Guarantee they always work,
and reject all usages that break this guarantee (like putting a struct
with dtor inside a class,
Seems to work when I try it:
bar.d:
-
import core.stdc.stdio;
struct S { ~this() { printf(S.~this()\n); } }
class C { S s; }
void main()
{
C c = new C();
c = null;
}
-
C:\cbx\marsbar
S.~this()
The proposal was to get rid of class dtors, in which case this code will
no longer work. Is that really the direction we want to move in?
T
--
Let's call it an accidental feature. -- Larry Wall
Re: Parallel execution of unittests
On Thursday, 1 May 2014 at 11:44:12 UTC, w0rp wrote: On Thursday, 1 May 2014 at 11:05:55 UTC, Jacob Carlborg wrote: On 2014-04-30 23:35, Andrei Alexandrescu wrote: Agreed. I think we should look into parallelizing all unittests. -- Andrei I recommend running the tests in random order as well. This is a bad idea. Tests could fail only some of the time. Even if bugs are missed, I would prefer it if tests did exactly the same thing every time. They _should_ do exactly the same thing every time. Which is why running in threads or at random is a great way to enforce that. Atila
Re: More radical ideas about gc and reference counting
On Thu, May 01, 2014 at 10:06:17AM +, monarch_dodra via Digitalmars-d wrote: On Wednesday, 30 April 2014 at 23:19:18 UTC, H. S. Teoh via Digitalmars-d wrote: On Wed, Apr 30, 2014 at 03:55:38PM -0700, Andrei Alexandrescu via Digitalmars-d wrote: On 4/30/14, 3:47 PM, H. S. Teoh via Digitalmars-d wrote: [...] I don't like the sound of that. I haven't found myself in a place where I needed to do something like this, but if I had to, I'd be very unhappy if struct dtors only work when they're not class members. Can we make them always work, and if necessary prohibit using them as class members? Then we're back to effectively class destructors. I think we've gathered quite a bit of evidence there are pernicious issues associated with them. -- Andrei [...] How so? If we prohibit structs with dtors from being class members, then it could work. Why would we prohibit structs with destructors from being class members? That don't make sense to me. The proposal was to get rid of class dtors at some point. Which will introduce the problem of what to do when a class member is a struct with a dtor, since that dtor will never run. A class is allowed to have a destructor = A class can have members with destructors. So we either kill off class destructor entirelly (which would mean no members with destructors) (But that seems like a bad idea), or we keep both. Or did I miss something in the argument? Yes that's the gist of it. But Andrei seems to be convinced that class dtors are a bad idea, so to me, that also implies that class members with dtors are an equally bad (or worse) idea, so therefore they must be gotten rid of too. T -- All problems are easy in retrospect.
Re: Parallel execution of unittests
On Thu, 01 May 2014 09:26:39 -0400, Byron byron.he...@gmail.com wrote: On Thu, 01 May 2014 11:44:11 +, w0rp wrote: On Thursday, 1 May 2014 at 11:05:55 UTC, Jacob Carlborg wrote: On 2014-04-30 23:35, Andrei Alexandrescu wrote: Agreed. I think we should look into parallelizing all unittests. -- Andrei I recommend running the tests in random order as well. This is a bad idea. Tests could fail only some of the time. Even if bugs are missed, I would prefer it if tests did exactly the same thing every time. Running tests in random order helps find hidden dependencies, but I wouldn't want it as a default. I lot of unittesting libraries offer this. If you don't run tests often it doesn't help much, but if you do TDD it can help. Note the order of unit tests is defined by druntime. It can easily be modified. -Steve
Re: Parallel execution of unittests
I dream of a day when TDD crap will be finally discarded and fade into oblivion. I think most people who don't like TDD don't fully understand it. At the same time I think people who like TDD tend to abuse it. Either way, I like it, do it, and want my workflow to be the best it is with it. Even if failing tests were exceptional, I still want everything I just mentioned. Probably. But will you still _need_ it? ;) Yes. :P And if test group is complex enough to require categorization then either my code is not procedural enough or module is just too big and needs to be split. And when I split them I put them into a subcategory. This is somewhat redundant as they are already categorized by module / package. Which is why my library uses the fully qualified name of the test to categorise them. Atila
Re: More radical ideas about gc and reference counting
On Wed, Apr 30, 2014 at 04:33:23PM -0700, Andrei Alexandrescu via Digitalmars-d wrote: On 4/30/14, 4:17 PM, H. S. Teoh via Digitalmars-d wrote: [...] and they stop working as one might expect. Wasn't this whole fiasco the whole reason std.stdio.ByLine was recently rewritten to use ref counting? It relied on struct dtors before, and look how well that turned out. But ref counting IS destructors. Which will stop working once class dtors are deprecated, yet you still allow putting File in a class member. Basically, you can't allow something to have dtors, yet have no way to guarantee it will be called at the expected time. I don't think so - it's navigating close enough to the so let's not use cars anymore fallacy. There are plenty many situations in which dtors are working fabulously well, and throwing them away because we can't guarantee they'll work absolutely well seems bad decision making to me. No, allowing the user to specify a struct with a dtor, and put that struct in a class, and then saying nyah nyah class dtors don't exist anymore so your struct will never destruct -- *that* is bad decision making. At the very least, the compiler should complain loudly and clearly that structs with dtors should not be put inside a class. Isn't this what D philosophy is supposed to be? Things should *work* by default, and unexpected behaviour (dtor never gets called) should only happen when the user explicitly asks for it. That kind of under-specification is the source of endless hard-to-trace bugs, gotchas, unfixable issues, and holes in the type system. If we're going to have dtors at all, let's do it *right*. Guarantee they always work, and reject all usages that break this guarantee (like putting a struct with dtor inside a class, putting it inside a dynamic array in GC memory, etc.). If we can't guarantee anything at all, then let's not have dtors at all. Trying to stay in the gray zone in between does nothing but cause endless problems down the road. Not to mention language smells. What's the use of a feature that only sometimes works, where sometimes is un(der)specified, left to implementation, or depends on undefined behaviour? A lot of other languages have such imperfections, and nobody raises a brow. [...] I thought D is here because we want to do better than that? T -- GEEK = Gatherer of Extremely Enlightening Knowledge
Re: Parallel execution of unittests
On 2014-05-01 14:00, Johannes Pfau wrote: But on a quick look I don't understand how this (or DRT #775) completely solves the issue. Now you don't have to import the modules anymore, which is a step forward, but let's say I want to used this to find all SubClasses of a class. Now I can inspect that code in CTFE, but how do I build a list of all subclasses? In the most generic way (with DLLs/shared libraries) this can only work if code can be executed at runtime, so we'd need a way to emit module constructors AFAICS. These should be able to avoid the usual constructor dependency rules though. RMInfo only helps with all the modules the compiler sees during a given compile run. -- /Jacob Carlborg
Re: More radical ideas about gc and reference counting
On Thu, May 01, 2014 at 01:37:28AM -0700, Jonathan M Davis via Digitalmars-d wrote: On Wed, 30 Apr 2014 13:21:33 -0700 Andrei Alexandrescu via Digitalmars-d digitalmars-d@puremagic.com wrote: [...] First off, we're considering eliminating destructor calls from within the GC entirely. It makes for a faster and better GC, but the real reason here is that destructors are philosophically bankrupt in a GC environment. I think there's no need to argue that in this community. The GC never guarantees calling destructors even today, so this decision would be just a point in the definition space (albeit an extreme one). I really don't like the fact that struct destructors are not called by the GC, and if anything, I'd be inclined to argue for finding a way to guarantee that they get run rather than guaranteeing that they never get run. It's just far too easy to have a struct expect that its destructor will be run and then have issues when it's not run. But it would be better to define struct destructors as never getting run rather than having it be undefined as it is now. +1. We're considering deprecating ~this() for classes in the future. While it's not good to rely on finalizers, they're good to have as backup if the appropriate cleanup function doesn't get called like it's supposed to. They're not as critical as they'd be in Java, since we have structs, but I'd be disinclined to remove finalizers from D without a really good reason. [...] I'd like to hear an enumeration of those reasons as well. While in principle I agree with the sentiment to get rid of class dtors, I'm concerned about the rippling side-effects this will have throughout the rest of the language. Such as class members that are structs with dtors, which will mean that the struct dtors will never get called. Using another poster's argument: if dtors are used for resource management, then it's a bad idea to mix dtors with GC (i.e. classes). So in this sense I agree with getting rid of class dtors. But we have to consider what happens to the case where you stick a struct with a dtor into a class. I'm inclined to say that we should outright prohibit that, because again, the dtor (struct dtor this time) is used for resource management, and therefore shouldn't be mixed with a GC'd resource (i.e., classes). I find it unacceptable that the compiler will silently accept such a usage, and yet have no semantic guarantees (that dtor will run). This is the kind of behaviour I expect from C/C++, but not from D. Next thing you know, somebody is gonna start wrapping structs inside classes as a way of suppressing the dtor. I don't think that's the kind of thing the language should allow. I contend that it will do the language a great deal of good to outright ban such a kind of usage. (Structs with no dtors, OTOH, can be safely used as class members, obviously.) T -- It only takes one twig to burn down a forest.
Re: D For A Web Developer
On Thursday, 1 May 2014 at 12:17:56 UTC, Ary Borenszweig wrote: On 5/1/14, 6:58 AM, Jacob Carlborg wrote: On 2014-04-30 22:11, Russel Winder via Digitalmars-d wrote: This cannot be a good idea. If the block says unittest then it contains unit tests, not integration tests or system tests, just unit tests. Then we need to come up with a separate framework for doing all other kinds of tests. Yes. And then you need two different commands to check if the system works. And if you want, for example, coverage analysis I wonder how you'd do that... That can't be good. Two commands effectively. `rdmd -unittest` for development cycle of single module and `make test` to verify everything after feature/bugfix is completed. This is what I tend to do with existing tools (by splitting higher level tests in separate applications), adding some in-language support will just make intention a bit more clear.
Re: Parallel execution of unittests
On Thu, 01 May 2014 00:49:53 -0400, Jonathan M Davis via Digitalmars-d
digitalmars-d@puremagic.com wrote:
On Wed, 30 Apr 2014 20:33:06 -0400
Steven Schveighoffer via Digitalmars-d digitalmars-d@puremagic.com
wrote:
On Wed, 30 Apr 2014 13:50:10 -0400, Jonathan M Davis via
Digitalmars-d digitalmars-d@puremagic.com wrote:
On Wed, 30 Apr 2014 08:59:42 -0700
Andrei Alexandrescu via Digitalmars-d digitalmars-d@puremagic.com
wrote:
On 4/30/14, 8:54 AM, bearophile wrote:
Andrei Alexandrescu:
A coworker mentioned the idea that unittests could be run in
parallel
In D we have strong purity to make more safe to run code in
parallel:
pure unittest {}
This doesn't follow. All unittests should be executable
concurrently. -- Andrei
In general, I agree. In reality, there are times when having state
across unit tests makes sense - especially when there's expensive
setup required for the tests.
int a;
unittest
{
// set up a;
}
unittest
{
// use a;
}
==
unittest
{
int a;
{
// set up a;
}
{
// use a;
}
}
It makes no sense to do it the first way, you are not gaining
anything.
It can make sense to do it the first way when it's more like
LargeDocumentOrDatabase foo;
unittest
{
// set up foo;
}
unittest
{
// test something using foo
}
unittest
{
// do other tests using foo which then take advantage of changes made
// by the previous test rather than doing all of those changes to
// foo in order to set up this test
}
In general, I agree that tests shouldn't be done that way, and I
don't think that I've ever done it personally, but I've seen it done,
and for stuff that requires a fair bit of initialization, it can save
time to have each test build on the state of the last. But even if we
all agree that that sort of testing is a horrible idea, the language
supports it right now, and automatically parallelizing unit tests will
break any code that does that.
I recommend optimizing using a function.
i.e.:
version(unittest)
{
LargeDocumentOrDatabase foo;
auto getFoo() {/* check to see if foo is set up, return it*/}
}
I understand what you are saying. I think the largest problem with
parallelizing unit tests is that people haven't been careful to make sure
that's possible. Now they should, or face the consequences.
The point I was making, however, is that within a module, you can choose
whether you want parallel or serial unit tests. If you want parallel,
separate them into multiple unittest blocks. If you want serial, put them
in one.
For the super-rare cases where it needs to be serial, put them in one.
It's not hard.
Honestly, the idea of running unit tests in parallel makes me very
nervous. In general, across modules, I'd expect it to work, but
there will be occasional cases where it will break.
Then you didn't write your unit-tests correctly. True unit
tests-anyway.
In fact, the very quality that makes unit tests so valuable (that
they are independent of other code) is ruined by sharing state across
tests. If you are going to share state, it really is one unit test.
All it takes is that tests in two separate modules which have separate
functionality access the file system or sockets or some other system
resource, and they could end up breaking due to the fact that the other
test is messing with the same resource. I'd expect that to be a
relatively rare case, but it _can_ happen, so simply parallelizing
tests across modules does risk test failures that would not have
occurred otherwise.
Right, and with the knowledge that unit tests are being run in parallel,
one can trivially change their design to fix the problem.
I agree my assumptions were not what you were thinking of. I wasn't
thinking of shared system resources. But this isn't too difficult to
figure out.
I do think there should be a way to mark a unit test as don't parallelize
this.
Across the unittest
blocks in a single module, I'd be _very_ worried about breakage.
There is nothing whatsoever in the language which guarantees that
running them in parallel will work or even makes sense. All that
protects us is the convention that unit tests are usually
independent of each other, and in my experience, it's common enough
that they're not independent that I think that blindly enabling
parallelization of unit tests across a single module is definitely
a bad idea.
I think that if we add the assumption, the resulting fallout would be
easy to fix.
Note that we can't require unit tests to be pure -- non-pure
functions need testing too :)
Sure, they need testing. Just don't test them in parallel, because
they're not guaranteed to work in parallel. That guarantee _does_ hold
for pure functions, because they don't access global, mutable state.
So, we can safely parallelize a unittest block that is pure, but we
_can't_ safely paralellize one that isn't - not in a guaranteed way.
A function may be impure, but run in a
Re: More radical ideas about gc and reference counting
On 5/1/14, 3:06 AM, monarch_dodra wrote: A class is allowed to have a destructor = A class can have members with destructors. No equivalence, but implication. If a class has at least one member with a destructor, the compiler might need to generate a destructor for the class. -- Andrei
Re: Parallel execution of unittests
On 5/1/14, 1:34 AM, Dicebot wrote: I have just recently went through some of out internal projects removing all accidental I/O tests for the very reason that /tmp was full Well a bunch of stuff will not work on a full /tmp. Sorry, hard to elicit empathy with a full /tmp :o). -- Andrei
Re: D For A Web Developer
On 5/1/14, 3:50 AM, John Colvin wrote: On Wednesday, 30 April 2014 at 19:25:40 UTC, Dicebot wrote: On Wednesday, 30 April 2014 at 19:08:15 UTC, Jacob Carlborg wrote: On 2014-04-30 11:43, Dicebot wrote: This is common complaint I still fail to understand. I have never ever wanted to run a single unit test, why would one need it? If running all module tests at once creates problems than either module is too big or unit tests are not really unit tests. Why would I run more tests than I have to? Because you hardly notice difference between 0.1 and 0.5 seconds The compilation time is often more of a problem than the runtime. That's the case for phobos. -- Andrei
Re: std.allocator: primitives for helping GC
On Thursday, 1 May 2014 at 04:06:02 UTC, Andrei Alexandrescu wrote: So far allocators have been quite GC-agnostic; deallocation would be needed for each allocation, and there was little structure to help tracing. The primitive resolveInternalPointer has made a step toward more precise structuring of heaps, and now it's time to look at some real GC helpers. These are markAllAsUnused, markAsUsed, and doneMarking. Find their description here: http://erdani.com/d/phobos-prerelease/std_allocator.html There are a few proof of concept implementations here: https://github.com/andralex/phobos/blob/allocator/std/allocator.d The basic idea is that the GC would initiate a collection by calling markAllAsUnused. That would cause the underlying untyped allocator to softly mark all memory as free, but without actually messing it up. (Many allocators are capable of implementing such a primitive cheaply.) Then, the GC would trace objects starting from roots. Whenever it finds a used pointer, it would mark the corresponding memory as allocated by using markAsUsed, effectively undoing the soft freeing for it. At the end of the process, what's left is the memory effectively in use. Everything else got free by construction. It seems to me that these three primitives (together with resolveInternalPointer) are sufficient for conservative tracing. I plan to make tracing more precise by adding more structure, but the scanning logic will stay the same. Now is the time to destroy. I might be missing something by a mile. This marking will usually be implemented by setting a flag near the memory in question, which is not COW and cache friendly, because it tends to write all over the place. For this reason, some GCs use a temporary bitmap for marking, and leave the marked memory untouched. By pushing the marking to the specific allocators, this can no longer be implemented globally. But I guess a GC is not required to use these helper functions, so I guess it's fine... And just to be sure: doneMarking is _not_ supposed to actually free the memory (and call the destructors), is it? That's still the GC's job, right?
Re: More radical ideas about gc and reference counting
On 5/1/14, 3:10 AM, monarch_dodra wrote: On Thursday, 1 May 2014 at 01:04:08 UTC, Steven Schveighoffer wrote: That means classes that need cleanup (either directly or by having fields that are structs with destructors) would need to garner that by other means, such as reference counting or manual. We're considering deprecating ~this() for classes in the future. So essentially, any class with a dtor needs reference counting? How does one clean up a cycle of them? Yeah, what he said. This has me very worried. Making cycles is actually incredibly easy. Any inteligently implemented node-based data structure, implemented with classes, such as a linked list, a tree or a graph, is virtually guaranteed to have a cycle somewhere... In my experience classes with destructors are simpler (files, sockets)... Lists, trees, and graphs tend to be in-memory objects. But of course this is just speculation. *Or* is the idea that reference counting only works for finalization, but the memory is still managed by the GC? In that case, the destructors would leak, but the memory still be released? That's still bad, of course, but not as bad... Yah, that would be the case. Essentially we're talking about making the entire GC heap passive. Andrei
Re: Parallel execution of unittests
On Thursday, 1 May 2014 at 14:55:50 UTC, Andrei Alexandrescu wrote: On 5/1/14, 1:34 AM, Dicebot wrote: I have just recently went through some of out internal projects removing all accidental I/O tests for the very reason that /tmp was full Well a bunch of stuff will not work on a full /tmp. Sorry, hard to elicit empathy with a full /tmp :o). -- Andrei So you are OK with your unit tests failing randomly with no clear diagnostics? Including automated test servers? Really? Bunch of stuff can go wrong. Higher level tests verify their expectations from environment (if written carefully), which is impractical to do in unit tests. Also such reliance on environment makes running in parallel impossible without explicit resource dependency tracking for any kind of bigger test suite.
Re: Parallel execution of unittests
On 5/1/14, 4:05 AM, Jacob Carlborg wrote: On 2014-04-30 23:35, Andrei Alexandrescu wrote: Agreed. I think we should look into parallelizing all unittests. -- Andrei I recommend running the tests in random order as well. Great idea! -- Andrei
Re: FYI - mo' work on std.allocator
On 5/1/14, 3:36 AM, Temtaime wrote: Hi Andrey. Have you even test your allocator on different arch(32/64) and/or with different compiler flags ? Thanks, I'll look into that! -- Andrei
Re: Parallel execution of unittests
On 5/1/14, 4:31 AM, Johannes Pfau wrote: @Andrei do you think having to explicitly import modules to be tested is an issue? Well it kinda is. All that's written on the package is unittest, we should add no fine print to it. -- Andrei
Re: std.allocator: primitives for helping GC
On 4/30/14, Andrei Alexandrescu via Digitalmars-d digitalmars-d@puremagic.com wrote: So far allocators have been quite GC-agnostic; deallocation would be needed for each allocation, and there was little structure to help tracing. The primitive resolveInternalPointer has made a step toward more precise structuring of heaps, and now it's time to look at some real GC helpers. These are markAllAsUnused, markAsUsed, and doneMarking. Find their description here: http://erdani.com/d/phobos-prerelease/std_allocator.html There are a few proof of concept implementations here: https://github.com/andralex/phobos/blob/allocator/std/allocator.d The basic idea is that the GC would initiate a collection by calling markAllAsUnused. That would cause the underlying untyped allocator to softly mark all memory as free, but without actually messing it up. (Many allocators are capable of implementing such a primitive cheaply.) Then, the GC would trace objects starting from roots. Whenever it finds a used pointer, it would mark the corresponding memory as allocated by using markAsUsed, effectively undoing the soft freeing for it. At the end of the process, what's left is the memory effectively in use. Everything else got free by construction. It seems to me that these three primitives (together with resolveInternalPointer) are sufficient for conservative tracing. I plan to make tracing more precise by adding more structure, but the scanning logic will stay the same. Now is the time to destroy. I might be missing something by a mile. Andrei I actually do have a design in mind for integrating your allocators with my GC design, but there are really only 3 methods that a user-land allocator needs to implement. Below are a few snippets from my current working design describing the `markAsReferenced`, `sweep`, and `free` methods, as implemented by the GC allocators. A user-space allocator will need to call `GC.takeOwnership` passing in the pointer to the memory and it's length in order to tell the GC that it wants to handle the scanning and sweeping of the block of memory itself. A method by the name of `markAsReferenced` accepting a single `size_t` containing a pointer to an object, or a pointer to the interior of an object that, if valid, lies within the allocator’s page. This should mark the object, and any heap-allocated value referenced by the object by passing a pointer to the object to `GC.markAsReferenced`, so that a subsequent call to `sweep` doesn’t result in one of the values referenced by this object being freed by another allocator. This should treat the final possible pointer as a special case to allow the compiler to perform tail-call optimizations. A method by the name of `sweep` accepting no parameters, returning a `size_t` with the number of objects freed. This return value will only ever be used for statistics, so, while it is encouraged for an allocator to implement this, it is permissible to unconditionally return `0`, as the return value will never be essential to the functioning of the GC. This method is to be invoked only after every root has been recursively marked as referenced. This method must not move any marked heap-allocated objects unless the global setting asyncSweep is disabled. An allocator is free to do whatever it wants with the freed allocations, however all freed allocations need to be passed to `GC.finalize` which will return true if the value is actually free due to pending finalizers. It is encouraged for allocator implementations to take lock contention into account when implementing this method, as the Allocator Dispatch may request an allocation while the sweep is still being performed. It is recommended for an allocator to treat a completely un-marked heap as a special case, so that, if there are finalizers pending, time is not wasted attempting to add them to the finalizer list. If there are no remaining live allocations belonging to an allocator, that allocator should pass a pointer to itself to `GC.releaseAllocator`, so that it can be released if it is no longer needed. A method by the name of `free` accepting a single pointer to an allocation owned by this allocator. This method should assume that the finalizer has already been called if applicable, and is free to use the block of memory in whatever way it wants. It is not required for an allocator to immediately make the memory available to be re-allocated, as it is permitted for it to wait until the next time that `sweep` is called. While generally errors should not be thrown in the GC, if a pointer to an object is passed to an allocator’s free method that is not in fact owned by that allocator, the allocator should throw an Error, due to the fact there are likely even bigger problems. It will be possible to catch this error via a global handler.
Re: Parallel execution of unittests
On 5/1/14, 4:35 AM, Johannes Pfau wrote: Am Wed, 30 Apr 2014 09:02:54 -0700 schrieb Andrei Alexandrescu seewebsiteforem...@erdani.org: To summarize: It provides a function pointer for every unit test to druntime or user code. This is actually easy to do. Naming tests requires changes in the parser, but I guess that shouldn't be difficult either. That's fantastic, would you be willing to reconsider that work? Are you still interested in this? Yes. I guess we could build a std.test phobos module to completely replace the current unittest implementation, see: http://forum.dlang.org/post/ljtbch$lg6$1...@digitalmars.com This seems to be a better solution, especially considering extension possibilities. I think anything that needs more than the user writing unittests and adding a line somewhere would be suboptimal. Basically we need all we have now, just run in parallel. Andrei
Re: More radical ideas about gc and reference counting
On Thu, 01 May 2014 00:07:42 -0400, Andrei Alexandrescu
seewebsiteforem...@erdani.org wrote:
On 4/30/14, 7:20 PM, Steven Schveighoffer wrote:
On Wed, 30 Apr 2014 21:51:38 -0400, Andrei Alexandrescu
seewebsiteforem...@erdani.org wrote:
On 4/30/14, 6:04 PM, Steven Schveighoffer wrote:
destructors are for cleaning up non-GC resources. File handles,
malloc'd
memory, etc. I don't see why these need to be eliminated.
Virtually all GCs are known to be horrible at managing scarce
resources (including memory itself).
The destructor can be a crutch, but it's not good to leave open
resources when the user of your code has not cleaned them up manually.
Sounds like defensive programming to me.
Not at all. I've written programs that run for weeks at a time without
ever crashing, who continually manage such resources via the GC.
If the GC cleans up your object, but your object wasn't finalized, you
would have to make that a hard error. i.e. crash.
Still not talked about that I have seen, is the fact that with RC, we need
a solution for cycles. If that's the GC then the GC WILL be cleaning up
objects, even if they are ref counted.
I can see no reason to disallow destruction from cleaning up resources
that nobody else has managed to clean up.
Make memory management faster and better for everyone. I'm not saying
it's reason enough, but it's a reason.
It doesn't make it better. The GC still is cleaning up the memory. Having
it call close(fd) doesn't delay or make worse anything.
The largest problem with D destructors comes from trying to clean up D
objects in destructors of structs, that you never expected to be done
via the GC.
Cleaning up files and malloc'd memory is not an issue.
Not getting this...
class C
{
private int fd = -1;
private ubyte[] buffer;
this(string fname) {
fd = open(fname, O_CREAT);
buffer = (cast(ubyte*).malloc(100))[0..100];
}
~this() {
if(fd != -1) { close(fd); fd = -1;}
if(buffer.ptr) { .free(buffer.ptr); buffer = null}
}
}
There is no problem with this code. It works correctly in all cases. It's
not inefficient. It's not cumbersome or awkward or confusing. It's exactly
what class destructors are for.
However, this *is* a problem:
class C
{
RCValue s;
}
I have no way to say how s will be destructed, unless I poison it somehow.
Even still, s.~this() will be called when C.~this() is done. This can be
run in a separate thread as another object that is calling s.~this() at
the same time.
Removing ~this() for classes just ADDS complication that we don't need.
You are trying to fix the problem for the second situation by introducing
a problem for the first that is unnecessary.
Solutions for the second problem are not predicated on jettisoning class
dtors. You can, for instance, try and run the dtor for the instance of C
in the same thread where it is owned. You could, as you say, force C into
a ref counted mechanism. But this doesn't mean you should prevent C from
cleaning up its resources!
Please understand that I'm not arguing for the status quo. What I'm
arguing is that class destructors DO have some valid uses, and removing
them would introduce new problems. I agree we need to find a solution to
the ref counting + GC problem.
-Steve
Re: Parallel execution of unittests
On Thursday, 1 May 2014 at 12:04:57 UTC, Xavier Bigand wrote: It's just a lot harder when you are under pressure. I am working for a very small company and our dead lines clearly doesn't help us with that, and because I am in the video game industry it's not really critical to have small bugs. Not every body have the capacity or resources (essentially time) to design his code in the pure conformance of unittests definition, and IMO isn't not an excuse to avoid tests completely. If a language/standard library can help democratization of tests it's a good thing, so maybe writing tests have to stay relatively simple and straightforward. My point is just when you are doing things only for you it's often simpler to them like they must be. I know that and don't have luxury of time for perfect tests either :) But it is more about state of mind than actual time consumption - once you start keeping higher level tests with I/O separate and making observation how some piece of functionality can be tested in contained way, you approach to designing modules changes. At some point one simply starts to write unit test friendly modules from the very first go, it is all about actually thinking into it. Using less OOP and more functional programming helps with that btw :) I can readily admit that in real industry projects one is likely to do many different dirty things and this is inevitable. What I do object to is statement that this is the way to go in general, especially in language standard library.
Re: Parallel execution of unittests
On 5/1/14, 4:41 AM, Jacob Carlborg wrote: On 2014-04-30 22:41, Andrei Alexandrescu wrote: Yah I think that's possible but I'd like the name to be part of the function name as well e.g. unittest__%s. Why is that necessary? To have the correct symbol name when debugging? It's nice to have the name available in other tools (stack trace, debugger). The Ruby syntax looks like this: [snip] The unit test runner can also print out a documentation, basically all text in the it and describe parameters. Something like this: https://coderwall-assets-0.s3.amazonaws.com/uploads/picture/file/1949/rspec_html_screen.png That's all nice, but I feel we're going gung ho with overengineering already. If we give unittests names and then offer people a button parallelize unittests to push (don't even specify the number of threads! let the system figure it out depending on cores), that's a good step to a better world. Andrei
Re: D For A Web Developer
On Thursday, 1 May 2014 at 13:33:50 UTC, Marc Schütz wrote: IMO the client shouldn't do any validation, unless you can really, really trust it. That's why I like to do things the following way: 1. user input on the client 2. post using ajax 3. server validates and stores the data 4a. if transaction or data is invalid fails, send errors to the client 4b. if everything's ok, tell the client to redirect to the next page 5. on the client, add error CSS classes to the relevant fields, or execute the redirect That's a lot of unnecessary back and forth to the server for a JS-based design. Plus it avoids some of the nicer UX enhancements JS can enable, like validate-as-you-type, and does so without the benefit of not requiring JS. Kind of a worst of both worlds (no offence). Naturally, the server needs to do validation no matter what. But there's nothing wrong with doing an optional preliminary validation on the client side first.
Re: Parallel execution of unittests
On 5/1/14, 4:44 AM, w0rp wrote: On Thursday, 1 May 2014 at 11:05:55 UTC, Jacob Carlborg wrote: On 2014-04-30 23:35, Andrei Alexandrescu wrote: Agreed. I think we should look into parallelizing all unittests. -- Andrei I recommend running the tests in random order as well. This is a bad idea. Tests could fail only some of the time. Even if bugs are missed, I would prefer it if tests did exactly the same thing every time. I do random testing all the time, and I print the seed of the prng upon startup. When something fails randomly, I take the seed and seed the prng with it to reproduce. -- Andrei
Re: More radical ideas about gc and reference counting
On 5/1/14, 5:04 AM, w0rp wrote: Removing class destructors would break my DQt library as it currently stands, which wraps C++ classes with D classes, Qt uses classes for polymorphism, for protected overrides and slots. I would have to consider some other means of wrapping C++. It might also be confusing for a few people. So, having said that... I am completely in favour of removing class destructors. It's a simple principle. When you have to do less work, things can run faster. So I can see removing class destructors as a feature leading to reduced pause times. Destructors exist only for resource management. Doing resource management through GC... is a bad idea. I've decided what I'll do. I'll implement both and let the user choose. Push policy up, implementation down! -- Andrei
Re: More radical ideas about gc and reference counting
On 5/1/14, 7:17 AM, H. S. Teoh via Digitalmars-d wrote: On Wed, Apr 30, 2014 at 04:33:23PM -0700, Andrei Alexandrescu via Digitalmars-d wrote: No, allowing the user to specify a struct with a dtor, and put that struct in a class, and then saying nyah nyah class dtors don't exist anymore so your struct will never destruct -- *that* is bad decision making. At the very least, the compiler should complain loudly and clearly that structs with dtors should not be put inside a class. Isn't this what D philosophy is supposed to be? Things should *work* by default, and unexpected behaviour (dtor never gets called) should only happen when the user explicitly asks for it. Problem is there's a lot of correct code that e.g. closes the File members properly etc. All that code would be disable together with the risky code. That kind of under-specification is the source of endless hard-to-trace bugs, gotchas, unfixable issues, and holes in the type system. If we're going to have dtors at all, let's do it *right*. Guarantee they always work, and reject all usages that break this guarantee (like putting a struct with dtor inside a class, putting it inside a dynamic array in GC memory, etc.). If we can't guarantee anything at all, then let's not have dtors at all. Trying to stay in the gray zone in between does nothing but cause endless problems down the road. Not to mention language smells. What's the use of a feature that only sometimes works, where sometimes is un(der)specified, left to implementation, or depends on undefined behaviour? A lot of other languages have such imperfections, and nobody raises a brow. [...] I thought D is here because we want to do better than that? Better != perfection at all costs. Andrei
Re: Parallel execution of unittests
On Thu, 01 May 2014 11:04:31 -0400, Andrei Alexandrescu seewebsiteforem...@erdani.org wrote: On 5/1/14, 4:05 AM, Jacob Carlborg wrote: On 2014-04-30 23:35, Andrei Alexandrescu wrote: Agreed. I think we should look into parallelizing all unittests. -- Andrei I recommend running the tests in random order as well. Great idea! -- Andrei I think we can configure this at runtime. Imagine, you have multiple failing unit tests. You see the first failure. You find the issue, try and fix the problem, or instrument it, and now a DIFFERENT test fails. Now focus on that one, yet a different one fails. This is just going to equal frustration. If you want to run random, we can do that. If you want to run in order, that also should be possible. In fact, while debugging, you need to run them in order, and serially. -Steve
Re: More radical ideas about gc and reference counting
On 5/1/14, 7:30 AM, H. S. Teoh via Digitalmars-d wrote: On Thu, May 01, 2014 at 01:37:28AM -0700, Jonathan M Davis via Digitalmars-d wrote: While it's not good to rely on finalizers, they're good to have as backup if the appropriate cleanup function doesn't get called like it's supposed to. They're not as critical as they'd be in Java, since we have structs, but I'd be disinclined to remove finalizers from D without a really good reason. [...] I'd like to hear an enumeration of those reasons as well. 1. Most scarce resources must be released eagerly. GC collections occur relatively rarely and are triggered by different signals (low on memory). 2. The notion of running the GC when one runs out of file handles or sockets is terribly inefficient. 3. Destructors are odd - they will run in a different thread than the one that created the object. They also are limited in surprising ways, i.e. may not allocate memory or block on other threads directly or indirectly. 4. Due to imprecision, there's no actual guarantee any given destructor will end up running. Leaving a scarce resources at the whim of a best-effort approach is poor design. While in principle I agree with the sentiment to get rid of class dtors, I'm concerned about the rippling side-effects this will have throughout the rest of the language. Such as class members that are structs with dtors, which will mean that the struct dtors will never get called. Using another poster's argument: if dtors are used for resource management, then it's a bad idea to mix dtors with GC (i.e. classes). So in this sense I agree with getting rid of class dtors. But we have to consider what happens to the case where you stick a struct with a dtor into a class. I'm inclined to say that we should outright prohibit that, That can't happen. Andrei
Re: std.allocator: primitives for helping GC
On 5/1/14, 8:00 AM, Marc Schütz schue...@gmx.net wrote: This marking will usually be implemented by setting a flag near the memory in question, which is not COW and cache friendly, because it tends to write all over the place. Depends. HeapBlockWithInternalPointers uses the same flag as the allocation flag (nice!), and all flags are together at the front of the managed chunk. One perk is that once tracing is done, there's nothing else to do - free memory stays free. For this reason, some GCs use a temporary bitmap for marking, and leave the marked memory untouched. By pushing the marking to the specific allocators, this can no longer be implemented globally. Yah, probably I'll implement such a tracer too. For more generality (to work on discontiguous blocks of varied granularity) I'm thinking of having it use an interval tree instead of a bitmap. But I guess a GC is not required to use these helper functions, so I guess it's fine... And just to be sure: doneMarking is _not_ supposed to actually free the memory (and call the destructors), is it? That's still the GC's job, right? Yah, doneMarking would only do whatever low-level postprocessing the untype GC would need to do. Calling dtors will be done right after that. Andrei
Re: Parallel execution of unittests
On 5/1/14, 8:04 AM, Dicebot wrote: On Thursday, 1 May 2014 at 14:55:50 UTC, Andrei Alexandrescu wrote: On 5/1/14, 1:34 AM, Dicebot wrote: I have just recently went through some of out internal projects removing all accidental I/O tests for the very reason that /tmp was full Well a bunch of stuff will not work on a full /tmp. Sorry, hard to elicit empathy with a full /tmp :o). -- Andrei So you are OK with your unit tests failing randomly with no clear diagnostics? I'm OK with my unit tests failing on a machine with a full /tmp. The machine needs fixing. -- Andrei
Re: Parallel execution of unittests
On Thursday, 1 May 2014 at 15:37:21 UTC, Andrei Alexandrescu wrote: On 5/1/14, 8:04 AM, Dicebot wrote: On Thursday, 1 May 2014 at 14:55:50 UTC, Andrei Alexandrescu wrote: On 5/1/14, 1:34 AM, Dicebot wrote: I have just recently went through some of out internal projects removing all accidental I/O tests for the very reason that /tmp was full Well a bunch of stuff will not work on a full /tmp. Sorry, hard to elicit empathy with a full /tmp :o). -- Andrei So you are OK with your unit tests failing randomly with no clear diagnostics? I'm OK with my unit tests failing on a machine with a full /tmp. The machine needs fixing. -- Andrei It got full because of tests (surprise!). Your actions?
Re: Parallel execution of unittests
On 5/1/14, 9:07 AM, Dicebot wrote: On Thursday, 1 May 2014 at 15:37:21 UTC, Andrei Alexandrescu wrote: On 5/1/14, 8:04 AM, Dicebot wrote: On Thursday, 1 May 2014 at 14:55:50 UTC, Andrei Alexandrescu wrote: On 5/1/14, 1:34 AM, Dicebot wrote: I have just recently went through some of out internal projects removing all accidental I/O tests for the very reason that /tmp was full Well a bunch of stuff will not work on a full /tmp. Sorry, hard to elicit empathy with a full /tmp :o). -- Andrei So you are OK with your unit tests failing randomly with no clear diagnostics? I'm OK with my unit tests failing on a machine with a full /tmp. The machine needs fixing. -- Andrei It got full because of tests (surprise!). Your actions? Fix the machine and reduce the output created by the unittests. It's a simple engineering problem. -- Andrei
Re: std.allocator: primitives for helping GC
On 5/1/14, 12:43 AM, Dmitry Olshansky wrote: 01-May-2014 08:05, Andrei Alexandrescu пишет: So far allocators have been quite GC-agnostic; deallocation would be needed for each allocation, and there was little structure to help tracing. The primitive resolveInternalPointer has made a step toward more precise structuring of heaps, and now it's time to look at some real GC helpers. These are markAllAsUnused, markAsUsed, and doneMarking. Find their description here: http://erdani.com/d/phobos-prerelease/std_allocator.html There are a few proof of concept implementations here: https://github.com/andralex/phobos/blob/allocator/std/allocator.d The basic idea is that the GC would initiate a collection by calling markAllAsUnused. That would cause the underlying untyped allocator to softly mark all memory as free, but without actually messing it up. (Many allocators are capable of implementing such a primitive cheaply.) It would be interesting to define where GC sits and how it integrates with allocators to begin with. Given that currently GCAllocator is mentioned somewhere at the bottom of heap layers I'm really at loss about how these helpers fit into the picture. Well currently GCAllocator is only included for completeness. The purpose of std.allocator/std.typed_allocator is to be a complete redesign, not an adaptation, of the current GC. Then, the GC would trace objects starting from roots. Whenever it finds a used pointer, it would mark the corresponding memory as allocated by using markAsUsed, effectively undoing the soft freeing for it. How GC determines which allocator an object belongs to ? The idea is, for each root (whether conservative or typed), the GC first calls resolveInternalPointer. That gives the memory chunk encompassing that pointer. The GC uses that chunk to retrieve metadata about it and then passes it to setAsUsed. Andrei
Re: Parallel execution of unittests
Le 01/05/2014 13:44, w0rp a écrit : On Thursday, 1 May 2014 at 11:05:55 UTC, Jacob Carlborg wrote: On 2014-04-30 23:35, Andrei Alexandrescu wrote: Agreed. I think we should look into parallelizing all unittests. -- Andrei I recommend running the tests in random order as well. This is a bad idea. Tests could fail only some of the time. Even if bugs are missed, I would prefer it if tests did exactly the same thing every time. I am in favor of randomized order, cause it can help to find real bugs.
Re: Parallel execution of unittests
Le 01/05/2014 16:01, Atila Neves a écrit : On Thursday, 1 May 2014 at 11:44:12 UTC, w0rp wrote: On Thursday, 1 May 2014 at 11:05:55 UTC, Jacob Carlborg wrote: On 2014-04-30 23:35, Andrei Alexandrescu wrote: Agreed. I think we should look into parallelizing all unittests. -- Andrei I recommend running the tests in random order as well. This is a bad idea. Tests could fail only some of the time. Even if bugs are missed, I would prefer it if tests did exactly the same thing every time. They _should_ do exactly the same thing every time. Which is why running in threads or at random is a great way to enforce that. Atila +1
