Re: Dub, Cargo, Go, Gradle, Maven
On Tue, 13 Feb 2018 10:21:49 +, Abdulhaq wrote: > On Tuesday, 13 February 2018 at 10:06:43 UTC, welkam wrote: >> ADG? Google doesnt find anything relevant > > Acyclic directed graph A.K.A "DAG" - directed acyclic graph. I am also interested in build systems, and am currently convinced that the problem space is complex enough to warrant build systems being separate from package management systems. Maybe a compromise position would be for a package management system to define an interface through which it can do things like: * Discover what the external dependencies are, * Provide those external dependencies, and * Invoke a full build. Then, any number of build systems (and deployment systems?) could be adapted to work with the package management system.
Re: Announcing bottom-up-build - a build system for C/C++/D
On Thu, 27 Jun 2013 22:49:41 +, Graham St Jack wrote: On Thu, 27 Jun 2013 07:44:07 +0200, Rob T wrote: This build system seems to be very well suited for building complex large projects in a sensible way. I successfully tested the example build on Debian linux. I will definitely explore this further using one of my own projects. One issue I immediately ran into, is when I run bub incorrectly it hangs after writing the bail message to console. ctrl-c does not kill it, and I have to run a process kill commandto terminate. Seems it gets stuck in doBailer() while(true) loop, but I only glanced at the source quickly before posting back here. --rt I recently made a big change to the inter-thread communication code, and there are a few problems there still. I will look into it. Fixed the problem, and also: * split bub.d up into several smaller files, * added a Makefile to bootstrap the building of bub, and * added a bub.cfg and Bubfile as a trivial example of how to use bub.
Re: Announcing bottom-up-build - a build system for C/C++/D
On Sat, 29 Jun 2013 00:59:15 +0200, John Colvin wrote: On Thursday, 27 June 2013 at 00:10:37 UTC, Graham St Jack wrote: Having side-by-side comparisons of D against bash scripts and C++ modules had the effect of turning almost all the other team members into D advocates. Any chance we could know what team this is? (Sorry if this is common knowledge) It is the development team at my previous workplace. I haven't asked them for permission, so I would rather not say who they are. David Bryant (the previous poster) was a member of that team though, and he will be happy to provide details.
Re: Announcing bottom-up-build - a build system for C/C++/D
On Fri, 28 Jun 2013 08:05:08 +0200, Rob T wrote: On Thursday, 27 June 2013 at 23:03:40 UTC, Graham St Jack wrote: This isn't a build tool for everyone, but it really does make a big difference on big projects. Well I'm noticing some interesting concepts, such as being able to associate an include or import file with the library to link in, so that it gets done automatically simply by using the include/import file, great idea assuming I understood that correctly. That is correct. A side effect of assigning headers to libraries is that bub can then enforce the visibility rules and give understandable error messages almost instantaneously when you transgress them. Another consequence is that the compiler only needs to be given a few -I directives because you aren't using them to control visibility. One thing you may want to consider, or maybe this is already possible, is add the ability to optionally specify something like *.d so that all .d files get included from a folder, rather than have to always individually specify them manually. A useful idea. I will add it to the list. Also more concise documentation with clearer examples would be invaluable. I'm currently unsure if I need to restructure my existing project folders to fit bub or if bub can be configured to fit my existing projects. Documentation is always tough to pitch at the right level. I will see what I can do. Whether you need to restructure your project depends on whether you have already thought about dependencies and arranged things accordingly. My suggestion is to extend the example a bit and get a feel for how it works, then think about what that means for your project.
Re: Announcing bottom-up-build - a build system for C/C++/D
On Fri, 28 Jun 2013 08:28:03 +0200, Marco Leise wrote: How does this build tool handle projects with multiple executables ? For example the util-linux package contains dozens of utilities or a project might have a CLI and a GUI version. Or there might be slight alterations like setting a version or debug flag: -debug=threading -version=demo A central theme in bottom-up-build is to produce any number of build artifacts: libraries, executables, tests, etc. Different versions are catered for with: bub-config --mode=whatever build-dir The mode is described in your configuration file, and results in a build directory set up specifically for that mode. If you want multiple modes, use multiple build directories. Take a look in example/bub.cfg.
Re: Announcing bottom-up-build - a build system for C/C++/D
On Thu, 27 Jun 2013 07:44:07 +0200, Rob T wrote: This build system seems to be very well suited for building complex large projects in a sensible way. I successfully tested the example build on Debian linux. I will definitely explore this further using one of my own projects. One issue I immediately ran into, is when I run bub incorrectly it hangs after writing the bail message to console. ctrl-c does not kill it, and I have to run a process kill commandto terminate. Seems it gets stuck in doBailer() while(true) loop, but I only glanced at the source quickly before posting back here. --rt I recently made a big change to the inter-thread communication code, and there are a few problems there still. I will look into it.
Announcing bottom-up-build - a build system for C/C++/D
Bottom-up-build (bub) is a build system written in D which supports building of large C/C++/D projects. It works fine on Linux, with a Windows port nearly completed. It should work on OS-X, but I haven't tested it there. Bub is hosted on https://github.com/GrahamStJack/bottom-up-build. Some of bub's features that are useful on large projects are: Built files are located outside the source directory, using a different build directory for (say) debug, release, profile, etc. Very simple configuration files, making the build infrastructure easy to maintain. Automatic deduction of which libraries to link with. Automatic execution and evaluation of tests. Enforcement of dependency control, with prevention of circularities between modules and directories. Generated files are not scanned for imports/includes until after they are up to date. This is a real enabler for code generation. Files in the build directory that should not be there are automatically deleted. It is surprising how often a left-over build artifact can make you think that something works, only to discover your mistake after a commit. This feature eliminates that problem. The dependency graph is accurate, maximising opportunity for multiple build jobs and thus speeding up builds significantly. An early precursor to bub was developed to use on a large C++ project that had complex dependencies and used a lot of code generation. Bub is a major rewrite designed to be more general-purpose. The positive effect of the bub precursor on the project was very significant. Examples of positive consequences are: Well-defined dependencies and elimination of circularities changed the design so that implementation and testing proceeded from the bottom up. Paying attention to dependencies eliminated many unnecessary ones, resulting in a substantial increase in the reusability of code. This was instrumental in changing the way subsequent projects were designed, so that they took advantage of the large (and growing) body of reusable code. The reusable code improved in design and quality with each project that used it. Tests were compiled, linked and executed very early in the build - typically immediately after the code under test. This meant that regressions were usually detected within a few seconds of initiating a build. This was transformative to work rate, and willingness to make sweeping changes. Doing a clean is hardly ever necessary. This is important because it dramatically reduces the total amount of time that builds take, which matters on a large project (especially C++). Having a build system that works with both C++ and D meant that it was easy to slip some D code into the project. Initially as scripts, then as utilities, and so on. Having side-by-side comparisons of D against bash scripts and C++ modules had the effect of turning almost all the other team members into D advocates.
Re: Formal Review of std.process
Not sure if this is the right place for this, but I have found a problem in my fairly old version of the new process module. The problem was in the POSIX spawnProcessImpl() just after the fork() call in the child process branch. The child would occasionally block forever on an attempt by the D code to lock a mutex. The problem went away when I moved the toStringx() and toArgz() calls to before the fork() call. I have no idea what is going on here, but it seems like fork() leaves the child in bad shape, so my 'fix' is very much a band-aid.
Re: New std.process?
On Sun, 03 Feb 2013 19:56:54 +, Dejan Lekic wrote: simendsjo wrote: On Saturday, 20 October 2012 at 18:17:31 UTC, Alex Rønne Petersen wrote: It's time for the periodic new std.process ping. ;) Seriously, though, what's the state of it? Can we get it into the review queue soon? It would be great to have it in 2.060. I just created a small script using std.process, and the *pain*.. Took a look at the new std.process which easily lets you spawn a process using custom stdin/out/err, get the result etc.etc.. Looks a lot better than what we currently have. Unfortunately, it needs some druntime changes, so I couldn't just plug it in without building dmd myself. So.. Ping? :) Inclusion in 2.062? +1 I join Alex on this. I can't wait for improved std.process, honestly... +1. I have been using a hacked copy of the new one for ages, and would love to see it become part of phobos.
Re: Something needs to happen with shared, and soon.
On Sun, 11 Nov 2012 22:19:08 +0100, martin wrote: On Sunday, 11 November 2012 at 20:08:25 UTC, Benjamin Thaut wrote: Fully agree. +1 +1. I find it so broken that I have to avoid using it in all but the most trivial situations.
Re: New std.process?
On Sun, 21 Oct 2012 21:36:32 +0400, Denis Shelomovskij wrote: 21.10.2012 1:52, David Nadlinger пишет: On Saturday, 20 October 2012 at 18:56:01 UTC, Alex Rønne Petersen wrote: Oh, and could somebody please post a link to the latest version of the new std.process draft? I will probably add cross-platform support for constraining execution time and resource (RAM, mostly) usage, but there is no point in reimplementing it if it's already there. David Probably original discussion with links: http://www.digitalmars.com/d/archives/digitalmars/D/ The_new_std.process_163694.html Links from that thread: * std.process overhaul: https://github.com/kyllingstad/phobos/commits/new-std-process * druntime changes: https://github.com/schveiguy/druntime/commits/new-std-process I am also hanging out for the new std.process. Any idea when the required druntime changes will go in, or if they have already?
Re: Message passing between threads: Java 4 times faster than D
I suggest using a template-generated type that can contain any of the messages to be sent over a channel. It is reasonably straightforward to generate all the boilerplate code necessary to make this happen. My prototype (attached) still needs work to remove linux dependencies and tighten it up, but it works ok. Another advantage of this approach (well, I see it as an advantage) is that you declare in a single location all the messages that can be sent over the channel, and of course the messages are type-safe. The file of interest is concurrency.d. On 10/02/12 02:14, Sean Kelly wrote: So a queue per message type? How would ordering be preserved? Also, how would this work for interprocess messaging? An array-based queue is an option however (though it would mean memmoves on receive), as are free-lists for nodes, etc. I guess the easiest thing there would be a lock-free shared slist for the node free-list, though I couldn't weigh the chance of cache misses from using old memory blocks vs. just expecting the allocator to be fast. On Feb 9, 2012, at 6:10 AM, Gor Gyolchanyangor.f.gyolchan...@gmail.com wrote: Generally, D's message passing is implemented in quite easy-to-use way, but far from being fast. I dislike the Variant structure, because it adds a huge overhead. I'd rather have a templated message passing system with type-safe message queue, so no Variant is necessary. In specific cases Messages can be polymorphic objects. This will be way faster, then Variant. On Thu, Feb 9, 2012 at 3:12 PM, Alex_Dovhalalex_dov...@yahoo.com wrote: Sorry, my mistake. It's strange to have different 'n', but you measure speed as 1000*n/time, so it's doesn't matter if n is 10 times bigger. -- Bye, Gor Gyolchanyan. -- Graham St Jack delve.tar.gz Description: GNU Zip compressed data
Re: Suggestion for std.process upgrade
On 13/12/11 09:13, Steven Schveighoffer wrote: On Mon, 12 Dec 2011 17:08:53 -0500, Bane branimir.milosavlje...@gmail.com wrote: I have been playing with std.process (D2) lately, and have 2 suggestions and more or less tested code if somebody other than my self have use for it. One is shell() function (not mentioned in the docs, curious), I see it can be made more efficient on Windows. It executes shell command and returns standard output as string. Current implementation do it by piping stdout to temporary file on disk and reading that file back. Using CreateProccess Windows API it can do same job 3 times faster and remove need for temporary files and disk writes. I think that is beneficial gain for some applications. Other is fork-exec implementation eg. starting a program using command line and detaching it from parent, so it continues to run after parent is dead. On Posix it is implemented using fork() and exec() calls, on Windows using CreateProcess. There is a completely revamped version of std.process. It is being held up right now because DMD on windows depends on DMC for it's C runtime, and DMC has issues supporting pipes. I have recently opened a pull request for Walter to merge, I'm going to ping him about it right after 2.057 is released. For more info, see the docs Lars posted here: http://kyllingen.net/code/ltk/doc/process.html Once the DMC issue is fixed, you should see this improvement getting much more attention. It's very low hanging fruit. -Steve I took a look at the link, and it looks very nice. I am currently trying to port an application over from Linux to Windows and the lack of wait(pid) is currently a blocker. -- Graham St Jack
Re: Immutable Message Passing
On 11/12/11 12:59, Andrew Gough wrote:
On Mon, 05 Dec 2011 08:21:45 +0100
Jacob Carlborgd...@me.com wrote:
On 2011-12-04 21:46, Andrew Wiley wrote:
On Sun, Dec 4, 2011 at 1:42 AM, Jonathan M
Davisjmdavisp...@gmx.com wrote:
On Sunday, December 04, 2011 01:24:02 Andrew Wiley wrote:
This should work, right? I'm not just going crazy or something?
import std.concurrency;
import std.stdio;
class Bob {
}
void main() {
auto tid = spawn(b);
tid.send(new immutable(Bob)());
}
void b() {
receive(
(immutable(Bob) b) {
writeln(got it);
}
);
}
I'm getting this from both GDC (trunk) and DMD (2.056 and trunk -
I can't seem to get 2.055 to run):
core.exception.AssertError@/usr/include/d/std/variant.d(286):
immutable(Bob)
Seems like some type conversion isn't working properly and
variant is flagging immutable types as not assignable?
Well, that particular assertion seems to be because of the
immutability. It seems that Variant can't handle it - the comment
with it being
// type is not assignable
But I'd have to go digging to see why on earth this is failing
(and that's assuming that it's not a compiler bug, since then it
would be completely out of my league). It _looks_ like it should
work.
- Jonathan M Davis
Actually, it looks like I am going crazy. I went back to dmd 2.052
(and every version in between), and this doesn't seem to have ever
worked.
However, not allowing message passing with immutable messages
eliminates one of the main use cases for immutable. I'll see if I
can find a workaround.
The actual issue is that once Variant finds a matching type, it
tries to assign `immutable(Bob) = immutable(Bob)` , which is
illegal. However, all it needs to do is assign a *reference* to an
immutable object. Unfortunately, we don't support that.
Incidentally, this makes a compelling argument for something like
immutable(Object) ref.
Have message passing ever worked for other than primitive types
(including strings) ?
It works for shared classes
Yes, but using shared classes brings its own problems that aren't easy
to solve without something like tail shared/const/immutable.
--
Graham St Jack
Re: Why D const is annoying
On 11/12/11 06:55, Jonathan M Davis wrote:
On Saturday, December 10, 2011 05:45:11 bearophile wrote:
Timon Gehr:
Just slice the const array to get a range. The specialization for ranges
does not have the bug.
import std.algorithm;
void main() {
const arr = [1, 2, 3];
reduce!a*b(arr[]); // It works.
}
Wasn't arr a range already?
Per isInputRange!(typeof(arr)), no. It has the right functions, but it can't
use them, because it's const. A const range is essentially useless, because
you can't iterate over it.
When a template is instantiated, it's instantiated on the exact types that
it's given. So, if you given a const or immutable array, it's going to
instantiate on that type, even though it _could_ theoretically instantiate
itself with a mutable array with const or immutable elements. And since, a
const or immutable array won't work as a range, the template instantiation
fails.
The range-based functions in std.array and std.string work with const and
immutable arrays simply because they are either coded specifically for arrays
or have specializations for them. You need a function which takes const(T)[]
rather than one which takes const T[]. std.array and std.string typically take
const(T)[] rather than const T[], whereas a more general range function is
taking R, which in the case above is determined to be const int[], which won't
work as a range.
It used to be that the slice of an array was the exact type of that array,
meaning Timon's solution wouldn't work without a cast, because the slice would
be just as const as the original. Fortunately, that has been changed, so now a
slice of a const or immutable array will have its elements be const or
immutable but not the slice itself. So, now const and immutable arrays are
like static arrays in that you need to slice them to use them with range-based
functions, but you can't use them directly with range-based functions.
A bigger problem is const or immutable ranges which are structs. Unless the
programmer who defined the range type managed to have an opSlice which returned
a version with const or immutable elements but where the range itself wasn't
const or immutable (i.e. a tail-const slice), then you can't even get slicing
to work. And even if templates were improved to the point that they would
instantiate const int[] as const(int)[] (which I expect is a change which will
never happen due to the difficulties in doing so), that wouldn't solve the
problem for ranges which aren't arrays.
Instead of an immutable struct trying to be a range, why not have it
return a mutable range over its immutable elements? This would be
analogous to slicing an immutable array.
Also, const and immutable ranges which aren't arrays which have no opSlice
will _never_ work, because there's no way to get a mutable slice of them to
operate on, so you're stuck with a const or immutable range.
Really what this means is that you need to slice const and immutable ranges
when you pass them to range-based functions, and then as long as they're
arrays or their opSlices have been defined properly, it'll work just fine.
If ranges had been designed more like slists (i.e. they have head and tail
rather than front and popFront), then they would have been inherently tail-
const and we wouldn't be having these problems. But that's less efficent,
because you have to copy the range every time that you want to remove an
element from it. Regardless, it's too late now. Ranges are too heavily used
with their current API for us to change it that drastically now.
- Jonathan M Davis
--
Graham St Jack
Re: Immutable Message Passing
On 05/12/11 18:48, Jacob Carlborg wrote: On 2011-12-05 08:25, Graham St Jack wrote: I always use arrays or structs. Until the tail-const thing (or something like it) happens, classes don't seem to be viable in messages between threads. You can always serialize the object, if a copy is acceptable. Sure - I was counting that as an array of bytes, and I sometimes resort to that too. -- Graham St Jack
Re: Immutable Message Passing
My vote is for something like immutable(Object) ref, as Andrew suggested
earlier. This would allow mutable references to immutable objects to be
passed through a message channel without nasty typecasting.
std.typecons.Rebindable has always been an ugly hack that doesn't quite
do the job. Certainly every attempt I have made to use it has ended
unhappily, and I end up redesigning to not pass objects between threads.
What is the status of the immutable(Object) ref proposal? Is it on the
list of things to do, or is it ruled out? If it is ruled out, then what
is the superior proposal?
On 05/12/11 10:49, Timon Gehr wrote:
On 12/04/2011 11:32 PM, Andrew Wiley wrote:
On Sun, Dec 4, 2011 at 4:23 PM, Andrei Alexandrescu
seewebsiteforem...@erdani.org wrote:
On 12/4/11 4:16 PM, Andrew Wiley wrote:
So it looks like right now, message passing is copying objects, which
seems very bad. Check this out:
import std.stdio;
import std.concurrency;
class Bob {
}
void main() {
auto tid = spawn(func);
auto bob = new shared(Bob)();
writeln(bob is currently at , cast(void*)(bob));
This is the address of the reference, not that of the object. Use
cast(void*)(bob).
Andrei
Ah, I am covered with shame.
In that case, no object copying is occurring, and I have message
passing for immutable objects working, although my current solution is
basically to check whether the object is immutable, and if so, memcpy
the reference. That breaks immutability, but only for the reference,
and I don't think there's any alternative unless we get tail
const/immutable into the language.
I'm guessing this is too hackish to get merged into std.variant?
You might want to have a look at std.typecons.Rebindable.
--
Graham St Jack
Re: Immutable Message Passing
On 05/12/11 17:51, Jacob Carlborg wrote:
On 2011-12-04 21:46, Andrew Wiley wrote:
On Sun, Dec 4, 2011 at 1:42 AM, Jonathan M
Davisjmdavisp...@gmx.com wrote:
On Sunday, December 04, 2011 01:24:02 Andrew Wiley wrote:
This should work, right? I'm not just going crazy or something?
import std.concurrency;
import std.stdio;
class Bob {
}
void main() {
auto tid = spawn(b);
tid.send(new immutable(Bob)());
}
void b() {
receive(
(immutable(Bob) b) {
writeln(got it);
}
);
}
I'm getting this from both GDC (trunk) and DMD (2.056 and trunk - I
can't seem to get 2.055 to run):
core.exception.AssertError@/usr/include/d/std/variant.d(286):
immutable(Bob)
Seems like some type conversion isn't working properly and variant is
flagging immutable types as not assignable?
Well, that particular assertion seems to be because of the
immutability. It
seems that Variant can't handle it - the comment with it being
// type is not assignable
But I'd have to go digging to see why on earth this is failing (and
that's
assuming that it's not a compiler bug, since then it would be
completely out
of my league). It _looks_ like it should work.
- Jonathan M Davis
Actually, it looks like I am going crazy. I went back to dmd 2.052
(and every version in between), and this doesn't seem to have ever
worked.
However, not allowing message passing with immutable messages
eliminates one of the main use cases for immutable. I'll see if I can
find a workaround.
The actual issue is that once Variant finds a matching type, it tries
to assign `immutable(Bob) = immutable(Bob)` , which is illegal.
However, all it needs to do is assign a *reference* to an immutable
object. Unfortunately, we don't support that.
Incidentally, this makes a compelling argument for something like
immutable(Object) ref.
Have message passing ever worked for other than primitive types
(including strings) ?
I always use arrays or structs. Until the tail-const thing (or something
like it) happens, classes don't seem to be viable in messages between
threads.
--
Graham St Jack
Re: Added --makedepend flag to rdmd
Thanks very much for this - I need it for the build tool I am working on. On 06/06/11 01:18, Andrei Alexandrescu wrote: I just added a --makedepend flag to rdmd: https://github.com/D-Programming-Language/tools/commit/451ffed8ff985465a52124f7671494ac1d3744b4 It instructs rdmd to simply print to stdout the name of the input file followed by a colon and then by the space-separated files that the input file depends on, directly or indirectly. Example: // file test1.d; import test2.d; // file test2.d; import test3.d, mylib.test4.d; With this setup, assuming test3.d and test4.d contain no further non-system imports, the command: rdmd --makedepend test1.d will print test1.d : ./test2.d ./mylib/test4.d ./test3.d This flag is intended to be useful to larger-scale build tools that need to store and track module interdependencies. In the simplest use case, directing the output of rdmd --makedepend (for each of a project's root files) to a file and then including that file in a makefile will ensure that dependencies are properly maintained. Cheers, Andrei -- Graham St Jack
Re: Message Passing and Shared Data
On 11/04/11 15:18, Jonathan M Davis wrote: I'm working on an application that makes use of message passing to separate things that need to be responsive from tasks that may take some time, and I'm having trouble because my message passing discipline isn't meshing with D's type system. I'm pretty much following the contract that when I send a message from one thread to another, the original thread doesn't store any references to the message or what's in it, IE ownership is transferred. The receiving thread may then send some of the same objects back after some processing, but again, ownership is transferred. This has the benefit that I avoid a lot of memory allocations, but it means that right now, I'm constructing everything as shared, then casting it away to populate the object, then sending the object, then casting shared away again in the receiver, and so on. The messages are generally passed around for things like requesting a large chunk of data that takes a while to generate and receiving objects from an IO thread that's decoding objects from sockets. Any suggestions as to how I could avoid doing casts everywhere and/or restore some sort of safety while avoiding having to reallocate every object I mess with? I've tried fiddling with __gshared, but couldn't figure out any way to use it with message passing. There has been some talk of making it possible to do what you want to do with a unique template or attribute, but it hasn't gone anywhere that I know of. In particular, if it requires actual language changes, then it's unlikely to change in D2. We might get some sort of template that solves the problem though. I don't know. For the moment, however, message passing only works with immutable values. End of story. So, you have to do stuff like what you've been doing if you want to get around it. Essentially, you either deal just with immutable values, cast to and from immutable, or use shared. However, at the moment, there's no real way to transfer ownership of an object between threads, so message passing just doesn't do quite what you want to do. There may be a solution for it in the future though. - Jonathan M Davis Slight correction - it works with a type that passes this test: !hasLocalAliasing!(T) That means anything without references to non-immutable data. In the example in question, it amounts to the same thing. The difference is that you can send plain-old-data too. -- Graham St Jack
Re: a 'Shared libraries for Linux' question
On 29/03/11 15:35, Jonathan M Davis wrote: On 2011-03-28 21:38, Long Chang wrote: The shared lib support for Linux is very important for me . I ask people is there a schedule or plan once, but not get any responded . I hope the somebody can tell us some information about this . There's never really a schedule. Things get done when they get done. Shared libraries are near the top of the todo list, but they aren't being work on yet as far as I know. Walter has stated that he's currently focusing on fixing bugs with regards to destructors and temporaries, and then he intends to work on the issues with const (e.g. http://d.puremagic.com/issues/show_bug.cgi?id=1824 ). There's a good chance that shared libraries will be after that. But as to when exactly that'll be, I don't know. They _are_ among the higher priorities, but there's a lot of things that need to get done only and so much Walter to go around. - Jonathan M Davis That sounds fantastic. Those issues are the most troublesome for me, and have been for a long time. -- Graham St Jack
Re: Strategies for resolving cyclic dependencies in static ctors
I sounds like we actually agree with each other on all the important points - its just the different starting positions made our near-identical ideas about testing to look different. Thanks for the discussion. -- Graham St Jack
Re: Strategies for resolving cyclic dependencies in static ctors
On 25/03/11 06:09, Steven Schveighoffer wrote: On Thu, 24 Mar 2011 00:17:03 -0400, Graham St Jack graham.stj...@internode.on.net wrote: Regarding unit tests - I have never been a fan of putting unit test code into the modules being tested because: * Doing so introduces stacks of unnecessary imports, and bloats the module. As Jonathan says, version(unittest) works. No need to bloat unnecessarily. Agreed. However, all the circularity problems pop up when you compile with -unittest. * Executing the unittests happens during execution rather than during the build. Compile-time code execution is not a good idea for unit tests. It is always more secure and accurate to execute tests in the environment of the application, not the compiler. I didn't say during compilation - the build tool I use executes the test programs automatically. Besides, this is an implementation detail. It is easily mitigated. For example, phobos' unit tests can be run simply by doing: make -f posix.mak unittest and it builds + runs all unit tests. This can be viewed as part of the Build process. The problem I have with this is that executing the tests requires a special build and run which is optional. It is the optional part that is the key problem. In my last workplace, I set up a big test suite that was optional, and by the time we got around to running it, so many tests were broken that it was way too difficult to maintain. In my current workplace, the tests are executed as part of the build process, so you discover regressions ASAP. All unittests (as in the keyword) seem to have going for them is to be an aid to documentation. The huge benefit of D's unit tests are that the test is physically close to the code it's testing. This helps in debugging and managing updates. When you update the code to fix a bug, the unit test is right there to modify as well. I guess that was what I was alluding to as well. I certainly agree that having the tests that close is handy for users of a module. The extra point you make is that the unittest approach is also easier for the maintainer, which is fair enough. The whole point of unittests are, if they are not easy to do and conveniently located, people won't do them. You may have a really good system and good coding practices that allows you to implement tests the way you do. But I typically will forget to update tests when I'm updating code. It's much simpler if I can just add a new line right where I'm fixing the code. In practice I find that unit tests are often big and complex, and they deserve to be separate programs in their own right. The main exception to this is low-level libraries (like phobos?). What I do instead is put unit tests into separate modules, and use a custom build system that compiles, links AND executes the unit test modules (when out of date of course). The build fails if a test does not pass. The separation of the test from the code under test has plenty of advantages and no down-side that I can see - assuming you use a build system that understands the idea. Some of the advantages are: * No code-bloat or unnecessary imports. Not a real problem with version(unittest). * Much easier to manage inter-module dependencies. Not sure what you mean here. I mean that the tests typically have to import way more modules than the code under test, and separating them is a key step in eliminating circular imports. * The tests can be fairly elaborate, and can serve as well-documented examples of how to use the code under test. This is not an against for unit tests, they can be this way as well. Unit testing phobos takes probably a minute on my system, including building the files. They are as complex as they need to be. Conceded - it doesn't matter where the tests are, they can be as big as they need to be. As for the time tests take, an important advantage of my approach is that the test programs only execute if their test-passed file is out of date. This means that in a typical build, very few (often 0 or 1) tests have to be run, and doing so usually adds way less than a second to the build time. After every single build (even in release mode), you know for sure that all the tests pass, and it doesn't cost you any time or effort. * Since they only execute during the build, and even then only when out of date, they can afford to be more complete tests (ie use plenty of cpu time) IMO unit tests should not be run along with the full application. I'd suggest a simple unit test blank main function. I think even dmd (or rdmd?) will do this for you. There is no requirement to also run your application when running unit tests. That is my point exactly. I don't run tests as part of the application - the tests are separate utilities intended to be run automatically by the build tool. They can also be run manually to assist in debugging when something goes
Re: Strategies for resolving cyclic dependencies in static ctors
Regarding unit tests - I have never been a fan of putting unit test code into the modules being tested because: * Doing so introduces stacks of unnecessary imports, and bloats the module. * Executing the unittests happens during execution rather than during the build. All unittests (as in the keyword) seem to have going for them is to be an aid to documentation. What I do instead is put unit tests into separate modules, and use a custom build system that compiles, links AND executes the unit test modules (when out of date of course). The build fails if a test does not pass. The separation of the test from the code under test has plenty of advantages and no down-side that I can see - assuming you use a build system that understands the idea. Some of the advantages are: * No code-bloat or unnecessary imports. * Much easier to manage inter-module dependencies. * The tests can be fairly elaborate, and can serve as well-documented examples of how to use the code under test. * Since they only execute during the build, and even then only when out of date, they can afford to be more complete tests (ie use plenty of cpu time) * If the code builds, you know all the unit tests pass. No need for a special unittest build and manual running of assorted programs to see if the tests pass. * No need for special builds with -unittest turned on. -- Graham St Jack
Re: Strategies for resolving cyclic dependencies in static ctors
On 24/03/11 15:19, Jonathan M Davis wrote: Regarding unit tests - I have never been a fan of putting unit test code into the modules being tested because: * Doing so introduces stacks of unnecessary imports, and bloats the module. * Executing the unittests happens during execution rather than during the build. All unittests (as in the keyword) seem to have going for them is to be an aid to documentation. What I do instead is put unit tests into separate modules, and use a custom build system that compiles, links AND executes the unit test modules (when out of date of course). The build fails if a test does not pass. The separation of the test from the code under test has plenty of advantages and no down-side that I can see - assuming you use a build system that understands the idea. Some of the advantages are: * No code-bloat or unnecessary imports. * Much easier to manage inter-module dependencies. * The tests can be fairly elaborate, and can serve as well-documented examples of how to use the code under test. * Since they only execute during the build, and even then only when out of date, they can afford to be more complete tests (ie use plenty of cpu time) * If the code builds, you know all the unit tests pass. No need for a special unittest build and manual running of assorted programs to see if the tests pass. * No need for special builds with -unittest turned on. Obviously, it wouldn't resolve all of your concerns, but I would point out that you can use version(unittest) to enclose stuff that's only supposed to be in the unit tests build. And that includes using version(unittest) on imports, avoiding having to import stuff which is only needed for unit tests during normal builds. - Jonathan M Davis That is a good point, but as you say, it doesn't address all the concerns. I would be interested to hear some success stories for the unittest-keyword approach. So far I can't see any up-side. -- Graham St Jack
Re: Strategies for resolving cyclic dependencies in static ctors
On 23/03/11 03:41, Steven Schveighoffer wrote:
On Mon, 21 Mar 2011 20:12:55 -0400, Nick Sabalausky a@a.a wrote:
I'm intending this thread as somewhat of a roundtable-like discussion.
Hopefully we can come up with good material for a short article on
Wiki4D,
or maybe the D website, or wherever.
The scenario: A coder is writing some D, compiles, runs and gets a
Cyclic
dependency in static ctors error. Crap! A pain for experienced D
users, and
very bad PR for new D users. (Hopefully we'll eventually get some
sort of
solution for this, but in the meantime D users just have to deal with
it.)
The question: What now? What strategies do people find useful for
dealing
with this? Any specific first steps to take? Best practices? Etc.
What one can try is to factor out the initialization code into a
separate module. Essentially if you have:
module a;
import f : someFunction;
import b; // conflicts because of circular dependencies
int aglobal;
static this()
{
aglobal = someFunction();
}
You can do something like:
module a_static;
import f : someFunction;
int aglobal;
static this()
{
aglobal = someFunction();
}
in a.d:
module a;
public import a_static;
import b;
Of course, there can be reprecussions -- you may need to have aglobal
declared in a.d. In those cases, one can try to hide the cycle as Max
Samuckha stated, but I'd rather see a compiler option than these kinds
of workarounds. The workarounds can be unobvious, but can be just as
dangerous.
-Steve
My own solution to this problem is to never have circular imports at
all. The build system I use prohibits them, so any careless introduction
of a circularity is spotted immediately and I refactor the code to
eliminate the circularity. I have never come across a valid need for
circularities, and have never had any trouble eliminating any that creep in.
Avoiding circularities has plenty of advantages, like progressive
development, testing and integration. On bigger projects these
advantages are very important, and even on small ones they are useful.
--
Graham St Jack
Re: What To Do About Shared?
On 23/03/11 11:08, dsimcha wrote: Some discussions about std.parallelism have prompted an examination of how far D's guarantees against low level data races should extend and how safety and practicality should be balanced. On the one hand, coarse-grained multithreading with hard guarantees against low-level races is a great thing if it's flexible enough to do what you need it to. On the other hand, not everything is implementable (at least not efficiently or easily) in such a paradigm. D is a systems language and should not force people who want unchecked shared state multithreading to either do without it for fight the type system every inch of the way (by casting all over the place) to get it. I've come up with the following proposal, which is implicitly used in the design of std.parallelism, but which I think should be made explicit. 1. All @safe code must be statically checkable and provably free from low level data races provided that all @trusted code it calls is correctly implemented. It may not cast away shared, etc. 2. All @trusted code must guarantee to its clients that calling such code from @safe code will not result in low level data races. 3. All modules that deal with multithreading must document either that: a. They will use the type system to guarantee that low-level data races can't happen. b. They will share state freely. c. They will mostly share state freely, but will make guarantees about some specific subset. std.concurrency would be in category a. core.thread would be in category b. std.parallelism would be in category c. All code that only uses modules from category a, does not cast away shared and does not use __gshared variables can be guaranteed free from low level data races even if it is not @safe. If you want hard guarantees about low level data races, these can be achieved with a very small amount of discipline: Only use modules from category a or only use @safe code. This is easily checkable. Using modules from category b or modules from category c in non-@safe code should be considered equivalent to casting away shared: You may do so, but you're on your own when it comes to thread safety and you may not do it in @safe code. Sounds good in principal. I assume that category a code could be @trusted, and that category b and c must not be @trusted. I agree that trying to use the language to discriminate between category b and c would be too tricky, especially when you consider the subtleties of what is shared and what is not. Documentation is the only viable option there. Are these static checks feasible, and if so, what are the chances of getting them into the language anytime soon? -- Graham St Jack
Re: Strategies for resolving cyclic dependencies in static ctors
On 23/03/11 15:12, Nick Sabalausky wrote: Graham St Jackgraham.stj...@internode.on.net wrote in message news:imbai9$2jb9$1...@digitalmars.com... My own solution to this problem is to never have circular imports at all. The build system I use prohibits them, so any careless introduction of a circularity is spotted immediately and I refactor the code to eliminate the circularity. I have never come across a valid need for circularities, and have never had any trouble eliminating any that creep in. Avoiding circularities has plenty of advantages, like progressive development, testing and integration. On bigger projects these advantages are very important, and even on small ones they are useful. That's certainly good in many cases, but I find there are many times when a one-way dependency graph just doesn't fit the given problem and causes more trouble than it solves. You often end up needing to re-invent the wheel to avoid a dependency, or split/arrange/merge modules in confusing unintuitive ways that have more to do with implementation detail than high-level purpose. I'm happy to admit that these cases could come up, but I have never yet seen one where the design wasn't improved by removing the circularity. -- Graham St Jack
Re: Proposal for std.path replacement
On 04/03/11 02:59, Lars T. Kyllingstad wrote: As mentioned in the std.path.getName(): Screwy by design? thread, I started working on a rewrite of std.path a long time ago, but I got sidetracked by other things. The recent discussion got me working on it again, and it turned out there wasn't that much left to be done. So here it is, please comment: http://kyllingen.net/code/ltk/doc/path.html https://github.com/kyllingstad/ltk/blob/master/ltk/path.d Features: - Most functions work with all string types, i.e. all permutations of mutable/const/immutable(char/wchar/dchar)[]. Notable exceptions are toAbsolute() and toCanonical, because they rely on std.file.getcwd() which returns an immutable(char)[]. - Correct behaviour in corner cases that aren't covered by the current std.path. See the other thread for some examples, or take a look at the unittests for a more complete picture. - Saner naming scheme. (Still not set in stone, of course.) -Lars I like it. It certainly looks a lot cleaner than the current std.path. I am interested in why you chose to use templates to allow not only char, dchar and wchar arrays, but also const, mutable, and immutable. My first instinct would be to use non-templated functions that take const char[]. -- Graham St Jack
Re: Proposal for std.path replacement
On 04/03/11 12:34, Bekenn wrote: On 3/3/11 3:30 PM, Graham St Jack wrote: My first instinct would be to use non-templated functions that take const char[]. Please don't ever restrict encodings like that. As much as possible, libraries should seek to be encoding agnostic (though I'm all for const-qualifying parameters). This is one area where I feel the standard library severely lacks at present. As a Windows developer, I prefer to use wchar strings by default and use only the W versions of the Windows API functions, because the A versions severely limit functionality. Only the W versions have full support for Unicode; the A versions are entirely dependent on the current (8-bit) code page. This means no support for UNC paths or paths longer than 260 characters, and also means that international characters commonly end up completely garbled. Good practice in Windows is to consider the A versions deprecated and avoid them like the plague. Ok, I don't mind supporting wchar and dchar in addition to char, especially if Windows insists on using them. My main issue here is with the constness of the parameters. I think the correct parameter to pass is const C[]. This has the advantages of: * Accepting both mutable and immutable data. * Declares that the function won't mutate the data. * Declares that the function doesn't expect the data to be immutable. It would be even better to use const scope char[], declaring that a reference won't be kept, but it seems that scope in this context is deprecated. Once upon a time in meant const scope. Does anyone know what it means now? -- Graham St Jack
Re: dmd 1.067 and 2.052 release
On 21/02/11 16:14, Michel Fortin wrote: On 2011-02-20 20:21:20 -0500, Graham St Jack graham.stj...@internode.on.net said: In particular, are there any plans to re-examine the tail-const issue in light of the compiler patch proposed by Michel Fortin in his post: const(Object)ref is here! back in December? Note that there's now a pull request for that: https://github.com/D-Programming-Language/dmd/pull/3 And if someone wants to test it, just download and compile the const-object-ref branch of my dmd fork: https://github.com/michelf/dmd/tree/const-object-ref I'm currently waiting for feedback from Walter about this (and possibly others who dare to test it before it's in the mainline) before putting more work on it. Excellent - hopefully Walter will like it. I will have a go at compiling your branch and trying it out, but my use cases aren't all that stressful. -- Graham St Jack
Re: dmd 1.067 and 2.052 release
Fantastic news! Well done once again to the whole team. Now that the 64-bit bugbear is in the bag (along with a big pile of bugs), what is next? Is there a list somewhere detailing the planned language/toolchain changes that will make it into D2? Are we converging on a stable release of the language anytime soon? In particular, are there any plans to re-examine the tail-const issue in light of the compiler patch proposed by Michel Fortin in his post: const(Object)ref is here! back in December? Some other issues I assume are still in flux (and can remember) are: * Tweaks to usability of const/immutable/shared. * Tweaks to usability of nothrow, pure, etc. * Rollout of const, nothrow, pure, etc thoughout phobos. * Fate of the delete keyword. * Fate of the scope keyword used in object declaration and function parameters. * Meaning of in keyword for function parameters. Is it just const, and if so, why not just use const? I don't have (much) of a personal agenda here - I just want the rough edges smoothed off and a stable language. -- Graham St Jack
Re: How will we fix opEquals?
This isn't really true. If you make opEquals const, then anything opEquals calls must be const. Inevitably, you need to make a lot of your object const, which then goes viral to things that object uses, etc. On the other hand, I don't think opt-in const is that worthy a goal. Things that are const, should be const. -Steve I totally agree - we need to take the plunge and roll const out through phobos. If const (and immutable too for that matter) is broken in subtle ways, then it needs to be fixed and then embraced. -- Graham St Jack
Re: How will we fix opEquals?
Vote++ -- Graham St Jack
Re: findSkip signature
On 18/01/11 08:04, Andrei Alexandrescu wrote: On 1/17/11 3:21 PM, Ali Çehreli wrote: Andrei Alexandrescu wrote: I just added a useful function to Phobos: findSkip. Refer to http://d-programming-language.org/cutting-edge/phobos/std_algorithm.html#findSkip and http://www.dsource.org/projects/phobos/changeset/2339 I'm unsure about the signature. Currently the function returns Tuple!(R1, balance, bool, found), i.e. the balance of the range being searched and a Boolean telling whether or not the other range was found. The Boolean is necessary because returning an empty range is ambiguous - did you find the needle at the very end of the haystack, or not at all? I think a signature that's easier to use is: bool findSkip(alias pred = a == b, R1, R2)(ref R1 haystack, R2 needle); i.e. the haystack is passed by reference and modified if the find was successful. This is in keep with skipOver's signature, but not in keep with find's signature. Opinions? Andrei I am not sure where findSkip is useful but can we assume that the caller is not interested in whether it was found or not? If the callers are interested, they should have the option of calling find and skip separately. If they are always interested, then find and skip should not be merged together. findSkip is useful in parsing. One simple case in which it was useful was count() that counts how many times a forward range occurs in another. You can't do that with find(). (Try it!) I made the executive decision to pass by reference and return a bool. vote++ http://d-programming-language.org/cutting-edge/phobos/std_algorithm.html#findSkip Andrei -- Graham St Jack
Re: Portability bug in integral conversion
On 16/01/11 08:52, Andrei Alexandrescu wrote: We've spent a lot of time trying to improve the behavior of integral types in D. For the most part, we succeeded, but the success was partial. There was some hope with the polysemy notion, but it ultimately was abandoned because it was deemed too difficult to implement for its benefits, which were considered solving a minor annoyance. I was sorry to see it go, and I'm glad that now its day of reckoning has come. Some of the 32-64 portability bugs have come in the following form: char * p; uint a, b; ... p += a - b; On 32 bits, the code works even if a b: the difference will become a large unsigned number, which is then converted to a size_t (which is a no-op since size_t is uint) and added to p. The pointer itself is a 32-bit quantity. Due to two's complement properties, the addition has the same result regardless of the signedness of its operands. On 64-bits, the same code has different behavior. The difference a - b becomes a large unsigned number (say e.g. 4 billion), which is then converted to a 64-bit size_t. After conversion the sign is not extended - so we end up with the number 4 billion on 64-bit. That is added to a 64-bit pointer yielding an incorrect value. For the wraparound to work, the 32-bit uint should have been sign-extended to 64 bit. To fix this problem, one possibility is to mark statically every result of one of uint-uint, uint+int, uint-int as non-extensible, i.e. as impossible to implicitly extend to a 64-bit value. That would force the user to insert a cast appropriately. Thoughts? Ideas? Andrei It seems to me that the real problem here is that it isn't meaningful to perform (a-b) on unsigned integers when (ab). Attempting to clean up the resultant mess is really papering over the problem. How about a runtime error instead, much like dividing by 0? -- Graham St Jack
Re: Portability bug in integral conversion
On 17/01/11 10:39, Andrei Alexandrescu wrote:
On 1/16/11 5:24 PM, Graham St Jack wrote:
On 16/01/11 08:52, Andrei Alexandrescu wrote:
We've spent a lot of time trying to improve the behavior of integral
types in D. For the most part, we succeeded, but the success was
partial. There was some hope with the polysemy notion, but it
ultimately was abandoned because it was deemed too difficult to
implement for its benefits, which were considered solving a minor
annoyance. I was sorry to see it go, and I'm glad that now its day of
reckoning has come.
Some of the 32-64 portability bugs have come in the following form:
char * p;
uint a, b;
...
p += a - b;
On 32 bits, the code works even if a b: the difference will become a
large unsigned number, which is then converted to a size_t (which is a
no-op since size_t is uint) and added to p. The pointer itself is a
32-bit quantity. Due to two's complement properties, the addition has
the same result regardless of the signedness of its operands.
On 64-bits, the same code has different behavior. The difference a - b
becomes a large unsigned number (say e.g. 4 billion), which is then
converted to a 64-bit size_t. After conversion the sign is not
extended - so we end up with the number 4 billion on 64-bit. That is
added to a 64-bit pointer yielding an incorrect value. For the
wraparound to work, the 32-bit uint should have been sign-extended to
64 bit.
To fix this problem, one possibility is to mark statically every
result of one of uint-uint, uint+int, uint-int as non-extensible,
i.e. as impossible to implicitly extend to a 64-bit value. That would
force the user to insert a cast appropriately.
Thoughts? Ideas?
Andrei
It seems to me that the real problem here is that it isn't meaningful to
perform (a-b) on unsigned integers when (ab). Attempting to clean up
the resultant mess is really papering over the problem. How about a
runtime error instead, much like dividing by 0?
That's too inefficient.
Andrei
If that is the case, then a static check like you are suggesting seems
like a good way to go. Sure it will be annoying, but it will pick up a
lot of bugs.
This particular problem is one that bights me from time to time because
I tend to use uints wherever it isn't meaningful to have negative
values. It is great until I need to do a subtraction, when I sometimes
forget to check which is greater. Would the check you have in mind
statically check the following as ok?
where a and b are uints and ptr is a pointer:
if (a b) {
ptr += (a-b);
}
--
Graham St Jack
Re: Portability bug in integral conversion
On 17/01/11 13:30, Andrei Alexandrescu wrote:
On 1/16/11 7:51 PM, Graham St Jack wrote:
On 17/01/11 10:39, Andrei Alexandrescu wrote:
On 1/16/11 5:24 PM, Graham St Jack wrote:
On 16/01/11 08:52, Andrei Alexandrescu wrote:
We've spent a lot of time trying to improve the behavior of integral
types in D. For the most part, we succeeded, but the success was
partial. There was some hope with the polysemy notion, but it
ultimately was abandoned because it was deemed too difficult to
implement for its benefits, which were considered solving a minor
annoyance. I was sorry to see it go, and I'm glad that now its day of
reckoning has come.
Some of the 32-64 portability bugs have come in the following form:
char * p;
uint a, b;
...
p += a - b;
On 32 bits, the code works even if a b: the difference will
become a
large unsigned number, which is then converted to a size_t (which
is a
no-op since size_t is uint) and added to p. The pointer itself is a
32-bit quantity. Due to two's complement properties, the addition has
the same result regardless of the signedness of its operands.
On 64-bits, the same code has different behavior. The difference a
- b
becomes a large unsigned number (say e.g. 4 billion), which is then
converted to a 64-bit size_t. After conversion the sign is not
extended - so we end up with the number 4 billion on 64-bit. That is
added to a 64-bit pointer yielding an incorrect value. For the
wraparound to work, the 32-bit uint should have been sign-extended to
64 bit.
To fix this problem, one possibility is to mark statically every
result of one of uint-uint, uint+int, uint-int as non-extensible,
i.e. as impossible to implicitly extend to a 64-bit value. That would
force the user to insert a cast appropriately.
Thoughts? Ideas?
Andrei
It seems to me that the real problem here is that it isn't
meaningful to
perform (a-b) on unsigned integers when (ab). Attempting to clean up
the resultant mess is really papering over the problem. How about a
runtime error instead, much like dividing by 0?
That's too inefficient.
Andrei
If that is the case, then a static check like you are suggesting seems
like a good way to go. Sure it will be annoying, but it will pick up a
lot of bugs.
This particular problem is one that bights me from time to time because
I tend to use uints wherever it isn't meaningful to have negative
values. It is great until I need to do a subtraction, when I sometimes
forget to check which is greater. Would the check you have in mind
statically check the following as ok?
where a and b are uints and ptr is a pointer:
if (a b) {
ptr += (a-b);
}
That would require flow analysis. I'm not sure we want to embark on
that ship. In certain situations value range propagation could take
care of it.
Andrei
My fear is that if a cast is always required, people will just put one
in out of habit and we are no better off (just like exception-swallowing).
Is the cost of run-time checking really prohibitive? Correct code should
have some checking anyway. Maybe providing phobos functions to perform
various correct-usage operations with run-time checks like in my code
fragment above would by useful. They could do the cast, and most of the
annoyance factor would be dealt with. A trivial example:
int difference(uint a, uint b) {
if (a = b) {
return cast(int) a-b;
}
else {
return -(cast(int) b-a);
}
}
--
Graham St Jack
Re: Portability bug in integral conversion
On 17/01/11 14:16, Jonathan M Davis wrote: On Sunday 16 January 2011 19:38:55 Andrei Alexandrescu wrote: On 1/16/11 9:32 PM, Graham St Jack wrote: Is the cost of run-time checking really prohibitive? Yes. There is no question about that. This is not negotiable. Well, since it would mean checking a condition every time that you did arithmetic, that would likely _at least_ double the cost of doing any arithmetic. And particularly since arithmetic is such a basic operation that _everything else_ relies on, that could get really expensive, really fast. Yeah. I don't think that that's negotiable. Absolutely best case, I could see adding a compiler flag to enable it for debugging purposes, but it would definitely be expensive to do such checks and would be totally unacceptable in the release build of a systems programming language. - Jonathan M Davis Yes, I agree that checking all the time would be too expensive. What I meant was that we could provide functions that could do appropriate checking when it is needed. Andrei didn't like the functions idea, suggesting types that do policy-based checking, which I am happy with. -- Graham St Jack
Re: New syntax for string mixins
I've attached a part of how concurrency.d could look like translated to my suggested syntax. It probably contains a lot of errors because did a quick translation and I had some trouble understanding the mixins. Yes, even I couldn't understand them even a week later. Maintenance is a real problem. My initial reaction is that the proposed syntax helps a bit, but isn't really a game-changer. I will let you know if I change my mind on closer inspection. -- Graham St Jack
Re: New syntax for string mixins
On 14/12/10 20:33, Vladimir Panteleev wrote: On Tue, 14 Dec 2010 09:30:46 +0200, Graham St Jack graham.stj...@internode.on.net wrote: There is of course the worry that it could get so easy that everyone starts doing it, and we have (relatively) impenetrable code everywhere instead of just deep in the bowels of framework libraries. TBH, I'm more excited by AST macros which I understood are planned for D3, as mentioned here: http://s3.amazonaws.com/dconf2007/WalterAndrei.pdf They seem to promise the power of mixins, but without the mess. I took a look at the pdf, but couldn't see how the AST macros could come close to the kinds of things that are possible (but difficult) with mixins. Is there more information somewhere? Jacob, I can see how your proposed syntax would make simple mixins easier and clearer, but how would it do something more complex? For example taking a classname and a bunch of field types and names, and turning it into a class definition complete with constructor from field values, constructor from an input stream, a method to write to an output stream, and const getters. Or maybe std.typecons.AutoImplement. -- Graham St Jack
Re: New syntax for string mixins
I don't know, do you have an example ?
For example taking a classname and a bunch of field types and names, and
turning it into a class definition complete with constructor from field
values, constructor from an input stream, a method to write to an output
stream, and const getters. Or maybe std.typecons.AutoImplement.
Could you post an example of how that mixin would be used and the code
it would generate then I can see if I can translate it to my syntax.
AutoImplement seems to just contain template mixins which is something
else.
I have attached my concurrency framework, which relies heavily on
mixins, plus its unit test to show how it is used. I haven't included
the various dependencies because I assume you just want the example
code. Let me know if you want something buildable, or perhaps something
more cut-down.
What the code-generating template does is to create from something like
this (you can have any number of Messages in a Protocol):
alias Protocol!(Requests,Message!(job,string, name)).code
jobCode;
mixin(jobCode);
this code:
class Requests {
struct jobMsg {
string name;
this(string name) {
this.name = name;
}
void read(InStream stream) {
name = stream.get!string;
}
void write(OutStream stream) {
stream(name);
}
}
struct Message {
uint kind;
union {
jobMsg job;
}
this(ref jobMsg msg) {
kind = 0;
job = msg;
}
this(InStream stream) {
kind = stream.get!uint;
switch(kind) {
case 0: job.read(stream); break;
default: assert(0, Cannot read unsupported message kind);
}
}
void write(OutStream stream) {
stream(kind);
switch(kind) {
case 0: job.write(stream); break;
default: assert(0, Cannot write unsupported message
kind);
}
}
}
private alias Channel!(Message) _Chan;
private alias shared _Chan Chan;
private Chan channel;
this() { channel = new Chan(); }
ChannelSelectable newSelectable() { return channel.newSelectable(); }
void finalize() { channel.finalize; }
interface IHandler {
void job(string name);
}
void job(string name) {
channel.add(Message(jobMsg(name)));
}
void receive(IHandler handler) {
auto message = channel.remove;
switch (message.kind) {
case 0: handler.job(message.job.name); break;
default: assert(0, Cannot dispatch unsupported message kind);
}
}
}
I use this for inter-thread communications, and I use the discriminated
union to pass messages between processes. The manual mixin after the
alias is a debugging aid - the compiler errors when doing mixins aren't
helpful at all. I case you are wondering why I did all this, it was
partly a learning experience, but mostly an attempt to do something
properly thread-safe (!hasAliasing), using shared, const and immutable
properly.
--
Graham St Jack
module bedrock.maxim.concurrency;
public import bedrock.maxim.io;
public import bedrock.maxim.stream;
import bedrock.maxim.logging;
import std.algorithm;
import std.stdio;
import std.conv;
import std.traits;
import std.typecons;
import std.typetuple;
import core.thread;
import core.stdc.stdlib;
import core.stdc.errno;
import core.stdc.config;
import core.sys.posix.pthread;
import core.sys.posix.signal;
static import linux = std.c.linux.linux;
//
// Provides a message-passing concurrency implementation.
// * Messages cannot contain references to mutable data.
// * Message queues (Channels) are explicitly created and passed to threads as
// arguments to spawn().
// * Channels are synchronized (and thus shared).
// * The types of messages a Channel supports are well-defined.
// * Messages are processed in order.
// * Multiple threads can receive from the same Channel.
// * Channels have an internal pipe that can be used in a Selector to allow
// a thread to receive input from multiple Channels and file-descriptor devices
// such as sockets and pipes.
// * There is no automatic thread-termination behaviour.
//
// TODO - prevent non-shared aliased data from being passed to spawn().
// TODO - gathering non-fatal signals via signalfd, putting info on a Channel.
//
//-
// Condition - Adapted from core.sync.condition to be linux-specific,
// use an object's monitor and to be shared.
//-
public class ConditionException : Exception {
this(string text) { super(text); }
}
// clock_gettime definitions copied from commented-out and scattered
// definitions in druntime
private {
enum int CLOCK_REALTIME = 0;
enum int TIMER_ABSTIME = 0x01
Re: New syntax for string mixins
I have done a fair bit of mixin coding using recursive templates
(inspired by std.typecons). It was an amazing taste of what you can do
in D, and I am delighted with the result - HEAPS of boiler-plate coding
vanished before my eyes. However, the template code is virtually
impossible to understand.
What you are suggesting here seems to be a way to dramatically reduce
the complexity of code that generates source-code and mixes it in. I
think something like that is needed before this mind-bogglingly powerful
feature of D can realise its potential.
There is of course the worry that it could get so easy that everyone
starts doing it, and we have (relatively) impenetrable code everywhere
instead of just deep in the bowels of framework libraries.
On 14/12/10 07:07, Jacob Carlborg wrote:
This is an idea I've been thinking of for a while, it's not a really
suggestion (at least not yet) I just wanted to here what people think
about it.
If we take a step back and look at what string mixins actually do or
rather what they're used for, that would be: inserting a piece/block
of code where the mixin expression is used. If we then take a look at
how a block of code is represented in D (how you store it in variables
and how you pass it around). It's not as a string which is used by the
mixin expression, instead delegates are used to represent a block of
code in D that can be passed around. Therefore this is my idea:
Add a new mixin operator @ (1). When that operator is put in front
of a function call it would behave as a string mixin. The function
that is called needs to be CTFE and return a delegate or an array of
delegates:
class Foo
{
@get_set(int, bar);
}
The above code would be the same as the following code:
class Foo
{
mixin(get_set(int, bar));
}
If we now move to the declaration of get_set this is how it could
look like:
void delegate () get_set (string type, string name)
{
return
{
@type _...@name;
@type @name ()
{
return _...@name;
}
@type @name (@type @name)
{
return _...@name = @name;
}
};
}
In the above code when @ is used in the delegate literal it
basically behaves like string interpolation, so @type would be
replaced with the content of the type variable (2).
When the get_set function is called with the mixin symbol the
content of the returned delegate is inserted where the call is made.
If the function returns an array of delegates then the array would be
unfolded and the content of all the delegates would be inserted.
Taking it one step further:
Allow the mixin syntax to be placed in front of most of the
declarations and drop the need for string literals, commas and
parentheses, basically allowing the following syntax:
class Foo
{
@get_set int bar;
}
Would be translated into this:
class Foo
{
@get_set!(int)(bar);
}
Maybe one could do something like this as well:
@singleton class Foo
{
}
void delegate () singleton (string name, void delegate () classBody)
{
return { // a simple singleton implementation
class @name
{
static @name instance;
static this ()
{
instance = new @name;
}
private this () {}
@classBody;
}
};
}
Above, in the last example, @classBody would insert the content of the
delegate, basically the class body.
1. I will use @ in this example because I think it looks good but it
would probably conflict with @nothrow, @property and others.
2. I have no idea if this usage of the mixin symbol, @, will
conflict with the first usage.
--
Graham St Jack
Re: const(Object)ref is here!
On 07/12/10 05:51, Steven Schveighoffer wrote: On Mon, 06 Dec 2010 14:04:43 -0500, Jonathan M Davis jmdavisp...@gmx.com wrote: On Monday, December 06, 2010 07:37:27 Steven Schveighoffer wrote: It should be relatively painless. Just change the signatures of the base functions, and fix any compile errors. And watch all the code break... I'll definitely welcome the change (it's probably the first bug that I voted on in bugzilla), but there will be tons of code broken by it, since you just _know_ that a lot of user code doesn't bother to make toString() and its friends const. It's still a change that needs to be made though. Being unable to properly compare const and immutable objects is crippling. Of course, it would be more of an issue if it were easier to actually create immutable objects which are classes rather than strcuts, but that's a separate issue. Yes, one of the issues is that const has a viral effect. If you ignore const, none of your functions are const. So if you use functions inside opEquals, those have to be marked as const, and so on. But there are very few circumstances where opEquals needs to be marked as mutable. If that is the case, there is something broken with your code (and you should fix it) or there's something wrong with code you use (and you'll have to insert casts to deal with it for now). In a very small number of circumstances, non-const opEquals can be beneficial (such as caching data that is expensive to calculate). We need to find another way to deal with that (I suggest having logical const, but that's not a popular idea with Walter :). But it shouldn't detract from the requirements. You can always circumvent if you need special cases. I'd rather start from a solid const-supporting position and add holes where they make sense then start from a base that is full of holes and try to patch them slowly. In many cases affixing const to your code is not just a simple 'slap a const on here'. It can involve careful thought and possibly redesign. But without these changes, const is very useless, the standard library needs to eat its own dogfood if we want to peddle it to others. -Steve Vote++ -- Graham St Jack
Re: const(Object)ref is here!
First, I have to say that it is wonderful that someone is taking a
serious look at this area again, and even better, you have come up with
a compiler patch to make it happen!
Some questions (assuming your patch or something like it gets into dmd):
Does this mean that I would be able to write this:
immutable(Foo)ref foo; // create a reference
foo = new immutable(Foo)(); // re-bind it (not sure about new syntax
for immutables)
Are there any other show-stopping syntax issues that are holding up
widespread adoption/rollout of const-correctness?
What do Walter and Andrei think?
Does anyone know what the plan is for rolling const-correctness
thoughout druntime and phobos (and shared/immutable in std.concurrency)?
Having to do casting all the time is a real drag. I for one would be
happy to help if help is needed.
On 06/12/10 11:21, Michel Fortin wrote:
After a recent discussion on this list about tail-const class
references, it became rather clear that someone interested would have
to implement the thing if we were to have it. So I did it. See
enhancement request 5325 for an explanation and a patch.
http://d.puremagic.com/issues/show_bug.cgi?id=5325
Let's hope Walter likes my patch.
In the tail-const thread, I proposed the challenge of making this code
work when an array of const object is passed as an argument:
T[] giveMeASortedArray(alias Predicate, T)(T[] t) {
// creating new array of the same length but with assignable
elements
auto copy = new Unqual!(typeof(t[0]))[t.length];
foreach (index, value; t)
copy[index] = value;
// sorting the copy
sort!(Predicate)(copy);
return copy;
}
Well, with the patch I made it now works!... irrespective of the type
attribute (const,immutable,shared,inout). The only modification
required to Phobos is to make to!string() compile when passed a
const(Object) because somehow 'sort' requires that.
Here is how the function is invoked:
void main() {
int*[] a = giveMeASortedArray!(a b)(new int*[12]);
Object[] b = giveMeASortedArray!(a b)(new Object[12]);
const(int*)[] c = giveMeASortedArray!(a b)(new
const(int*)[12]);
const(Object)[] d = giveMeASortedArray!(cast(void*)a
cast(void*)b)(new const(Object)[12]);
}
See the strange predicate for the const(Object) version? That's
because opCmp() in Object doesn't work with const.
Two minor modifications are required in Phobos to make the above
compile. First, to!string(const(Object)) needs an implementation
because somehow 'sort' requires it. Also template std.traits.isMutable
needed an adjustment so that swap()'s template constrains are satisfied.
--
Graham St Jack
Re: Initialization of unions
Hi Justin,
I am using a struct as a discriminated union in my version of the
concurrency framework, and it all seems to work just fine. Here is a
code fragment from the code that implements a simple protocol. The code
is generated by a template, so it is very easy to crank out a message
struct with lots of different kinds of payload.
struct jobMsg {
string name;
this(string name) {
this.name = name;
}
void read(InStream stream) {
name = stream.get!string;
}
void write(OutStream stream) {
stream(name);
}
}
struct Message {
uint kind;
union {
jobMsg job;
}
this(ref jobMsg msg) {
kind = 0;
job = msg;
}
this(InStream stream) {
kind = stream.get!uint;
switch(kind) {
case 0: job.read(stream); break;
default: assert(0, Cannot read unsupported message kind);
}
}
void write(OutStream stream) {
stream(kind);
switch(kind) {
case 0: job.write(stream); break;
default: assert(0, Cannot write unsupported message kind);
}
}
}
On 23/09/10 21:58, Justin Johansson wrote:
On 23/09/2010 10:14 PM, bearophile wrote:
Justin Johansson:
One of the problems with C++ is that it is not possible
to create unions with non-primitive members (e.g. structs)
that have constructors.
Do you mean something like this?
struct S1 {
int y;
this(int x) { y = x; }
}
struct S2 {
string t;
this(string s) { t = s; }
}
union U {
S1 s1;
S2 s2;
}
static U u2 = { s2:S2(hello) };
void main() {
U u = U(S1(10));
assert(u.s1.y == 10);
u.s2 = S2(hello);
assert(u.s2.t == hello);
// U u3 = U(S2(hello)); // not possible
}
Yes, but not yes; something like that. You are obviously
one step ahead of me so perhaps I should give up or else
post the exact problem in C++. Still, it looks likes
from what you have shown that D has some better union
construction syntax than C++.
I hope others can throw in their 2 cents.
Bye,
Justin
--
Graham St Jack
Re: d.vim 0.20
On 31/08/10 03:38, Jesse Phillips wrote: There is a new version of the D syntax file for Vim. This is version 0.20 and is available at http://www.vim.org/scripts/script.php?script_id=379 * Added missing keywords for D 2.0 compiler 2.047 * Added special highlighting of known versions/scopes/annotations (thanks to Shougo Matsushita) * Added highlighting of ASM Op Codes in asm blocks. I highly suggest you drop this latest version into your syntax folder ~/.vim/syntax Linux and ~/_vimfiles/syntax Windows (where ~ is users directory, not documents). I am the new maintainer of this file so please direct any comments, suggestions, patches my way at jesse.k.phillip...@gmail.com thanks++ -- Graham St Jack
Re: The Status of Const
On 14/08/10 14:39, Denis Koroskin wrote: Graham St Jack Wrote: For me, the key problem is that a class object reference has the same const/immutable/shared attribute as the object on the heap that it refers to. This is sometimes what you need, but more often you want a non-shared, mutable reference to a const/immutable/shared object. You can achieve this with pointers for arrays, structs and primitive types, but not with classes because a class pointer is just a pointer to a reference. I discussed a possibility of requiring '*' to denote both references AND pointers before TDPL was out. It would solve a whole bunch of language issues, including this one: Rebindable!(immutable(Foo)) bar; - immutable(Foo)* bar; There is a lot more benefits in the proposal than you might think at first. Here is the link: http://www.digitalmars.com/d/archives/digitalmars/D/What_if_D_would_require_for_reference_types_104816.html -- Graham St Jack Thanks for that. I must admit that the proposal looks really good to me, especially if it is compulsory to always use a pointer - i.e. cannot have a class object by value. But - for some reason I don't understand, this and the many other proposals I have seen don't get up. Maybe what is needed here is for someone (Walter?) to write up the results of the various const/immutable/shared discussions, and put it on the website, together with suggested usage idioms. Then we will all understand why things are the way they are, and how we should be using these language features. This issue has been a running sore for a long time, but maybe all it needs is an authoritative write-up to sort it out. -- Graham St Jack
Re: The Status of Const
On 14/08/10 05:55, Walter Bright wrote:
Graham St Jack wrote:
For me, the key problem is that a class object reference has the same
const/immutable/shared attribute as the object on the heap that it
refers to. This is sometimes what you need, but more often you want a
non-shared, mutable reference to a const/immutable/shared object.
I struggled with this for a long while before I eventually came to the
conclusion that I was thinking about class objects as a reference and
the instance, and that those two were separable concepts. They are not.
A reference type is implicitly treated like a value type as far as
accessing its members go. Trying to separate out the two is a
fundamental misunderstanding of what reference types are all about.
Semantically, they are much more than just a pointer to the instance.
You can achieve this with pointers for arrays, structs and primitive
types, but not with classes because a class pointer is just a pointer
to a reference.
The way to do it with class is take a pointer to the class:
class C { ... }
const(C)*[]; // array of pointers to const classes
just like you'd do with a struct.
I get it - I think. I will give this a try and see how it works in practice.
--
Graham St Jack
Re: The Status of Const
I have tried using const/immutable/shared several times, and am
currently in the process of giving it another shot, with much more
success that before, but it is hard work.
The major language hassle for me is that a class reference has the same
const/immutable/shared type as the object is refers to, unlike pointers.
I know that Rebindable is a workaround, but I don't find it satisfactory
- essentially it uses brute-force casting to defeat the type system, and
isn't syntactically identical to the bound type.
The data I pass between threads has to pass the !hasAliasing test, and I
haven't found a practical way to make it work for classes. What works
best for me is a struct with a pointer to immutable data like so:
struct Foo {
struct Payload {
...
}
immutable(Payload)* payload;
...
}
Foo can then be passed, contained by value and assigned to, just like
any other value type. Containing an immutable instance of Foo is ok too,
if you need another layer of !hasAliasing. I tend to define all my
non-trivial building-block data types this way, and pass them by value.
With careful design of my multi-threaded code, I can define a set of
messages to pass between threads that don't need to contain complex data
structures. The resultant design is always an improvement over the more
complex interface that first comes to mind, so I find that the type
system is pushing me in a good direction.
The next big hurdle (as you pointed out) is that druntime and phobos
aren't designed for const/immutable/shared. For example, InternetAddress
fails the !hasAliasing test, and concurrency Mailboxes defeat the type
system rather than being shared themselves, and don't insisting that
message parameters have to be !hasAliasing. So far I have had to
implement my own concurrency, socket and condition modules to overcome
these issues, and I'm sure those are just the beginning.
So there are plenty of bumps in the road, but it does look like it is
finally possible to use const/immutable/shared if you are determined
enough. That said, it needs to be way, way easier before wide adoption.
The library issues can be addressed easily enough - It didn't take me
long to pull off versions that worked ok (Linux only I'm afraid).
However, the issue with classes doesn't look so easy, and that seriously
limits what can be passed in messages.
Is there any plan to introduce some way of having a mutable reference to
an immutable class object on the heap? Do others see this as a problem
at all?
On 13/08/10 08:26, dsimcha wrote:
This is from a discussion that originated on the Phobos mailing list, but I
thought I'd bring up the question of what should be done about const on the
newsgroup to see what others think:
Despite its theoretical beauty, I find D's const/immutable system to be
utterly useless for all but the simplest use cases. I made a serious attempt
a while back to use it in a real multithreaded program. In hindsight it was
more trouble than it was worth, largely for three reasons:
1. It's difficult to create non-trivial immutable data structures, and often
impossible without relying on either unchecked casts or unnecessary copying.
2. Much generic code in Phobos (even things as simple as std.math.pow()
before I recently fixed it) behaves incorrectly when given const/immutable
data. This also applies to other libraries I use, including ones that I'm the
main author of, so I'm just as guilty of it as anyone. Given that noone,
including me, seems to be able to get const to interact well with generic
code, perhaps we need a language-level solution.
3. inout is currently so bug-ridden it's not even funny. (Though this is
clearly fixable long-term, once we get higher priority stuff off our plates.)
It would have probably been better if this was brought to a head sooner, but
it's better late than never. Do others agree that D's const system is
difficult to impossible to use properly? Has anyone successfully used D's
const system in a non-trivial setting despite these limitations? If so, was
it more trouble than it was worth in hindsight? How can these limitations be
worked around and/or fixed?
--
Graham St Jack
Re: The Status of Const
On 13/08/10 09:51, Brad Roberts wrote: On Thu, 12 Aug 2010, dsimcha wrote: This is from a discussion that originated on the Phobos mailing list, but I thought I'd bring up the question of what should be done about const on the newsgroup to see what others think: Despite its theoretical beauty, I find D's const/immutable system to be utterly useless for all but the simplest use cases. I made a serious attempt a while back to use it in a real multithreaded program. In hindsight it was more trouble than it was worth, largely for three reasons: 1. It's difficult to create non-trivial immutable data structures, and often impossible without relying on either unchecked casts or unnecessary copying. 2. Much generic code in Phobos (even things as simple as std.math.pow() before I recently fixed it) behaves incorrectly when given const/immutable data. This also applies to other libraries I use, including ones that I'm the main author of, so I'm just as guilty of it as anyone. Given that noone, including me, seems to be able to get const to interact well with generic code, perhaps we need a language-level solution. 3. inout is currently so bug-ridden it's not even funny. (Though this is clearly fixable long-term, once we get higher priority stuff off our plates.) It would have probably been better if this was brought to a head sooner, but it's better late than never. Do others agree that D's const system is difficult to impossible to use properly? Has anyone successfully used D's const system in a non-trivial setting despite these limitations? If so, was it more trouble than it was worth in hindsight? How can these limitations be worked around and/or fixed? For discussions like this, I think it's essential to distinguish between the language vs the runtime + core libraries. I recognize what matters is the end result usability, but examining the layers independently is really important. So, which are you talking about (could well be both)? For me, the key problem is that a class object reference has the same const/immutable/shared attribute as the object on the heap that it refers to. This is sometimes what you need, but more often you want a non-shared, mutable reference to a const/immutable/shared object. You can achieve this with pointers for arrays, structs and primitive types, but not with classes because a class pointer is just a pointer to a reference. -- Graham St Jack
Re: The Status of Const
On 13/08/10 10:08, Andrei Alexandrescu wrote: Trass3r wrote: Well isn't it natural that a constness system has a much larger impact than just some syntax additions. I don't see any flaws in its design. The implementation is of course still buggy though and needs to mature. I agree. There are very severe bugs and undue limitations in today's const. Having a comprehensive discussion of const's status and role in current and future D idioms is a great idea. We should start it with a scrutiny of the reported and possibly unreported bugs in the feature. Andrei I also agree that cost is worth the effort. Most of the roadblocks are relatively easy to overcome by rolling const-correctness through druntime and phobos. However, I still regard the language design decision of a class reference having the same constness as the object it refers to as a major language design problem. I would be delighted if someone could point out to me how to neatly work around this though. -- Graham St Jack
Re: The Status of Const
On 13/08/10 10:18, Jonathan M Davis wrote: On Thursday, August 12, 2010 17:38:28 Graham St Jack wrote: For me, the key problem is that a class object reference has the same const/immutable/shared attribute as the object on the heap that it refers to. This is sometimes what you need, but more often you want a non-shared, mutable reference to a const/immutable/shared object. You can achieve this with pointers for arrays, structs and primitive types, but not with classes because a class pointer is just a pointer to a reference. Hence the hack that is Rebindable!(). Oh, and you _can_ achieve pointers to classes, but what you normally use are references, which do have the problem of not being able to be split between the reference and referent types. - Jonathan M Davis So how do you get a pointer to an object? Taking the address of an object reference gives you a pointer to the reference, not the object, which is fair enough. As far as I know there isn't a way to get a pointer to the object itself, and even if you could, how do you use such a thing? -- Graham St Jack
Re: A working backtrace for linux
On 04/08/10 05:30, Sean Kelly wrote: Trass3r Wrote: Which reminds me that there are still no traces for Windoze, even though Tango has been providing for quite some time IIRC. Then perhaps the person who wrote the Tango Win32 backtrace will submit it to Phobos :-) I've been busy with other things recently, but Win32 backtrace is coming. It's just more work than the 30 minutes it took me to do the Linux/OSX one (which admittedly needs some work as well). Sean, is there any chance of you rolling some of the good stuff in this code into druntime? In particular, it would be very nice to get symbol names in a stacktrace rather than a bunch of addresses, and even nicer to get a stacktrace on a SIGSEGV. -- Graham St Jack
Re: A working backtrace for linux
On 04/08/10 12:47, Brad Roberts wrote: On 8/3/2010 8:09 PM, Sean Kelly wrote: Graham St Jack Wrote: Sean, is there any chance of you rolling some of the good stuff in this code into druntime? In particular, it would be very nice to get symbol names in a stacktrace rather than a bunch of addresses, and even nicer to get a stacktrace on a SIGSEGV. You can get the symbol names by adding -L--export-dynamic to your DFLAGS in dmd.conf. That option is on by default on OSX and I didn't realize it was different on Linux. I'll sort out demangling as well, it's just a bit more work. I like the stack trace code provided, but it uses a lot of modules that aren't available to druntime so it would be difficult to use directly. see also my notes in bug 1001, the one all about this topic. Thanks for the heads-up - I will keep a closer eye on the bug-list in future. Its great to see all the action going on under the hood. -- Graham St Jack
Re: poll about delete
On 28/07/10 05:47, Andrei Alexandrescu wrote: Max Samukha wrote: On 07/27/2010 10:53 PM, Andrei Alexandrescu wrote: and scope storage class. If scope storage class is going, we need a library equivalent. Current 'scoped' is not good due to 4500. Agreed. I know how to fix it. Andrei I missed the previous discussions about removing the scoped storage class. Would you mind re-stating the argument here, or post a link to it? FWIW I'm happy for delete to be removed. -- Graham St Jack
Re: poll about delete
On 28/07/10 07:50, Andrei Alexandrescu wrote: Graham St Jack wrote: On 28/07/10 05:47, Andrei Alexandrescu wrote: Max Samukha wrote: On 07/27/2010 10:53 PM, Andrei Alexandrescu wrote: and scope storage class. If scope storage class is going, we need a library equivalent. Current 'scoped' is not good due to 4500. Agreed. I know how to fix it. Andrei I missed the previous discussions about removing the scoped storage class. Would you mind re-stating the argument here, or post a link to it? FWIW I'm happy for delete to be removed. In brief scope is unsafe and impossible to make safe without extensive changes to the language. Andrei Thanks. I definitely agree that it should go if it can't be made to work properly. -- Graham St Jack
Concurrency
I have been using std.concurrency for a while, and have been very
impressed by it. In particular, it was easy to get a complex
multi-threaded application going with std.concurrency. The
thread-local-storage behaviour of D is really cool too.
However, I have had some problems with std.concurrency (most of which
I'm sure are already being addressed by Sean). In particular, I wanted a
way to select on multiple file-descriptors so that I could (say) send
commands to a thread that also reads from a blocking file-descriptor
like a serial port, UDP socket or GTK main loop. I also wanted to
experiment with the amazing template capabilities of D that I found out
about in the book, and see if I could generate boiler-plate
message-passing code that doesn't rely on magic like Variant.
The attached file is where I am at so far. It works, but still needs a
lot of work to bring it up to scratch. The reason for posting it here is
to stimulate some discussion about how D's standard concurrency module
should behave (and maybe get a few helpful tips).
Please take a look and let me know what you think.
--
Graham St Jack
module alt.concurrency;
//public import alt.io;
import std.algorithm;
import std.contracts;
import std.stdio;
import std.conv;
import std.traits;
import std.typecons;
import std.typetuple;
import core.sync.condition;
import core.sync.mutex;
import core.thread;
import core.stdc.stdlib;
//
// Provides an alternative message-passing concurrency implementation
// to std.concurrency. It is similar to std.concurrency, but is different in that:
// * Messages cannot contain references to mutable data.
// * Message queues (Channels) are explicitly created and passed to threads as
// arguments to spawn().
// * Channels are synchronized (and thus shared).
// * The types of messages a Channel supports are well-defined.
// * Messages are processed in order.
// * Multiple threads can receive from the same Channel.
// * Out of band messages are not supported directly, but can be implemented via
// additional Channels.
// * Channels have an internal pipe that can be used in a Selector to allow
// a thread to receive input from multiple Channels and file-descriptor devices
// such as sockets and pipes.
// * There is no automatic thread-termination behaviour.
//
// The motivation for this module is to stimulate some discussion about
// how D's standard message-passing support should behave.
// It is also a learning experience for the author.
// The std.concurrency issues of most interest to the author are:
// * Messages are not named - they are inferred from the types.
// * Lack of compile-time checking.
// * It allows mutable references to be sent in messages.
//
// Graham St Jack, July 2010.
// This is free software - do with it as you will.
//
// Currently only Linux is supported.
//
// TODO - prevent non-shareable data from being passed to spawn().
// TODO - handling of signals.
// TODO - print demangled stack trace on unexpected thread death.
// TODO - provide Windows and Mac support in alt.io, tidy up alt.io
//and debug it properly.
//
//
// A Channel for sending messages between threads.
// The internal pipe is used to block removal until a message is available,
// and to provide a file-descriptor to select on. Multiple threads
// may add and remove messages from the queue, but usually one thread
// adds and one other thread removes.
//
public class ChannelFull : Exception {
this() { super(channel full); }
}
public class ChannelFinalized: Exception {
this() { super(channel finalized); }
}
// A SelectableReader returned from a Channel for use in a Selector
// The Channel itself cannot be used because it is shared.
/*
class Selectable : ISelectableReader {
private int mFd;
this(int fd) {
mFd = fd;
}
// ISelectableReader implementation:
override uint read(void * buffer, uint count) {
assert(false, read is not supported);
}
override int read_descriptor() {
return mFd;
}
}
*/
// FIXME - class should be synchronized, but druntime.core.sync's
// Condition class doesn't work with synchronized classes or methods yet.
class Channel(T) {
private {
struct Node {
T payload;
Node* next;
this(T payload) {
this.payload = payload;
}
}
Node* mFront;
Node* mBack;
uint mCapacity;
uint mCount;
//ubyte mZero;
//Pipe mPipe;
bool mFinalized;
Mutex mMutex;
Condition mReadCondition;
Condition mWriteCondition;
}
this(uint capacity = 0) {
mCapacity = capacity;
//mPipe = new Pipe();
mMutex = new Mutex();
mReadCondition = new Condition(mMutex);
mWriteCondition = new Condition(mMutex);
}
// Return a new Selectable that can be used in a Selector
Re: Concurrency
On 26/07/10 13:48, Sean Kelly wrote: Graham St Jack Wrote: However, I have had some problems with std.concurrency (most of which I'm sure are already being addressed by Sean). In particular, I wanted a way to select on multiple file-descriptors so that I could (say) send commands to a thread that also reads from a blocking file-descriptor like a serial port, UDP socket or GTK main loop. Hm... this is essentially what libev and libevent do. The trick is mostly making this work in a consistent and performant manner on both Posix and Windows. On Posix, the read event simply tells you data is available, while on Windows the read event hands the data to you directly. The former works quite well with messaging while the latter... well, it's kind of weird. I don't know much about Windows programming, but I was expecting this to be the sticking point. So do we have a solution in D for this? For me, it is the key issue with std.concurrency as it stands. Using the Pipe approach as I did in Channel, maybe a Windows version could allow a byte to be removed from the Pipe for each message, and another byte immediately put on to replace it if the Channel was not then empty. I like that you're playing with Channels, by the way. I chose the messaging API in std.concurrency because it can have a range of more structured models built on top of it, so I think it's more likely to see broad use done this way. Maybe channels will be the first :-) The Channels I proposed were really just a first try at doing code generation, but they do work really well. I particularly like being able to print out the generated code to see what you are getting. I must admit that I find Variant perplexing, and had issues with the message size limitation and no prohibition on aliased message parameters. In working with both std.concurrency and alt.concurrency, I have found that alt.concurrency has some advantages such as: Creating Channels first and providing them as arguments to spawn() is simpler than spawning threads and passing Tids in messages. Being able to use the same set of parameter types in two different messages, and having names for the messages is clearer. Being able to have multiple threads reading messages from the same Channel - something I need (well, want because it is much easier than double-handling all the messages through an intermediary) on my current project. The benefit of compile-time checking. Of course std.concurrency has lots of bonuses too. What I am after is a discussion that leads to a std.concurrency that is better still. By the way, how are you getting on with rolling out shared through core.sync? I'm very keen to find out if shared can at last be used as intended - such as for implementing a Channel. I could lend a hand if you need one. -- Graham St Jack
Re: TDPL, shared data, and Phobos
On 23/07/10 10:23, Sean Kelly wrote: awishformore Wrote: On 22/07/2010 01:49, Robert Jacques wrote: Have you tried core.sync.rwmutex? Also, please remember that CREW locks are not composable and can easily lead to dead-locks. Afaik, the current rwmutex is a wrapper around two separate mutexes (one for readers, one for writers) and you have to decide whether readers or writers get precedence, meaning that ether all writers in the queue have to wait if just one reader has to write or all writers in the queue have to wait if there is a single reader comes up. This is very unlike the behaviour I would like to see; I would expect readers and writers to be in the same queue, meaning the only difference between the rw and the normal mutex would be that all subsequent readers in the queue can read at the same time. ReadWriteMutex exposes a read and write interface, but there certainly aren't two actual mutexes underneath. It's true that the implementation doesn't explicitly maintain a queue, but this is intentional. If readers and writers in the queue have different thread priorities set, those priorities should be honored, and it's pointless to write all that code in druntime when the OS takes care of it for us. Instead, those waiting for access to the mutex all block on a condition variable and whoever wakes up first wins. It's up the OS to make sure that thread priorities are honored and starvation doesn't occur. It isn't clear that thread priorities will do the job here. I have been burned before by things like priority inheritance chaining, and other ways that thread priorities can be elevated for potentially long periods of time. Priority inheritance chaining goes like this: Thread low locks mutex A, then mutex B Thread high tries to lock mutex B, elevating low's priority to high's so that high can get the mutex quickly. When thread low releases mutex B (letting high get it), the OS has trouble figuring out what low's priority should now be, and leaves it elevated until it releases all mutexes it still has (mutex A in this case). Low is now running at a high priority, preventing thread medium from getting any CPU. This scenario happened for me with vxWorks some time back, and is the reason I no longer do much work at all while I have a mutex locked. I am confident that it is a real problem to this day. -- Graham St Jack
Re: TDPL, shared data, and Phobos
On Sun, 18 Jul 2010 16:05:08 +, Sean Kelly wrote:
Graham St Jack graham.stj...@internode.on.net wrote:
On Sat, 17 Jul 2010 11:42:03 -0400, Sean Kelly wrote:
The casts are necessary because I haven't yet applied 'shared' to
druntime. I ran into a few issues when doing so and rolled back my
changes. I'll give it another shot before the next release.
I'm glad you announced you intention - I was just about to roll up my
sleeves and give it a go for Condition, but will wait for the next
release.
Like all my previous attempts to use shared, I have waited for quite a
while for things to improve, tried using shared again, hit a brick wall
and resorted to defeating the compiler's detection of shared data.
TDPL raised my hopes without actually making it clear how to use
synchronized classes. Alas, it seems to me that they still aren't
usable
in practice. With any luck the problems are just library issues which
can
be fixed relatively easily.
Like Brian Palmer, I am frustrated by the lack of documentation about
shared and druntime's sync package, and am happy to lend a hand if that
would be helpful.
The code I am trying to write is a simple synchronized class with a
Condition, but I can't create a Condition on a shared this.
A cut-down version of what I want to write is:
synchronized class Foo {
Condition mCondition;
this() {
mCondition = cast(shared) new Condition(this);
}
void some_method() {
}
}
I realise that Condition wants a Mutex, but a synchronized class
already
has an implicit one which is implicitly used by all the methods, so the
above is definitely what I want to write.
What I have to write instead (which avoids the compiler noticing that
anything is being shared) is:
class Foo {
Mutex mMutex;
Condition mCondition;
this() {
mMutex = new Mutex();
mCondition = new Condition(mMutex);
}
void some_method() {
synchronized(mMutex) {
}
}
}
The latter works just fine, but it is very disappointing after all the
fuss about how important shared is that you can't actually use it for
the
most mainstream of all uses - a synchronized class with a condition
(which is what a message queue between threads is supposed to be).
new Mutex(this) makes the mutex the object monitor, so it will be what's
locked for synchronized functions.
That's cool. I look forward to shared not being an issue ;-)
I assume that when Condition and Mutex are shareable, I will then (from
your other post) write:
synchronized class Foo {
Mutex mMutex;
Condition mCondition;
this() {
mMutex = cast(shared) new Mutex(this);
mCondition = cast(shared) new Condition(mMutex);
}
void some_method() {
...
mCondition.notify; // ok because mMutex is locked
}
}
Re: Debugging
On Fri, 16 Jul 2010 21:47:50 +, Sean Kelly wrote: -profile currently doesn't work with multithreaded apps. Darn. Are there plans to sort that out?
Re: TDPL, shared data, and Phobos
On Sat, 17 Jul 2010 11:42:03 -0400, Sean Kelly wrote:
The casts are necessary because I haven't yet applied 'shared' to
druntime. I ran into a few issues when doing so and rolled back my
changes. I'll give it another shot before the next release.
I'm glad you announced you intention - I was just about to roll up my
sleeves and give it a go for Condition, but will wait for the next
release.
Like all my previous attempts to use shared, I have waited for quite a
while for things to improve, tried using shared again, hit a brick wall
and resorted to defeating the compiler's detection of shared data.
TDPL raised my hopes without actually making it clear how to use
synchronized classes. Alas, it seems to me that they still aren't usable
in practice. With any luck the problems are just library issues which can
be fixed relatively easily.
Like Brian Palmer, I am frustrated by the lack of documentation about
shared and druntime's sync package, and am happy to lend a hand if that
would be helpful.
The code I am trying to write is a simple synchronized class with a
Condition, but I can't create a Condition on a shared this.
A cut-down version of what I want to write is:
synchronized class Foo {
Condition mCondition;
this() {
mCondition = cast(shared) new Condition(this);
}
void some_method() {
}
}
I realise that Condition wants a Mutex, but a synchronized class already
has an implicit one which is implicitly used by all the methods, so the
above is definitely what I want to write.
What I have to write instead (which avoids the compiler noticing that
anything is being shared) is:
class Foo {
Mutex mMutex;
Condition mCondition;
this() {
mMutex = new Mutex();
mCondition = new Condition(mMutex);
}
void some_method() {
synchronized(mMutex) {
}
}
}
The latter works just fine, but it is very disappointing after all the
fuss about how important shared is that you can't actually use it for the
most mainstream of all uses - a synchronized class with a condition
(which is what a message queue between threads is supposed to be).
Re: Debugging
On Sat, 17 Jul 2010 14:43:38 -0400, Sean Kelly wrote: Graham St Jack Wrote: On Fri, 16 Jul 2010 21:47:50 +, Sean Kelly wrote: -profile currently doesn't work with multithreaded apps. Darn. Are there plans to sort that out? It's been on my to do list for ages. Timing data is less/differently useful for multithreaded code, but it should at least not crash, unlike now. I find profiling very useful in multi-threaded programs for assisting with optimisation, and am keen for it to be possible. I use C++ in my day job (with gcc), and recently went through a lot of pain trying to get gprof to work, only to discover that it fundamentally doesn't work on multi-threaded code. I guess that is fair enough given that instrumented code can't use something as simple as a high-resolution clock to measure how long things take. What I ended up using was sysprof, which is an external program that does statistical sampling of the whole system. It works really well, and produces a respectable call graph that shows very clearly where the time is being spent. However, when I try it with a D2 program, I don't get a call graph - presumably because it doesn't understand the stack frames and so can't work out the call graph. This renders it almost useless for D programs.
Debugging
I have jumped back onto the D2 band-wagon after a long absence, read
Andrei's book, and am having a great time cutting heaps of D code. My
increase in productivity and happiness when compared to working in C++ is
enormous.
Bug thumbs up to everyone involved with D2, Phobos and the book.
One area I am having a bit of trouble with is debugging. Can anyone help
me out with how to debug a D2 program in Linux? Currently I am reduced to
printing out heaps of debug text.
Specifically:
Stack Trace:
I can't get the D2 stack-trace to work properly. All I get is something
like this, which isn't helpful:
Segmentation fault
The code I used to generate this was:
import std.stdio;
import std.file;
void foo(File file) {
file.flush;
}
void main(string[] args) {
File file;
foo(file);
}
Is stack-trace support broken, or do I have to do something to enable it?
GDB:
What is the status of D support in GDB? The last post I saw was back in
April. It is currently hard work to debug with gdb when it doesn't
understand D name mangling.
Profiling:
--
How do I profile a D2 program? When I try -profile in dmd, the resultant
executable crashes with a segfault when I try to run it. When I try
sysprof, I get a handful of mangled names, and no call graph.
Re: Debugging
On Tue, 13 Jul 2010 03:55:08 -0400, bearophile wrote: Graham St Jack: How do I profile a D2 program? When I try -profile in dmd, the resultant executable crashes with a segfault when I try to run it. Are you able to produce a small test case of this bug? Bye, bearophile Small test cases don't crash. I will spend a bit of time experimenting with my way-too-big test case to find out why it crashes when run after being compiled with -profile, and post my results here. One thing I noticed was that the trace.log file doesn't demangle the names properly either.
Re: Debugging
On Tue, 13 Jul 2010 08:48:50 -0400, Steven Schveighoffer wrote:
On Tue, 13 Jul 2010 02:26:22 -0400, Graham St Jack
graham.stj...@internode.on.net wrote:
I have jumped back onto the D2 band-wagon after a long absence, read
Andrei's book, and am having a great time cutting heaps of D code. My
increase in productivity and happiness when compared to working in C++
is enormous.
Bug thumbs up to everyone involved with D2, Phobos and the book.
One area I am having a bit of trouble with is debugging. Can anyone
help me out with how to debug a D2 program in Linux? Currently I am
reduced to printing out heaps of debug text.
Specifically:
Stack Trace:
I can't get the D2 stack-trace to work properly. All I get is something
like this, which isn't helpful:
Segmentation fault
The code I used to generate this was: import std.stdio;
import std.file;
void foo(File file) {
file.flush;
}
void main(string[] args) {
File file;
foo(file);
}
Is stack-trace support broken, or do I have to do something to enable
it?
Seg faults do not generate stack traces in Linux/MacOS. This is because
a segmentation fault is generated by a signal, and it's unsafe to throw
exceptions from signals.
I believe seg faults can generate exceptions in Windows, but I'm not
sure.
Your best bet is to get a debugger working, and it will halt on the
signal. As I understand it, a lot of good work was done recently on dmd
(can't remember who did it) to get it working better with gdb.
-Steve
Thanks. I will persist with gdb and look forward to the D support coming
through.
Re: Debugging
On Tue, 13 Jul 2010 10:34:41 -0500, Andrei Alexandrescu wrote: On 07/13/2010 01:26 AM, Graham St Jack wrote: I have jumped back onto the D2 band-wagon after a long absence, read Andrei's book, and am having a great time cutting heaps of D code. My increase in productivity and happiness when compared to working in C++ is enormous. Bug thumbs up to everyone involved with D2, Phobos and the book. [snip] Thanks. Bug thumbs up? Clearly you had debugging on your mind :o). Check http://www.zerobugs.org/. Andrei Freudian slip... I love the book by the way - I couldn't put it down. D is totally awesome now, and there were heaps of cool new features that I didn't know about. Thanks again.
Re: Debugging
On Tue, 13 Jul 2010 15:29:35 -0400, Jesse Phillips wrote:
Graham St Jack Wrote:
Specifically:
Stack Trace:
I can't get the D2 stack-trace to work properly. All I get is something
like this, which isn't helpful:
Segmentation fault
The code I used to generate this was: import std.stdio;
import std.file;
void foo(File file) {
file.flush;
}
void main(string[] args) {
File file;
foo(file);
}
Is stack-trace support broken, or do I have to do something to enable
it?
While you aren't asking why, 'file' is initialized to null and is not a
stack allocated class as it would be in C++.
Yes - it was a contrived example designed to deliberately segfault.
The first step to debugging is to use the -gc flag instead of -g. The
unreleased version of GDB has the patch for D mangling, otherwise you
need the symbols to mimic C. Then you can run your program with GDB.
Thanks - I have been doing that, but like I said, it is hard work
mentally demangling the names.
As you should be familiar with. You can get the stack trace form GDB if
you enable core dumps: $ ulimit -c 5000 dmd -gc test.d ./test
gdb ./test core
I didn't know about that, thanks. It looks like a more convenient way of
debugging segfaults than running the program in gdb directly. I will give
it a go.
Re: Debugging
On Tue, 13 Jul 2010 21:38:09 +, Graham St Jack wrote: On Tue, 13 Jul 2010 03:55:08 -0400, bearophile wrote: Graham St Jack: How do I profile a D2 program? When I try -profile in dmd, the resultant executable crashes with a segfault when I try to run it. Are you able to produce a small test case of this bug? Bye, bearophile Small test cases don't crash. I will spend a bit of time experimenting with my way-too-big test case to find out why it crashes when run after being compiled with -profile, and post my results here. One thing I noticed was that the trace.log file doesn't demangle the names properly either. Here is a gdb stacktrace from a program that was compiled with -profile and -gc. The program is too big to post here, but it works just fine without the -profile. Note that the problem seems to be something to do with variant and/or concurrency. I don't like my chances, but I will dig deeper. Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0xf7521b70 (LWP 32218)] 0x080834cc in trace_epi () (gdb) bt #0 0x080834cc in trace_epi () #1 0x0806ee56 in _trace_epi_n () #2 0xf7520ee4 in ?? () #3 0x08057b4d in _D3std7variant17__T8VariantNVk24Z8VariantN59__T7handlerTS3std8typecons24__T5TupleTC3bob7NewFileZ5TupleZ7handlerFE3std7variant17__T8VariantNVk24Z8VariantN4OpIDPG24hPvZi (parm=0xf7520eac, pStore=0x0, selector=3) at /home/grahams/local/dmd2/linux/bin/../../src/phobos/ std/variant.d:248 #4 0x08061310 in _D3std7variant17__T8VariantNVk24Z8VariantN62__T10convertsToTS3std8typecons23__T5TupleTC3bob6ActionZ5TupleZ10convertsToMFZb (this=0xf7520ec8) at /home/grahams/local/dmd2/linux/bin/../../src/phobos/std/ variant.d:584 #5 0x080607ec in _D3std11concurrency10MessageBox130__T3getTDFS3std11concurrency3TidZvTDFC3bob6ActionZvTDFC3bob7NewFileZvTDFAyaZvTDFC3bob11UpdatedFileZvTDFC3bob11ScannedFileZvTDFbZvZ3getMFDFS3std11concurrency3TidZvDFC3bob6ActionZvDFC3bob7NewFileZvDFAyaZvDFC3bob11UpdatedFileZvDFC3bob11ScannedFileZvDFbZvZv9onUserMsgMFS3std11concurrency7MessageZb (this=0xf7521038, msg=...) at /home/grahams/local/dmd2/linux/bin/../../src/phobos/std/ concurrency.d:360 #6 0x08061213 in _D3std11concurrency10MessageBox130__T3getTDFS3std11concurrency3TidZvTDFC3bob6ActionZvTDFC3bob7NewFileZvTDFAyaZvTDFC3bob11UpdatedFileZvTDFC3bob11ScannedFileZvTDFbZvZ3getMFDFS3std11concurrency3TidZvDFC3bob6ActionZvDFC3bob7NewFileZvDFAyaZvDFC3bob11UpdatedFileZvDFC3bob11ScannedFileZvDFbZvZv4scanMFKS3std11concurrency36__T4ListTS3std11concurrency7MessageZ4ListZb (this=0xf7521038, list=0xf7d26dc8) at /home/grahams/local/dmd2/linux/bin/../../src/phobos/std/ concurrency.d:449 #7 0x08060555 in _D3std11concurrency10MessageBox130__T3getTDFS3std11concurrency3TidZvTDFC3bob6ActionZvTDFC3bob7NewFileZvTDFAyaZvTDFC3bob11UpdatedFileZvTDFC3bob11ScannedFileZvTDFbZvZ3getMFDFS3std11concurrency3TidZvDFC3bob6ActionZvDFC3bob7NewFileZvDFAyaZvDFC3bob11UpdatedFileZvDFC3bob11ScannedFileZvDFbZvZv (this=0xf7d26dc0, _param_6=57795602383664, _param_5=577955439440031408, _param_4=577954855324479152, _param_3=577954408647880368, _param_2=577953807352458928, _param_1=577952742200569520, _param_0=577952226804494000) at /home/ grahams/local/dmd2/linux/bin/../../src/phobos/std/concurrency.d:463 #8 0x080603f0 in _D3std11concurrency134__T7receiveTDFS3std11concurrency3TidZvTDFC3bob6ActionZvTDFC3bob7NewFileZvTDFAyaZvTDFC3bob11UpdatedFileZvTDFC3bob11ScannedFileZvTDFbZvZ7receiveFDFS3std11concurrency3TidZvDFC3bob6ActionZvDFC3bob7NewFileZvDFAyaZvDFC3bob11UpdatedFileZvDFC3bob11ScannedFileZvDFbZvZv (_param_6=57795602383664, _param_5=577955439440031408, _param_4=577954855324479152, _param_3=577954408647880368, _param_2=577953807352458928, _param_1=577952742200569520, _param_0=577952226804494000) at /home/grahams/local/dmd2/linux/bin/../../ src/phobos/std/concurrency.d:228 #9 0x08054c18 in _D3bob13do_schedulingFbS3std11concurrency3TidZv (done_tid=..., print_deps=false) at /home/grahams/source/squid/open/ bedrock/build-tool/util/bob.d:2127 #10 0x08067bfc in _D3std11concurrency35__T5spawnTbTS3std11concurrency3TidZ5spawnFPFbS3std11concurrency3TidZvbS3std11concurrency3TidZS3std11concurrency3Tid4execMFZv (this=0xf7d27b80) at /home/grahams/local/dmd2/linux/bin/../../src/phobos/std/ concurrency.d:154 #11 0x0806c00e in _D4core6thread6Thread3runMFZv () #12 0x0807df2a in thread_entryPoint () #13 0xf7faa96e in start_thread (arg=0xf7521b70) at pthread_create.c:300 #14 0xf7ef3b5e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130
Re: dcollections 1.0 and 2.0a beta released
While I haven't read dcollections yet, I definitely agree with you about not liking container hierarchies, and about the importance of support for ranges. I hope Steven can be convinced that this is a good way to go :-).
Re: Concurrency architecture for D2
On Sun, 27 Dec 2009 14:32:52 -0600, Andrei Alexandrescu wrote: I think we are now in the position of defining a solid set of concurrency primitives for D. This follows many months of mulling over models and options. It would be great to open the participation to the design as broadly as possible, but I think it's realistic to say we won't be able to get things done on the newsgroup. When we discuss a topic around here, there's plenty of good ideas but also the inevitable bikeshed discussions, explanations being asked, explanations being given, and other sources of noise. We simply don't have the time to deal with all that - the time is short and we only have one shot at this. That's why I'm thinking of creating a mailing list or maybe another group for this. Any ideas on what would be the best approach? I also want to gauge interest from threading experts who'd like to participate. Please advise: (a) whether you would like to participate to the design; (b) keep discussions on the general group; (c) create a separate newsgroup; (d) create a mailing list. The latter would have open enrollment. Andrei I would love to be involved too.
Re: TDPL goes out for preliminary review
On Thu, 17 Dec 2009 03:04:59 +, Graham St Jack wrote: On Thu, 17 Dec 2009 01:13:36 +0100, grauzone wrote: Andrei Alexandrescu wrote: But let's not forget we have concurrency ahead of us. I encourage you all to chime in with your thoughts and ideas regarding all aspects of concurrency. The recent multicore performance bug is a great starting point. If you try e.g. shared and it's broken, let us know. If you try it and it works, push it til it breaks. If you have ideas on how to make semantic checking better, pipe up. There's a guy on the NG who posts once in a while how broken shared is and how he has to use __gshared instead. His threads are being pretty much ignored. And now? Andrei I'm certainly in that category. I will be trying (soon) to put a post together that sets out my issues with 'shared'. Here is my attempt to set out my problems with the shared keyword as it now stands. First the good part: writing multi-threaded programs is not easy, and anything the language and compiler can do to make it easier is good. Now for the way I approach writing multi-threaded programs: Keep threads apart from each other wherever possible. Specifically, don't let them access or modify the same data except in well-understood and carefully controlled ways. My preferred way of doing this is for threads to share only a very small amount of data, all of which is mutex protected, and carefully designed to be safe to access by multiple threads. My favourite kind of these is a templated queue (like a go channel). Any data passed out from such a shareable object has to be either immutable or cloned, so that each thread can rely on its data not being trampled on by other threads. And my wish-list: What I would like is for D to provide a clean way for me to be able to say this object and all its methods are nice and safe to access from multiple threads, and to also say all instances of this type are immutable. And I also want the compiler to tell me if it thinks I have multiple threads accessing data in an unsafe way. Finally my issues with D as it stands: Immutable or const objects are a real pain because I can't have a mutable reference to them. Rebindable doesn't seem to work, and I haven't been able to make a version that works well enough. Immutable types don't add any value because you have to keep stating that objects of them are immutable everywhere. Having first-class immutable types would make it MUCH easier to reap the benefits of immutable data. I can't currently see a use for the shared keyword as it stands. It seems to me that what is needed is a keyword more like shareable, meaning that this object (or data or function?) can be safely accessed by multiple threads. It should be an error to access a non-shareable object with multiple threads. It should also be an error to claim that something is shareable unless it meets some well thought out criteria that the compiler can check. I haven't thought these out yet, but some candidates I like the look of are: * All outputs from the object must be immutable or passed by value (cloned). * All externally accessible methods must be synchronized on the object's monitor. I'm happy that some sort of back-door override like __gshared has to be there for those cases that are actually ok, but only because of some grubby detail that the compiler can't figure out.
Re: TDPL goes out for preliminary review
On Thu, 17 Dec 2009 01:13:36 +0100, grauzone wrote: Andrei Alexandrescu wrote: But let's not forget we have concurrency ahead of us. I encourage you all to chime in with your thoughts and ideas regarding all aspects of concurrency. The recent multicore performance bug is a great starting point. If you try e.g. shared and it's broken, let us know. If you try it and it works, push it til it breaks. If you have ideas on how to make semantic checking better, pipe up. There's a guy on the NG who posts once in a while how broken shared is and how he has to use __gshared instead. His threads are being pretty much ignored. And now? Andrei I'm certainly in that category. I will be trying (soon) to put a post together that sets out my issues with 'shared'.
Re: TDPL goes out for preliminary review
On Thu, 17 Dec 2009 00:51:48 +, dsimcha wrote: == Quote from Andrei Alexandrescu (seewebsiteforem...@erdani.org)'s article But let's not forget we have concurrency ahead of us. I encourage you all to chime in with your thoughts and ideas regarding all aspects of concurrency. The recent multicore performance bug is a great starting point. If you try e.g. shared and it's broken, let us know. If you try it and it works, push it til it breaks. If you have ideas on how to make semantic checking better, pipe up. Andrei I think for this to happen, there needs to be a tutorial somewhere explaining what shared is supposed to do (there were so many ideas thrown around that I don't remember them all and don't know which ones got adopted), and how much of it is already implemented. I agree. The whole thing is very confused right now, and certainly everything I try in my code doesn't work out. The only way I can use threads tight now is to avoid use of shared at all. If anyone knows how it is supposed to work now, please write out a description.
Re: Various shared bugs
On Mon, 07 Dec 2009 22:45:17 -0500, Jason House wrote: So, after months of avoiding shared, I decided to see if I could remove all my casting away of shared. It looks like things are still pretty buggy (or at least not particularly easy to use). is(T : shared) gives a parse error alias shared T U silently does the wrong thing. (Use alias shared(T) U instead) is not callable using argument types () is just awful. Typically, it means you're calling a non-shared function with a shared type or a shared function with a non-shared type. I hit into a case where I got that message with a shared type and a shared method, but I have mad no attempt to reproduce. The error Can not implicitly convert expression of type(this) of type shared(xxx) to full.name.xxx can pop up when defining shared this() constructor gives no line number where the offending usage occurs. I've backed out most of my pro-shared changes and will try again in a few months :( I have also given up on shared and am also adopting a waiting strategy. I would love to get some tips from anyone (like Walter, for example) who thinks they have a way of using shared successfully. My recent post on the subject got no meaningful responses - just one from bearophile that was supportive, but alas not helpful.
Ongoing problems with shared and immutable
I have been dabbling with the shared and immutable keywords in an effort
to find out if they are usable, and aren't getting anywhere. My current
code base avoids them like the plague, but I can see the advantages of
them if only they were usable.
The previous newsgroup threads on this issue have all petered out without
any resolution.
Here is a small program that shows the sort of thing I am trying to do.
There is nothing tricky here, just passing of immutable data via a shared
queue between what could be two threads.
import std.typecons;
// immutable message
immutable class Message {
public int data;
this(int value) {
data = value;
}
}
// Queue-like shared synchronized container.
// The Condition on remove() is missing for brevity.
shared class Queue {
private Rebindable!(Message) mMsg;
this() {
mMsg = null;
}
synchronized void add(Message msg) {
mMsg = msg;
}
synchronized Message remove() {
return mMsg.get;
}
}
int main(string args[]) {
// pretend to be one thread queuing an object
auto queue = new Queue();
auto tx_message = new Message(1);
queue.add(tx_message);
// pretend to be another thread taking the data.
auto rx_message = queue.remove;
return rx_message.data;
}
Here are the error messages when I compile it with dmd 2.036:
Error: cannot implicitly convert expression (this) of type immutable
(Message) to shared_problem.Message
shared_problem.d(22): Error: function std.typecons.Rebindable!(immutable
(Message)).Rebindable.opAssign (immutable(Message) another) is not
callable using argument types (void*) shared
Error: cannot implicitly convert expression (this) of type shared(Queue)
to shared_problem.Queue
shared_problem.d(26): Error: function std.typecons.Rebindable!(immutable
(Message)).Rebindable.opAssign (immutable(Message) another) is not
callable using argument types (immutable(Message)) shared
shared_problem.d(29): Error: struct std.typecons.Rebindable!(immutable
(Message)).Rebindable member original is not accessible
shared_problem.d(29): Error: struct std.typecons.Rebindable!(immutable
(Message)).Rebindable member original is not accessible
Apart from annoyances like not getting a line number on some of the
errors, it looks like:
immutable types (at least for classes) aren't usable.
shared seems to be applied to immutable variables when it shouldn't,
confusing the type system.
shared types don't seem to be usable.
Rebindable seems to be broken.
Note that I'm trying to use immutable and shared types in the hope that I
can then avoid having to declare the instances of them as being immutable
or shared everywhere. In any case, they are in the documentation, so they
should work ;-).
Does anyone have an approach that works without bypassing shared and
immutable completely?
Re: Condition Mutexes
On Wed, 21 Oct 2009 00:56:13 +, dsimcha wrote:
I'm messing around w/ core.sync. Does anyone know what I'm doing wrong
in this program? It just hangs. If I could figure out ()##$) condition
mutexes (right now, I'm using busy spinning), I might have a decent
implementation of parallelForeach over ranges.
import core.sync.mutex, core.sync.condition, core.thread, std.stdio;
__gshared Condition condition;
void waitThenPrint() {
condition.wait();
writeln(FOO);
}
void main() {
condition = new Condition( new Mutex() ); auto T = new
Thread(waitThenPrint);
T.start();
condition.notify(); // Never wakes up and prints FOO.
}
There are a few problems. The most serious is that you have to lock the
mutex before calling condition.wait(). The underlying operating-system
stuff atomically
This means that the mutex needs to be an attribute of the class, and
waitThenPrint() should be more like this:
void waitThenPrint() {
synchronized(myMutex) {
condition.wait();
}
writeln(FOO);
}
While it isn't strictly necessary in this case, you should also:
Put the condition.notify() call into a synchronized(myMutex) block.
When some state variables are involved in the condition, you should do
something like this:
void waitThenPrint() {
synchronized(myMutex) {
while (state_not_right()) {
condition.wait();
}
}
writeln(FOO);
}
and
synchronized(myMutex) {
set_state_to_right();
condition.notify();
}
Re: Phobos.testing
This discussion is great news. I will happily contribute to Phobos if the barriers are lowered enough. It would be worthwhile posting something on the announce newsgroup when you have some sort of improved contribution procedure worked out. Also, I would be happier with mercurial or git than with subversion.
Re: It's official: One-day D tutorial at the ACCU Conference 2010 in Oxford, England
I would love to get my hands on the transcript and video of the event... Will it be recorded?
Re: It's official: One-day D tutorial at the ACCU Conference 2010 in Oxford, England
I would love to get my hands on the transcript and video of the event... Will it be recorded?
Re: shared adventures in the realm of thread-safety.
On Wed, 16 Sep 2009 08:00:40 -0400, Jason House wrote: Graham St Jack Wrote: So, what is the design of shared supposed to be then? Its time for Walter to buy in and tell us where this is all going - I for one am very confused right now. Thanks for that. Its good to know that there is a plan in there somewhere, even if the details are still very fuzzy. I agree that the lofty goal of improving thread-safety for mere mortals is worthwhile, and that it won't be easy to pull off. What I was really after though is what the plan is for D2 right now. The whole shared situation in D2 looks like a mess to me, and I would like some reassurance that something simple and tidy will be happening soon. Here's what I know: Bartosz's ownership scheme is delayed until at least D3 Shared code will be sequentially consistent Walter likes the idea of optimizing away memory barriers that the compiler can prove are unneeded (some barriers in synchronized sections) Bartosz is rewriting how threads are done similar to what his blogs hint at Issues that Bartosz hits with shared are fixed immediately Here's what I suspect from a number of emails: Because every class contains a monitor, Walter/dmd will treat every class as its own monitor for the purposes of optimization. I too wish Walter would advertise the design, but I think the simple fact is that he doesn't know what the design is!
Re: shared adventures in the realm of thread-safety.
So, what is the design of shared supposed to be then? Its time for Walter to buy in and tell us where this is all going - I for one am very confused right now. Currently I am working around it by not using synchronized methods (I put synchronized blocks inside the methods), which is very bad form, but what else can I do?
Re: shared adventures in the realm of thread-safety.
I'm also having the same problems.
As Jeremie said, as soon as you start introducing shared methods (via
synchronized for example), you rapidly get into trouble that can only be
overcome by excessive casting.
It may be possible to contain the problem by refactoring multi-threaded
code so that the shared objects are very small and simple, but even then
the casting required is too much. This approach might be ok if you could
define classes as being shared or immutable, and ALL instance of them
were then implicitly shared or immutable. Also, immutable objects should
be implicitly shareable.
On Sat, 12 Sep 2009 15:32:05 -0400, Jason House wrote:
I'm glad to see I'm not the only one trying to use shared. I tried to
use it with 2.031 and rapidly hit bug after bug... I submitted several
bug reports for basic functionality, and none of it appeared in the
changelog.
http://d.puremagic.com/issues/show_bug.cgi?id=3089
http://d.puremagic.com/issues/show_bug.cgi?id=3090
http://d.puremagic.com/issues/show_bug.cgi?id=3091
Jeremie Pelletier Wrote:
I decided to play once again with shared and see what 2.032 is capable
of. Turns out a lot of the previous issues I was having last time are
gone, however, there are still a few things left which prevent me from
rewriting my code.
The first issue that jumped to my face straight away was how 'shared
const' methods are not callable from 'shared' objects.
shared class Foo {
void bar() const;
}
auto foo = new Foo; // foo is of type shared(Foo) foo.bar; // Error:
function Foo.bar () shared const is not callable using argument types
() shared
Considering how 'const' methods can be called from mutable objects,
this looks like either a bug or a really awkward feature to me. Sending
a shared(Foo) to a method expecting a shared(const(Foo)) also triggers
a similar error from the compiler.
The other issue may be an intended feature, but it doesn't sound
practical to me. Marking a method as shared assumes all used properties
in the method's scope are also shared. Here is an example to illustrate
my point:
class SimpleReader {
this(LocalFile file) { _stream = new FileInputStream(file); } ...
private:
synchronized void read(ubyte[] buf, long offset) {
_stream.seek(offset);
_stream.read(buf);
}
FileInputStream _stream;
}
The FileInputStream here is a generic blocking binary stream which is
not thread-safe by design. The reader is a composite class where every
instance has its own unique stream instance and use it to implement
asynchronous reads over the file format it abstracts, which in my case
is a specialized read-only archive using a lot of random accesses from
different threads.
This is where the issue shows its ugly head. The 'synchronized' keyword
tags the read method as shared, which in itself is quite neat, what is
annoying however is that it also changes the type of _stream in the
method's scope to shared(FileInputStream) and therefore triggers
compiler errors because _stream.seek and _stream.read are not shared:
Error: function FileInputStream.read (ubyte[]) is not callable using
argument types (ubyte[]) shared
While it may be an attempt to keep shared usage safe, it isn't very
practical. The stream object here is not shared because it is not
thread-safe. While it may be used by different threads, it is unique to
the reader's context and its accesses are synchronized by the reader,
the stream should therefore be completely oblivious to the fact it is
being used by different threads.
Maybe this could be the time to implement an unique qualifier; this is
a context where having _stream be of type unique(FileInputStream) would
solve the problem and allow further compiler optimizations. I don't
know if it can be done with templates, and without any overhead
whatsoever. I know I would much rather see unique(Foo) than Unique!Foo,
and it would allow the use of 'is(foo : unique)'.
Furthermore, tagging a method with shared does not make it thread-safe,
it may however use synchronized within its scope to protect its shared
or unique data. This may be confusing when calling shared methods vs
calling synchronized methods; one may think the shared one is not
thread-safe and optionally synchronize the call, resulting in another
monitor being used for nothing, or no monitor being used at all:
class Foo {
shared void bar() {
// Do stuff with local or immutable data synchronized(this) {
/* do stuff with shared data */ }
}
shared void bar2() {
// Do stuff on shared data
}
}
Someone seeing only the prototype of Foo.bar may assume the method is
not thread-safe and call it as 'synchronized(foo) foo.bar()'. Just like
they could see the prototype of bar2 and assume it is thread-safe,
calling it as 'foo.bar2()'.
What could be a good design against this sort of misleading behavior?
Phew, that's about
Re: dmd 1.047 and 2.032 releases
Its great to see so many bugs being sorted out.
However, I am having all kinds of trouble with shared, which up until
now I have been able to fairly easily sidestep. Here is a cut-down
example of what I am trying to do, which is to have one thread acquiring
data and passing it on to another thread via a queue which uses
appropriate synchronization mechanisms to make things thread-safe. The
data passed through is immutable, so theoretically everything should be
fine.
This kind of thing is very routine in multi-threaded programs, and should
be easy to pull off. I find I have to do heaps of nasty casting to get
past the compiler (the immutable assignment thing is a side issue).
Is there a neater syntax for this? How about being able to say shared
immutable class Message {...} (or just shared or just immutable or just
const), and then all instances (new, local, parameter, return) would
automatically be shared and immutable?
class Message {
int data_;
this(int value) {
data_ = value;
}
int get() immutable {
return data_;
}
}
// queue-like synchronized container
class Queue {
immutable(Message) msg_;
this() {
msg_ = null;
}
synchronized void add(immutable(Message) msg) {
Message * tmp = cast(Message *) msg_;
*tmp = cast(Message) msg;
}
synchronized immutable(Message) remove() {
return msg_;
}
}
int main(string args[]) {
// pretend to be one thread queueing an object
auto queue = cast(shared) new Queue();
auto message = cast(immutable) cast(shared) new Message(1);
queue.add(message);
// pretend to be another thread taking the data and working with it
return queue.remove.get;
}
Re: const and immutable objects
On Mon, 31 Aug 2009 15:20:21 -0500, Andrei Alexandrescu wrote:
Graham St Jack wrote:
I have been trying (again) to start using const and immutable objects
in a project, and found I kept getting stuck with the need to change
the value of object references, which isn't allowed. I don't quite
follow the arguments that mean that this has to be the case, but I'm
prepared to accept them.
After some experimenting with Andrei's Rebindable template, I still
couldn't get past the problems. For a start, the get method is private
so I couldn't test for null.
In the end I cooked up the following, which is a very simple workaround
that just lets me assign to a const or immutable object reference. It
works just fine, and can be used in templates like containers without
forcing the container to care about what it is containing.
Comments?
[snip code]
Hi Graham,
Could you please post a short snippet (or a few) that illustrate the
problems you ran into with Rebindable?
Thanks,
Andrei
The code fragments are lost in the sands of time - I can recreate them if
necessary, but here is an outline of what the problems were:
Rebindable!(Foo) foo = null;
foo = cast(immutable) new Foo(); // ok so far
foo = Foo.init; // still ok
if (foo !is null) {} // compiler error
if (foo.get !is null) {} // another compiler error, and nasty!
The main thing that has to change in Rebindable is that it needs to be a
transparent wrapper, so that if (foo !is null) {} works.
What I was doing to get into trouble was writing some communications code
using Fibers, and I wanted the message objects to be immutable (call me
silly, but I wanted to give it a go). This meant I needed to:
* Have a templated queue class that could contain any sort of mutable
types, and const or immutable objects. In particular, I didn't want the
queue to have to use different code for objects and (say) integers.
* Pass messages between the Fibre's stack frame and the main thread, so
for an immutable object reference I needed to: test for null, assign to
null and assign to a new message.
The motivation for passing immutable messages around is that there is an
implicit assumption that these messages don't mutate as they fan out to
various destinations, and I wanted some compiler enforcement.
Re: const and immutable objects
On Mon, 31 Aug 2009 09:12:33 -0400, Steven Schveighoffer wrote:
On Sun, 30 Aug 2009 18:31:33 -0400, Graham St Jack
graham.stj...@internode.on.net wrote:
I have been trying (again) to start using const and immutable objects
in a project, and found I kept getting stuck with the need to change
the value of object references, which isn't allowed. I don't quite
follow the arguments that mean that this has to be the case, but I'm
prepared to accept them.
After some experimenting with Andrei's Rebindable template, I still
couldn't get past the problems. For a start, the get method is private
so I couldn't test for null.
In the end I cooked up the following, which is a very simple workaround
that just lets me assign to a const or immutable object reference. It
works just fine, and can be used in templates like containers without
forcing the container to care about what it is containing.
Comments?
First, a little history lesson. An earlier version of const had this
distinction:
class C {}
const(C) c; // c is rebindable, what it points to is const
const C c2; // c2 is not rebindable.
The const(C) format was modeled after pointers and arrays, i.e.
const(C)[] and const(C)*, but since the reference is not explicitly
stated, it's invisible in the const format also :)
So that was thrown out because it was determined that const(...) should
mean that anything inside the parentheses should be const (including a
class reference). What followed was several attempts (including some by
me) to come up with a way to denote a tail-const class reference. Most
of them centered around pulling out the reference part, i.e. ref
const(C) or const(*C)*. Others included new keywords. In the end, none
of them looked great, and none of them made Walter change his mind. So
via syntax, there is no way to say rebindable reference to const class
data. So to answer your first question, there's no objection by Walter
and crew as to being able to rebind a const class reference, there's
just no good syntax to denote it (that has been presented so far
anyways).
Then Andrei came along with Rebindable, and the argument died down. I
had a feeling that Rebindable would be somewhat unusable in its current
form, but given that we were about to get opImplicitCast and other
niceties, it seemed Rebindable would be sufficient in the future.
OK, so now we have alias this which is supposed to be the implementation
of opImplicitCast, and you say it's still too difficult. Looking at the
source, it looks like Rebindable needs some attention, as it still
doesn't use alias this. I have no idea if there is a way to implement
!is null, but I assume that it should be forwarded to the alias this
member.
So I think a bug report is in order. Rebindable should not use opDot,
but rather alias this. Implicit casting is paramount to make Rebindable
look like a transparent wrapper.
-Steve
I remember the history - it was long and complicated, with no answer that
looked good. I also agree that Rebindable as it stands needs work, and
definitely needs to be a transparent wrapper. Since I don't understand
the issues all that well, how about you submit the bug report?
const and immutable objects
I have been trying (again) to start using const and immutable objects in
a project, and found I kept getting stuck with the need to change the
value of object references, which isn't allowed. I don't quite follow the
arguments that mean that this has to be the case, but I'm prepared to
accept them.
After some experimenting with Andrei's Rebindable template, I still
couldn't get past the problems. For a start, the get method is private so
I couldn't test for null.
In the end I cooked up the following, which is a very simple workaround
that just lets me assign to a const or immutable object reference. It
works just fine, and can be used in templates like containers without
forcing the container to care about what it is containing.
Comments?
//
// Assign dest to src, using brute-force methods as necessary if T
// is an immutable or const object.
//
// The intent is to avoid needing routine nasty tricks in
// application code, especially in template code which should be
// spared the complication of handling this very annoying
// restriction on objects. Use of this template function doesn't
// let you change the object itself, just the reference to it - so
// all should be well.
//
void forcefulAssign(T)(ref T dest, T src) {
static if (!is(T X == const(U), U) !is(T X == invariant(U), U)) {
// T is not const or immutable - just assign normally
dest = src;
}
else static if (is(T : Object)) {
// T is a const or immutable object - use brute force
U * ptr = cast(U *) dest;
*ptr = cast(U) src;
}
else {
// T is some other const or immutable type - not assignable
static assert(false, cannot assign to ~ T.stringof);
}
}
// assign the default initializer to something
void forcefulInitialize(T)(ref T dest) {
static if (!is(T X == const(U), U) !is(T X == invariant(U), U)) {
// T is not const or immutable - just initialise normally
dest = T.init;
}
else static if (is(T : Object)) {
// T is a const or immutable object - use brute force
U * ptr = cast(U *) dest;
*ptr = U.init;
}
else {
// T is some other const or immutable type - not assignable
static assert(false, cannot assign to ~ T.stringof);
}
}
Re: const and immutable objects
On Sun, 30 Aug 2009 21:28:18 -0400, Jeremie Pelletier wrote:
I agree that D lacks a mechanism to separate the object from it's
reference. Maybe syntax like the following could be used to apply the
storage class to the object value, and not the reference value:
class Foo;
void bar(in Foo foo) {}
It's quite ugly and C-like, but that's the first thing that came to
mind. The reference value is unique to the current method and shouldn't
share the same storage qualifiers as it's referenced memory.
I think the time for pining over this particular syntax feature of D is
over - as nice as it would be to be able to fix the problem in the
language, it would be too disruptive right now. What we need is something
in phobos that works around the problem, and Rebindable isn't suitable as
it stands because:
* It presents different interfaces for different kinds of wrapped types.
* It doesn't let you access the wrapped object except via opDot.
My suggested workaround is perhaps a bit rough, but it does work. I would
like to hear from anyone who is using const and immutable objects in D2
who has something better, or has suggestions to improve my code.
Re: T[new]
This sounds excellent.
Re: project oriented
On Tue, 12 May 2009 21:12:51 +, BCS wrote: Hello davidl, The module package system still stays in the state of the C age. It's file oriented. I think there's no more sound package system than C# one. The namespace and distributed packaging is a must nowadays and the compiler should be project oriented and take project information as the compiling base. Also an IDE is quite useful for providing project templates. The up side to file based packaging is that the compiler can find the files without needing extra information. There are several tools that can build a project from nothing but a set of .d files. With the c# type of system, the compiler/build system needs to have a metadata file that list all the .d files to be built adding yet another piece of redundant complexity. The c# solution works well if you will *only* develop from the IDE but is a total pain as soon as you need to work with non-language aware tools. Good point. I like the current system's simplicity, and changing it as suggested would add a lot of hassle.
Re: When will D1 be finished?
On Mon, 11 May 2009 12:11:53 -0700, Walter Bright wrote: Luís Marques wrote: When D1 was declared finished I thought it meant it would progress to a stable state, with nearly all non-minor problems fixed and a large set of companion libraries. I'm afraid I don't see that happening at an animating rate. D1 regularly gets around 20 bug fixes a month. I don't understand why this is not seen as progress to a stable state. About 80% of bug fixes are common to both D2 and D1. I agree that D1 is getting its fair share of attention, and I support the earlier post that suggested getting in there and making contributions yourself if you don't like the rate of progress in your personal area of interest. D (especially D2 ;-) ) is far and away the best language I have ever worked with already, and I think the team needs to be congratulated for its efforts. I'm embarrassed that I can't contribute more because of other commitments, but when I do, it is done as constructively as possible.
Re: Weird std.stdio threading bug?
I posted a bug report for this a few days ago (2890). I got as far as
finding out that it is a file locking problem caused by what looks like a
compiler bug re calling the destructor of a struct. The following patch
to std.stdio works around the problem, but is hardly a fix.
$ diff dmd/src/phobos/std/stdio.d stdio.d
922c922
//return LockingTextWriter(this);
---
return LockingTextWriter(this);
925,926c925,926
auto result = LockingTextWriter(this);
return result;
---
//auto result = LockingTextWriter(this);
//return result;
On Mon, 27 Apr 2009 22:53:04 +, dsimcha wrote:
The following small test program seems to have a weird deadlock or
something:
It should keep printing the phrase Doing stuff. forever, but it only
gets
through maybe two iterations before its CPU usage does to zero and it
stops printing, at least on my computer. Has anyone noticed any bad
behavior with std.stdio and multithreading?
import core.thread, std.stdio;
void main() {
Thread[] myThreads;
foreach(i; 0..4) {
myThreads ~= new Thread( { doStuff(); }); myThreads[$ -
1].start;
}
}
void doStuff() {
while(true) {
synchronized {
writeln(Doing stuff.);
}
}
}
If the writeln line is commented out, this thing keeps executing the
empty loop with measurable CPU usage.
Re: Bug in std.socket
On Wed, 15 Apr 2009 00:38:33 -0700, Unknown W. Brackets wrote: Does waiting for a keypress in the client thread do you any favors? Also, you can get some mileage by doing a select() (see SocketSet) on the socket first. This will tell you if you need to accept(), and also allow you to do timeouts. I'm a big fan of non-blocking sockets, you can improve connect() this way too. -[Unknown] Thanks for all the useful suggestions. I'm on the ticket bandwagon at last. I realise that the code I posted was a bit rough (although I missed the race, which is definitely there). It was a heavily cut-down version of something a lot larger that does indeed use select, uses non-blocking I/ O, retries connection attempts, and even uses Fibers. What I wanted was a small and simple test case to attach to the ticket, not something serious.
Re: Bug in std.socket
On Wed, 15 Apr 2009 09:35:45 -0700, Brad Roberts wrote: Graham St Jack wrote: On Wed, 15 Apr 2009 00:38:33 -0700, Unknown W. Brackets wrote: Does waiting for a keypress in the client thread do you any favors? Also, you can get some mileage by doing a select() (see SocketSet) on the socket first. This will tell you if you need to accept(), and also allow you to do timeouts. I'm a big fan of non-blocking sockets, you can improve connect() this way too. -[Unknown] Thanks for all the useful suggestions. I'm on the ticket bandwagon at last. I realise that the code I posted was a bit rough (although I missed the race, which is definitely there). It was a heavily cut-down version of something a lot larger that does indeed use select, uses non-blocking I/ O, retries connection attempts, and even uses Fibers. What I wanted was a small and simple test case to attach to the ticket, not something serious. I was about to close the ticket as invalid due to buggy reproduction code. Can you demonstrate the problem with an example app that's _not_ buggy? If so, please attach it to the bug report. Later, Brad Someone else already provided a much simpler test case without bugs.
Bug in std.socket
Forgive my ignorance, but can anyone tell me how to post a ticket for
phobos?
I want to report a bug in std.socket's TcpSocket.accept() (at least it
manifests itself there for me) introduced in D 2.0.27 and still there in
2.0.28. The following code runs to completion in 2.026 and blocks on
accept in later compiler versions.
import std.socket;
import std.stdio;
import core.thread;
class Server {
private {
InternetAddress mAddress;
Thread mThread;
}
this(InternetAddress address) {
mAddress = address;
mThread = new Thread(run);
mThread.name = server.dup;
mThread.start;
}
void join() {
mThread.join;
}
void run() {
try {
writefln(server - setting up server socket);
auto listener = new TcpSocket();
listener.bind(mAddress);
listener.listen(5);
// wait for a connection
writefln(server - waiting for a client connection);
auto socket = listener.accept();
writefln(server - got a client connection);
// read some data and write it back
writefln(server - reading data from the client);
ubyte[100] data;
int qty = socket.receive(data);
writefln(server - writing the data back to the client);
socket.send(data[0..qty]);
}
catch (Exception ex) {
writefln(server - server got exception: %s, ex);
}
// terminate
writefln(server - terminating);
}
}
class Client {
private {
InternetAddress mAddress;
Thread mThread;
}
this(InternetAddress address) {
mAddress = address;
mThread = new Thread(run);
mThread.name = client.dup;
mThread.start;
}
void join() {
mThread.join;
}
void run() {
try {
// connect
writefln(client - connecting to server);
auto socket = new TcpSocket(mAddress);
writefln(client - connected to server);
// send and receive some data
writefln(client - sending data to server);
ubyte[3] send_data;
send_data[0] = 41;
send_data[1] = 42;
send_data[2] = 43;
int qty = socket.send(send_data);
assert(qty == send_data.length);
writefln(client - receiving data from server);
ubyte[100] receive_data;
qty = socket.receive(receive_data);
writefln(client - got %s bytes from server, qty);
}
catch (Exception ex) {
writefln(client - client got exception %s, ex);
}
// terminate
writefln(client - terminating);
}
}
int main(string[] args) {
writefln(test starting);
try {
InternetAddress address = new InternetAddress(localhost, 12345);
Server server = new Server(address);
Client client = new Client(address);
writefln(joining with server);
server.join;
writefln(joining with client);
client.join;
writefln(finished);
}
catch (Exception ex) {
writefln(unexpected exception %s, ex);
}
return 0;
}
Re: Proposal: adding condition variable to object monitors
class C
{
Mutex m;
Condition c;
this()
{
// make m this object's monitor
m = new Mutex( this );
c = new Condition( m );
}
synchronized void foo()
{
// m is locked
c.notify();
}
}
I like this approach, and agree that it is very useful to be able to have
more than one condition share the same mutex.
I don't mind the handraulic creation of the mutex and condition when you
are using a condition, because of the added flexibility and the fact that
it doesn't come up much.
Being able to use the Object's monitor for the mutex is really good too.
Please release it soon!
Re: dmd platform support - poll
On Thu, 25 Dec 2008 15:56:29 -0800, Sean Kelly wrote: Walter Bright wrote: What platforms for dmd would you be most interested in using? In order of preference: mac osx 32 bit intel linux 64 bit mac osx 64 bit intel Ditto
Re: DMD 1.036 and 2.020 releases
On Mon, 20 Oct 2008 16:29:36 -0700, Walter Bright wrote: http://www.digitalmars.com/d/1.0/changelog.html http://ftp.digitalmars.com/dmd.1.036.zip The 2.0 version splits phobos into druntime and phobos libraries (thanks to Sean Kelly). This will enable both Tango and Phobos to share a common core library. http://www.digitalmars.com/d/2.0/changelog.html http://ftp.digitalmars.com/dmd.2.020.zip There are a lot of structural changes that go along with this, so expect some rough patches with this release. It may take a followup release to file them down. There's also some renaming of imports and function names, as a compromise with Tango names. This is FANTASTIC news. Many thanks to everyone involved, especially Sean for all the hard work.
