date:20110419

Lars T. Kyllingstad:

> Please vote in this thread, by replying with
> 
>   - "YES" if you think std.parallelism should be included in Phobos
> in its present form,
> 
>   - "NO" if you think it shouldn't.

If my vote counts then Yes.

Bye,
bearophile

Re: std.parallelism: VOTE IN THIS THREAD

2011-04-19 Thread Jonathan M Davis

> Lars T. Kyllingstad:
> > Please vote in this thread, by replying with
> > 
> >   - "YES" if you think std.parallelism should be included in Phobos
> >   
> > in its present form,
> >   
> >   - "NO" if you think it shouldn't.
> 
> If my vote counts then Yes.

Everyone's vote on the newsgroup counts unless it becomes evident that we have 
a sock puppet problem or a ton of people that never post here start voting or 
some other problem occurs where it's evident that there's a problem which is 
skewing the votes. As long as things continue to go smoothly with the voting 
and people don't abuse it, anyone on the newsgroup can vote.

- Jonathan M Davis

Re: std.parallelism: VOTE IN THIS THREAD

2011-04-19 Thread Caligo

On Tue, Apr 19, 2011 at 5:49 AM, Jonathan M Davis  wrote:
>> Lars T. Kyllingstad:
>> > Please vote in this thread, by replying with
>> >
>> >   - "YES" if you think std.parallelism should be included in Phobos
>> >
>> >     in its present form,
>> >
>> >   - "NO" if you think it shouldn't.
>>
>> If my vote counts then Yes.
>
> Everyone's vote on the newsgroup counts unless it becomes evident that we have
> a sock puppet problem or a ton of people that never post here start voting or
> some other problem occurs where it's evident that there's a problem which is
> skewing the votes. As long as things continue to go smoothly with the voting
> and people don't abuse it, anyone on the newsgroup can vote.
>
> - Jonathan M Davis
>

I would like to make a comment if that's okay.  If a person is not an
expert on parallelism, library development, or we can't verify his or
her background and such, I don't see why their vote should count.  I'm
not voting because I'm just an ordinary D user, and I have no
expertise in parallelism.  And since this a public vote, if would be
great if people who are voting did not hide behind an alias, such as
bearophile.

P.S.
I can't wait for std.parallelism to become part of Phobos.

Re: std.parallelism: VOTE IN THIS THREAD

He's not hiding behind an alias, he made numerous blog posts here:
http://leonardo-m.livejournal.com/

My vote is YES.

Re: std.parallelism: VOTE IN THIS THREAD

Caligo:

> I would like to make a comment if that's okay.  If a person is not an
> expert on parallelism, library development, or we can't verify his or
> her background and such, I don't see why their vote should count.  I'm
> not voting because I'm just an ordinary D user, and I have no
> expertise in parallelism.  And since this a public vote, if would be
> great if people who are voting did not hide behind an alias, such as
> bearophile.

You are right regarding parallelism experience, I don't know enough about this 
topic, this is why I was not sure about voting. Feel free to ignore my vote 
because of this.

Regarding library development experience, alias problems, etc, they aren't 
problems in this case.

Bye,
bearophile

Re: std.parallelism: VOTE IN THIS THREAD

2011-04-19 Thread Caligo

On Tue, Apr 19, 2011 at 7:11 AM, bearophile  wrote:
> Caligo:
>
>> I would like to make a comment if that's okay.  If a person is not an
>> expert on parallelism, library development, or we can't verify his or
>> her background and such, I don't see why their vote should count.  I'm
>> not voting because I'm just an ordinary D user, and I have no
>> expertise in parallelism.  And since this a public vote, if would be
>> great if people who are voting did not hide behind an alias, such as
>> bearophile.
>
> You are right regarding parallelism experience, I don't know enough about 
> this topic, this is why I was not sure about voting. Feel free to ignore my 
> vote because of this.
>
> Regarding library development experience, alias problems, etc, they aren't 
> problems in this case.
>
> Bye,
> bearophile
>


Sorry Leonardo, I didn't mean to pick on you.  It's just that I don't
believe in voting, and I really care about D.

Re: std.parallelism: VOTE IN THIS THREAD

2011-04-19 Thread Stephan

YES

TempAlloc: Request for Review For Inclusion in core.memory

My TempAlloc module's been ready for review for a while, but 
std.net.isemail and std.parallelism were ahead of it in the review 
queue.  Now that the reviews for both are done (I doubt running a review 
concurrently with a vote is problematic), I want to put this next in 
line.  I'd like to get it in druntime soon because Don wants to use it 
in BigInt and I also have some things I want to contribute to Phobos 
that TempAlloc would be useful for.


TempAlloc is a segmented stack for allocating and freeing temporary 
buffers efficiently.  It works by allocating memory in large chunks 
(currently 4 MB, but this number may be revised) at a time from the C 
heap, and parceling them out with the standard last in first out stack 
restriction.  The proposal also contains a few small helper functions 
that I think would be useful, but these can be made private or removed 
if people don't like them.


TempAlloc has the following advantages compared to allocation on the
call stack:

1. Pointers to memory allocated on the TempAlloc stack are still
valid when the function they were allocated from returns.
Functions can be written to create and return data structures on the
TempAlloc stack.

2. Since it is a segmented stack, large allocations can be performed 
with no danger of stack overflow errors.


It has the following advantages compared to heap allocation:

1. Both allocation and deallocation are extremely fast. Most allocations
consist of verifying enough space is available, incrementing a pointer 
and a performing a few cheap bookkeeping operations. Most deallocations

consist decrementing a pointer and performing a few cheap bookkeeping
operations.

2. The segmented stack is thread-local, so synchronization is only 
needed when a segment needs to be allocated or freed.


3. Fragmentation is not an issue when allocating memory on the
TempAlloc stack, though it can be an issue when trying to allocate
a new segment.

Code:

https://github.com/dsimcha/TempAlloc/blob/master/tempalloc.d

Docs:

http://cis.jhu.edu/~dsimcha/d/phobos/core_tempalloc.html

Re: Ben Gertzfield is our 11th mentor

2011-04-19 Thread Daniel Gibson

Am 19.04.2011 06:09, schrieb Eric Poggel (JoeCoder):
> On 4/18/2011 9:20 PM, Daniel Gibson wrote:
> 
>> What about successful languages like C(++) or Python? I think they were
>> and still are pretty neutral (even though at least C and C++ were
>> developed at a commercial company).
> 
> C++ is horrid but fast.  Python is beautiful but slow.  D is the best of
> both, or at least moving in that direction.

Yeah my point was that languages can be successful without being
political, disproving Jasons "No successful platform is politically
neutral."

> 
>> So no, it's not a worthy goal to become Facebooks (or anybody elses)
>> weapon (and nothing suggests they want that) and I personally would stop
>> using D if something like that happened.
> 
> I don't understand how Facebook using it "as a weapon" could impair the
> freedoms we already enjoy with D.  AFAIK, everything except the back-end
> of one of the three compilers is open source and redistributable.
> 

Look at how Oracle is using Java as a weapon..
And probably the development of D would be focused on their goals and
everything else would be neglected (like for example Oracle neglects
Java for game development). And even though the front end currently is
Open Source that could  be changed for future versions.

But this discussion is quite pointless, nothing suggests that Facebook
plans to use D or any other technology as a weapon or for vendor
lock-ins - in the contrary, they made some great Open Source
contributions in the past. And nothing suggests that Walter is
interested in something like this.

Cheers,
- Daniel

Re: std.parallelism: VOTE IN THIS THREAD

2011-04-19 Thread Russel Winder

On Tue, 2011-04-19 at 06:13 -0500, Caligo wrote:
[ . . . ]
> I would like to make a comment if that's okay.  If a person is not an
> expert on parallelism, library development, or we can't verify his or
> her background and such, I don't see why their vote should count.  I'm
> not voting because I'm just an ordinary D user, and I have no
> expertise in parallelism.  And since this a public vote, if would be
> great if people who are voting did not hide behind an alias, such as
> bearophile.

I think there is an very interesting and important issue here.  There
are really four (or more/less) roles of people who might vote:

Has detailed knowledge of implementation and usage.
Has some knowledge of implementation and/or usage.
Perhaps just using the API.
No actual interest. 

And then there are:

Sock puppet aka shill
Troll

but let's ignore them.

Although the general belief is "one person, one vote", some decisions
are best influenced by considering the weighting to a vote provided by
annotating with role.

Two cases perhaps highlight this:

If a person using the library but having no knowledge of the
implementation finds a problematic API issue then this should count as
much as any view of people more knowledgeable about the internals.

If a person without knowledge of the theory and practice votes yes where
the domain experts are able to argue there is a blocking problem, then
the person leading the vote should have the option of cancelling and
re-entering review even if there was a clear majority vote for.

I think the issue here is not to be bound too rigidly by votes and
statistics, this is not after all politics, but instead to ensure that
everyone has the right sort of say about these things and that the
majority of people always feel the right decisions are getting made.

Consider the system not being one of the review leader managing the
votes, but of the review leader having a single golden vote which then
has to be justified by argumentation.

> P.S.
> I can't wait for std.parallelism to become part of Phobos.

Me too.

-- 
Russel.
=
Dr Russel Winder  t: +44 20 7585 2200   voip: sip:russel.win...@ekiga.net
41 Buckmaster Roadm: +44 7770 465 077   xmpp: rus...@russel.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder

signature.asc
Description: This is a digitally signed message part

Re: std.parallelism: VOTE IN THIS THREAD

2011-04-19 Thread Michel Fortin


I say yes.

I want to mention that I am a left a little skeptical about the lack of 
low-level race safety in this module, but I understand that this is 
more a language- or compiler-level problem than one with the module 
itself. I hope these problem will be addressed eventually and that 
std.parallelism can become safe that day.


--
Michel Fortin
michel.for...@michelf.com
http://michelf.com/

Re: std.parallelism: VOTE IN THIS THREAD

== Quote from Michel Fortin (michel.for...@michelf.com)'s article
> I say yes.
> I want to mention that I am a left a little skeptical about the lack of
> low-level race safety in this module, but I understand that this is
> more a language- or compiler-level problem than one with the module
> itself. I hope these problem will be addressed eventually and that
> std.parallelism can become safe that day.

I **hope** so, too.  I'm just more skeptical than you that it can be done, even 
in
principle, without major sacrifices in terms of efficiency, expressiveness or
flexibility.

Re: std.parallelism: VOTE IN THIS THREAD

2011-04-19 Thread Jesse Phillips

Lars T. Kyllingstad Wrote:

> As announced a week ago, the formal review process for David Simcha's 
> std.parallelism module is now over, and it is time to vote over whether 
> the module should be included in Phobos.  See below for more information 
> on the module and on previous reviews.
> 
> Please vote in this thread, by replying with
> 
>   - "YES" if you think std.parallelism should be included in Phobos
> in its present form,
> 
>   - "NO" if you think it shouldn't.

Yes.

Re: std.parallelism: VOTE IN THIS THREAD

2011-04-19 Thread Steven Schveighoffer

On Tue, 19 Apr 2011 03:25:06 -0400, Lars T. Kyllingstad  
 wrote:



As announced a week ago, the formal review process for David Simcha's
std.parallelism module is now over, and it is time to vote over whether
the module should be included in Phobos.  See below for more information
on the module and on previous reviews.

Please vote in this thread, by replying with

  - "YES" if you think std.parallelism should be included in Phobos
in its present form,

  - "NO" if you think it shouldn't.



YES.

Note that I have no practical experience with the library or the concepts,  
but the docs look good enough and I trust David's abilities.


-Steve

Re: TempAlloc: Request for Review For Inclusion in core.memory

Nice work. I've tried it out with allocating some large float arrays,
quick two test cases were a 94MB and 282MB audio file.

File 1 w/o TempAlloc: 410msecs
File 1 w/ TempAlloc: 184msecs

File 2 w/o TempAlloc 1321 msecs
File 2 w/ TempAlloc 567 msecs

Btw, how much *can* I allocate at once by using new (not specific to
TempAlloc)? I've tried a 500MB file, but allocation fails even though
I have plenty of free memory on my system. Is there a limit to using
'new'?

Of course, ideally I'd use some kind of buffering mechanism where a
portion of a file is loaded into memory, and when that's exhausted I'd
load another chunk. But I'd like to know the limits for a single
memory allocation on the garbage-collected heap. Is there some
internal limit for D? I know I can't allocate more than 3GB in the
entire app, but I'm looking at single allocations.

Pardon me if this all sounds stupid, I'm a complete newb when it comes
to memory allocation mechanics. :)

Re: TempAlloc: Request for Review For Inclusion in core.memory

Oh, I should mention I was using a C library to actually load the data
into the buffer, so I didn't use -J imports or anything like that.

Re: TempAlloc: Request for Review For Inclusion in core.memory

== Quote from Andrej Mitrovic (andrej.mitrov...@gmail.com)'s article
> Nice work. I've tried it out with allocating some large float arrays,
> quick two test cases were a 94MB and 282MB audio file.
> File 1 w/o TempAlloc: 410msecs
> File 1 w/ TempAlloc: 184msecs
> File 2 w/o TempAlloc 1321 msecs
> File 2 w/ TempAlloc 567 msecs
> Btw, how much *can* I allocate at once by using new (not specific to
> TempAlloc)? I've tried a 500MB file, but allocation fails even though
> I have plenty of free memory on my system. Is there a limit to using
> 'new'?
> Of course, ideally I'd use some kind of buffering mechanism where a
> portion of a file is loaded into memory, and when that's exhausted I'd
> load another chunk. But I'd like to know the limits for a single
> memory allocation on the garbage-collected heap. Is there some
> internal limit for D? I know I can't allocate more than 3GB in the
> entire app, but I'm looking at single allocations.
> Pardon me if this all sounds stupid, I'm a complete newb when it comes
> to memory allocation mechanics. :)

Ironically, these benchmarks just show that the C heap is faster than the D GC
heap.  They are well over the 4 MB mark, where TempAlloc just falls back on the 
C
heap.  As far as how much you can allocate on the GC heap, it boils down to how
much **contiguous** free space you have in your address space.  Since 32-bit
address space is only 4 GB (and only 2 GB for user space on Win32), I'm not
surprised you can't allocate more than 500 MB.

Re: TempAlloc: Request for Review For Inclusion in core.memory

Ok, perhaps you should put a note in the docs about falling back to
the C heap. Or maybe that's just common sense for people in the
know-how?

I've realized the allocation attempt was actually quite a bit more
than 500Mb. The C lib returned the number of channels and frame count,
which totalled to a frame count of 291_335_184. Multiply that by the
float size (4), and you get: 1_165_340_736, or 1.65 GB.

The GC heap can actually allocate even 200_000_000 elements on my
system, so that's around 800MB if I'm right.

Re: TempAlloc: Request for Review For Inclusion in core.memory

== Quote from Andrej Mitrovic (andrej.mitrov...@gmail.com)'s article
> Ok, perhaps you should put a note in the docs about falling back to
> the C heap. Or maybe that's just common sense for people in the
> know-how?

Right.  This is roughly equivalent to allocating a new segment.  I might put 
more
details into the docs about the heuristics used, but I'm not sure how much I
should document them.  They're somewhat complicated, may change, and are
implementation details.  On the other hand, the whole point of TempAlloc is a
performance optimization, and knowing such details is useful for predicting
performance.  On the third hand, it's open source.  If you really want to know
**exactly** how it works instead of just how to use it, you can always read the 
code.

Re: TempAlloc: Request for Review For Inclusion in core.memory

2011-04-19 Thread Denis Koroskin


On Tue, 19 Apr 2011 18:34:33 +0400, dsimcha  wrote:


== Quote from Andrej Mitrovic (andrej.mitrov...@gmail.com)'s article

Nice work. I've tried it out with allocating some large float arrays,
quick two test cases were a 94MB and 282MB audio file.
File 1 w/o TempAlloc: 410msecs
File 1 w/ TempAlloc: 184msecs
File 2 w/o TempAlloc 1321 msecs
File 2 w/ TempAlloc 567 msecs
Btw, how much *can* I allocate at once by using new (not specific to
TempAlloc)? I've tried a 500MB file, but allocation fails even though
I have plenty of free memory on my system. Is there a limit to using
'new'?
Of course, ideally I'd use some kind of buffering mechanism where a
portion of a file is loaded into memory, and when that's exhausted I'd
load another chunk. But I'd like to know the limits for a single
memory allocation on the garbage-collected heap. Is there some
internal limit for D? I know I can't allocate more than 3GB in the
entire app, but I'm looking at single allocations.
Pardon me if this all sounds stupid, I'm a complete newb when it comes
to memory allocation mechanics. :)


Ironically, these benchmarks just show that the C heap is faster than  
the D GC
heap.  They are well over the 4 MB mark, where TempAlloc just falls back  
on the C
heap.  As far as how much you can allocate on the GC heap, it boils down  
to how
much **contiguous** free space you have in your address space.  Since  
32-bit
address space is only 4 GB (and only 2 GB for user space on Win32), I'm  
not

surprised you can't allocate more than 500 MB.


I think you are wrong about contiguous memory block. I believe virtual  
address space is divided into blocks (which are, indeed, contiguous) but  
the address space itself is not. What I mean is that if you try to  
allocate 1GB of memory, an OS might succeed that even if the physical  
memory is "fragmented" by other processes.


This isn't the case for ALL operating systems, though.

I've believe some OSes don't even allocate memory physically until you  
actually access it (by reading or writing to it), and even then they may  
only allocate memory for requested pages only.


I might be wrong though.

Re: TempAlloc: Request for Review For Inclusion in core.memory

2011-04-19 Thread Daniel Gibson

Am 19.04.2011 17:02, schrieb Denis Koroskin:
> On Tue, 19 Apr 2011 18:34:33 +0400, dsimcha  wrote:
> 
>> == Quote from Andrej Mitrovic (andrej.mitrov...@gmail.com)'s article
>>> Nice work. I've tried it out with allocating some large float arrays,
>>> quick two test cases were a 94MB and 282MB audio file.
>>> File 1 w/o TempAlloc: 410msecs
>>> File 1 w/ TempAlloc: 184msecs
>>> File 2 w/o TempAlloc 1321 msecs
>>> File 2 w/ TempAlloc 567 msecs
>>> Btw, how much *can* I allocate at once by using new (not specific to
>>> TempAlloc)? I've tried a 500MB file, but allocation fails even though
>>> I have plenty of free memory on my system. Is there a limit to using
>>> 'new'?
>>> Of course, ideally I'd use some kind of buffering mechanism where a
>>> portion of a file is loaded into memory, and when that's exhausted I'd
>>> load another chunk. But I'd like to know the limits for a single
>>> memory allocation on the garbage-collected heap. Is there some
>>> internal limit for D? I know I can't allocate more than 3GB in the
>>> entire app, but I'm looking at single allocations.
>>> Pardon me if this all sounds stupid, I'm a complete newb when it comes
>>> to memory allocation mechanics. :)
>>
>> Ironically, these benchmarks just show that the C heap is faster than
>> the D GC
>> heap.  They are well over the 4 MB mark, where TempAlloc just falls
>> back on the C
>> heap.  As far as how much you can allocate on the GC heap, it boils
>> down to how
>> much **contiguous** free space you have in your address space.  Since
>> 32-bit
>> address space is only 4 GB (and only 2 GB for user space on Win32),
>> I'm not
>> surprised you can't allocate more than 500 MB.
> 
> I think you are wrong about contiguous memory block. I believe virtual
> address space is divided into blocks (which are, indeed, contiguous) but
> the address space itself is not. What I mean is that if you try to
> allocate 1GB of memory, an OS might succeed that even if the physical
> memory is "fragmented" by other processes.
> 

That doesn't help if the logical memory of your process is fragmented as
well :)

> This isn't the case for ALL operating systems, though.
> 
> I've believe some OSes don't even allocate memory physically until you
> actually access it (by reading or writing to it), and even then they may
> only allocate memory for requested pages only.
> 
> I might be wrong though.

Re: std.parallelism: VOTE IN THIS THREAD

== Quote from Michel Fortin (michel.for...@michelf.com)'s article
> I say yes.
> I want to mention that I am a left a little skeptical about the lack of
> low-level race safety in this module, but I understand that this is
> more a language- or compiler-level problem than one with the module
> itself. I hope these problem will be addressed eventually and that
> std.parallelism can become safe that day.

One other point about safety:  D's flagship concurrency model doesn't hold your
hand on high-level invariants, only low-level data races.  Probably nothing will
ever be able to hold your hand about high level invariants.  With 
std.parallelism
getting the high-level invariants (i.e. what has data dependencies and what
doesn't) wrong can lead to low-level races, but as long as you get the 
high-level
invariants right (only parallelize stuff without data dependencies) all the low
level concurrency issues are taken care of and there can be no low-level data 
races.

Re: std.parallelism: VOTE IN THIS THREAD

YES

Re: std.parallelism: VOTE IN THIS THREAD

2011-04-19 Thread Jonas Drewsen

YES

Re: std.parallelism: VOTE IN THIS THREAD

2011-04-19 Thread Mike Parker

YES

Re: std.parallelism: VOTE IN THIS THREAD

2011-04-19 Thread Andrei Alexandrescu

YES

Re: TempAlloc: Request for Review For Inclusion in core.memory

2011-04-19 Thread Dmitry Olshansky


slackCat should probably use the common type, not the first one passed?
Also it would be cool to have addRange/removeRange somehow automated for 
types that have indirections.

Or it's too much state to carry around?

--
Dmitry Olshansky

Re: TempAlloc: Request for Review For Inclusion in core.memory

== Quote from Dmitry Olshansky (dmitry.o...@gmail.com)'s article
> slackCat should probably use the common type, not the first one passed?

Good point.  Will fix.

> Also it would be cool to have addRange/removeRange somehow automated for
> types that have indirections.
> Or it's too much state to carry around?

Yeah, I don't want this because a major purpose of TempAlloc is to allow 
efficient
parallelism, and addRange/removeRange involves locks.  Besides, I think they use
linear search in the current GC implementation rather than something more
efficient.  Once you're doing stuff like this, the advantages of TempAlloc go 
away
rapidly.  Besides, it's perfectly safe (at least now, while the GC isn't moving)
to store a reference to GC memory in TempAlloc, as long as it's not the **only**
reference.  It's also perfectly safe to have indirections from TempAlloc data 
into
other TempAlloc data, as long as the LIFO freeing order isn't violated.

std.parallelism Question

2011-04-19 Thread Jonathan Crapuchettes

I've been looking at your std.parallelism module and am really interested in it, 
but there is one problem that I have run into that I can't seem to find a 
solution. If I have code that requires a database connection in each thread, I 
don't really want to be creating and then deleting a lot (e.g. 500,000) of 
connections while using taskPool.parallel. Normally I would create a set of 
objects that extend Thread and create a single connection in each. Can you offer 
a suggestion how I might handle this situation? Here is an example of my problem:


string[] array = ...;
foreach (string element, taskPool.parallel(array))
{
//Create database connection
//Run some complex code with database query
}

I did come up with a solution, but it was a bit of a hack. I changed the 
definition of TaskPool to be a templated class:


final class TaskPool(ThreadType : Thread = Thread)

And this allowed me to create instances of my own worker class. The final usage 
looks like the following:


//Init the thread pool
TaskPool!(Worker) pool = new TaskPool!(Worker)(THREADS);

Worker mainThreadWorker = new Worker();
try {
foreach (string element; pool.parallel(list, 4))
{
if (thread_isMainThread())
mainThreadWorker.work(element);
else
(cast(Worker)Thread.getThis()).work(element);
}
} finally {
pool.finish();
thread_joinAll();
}

class Worker : Thread
{
private MysqlConnection db;

public this(void delegate() dg)
{
super(dg);
db = new MysqlConnection();
}
}


The code works and I am using it to do millions of calculations. Thoughts?
Thank you,
Jonathan

Re: std.parallelism Question

== Quote from Jonathan Crapuchettes (jcrapuchet...@gmail.com)'s article
> I've been looking at your std.parallelism module and am really interested in 
> it,
> but there is one problem that I have run into that I can't seem to find a
> solution. If I have code that requires a database connection in each thread, I
> don't really want to be creating and then deleting a lot (e.g. 500,000) of
> connections while using taskPool.parallel. Normally I would create a set of
> objects that extend Thread and create a single connection in each. Can you 
> offer
> a suggestion how I might handle this situation?

These kinds of situations are **exactly** what worker-local storage is for.  
Roughly:

// new MysqlConnection() is lazy, you get one per worker thread.
auto connections = taskPool.workerLocalStorage(new MysqlConnection());

foreach(element; taskPool.parallel(array)) {
// Get the worker-local instance of connection.
auto connection = connections.get;

// Do stuff.
}

// Now that we're done with the parallel part of the algorithm,
// we can access all the connections to free them.

foreach(c; connections.toRange) {
c.free();
}

Re: std.parallelism: VOTE IN THIS THREAD

dsimcha:

> Probably nothing will ever be able to hold your hand about high level 
> invariants.

Who knows? Probably there are papers with ideas about such problems.

Bye,
bearophile

More rational language design

A bit of quantitative design for computer languages (about Java):
http://www.ibm.com/developerworks/java/library/j-ldn1/

>From the article:

>If one of the challenges of evolving a widely used language is that there's a 
>lot of code out there that we don't want to break, one of the benefits is that 
>there's a lot of code out there that we can use to analyze how often certain 
>idioms come up. By grabbing a big body of code (a corpus) and running an 
>analysis tool (often an instrumented version of the compiler) over it, you can 
>learn a lot about how often certain idioms come up in the real world, and 
>whether they are prevalent enough to influence how we evolve the language.<

Bye,
bearophile

Re: lame question

2011-04-19 Thread Timon Gehr

Steven Schveighoffer wrote:
> And one other note -- delete will eventually be deprecated.  In order to
> free memory, you must use clear and GC.free.

> -Steve

Well, why? It seems like a bad decision to me.

Re: std.parallelism: VOTE IN THIS THREAD

2011-04-19 Thread Sean Kelly

YES

Re: lame question

2011-04-19 Thread Steven Schveighoffer


On Tue, 19 Apr 2011 14:04:38 -0400, Timon Gehr  wrote:


Steven Schveighoffer wrote:

And one other note -- delete will eventually be deprecated.  In order to
free memory, you must use clear and GC.free.



-Steve


Well, why? It seems like a bad decision to me.


I'm neutral on the subject, you'd have to ask Andrei.  I rarely use delete  
anyways.


-Steve

Re: std.parallelism Question

2011-04-19 Thread Jonathan Crapuchettes


Thank you for your quick reply. I am now starting to see how it should be used.

As a little bit of encouragement, your code is being used in in-house 
applications doing millions to trillions of calculations that are creating 
leading edge economic data.


Thank you again,
Jonathan

dsimcha wrote:

== Quote from Jonathan Crapuchettes (jcrapuchet...@gmail.com)'s article

I've been looking at your std.parallelism module and am really interested in it,
but there is one problem that I have run into that I can't seem to find a
solution. If I have code that requires a database connection in each thread, I
don't really want to be creating and then deleting a lot (e.g. 500,000) of
connections while using taskPool.parallel. Normally I would create a set of
objects that extend Thread and create a single connection in each. Can you offer
a suggestion how I might handle this situation?


These kinds of situations are **exactly** what worker-local storage is for.  
Roughly:

// new MysqlConnection() is lazy, you get one per worker thread.
auto connections = taskPool.workerLocalStorage(new MysqlConnection());

foreach(element; taskPool.parallel(array)) {
 // Get the worker-local instance of connection.
 auto connection = connections.get;

 // Do stuff.
}

// Now that we're done with the parallel part of the algorithm,
// we can access all the connections to free them.

foreach(c; connections.toRange) {
 c.free();
}

Re: std.parallelism: VOTE IN THIS THREAD

2011-04-19 Thread Rainer Schuetze


Yes.

This module is a great example of the expressive powers of D.

Maybe it's too late for comments, but skipping over the documentation 
again, I think parallel foreach is the flagship of the module and the 
simplest concept to understand, so I would expect the example from 
parallel(R) to be shown in the synopsis.

Re: std.parallelism: VOTE IN THIS THREAD

2011-04-19 Thread Mike Wey


YES

--
Mike Wey

Re: Floating Point + Threads?

2011-04-19 Thread Sean Kelly

On Apr 16, 2011, at 1:02 PM, Robert Jacques wrote:

> On Sat, 16 Apr 2011 15:32:12 -0400, Walter Bright 
>  wrote:
>> 
>> 
>> The dmd startup code (actually the C startup code) does an fninit. I never 
>> thought about new thread starts. So, yeah, druntime should do an fninit on 
>> thread creation.
> 
> The documentation I've found on fninit seems to indicate it defaults to 
> 64-bit precision, which means that by default we aren't seeing the benefit of 
> D's reals. I'd much prefer 80-bit precision by default.

There is no option to set "80-bit precision" via the FPU control word.  Section 
8.1.5.2 of the Intel 64 SDM says the following:

"The precision-control (PC) field (bits 8 and 9 of the x87 FPU control word) 
determines the precision (64, 53, or 24 bits) of floating-point calculations 
made by the x87 FPU (see Table 8-2). The default precision is double extended 
precision, which uses the full 64-bit significand available with the double 
extended-precision floating-point format of the x87 FPU data registers. This 
setting is best suited for most applications, because it allows applications to 
take full advantage of the maximum precision available with the x87 FPU data 
registers."

So it sounds like finit/fninit does what we want.

Re: std.parallelism: VOTE IN THIS THREAD

2011-04-19 Thread Hamad Mohammad

YES

Re: std.parallelism: VOTE IN THIS THREAD

2011-04-19 Thread Timon Gehr

YES.

Re: lame question

2011-04-19 Thread Andrei Alexandrescu


On 4/19/11 1:04 PM, Timon Gehr wrote:

Steven Schveighoffer wrote:

And one other note -- delete will eventually be deprecated.  In order to
free memory, you must use clear and GC.free.



-Steve


Well, why? It seems like a bad decision to me.


The feature is not going away, just the keyword.

"delete" is a gratuitous carryover from C++, which had to invent it 
because the need for manual object disposal predated the introduction of 
templates.


D stays a systems programming language but also has a safe subset and 
generally offers better safety guarantees than C++. It is excessive to 
allocate a keyword to a feature that's fundamentally unsafe, 
particularly since the underlying feature offers considerably fewer 
guarantees than its C++ counterpart. (Some GCs are unable to implement 
"delete" meaningfully.)


Manual memory disposal for the GC heap will be implemented as a template 
function with semantics defined by the GC implementation.




Andrei

Re: lame question

Andrei:

> (Some GCs are unable to implement 
> "delete" meaningfully.)
> 
> Manual memory disposal for the GC heap will be implemented as a template 
> function with semantics defined by the GC implementation.

Now the function(s) that replace "delete" are meant to be hints for the GC. In 
a system language that uses a GC you probably enjoy some different kinds of GC 
hints (as in CLisp GC).

Bye,
bearophile

Hidden argument kind antipattern

2011-04-19 Thread Vladimir Panteleev

To elaborate, I mean allowing code which appears to behave surprisingly  
different from the at-a-glance interpretation, unless the programmer knows  
the function's signature. I've noticed a worrying adoption in D of this  
"antipattern", which, frankly, I believe doesn't belong in a well-designed  
programming language. One classic example of this is passing arguments by  
reference, something D inherited from C++. For example:


int x, y;
// ...
someObject.updateCoords(x, y);

What can you say about this code? The unobvious and surprising  
interpretation of it is that updateCoords will change the values of x and  
y. C# solved this problem neatly by requiring to specify the "ref" keyword  
before the function arguments:


someObject.updateCoords(ref x, ref y); // much clearer

This problem carries over to lazy parameters, as well. I'll quote a line  
of code from a recent post by David Simcha:



auto connections = taskPool.workerLocalStorage(new MysqlConnection());


Anyone who is not familiar with std.parallelism and D's lazy parameter  
feature is going to be very surprised to find out what this code really  
does. This might be OK for us, experienced D users, but think about the  
poor bloke who will someday, somewhere try to debug a D program written by  
someone else, and tear his hair out trying to figure out why an expression  
passed as an argument to some function isn't being evaluated before/during  
the function call. (The solution I would suggest is simply having to  
specify the "lazy" keyword before the expression of each lazy parameter,  
same as C#'s "ref". This will require updating all usage of "enforce"  
among other changes.)


I know that I should have brought this up during the module's review  
phase, so take the following with a grain of salt: in my opinion,  
std.parallelism's usage of lazy parameters overreaches their intended use  
and borders into abuse of this language feature. For this reason (and  
meaning no disrespect towards David and everyone who participated in  
shaping the module), I am uncomfortable with the inclusion of  
std.parallelism into the standard D library, as it would carry the message  
that D's maintainers encourage abusing the language in such ways. (This  
probably doesn't count as a vote.) For the moment, I would suggest  
changing all lazy parameters which are evaluated in different contexts  
(outside of the said function) into delegates, and replacing their usage  
with delegate literals.


--
Best regards,
 Vladimirmailto:vladi...@thecybershadow.net

Re: Hidden argument kind antipattern

== Quote from Vladimir Panteleev (vladi...@thecybershadow.net)'s article
> To elaborate, I mean allowing code which appears to behave surprisingly
> different from the at-a-glance interpretation, unless the programmer knows
> the function's signature. I've noticed a worrying adoption in D of this
> "antipattern", which, frankly, I believe doesn't belong in a well-designed
> programming language. One classic example of this is passing arguments by
> reference, something D inherited from C++. For example:
> int x, y;
> // ...
> someObject.updateCoords(x, y);
> What can you say about this code? The unobvious and surprising
> interpretation of it is that updateCoords will change the values of x and
> y. C# solved this problem neatly by requiring to specify the "ref" keyword
> before the function arguments:
> someObject.updateCoords(ref x, ref y); // much clearer
> This problem carries over to lazy parameters, as well. I'll quote a line
> of code from a recent post by David Simcha:
> > auto connections = taskPool.workerLocalStorage(new MysqlConnection());
> Anyone who is not familiar with std.parallelism and D's lazy parameter
> feature is going to be very surprised to find out what this code really
> does. This might be OK for us, experienced D users, but think about the
> poor bloke who will someday, somewhere try to debug a D program written by
> someone else, and tear his hair out trying to figure out why an expression
> passed as an argument to some function isn't being evaluated before/during
> the function call. (The solution I would suggest is simply having to
> specify the "lazy" keyword before the expression of each lazy parameter,
> same as C#'s "ref". This will require updating all usage of "enforce"
> among other changes.)
> I know that I should have brought this up during the module's review
> phase, so take the following with a grain of salt: in my opinion,
> std.parallelism's usage of lazy parameters overreaches their intended use
> and borders into abuse of this language feature. For this reason (and
> meaning no disrespect towards David and everyone who participated in
> shaping the module), I am uncomfortable with the inclusion of
> std.parallelism into the standard D library, as it would carry the message
> that D's maintainers encourage abusing the language in such ways. (This
> probably doesn't count as a vote.) For the moment, I would suggest
> changing all lazy parameters which are evaluated in different contexts
> (outside of the said function) into delegates, and replacing their usage
> with delegate literals.

Interesting point.  I was questioning whether this was really a good idea 
myself.
 I did it because it seemed like things like this were kind of the point of lazy
parameters.  Let's have a good discussion on this now, and if we decide that my
idiom is an abuse of lazy parameters, then I'll switch it over to a delegate
literal.  My vote is weakly in favor of lazy parameters, since a maintenance
programmer would probably look at what such a function does and not be surprised
for very long.  However, I'm not dead-set against changing it if there's a 
strong
consensus that it should be changed.

Re: Hidden argument kind antipattern

2011-04-19 Thread spir


On 04/20/2011 01:09 AM, Vladimir Panteleev wrote:

To elaborate, I mean allowing code which appears to behave surprisingly
different from the at-a-glance interpretation, unless the programmer knows the
function's signature. I've noticed a worrying adoption in D of this
"antipattern", which, frankly, I believe doesn't belong in a well-designed
programming language. One classic example of this is passing arguments by
reference, something D inherited from C++. For example:

int x, y;
// ...
someObject.updateCoords(x, y);

What can you say about this code? The unobvious and surprising interpretation
of it is that updateCoords will change the values of x and y. C# solved this
problem neatly by requiring to specify the "ref" keyword before the function
arguments:

someObject.updateCoords(ref x, ref y); // much clearer

This problem carries over to lazy parameters, as well. I'll quote a line of
code from a recent post by David Simcha:


auto connections = taskPool.workerLocalStorage(new MysqlConnection());


Anyone who is not familiar with std.parallelism and D's lazy parameter feature
is going to be very surprised to find out what this code really does. This
might be OK for us, experienced D users, but think about the poor bloke who
will someday, somewhere try to debug a D program written by someone else, and
tear his hair out trying to figure out why an expression passed as an argument
to some function isn't being evaluated before/during the function call. (The
solution I would suggest is simply having to specify the "lazy" keyword before
the expression of each lazy parameter, same as C#'s "ref". This will require
updating all usage of "enforce" among other changes.)

[...current vote about std.parallel...]
For the moment, I would suggest
changing all lazy parameters which are evaluated in different contexts (outside
of the said function) into delegates, and replacing their usage with delegate
literals.


Sounds very good to me.

Denis
--
_
vita es estrany
spir.wikidot.com

Re: Hidden argument kind antipattern

2011-04-19 Thread spir


On 04/20/2011 01:18 AM, dsimcha wrote:

== Quote from Vladimir Panteleev (vladi...@thecybershadow.net)'s article

To elaborate, I mean allowing code which appears to behave surprisingly
different from the at-a-glance interpretation, unless the programmer knows
the function's signature. I've noticed a worrying adoption in D of this
"antipattern", which, frankly, I believe doesn't belong in a well-designed
programming language. One classic example of this is passing arguments by
reference, something D inherited from C++. For example:
int x, y;
// ...
someObject.updateCoords(x, y);
What can you say about this code? The unobvious and surprising
interpretation of it is that updateCoords will change the values of x and
y. C# solved this problem neatly by requiring to specify the "ref" keyword
before the function arguments:
someObject.updateCoords(ref x, ref y); // much clearer
This problem carries over to lazy parameters, as well. I'll quote a line
of code from a recent post by David Simcha:

auto connections = taskPool.workerLocalStorage(new MysqlConnection());

Anyone who is not familiar with std.parallelism and D's lazy parameter
feature is going to be very surprised to find out what this code really
does. This might be OK for us, experienced D users, but think about the
poor bloke who will someday, somewhere try to debug a D program written by
someone else, and tear his hair out trying to figure out why an expression
passed as an argument to some function isn't being evaluated before/during
the function call. (The solution I would suggest is simply having to
specify the "lazy" keyword before the expression of each lazy parameter,
same as C#'s "ref". This will require updating all usage of "enforce"
among other changes.)
I know that I should have brought this up during the module's review
phase, so take the following with a grain of salt: in my opinion,
std.parallelism's usage of lazy parameters overreaches their intended use
and borders into abuse of this language feature. For this reason (and
meaning no disrespect towards David and everyone who participated in
shaping the module), I am uncomfortable with the inclusion of
std.parallelism into the standard D library, as it would carry the message
that D's maintainers encourage abusing the language in such ways. (This
probably doesn't count as a vote.) For the moment, I would suggest
changing all lazy parameters which are evaluated in different contexts
(outside of the said function) into delegates, and replacing their usage
with delegate literals.


Interesting point.  I was questioning whether this was really a good idea 
myself.
  I did it because it seemed like things like this were kind of the point of 
lazy
parameters.  Let's have a good discussion on this now, and if we decide that my
idiom is an abuse of lazy parameters, then I'll switch it over to a delegate
literal.  My vote is weakly in favor of lazy parameters, since a maintenance
programmer would probably look at what such a function does and not be surprised
for very long.  However, I'm not dead-set against changing it if there's a 
strong
consensus that it should be changed.


And what about requiring "lazy" (and "ref") at the call site; I mean in the 
long term? I find this very sensible.


Denis
--
_
vita es estrany
spir.wikidot.com

Re: Hidden argument kind antipattern

2011-04-19 Thread Vladimir Panteleev


On Wed, 20 Apr 2011 02:18:39 +0300, dsimcha  wrote:

My vote is weakly in favor of lazy parameters, since a maintenance  
programmer would probably look at what such a function does and not be  
surprised for very long.


Not contradicting you, but I'd like to clarify my point: given a large  
function with unknown code, it is my opinion that, with a well-designed  
programming language / library, a maintainer will not have to inspect the  
signature of each function referenced from the code to understand the  
basic _data flow_ and _execution order_ within the function.


--
Best regards,
 Vladimirmailto:vladi...@thecybershadow.net

Re: Hidden argument kind antipattern

dsimcha:

>  I did it because it seemed like things like this were kind of the point of 
> lazy
> parameters.

I agree. In my opinion the problem is not you using in your module a language 
feature designed for that very purpose. What's in question is the syntax of 
that language feature. I don't mind lazy arguments, but it can be argued that's 
better for them to require a "lazy" prefix at the calling point too. This 
little syntax problem that was already raised during the D2 design phase.

Two little disadvantages of requiring "lazy" prefix at the calling point too 
are:
- A little more code to write. This is not really a typing problem of five 
extra lowercase chars (there is a space too), but a problem of succinctness and 
syntax noise. I think being it lower case the added syntax noise is not so high.
- It's less general code. Changing a normal argument from lazy to not lazy or 
from not lazy to lazy becomes not transparent any more, it requires changes at 
all call points. Recent versions of C# have softened the requirements of the 
"ref" annotation at the calling point, when using COM:
http://en.wikipedia.org/wiki/C_Sharp_4.0#Optional_ref_Keyword_when_using_COM

I am undecided still. I weakly agree to require "lazy" suffix at the calling 
point too.

Bye,
bearophile

Re: Hidden argument kind antipattern

2011-04-19 Thread Kai Meyer


On 04/19/2011 05:18 PM, dsimcha wrote:

== Quote from Vladimir Panteleev (vladi...@thecybershadow.net)'s article

To elaborate, I mean allowing code which appears to behave surprisingly
different from the at-a-glance interpretation, unless the programmer knows
the function's signature. I've noticed a worrying adoption in D of this
"antipattern", which, frankly, I believe doesn't belong in a well-designed
programming language. One classic example of this is passing arguments by
reference, something D inherited from C++. For example:
int x, y;
// ...
someObject.updateCoords(x, y);
What can you say about this code? The unobvious and surprising
interpretation of it is that updateCoords will change the values of x and
y. C# solved this problem neatly by requiring to specify the "ref" keyword
before the function arguments:
someObject.updateCoords(ref x, ref y); // much clearer
This problem carries over to lazy parameters, as well. I'll quote a line
of code from a recent post by David Simcha:

auto connections = taskPool.workerLocalStorage(new MysqlConnection());

Anyone who is not familiar with std.parallelism and D's lazy parameter
feature is going to be very surprised to find out what this code really
does. This might be OK for us, experienced D users, but think about the
poor bloke who will someday, somewhere try to debug a D program written by
someone else, and tear his hair out trying to figure out why an expression
passed as an argument to some function isn't being evaluated before/during
the function call. (The solution I would suggest is simply having to
specify the "lazy" keyword before the expression of each lazy parameter,
same as C#'s "ref". This will require updating all usage of "enforce"
among other changes.)
I know that I should have brought this up during the module's review
phase, so take the following with a grain of salt: in my opinion,
std.parallelism's usage of lazy parameters overreaches their intended use
and borders into abuse of this language feature. For this reason (and
meaning no disrespect towards David and everyone who participated in
shaping the module), I am uncomfortable with the inclusion of
std.parallelism into the standard D library, as it would carry the message
that D's maintainers encourage abusing the language in such ways. (This
probably doesn't count as a vote.) For the moment, I would suggest
changing all lazy parameters which are evaluated in different contexts
(outside of the said function) into delegates, and replacing their usage
with delegate literals.


Interesting point.  I was questioning whether this was really a good idea 
myself.
  I did it because it seemed like things like this were kind of the point of 
lazy
parameters.  Let's have a good discussion on this now, and if we decide that my
idiom is an abuse of lazy parameters, then I'll switch it over to a delegate
literal.  My vote is weakly in favor of lazy parameters, since a maintenance
programmer would probably look at what such a function does and not be surprised
for very long.  However, I'm not dead-set against changing it if there's a 
strong
consensus that it should be changed.


I would agree with the "ref" and "lazy" modifiers to parameters. If they 
exist in the function signature, it seems potentially confusing to 
"silently" accept a built-in or other stack-based parameter as a 
reference with out an explicit "ref" in the call. For instance:


import std.stdio;
struct Bar { public int a; }
class Foo { public int a; }
void main()
{
int a;
Bar b = Bar();
Foo f = new Foo();
a   = 1;
b.a = 1;
f.a = 1;
writef("a=%d b.a=%d f.a=%d\n", a, b.a, f.a);
changeit(a);
changeit(b);
changeit(f);
writef("a=%d b.a=%d f.a=%d\n", a, b.a, f.a);
changeit_ref(a);
changeit_ref(b);
changeit_ref(f);
writef("a=%d b.a=%d f.a=%d\n", a, b.a, f.a);
}
void changeit(int baz) { baz =   2; }
void changeit(Bar bar) { bar.a = 2; }
void changeit(Foo foo) { foo.a = 2; }
void changeit_ref(ref int baz) { baz =   3; }
void changeit_ref(ref Bar bar) { bar.a = 3; }
void changeit_ref(Foo foo) { foo.a = 3; }
//Output:
//a=1 b.a=1 f.a=1
//a=1 b.a=1 f.a=2
//a=3 b.a=3 f.a=3


The call signatures for changeit and changeit_ref are exactly the same, 
but the function definition signatures are different. In the example 
above, I can see the value in forcing the caller to use 
"changeit_ref(ref a)" to match the function declaration.

Re: Hidden argument kind antipattern

2011-04-19 Thread Vladimir Panteleev

On Wed, 20 Apr 2011 02:43:50 +0300, bearophile   
wrote:


I agree. In my opinion the problem is not you using in your module a  
language feature designed for that very purpose. What's in question is  
the syntax of that language feature. I don't mind lazy arguments, but it  
can be argued that's better for them to require a "lazy" prefix at the  
calling point too. This little syntax problem that was already raised  
during the D2 design phase.


I would like to clarify my point:

Inarguably (IMO), the original intention of lazy parameters was to allow  
passing expressions to functions that were evaluated only when required.


David's library evaluates these expressions *in different threads*!

Generally, I would count that any usage of "lazy" counts as abuse except  
for evaluating the argument, at most once, during the function's execution.


--
Best regards,
 Vladimirmailto:vladi...@thecybershadow.net

Loss of precision errors in FP conversions

In Bugzilla I have just added an enhancement request that asks for a little 
change in D, I don't know if it was already discussed or if it's already 
present in Bugzilla:
http://d.puremagic.com/issues/show_bug.cgi?id=5864

In a program like this:

void main() {
uint x = 10_000;
ubyte b = x;
}


DMD 2.052 raises a compilation error like this, because the b=x assignment may 
lose some information, some bits of x:

test.d(3): Error: cannot implicitly convert expression (x) of type uint to ubyte

I think that a safe and good system language has to help avoid unwanted 
(implicit) loss of information during data conversions.

This is a case of loss of precision where D generates no compile errors:


import std.stdio;
void main() {
real f1 = 1.1112333;
writefln("%.19f", f1);
double f2 = f1; // loss of FP precision
writefln("%.19f", f2);
float f3 = f2; // loss of FP precision
writefln("%.19f", f3);
}

Despite some information is lost, see the output:
1.1112332
1.1113261
1.110864639282226

So one possible way to face this situation is to statically disallow 
double=>float, real=>float, and real=>double conversions (on some computers 
real=>double conversions don't cause loss of information, but I suggest to 
ignore this, to increase code portability), and introduce compile-time errors 
like:

test.d(5): Error: cannot implicitly convert expression (f1) of type real to 
double
test.d(7): Error: cannot implicitly convert expression (f2) of type double to 
float


Today float values seem less useful, because with serial CPU instructions the 
performance difference between operations on float and double is often not 
important, and often you want the precision of doubles. But modern CPUs (and 
current GPUs) have vector operations too. They are currently able to perform 
operations on 4 float values or 2 double values (or 8 float or 4 doubles) at 
the same time for each instruction. Such vector instructions are sometimes used 
directly in C-GCC code using SSE intrinsics, or come out of auto-vectorization 
optimization of loops done by GCC on normal serial C code. In this situation 
the usage of float instead of double gives almost a twofold performance 
increase. There are programs (like certain ray-tracing code) where the 
precision of a float is enough. So a compile-time error that catches currently 
implicit double->float conversions may help the programmer avoid unwanted 
usages of doubles that allow the compiler to pack 4/8 floats in a vector !
 register during loop vectorizations.


Partially related note: currently std.math doesn't seem to use the cosf, sinf C 
functions, but it uses sqrtf:

import std.math: sqrt, sin, cos;
void main() {
float x = 1.0f;
static assert(is(typeof(  sqrt(x)  ) == float)); // OK
static assert(is(typeof(  sin(x)   ) == float)); // ERR
static assert(is(typeof(  cos(x)   ) == float)); // ERR
}

Bye,
bearophile

Re: Hidden argument kind antipattern


On 4/19/2011 7:30 PM, spir wrote:

And what about requiring "lazy" (and "ref") at the call site; I mean in
the long term? I find this very sensible.

Denis


I wouldn't mind this for "lazy" but would argue heavily against it for 
"ref".  ref parameters are both more frequently used and less surprising 
than lazy parameters and usually the name and usage of the function give 
a stronger hint about what it does.  Secondly, functions may modify 
their input parameters in surprising ways without ref, if the parameters 
contain indirection (for example, classes, pointers, arrays). 
Furthermore, requiring ref at the call site would break uniform function 
call syntax.  For example:


void popFront(T)(ref T[] array) {
array = array[1..$];
}

void main() {
auto foo = [1, 2, 3];
foo.popFront();
}

Re: Hidden argument kind antipattern


On 4/19/2011 7:47 PM, Vladimir Panteleev wrote:

David's library evaluates these expressions *in different threads*!



Minor correction:  No it doesn't.  It constructs a WorkerLocalStorage 
object in a regular class method in the thread from which 
workerLocalStorage() was called.  It does evaluate the lazy parameter 
more than once, but all the multithreading stuff happens **after** 
workerLocalStorage() returns.  I still see your point, though.

Re: Loss of precision errors in FP conversions


On 4/19/2011 7:49 PM, bearophile wrote:

In Bugzilla I have just added an enhancement request that asks for a little 
change in D, I don't know if it was already discussed or if it's already 
present in Bugzilla:
http://d.puremagic.com/issues/show_bug.cgi?id=5864

In a program like this:

void main() {
 uint x = 10_000;
 ubyte b = x;
}


DMD 2.052 raises a compilation error like this, because the b=x assignment may 
lose some information, some bits of x:

test.d(3): Error: cannot implicitly convert expression (x) of type uint to ubyte

I think that a safe and good system language has to help avoid unwanted 
(implicit) loss of information during data conversions.

This is a case of loss of precision where D generates no compile errors:


import std.stdio;
void main() {
 real f1 = 1.1112333;
 writefln("%.19f", f1);
 double f2 = f1; // loss of FP precision
 writefln("%.19f", f2);
 float f3 = f2; // loss of FP precision
 writefln("%.19f", f3);
}

Despite some information is lost, see the output:
1.1112332
1.1113261
1.110864639282226

So one possible way to face this situation is to statically disallow double=>float, 
real=>float, and real=>double conversions (on some computers real=>double 
conversions don't cause loss of information, but I suggest to ignore this, to increase code 
portability), and introduce compile-time errors like:

test.d(5): Error: cannot implicitly convert expression (f1) of type real to 
double
test.d(7): Error: cannot implicitly convert expression (f2) of type double to 
float


Today float values seem less useful, because with serial CPU instructions the 
performance difference between operations on float and double is often not 
important, and often you want the precision of doubles. But modern CPUs (and 
current GPUs) have vector operations too. They are currently able to perform 
operations on 4 float values or 2 double values (or 8 float or 4 doubles) at the 
same time for each instruction. Such vector instructions are sometimes used 
directly in C-GCC code using SSE intrinsics, or come out of auto-vectorization 
optimization of loops done by GCC on normal serial C code. In this situation the 
usage of float instead of double gives almost a twofold performance increase. 
There are programs (like certain ray-tracing code) where the precision of a float 
is enough. So a compile-time error that catches currently implicit 
double->float conversions may help the programmer avoid unwanted usages of 
doubles that allow the compiler to pack 4/8 floats in a vecto

r register during loop vectorizations.



Partially related note: currently std.math doesn't seem to use the cosf, sinf C 
functions, but it uses sqrtf:

import std.math: sqrt, sin, cos;
void main() {
 float x = 1.0f;
 static assert(is(typeof(  sqrt(x)  ) == float)); // OK
 static assert(is(typeof(  sin(x)   ) == float)); // ERR
 static assert(is(typeof(  cos(x)   ) == float)); // ERR
}

Bye,
bearophile


Please, _NOOO!!_  The integer conversion errors are already arguably 
too pedantic, make generic code harder to write and get in the way about 
as often as they help.  Floating point tends to degrade much more 
gracefully than integer.  Where integer narrowing can just be silently, 
non-obviously and completely wrong, floating point narrowing will at 
least be approximately right, or become infinity and be wrong in an 
obvious way.  I know what you suggest could prevent bugs in a lot of 
cases, but it also has the potential to get in the way in a lot of cases.


Generally I worry about D's type system becoming like the Boy Who Cried 
Wolf, where it flags so many potential errors (as opposed to things that 
are definitely errors) that people become conditioned to just put in 
whatever casts they need to shut it up.  I definitely fell into that 
when porting some 32-bit code that was sloppy with size_t vs. int to 
64-bit.  I knew there was no way it was going to be a problem, because 
there was no way any of my arrays were going to be even within a few 
orders of magnitude of int.max, but the compiler insisted on nagging me 
about it and I reflexively just put in casts everywhere.  A warning 
_may_ be appropriate, but definitely not an error.

Re: TempAlloc: Request for Review For Inclusion in core.memory


On 4/19/2011 12:08 PM, Dmitry Olshansky wrote:

slackCat should probably use the common type, not the first one passed?


Fixed.

https://github.com/dsimcha/TempAlloc/commit/5e3fd0db9c905d361695d267e460a436c2b2

https://github.com/dsimcha/TempAlloc/commit/5e74dd20d92c545cb4e3945c81175307c60b1253

Re: Loss of precision errors in FP conversions

dsimcha:

> I know what you suggest could prevent bugs in a lot of
> cases, but it also has the potential to get in the way in a lot of cases.

You are right, and I am ready to close that enhancement request at once if the
consensus is against it.

double->float and real->float cases are not so common. How often do you use
floats in your code? In my code it's uncommon to use floats, generally I use
doubles.

A problem may be in real->double conversions, because I think D feels free to
use intermediate real values in some FP computations.

Another possible problem: generic code like this is going to produce an error
because 2.0 literal is double, so x*2.0 is a double even if x is float:

T foo(T)(T x) {
return x * 2.0;
}

But when I use floats, I have found it good a C lint that spots double->float
conversions, because it has actually allowed me to speed up some code that was
doing float->double->float conversions without me being aware of it.

> A warning _may_ be appropriate, but definitely not an error.

Another option is a -warn_fp_precision_loss compiler switch, that produces
warnings only when you use it. For my purposes this is is enough.

Regarding the actual amount of troubles this errors messages are going to
cause, I have recently shown a link that argues for quantitative analysis of
language changes:
http://www.digitalmars.com/webnews/newsgroups.php?art_group=digitalmars.D&article_id=135049
The idea here is to introduce those three FP conversions errors, compile Phobos
and some other D2 code, and count how many problems it causes.

Bye,
bearophile

Re: Loss of precision errors in FP conversions


On 4/19/2011 8:37 PM, bearophile wrote:

dsimcha:


I know what you suggest could prevent bugs in a lot of
cases, but it also has the potential to get in the way in a lot of cases.


You are right, and I am ready to close that enhancement request at once if the 
consensus is against it.

double->float and real->float cases are not so common. How often do you use 
floats in your code? In my code it's uncommon to use floats, generally I use doubles.


Very often, actually.  Basically, any time I have a lot of floating 
point numbers that aren't going to be extremely big or small in 
magnitude and I'm interested in storing them and maybe performing a few 
_simple_ computations with them (sorting, statistical tests, most 
machine learning algorithms, etc.).  Good examples are gene expression 
levels or transformations thereof and probabilities.  Single precision 
is plenty unless your numbers are extremely big or small, you need a 
ridiculous number of significant figures, or you're performing intense 
computations (for example matrix factorizations) where rounding error 
may accumulate and turn a small loss of precision into a catastrophic one.

Re: Loss of precision errors in FP conversions

2011-04-19 Thread Walter Bright


On 4/19/2011 5:02 PM, dsimcha wrote:

Generally I worry about D's type system becoming like the Boy Who Cried Wolf,
where it flags so many potential errors (as opposed to things that are
definitely errors) that people become conditioned to just put in whatever casts
they need to shut it up. I definitely fell into that when porting some 32-bit
code that was sloppy with size_t vs. int to 64-bit. I knew there was no way it
was going to be a problem, because there was no way any of my arrays were going
to be even within a few orders of magnitude of int.max, but the compiler
insisted on nagging me about it and I reflexively just put in casts everywhere.
A warning _may_ be appropriate, but definitely not an error.


That's definitely a worry. Having a Nagging Nellie giving false alarms on errors 
too often will:


1. cause programmers to hate D

2. lead to MORE bugs and harder to find ones, as Bruce Eckel pointed out, 
because people will put in things "just to shut up the compiler"


Hence my reluctance to add in a lot of these suggestions.

As to the specific about erroring on reducing precision, my talks with people 
who actually do write a lot of FP code for a living is NO. They don't want it. 
Losing precision in FP calculations is a fact of life, and FP programmers simply 
must understand it and deal with it. Having the compiler annoy you about it 
would be less than helpful.

Re: Loss of precision errors in FP conversions

Walter:

> Hence my reluctance to add in a lot of these suggestions.

In an answer I've suggested the alternative solution of a 
-warn_fp_precision_loss compiler switch, that produces warnings only when you 
use it. In theory this avoids most of the Nagging Nellie problem, because you 
use this switch only in special situations. But I am aware you generally don't 
like warnings.


> As to the specific about erroring on reducing precision, my talks with people
> who actually do write a lot of FP code for a living is NO. They don't want it.
> Losing precision in FP calculations is a fact of life, and FP programmers 
> simply
> must understand it and deal with it. Having the compiler annoy you about it
> would be less than helpful.

Loss of some precision bits in normal FP operations is a fact of life, but 
double->float conversions usually lose a much more significant amount of 
precision, and it's not a fact of life, it's the code that in some way asks for 
this irreversible conversion.

A related problem your answer doesn't keep in account are unwanted 
float->double conversions (that get spotted by those error messages just 
because the code actually performs float->double->float conversions). Such 
unwanted conversions have caused performance loss on a 32-bit-mode CPU in some 
of my C code (maybe this problem is not present with CPU on 64 bit code), 
because the code was actually using doubles. A C lint has allowed me to spot 
such problems and fix them.

Thank you for your answers,
bye,
bearophile

Re: Loss of precision errors in FP conversions

2011-04-19 Thread Walter Bright


On 4/19/2011 6:46 PM, bearophile wrote:

Walter:


Hence my reluctance to add in a lot of these suggestions.


In an answer I've suggested the alternative solution of a
-warn_fp_precision_loss compiler switch, that produces warnings only when you
use it. In theory this avoids most of the Nagging Nellie problem, because you
use this switch only in special situations. But I am aware you generally
don't like warnings.


Yes, I've argued strongly against warnings.

Re: Loss of precision errors in FP conversions

2011-04-19 Thread Brad Roberts

On Tue, 19 Apr 2011, Walter Bright wrote:

> On 4/19/2011 6:46 PM, bearophile wrote:
> > Walter:
> > 
> > > Hence my reluctance to add in a lot of these suggestions.
> > 
> > In an answer I've suggested the alternative solution of a
> > -warn_fp_precision_loss compiler switch, that produces warnings only when
> > you
> > use it. In theory this avoids most of the Nagging Nellie problem, because
> > you
> > use this switch only in special situations. But I am aware you generally
> > don't like warnings.
> 
> Yes, I've argued strongly against warnings.

The stronger argument, that I agree with, is not having flag based 
sometimes warnings.  The more flags you have, the more complex the matrix 
of landmines there are.  I hate micro-managment, in all it's forms.

Re: Loss of precision errors in FP conversions

2011-04-19 Thread Andrei Alexandrescu


On 4/19/11 7:37 PM, bearophile wrote:

dsimcha:


I know what you suggest could prevent bugs in a lot of
cases, but it also has the potential to get in the way in a lot of cases.


You are right, and I am ready to close that enhancement request at once if the 
consensus is against it.


Yes please. I once felt the same way, but learned better since.

Andrei

Re: Loss of precision errors in FP conversions

Andrei:

> Yes please. I once felt the same way, but learned better since.

OK :-) But I will probably write an article about this, because I have found a 
performance problem, that currently DMD doesn't help me avoid, that a C lint 
has avoided me.

Bye,
bearophile

Re: Loss of precision errors in FP conversions

On Tue, 19 Apr 2011 20:37:47 -0400, bearophile   
wrote:

dsimcha:


I know what you suggest could prevent bugs in a lot of
cases, but it also has the potential to get in the way in a lot of  
cases.


You are right, and I am ready to close that enhancement request at once  
if the consensus is against it.


double->float and real->float cases are not so common. How often do you  
use floats in your code? In my code it's uncommon to use floats,  
generally I use doubles.


I do GP GPU work, so I use floats all the time. They're also useful for  
data storage purposes.


A problem may be in real->double conversions, because I think D feels  
free to use intermediate real values in some FP computations.


For your information, the x87 can only perform computations at 80-bits. So  
all intermediate values are performed using reals. It's just how the  
hardware works. Now I now some compilers (i.e. VS) allow you to set a  
flag, which basically causes the system to avoid intermediate values  
altogether or to use SIMD instructions instead in order to be properly  
compliant.


Another possible problem: generic code like this is going to produce an  
error because 2.0 literal is double, so x*2.0 is a double even if x is  
float:


T foo(T)(T x) {
return x * 2.0;
}

But when I use floats, I have found it good a C lint that spots  
double->float conversions, because it has actually allowed me to speed  
up some code that was doing float->double->float conversions without me  
being aware of it.




Yes, this auto-promotion of literals is very annoying, and it would be  
nice if constants could smartly match the expression type. By the way,  
C/C++ also behave this way, which has gotten me into the habit of adding f  
after all my floating point constants.

Re: Floating Point + Threads?

On Tue, 19 Apr 2011 14:18:46 -0400, Sean Kelly   
wrote:



On Apr 16, 2011, at 1:02 PM, Robert Jacques wrote:

On Sat, 16 Apr 2011 15:32:12 -0400, Walter Bright  
 wrote:



The dmd startup code (actually the C startup code) does an fninit. I  
never thought about new thread starts. So, yeah, druntime should do an  
fninit on thread creation.


The documentation I've found on fninit seems to indicate it defaults to  
64-bit precision, which means that by default we aren't seeing the  
benefit of D's reals. I'd much prefer 80-bit precision by default.


There is no option to set "80-bit precision" via the FPU control word.   
Section 8.1.5.2 of the Intel 64 SDM says the following:


"The precision-control (PC) field (bits 8 and 9 of the x87 FPU control  
word) determines the precision (64, 53, or 24 bits) of floating-point  
calculations made by the x87 FPU (see Table 8-2). The default precision  
is double extended precision, which uses the full 64-bit significand  
available with the double extended-precision floating-point format of  
the x87 FPU data registers. This setting is best suited for most  
applications, because it allows applications to take full advantage of  
the maximum precision available with the x87 FPU data registers."


So it sounds like finit/fninit does what we want.


Yes, that sounds right. Thanks for clarifying.

Re: Loss of precision errors in FP conversions


On 4/19/2011 10:42 PM, bearophile wrote:

Andrei:


Yes please. I once felt the same way, but learned better since.


OK :-) But I will probably write an article about this, because I have found a 
performance problem, that currently DMD doesn't help me avoid, that a C lint 
has avoided me.

Bye,
bearophile


...or write a Lint tool for D.

Re: Hidden argument kind antipattern