[GSOC] Container Project

[Ian Bishop]

Hi,

My name is Ian Bishop and I'm a 4th year BCS student at the University of New 
Brunswick. I'm interested
in the GSOC Project of working on containers for D. This sort of project is 
right up my alley, my major is
Theory and Computation and I have taken related courses such as data 
structures, algorithms, compiler
construction and programming language theory.

I realize there's already another student interested in working on this stuff. 
Is the project taken or is
there room for one more?

Either way, I have a few questions about the project itself:

What kind of data structures are most wanted (or needed)? There seems to be a 
lot of discussion in the
previous threads about the need for a doubly-linked list and deque. Is the 
associative array
implementation supposed to be a permanent replacement for a proper map or is 
this also required in
practice?

For more abstract types, such as stacks, are these intended to be implemented 
as concretely or using
container adapters as they are in the STL?

What about the stuff over at ? 
It's licensed under Boost,
I haven't looked at it too hard but it seems like some of this stuff would be a 
good fit for the
std.container. Is there any reason (other than lack of time/resources) that 
this hasn't been done?

Thanks for your time,

Ian

Re: CTFE Overhaul (Was: RFC: Thrift project proposal (draft))

[Andrej Mitrovic]

Ok this is the thing that really gets me with CTFE:

void printFields(T)(T t)
{
enum fields = [__traits(allMembers, T)];

foreach (string field; fields)
{
mixin("writeln(t." ~ to!string(field) ~ ");");  // fail
mixin("writeln(t." ~ to!string(fields[0]) ~ ");");  // ok
}
}

Even though the foreach loop will work, `field` can't be accessed.

Once we have that working, its heaven.

Re: GCC 4.6

[bearophile]

Christopher Bergqvist:

> Why not split this NG in two?
> d-pragmatism - Concrete stuff, TDPL + absolutely necessary adjustments
> which are probably discussed first in the other ng...
> d-theory - A place to discuss the future of D, stuff with a longer timeline.
> 
> Or maybe we should accept this NG for being a mix of both and that at
> least d-announce is d-pragmatism condensed, kind of.

I am against a further NG split, I have explained why in another post. But here 
are some more comments.

On one hand online groups of people that work to develop some software need 
some discipline and organization of their work, to improve their productivity.

On the other hand it's all voluntary service, most people don't get paid to 
help D development, so they _can't_ be managed as employed people, especially 
in a public forum designed for generic discussions about a language.

Some people are more interested in coding a lot, other people are more 
interested in talking and ideas, and so on. If such desires aren't respected, 
some people may just decrease their interest in the community, if not leave. 
And given the small size of this group, this is bad.

This is why I have said those discussions are not eating away lot of coding 
time that's usable for Phobos and GSoC: because some people are not interested 
in writing lot of Phobos code or doing GSoC, so if they get shut up, the total 
coding time/mind share for Phobos/GSoC don't increase significantly.

And from what I have seen on myself (I have written more than 100_000 lines of 
D1 code), even people that do want to write lot of code don't waste a lot of 
"coding time" on the discussions, because their brain needs rest once in a 
while while they write down code. Writing down some answers about ideas doesn't 
reduce a lot the time they spend coding.

That said, now work on the already designed/planned features, on Phobos and on 
GSoC is more important than inventing new language features for D3. Still the 
group must keep being friendly toward people that want to talk about more 
general things. There are already groups focused on DMD development, Phobos, 
etc.

Bye,
bearophile

Re: Asynchronicity in D

[Max Klyga]

On 2011-04-01 01:45:54 +0300, Jonas Drewsen said:


On 31/03/11 23.20, Max Klyga wrote:

On 2011-03-31 22:35:43 +0300, Jonas Drewsen said:


On 31/03/11 18.26, Andrei Alexandrescu wrote:

snip


I believe that we would need both the threaded async IO that you
describe but also a select based one. The thread based is important
e.g. in order to keep buffering incoming data while processing
elements in the range (the OS will only buffer the number of bytes
allowed by sysadmin). The select based is important in order to handle
_many_ connections at the same time (think D as the killer app for
websockets). As Robert mentions fibers would be nice to take into
consideration as well.

What I also see as an unresolved issue is non-blocking handling in
http://erdani.com/d/phobos/std_stream2.html which fits in naturally
with this topic I think.

I may very well agree mentoring if we get a solid proposal out of this.


I'm very glad to hear this. Now my motivation doubled!


/Jonas


Any comments, if this proposal be more focused on asyncronous networking
or should it address asyncronisity in Phobos in general?

I researched a little about libev and libevent. Both seem to have some
limitations on Windows platform.

libev can only be used to deal with sockets on Windows and uses select,
which limits libev to 64 file handles per thread.


Actually it seems the limit is OS version dependent and for NT it is 
32767 per process: http://support.microsoft.com/kb/111855


That page also mentions that actual limit is 64 by default and is 
adjustable, but requires recompilation, because it is defined in a 
macro (FD_SETSIZE).



libevent uses Windows overlaping I/O, but this thread[1] shows that
current implementation has perfomance limitations.
So one option may be to use either libev or libevent, and implement
things on top of them.
Another is to make a new implementation (from scratch, or reuse some
code from Boost.ASIO[2]) using threads or fibers, or maybe both.

1. http://www.mail-archive.com/libevent-users@monkey.org/msg01730.html
2. http://www.boost.org/doc/libs/1_46_1/doc/html/boost_asio.html

Re: shared and cryptic error messages

[Jose Armando Garcia]

On Thu, Mar 31, 2011 at 7:59 PM, Jason House
 wrote:
> Jose Armando Garcia Wrote:
>
>> On Wed, Mar 30, 2011 at 9:39 PM, Jason House
>>  wrote:
>> > Jose Armando Garcia Wrote:
>> >
>> >> How do I get around this error?
>> >
>> > That's not easy to answer...  To get the compiler to shut up, you can copy 
>> > and paste FILE's destructor and mark it as shared.  Of course, every 
>> > method call to your file will similarly required shared methods.  If you 
>> > make no copies of the file object, and all accesses are from within your 
>> > synchronized methods, you can probably cast away shared while making the 
>> > method calls.  Of course, the common theme here is that you're verifying 
>> > the multi-threaded safety of all this stuff along the way.  If you're not 
>> > comfortable doing that, then you probably want to do another design that 
>> > you can prove works.
>> >
>> >
>>
>> This is a great suggestion and that is what I do in my implementation
>> of the synchronized class but I can't cast away the call to the dtor.
>> That call is done by the compiler. I am going to look into __gshared.
>
> For one, definitely file a bug report!
>
> You might be able to mark your FILE as __gshared or make it a pointer. As a 
> pointer, you manage the lifetime, and can cast away shared before calling 
> delete, clear, or whatever.
>
> If file is using TLS globals, or is otherwise unsafe to use the way you want, 
> then you need messages passing.
>
>

I'll file a bug report when I get back to my computer and I can verify
that my memory is not failing me.

>> >> Has anyone tried to write a
>> >> multithreaded application using D?
>> >
>> > Yes, and even succeeded :)
>> >
>>
>> Which? And it it cannot be named what does it do? How many lines of code?
>>
>
> It was an open source hobby project called housebot. I don't have the LOC 
> handy, but it is a program to play the game of go. It has a shared search 
> tree and a hash tree filled with weak pointers. Control messages are done 
> through message passing. Message passing did not exist in Phobos when I wrote 
> it, and I had to take atomic increment code from Tango.
>
>

Thanks!

Re: Asynchronicity in D

[dsimcha]

== Quote from Jonas Drewsen (jdrew...@nospam.com)'s article
> On 31/03/11 23.20, Max Klyga wrote:
> > On 2011-03-31 22:35:43 +0300, Jonas Drewsen said:
> >
> >> On 31/03/11 18.26, Andrei Alexandrescu wrote:
> >>> snip
> >>
> >> I believe that we would need both the threaded async IO that you
> >> describe but also a select based one. The thread based is important
> >> e.g. in order to keep buffering incoming data while processing
> >> elements in the range (the OS will only buffer the number of bytes
> >> allowed by sysadmin). The select based is important in order to handle
> >> _many_ connections at the same time (think D as the killer app for
> >> websockets). As Robert mentions fibers would be nice to take into
> >> consideration as well.
> >>
> >> What I also see as an unresolved issue is non-blocking handling in
> >> http://erdani.com/d/phobos/std_stream2.html which fits in naturally
> >> with this topic I think.
> >>
> >> I may very well agree mentoring if we get a solid proposal out of this.
> >
> > I'm very glad to hear this. Now my motivation doubled!
> >>
> >> /Jonas
> >
> > Any comments, if this proposal be more focused on asyncronous networking
> > or should it address asyncronisity in Phobos in general?
> >
> > I researched a little about libev and libevent. Both seem to have some
> > limitations on Windows platform.
> >
> > libev can only be used to deal with sockets on Windows and uses select,
> > which limits libev to 64 file handles per thread.
> Actually it seems the limit is OS version dependent and for NT it is
> 32767 per process: http://support.microsoft.com/kb/111855

Again forgive my naiveness, as most of my experience with concurrency is
concurrency to implement parallelism, not concurrency for its own sake.  
Shouldn't
32,000 threads be more than enough for anything?  I can't imagine what kinds of
programs would really need this level of concurrency, or how bad performance on
any specific thread would be when you have this many.  Right now in my Task
Manager the program with the most threads is explorer.exe, with 28.

Re: Asynchronicity in D

[dsimcha]

== Quote from Sean Kelly (s...@invisibleduck.org)'s article
> On Mar 31, 2011, at 11:48 AM, dsimcha wrote:
> > == Quote from Andrej Mitrovic (andrej.mitrov...@gmail.com)'s
> article
> >> Are fibers really better/faster than threads? I've heard rumors that
> >> they perform exactly the same, and that there's no benefit of using
> >> fibers over threads. Is that true?
> >
> > Here are some key differences between fibers (as currently implemented
> in
> > core.thread; I have no idea how this applies to the general case in
> other
> > languages) and threads:
> >
> > 1.  Fibers can't be used to implement parallelism.  If you have N > 1
> fibers
> > running on one hardware thread, your code will only use a single core.
> It bears mentioning that this has interesting implications for the
> default thread-local storage of statics.  All fibers running on a thread
> will currently share the thread's static data.  This could be worked
> around by doing TLS manually at the fiber level, but it's a non-trivial
> change.

Let's assume for the sake of argument that we are otherwise ready to make said
change.  What would the performance implications of this be for programs using 
TLS
heavily but not using fibers?  My gut feeling is that, if this has considerable
performance implications for non-fiber-using programs, it should be left alone
long-term, or fiber-local storage should be added as a separate entity.

std.event

[teo]

https://github.com/teo-/phobos/tree/std-event

Instead of changing std.signals I decided to insert a new module and call 
it std.event.

There is a problem though: The observers are notified via delegates when 
the event is fired. And if an observer is destroyed the event will be 
left with a dangling reference to that observer.

I can think of two workarounds:
  1. all observers must be disconnected from the event before destroying 
them
  2. asking the runtime via a hook to inform the event about the 
destruction of an object which has registered itself as an observer

The second workaround works only with class objects, but I cannot decide 
given a delegate whether I am dealing with a class or with something 
else. The std.signals module imposes the limitation to construct 
delegates only from classes. In contrast std.event has no such a 
limitation. I've given an example with a structure. However the problem 
with dangling references remains.

So, I would like to hear your comments about the module.

Re: shared and cryptic error messages

[Jason House]

Jose Armando Garcia Wrote:

> On Wed, Mar 30, 2011 at 9:39 PM, Jason House
>  wrote:
> > Jose Armando Garcia Wrote:
> >
> >> How do I get around this error?
> >
> > That's not easy to answer...  To get the compiler to shut up, you can copy 
> > and paste FILE's destructor and mark it as shared.  Of course, every method 
> > call to your file will similarly required shared methods.  If you make no 
> > copies of the file object, and all accesses are from within your 
> > synchronized methods, you can probably cast away shared while making the 
> > method calls.  Of course, the common theme here is that you're verifying 
> > the multi-threaded safety of all this stuff along the way.  If you're not 
> > comfortable doing that, then you probably want to do another design that 
> > you can prove works.
> >
> >
> 
> This is a great suggestion and that is what I do in my implementation
> of the synchronized class but I can't cast away the call to the dtor.
> That call is done by the compiler. I am going to look into __gshared.

For one, definitely file a bug report!

You might be able to mark your FILE as __gshared or make it a pointer. As a 
pointer, you manage the lifetime, and can cast away shared before calling 
delete, clear, or whatever.

If file is using TLS globals, or is otherwise unsafe to use the way you want, 
then you need messages passing.


> >> Has anyone tried to write a
> >> multithreaded application using D?
> >
> > Yes, and even succeeded :)
> >
> 
> Which? And it it cannot be named what does it do? How many lines of code?
> 

It was an open source hobby project called housebot. I don't have the LOC 
handy, but it is a program to play the game of go. It has a shared search tree 
and a hash tree filled with weak pointers. Control messages are done through 
message passing. Message passing did not exist in Phobos when I wrote it, and I 
had to take atomic increment code from Tango.

Re: Asynchronicity in D

[Jonas Drewsen]

On 31/03/11 23.20, Max Klyga wrote:

On 2011-03-31 22:35:43 +0300, Jonas Drewsen said:


On 31/03/11 18.26, Andrei Alexandrescu wrote:

snip


I believe that we would need both the threaded async IO that you
describe but also a select based one. The thread based is important
e.g. in order to keep buffering incoming data while processing
elements in the range (the OS will only buffer the number of bytes
allowed by sysadmin). The select based is important in order to handle
_many_ connections at the same time (think D as the killer app for
websockets). As Robert mentions fibers would be nice to take into
consideration as well.

What I also see as an unresolved issue is non-blocking handling in
http://erdani.com/d/phobos/std_stream2.html which fits in naturally
with this topic I think.

I may very well agree mentoring if we get a solid proposal out of this.


I'm very glad to hear this. Now my motivation doubled!


/Jonas


Any comments, if this proposal be more focused on asyncronous networking
or should it address asyncronisity in Phobos in general?

I researched a little about libev and libevent. Both seem to have some
limitations on Windows platform.

libev can only be used to deal with sockets on Windows and uses select,
which limits libev to 64 file handles per thread.


Actually it seems the limit is OS version dependent and for NT it is 
32767 per process: http://support.microsoft.com/kb/111855



libevent uses Windows overlaping I/O, but this thread[1] shows that
current implementation has perfomance limitations.
So one option may be to use either libev or libevent, and implement
things on top of them.
Another is to make a new implementation (from scratch, or reuse some
code from Boost.ASIO[2]) using threads or fibers, or maybe both.

1. http://www.mail-archive.com/libevent-users@monkey.org/msg01730.html
2. http://www.boost.org/doc/libs/1_46_1/doc/html/boost_asio.html

Re: Asynchronicity in D

[Sean Kelly]

On Mar 31, 2011, at 11:48 AM, dsimcha wrote:

> == Quote from Andrej Mitrovic (andrej.mitrov...@gmail.com)'s article
>> Are fibers really better/faster than threads? I've heard rumors that
>> they perform exactly the same, and that there's no benefit of using
>> fibers over threads. Is that true?
> 
> Here are some key differences between fibers (as currently implemented in
> core.thread; I have no idea how this applies to the general case in other
> languages) and threads:
> 
> 1.  Fibers can't be used to implement parallelism.  If you have N > 1 fibers
> running on one hardware thread, your code will only use a single core.

It bears mentioning that this has interesting implications for the default 
thread-local storage of statics.  All fibers running on a thread will currently 
share the thread's static data.  This could be worked around by doing TLS 
manually at the fiber level, but it's a non-trivial change.

Re: Asynchronicity in D

[Max Klyga]

On 2011-03-31 22:35:43 +0300, Jonas Drewsen said:


On 31/03/11 18.26, Andrei Alexandrescu wrote:

snip


I believe that we would need both the threaded async IO that you 
describe but also a select based one. The thread based is important 
e.g. in order to keep buffering incoming data while processing elements 
in the range (the OS will only buffer the number of bytes allowed by 
sysadmin). The select based is important in order to handle _many_ 
connections at the same time (think D as the killer app for 
websockets). As Robert mentions fibers would be nice to take into 
consideration as well.


What I also see as an unresolved issue is non-blocking handling in 
http://erdani.com/d/phobos/std_stream2.html which fits in naturally 
with this topic I think.


I may very well agree mentoring if we get a solid proposal out of this.


I'm very glad to hear this. Now my motivation doubled!


/Jonas


Any comments, if this proposal be more focused on asyncronous 
networking or should it address asyncronisity in Phobos in general?


I researched a little about libev and libevent. Both seem to have some 
limitations on Windows platform.


libev can only be used to deal with sockets on Windows and uses select, 
which limits libev to 64 file handles per thread.
libevent uses Windows overlaping I/O, but this thread[1] shows that 
current implementation has perfomance limitations.


So one option may be to use either libev or libevent, and implement 
things on top of them.
Another is to make a new implementation (from scratch, or reuse some 
code from Boost.ASIO[2]) using threads or fibers, or maybe both.


1. http://www.mail-archive.com/libevent-users@monkey.org/msg01730.html
2. http://www.boost.org/doc/libs/1_46_1/doc/html/boost_asio.html

Re: Asynchronicity in D

[Max Klyga]

On 2011-03-31 19:26:45 +0300, Andrei Alexandrescu said:


On 3/31/11 6:35 AM, Max Klyga wrote:

snip


I think that would be a good contribution that would complement Jonas'. 
You'll need to discuss cooperation with him and at best Jonas would 
agree to become a mentor.


Jonas agreed to become a mentor if I make this proposal 
strong/interesting enough.


I've posted a couple of weeks earlier how I think that could work with 
ranges: the range maintains the asynchronous state and has a queue of 
already-available buffers received. The network traffic occurs in a 
different thread; the range throws requests over the fence to libcurl 
and libcurl throws buffers over the fence back to the range. The range 
offers a seemingly synchronous interface:


foreach (line; byLineAsync("http://d-programming-language.org";))
{
... use line ...
}

except that the processing and the fetching of data occur in distinct threads.


I thought about the same.


Server-side code such as network servers etc. would also be an 
interesting topic. Let me know if you're versed in the likes of 
libev(ent).


I have no experience with libev/libevent, but I see no problem working 
with either, after reading some exapmles or/and documentation.

Re: [GSOC idea] enhance regular expressions?

[Dmitry Olshansky]

On 31.03.2011 22:16, petevi...@yahoo.com.au wrote:

Dmitry Olshansky  writes:

--- somewhat informal draft ---


Hi Dmitry,

It's good to know that there is interest in improving regular
expressions in the D standard library. I've run into a number of
problems using std.regex myself, so I'm keen so see either fixes for
std.regex or an improved replacement.

Thanks for the interest.
Indeed there are problems, though the engine itself is quite capable and 
optimized, I think it could be fixed.

I've been working on a regular expression library myself based around
Russ Cox article at http://swtch.com/~rsc/regexp/regexp2.html and the
std.regex range interface. It includes a backtracking and Thompson
engine. If you are interested it is on github:

https://github.com/PeteChadwick/pcregexeng

The ddoc documentation is here:
http://petechadwick.github.com/pcregexeng/


Wow, I'll definitely look into it.
Overall it seems solid even though incomplete. From glance it seems that 
parsing a pattern is somewhat convulted, but I'll defer any critics or 
decisions until I looked in closer detail.
I see you also using the test from std.regex, sadly it's somewhat broken 
(the one with test vectors).

All it's checking proves that *there is a match*, not *what* was matched.

Anyway, I'm gathering what I have in a way of fixes as my first pull 
request, wish me luck ;)

Having regular expression (regex) engine in standard library is
totally expected for any modern language.
The solid and performant implementation of regex is one of the
greatest points of pride (if not of sale).

Couldn't agree more.


While concrete results and benchmarks got frequently biased, there are
some general conclusions:
regexes usually came in two flavors: Finite Automation (FA, be it
deterministic  or nondeterministic) and  backtracking, each of these
with their set of traits.
FA are more stable O(N) in time on input, however they are usually
more limited as implementing features such as backreferences becomes
problematic, also not to mention the time for constructing DFA from
pattern.

For an NFA, the complexity of the regular expression also affects the
execution time as there will probably me more states to process per
character. There is also more storage and copying required for
captures. I haven't looked closely at using a DFA, and I'm not sure how
captures would work with one.

Right, for NFA the execution time in worst case O(n*m) n - input, m - 
pattern length.
As for DFA, it seems the process of construction full DFA eagerly from 
NFA would grab a lot of ram & time O(2^^m) in worst case, which makes it 
useful only on certain amounts of input, hence the idea to compute it at 
compile time.



Backtracking allows easily implementing some interesting features like
backreferencing, lookahead/lookbehind and such, but has a dreadful
exponential time on input in worst case.
Not willing to go deeper into the performance/feature/usage topic,
fairly good picture of various regular expression engines can be found
here:  http://lh3lh3.users.sourceforge.net/reb.shtml. I though have
now idea why he claims RE2 is feature-rich.

Now speaking of D, what we have now is essentially an optimized
backtracking with some not implemented features it may very well have
had. Also consider important issue that need proper resolution
http://d.puremagic.com/issues/show_bug.cgi?id=941. This situation begs
for a change.

The proposal consists of three parts:

1) Enhance and bugfix the existing engine in std.regex. Sometime ago I
got accustomed with this engine internals and confident that I can get
it done. That's a short-term solution at best and something I plan to
do in spare time even if it's not considered a compelling GSOC
proposal ;)

I would really like to see this.


2)  Make another one FA-based from scratch, here things go somewhat
speculative.
I plan to start with NFA, leaving DFA on later. Probably NFA with
cached DFA states as optimization tuning aka memory vs time. Someone
brought  http://swtch.com/~rsc/regexp/regexp1.html, it could be a
starting point, I see some optimization opportunities I'd like to
try. Supporting unicode of all flavors is of no question. The API for
this engine may stay the same or it even  may be integrated into
existing one, making the type of engine a compile-time parameter with
default being Regex.Backtracking. Suppose:
auto dfa = regex!Regex.NFA("(D|N)?FA","g");

I think the existing range based API is very good. I also like the idea
of the regular expression engine being a compile time parameter.


3) When I saw Don's comments on fixing CTFE, I remembered something D
can do _better_  than most others mainstream compiled languages -
metaprogramming. So having a StaticRegex compiled at compile-time
(inevitable calambour) would give us an  edge over comparative
implementations. Also I can imagine quite a lot of applications not
requiring *any* regex from non-constant pattern. Major exceptions are
editors with find/replace and c

Re: GCC 4.6

[Christopher Bergqvist]

Why not split this NG in two?
d-pragmatism - Concrete stuff, TDPL + absolutely necessary adjustments
which are probably discussed first in the other ng...
d-theory - A place to discuss the future of D, stuff with a longer timeline.

Or maybe we should accept this NG for being a mix of both and that at
least d-announce is d-pragmatism condensed, kind of.

On Thu, Mar 31, 2011 at 10:26 AM, so  wrote:
> On Thu, 31 Mar 2011 05:09:44 +0300, jasonw  wrote:
>
>> You hit the nail on the head here. I see two real problems with his
>> messages:
>>
>> 1) he's "force fitting" every possible language feature he learns into D.
>> Clearly some features are useful, others are not, and this is why many of
>> bearophile's ideas fail and generate endless debates and unnecessary noise.
>> He can't see that the features just don't fit in.
>
> This is not true, there are ideas here from many others as well that
> generate endless debates.
> The reason is as far as i can see not always the ideas "just don't fit in".
> The reasons IMO are the chain of command and the resources.
> Take the last long discussion on named arguments, i don't think anyone was
> against it.
> One another thing is that a few of us evasive to some questions.
>
>> If you lack the vision of good language design as a whole, you shouldn't
>> start suggesting new features like this. I'd appreciate it more if we won't
>> introduce "new" concepts in this way.
>
> This is oxymoron, by that logic there is not a single soul on earth with
> that vision.
> You just dismissed whole academia, isn't this the way it operates?
> Don't you think this is harmful? Why does D2 exist? D1 wasn't enough?
>
>> 2) Programming language design requires rigorous definition of terms and
>> other things. The D community doesn't encourage using precise, well-defined,
>> unique terms. This leads to some subtleties and other problems in the
>> discussions. Again I think the best place for general PL discussion is
>> somewhere else, preferable in the academia. I'm sorry to say this, but I
>> likely need to study how to put him in the kill file. The whole bearophile
>> phenomenon takes place on an isolated island somewhere in the dark corners
>> of D's history. The bug reports and benchmarks are priceless, but these
>> "lectures" about other language often aren't.
>
> I agree he should slow down proposing, at the same time people better stop
> ad hominem attacks.
> If it is way to go, we all need to shut up.
>

Re: [GSoC Proposal draft] High-Level Networking

[Jonas Drewsen]

On 30/03/11 23.58, Max Klyga wrote:

I read your "Curl RFC" thread, great job. Somehow I missed it when it
was originaly posted.

As you stated earlier, you plan to continue working on your curl
wrapper, so to boost my proposal usefullness I think it needs to address
unsolved problems.

Could you please be more specific about your plans on networking
modules? This information might help me make a stronger proposal.


I written about some of this in the other thread you've started 
"Asynchronicity in D".



Thank you.

Things that phobos currently lacks, but would benefit from:
- Tcp Server - a class implementing TCP-based server, handling incoming
connections. (QTcpServer[1] ot Net::TcpServer[2] might act as a good
reference)
Also have a look at the C# .net networking library and the boost asio 
library.



- Application level protocol servers.
Especially HTTP but I guess you knew that. A framework for such servers 
in general would be very nice.


/Jonas


References
1. http://doc.qt.nokia.com/latest/qtcpserver.html#details
2. http://www.ruby-doc.org/stdlib/libdoc/socket/rdoc/classes/TCPServer.html

Re: Asynchronicity in D

[Jonas Drewsen]

On 31/03/11 21.19, Torarin wrote:

I'm currently working on an http and networking library that uses
asynchronous sockets running in fibers and an event loop a la libev.
These async sockets have the same interface as regular Berkeley
sockets, so clients can choose whether to be synchronous, asynchronous
or threaded with template arguments.

For instance, it has HttpClient!AsyncSocket and HttpClient!Socket.

Torarin


Very interesting! Do you have a github repos we can see?

/Jonas

Re: Asynchronicity in D

[Jonas Drewsen]

On 31/03/11 20.48, dsimcha wrote:

== Quote from Andrej Mitrovic (andrej.mitrov...@gmail.com)'s article

Are fibers really better/faster than threads? I've heard rumors that
they perform exactly the same, and that there's no benefit of using
fibers over threads. Is that true?


Here are some key differences between fibers (as currently implemented in
core.thread; I have no idea how this applies to the general case in other
languages) and threads:

1.  Fibers can't be used to implement parallelism.  If you have N>  1 fibers
running on one hardware thread, your code will only use a single core.


The fastest webservers out there (e.g. zeus, nginx, lighttpd) also use 
some kind of fibers and they solve this problem by simply forking the 
process and sharing the listening socket between processes. That way you 
get the best of two worlds.


/Jonas


2.  Fibers use cooperative concurrency, threads use preemptive concurrency.  
This
means three things:

 a.  It's the programmer's responsibility to determine how execution time is
split between a group of fibers, not the OS's.

 b.  If one fiber goes into an endless loop, all fibers executing on that
thread will hang.

 c.  Getting concurrency issues right is easier, since fibers can't be
implicitly pre-empted by other fibers in the middle of some operation.  All
context switches are explicit, and as mentioned there is no true parallelism.

3.  Fibers are implemented in userland, and context switches are a lot cheaper
(IIRC an order of magnitude or more, on the order of 100 clock cycles for fibers
vs. 1000 for OS threads).

Re: Asynchronicity in D

[Jonas Drewsen]

On 31/03/11 18.26, Andrei Alexandrescu wrote:

On 3/31/11 6:35 AM, Max Klyga wrote:

I've been thinking on things I can change in my GSoC proposal to make it
stronger and noticed that currently Phobos does not address asynchronous
I/O of any kind.

A number of threads on thid newsgroup mentioned about this problem or
shown ways other languages address asynchronicity.

I want to ask D community about plans on asynchronicity in Phobos.
Did somenone in Phobos team thought about possible design?
How does asynchronicity stacks with ranges?
What model should D adapt?
etc.


I think that would be a good contribution that would complement Jonas'.
You'll need to discuss cooperation with him and at best Jonas would
agree to become a mentor.

I've posted a couple of weeks earlier how I think that could work with
ranges: the range maintains the asynchronous state and has a queue of
already-available buffers received. The network traffic occurs in a
different thread; the range throws requests over the fence to libcurl
and libcurl throws buffers over the fence back to the range. The range
offers a seemingly synchronous interface:

foreach (line; byLineAsync("http://d-programming-language.org";))
{
... use line ...
}

except that the processing and the fetching of data occur in distinct
threads.

Server-side code such as network servers etc. would also be an
interesting topic. Let me know if you're versed in the likes of libev(ent).


Thanks,

Andrei


I believe that we would need both the threaded async IO that you 
describe but also a select based one. The thread based is important e.g. 
in order to keep buffering incoming data while processing elements in 
the range (the OS will only buffer the number of bytes allowed by 
sysadmin). The select based is important in order to handle _many_ 
connections at the same time (think D as the killer app for websockets). 
As Robert mentions fibers would be nice to take into consideration as well.


What I also see as an unresolved issue is non-blocking handling in 
http://erdani.com/d/phobos/std_stream2.html which fits in naturally with 
this topic I think.


I may very well agree mentoring if we get a solid proposal out of this.

/Jonas

Re: shared and cryptic error messages

[Jose Armando Garcia]

On Wed, Mar 30, 2011 at 9:39 PM, Jason House
 wrote:
> Jose Armando Garcia Wrote:
>
>> Why am I getting this error? I suspect that synchronized is the
>> problem.
>
> A synchronized class is implicitly shared and most of the methods are 
> synchronized.  I say most because at a minimum, the constructor isn't 
> synchronized on anything.  As you probably know shared is transitive, so all 
> members are implicitly shared as well.
>
>
>> How do I get around this error?
>
> That's not easy to answer...  To get the compiler to shut up, you can copy 
> and paste FILE's destructor and mark it as shared.  Of course, every method 
> call to your file will similarly required shared methods.  If you make no 
> copies of the file object, and all accesses are from within your synchronized 
> methods, you can probably cast away shared while making the method calls.  Of 
> course, the common theme here is that you're verifying the multi-threaded 
> safety of all this stuff along the way.  If you're not comfortable doing 
> that, then you probably want to do another design that you can prove works.
>
>

This is a great suggestion and that is what I do in my implementation
of the synchronized class but I can't cast away the call to the dtor.
That call is done by the compiler. I am going to look into __gshared.

>> Does this mean that
>> synchronized classes are not allowed to have as member
>> unsynchronized/regular classes?
>
> As I said above, shared is transitive.  It's implicit that all members are 
> shared even though you don't mark them as such.  No shared objects can have 
> non-shared members.  If there are parts that are not accessed by more than 
> one thread, you can either restructure your code a bit, or make liberal use 
> of casts.
>
>
>> Has anyone tried to write a
>> multithreaded application using D?
>
> Yes, and even succeeded :)
>
>

Which? And it it cannot be named what does it do? How many lines of code?

>> Why is the error showing up in
>> std/stdio.d and not in my file?
>
> It's an awful error message.  I complained about it back in 2009: 
> http://d.puremagic.com/issues/show_bug.cgi?id=3642
>
> When this kind of thing pops up, it's either misusing the object (shouldn't 
> be shared), or the object is missing a shared method.  The error is really 
> awful.  After finding this case, I spotted a few others.  This error message 
> is extremely generic and used in many other cases.  In more mature areas of 
> the code, the compiler issues a much more helpful message on the very next 
> line.  For shared, there's no specialized error message and it leaves the 
> user scratching their head...
>
>
>> Can anyone point me to a decent
>> documentation of how "shared" works? I have read chapter 13 of The D
>> Programming Language and it is not sufficient for the level of
>> understanding I seek.
>
> Nope.  D is notorious for having no specs or other documentation to give a 
> detailed understanding.  I would have assumed TDPL was the best place.  (I 
> don't have a copy)
>
>
>> That is a lot of question. Hopefully I get answer to at least one.
>
>

Re: shared and cryptic error messages

[Jose Armando Garcia]

Adding a shared dtor to File doesn't sound like the correct solution.

On Wed, Mar 30, 2011 at 8:14 PM, KennyTM~  wrote:
> On Mar 31, 11 06:34, Jason House wrote:
>>
>> he compiler wants "argument types () shared" instead of "argument types
>> ()". It's an awful error message, and I'm certain I filed a bug for it at
>> least a year ago. In the toy example, mark the destructor as shared, and it
>> should compile.
>
> Yes you have. http://d.puremagic.com/issues/show_bug.cgi?id=3642
>
> Your solution works only if OP wills to dig into Phobos and add a shared
> destructor to File.
>

Re: Asynchronicity in D

[Torarin]

I'm currently working on an http and networking library that uses
asynchronous sockets running in fibers and an event loop a la libev.
These async sockets have the same interface as regular Berkeley
sockets, so clients can choose whether to be synchronous, asynchronous
or threaded with template arguments.

For instance, it has HttpClient!AsyncSocket and HttpClient!Socket.

Torarin

Re: shared and cryptic error messages

[Jose Armando Garcia]

First of all, thanks for the reply.

That means that it is imposible to use a struct or class that
overrides ~this() inside of a synchronized class. As long as I take
the extra care of making sure that I don't expose an object of this
class/struct outside of the synchronized class. It is impossible
because the compiler is responsible of calling ~this() so how can I
cast it to a non-shared object?

I really like D and its design but I it is hard to use in its current
state. At this point I am only writing toy applications and have
encounter many bugs. For example the following segfaults:

File[int] writers;
writers[1] = File("filename", "w");

I don't have the code infront of me but it segfaults in File.opAssign.
It makes sense that opAssign gets called on 'writers[1] =' but after
investigation, 'writers[1]' was never fully constructed (default ctor
was never called).

On Wed, Mar 30, 2011 at 7:34 PM, Jason House
 wrote:
> he compiler wants "argument types () shared" instead of "argument types ()". 
> It's an awful error message, and I'm certain I filed a bug for it at least a 
> year ago. In the toy example, mark the destructor as shared, and it should 
> compile.
>
> Jose Armando Garcia Wrote:
>
>> It looks like the following works:
>>
>> struct B {}
>> synchronized class A { private B b }
>>
>> but this doesn't:
>>
>> struct B { ~this() {} }
>> synchronized class A { private B b }
>>
>> On Wed, Mar 30, 2011 at 4:59 PM, Jose Armando Garcia  
>> wrote:
>> > import std.stdio;
>> >
>> > class B
>> > {
>> >  private File file;
>> > }
>> > synchronized class A
>> > {
>> >  private File file;
>> > }
>> >
>> > void main()
>> > {
>> > }
>> >
>> > /usr/include/d/dmd/phobos/std/stdio.d(292): Error: destructor
>> > std.stdio.File.~this () is not callable using argument types ()
>> >
>> > Why am I getting this error? I suspect that synchronized is the
>> > problem. How do I get around this error? Does this mean that
>> > synchronized classes are not allowed to have as member
>> > unsynchronized/regular classes? Has anyone tried to write a
>> > multithreaded application using D? Why is the error showing up in
>> > std/stdio.d and not in my file? Can anyone point me to a decent
>> > documentation of how "shared" works? I have read chapter 13 of The D
>> > Programming Language and it is not sufficient for the level of
>> > understanding I seek.
>> >
>> > That is a lot of question. Hopefully I get answer to at least one.
>> >
>
>

Re: Asynchronicity in D

[Steven Schveighoffer]

On Thu, 31 Mar 2011 14:48:13 -0400, dsimcha  wrote:


== Quote from Andrej Mitrovic (andrej.mitrov...@gmail.com)'s article

Are fibers really better/faster than threads? I've heard rumors that
they perform exactly the same, and that there's no benefit of using
fibers over threads. Is that true?


Here are some key differences between fibers (as currently implemented in
core.thread; I have no idea how this applies to the general case in other
languages) and threads:

1.  Fibers can't be used to implement parallelism.  If you have N > 1  
fibers

running on one hardware thread, your code will only use a single core.

2.  Fibers use cooperative concurrency, threads use preemptive  
concurrency.  This

means three things:

a.  It's the programmer's responsibility to determine how execution  
time is

split between a group of fibers, not the OS's.

b.  If one fiber goes into an endless loop, all fibers executing on  
that

thread will hang.

c.  Getting concurrency issues right is easier, since fibers can't be
implicitly pre-empted by other fibers in the middle of some operation.   
All
context switches are explicit, and as mentioned there is no true  
parallelism.


3.  Fibers are implemented in userland, and context switches are a lot  
cheaper
(IIRC an order of magnitude or more, on the order of 100 clock cycles  
for fibers

vs. 1000 for OS threads).


4. often there is an OS limit on how many threads a process can create.   
There is no such limit on fibers (only memory).  Using fibers can increase  
the number of simultaneous tasks that can be run by quite a bit.


-Steve

Re: Asynchronicity in D

[dsimcha]

== Quote from Andrej Mitrovic (andrej.mitrov...@gmail.com)'s article
> Are fibers really better/faster than threads? I've heard rumors that
> they perform exactly the same, and that there's no benefit of using
> fibers over threads. Is that true?

Here are some key differences between fibers (as currently implemented in
core.thread; I have no idea how this applies to the general case in other
languages) and threads:

1.  Fibers can't be used to implement parallelism.  If you have N > 1 fibers
running on one hardware thread, your code will only use a single core.

2.  Fibers use cooperative concurrency, threads use preemptive concurrency.  
This
means three things:

a.  It's the programmer's responsibility to determine how execution time is
split between a group of fibers, not the OS's.

b.  If one fiber goes into an endless loop, all fibers executing on that
thread will hang.

c.  Getting concurrency issues right is easier, since fibers can't be
implicitly pre-empted by other fibers in the middle of some operation.  All
context switches are explicit, and as mentioned there is no true parallelism.

3.  Fibers are implemented in userland, and context switches are a lot cheaper
(IIRC an order of magnitude or more, on the order of 100 clock cycles for fibers
vs. 1000 for OS threads).

Re: Asynchronicity in D

[Andrej Mitrovic]

Are fibers really better/faster than threads? I've heard rumors that
they perform exactly the same, and that there's no benefit of using
fibers over threads. Is that true?

Re: [GSOC idea] enhance regular expressions?

[petevik38]

Dmitry Olshansky  writes:
> --- somewhat informal draft ---
>

Hi Dmitry,

It's good to know that there is interest in improving regular
expressions in the D standard library. I've run into a number of
problems using std.regex myself, so I'm keen so see either fixes for
std.regex or an improved replacement.

I've been working on a regular expression library myself based around
Russ Cox article at http://swtch.com/~rsc/regexp/regexp2.html and the
std.regex range interface. It includes a backtracking and Thompson
engine. If you are interested it is on github:

https://github.com/PeteChadwick/pcregexeng

The ddoc documentation is here:
http://petechadwick.github.com/pcregexeng/

> Having regular expression (regex) engine in standard library is
> totally expected for any modern language.
> The solid and performant implementation of regex is one of the
> greatest points of pride (if not of sale).

Couldn't agree more.

> While concrete results and benchmarks got frequently biased, there are
> some general conclusions:
> regexes usually came in two flavors: Finite Automation (FA, be it
> deterministic  or nondeterministic) and  backtracking, each of these
> with their set of traits.
> FA are more stable O(N) in time on input, however they are usually
> more limited as implementing features such as backreferences becomes
> problematic, also not to mention the time for constructing DFA from
> pattern.

For an NFA, the complexity of the regular expression also affects the
execution time as there will probably me more states to process per
character. There is also more storage and copying required for
captures. I haven't looked closely at using a DFA, and I'm not sure how
captures would work with one.

> Backtracking allows easily implementing some interesting features like
> backreferencing, lookahead/lookbehind and such, but has a dreadful
> exponential time on input in worst case.
> Not willing to go deeper into the performance/feature/usage topic,
> fairly good picture of various regular expression engines can be found
> here:  http://lh3lh3.users.sourceforge.net/reb.shtml. I though have
> now idea why he claims RE2 is feature-rich.
>
> Now speaking of D, what we have now is essentially an optimized
> backtracking with some not implemented features it may very well have
> had. Also consider important issue that need proper resolution
> http://d.puremagic.com/issues/show_bug.cgi?id=941. This situation begs
> for a change.
>
> The proposal consists of three parts:
>
> 1) Enhance and bugfix the existing engine in std.regex. Sometime ago I
> got accustomed with this engine internals and confident that I can get
> it done. That's a short-term solution at best and something I plan to
> do in spare time even if it's not considered a compelling GSOC
> proposal ;)

I would really like to see this.

> 2)  Make another one FA-based from scratch, here things go somewhat
> speculative.
> I plan to start with NFA, leaving DFA on later. Probably NFA with
> cached DFA states as optimization tuning aka memory vs time. Someone
> brought  http://swtch.com/~rsc/regexp/regexp1.html, it could be a
> starting point, I see some optimization opportunities I'd like to
> try. Supporting unicode of all flavors is of no question. The API for
> this engine may stay the same or it even  may be integrated into
> existing one, making the type of engine a compile-time parameter with
> default being Regex.Backtracking. Suppose:
> auto dfa = regex!Regex.NFA("(D|N)?FA","g");

I think the existing range based API is very good. I also like the idea
of the regular expression engine being a compile time parameter.

> 3) When I saw Don's comments on fixing CTFE, I remembered something D
> can do _better_  than most others mainstream compiled languages - 
> metaprogramming. So having a StaticRegex compiled at compile-time
> (inevitable calambour) would give us an  edge over comparative
> implementations. Also I can imagine quite a lot of applications not
> requiring *any* regex from non-constant pattern. Major exceptions are
> editors with find/replace and certain scripting/parsing toolkits.
> It looks quite natural:
> ...
> enum sregNum= StaticRegex!"[-+]?[0-9]*\.?[0-9]+([eE][-+]?[0-9]+)";
> ...
> assert(match("PI: 3.1415926",sregNum).hit == "3.1415926");
>
> The best speed gain most likely would be obtained by generating the
> whole DFA at compile time, I believe such possibility  was previously
> discussed on the NG.

This would be a real nice feature.

> This diversity in kinds of regexes may warrant making regex engine an
> interface ( for homogeneous storage and parameter passing) with
> concrete implementations being classes.
>
> Another thing to sort out is style: are we sticking with ECMA or
> switching to Perl / GNU? Shall we provide also different versions of
> syntax?

Re: D grammar overhaul

[Jonathan M Davis]

On 2011-03-31 11:04, Brad Roberts wrote:
> On 3/31/2011 10:34 AM, Jonathan M Davis wrote:
> > On 2011-03-31 02:32, Trass3r wrote:
> >> Rainer Schuetze Wrote:
> >>> How can I contribute patches? I guess this would be with pull
> >>> requests, so I'll need a github repository, too? The discussion page
> >>> for pull requests looks like a nice feature...
> >> 
> >> Yep, you fork the original repo and apply your changes. Once you think
> >> you're somewhat done you can open a pull request. I think you can even
> >> comment on single lines in commits.
> >> 
> >> The advantage of this approach is that your changes don't appear as a
> >> single huge diff but as atomic changesets (as long as the author groups
> >> changes that belong together in single commits)
> > 
> > I would point out though that pull requests are done by branch, so if you
> > have commits that you don't want to be grouped with others or put in the
> > pull request, then they need to be on a separate branch. If you go and
> > make several commits and then only want some of them pulled, that
> > doesn't work so well. It's technically feasible via git, but it's more
> > of a pain, and it's not how github itself manages the process. So,
> > typically, you'd make a set of changes on a branch created for those
> > changes - be they one commit or several. Then you do a pull request on
> > that branch. Unrelated changes would be on other branches.
> > 
> > - Jonathan M Davis
> 
> Um.. no.  Github allows you to pick both a subrange of a branch.  Take a
> look at the help docs:
> 
>http://help.github.com/pull-requests/

I stand corrected. From everything that I've seen (or remembered - I'm not 
sure if I'd read that page before or not), it always takes the whole branch, 
but apparently, you can tell it to do otherwise. That's certainly not the 
default though. Then again, I wouldn't generally be very comfortable with 
someone grabbing just portions of a branch like that. I'm too paranoid about 
likely breaks or merge conflicts, I guess. Still, good to know.

- Jonathan M Davis

Re: D grammar overhaul

[Brad Roberts]

On 3/31/2011 10:34 AM, Jonathan M Davis wrote:
> On 2011-03-31 02:32, Trass3r wrote:
>> Rainer Schuetze Wrote:
>>> How can I contribute patches? I guess this would be with pull
>>> requests, so I'll need a github repository, too? The discussion page for
>>> pull requests looks like a nice feature...
>>
>> Yep, you fork the original repo and apply your changes. Once you think
>> you're somewhat done you can open a pull request. I think you can even
>> comment on single lines in commits.
>>
>> The advantage of this approach is that your changes don't appear as a
>> single huge diff but as atomic changesets (as long as the author groups
>> changes that belong together in single commits)
> 
> I would point out though that pull requests are done by branch, so if you 
> have 
> commits that you don't want to be grouped with others or put in the pull 
> request, then they need to be on a separate branch. If you go and make 
> several 
> commits and then only want some of them pulled, that doesn't work so well. 
> It's technically feasible via git, but it's more of a pain, and it's not how 
> github itself manages the process. So, typically, you'd make a set of changes 
> on a branch created for those changes - be they one commit or several. Then 
> you do a pull request on that branch. Unrelated changes would be on other 
> branches.
> 
> - Jonathan M Davis

Um.. no.  Github allows you to pick both a subrange of a branch.  Take a look 
at the help docs:

   http://help.github.com/pull-requests/

Re: __traits so long and ugly, what about ::?

[Jonathan M Davis]

On 2011-03-31 06:35, Kagamin wrote:
> Ary Manzana Wrote:
> > I just hate it when you have to write too much
> 
> hasMember!(S, "m")
> is only 1 character longer than
> S::hasMember("m")
> 
> not too much for me

And hasMember!(S, "m") is actually consistent with the rest of the language 
and straight forward to read for those who know the language. 
S::hasMember("m") just adds more syntax where there's no need for it, and 
makes it so that there's that much more syntax to learn and keep straight.

- Jonathan M Davis

Re: D grammar overhaul

[Jonathan M Davis]

On 2011-03-31 02:32, Trass3r wrote:
> Rainer Schuetze Wrote:
> > How can I contribute patches? I guess this would be with pull
> > requests, so I'll need a github repository, too? The discussion page for
> > pull requests looks like a nice feature...
> 
> Yep, you fork the original repo and apply your changes. Once you think
> you're somewhat done you can open a pull request. I think you can even
> comment on single lines in commits.
> 
> The advantage of this approach is that your changes don't appear as a
> single huge diff but as atomic changesets (as long as the author groups
> changes that belong together in single commits)

I would point out though that pull requests are done by branch, so if you have 
commits that you don't want to be grouped with others or put in the pull 
request, then they need to be on a separate branch. If you go and make several 
commits and then only want some of them pulled, that doesn't work so well. 
It's technically feasible via git, but it's more of a pain, and it's not how 
github itself manages the process. So, typically, you'd make a set of changes 
on a branch created for those changes - be they one commit or several. Then 
you do a pull request on that branch. Unrelated changes would be on other 
branches.

- Jonathan M Davis

Re: A few thoughts on the existing GSoC student proposals

[Piotr Szturmaj]

Andrei Alexandrescu wrote:

4. Database API

This is a high impact item. I'd need to collect more information about
the specifics of the application before creating an opinion about its
chances of success. I see Piotr and others have some related ideas; I
suggest them to apply appropriately (either as mentors or students). A
solid mentor(s)/student(s) combo is a key ingredient of a successful
project.


Thanks,

Andrei


I do not feel very strong as a mentor and I'm not a student, so I'd 
rather not apply. However, I'll certainly try to help with this item :)

Re: Asynchronicity in D

[Robert Clipsham]

On 31/03/2011 17:53, Robert Clipsham wrote:

On 31/03/2011 17:26, Andrei Alexandrescu wrote:

foreach (line; byLineAsync("http://d-programming-language.org";))
{
... use line ...
}


What would be awesome is if this was backed by fibers, then you have a
really simple and easy wrapper for doing async io, handling lots of
connections as the data comes in one thread. Of course a none-by-line
version would also be excellent given that a lot of IO doesn't care
about new lines.

--
Robert
http://octarineparrot.com/


To clarify, this isn't much use for clients, but for servers it could be 
useful, or if you're wanting to act as multiple clients.


--
Robert
http://octarineparrot.com/

Re: Asynchronicity in D

[Robert Clipsham]

On 31/03/2011 17:26, Andrei Alexandrescu wrote:

foreach (line; byLineAsync("http://d-programming-language.org";))
{
... use line ...
}


What would be awesome is if this was backed by fibers, then you have a 
really simple and easy wrapper for doing async io, handling lots of 
connections as the data comes in one thread. Of course a none-by-line 
version would also be excellent given that a lot of IO doesn't care 
about new lines.


--
Robert
http://octarineparrot.com/

Re: Asynchronicity in D

[dsimcha]

== Quote from Andrei Alexandrescu (seewebsiteforem...@erdani.org)'s article
> On 3/31/11 11:43 AM, dsimcha wrote:
> > == Quote from Andrei Alexandrescu (seewebsiteforem...@erdani.org)'s article
> >> On 3/31/11 6:35 AM, Max Klyga wrote:
> >>> I've been thinking on things I can change in my GSoC proposal to make it
> >>> stronger and noticed that currently Phobos does not address asynchronous
> >>> I/O of any kind.
> >>>
> >>> A number of threads on thid newsgroup mentioned about this problem or
> >>> shown ways other languages address asynchronicity.
> >>>
> >>> I want to ask D community about plans on asynchronicity in Phobos.
> >>> Did somenone in Phobos team thought about possible design?
> >>> How does asynchronicity stacks with ranges?
> >>> What model should D adapt?
> >>> etc.
> >> I think that would be a good contribution that would complement Jonas'.
> >> You'll need to discuss cooperation with him and at best Jonas would
> >> agree to become a mentor.
> >> I've posted a couple of weeks earlier how I think that could work with
> >> ranges: the range maintains the asynchronous state and has a queue of
> >> already-available buffers received. The network traffic occurs in a
> >> different thread; the range throws requests over the fence to libcurl
> >> and libcurl throws buffers over the fence back to the range. The range
> >> offers a seemingly synchronous interface:
> >> foreach (line; byLineAsync("http://d-programming-language.org";))
> >> {
> >>  ... use line ...
> >> }
> >> except that the processing and the fetching of data occur in distinct
> >> threads.
> >> Server-side code such as network servers etc. would also be an
> >> interesting topic. Let me know if you're versed in the likes of libev(ent).
> >> Thanks,
> >> Andrei
> >
> > Is this basically std.parallelism.asyncBuf
> > (http://cis.jhu.edu/~dsimcha/d/phobos/std_parallelism.html#asyncBuf) or 
> > something
> > different?
> asyncBuf would be an excellent backend for that, but the entire thing
> needs encapsulation so as to not expose user code to the risks of undue
> sharing.
> Andrei

Ok.  If there are any enhancements that would make asyncBuf work better for 
this,
let me know.

Re: Asynchronicity in D

[Andrei Alexandrescu]

On 3/31/11 11:43 AM, dsimcha wrote:

== Quote from Andrei Alexandrescu (seewebsiteforem...@erdani.org)'s article

On 3/31/11 6:35 AM, Max Klyga wrote:

I've been thinking on things I can change in my GSoC proposal to make it
stronger and noticed that currently Phobos does not address asynchronous
I/O of any kind.

A number of threads on thid newsgroup mentioned about this problem or
shown ways other languages address asynchronicity.

I want to ask D community about plans on asynchronicity in Phobos.
Did somenone in Phobos team thought about possible design?
How does asynchronicity stacks with ranges?
What model should D adapt?
etc.

I think that would be a good contribution that would complement Jonas'.
You'll need to discuss cooperation with him and at best Jonas would
agree to become a mentor.
I've posted a couple of weeks earlier how I think that could work with
ranges: the range maintains the asynchronous state and has a queue of
already-available buffers received. The network traffic occurs in a
different thread; the range throws requests over the fence to libcurl
and libcurl throws buffers over the fence back to the range. The range
offers a seemingly synchronous interface:
foreach (line; byLineAsync("http://d-programming-language.org";))
{
 ... use line ...
}
except that the processing and the fetching of data occur in distinct
threads.
Server-side code such as network servers etc. would also be an
interesting topic. Let me know if you're versed in the likes of libev(ent).
Thanks,
Andrei


Is this basically std.parallelism.asyncBuf
(http://cis.jhu.edu/~dsimcha/d/phobos/std_parallelism.html#asyncBuf) or 
something
different?


asyncBuf would be an excellent backend for that, but the entire thing 
needs encapsulation so as to not expose user code to the risks of undue 
sharing.


Andrei

Re: Asynchronicity in D

[dsimcha]

== Quote from Andrei Alexandrescu (seewebsiteforem...@erdani.org)'s article
> On 3/31/11 6:35 AM, Max Klyga wrote:
> > I've been thinking on things I can change in my GSoC proposal to make it
> > stronger and noticed that currently Phobos does not address asynchronous
> > I/O of any kind.
> >
> > A number of threads on thid newsgroup mentioned about this problem or
> > shown ways other languages address asynchronicity.
> >
> > I want to ask D community about plans on asynchronicity in Phobos.
> > Did somenone in Phobos team thought about possible design?
> > How does asynchronicity stacks with ranges?
> > What model should D adapt?
> > etc.
> I think that would be a good contribution that would complement Jonas'.
> You'll need to discuss cooperation with him and at best Jonas would
> agree to become a mentor.
> I've posted a couple of weeks earlier how I think that could work with
> ranges: the range maintains the asynchronous state and has a queue of
> already-available buffers received. The network traffic occurs in a
> different thread; the range throws requests over the fence to libcurl
> and libcurl throws buffers over the fence back to the range. The range
> offers a seemingly synchronous interface:
> foreach (line; byLineAsync("http://d-programming-language.org";))
> {
> ... use line ...
> }
> except that the processing and the fetching of data occur in distinct
> threads.
> Server-side code such as network servers etc. would also be an
> interesting topic. Let me know if you're versed in the likes of libev(ent).
> Thanks,
> Andrei

Is this basically std.parallelism.asyncBuf
(http://cis.jhu.edu/~dsimcha/d/phobos/std_parallelism.html#asyncBuf) or 
something
different?

Re: Asynchronicity in D

[Andrei Alexandrescu]

On 3/31/11 6:35 AM, Max Klyga wrote:

I've been thinking on things I can change in my GSoC proposal to make it
stronger and noticed that currently Phobos does not address asynchronous
I/O of any kind.

A number of threads on thid newsgroup mentioned about this problem or
shown ways other languages address asynchronicity.

I want to ask D community about plans on asynchronicity in Phobos.
Did somenone in Phobos team thought about possible design?
How does asynchronicity stacks with ranges?
What model should D adapt?
etc.


I think that would be a good contribution that would complement Jonas'. 
You'll need to discuss cooperation with him and at best Jonas would 
agree to become a mentor.


I've posted a couple of weeks earlier how I think that could work with 
ranges: the range maintains the asynchronous state and has a queue of 
already-available buffers received. The network traffic occurs in a 
different thread; the range throws requests over the fence to libcurl 
and libcurl throws buffers over the fence back to the range. The range 
offers a seemingly synchronous interface:


foreach (line; byLineAsync("http://d-programming-language.org";))
{
   ... use line ...
}

except that the processing and the fetching of data occur in distinct 
threads.


Server-side code such as network servers etc. would also be an 
interesting topic. Let me know if you're versed in the likes of libev(ent).



Thanks,

Andrei

Re: Review of std.net.isemail part 2

[Jacob Carlborg]

On 2011-03-31 01:12, Jonathan M Davis wrote:

On 2011-03-30 15:27, Andrei Alexandrescu wrote:

On 3/30/11 5:24 PM, bearophile wrote:

Andrei:

Beyond naming:

Some standard for member attributes? Like m_something, etc? I don't like
this a lot, but having a style guide on this too is useful for Phobos
(and user code).

Bye,
bearophile


I usually prepend private data members with an underscore.


That's what std.datetime and std.file do. Regardless, member variables often
need a different naming scheme from normal variables thanks to properties that
have the same name. That can go in the style guide, but I don't think that the
naming scheme for private member variables or local variables is as important
as the public ones, since they're part of the API. It would still be good to
be consistent (like it's good to be consistent with braces), but when it comes
to names, the primary concern is the public API.

- Jonathan M Davis


I think this is only valid if the member variables have a corresponding 
getter/setter with the same name.


--
/Jacob Carlborg

Re: __traits so long and ugly, what about ::?

[Jacob Carlborg]

On 2011-03-31 01:05, KennyTM~ wrote:

On Mar 31, 11 06:32, Ary Manzana wrote:

On 3/30/11 5:52 PM, KennyTM~ wrote:

On Mar 31, 11 04:19, Alix Pexton wrote:

On 30/03/2011 20:45, KennyTM~ wrote:

This is confusing as :: is used to separate scopes in C++ (and PHP
too).


The first thing it reminded me of was Lua, where a single colon makes
the left hand side into the first argument of the function on the
right.

foo:bar(x) ==> bar.(foo, x)

So it felt kinda familiar to me ^^

A...


That is almost like UFCS in D.

int foo(string x, int y) { return x.length - y; }

assert (foo("testing", 3) == 4);
assert ("testing".foo(3) == 4);

But OP's proposal is restricted to __traits only.

__traits is a relatively advanced part of the language, plus many of its
features has already been exposed via the library std.traits, e.g.
std.traits.hasMember!(S, "m"), I don't think it really needs a very
short syntax.


Look, metaprogramming in Ruby is a relatively advanced part of the
language. And you know why it is heavily used and everyone can jump and
start using it in a matter of seconds and build the most amazing things?
Because it's very, very, very, (add 1000 very words here), very easy to
use.



We're talking about __traits, a feature for compile-time reflection
where most of its capability is already wrapped up to std.traits. It
isn't comparable with the whole metaprogramming capability in another
language.

You know, metaprogramming is also heavily used in D, because of a
sensible template system, mixins and CTFE, not __traits.


Why make *anything* hard to use if you can do it in an easier way?


Is

meta.hasMember(S, "m")

or even

import std.traits;
...
hasMember!(S, "m")

really that hard to use?

I think the 'meta' namespace is enough. 'A::B' is too much for such a
low-level feature.


In Ruby it would just be: S.hasMember("m") and "hasMember" would just be 
a regular plain old method, nothing special about it.


--
/Jacob Carlborg

Re: __traits so long and ugly, what about ::?

[Jacob Carlborg]

On 2011-03-31 00:32, Ary Manzana wrote:

On 3/30/11 5:52 PM, KennyTM~ wrote:

On Mar 31, 11 04:19, Alix Pexton wrote:

On 30/03/2011 20:45, KennyTM~ wrote:

This is confusing as :: is used to separate scopes in C++ (and PHP
too).


The first thing it reminded me of was Lua, where a single colon makes
the left hand side into the first argument of the function on the right.

foo:bar(x) ==> bar.(foo, x)

So it felt kinda familiar to me ^^

A...


That is almost like UFCS in D.

int foo(string x, int y) { return x.length - y; }

assert (foo("testing", 3) == 4);
assert ("testing".foo(3) == 4);

But OP's proposal is restricted to __traits only.

__traits is a relatively advanced part of the language, plus many of its
features has already been exposed via the library std.traits, e.g.
std.traits.hasMember!(S, "m"), I don't think it really needs a very
short syntax.


Look, metaprogramming in Ruby is a relatively advanced part of the
language. And you know why it is heavily used and everyone can jump and
start using it in a matter of seconds and build the most amazing things?
Because it's very, very, very, (add 1000 very words here), very easy to
use.

Why make *anything* hard to use if you can do it in an easier way?


Yes, because there is nothing special with how the meta information is 
accessed, just regular methods, no additional language support.


--
/Jacob Carlborg

Re: opSlice() magic for foreach now works?

[Jacob Carlborg]

On 2011-03-30 22:35, Mafi wrote:

Am 30.03.2011 20:35, schrieb Jacob Carlborg:

On 2011-03-30 16:47, David Nadlinger wrote:

On 3/30/11 4:43 PM, Jacob Carlborg wrote:

Do we have three ways now to implement iteration, opApply, opSlice and
ranges?


Wouldn't opSlice only be syntatic sugar for ranges?

David


I have no idea, that's why I'm asking.



There are anly two ways to be truly iteratable: opApplay and Ranges.
Now, if the object does not implememhnt either, opSlice is checked: "Do
you know somebody that can iterate over you?".
If opSlice (ie a[]) gives something iterable - ie something with
opApply, a Range or something which itself has a opSlice - the foreach
is rewritten to use that thing.

Mafi


Ok, thanks.

--
/Jacob Carlborg

Re: Asynchronicity in D

[Aleksandar Ružičić]

I really like design of node.js (http://nodejs.org) it's internally
based on libev and everything runs in a single-threaded event loop.
It's proven to be highly concurrent and memory efficient.

Maybe a wrapper around libev(ent) for D ala node.js would be good
solution for asynchronous API, other than thread approach (I always
like to have more than one option and choose one which suits better
for concrete task I'm dealing with).

Whatever solution to be chosen I'd like to have an API like this:

readTextAsync(filename, (string contents) {
   // do something with contents
});


On Thu, Mar 31, 2011 at 2:04 PM, Piotr Szturmaj  wrote:
> Max Klyga wrote:
>>
>> I've been thinking on things I can change in my GSoC proposal to make it
>> stronger and noticed that currently Phobos does not address asynchronous
>> I/O of any kind.
>>
>> A number of threads on thid newsgroup mentioned about this problem or
>> shown ways other languages address asynchronicity.
>>
>> I want to ask D community about plans on asynchronicity in Phobos.
>> Did somenone in Phobos team thought about possible design?
>> How does asynchronicity stacks with ranges?
>> What model should D adapt?
>> etc.
>>
>
> Yes, asynchronous networking API would be more scalable. If you're
> collecting information about async IO, please take a look at libevent and
> libev, also NT's completion ports, FreeBSD's kqueue and Linux epoll.
>
> Protocols implemented using event-driven APIs should scale to thousands of
> connections using few working threads. Moreover async protocols could be
> wrapped to be synchronous (but not other way around) and used just like well
> known blocking API's.
>
> Basically, while using async IO, you request some data to be written and
> then wait for completion event (usually by callback function). Then you
> request some allocated buffer to be read and then you wait until network
> stack fills it up. You do not wait for blocking operation like with using
> send() or recv(), instead you may do some useful processing between events.
>

Re: Asynchronicity in D

[Piotr Szturmaj]

dsimcha wrote:

== Quote from Piotr Szturmaj (bncr...@jadamspam.pl)'s article

Max Klyga wrote:

I've been thinking on things I can change in my GSoC proposal to make it
stronger and noticed that currently Phobos does not address asynchronous
I/O of any kind.

A number of threads on thid newsgroup mentioned about this problem or
shown ways other languages address asynchronicity.

I want to ask D community about plans on asynchronicity in Phobos.
Did somenone in Phobos team thought about possible design?
How does asynchronicity stacks with ranges?
What model should D adapt?
etc.


Yes, asynchronous networking API would be more scalable. If you're
collecting information about async IO, please take a look at libevent
and libev, also NT's completion ports, FreeBSD's kqueue and Linux epoll.
Protocols implemented using event-driven APIs should scale to thousands
of connections using few working threads. Moreover async protocols could
be wrapped to be synchronous (but not other way around) and used just
like well known blocking API's.
Basically, while using async IO, you request some data to be written and
then wait for completion event (usually by callback function). Then you
request some allocated buffer to be read and then you wait until network
stack fills it up. You do not wait for blocking operation like with
using send() or recv(), instead you may do some useful processing
between events.


Forgive any naiveness here, but isn't this just a special case of future promise
parallelism?  Using my proposed std.parallelism module:

auto myTask = task(&someNetworkClass.recv);

// Use a new thread, but this could also be executed on a task
// queue to keep the number of threads down.
myTask.executeInNewThread();

// Do other stuff.

auto recvResults = myTask.yieldWait();
// Do stuff with recvResults

If I understand correctly (though it's very likely I don't since I've never
written any serious networking code before) such a thing can and should be
implemented on top of more general parallelism primitives rather than being 
baked
directly into the networking design.


Asynchronous tasks are great thing, but async networking IO aka 
overlapped IO is something different. Its efficency comes from direct 
interaction with operating system. In case of tasks you need one thread 
for each task, whereas in overlapped IO, you just request some well 
known IO operation, which is completed by the OS in the background. You 
don't need any threads, besides those which handle completion events.
Here is a good explanation of how it works in WinNT: 
http://en.wikipedia.org/wiki/Overlapped_I/O

Re: Asynchronicity in D

[Aleksandar Ružičić]

I really like design of node.js (http://nodejs.org) it's internally
based on libev and everything runs in a single-threaded event loop.
It's proven to be highly concurrent and memory efficient.

Maybe a wrapper around libev(ent) for D ala node.js would be good
solution for asynchronous API, other than thread approach (I always
like to have more than one option and choose one which suits better
for concrete task I'm dealing with).

Whatever solution to be chosen I'd like to have an API like this:

readTextAsync(filename, (string contents) {
  // do something with contents
});

Re: Asynchronicity in D

[dsimcha]

== Quote from Piotr Szturmaj (bncr...@jadamspam.pl)'s article
> Max Klyga wrote:
> > I've been thinking on things I can change in my GSoC proposal to make it
> > stronger and noticed that currently Phobos does not address asynchronous
> > I/O of any kind.
> >
> > A number of threads on thid newsgroup mentioned about this problem or
> > shown ways other languages address asynchronicity.
> >
> > I want to ask D community about plans on asynchronicity in Phobos.
> > Did somenone in Phobos team thought about possible design?
> > How does asynchronicity stacks with ranges?
> > What model should D adapt?
> > etc.
> >
> Yes, asynchronous networking API would be more scalable. If you're
> collecting information about async IO, please take a look at libevent
> and libev, also NT's completion ports, FreeBSD's kqueue and Linux epoll.
> Protocols implemented using event-driven APIs should scale to thousands
> of connections using few working threads. Moreover async protocols could
> be wrapped to be synchronous (but not other way around) and used just
> like well known blocking API's.
> Basically, while using async IO, you request some data to be written and
> then wait for completion event (usually by callback function). Then you
> request some allocated buffer to be read and then you wait until network
> stack fills it up. You do not wait for blocking operation like with
> using send() or recv(), instead you may do some useful processing
> between events.

Forgive any naiveness here, but isn't this just a special case of future promise
parallelism?  Using my proposed std.parallelism module:

auto myTask = task(&someNetworkClass.recv);

// Use a new thread, but this could also be executed on a task
// queue to keep the number of threads down.
myTask.executeInNewThread();

// Do other stuff.

auto recvResults = myTask.yieldWait();
// Do stuff with recvResults

If I understand correctly (though it's very likely I don't since I've never
written any serious networking code before) such a thing can and should be
implemented on top of more general parallelism primitives rather than being 
baked
directly into the networking design.

Re: __traits so long and ugly, what about ::?

[Steven Schveighoffer]

On Thu, 31 Mar 2011 10:39:29 -0400, Andrei Alexandrescu  
 wrote:



On 03/31/2011 09:27 AM, Steven Schveighoffer wrote:

On Thu, 31 Mar 2011 10:16:58 -0400, Andrei Alexandrescu
 wrote:


On 03/31/2011 03:52 AM, bearophile wrote:

KennyTM~:


I think the GSoC group and language feature discussion group should
be separated.


Regarding D/Phobos feature discussions, they don't damage the summer
of code discussions significantly because they put in use almost a
different part of the brain.


They do share the same time budget.


So don't read/respond to the posts. I think any discouragement for open
discussion for any topic related to D is not a healthy attitude for an
open community.


Well you know it's considerably less simpler than that. Besides, I  
probably wasn't clear enough: I'm not discouraging anything as much as  
encouraging discussions that mark progress in important matters.


That's a better statement.  I just don't think such "unimportant"  
discussions are harmful or damaging, just don't participate if you don't  
want to be distracted.  We're all (mostly) adults in charge of our own  
schedule.  We don't need to be insulated from things.



If you want to have a GSOC-only discussion group, create a new newsgroup
or mailing list. Do not hijack this group.


This is not about GSoC (it's but one of several items I mentioned), it's  
simply about making headway in topics that are important.


Sorry if I got the wrong impression.  I don't always read an entire thread  
when I see a post I want to respond to.


-Steve

Re: __traits so long and ugly, what about ::?

[Andrei Alexandrescu]

On 03/31/2011 09:27 AM, Steven Schveighoffer wrote:

On Thu, 31 Mar 2011 10:16:58 -0400, Andrei Alexandrescu
 wrote:


On 03/31/2011 03:52 AM, bearophile wrote:

KennyTM~:


I think the GSoC group and language feature discussion group should
be separated.


Regarding D/Phobos feature discussions, they don't damage the summer
of code discussions significantly because they put in use almost a
different part of the brain.


They do share the same time budget.


So don't read/respond to the posts. I think any discouragement for open
discussion for any topic related to D is not a healthy attitude for an
open community.


Well you know it's considerably less simpler than that. Besides, I 
probably wasn't clear enough: I'm not discouraging anything as much as 
encouraging discussions that mark progress in important matters.



If you want to have a GSOC-only discussion group, create a new newsgroup
or mailing list. Do not hijack this group.


This is not about GSoC (it's but one of several items I mentioned), it's 
simply about making headway in topics that are important.



Andrei

Re: __traits so long and ugly, what about ::?

[Steven Schveighoffer]

On Thu, 31 Mar 2011 10:16:58 -0400, Andrei Alexandrescu  
 wrote:



On 03/31/2011 03:52 AM, bearophile wrote:

KennyTM~:


I think the GSoC group and language feature discussion group should
be separated.


Regarding D/Phobos feature discussions, they don't damage the summer
of code discussions significantly because they put in use almost a
different part of the brain.


They do share the same time budget.


So don't read/respond to the posts.  I think any discouragement for open  
discussion for any topic related to D is not a healthy attitude for an  
open community.


If you want to have a GSOC-only discussion group, create a new newsgroup  
or mailing list.  Do not hijack this group.


-Steve

Re: __traits so long and ugly, what about ::?

[Andrei Alexandrescu]

On 03/31/2011 03:52 AM, bearophile wrote:

KennyTM~:


I think the GSoC group and language feature discussion group should
be separated.


Regarding D/Phobos feature discussions, they don't damage the summer
of code discussions significantly because they put in use almost a
different part of the brain.


They do share the same time budget.

Andrei

Re: __traits so long and ugly, what about ::?

[Ary Manzana]

On 3/31/11 10:35 AM, Kagamin wrote:

Ary Manzana Wrote:


I just hate it when you have to write too much


hasMember!(S, "m")
is only 1 character longer than
S::hasMember("m")

not too much for me


Ah, so then I change it to:

S::hm("M")

Re: __traits so long and ugly, what about ::?

[Kagamin]

Ary Manzana Wrote:

> I just hate it when you have to write too much

hasMember!(S, "m")
is only 1 character longer than
S::hasMember("m")

not too much for me

Re: expression templates

[enuhtac]

Am 29.03.2011 21:15, schrieb dsimcha:
> == Quote from bearophile (bearophileh...@lycos.com)'s article
>> The operator overloading done with opCmp is too much coarse even if you want 
>> to
> implement sets with operators like <= < > >= == for subset, etc.
>
> Can you please give an example of where <=, >, etc. are useful for 
> representing
> set operations?  My naive opinion (i.e. without understanding your use case) 
> is
> that using comparison operators to represent anything besides partial or total
> ordering is a severe abuse of operator overloading.  (Their use to represent
> ordering of corresponding elements of a vector or matrix is a borderline 
> case.)
You're right. You rarely need comparison operators for sets. This is
possible of course and Matlab or Python use this feature which
significantly simplifies some operations, but if you consider
performance (and that is what you do if you think about expression
templates) this is not a good practice as temporary bool vectors are
generated.

In my case the use of expression templates is twofold: I expect to have
some multidimensional templated array variable type, let's call it
'Array', that should support expression templates for algebraic
operations. For this purpose comparison operators are not needed
(normaly you use reductions instead). But additionally I would like to
use expression templates to parse expressions that are used to
automatically generate device depend code. In one of my previous posts I
gave an example. These expression templates do not work on sets but on
scalar expressions. So comparison operators can be useful. E.g.:

auto case1 = v[I] < 0.0;
auto case2 = !case1;
auto result = case1 * v[I] + case2 * v[I+1];

This code mimics the behaviour of an if ... else construct. The
difference is that both cases are computed and combined into the result
so internally no branches are necessary. This is used to circumvent
program path divergence on GPU devices. Also I suppose that this type of
code is more efficient on standard CPUs as there are no branch
prediction failures.

I use scalar expressions to build complex operators that work on arrays.
To give an example that includes both types of expression templates I
imagine code like this:

class Array( T )
{ ... };

class Operator( Expr )
{ ... };

Operator!( result[I] = (v[I+1] - v[I-1])/h ) derivative;

Array!double a, b, c;

b = derivative( a );
c = a + b;

result, v and I need to be predefined (global) variables of whatever
type that encode input and output parameters of operators (maybe result
could be omitted) and the position in the array.

As I now understand, expression templates are normally used for set
operations. For this purpose you normally do not need comparison
operators so D is well suited. But I would like to use expression
templates in a more general way that is obviously not standard. So D
lacks the necessary support. It's a pity...

Re: Asynchronicity in D

[Piotr Szturmaj]

Max Klyga wrote:

I've been thinking on things I can change in my GSoC proposal to make it
stronger and noticed that currently Phobos does not address asynchronous
I/O of any kind.

A number of threads on thid newsgroup mentioned about this problem or
shown ways other languages address asynchronicity.

I want to ask D community about plans on asynchronicity in Phobos.
Did somenone in Phobos team thought about possible design?
How does asynchronicity stacks with ranges?
What model should D adapt?
etc.



Yes, asynchronous networking API would be more scalable. If you're 
collecting information about async IO, please take a look at libevent 
and libev, also NT's completion ports, FreeBSD's kqueue and Linux epoll.


Protocols implemented using event-driven APIs should scale to thousands 
of connections using few working threads. Moreover async protocols could 
be wrapped to be synchronous (but not other way around) and used just 
like well known blocking API's.


Basically, while using async IO, you request some data to be written and 
then wait for completion event (usually by callback function). Then you 
request some allocated buffer to be read and then you wait until network 
stack fills it up. You do not wait for blocking operation like with 
using send() or recv(), instead you may do some useful processing 
between events.

Re: __traits so long and ugly, what about ::?

[spir]

On 03/31/2011 05:24 AM, Andrei Alexandrescu wrote:

My problem is that when you start using D's cool features you end up
with a really hard to understand and obscure code...


I understand and to a good extent agree with the sentiment.


Pleased to read that :-) I've had this sentiment for a while as well. I'm 
afraid this is frightening. Large pieces of Phobos are hard to read, imo (eg 
std.format).


Denis
--
_
vita es estrany
spir.wikidot.com

Asynchronicity in D

[Max Klyga]

I've been thinking on things I can change in my GSoC proposal to make 
it stronger and noticed that currently Phobos does not address 
asynchronous I/O of any kind.


A number of threads on thid newsgroup mentioned about this problem or 
shown ways other languages address asynchronicity.


I want to ask D community about plans on asynchronicity in Phobos.
Did somenone in Phobos team thought about possible design?
How does asynchronicity stacks with ranges?
What model should D adapt?
etc.

Re: D grammar overhaul

[Trass3r]

Rainer Schuetze Wrote:
> How can I contribute patches? I guess this would be with pull 
> requests, so I'll need a github repository, too? The discussion page for 
> pull requests looks like a nice feature...

Yep, you fork the original repo and apply your changes. Once you think you're 
somewhat done you can open a pull request.
I think you can even comment on single lines in commits.

The advantage of this approach is that your changes don't appear as a single 
huge diff but as atomic changesets (as long as the author groups changes that 
belong together in single commits)

> As a starter, here are some syntactic problems in the ddoc sources that 
> cause the generation of grammar.txt to fail:
> 
> - in template.dd, $(GNAME Constraint) is not enclosed by $(GRAMMAR)
> - in attribute.dd, $(GRAMMAR $(B const)), $(GRAMMAR $(B override)), etc 
> are used for formatting without having to do with the grammar. The boxes 
> they generate don't make much sense anyway, so they can just be removed.
> - MulExpression is only in $(V1), so it does not show for D2
> 
> I've attached the patch for these

Nice, I already incorporated some of your changes from the wiki. Will push soon 
so you can have a look what's already done.

Re: __traits so long and ugly, what about ::?

[bearophile]

KennyTM~:

> I think the GSoC group and language feature discussion group should be 
> separated.

Regarding D/Phobos feature discussions, they don't damage the summer of code 
discussions significantly because they put in use almost a different part of 
the brain. The skills and attention required to write real code (or discuss 
closely about it, like discussion about what data structures to write down for 
the GSoC) are quite different from the skills needed to think about new design 
ideas for D or Phobos. For me such two things are so separated that I use one 
to relax my mind from doing the other too much.

Said that, and given the limited number of people here, a further newsgroup 
split is not good.

Bye,
bearophile

Re: __traits so long and ugly, what about ::?

[so]

On Wed, 30 Mar 2011 22:28:27 +0300, Ary Manzana   
wrote:



I think :: is not used in the language.

In a recent article about D I saw:

mixin(__traits(identifier, T) ~ " " ~
   to!string(tolower(__traits(identifier, T)[0])) ~
   __traits(identifier, T)[1..$] ~ ";");

What if foo::bar were an alias for __traits(foo, bar) ?

The code would look like this:

mixin(T::identifier ~ " " ~
   to!string(tolower(T::identifier[0])) ~
   T::identifier[1..$] ~ ";");

What do you think?

Another uses:

__traits(int, isArithmetic) ==> int::isArithmetic
__traits(C, isAbstractClass) ==> C::isAbstractClass

__traits(hasMember, S, "m") ==> S::hasMember("m")

Well, you get the idea...

:: might be applied to other compile time uses, but I just came up with  
this...


I agree it is a problem (a small one) but i don't believe it deserves a  
new syntax. Especially not a major one like this.

Like many others i think "meta" is a much better choice.

Re: GCC 4.6

[so]

On Thu, 31 Mar 2011 05:09:44 +0300, jasonw  wrote:

You hit the nail on the head here. I see two real problems with his  
messages:


1) he's "force fitting" every possible language feature he learns into  
D. Clearly some features are useful, others are not, and this is why  
many of bearophile's ideas fail and generate endless debates and  
unnecessary noise. He can't see that the features just don't fit in.


This is not true, there are ideas here from many others as well that  
generate endless debates.
The reason is as far as i can see not always the ideas "just don't fit  
in". The reasons IMO are the chain of command and the resources.
Take the last long discussion on named arguments, i don't think anyone was  
against it.

One another thing is that a few of us evasive to some questions.

If you lack the vision of good language design as a whole, you shouldn't  
start suggesting new features like this. I'd appreciate it more if we  
won't introduce "new" concepts in this way.


This is oxymoron, by that logic there is not a single soul on earth with  
that vision.

You just dismissed whole academia, isn't this the way it operates?
Don't you think this is harmful? Why does D2 exist? D1 wasn't enough?

2) Programming language design requires rigorous definition of terms and  
other things. The D community doesn't encourage using precise,  
well-defined, unique terms. This leads to some subtleties and other  
problems in the discussions. Again I think the best place for general PL  
discussion is somewhere else, preferable in the academia. I'm sorry to  
say this, but I likely need to study how to put him in the kill file.  
The whole bearophile phenomenon takes place on an isolated island  
somewhere in the dark corners of D's history. The bug reports and  
benchmarks are priceless, but these "lectures" about other language  
often aren't.


I agree he should slow down proposing, at the same time people better stop  
ad hominem attacks.

If it is way to go, we all need to shut up.

Re: Review of std.net.isemail part 2

[Jonathan M Davis]

On 2011-03-31 00:09, Jacob Carlborg wrote:
> On 3/30/11 10:07 PM, Andrei Alexandrescu wrote:
> > On 3/30/11 2:47 PM, Jonathan M Davis wrote:
> >> I've tried to get Andrei to agree to a style guide a few times, but he's
> >> generally pushed back on it. I definitely think that we should have
> >> one if we
> >> want to actually have a consistent style, but thus far, he hasn't
> >> agreed to
> >> have one.
> > 
> > I think that's not representing my viewpoint quite accurately, but
> > getting to the bottom of whatever misunderstanding was there is not
> > important.
> > 
> > It would be helpful to have a broad style guide. The existing one is a
> > good start and is in fact already observed by much of Phobos (and in
> > particular by most of my code). The problem with writing a more
> > elaborate guide is finding the person(s) with the time and inclination
> > to write a draft, get it vetted by the major contributors, and take it
> > to completion.
> 
> Which one is the existing one, it it available on the DigitalMars site?
> 
> > For my money, just take the first that applies:
> > 
> > - Is it a function name? Use thisStyle.
> > 
> > - Is it a value (be it constant or variable)? Use thisStyle.
> > 
> > - Is it a type? Use ThisStyle.
> > 
> > - Is it a module name? Use this_style.
> > 
> > Beyond naming:
> > 
> > - Define variables as late as possible.
> > 
> > - Define constants and other names scoped appropriately.
> > 
> > - Prefer anonymous temporaries to named values within reason.
> > 
> > - Prefer ? : to if/else within reason.
> > 
> > - Prefer compact, effective code to verbose code within reason. Make
> > every line count.
> 
> I don't like this one. I would say prefer readable code to compact code.
> Don't be afraid of having long descriptive names of variables and having
> vertical space in your code. I don't mean you should put three newlines
> between two function declarations but I usually put a newline before and
> after statements:
> 
> int a;
> int b;
> 
> foo();
> bar();
> 
> if (a == b)
> foo();
> 
> else
> bar();
> 
> int c = 3;
> 
> > Nitpicks that 9 people have 10 opinions of:
> > 
> > - No 2+ empty lines please. Within a function, an empty line at best
> > should be replaced by a comment describing the meaning of the next block.
> > 
> > - Try to fit functions loosely on one editor page.
> > 
> > - Brace on its own line. (In fact I don't care much either way, but this
> > is the way the majority of the code is.) Note that this does cost more
> > vertical space so it somewhat clashes with the previous point.
> > 
> > - Please avoid > 80 columns unless you feel it induces sterility in the
> > long term.
> 
> Do you know how narrow 80 columns look on a wide screen. I think it
> looks narrow on a regular (non-wide) screen.

I definitely have to agree with you on that one (most everything you said 
actually), but there are obviously differing opinions and it needs to be 
discussed. I've put together an initial proposal for a style guide (based on 
what Phobos has been doing and what's been discussed on the newsgroup) and 
posted it to the Phobos list. So, discuss away on that thread. Hopefully, we 
can come to a reasonable consensus.

- Jonathan M Davis

Re: D grammar overhaul

[Rainer Schuetze]

Trass3r wrote:

Original discussion forked in the D grammar thread, so I'm opening a new one 
for this specific issue.
The D grammar is in dire need of an overhaul.

I suggest we discuss changes here and then put everything into github to issue 
a pull request once everything's done.
I already forked the d-programming-language.org repo to get started by fixing 
some mistakes. Hereafter I list some issues that come to my mind.


I'm not yet friend with git, but this might be a chance to get used to 
it. How can I contribute patches? I guess this would be with pull 
requests, so I'll need a github repository, too? The discussion page for 
pull requests looks like a nice feature...


I'm not sure though it is a good place to keep an overview of the 
grammar in a readable style. The grammar in the ddoc source is sometimes 
rather polluted with formatting information and links.


As a starter, here are some syntactic problems in the ddoc sources that 
cause the generation of grammar.txt to fail:


- in template.dd, $(GNAME Constraint) is not enclosed by $(GRAMMAR)
- in attribute.dd, $(GRAMMAR $(B const)), $(GRAMMAR $(B override)), etc 
are used for formatting without having to do with the grammar. The boxes 
they generate don't make much sense anyway, so they can just be removed.

- MulExpression is only in $(V1), so it does not show for D2

I've attached the patch for these.


Also there are 3 different instances of mixin: MixinExpression, MixinStatement 
and MixinDeclaration. Even though they all come down to the same thing.
I think, the different mixin variants are ok, because you might want to 
generate different AST for these.

Well I can imagine that MixinStatement is necessary because if you have an 
ExpStatement with a MixinExpression the result is probably discarded (just like 
'a' is discarded in a = 5;).
But why do you need another MixinDeclaration?


MixinDeclaration expands to declarations, while MixinStatement expands 
to statements, so the difference is not interesting during parsing, but 
might be important in the semantic pass (which restarts parsing for the 
string).


From 7326cf6c2cb08069ef29b281244ae57f90171ce1 Mon Sep 17 00:00:00 2001
From: Rainer 
Date: Thu, 31 Mar 2011 09:03:25 +0200
Subject: [PATCH] grammar formatting corrections

---
 attribute.dd  |   22 +-
 expression.dd |4 +---
 template.dd   |4 ++--
 3 files changed, 4 insertions(+), 26 deletions(-)

diff --git a/attribute.dd b/attribute.dd
index 050a385..5db0dc8 100644
--- a/attribute.dd
+++ b/attribute.dd
@@ -277,10 +277,6 @@ $(GNAME ProtectionAttribute):
 
 $(LNAME2 const, Const Attribute)
 
-$(GRAMMAR
-$(B const)
-)
-
$(P The $(B const) attribute declares constants that can be
evaluated at compile time. For example:
)
@@ -392,10 +388,6 @@ void main()
 
 $(LNAME2 override, Override Attribute)
 
-$(GRAMMAR
-$(B override)
-)
-
$(P The $(B override) attribute applies to virtual functions.
It means that the function must override a function with the
same name and parameters in a base class. The override attribute
@@ -423,10 +415,6 @@ class Foo2 : Foo
 
 Static Attribute
 
-$(GRAMMAR
-$(B static)
-)
-
$(P The $(B static) attribute applies to functions and data.
It means that the declaration does not apply to a particular
instance of an object, but to the type of the object. In
@@ -472,10 +460,6 @@ private int y = 4; // y is local to module foo
 
 $(LNAME2 auto, Auto Attribute)
 
-$(GRAMMAR
-$(B auto)
-)
-
$(P The $(B auto) attribute is used when there are no other attributes
and type inference is desired.
)
@@ -486,10 +470,6 @@ auto i = 6.8;  // declare i as a double
 
 $(LNAME2 scope, Scope Attribute)
 
-$(GRAMMAR
-$(B scope)
-)
-
 $(P
The $(B scope) attribute is used for local variables and for class
declarations. For class declarations, the $(B scope) attribute creates
@@ -541,5 +521,5 @@ $(P
 Macros:
TITLE=Attributes
WIKI=Attribute
-   CATEGORY_SPEC=$0
+   CATEGORY_SPEC=$0
 
diff --git a/expression.dd b/expression.dd
index cfee8d2..615441e 100644
--- a/expression.dd
+++ b/expression.dd
@@ -656,7 +656,6 @@ $(GNAME CatExpression):
 
 Mul Expressions
 
-$(V1
 $(GRAMMAR
 $(GNAME MulExpression):
$(GLINK UnaryExpression)
@@ -664,7 +663,6 @@ $(GNAME MulExpression):
$(I MulExpression) $(B /) $(GLINK UnaryExpression)
$(I MulExpression) $(B %) $(GLINK UnaryExpression)
 )
-)
 
$(P The operands must be arithmetic types. They undergo integral
promotions, and then are brought to a common type using the
@@ -1931,7 +1929,7 @@ void main()
 Macros:
TITLE=Expressions
WIKI=Expression
-   CATEGORY_SPEC=$0
+   CATEGORY_SPEC=$0
GLINK=$(LINK2 #$0, $(I $0))
GNAME=$(I $0)
DOLLAR=$
diff --git a/template.dd b/template.dd
index e46fe4d..8bbd9db 100644
--- a/template.dd
+++ b/template.dd
@@ -1000,13 +1000,

Re: __traits so long and ugly, what about ::?

[Jacob Carlborg]

On 3/30/11 9:28 PM, Ary Manzana wrote:

I think :: is not used in the language.

In a recent article about D I saw:

mixin(__traits(identifier, T) ~ " " ~
to!string(tolower(__traits(identifier, T)[0])) ~
__traits(identifier, T)[1..$] ~ ";");

What if foo::bar were an alias for __traits(foo, bar) ?

The code would look like this:

mixin(T::identifier ~ " " ~
to!string(tolower(T::identifier[0])) ~
T::identifier[1..$] ~ ";");

What do you think?

Another uses:

__traits(int, isArithmetic) ==> int::isArithmetic
__traits(C, isAbstractClass) ==> C::isAbstractClass

__traits(hasMember, S, "m") ==> S::hasMember("m")

Well, you get the idea...

:: might be applied to other compile time uses, but I just came up with
this...


I like it.

--
/Jacob Carlborg

Re: __traits so long and ugly, what about ::?

[Jacob Carlborg]

On 3/30/11 9:45 PM, KennyTM~ wrote:

On Mar 31, 11 03:28, Ary Manzana wrote:

I think :: is not used in the language.

In a recent article about D I saw:

mixin(__traits(identifier, T) ~ " " ~
to!string(tolower(__traits(identifier, T)[0])) ~
__traits(identifier, T)[1..$] ~ ";");

What if foo::bar were an alias for __traits(foo, bar) ?

The code would look like this:

mixin(T::identifier ~ " " ~
to!string(tolower(T::identifier[0])) ~
T::identifier[1..$] ~ ";");

What do you think?

Another uses:

__traits(int, isArithmetic) ==> int::isArithmetic
__traits(C, isAbstractClass) ==> C::isAbstractClass



You've got the order wrong.


__traits(hasMember, S, "m") ==> S::hasMember("m")

Well, you get the idea...

:: might be applied to other compile time uses, but I just came up with
this...


-1.

This is confusing as :: is used to separate scopes in C++ (and PHP too).
e.g.

struct A {
int x;
bool isSame(const A other) pure const { return x == other.x; }
}

void main () {
A a = A(2), b = A(2);
assert ( a.isSame(b)); // ok
assert (! a::isSame(b)); // ???
}

(How about that 'meta' namespace proposal? meta.hasMember(S, "m") )


I think think this is a valid reason. D does many things differently 
than C++. I don't know how many times I heard someone complaining about 
D's lack of struct inheritance.


--
/Jacob Carlborg

Re: Review of std.net.isemail part 2

[Jacob Carlborg]

On 3/30/11 10:07 PM, Andrei Alexandrescu wrote:

On 3/30/11 2:47 PM, Jonathan M Davis wrote:

I've tried to get Andrei to agree to a style guide a few times, but he's
generally pushed back on it. I definitely think that we should have
one if we
want to actually have a consistent style, but thus far, he hasn't
agreed to
have one.


I think that's not representing my viewpoint quite accurately, but
getting to the bottom of whatever misunderstanding was there is not
important.

It would be helpful to have a broad style guide. The existing one is a
good start and is in fact already observed by much of Phobos (and in
particular by most of my code). The problem with writing a more
elaborate guide is finding the person(s) with the time and inclination
to write a draft, get it vetted by the major contributors, and take it
to completion.


Which one is the existing one, it it available on the DigitalMars site?


For my money, just take the first that applies:

- Is it a function name? Use thisStyle.

- Is it a value (be it constant or variable)? Use thisStyle.

- Is it a type? Use ThisStyle.

- Is it a module name? Use this_style.

Beyond naming:

- Define variables as late as possible.

- Define constants and other names scoped appropriately.

- Prefer anonymous temporaries to named values within reason.

- Prefer ? : to if/else within reason.

- Prefer compact, effective code to verbose code within reason. Make
every line count.


I don't like this one. I would say prefer readable code to compact code. 
Don't be afraid of having long descriptive names of variables and having 
vertical space in your code. I don't mean you should put three newlines 
between two function declarations but I usually put a newline before and 
after statements:


int a;
int b;

foo();
bar();

if (a == b)
   foo();

else
   bar();

int c = 3;


Nitpicks that 9 people have 10 opinions of:

- No 2+ empty lines please. Within a function, an empty line at best
should be replaced by a comment describing the meaning of the next block.

- Try to fit functions loosely on one editor page.

- Brace on its own line. (In fact I don't care much either way, but this
is the way the majority of the code is.) Note that this does cost more
vertical space so it somewhat clashes with the previous point.

- Please avoid > 80 columns unless you feel it induces sterility in the
long term.


Do you know how narrow 80 columns look on a wide screen. I think it 
looks narrow on a regular (non-wide) screen.



- No tabs please.

- Comments should be high level (describing 3-10 lines) instead of
low-level (describing the mechanics of the next line, which are already
obvious in code).


Thanks,

Andrei



--
/Jacob Carlborg

Re: Review of std.net.isemail part 2

[Jacob Carlborg]

On 3/30/11 9:42 PM, Andrei Alexandrescu wrote:

On 3/30/11 1:47 PM, Jacob Carlborg wrote:

So, should I change the enum members to start with lowercase or leave it
like it is?


Change please.

Thanks,

Andrei


Ok, will do.

--
/Jacob Carlborg