Vote for std.process

[Jesse Phillips]

It is that time, If you would like to see the proposed 
std.process include into Phobos please vote yes. If one condition 
must be met specify under what condition, otherwise vote no.


std.process by Lars Kyllingstad and Steven Schveighoffer is a 
suggested improvement to the existing std.process and is a major 
change to the API. The original API remains but these will be 
going through deprecation.


In summary of the discussion there was concern of the use of 
Error and Exception. Lars is very interested in getting the 
standard hierarchy http://wiki.dlang.org/DIP33 thus leaving these 
at the safest level to allow for change without breaking code.


Please place any further comments in the official review thread 
leaving only your vote and a short comment (there should be no 
need to reply to anyone).


Docs: 
http://www.kyllingen.net/code/std-process2/phobos-prerelease/std_process.html


Source: 
https://github.com/kyllingstad/phobos/blob/std-process2/std/process.d


Friday April 19 PST will be the last day of voting.

Re: Vote for std.process

[Vladimir Panteleev]

On Friday, 12 April 2013 at 04:46:52 UTC, Jesse Phillips wrote:
It is that time, If you would like to see the proposed 
std.process include into Phobos please vote yes.


Yes!

Re: Vote for std.process

[Nick Sabalausky]

On Fri, 12 Apr 2013 06:46:51 +0200
"Jesse Phillips"  wrote:

> It is that time, If you would like to see the proposed 
> std.process include into Phobos please vote yes. If one condition 
> must be met specify under what condition, otherwise vote no.
> 

Yes

Re: Vote for std.process

[Manu]

Sorry to derail the topic, but I'd just like to raise a major gripe I've
had with introducing anything into phobos recently.

I see this pattern where something is designed, discussed, and then voted
into phobos. At this time the design looks good on paper, but there is very
little practical experience using the library.
The problem then is, once accepted, people start using it, and at some
point some issues are found, or ideas for improvement are made based on
user experience, but the module can no longer be touched due to the general
phobia of making breaking changes...

Can I suggest that ALL new modules should be added to exp. rather than
std.? Here they will stay for at least 1 year, and while they live in exp,
it is understood that they are still in an experimental introductory phase.
Users who choose to use modules in exp are accepting that the API may
receive changes, and consequently, accepting the responsibility to update
their code (and not complain about it breaking their program), should the
library be amended in its introductory phase.
At some time later when the module has been used in a decent amount of
software, and the API has stabilised, it can then be moved to std. This
move is obviously a breaking change its self, but anyone who has agrees to
import exp modules has already accepted the responsibility to update their
code as such.

Thoughts?


On 12 April 2013 16:12, Nick Sabalausky  wrote:

> On Fri, 12 Apr 2013 06:46:51 +0200
> "Jesse Phillips"  wrote:
>
> > It is that time, If you would like to see the proposed
> > std.process include into Phobos please vote yes. If one condition
> > must be met specify under what condition, otherwise vote no.
> >
>
> Yes
>

Re: Vote for std.process

[Paulo Pinto]

On Friday, 12 April 2013 at 06:25:10 UTC, Manu wrote:
Sorry to derail the topic, but I'd just like to raise a major 
gripe I've

had with introducing anything into phobos recently.

I see this pattern where something is designed, discussed, and 
then voted
into phobos. At this time the design looks good on paper, but 
there is very

little practical experience using the library.
The problem then is, once accepted, people start using it, and 
at some
point some issues are found, or ideas for improvement are made 
based on
user experience, but the module can no longer be touched due to 
the general

phobia of making breaking changes...

Can I suggest that ALL new modules should be added to exp. 
rather than
std.? Here they will stay for at least 1 year, and while they 
live in exp,
it is understood that they are still in an experimental 
introductory phase.
Users who choose to use modules in exp are accepting that the 
API may
receive changes, and consequently, accepting the responsibility 
to update
their code (and not complain about it breaking their program), 
should the

library be amended in its introductory phase.
At some time later when the module has been used in a decent 
amount of
software, and the API has stabilised, it can then be moved to 
std. This
move is obviously a breaking change its self, but anyone who 
has agrees to
import exp modules has already accepted the responsibility to 
update their

code as such.

Thoughts?


On 12 April 2013 16:12, Nick Sabalausky 

wrote:



On Fri, 12 Apr 2013 06:46:51 +0200
"Jesse Phillips"  wrote:

> It is that time, If you would like to see the proposed
> std.process include into Phobos please vote yes. If one 
> condition

> must be met specify under what condition, otherwise vote no.
>

Yes



Fully agree

Re: Vote for std.process

[Jacob Carlborg]

On 2013-04-12 08:24, Manu wrote:


Thoughts?


I think it's a good idea.

--
/Jacob Carlborg

Re: Vote for std.process

[Andrej Mitrovic]

On 4/12/13, Jesse Phillips  wrote:
> It is that time, If you would like to see the proposed
> std.process include into Phobos please vote yes. If one condition
> must be met specify under what condition, otherwise vote no.

Yes.

Re: Vote for std.process

[Manu]

On 12 April 2013 14:46, Jesse Phillips  wrote:

> It is that time, If you would like to see the proposed std.process include
> into Phobos please vote yes. If one condition must be met specify under
> what condition, otherwise vote no.
>

I didn't think I had much of an opinion on std.process, but I just gave it
a quick once over and noticed, relating to our recent discussions about
flagrant GC usage throughout phobos, that this is riddled with the exact
sorts of issues I've been talking about:

string[string] is used in the main API to receive environment variables;
perhaps kinda convenient, but it's impossible to supply environment
variables with loads of allocations.

toStringz is used liberally; alternatively, alloca() could allocate the
c-string's on the stack and zero terminate them there, passing a pointer to
the stack string to the OS functions.

String concatenation is rampant! Look at this code to parse the env
variables (which are already an AA):

foreach (var, val; childEnv)
envz[pos++] = (var~'='~val~'\0').ptr;


And many many more of the same...

This is a lib that spawns a process, I can't imagine why this library
should need to allocate a single byte, but it allocates wildly just to
perform a trivial pass-through to the OS.
This module is no exception either, basically all of phobos is like this,
and I'd like to see careful consideration made to these details in the
future.

I've said before, I sadly have to avoid phobos like the plague. Some
modules (like this one) that provide fundamental functionality - not just
helper functions - can't be avoided. Requirements for those should be extra
strict in my opinion.

Re: Vote for std.process

[Vladimir Panteleev]

On Friday, 12 April 2013 at 06:25:10 UTC, Manu wrote:
Can I suggest that ALL new modules should be added to exp. 
rather than

std.?


Java tried this. As a result, they are now stuck with "javax" 
because they realized breaking backwards compatibility wasn't 
worth the benefit of "graduating" packages from "javax" to "java".


If the goal is to include new or newly-overhauled modules in 
official releases, I think simply putting a big red 
"EXPERIMENTAL" warning at the top of the documentation would be 
simpler. Then, the warning can be removed when there is a D 
release which has no breaking changes in said module since its 
last released version.


However, another direction would be to make it easier to test 
such fresh modules. For example, it would be nice to have 
"nightly" builds of D, both for the "master" version, and a "mob" 
version, which includes some work-in-progress developments.

Re: Vote for std.process

[angel]

...
Thoughts?


Right suggestion.

As it was already discussed once, D community could follow Linux 
in this regard: each new module enters 'staging' folder. At some 
point in time it goes into 'mainline'.
This could also streamline the process of deprecating the old 
library.

Re: Vote for std.process

[Vladimir Panteleev]

On Friday, 12 April 2013 at 07:04:23 UTC, Manu wrote:

And many many more of the same...


So how many of these will produce a measurable performance 
increase when optimized, considering the relative cost of process 
creation?


This is a lib that spawns a process, I can't imagine why this 
library should need to allocate a single byte


So that the library fits in 3500 lines instead of many more, and 
has an API that can be invoked using one line, instead of ten.

Re: Vote for std.process

[Paulo Pinto]

On Friday, 12 April 2013 at 07:00:25 UTC, Vladimir Panteleev 
wrote:

On Friday, 12 April 2013 at 06:25:10 UTC, Manu wrote:
Can I suggest that ALL new modules should be added to exp. 
rather than

std.?


Java tried this. As a result, they are now stuck with "javax" 
because they realized breaking backwards compatibility wasn't 
worth the benefit of "graduating" packages from "javax" to 
"java".


Actually they also have another issue.

Java supported since the begging something similar to a System
package for dirty tricks, officially it was a Sun only API.

http://www.docjar.com/docs/api/sun/misc/Unsafe.html

So actually all Java implementations provide a compatible API, 
because many do use it.


Younger languages seem not to have any issue breaking backwards 
compatibility for experimental code.


--
Paulo

Re: Vote for std.process

[Manu]

On 12 April 2013 17:13, Vladimir Panteleev wrote:

> On Friday, 12 April 2013 at 07:04:23 UTC, Manu wrote:
>
>> And many many more of the same...
>>
>
> So how many of these will produce a measurable performance increase when
> optimized, considering the relative cost of process creation?


I agree that spawning processes is a low-frequency operation, but it's a
principle I'm trying to illustrate here.


 This is a lib that spawns a process, I can't imagine why this library
>> should need to allocate a single byte
>>
>
> So that the library fits in 3500 lines instead of many more, and has an
> API that can be invoked using one line, instead of ten.
>

You can argue whatever you like. I've said my part, and I wherever it lands
is not my call.
I'm just illustrating the point that phobos, like STL, and to a lesser
extent the CRT, are to continue to be avoided like the plague in some
fields unless these details are considered in future.

Most of the required changes would be self-contained, and not affect the
API at all.

Re: Vote for std.process

[Vladimir Panteleev]

On Friday, 12 April 2013 at 07:22:42 UTC, Manu wrote:
I agree that spawning processes is a low-frequency operation, 
but it's a

principle I'm trying to illustrate here.


My point was that it is not that it's low-frequency, it's that 
the OS process creation operation is so expensive, that a few 
memory allocations will not make much of a difference in 
comparison. It's the same as optimizing memory allocations in a 
program which is intrinsically disk- or network-bound.


You can argue whatever you like. I've said my part, and I 
wherever it lands is not my call.


Well, that's just not very constructive.

Your complaint in valid in general, but I was pointing out that 
it is not much so when specifically aimed at std.process.

Re: Vote for std.process

[Steven Schveighoffer]

On Fri, 12 Apr 2013 03:04:08 -0400, Manu  wrote:

On 12 April 2013 14:46, Jesse Phillips   
wrote:


It is that time, If you would like to see the proposed std.process  
include

into Phobos please vote yes. If one condition must be met specify under
what condition, otherwise vote no.



I didn't think I had much of an opinion on std.process, but I just gave  
it

a quick once over and noticed, relating to our recent discussions about
flagrant GC usage throughout phobos, that this is riddled with the exact
sorts of issues I've been talking about:


It's spawning a new process.  What is the issue with allocating a bit of  
memory?  The spawning of the process performance is going to dwarf that of  
any memory allocations.



string[string] is used in the main API to receive environment variables;
perhaps kinda convenient, but it's impossible to supply environment
variables with loads of allocations.


I think you meant "without"?

What would be your suggestion?  string[string] is the built-in map type.   
How do you pass an environment map without having some allocations?



toStringz is used liberally; alternatively, alloca() could allocate the
c-string's on the stack and zero terminate them there, passing a pointer  
to

the stack string to the OS functions.


This would be a lot of effort for pretty much zero gain for the majority  
of cases.



String concatenation is rampant! Look at this code to parse the env
variables (which are already an AA):

foreach (var, val; childEnv)
envz[pos++] = (var~'='~val~'\0').ptr;


This could be improved.  It could also be optimized into a single  
allocation automatically by the compiler (it might already be).  The API  
would not be affected by this improvement, though.



And many many more of the same...

This is a lib that spawns a process, I can't imagine why this library
should need to allocate a single byte, but it allocates wildly just to
perform a trivial pass-through to the OS.


It is not a trivial pass-through.  Different OSes require different  
parameter types.  Look at the difference between the windows and posix  
implementations.  We are writing a cross-platform library, so whatever we  
pick will have to be converted on at least one OS.



This module is no exception either, basically all of phobos is like this,
and I'd like to see careful consideration made to these details in the
future.


I too like to avoid memory allocations.  But the level to which you  
require is too stringent.  Not even Tango, which made low heap-allocations  
a priority, avoided allocations in it's process library.



I've said before, I sadly have to avoid phobos like the plague. Some
modules (like this one) that provide fundamental functionality - not just
helper functions - can't be avoided. Requirements for those should be  
extra

strict in my opinion.


We cannot cater to all developers.  To put all developers through a lot of  
pain in order to attract the fringe ones is not a practical goal.


I think the convenience of std.process is what it is there for.  For the  
highest performance, you are able to call the OS functions directly.


-Steve

Re: Vote for std.process

[Manu]

On 12 April 2013 17:30, Vladimir Panteleev wrote:

> On Friday, 12 April 2013 at 07:22:42 UTC, Manu wrote:
>
>> I agree that spawning processes is a low-frequency operation, but it's a
>> principle I'm trying to illustrate here.
>>
>
> My point was that it is not that it's low-frequency, it's that the OS
> process creation operation is so expensive, that a few memory allocations
> will not make much of a difference in comparison. It's the same as
> optimizing memory allocations in a program which is intrinsically disk- or
> network-bound.


Which OS are we talking about?
What OS runs on an a Nintendo Wii? There's only 24mb of system memory in
that machine, can we afford to allocate it frivolously?

Will I avoid phobos as a policy? Yes.


 You can argue whatever you like. I've said my part, and I wherever it
>> lands is not my call.
>>
>
> Well, that's just not very constructive.
>
> Your complaint in valid in general, but I was pointing out that it is not
> much so when specifically aimed at std.process.
>

I don't necessarily disagree with your point (although I don't agree
either). Perhaps in the context of std.process it's not so important... but
it's still an opportunity to set a precedent.
Honestly, any new module could have appeared for approval at this
particular moment and I would have made the same criticisms. Not
necessarily picking on std.process in particular, I'm making a point about
phobos, and what is considered acceptable.

Re: Vote for std.process

[Dicebot]

Yes.

Re: Vote for std.process

[Manu]

On 12 April 2013 17:35, Steven Schveighoffer  wrote:

> On Fri, 12 Apr 2013 03:04:08 -0400, Manu  wrote:
>
 string[string] is used in the main API to receive environment variables;
>> perhaps kinda convenient, but it's impossible to supply environment
>> variables with loads of allocations.
>>
>
> I think you meant "without"?
>

Yes.


What would be your suggestion?  string[string] is the built-in map type.
>  How do you pass an environment map without having some allocations?


I'd use string[].


 toStringz is used liberally; alternatively, alloca() could allocate the
>> c-string's on the stack and zero terminate them there, passing a pointer
>> to
>> the stack string to the OS functions.
>>
>
> This would be a lot of effort for pretty much zero gain for the majority
> of cases.


A trivial ... mixin template (?) could wrap it up, I don't see it
particularly less convenient than calling toStringz().
Perhaps there are other tools missing from phobos if this is somehow hard...


 String concatenation is rampant! Look at this code to parse the env
>> variables (which are already an AA):
>>
>> foreach (var, val; childEnv)
>> envz[pos++] = (var~'='~val~'\0').ptr;
>>
>
> This could be improved.  It could also be optimized into a single
> allocation automatically by the compiler (it might already be).  The API
> would not be affected by this improvement, though.
>

I've never seem the compiler apply that optimisation, although I often wish
it would.
I saw an appender appear a few pages below, that would be an improvement
here too I guess.


 And many many more of the same...
>>
>> This is a lib that spawns a process, I can't imagine why this library
>> should need to allocate a single byte, but it allocates wildly just to
>> perform a trivial pass-through to the OS.
>>
>
> It is not a trivial pass-through.  Different OSes require different
> parameter types.  Look at the difference between the windows and posix
> implementations.  We are writing a cross-platform library, so whatever we
> pick will have to be converted on at least one OS.


Writing cross-platform code is my life.
They are trivial conversions. It could be done on the stack easily.
Granted, ideally the compiler could theoretically perform some of those
improvements, but it doesn't, and probably depends on other changes to be
able to do so anyway.
For instance, the compiler couldn't confidently lower the allocations to
the stack unless 'scope' arguments worked, so it knew it wouldn't escape
the OS call it was being handed to.


 This module is no exception either, basically all of phobos is like this,
>> and I'd like to see careful consideration made to these details in the
>> future.
>>
>
> I too like to avoid memory allocations.  But the level to which you
> require is too stringent.  Not even Tango, which made low heap-allocations
> a priority, avoided allocations in it's process library.


But it would be trivial to use the stack in these cases. It doesn't matter
if it adds a few lines under the hood.

I'm really just trying to make the point that it should ALWAYS be a key
consideration, and people should make it habit to think about/criticise
these things when accepting code into phobos.
This is the standard library, it will be used in more D code than any other
library ever. Why skimp here?


 I've said before, I sadly have to avoid phobos like the plague. Some
>> modules (like this one) that provide fundamental functionality - not just
>> helper functions - can't be avoided. Requirements for those should be
>> extra
>> strict in my opinion.
>>
>
> We cannot cater to all developers.


In this case you certainly can, it would be fairly trivial to do.


 To put all developers through a lot of pain in order to attract the fringe
> ones is not a practical goal.
>

What pain? Are you saying a stack variable is somehow hard?
If that's the case, then there's clearly some other fundamental tools
missing from phobos.


I think the convenience of std.process is what it is there for.  For the
> highest performance, you are able to call the OS functions directly.
>

No, it's to save (ideally) _all_ programmers the effort of rewriting this
module themselves.
This is exactly the sort of thing I waste loads of my life doing, rewriting
wheels because the author was a PC programmer, and simply didn't give a
shit.
What a waste of life >_<

I re-iterate, maybe my points aren't of critical importance in relation to
std.process specifically, I'm trying to raise some general issues with
phobos submissions (that shouldn't exclude std.process either).

Re: Vote for std.process

[Vladimir Panteleev]

On Friday, 12 April 2013 at 07:53:04 UTC, Manu wrote:

Which OS are we talking about?
What OS runs on an a Nintendo Wii? There's only 24mb of system 
memory in

that machine, can we afford to allocate it frivolously?


I wasn't aware that writing games for the Wii involves creating 
processes in the thousands.


I don't necessarily disagree with your point (although I don't 
agree
either). Perhaps in the context of std.process it's not so 
important... but

it's still an opportunity to set a precedent.
Honestly, any new module could have appeared for approval at 
this

particular moment and I would have made the same criticisms. Not
necessarily picking on std.process in particular, I'm making a 
point about

phobos, and what is considered acceptable.


Don't forget that shifting focus when adjusting goals is always a 
question of tradeoffs. Sacrificing tersity, maintainability, 
usability, safety etc. for the sake of performance would be a 
poor move in the case of std.process.

Re: Vote for std.process

[Dicebot]

On Friday, 12 April 2013 at 08:14:25 UTC, Manu wrote:

...


While I generally agree with your point, std.process is hardly 
the place to start from. It is not performance-critical 
functionality and it is not basic functionality that may be used 
in other critical modules. It is blaming consequences instead of 
reasons, not really practical.

Re: Vote for std.process

[Manipulator]

Please correct me if I'm wrong.
It's not the allocation itself that is the problem it's that it 
will trigger the GC to stop the world at some point.

Re: Vote for std.process

[Vladimir Panteleev]

On Friday, 12 April 2013 at 07:53:04 UTC, Manu wrote:
Honestly, any new module could have appeared for approval at 
this

particular moment and I would have made the same criticisms.


Observation: this move wasn't something I would encourage. Jesse, 
the review manager for std.process, has specifically asked:


Please place any further comments in the official review thread 
leaving only your vote and a short comment (there should be no 
need to reply to anyone).


There was ample time to discuss std.process during the informal 
and formal review. However, the situation with your replies is 
even worse: they don't even concern std.process in particular. 
Please consider starting a new thread next time, and stating your 
criticism aside to whatever poor module comes up the review queue 
;)

Re: Vote for std.process

[Paulo Pinto]

On Friday, 12 April 2013 at 07:53:04 UTC, Manu wrote:
On 12 April 2013 17:30, Vladimir Panteleev 
wrote:



On Friday, 12 April 2013 at 07:22:42 UTC, Manu wrote:

I agree that spawning processes is a low-frequency operation, 
but it's a

principle I'm trying to illustrate here.



My point was that it is not that it's low-frequency, it's that 
the OS
process creation operation is so expensive, that a few memory 
allocations
will not make much of a difference in comparison. It's the 
same as
optimizing memory allocations in a program which is 
intrinsically disk- or

network-bound.



Which OS are we talking about?
What OS runs on an a Nintendo Wii? There's only 24mb of system 
memory in

that machine, can we afford to allocate it frivolously?

Will I avoid phobos as a policy? Yes.




Do you create process at all?

I always had the impression those environments used real time OS 
like libraries and multitasking is achieved by co-routines or 
threads, not real processes.


Just a dummy comment from an industry outsider.

--
Paulo

Re: Vote for std.process

[Manu]

On 12 April 2013 18:18, Vladimir Panteleev wrote:

> On Friday, 12 April 2013 at 07:53:04 UTC, Manu wrote:
>
>> Which OS are we talking about?
>> What OS runs on an a Nintendo Wii? There's only 24mb of system memory in
>> that machine, can we afford to allocate it frivolously?
>>
>
> I wasn't aware that writing games for the Wii involves creating processes
> in the thousands.


Again, I'm just suggesting possibilities, and trying to illustrate that
it's a STANDARD library, you can never predict where users will want to use
it.


 I don't necessarily disagree with your point (although I don't agree
>> either). Perhaps in the context of std.process it's not so important...
>> but
>> it's still an opportunity to set a precedent.
>> Honestly, any new module could have appeared for approval at this
>> particular moment and I would have made the same criticisms. Not
>> necessarily picking on std.process in particular, I'm making a point about
>> phobos, and what is considered acceptable.
>>
>
> Don't forget that shifting focus when adjusting goals is always a question
> of tradeoffs. Sacrificing tersity, maintainability, usability, safety etc.
> for the sake of performance would be a poor move in the case of std.process.
>

Can you perhaps quantify how any of those things would be reduced by
addressing at least the details I highlight?
I'm basically advocating making a habit of using the stack where possible.
It's not exactly hard, or cryptic.

Re: Vote for std.process

[Manu]

On 12 April 2013 18:21, Manipulator  wrote:

> Please correct me if I'm wrong.
> It's not the allocation itself that is the problem it's that it will
> trigger the GC to stop the world at some point.
>

It's just another source of garbage. Which means this function of off
limits beneath a routine that's disabled collection, for instance. There
are lots of reasons to avoid making unnecessary garbage.

Re: Vote for std.process

[Manu]

On 12 April 2013 18:21, Dicebot  wrote:

> On Friday, 12 April 2013 at 08:14:25 UTC, Manu wrote:
>
>> ...
>>
>
> While I generally agree with your point, std.process is hardly the place
> to start from. It is not performance-critical functionality and it is not
> basic functionality that may be used in other critical modules. It is
> blaming consequences instead of reasons, not really practical.
>

And I generally agree with your point. Like I said, I would have made this
criticisms of whatever came alone ;)

Re: Vote for std.process

[Johannes Pfau]

Am Fri, 12 Apr 2013 17:04:08 +1000
schrieb Manu :

> 
> I've said before, I sadly have to avoid phobos like the plague. Some
> modules (like this one) that provide fundamental functionality - not
> just helper functions - can't be avoided. Requirements for those
> should be extra strict in my opinion.
> 

Most (GC) allocations could be fixed without breaking the API. I can
see 2 places where the API forces _GC_ allocations:
* string[string]
* thrown Exceptions are allocated with the GC

There is no simple solution for these. Maybe we'll have a hashtable in
the standard library at some point which will allow to use custom
allocators. Then std.process could use these. Exceptions require some
thinking. You can of course allocate them using any allocator, but
freeing them is difficult...

If you want to get rid of all (not only GC) allocations, there's another
issue: As D strings are not zero terminated we'll always have
allocations passing those to C code. Maybe we should have a cstring
type in phobos which would just be a string which is guaranteed to be
zero terminated.

http://dpaste.dzfl.pl/e76aa995 (needs some inout though)

Re: Vote for std.process

[Johannes Pfau]

Am Fri, 12 Apr 2013 06:46:51 +0200
schrieb "Jesse Phillips" :

> It is that time, If you would like to see the proposed 
> std.process include into Phobos please vote yes. If one condition 
> must be met specify under what condition, otherwise vote no.
> 
> std.process by Lars Kyllingstad and Steven Schveighoffer is a 
> suggested improvement to the existing std.process and is a major 
> change to the API. The original API remains but these will be 
> going through deprecation.
> 
> In summary of the discussion there was concern of the use of 
> Error and Exception. Lars is very interested in getting the 
> standard hierarchy http://wiki.dlang.org/DIP33 thus leaving these 
> at the safest level to allow for change without breaking code.
> 
> Please place any further comments in the official review thread 
> leaving only your vote and a short comment (there should be no 
> need to reply to anyone).
> 
> Docs: 
> http://www.kyllingen.net/code/std-process2/phobos-prerelease/std_process.html
> 
> Source: 
> https://github.com/kyllingstad/phobos/blob/std-process2/std/process.d
> 
> Friday April 19 PST will be the last day of voting.

Yes!

Re: Vote for std.process

[Vladimir Panteleev]

On Friday, 12 April 2013 at 08:26:48 UTC, Manu wrote:
Again, I'm just suggesting possibilities, and trying to 
illustrate that
it's a STANDARD library, you can never predict where users will 
want to use

it.


So you are talking about an imaginary system where memory 
allocation is expensive but process creation is cheap.


It's impossible to design and optimize software for some 
intangible goals. None of the systems that D targets, or aims to 
target, meet that criteria.


Can you perhaps quantify how any of those things would be 
reduced by

addressing at least the details I highlight?
I'm basically advocating making a habit of using the stack 
where possible.

It's not exactly hard, or cryptic.


Yes. However, I suggest the following instead:

Please rewrite some part of std.process with performance in mind, 
and post it here for review. This way, we can analyze the 
benefits and drawbacks based on a concrete example, instead of 
vapor and hot air.

Re: Vote for std.process

[Johannes Pfau]

Am Fri, 12 Apr 2013 17:04:08 +1000
schrieb Manu :


> 
> toStringz is used liberally; alternatively, alloca() could allocate
> the c-string's on the stack and zero terminate them there, passing a
> pointer to the stack string to the OS functions.
> 

Are you sure this is possible in this specific case?
The execve manpage says:

execve() does not return on success, and the text, data, bss, and
_stack_ of the calling process are overwritten by that of the program
loaded.

There is no documentation if the environment variables are copied
internally in execve.

Re: Vote for std.process

[Regan Heath]

On Fri, 12 Apr 2013 09:38:53 +0100, Vladimir Panteleev  
 wrote:



On Friday, 12 April 2013 at 08:26:48 UTC, Manu wrote:

Again, I'm just suggesting possibilities, and trying to illustrate that
it's a STANDARD library, you can never predict where users will want to  
use

it.


So you are talking about an imaginary system where memory allocation is  
expensive but process creation is cheap.


It's impossible to design and optimize software for some intangible  
goals. None of the systems that D targets, or aims to target, meet that  
criteria.


I don't understand why you're so "against" these comments.  There is no  
real "argument" here, you both want std.process to be as good as it can be  
and it will clearly be better if it performs faster and allocates less.


Yes, premature optimisation is generally frowned upon in "user" code, but  
library code should ideally be optimised.  Should this percieved lack of  
optimisation prevent it's inclusion now, no because optimisation wont  
change the API (it may extend it however).



Can you perhaps quantify how any of those things would be reduced by
addressing at least the details I highlight?
I'm basically advocating making a habit of using the stack where  
possible.

It's not exactly hard, or cryptic.


Yes. However, I suggest the following instead:

Please rewrite some part of std.process with performance in mind, and  
post it here for review. This way, we can analyze the benefits and  
drawbacks based on a concrete example, instead of vapor and hot air.


Fair point, after all someone has to do the work and it seems logical that  
the person with the most understand of the issue should so it.  Whether he  
has the time however is the Q/problem.


R

--
Using Opera's revolutionary email client: http://www.opera.com/mail/

Re: Vote for std.process

[Vladimir Panteleev]

On Friday, 12 April 2013 at 08:57:19 UTC, Regan Heath wrote:
I don't understand why you're so "against" these comments.  
There is no real "argument" here, you both want std.process to 
be as good as it can be and it will clearly be better if it 
performs faster and allocates less.


This is a fallacy.

Performance is not free.

Somebody must do the work.
D contributors do not have an infinity of time and motivation.

Optimizing code often implies making it more complicated.

Straight-forward code is self-documenting.

More complicated code is harder to read and review.

More complicated code more easily hides bugs - including security 
bugs, such as buffer overflows.


Maintaining optimized code requires understanding not only the 
high-level logic, but also the low-level optimization details.


The benefits of optimizing std.process are likely to be so small, 
as to be difficult to measure.


Would you still say that the above costs are worth the 
nearly-intangible gain?

Re: Vote for std.process

[Dicebot]

On Friday, 12 April 2013 at 08:57:19 UTC, Regan Heath wrote:

...


It is not something to discuss in this topic. Yes, phobos has 
allocations issues. Yes, it is worth discussing. Does std.process 
do something special in that sense? No, possible problems are 
more related to lack of general configurable allocator.


Please do not fill voting thread with noise if you have no 
_blocking_ objections.

Re: Vote for std.process

[Paulo Pinto]

On Friday, 12 April 2013 at 08:34:15 UTC, Johannes Pfau wrote:

Am Fri, 12 Apr 2013 17:04:08 +1000
schrieb Manu :



I've said before, I sadly have to avoid phobos like the 
plague. Some
modules (like this one) that provide fundamental functionality 
- not
just helper functions - can't be avoided. Requirements for 
those

should be extra strict in my opinion.



Most (GC) allocations could be fixed without breaking the API. 
I can

see 2 places where the API forces _GC_ allocations:
* string[string]
* thrown Exceptions are allocated with the GC

There is no simple solution for these. Maybe we'll have a 
hashtable in
the standard library at some point which will allow to use 
custom
allocators. Then std.process could use these. Exceptions 
require some
thinking. You can of course allocate them using any allocator, 
but

freeing them is difficult...

If you want to get rid of all (not only GC) allocations, 
there's another

issue: As D strings are not zero terminated we'll always have
allocations passing those to C code. Maybe we should have a 
cstring
type in phobos which would just be a string which is guaranteed 
to be

zero terminated.

http://dpaste.dzfl.pl/e76aa995 (needs some inout though)



This is the Object Pascal way, although you can reach to a point 
where you get language native strings and C strings mixed all 
over the place.


Not to mention that C strings should not be allowed in safe code.

--
Paulo

Re: Vote for std.process

[H. S. Teoh]

On Fri, Apr 12, 2013 at 06:46:51AM +0200, Jesse Phillips wrote:
> It is that time, If you would like to see the proposed std.process
> include into Phobos please vote yes. If one condition must be met
> specify under what condition, otherwise vote no.

Yes!


--T

Re: Vote for std.process

[John Colvin]

On Friday, 12 April 2013 at 04:46:52 UTC, Jesse Phillips wrote:
It is that time, If you would like to see the proposed 
std.process include into Phobos please vote yes.


Yes! :)

Re: Vote for std.process

[Regan Heath]

On Fri, 12 Apr 2013 10:17:54 +0100, Vladimir Panteleev  
 wrote:



On Friday, 12 April 2013 at 08:57:19 UTC, Regan Heath wrote:
I don't understand why you're so "against" these comments.  There is no  
real "argument" here, you both want std.process to be as good as it can  
be and it will clearly be better if it performs faster and allocates  
less.


This is a fallacy.


Nope, you're attacking a Strawman here.  At most my statement is  
incomplete because it doesn't address the costs (which I left out for  
brevity).



Performance is not free.


I never claimed it was (thus the Strawman).  I simply claimed std.process  
would be "better" if it performed faster and allocated less - which is  
true.



Somebody must do the work.
D contributors do not have an infinity of time and motivation.


I know this as well as anyone, my own free time is in short supply.


Optimizing code often implies making it more complicated.

Straight-forward code is self-documenting.

More complicated code is harder to read and review.

More complicated code more easily hides bugs - including security bugs,  
such as buffer overflows.


Maintaining optimized code requires understanding not only the  
high-level logic, but also the low-level optimization details.


The benefits of optimizing std.process are likely to be so small, as to  
be difficult to measure.


All true.  However, complexity can and should be packaged in such a way as  
to localise, and this localised complex code should be tested to death and  
maintained by someone who understands it.  It should be bracketed by  
sufficient comments and warnings about how/why it does what it does.  The  
resulting packaged complexity, with it's associated cost can be re-used  
many times over for all the benefit it gives.


Stack allocating the environment variables need not be a localised  
improvement but could be a standard library function which can be reused,  
for example.


Would you still say that the above costs are worth the nearly-intangible  
gain?


"nearly-intangible" is wrong. Library code is code which is used by  
(hopefully) millions of people, writing millions of applications, running  
for millions of hours, on millions of systems, creating thousands of  
processes, etc..  In short, a little effort now pays massive dividends  
over it's lifetime.


So, yes, IMO the costs shown above are worth the resulting gains.

D is a systems programming language, there is hope that it will penetrate  
a wide range of systems and environments - sure in many cases a little bit  
of memory use or performance loss is unimportant, but for many it will be  
the decisive factor which makes D unusable there.


D is constantly being compared to other languages on the basis of  
performance, so it's clearly an important aspect of D's success.


Library code needs to work first time and work well or people will roll  
their own wasting time, energy and in many cases getting some aspect of it  
just plain wrong.


Again, I'm not suggesting (nor was Manu I think) that this in any way  
blocks the inclusion of std.process.  But I do share his desire for  
performance to sometimes be a little higher up the priority list.


R

--
Using Opera's revolutionary email client: http://www.opera.com/mail/

Re: Vote for std.process

[Regan Heath]

On Fri, 12 Apr 2013 05:46:51 +0100, Jesse Phillips  
 wrote:


It is that time, If you would like to see the proposed std.process  
include into Phobos please vote yes. If one condition must be met  
specify under what condition, otherwise vote no.


Yes.

--
Using Opera's revolutionary email client: http://www.opera.com/mail/

Re: Vote for std.process

[Andrea Fontana]

On Friday, 12 April 2013 at 04:46:52 UTC, Jesse Phillips wrote:
It is that time, If you would like to see the proposed 
std.process include into Phobos please vote yes. If one 
condition must be met specify under what condition, otherwise 
vote no.


std.process by Lars Kyllingstad and Steven Schveighoffer is a 
suggested improvement to the existing std.process and is a 
major change to the API. The original API remains but these 
will be going through deprecation.


Yes.

A compiler warning like for deprecation should added for 
"experimental" IMO rather than use exp. vs std.

Re: Vote for std.process

[Jacob Carlborg]

On 2013-04-12 09:04, Manu wrote:


I've said before, I sadly have to avoid phobos like the plague. Some
modules (like this one) that provide fundamental functionality - not
just helper functions - can't be avoided. Requirements for those should
be extra strict in my opinion.


Have you tried Tango? For most functions that do some array or string 
processing you have the option of passing your own buffer. If the buffer 
is too small, the function will allocate, otherwise it won't.


Docs: http://www.dsource.org/projects/tango/docs/current/
Tango D2: https://github.com/SiegeLord/Tango-D2

--
/Jacob Carlborg

Re: Vote for std.process

[Vladimir Panteleev]

On Friday, 12 April 2013 at 10:14:35 UTC, Regan Heath wrote:
All true.  However, complexity can and should be packaged in 
such a way as to localise, and this localised complex code 
should be tested to death and maintained by someone who 
understands it.  It should be bracketed by sufficient comments 
and warnings about how/why it does what it does.  The resulting 
packaged complexity, with it's associated cost can be re-used 
many times over for all the benefit it gives.


Stack allocating the environment variables need not be a 
localised improvement but could be a standard library function 
which can be reused, for example.


Performant abstractions. Like the much-awaited allocator design?

Either way, they can only diminuate, not remove the costs.

Would you still say that the above costs are worth the 
nearly-intangible gain?


"nearly-intangible" is wrong. Library code is code which is 
used by (hopefully) millions of people, writing millions of 
applications, running for millions of hours, on millions of 
systems, creating thousands of processes, etc..  In short, a 
little effort now pays massive dividends over it's lifetime.


So, yes, IMO the costs shown above are worth the resulting 
gains.


D is constantly being compared to other languages on the basis 
of performance, so it's clearly an important aspect of D's 
success.


Library code needs to work first time and work well or people 
will roll their own wasting time, energy and in many cases 
getting some aspect of it just plain wrong.


But once again, you speak in vague terms.

Consider the following hypothetical decisions and outcomes:

1. std.process is left at is. One user is angry / turned away 
because it performs 0.1% slower than it can be.


2. std.process is rewritten to minimize allocations. Code 
complexity goes up, new improvements are challenging to add; bugs 
pop up and go unfixed for a while because fewer programmers are 
qualified or willing to commit the effort of making correct 
fixes. More people are angry / turned away from D because its 
standard library is buggy.


Of course, the above is an exaggerated illustration. But would 
optimizing all code left and right really make more D users 
happier?


There's also the question of priorities. Would you rather than 
effort is spent on optimizing std.process (and dealing with all 
the fallout from any such optimizations), or working on something 
that is acutely missing and hurting D?


D is a systems programming language, there is hope that it will 
penetrate a wide range of systems and environments - sure in 
many cases a little bit of memory use or performance loss is 
unimportant, but for many it will be the decisive factor which 
makes D unusable there.


This is surely an exaggeration.

D does not attempt to please everyone out there who is choosing a 
programming language for their next project. There is no such 
language, nor can one exist. One has to accept that D has a 
number of goals, none of which are absolute, but merely point 
towards a certain, but not overly specific, point in the 
multidimensional matrix of trade-offs. D never was about 
achieving maximum performance in all possible cases.

Re: Vote for std.process

[Dmitry Olshansky]

12-Apr-2013 12:26, Manu пишет:

On 12 April 2013 18:18, Vladimir Panteleev mailto:vladi...@thecybershadow.net>> wrote:

On Friday, 12 April 2013 at 07:53:04 UTC, Manu wrote:

Which OS are we talking about?
What OS runs on an a Nintendo Wii? There's only 24mb of system
memory in
that machine, can we afford to allocate it frivolously?


I wasn't aware that writing games for the Wii involves creating
processes in the thousands.


Again, I'm just suggesting possibilities, and trying to illustrate that
it's a STANDARD library, you can never predict where users will want to
use it.


Very true.

--
Dmitry Olshansky

Re: Vote for std.process

[Dmitry Olshansky]

12-Apr-2013 11:04, Manu пишет:

On 12 April 2013 14:46, Jesse Phillips mailto:jessekphillip...@gmail.com>> wrote:

It is that time, If you would like to see the proposed std.process
include into Phobos please vote yes. If one condition must be met
specify under what condition, otherwise vote no.


I didn't think I had much of an opinion on std.process, but I just gave
it a quick once over and noticed, relating to our recent discussions
about flagrant GC usage throughout phobos, that this is riddled with the
exact sorts of issues I've been talking about:

string[string] is used in the main API to receive environment variables;
perhaps kinda convenient, but it's impossible to supply environment
variables with loads of allocations.

toStringz is used liberally; alternatively, alloca() could allocate the
c-string's on the stack and zero terminate them there, passing a pointer
to the stack string to the OS functions.



Most of this is unsolvable until there is a STANDARD 
interface/policy/hooks for allocators in Phobos. All work to get around 
the cases like that at the moment is just a waste of time or specific 
hacks for particular bottlenecks.


I did a fair amount of these in std.regex to make matching GC-free and 
almost heap-free - it preallocates on the first call. It's an example of 
a necessary hack and it's not pretty. More then that I'm sure I'd have 
to re-write the whole memory allocation code once we (finally!) have he 
standard way to do it.


TL;DR - until using allocators is easy and straight-forward (and 
standard) there would be no end of GC-only hegeomny 'cause it's 
easier/simpler. Not much to discuss ;)



String concatenation is rampant! Look at this code to parse the env
variables (which are already an AA):

foreach (var, val; childEnv)

envz[pos++] = (var~'='~val~'\0').ptr;



Horrible...
But let's focus on the API parts. The internals could be fixed with a 
pull request or two. Freaking out because of a GC happy code doesn't 
help those waiting for *any* sane process management module in std.



--
Dmitry Olshansky

Re: Vote for std.process

[Manu]

On 12 April 2013 18:34, Johannes Pfau  wrote:

> Am Fri, 12 Apr 2013 17:04:08 +1000
> schrieb Manu :
>
> >
> > I've said before, I sadly have to avoid phobos like the plague. Some
> > modules (like this one) that provide fundamental functionality - not
> > just helper functions - can't be avoided. Requirements for those
> > should be extra strict in my opinion.
> >
>
> Most (GC) allocations could be fixed without breaking the API. I can
> see 2 places where the API forces _GC_ allocations:
> * string[string]
> * thrown Exceptions are allocated with the GC
>
> There is no simple solution for these. Maybe we'll have a hashtable in
> the standard library at some point which will allow to use custom
> allocators. Then std.process could use these. Exceptions require some
> thinking. You can of course allocate them using any allocator, but
> freeing them is difficult...
>

Exceptions are exceptional, I don't think that's an issue.
the string[string] in the api is guaranteed garbage, and will never be
changed. What's the advantage of using an AA in this case?

If you want to get rid of all (not only GC) allocations, there's another
> issue: As D strings are not zero terminated we'll always have
> allocations passing those to C code.


They are being passed through to a C function that doesn't retain the
pointer; they can be allocated on the stack.

Maybe we should have a cstring
> type in phobos which would just be a string which is guaranteed to be
> zero terminated.
>

Maybe, although the point here is to wrap up and hide the C API, so that's
not useful here. A helper in phobos for allocating a (c)string on the stack
may be useful...

Re: Vote for std.process

[Manu]

On 12 April 2013 18:34, Johannes Pfau  wrote:

> Am Fri, 12 Apr 2013 17:04:08 +1000
> schrieb Manu :
>
> >
> > I've said before, I sadly have to avoid phobos like the plague. Some
> > modules (like this one) that provide fundamental functionality - not
> > just helper functions - can't be avoided. Requirements for those
> > should be extra strict in my opinion.
> >
>
> Most (GC) allocations could be fixed without breaking the API. I can
> see 2 places where the API forces _GC_ allocations:
> * string[string]
> * thrown Exceptions are allocated with the GC
>
> There is no simple solution for these. Maybe we'll have a hashtable in
> the standard library at some point which will allow to use custom
> allocators. Then std.process could use these. Exceptions require some
> thinking. You can of course allocate them using any allocator, but
> freeing them is difficult...
>

Exceptions are exceptional, I don't think that's an issue.
the string[string] in the api is guaranteed garbage, and will never be
changed. What's the advantage of using an AA in this case?

If you want to get rid of all (not only GC) allocations, there's another
> issue: As D strings are not zero terminated we'll always have
> allocations passing those to C code.


They are being passed through to a C function that doesn't retain the
pointer; they can be allocated on the stack.

Maybe we should have a cstring
> type in phobos which would just be a string which is guaranteed to be
> zero terminated.
>

Maybe, although the point here is to wrap up and hide the C API, so that's
not useful here. A helper in phobos for allocating a (c)string on the stack
may be useful...

Re: Vote for std.process

[Jesse Phillips]

This is good feedback and all, but it would be good to have had 
this placed in the official review thread.


But it is too late now and I would not want to break the 
discussion up.

Re: Vote for std.process

[Jesse Phillips]

On Friday, 12 April 2013 at 06:25:10 UTC, Manu wrote:
I see this pattern where something is designed, discussed, and 
then voted
into phobos. At this time the design looks good on paper, but 
there is very

little practical experience using the library.
The problem then is, once accepted, people start using it, and 
at some
point some issues are found, or ideas for improvement are made 
based on
user experience, but the module can no longer be touched due to 
the general

phobia of making breaking changes...


I think this needs to happen prior to the formal review/voting. I 
would say it should be a precursor to starting the official 
review, however this would raise the bar too high for things like 
Jacob's Serialization library; he has a working library, but it 
isn't ready for Phobos and it would be silly to require the 
translation prior to approving it for Phobos.


How we choose to add to the exp module would need some 
consideration.

Re: Vote for std.process

[Andrej Mitrovic]

On 4/12/13, Manu  wrote:
> string[string] is used in the main API to receive environment variables;
> perhaps kinda convenient, but it's impossible to supply environment
> variables with loads of allocations.

I think the functions which take the environment as an associative
array can be changed without breaking any user code. E.g.:

Pid spawnProcess(Env)(in char[][] args,
 File stdin = std.stdio.stdin,
 File stdout = std.stdio.stdout,
 File stderr = std.stdio.stderr,

 /* was const string[string] env = null */
 const Env env = Env.init,

 Config config = Config.none)
if (isSomeAssociativeArray!Env)  // introduce constraint
{
}

Then the user can provide their custom stack-based type which implements opIn_r.

Re: Vote for std.process

[Steven Schveighoffer]

On Fri, 12 Apr 2013 11:24:37 -0400, Andrej Mitrovic  
 wrote:



On 4/12/13, Manu  wrote:

string[string] is used in the main API to receive environment variables;
perhaps kinda convenient, but it's impossible to supply environment
variables with loads of allocations.


I think the functions which take the environment as an associative
array can be changed without breaking any user code. E.g.:

Pid spawnProcess(Env)(in char[][] args,
 File stdin = std.stdio.stdin,
 File stdout = std.stdio.stdout,
 File stderr = std.stdio.stderr,

 /* was const string[string] env = null */
 const Env env = Env.init,

 Config config = Config.none)
if (isSomeAssociativeArray!Env)  // introduce constraint
{
}

Then the user can provide their custom stack-based type which implements  
opIn_r.


It's a good idea.  I believe it can be done later without breaking any  
code.


-Steve

Re: Vote for std.process

[Joshua Niehus]

yes

Re: Vote for std.process

[Steven Schveighoffer]

On Fri, 12 Apr 2013 04:14:15 -0400, Manu  wrote:


On 12 April 2013 17:35, Steven Schveighoffer  wrote:
What would be your suggestion?  string[string] is the built-in map type.

 How do you pass an environment map without having some allocations?



I'd use string[].


You mean with format "a=b"?  I suppose that's possible, though horrible to  
work with before passing in.  Plus what happens if you have ["a=b", "a=c"]  
?  AA's prevent these kinds of mistakes/contradictions.


I think someone suggested using a templated map type, that is probably a  
good idea.  Then you can provide whatever type you wish, with opIndex  
capability.



toStringz is used liberally; alternatively, alloca() could allocate the
 c-string's on the stack and zero terminate them there, passing a  
pointer to

 the stack string to the OS functions.


 This would be a lot of effort for pretty much zero gain for the  
majority of cases.


A trivial ... mixin template (?) could wrap it up, I don't see it  
particularly less convenient than calling toStringz().
Perhaps there are other tools missing from phobos if this is somehow  
hard...


alloca is hard to use, specifically because it can't be wrapped into a  
function.  I don't know how easy it would be to do with a mixin, I haven't  
tried it.


Note that toStringz likely is going to avoid reallocation or copying if  
possible -- it's very likely that a string already has a 0 after it, and  
also very likely that if it doesn't, it has room to append a 0.


I still feel that if it can't be done easily, it over-complicates code for  
very little gain.


As others have said too, this is an implementation detail.  It can be done  
as a later pull request if you feel up to making it work!


-Steve

Re: Vote for std.process

[Steven Schveighoffer]

On Fri, 12 Apr 2013 06:14:35 -0400, Regan Heath   
wrote:


On Fri, 12 Apr 2013 10:17:54 +0100, Vladimir Panteleev  
 wrote:



Optimizing code often implies making it more complicated.

Straight-forward code is self-documenting.

More complicated code is harder to read and review.

More complicated code more easily hides bugs - including security bugs,  
such as buffer overflows.


Maintaining optimized code requires understanding not only the  
high-level logic, but also the low-level optimization details.


The benefits of optimizing std.process are likely to be so small, as to  
be difficult to measure.


All true.  However, complexity can and should be packaged in such a way  
as to localise, and this localised complex code should be tested to  
death and maintained by someone who understands it.  It should be  
bracketed by sufficient comments and warnings about how/why it does what  
it does.  The resulting packaged complexity, with it's associated cost  
can be re-used many times over for all the benefit it gives.


I would say I agree with both of you.  In this particular instance,  
maintainability trumps performance, since creating processes is  
fundamentally not cheap.  But if someone is willing to step up an give us  
a good alternative, there is no reason not to use it.  It should be  
readable and well documented.


I think in order to achieve good performance, we should be focused on  
custom allocators.  That really is the roadblock to all of this.


-Steve

Re: Vote for std.process

[Nick Sabalausky]

On Fri, 12 Apr 2013 13:00:29 +0200
"Vladimir Panteleev"  wrote:
> 
> There's also the question of priorities. Would you rather than 
> effort is spent on optimizing std.process (and dealing with all 
> the fallout from any such optimizations), or working on something 
> that is acutely missing and hurting D?
> 

While I'm not necessarily disagreeing with the other points you raise,
Phobos's excess allocations *are* verifiability hurting D. A couple of
the biggest potential user bases for D are videogames and embedded. I
feel confident in saying that no other language has as much potential
in these areas as D has. But these groups have *already*, vocally, been
facing the problem of avoiding/rewriting potentially-large parts of
Phobos, or sticking with C/C++.

Now, you could argue that many of these people are simply being overly
fearful of a merely imagined problem, but even if that's true, the
problem is still real in it's effects: Hindering D adoption and causing
people to (perhaps needlessly) avoid/rewrite parts of Phobos.

Now, I'm not suggesting that we do *or* don't start a big effort to
minimize allocations throughout Phobos, I'm simply objecting to the
implication that unnecessary allocations in Phobos aren't hurting D in
any way.

Re: Vote for std.process

[Vladimir Panteleev]

On Friday, 12 April 2013 at 16:42:18 UTC, Nick Sabalausky wrote:
While I'm not necessarily disagreeing with the other points you 
raise,
Phobos's excess allocations *are* verifiability hurting D. A 
couple of
the biggest potential user bases for D are videogames and 
embedded. I
feel confident in saying that no other language has as much 
potential
in these areas as D has. But these groups have *already*, 
vocally, been
facing the problem of avoiding/rewriting potentially-large 
parts of

Phobos, or sticking with C/C++.

Now, you could argue that many of these people are simply being 
overly
fearful of a merely imagined problem, but even if that's true, 
the
problem is still real in it's effects: Hindering D adoption and 
causing

people to (perhaps needlessly) avoid/rewrite parts of Phobos.

Now, I'm not suggesting that we do *or* don't start a big 
effort to
minimize allocations throughout Phobos, I'm simply objecting to 
the
implication that unnecessary allocations in Phobos aren't 
hurting D in

any way.


No, that's not what I was trying to say. I agree with your post 
completely. My point was specifically about std.process in 
particular, since I was misunderstanding why Manu was using 
std.process as an example.

Re: Vote for std.process

[Mike Wey]

Yes.

--
Mike Wey

Re: Vote for std.process

[Tove]

On Friday, 12 April 2013 at 15:43:27 UTC, Steven Schveighoffer 
wrote:

On Fri, 12 Apr 2013 04:14:15 -0400, Manu 

I'd use string[].


You mean with format "a=b"?  I suppose that's possible, though 
horrible to work with before passing in.  Plus what happens if 
you have ["a=b", "a=c"] ?  AA's prevent these kinds of 
mistakes/contradictions.




I prefer Manu's idea with the API accepting string[], it's closer 
to the native format.


Then you could simply provide a convenience conversion from a 
map...

ex env!["foo" : "bar"] which would convert it to ["foo=bar"].

This also has the added benefit of being self 
documenting(considering the lack of named parameters).


But most importantly So the user has a free choice of 
constructing the env parameter manually in the most efficient way 
or using the lazy convenience function.

Re: Vote for std.process

[Steven Schveighoffer]

On Fri, 12 Apr 2013 15:26:12 -0400, Tove  wrote:


On Friday, 12 April 2013 at 15:43:27 UTC, Steven Schveighoffer wrote:

On Fri, 12 Apr 2013 04:14:15 -0400, Manu 

I'd use string[].


You mean with format "a=b"?  I suppose that's possible, though horrible  
to work with before passing in.  Plus what happens if you have ["a=b",  
"a=c"] ?  AA's prevent these kinds of mistakes/contradictions.




I prefer Manu's idea with the API accepting string[], it's closer to the  
native format.


That's great for the library on POSIX systems, but for the user, it's not  
as straightforward.  One must construct "x=y" strings instead of setting x  
to y.  Unless you are using literals, this involves an allocation anyway.


Also, this is not the native format on Windows which passes the  
environment in one large string delimited by nulls.




Then you could simply provide a convenience conversion from a map...
ex env!["foo" : "bar"] which would convert it to ["foo=bar"].

This also has the added benefit of being self documenting(considering  
the lack of named parameters).


So for the most convenient/common case, you want to add an allocation?

But most importantly So the user has a free choice of constructing the  
env parameter manually in the most efficient way or using the lazy  
convenience function.


I think the best suggestion so far is to allow a templated version for  
which you can provide an opIndex method, and a possible toEnv method  
(versioned for both Posix and Windows of course).  Then you have the power  
to avoid allocation as much as possible.  At the moment with AA's being  
the interface, we have no possibility to avoid allocation, and I  
understand that point.  But I don't want to eliminate that case or make it  
specifically more inefficient.


-Steve

Re: Vote for std.process

[Tove]

On Friday, 12 April 2013 at 20:24:05 UTC, Steven Schveighoffer 
wrote:
On Fri, 12 Apr 2013 15:26:12 -0400, Tove  
wrote:



So for the most convenient/common case, you want to add an 
allocation?




with the original proposal there is one anyway...

But with my suggested approach you could create many processes 
reusing the same env... only paying the conversion/allocation 
cost once(outside of the loop).

Re: Vote for std.process

[Steven Schveighoffer]

On Fri, 12 Apr 2013 16:32:37 -0400, Tove  wrote:


On Friday, 12 April 2013 at 20:24:05 UTC, Steven Schveighoffer wrote:

On Fri, 12 Apr 2013 15:26:12 -0400, Tove  wrote:


So for the most convenient/common case, you want to add an allocation?



with the original proposal there is one anyway...


I meant add an additional allocation to what is there.  There needs to be  
one allocation to collect all the variables into one long string (on  
Windows), and with your suggestion, it has to go through an intermediate  
"key=value" format.


But with my suggested approach you could create many processes reusing  
the same env... only paying the conversion/allocation cost once(outside  
of the loop).


This would be attractive.  But I still want a indexable object.  Having to  
generate a=b strings is awkward.


It could be something that does the right thing on POSIX (generate "a=b"  
under the hood) or Windows (Probably can cache the generated environment  
string).


-Steve

Re: Vote for std.process

[Tove]

On Friday, 12 April 2013 at 20:52:55 UTC, Steven Schveighoffer 
wrote:
On Fri, 12 Apr 2013 16:32:37 -0400, Tove  
wrote:


On Friday, 12 April 2013 at 20:24:05 UTC, Steven Schveighoffer 
wrote:
On Fri, 12 Apr 2013 15:26:12 -0400, Tove  
wrote:



So for the most convenient/common case, you want to add an 
allocation?




with the original proposal there is one anyway...


I meant add an additional allocation to what is there.  There 
needs to be one allocation to collect all the variables into 
one long string (on Windows), and with your suggestion, it has 
to go through an intermediate "key=value" format.


But with my suggested approach you could create many processes 
reusing the same env... only paying the conversion/allocation 
cost once(outside of the loop).


This would be attractive.  But I still want a indexable object.
 Having to generate a=b strings is awkward.

It could be something that does the right thing on POSIX 
(generate "a=b" under the hood) or Windows (Probably can cache 
the generated environment string).


-Steve


Hmhm, I see your point. Could our custom indexable object have

'alias this' to an 'union env'? which is the in param to the 
process api?

Re: Vote for std.process

[James Wirth]

All progress requires risk. My vote is:
YES

Re: Vote for std.process

[Matej Nanut]

On Friday, 12 April 2013 at 04:46:52 UTC, Jesse Phillips wrote:
It is that time, If you would like to see the proposed 
std.process include into Phobos please vote yes. If one 
condition must be met specify under what condition, otherwise 
vote no.


Yes.

Re: Vote for std.process

[Jacob Carlborg]

On 2013-04-12 16:04, Jesse Phillips wrote:


I think this needs to happen prior to the formal review/voting. I would
say it should be a precursor to starting the official review, however
this would raise the bar too high for things like Jacob's Serialization
library; he has a working library, but it isn't ready for Phobos and it
would be silly to require the translation prior to approving it for Phobos.


What's left is basically just to put it under the std namespace and 
convert tabs to spaces.


--
/Jacob Carlborg

Re: Vote for std.process

[Martin Nowak]

YES

Re: Vote for std.process

[Martin Nowak]

On 04/12/2013 10:14 AM, Manu wrote:

String concatenation is rampant! Look at this code to parse the env
variables (which are already an AA):

 foreach (var, val; childEnv)
 envz[pos++] = (var~'='~val~'\0').ptr;


This could be improved.  It could also be optimized into a single
allocation automatically by the compiler (it might already be).  The
API would not be affected by this improvement, though.


I've never seem the compiler apply that optimisation, although I often
wish it would.
I saw an appender appear a few pages below, that would be an improvement
here too I guess.


It is only a single runtime call to _d_arraycatnT which precomputes the 
resulting length.

https://github.com/D-Programming-Language/druntime/blob/e5415ed4f0638fcd1d0f8e06aec4197ad108449f/src/rt/lifetime.d#L2031

Re: Vote for std.process

[Kapps]

On Friday, 12 April 2013 at 04:46:52 UTC, Jesse Phillips wrote:
It is that time, If you would like to see the proposed 
std.process include into Phobos please vote yes. If one 
condition must be met specify under what condition, otherwise 
vote no.


Yes

Re: Vote for std.process

[Jesse Phillips]

Yes

Was: Re: Vote for std.process

[Regan Heath]

I've moved this to another thread to allay complaints.


"Vladimir Panteleev" 
On Friday, 12 April 2013 at 10:14:35 UTC, Regan Heath wrote:
All true.  However, complexity can and should be packaged insuch a way  
as to localise, and this localised complex codeshould be tested to  
death and maintained by someone whounderstands it.  It should be  
bracketed by sufficient commentsand warnings about how/why it does what  
it does.  The resultingpackaged complexity, with it's associated cost  
can be re-usedmany times over for all the benefit it gives.


Stack allocating the environment variables need not be alocalised  
improvement but could be a standard library functionwhich can be  
reused, for example.


Performant abstractions. Like the much-awaited allocator design?
Either way, they can only diminuate, not remove the costs.


non localised improvements have a fixed cost and ever increasing benefit -  
is the point I was making here.  Agreed, you cannot remove the cost, and  
in fact well written reusable code often carries a slightly higher cost by  
it's very nature.


Would you still say that the above costs are worth the 
nearly-intangible gain?


"nearly-intangible" is wrong. Library code is code which isused by  
(hopefully) millions of people, writing millions ofapplications,  
running for millions of hours, on millions ofsystems, creating  
thousands of processes, etc..  In short, alittle effort now pays  
massive dividends over it's lifetime.


So, yes, IMO the costs shown above are worth the resultinggains.

D is constantly being compared to other languages on the basisof  
performance, so it's clearly an important aspect of D'ssuccess.


Library code needs to work first time and work well or peoplewill roll  
their own wasting time, energy and in many casesgetting some aspect of  
it just plain wrong.


But once again, you speak in vague terms.


The initial point was a vague one, not a specific one.  Manu wasn't  
attempting to block std.process, he had a general concern which I share.



Consider the following hypothetical decisions and outcomes:

1. std.process is left at is. One user is angry / turned awaybecause it  
performs 0.1% slower than it can be.


It very much matters *who* that 1 user is.  And, the count may be higher,  
and we might never "hear" from these people as they find other solutions.   
We're lucky that some people who try D and have issues tell us about them,  
they may be 5% of the total for all we know.


2. std.process is rewritten to minimize allocations. Codecomplexity goes  
up, new improvements are challenging to add; bugspop up and go unfixed  
for a while because fewer programmers arequalified or willing to commit  
the effort of making correctfixes. More people are angry / turned away  
from D because itsstandard library is buggy.


Of course, the above is an exaggerated illustration.


Indeed, you've downplayed point #1 and exaggerated point #2.  In reality  
the suggested improvements would add only very minor complexity and  
prevent none of the current crop of contributors from working with/on  
std.process.


But would optimizing all code left and right really make more D users 
happier?


Yes, as well as the users of their applications.  True, none of them will  
even realise they could have been less happy, so none of them will realise  
the effort that went into it, but all of them will be better off.


There's also the question of priorities. Would you rather thaneffort is  
spent on optimizing std.process (and dealing with allthe fallout from  
any such optimizations), or working on somethingthat is acutely missing  
and hurting D?


Add the missing items, without a doubt - which is why no-one is suggesting  
blocking std.process over this issue.


D is a systems programming language, there is hope that it will 
penetrate a wide range of systems and environments - sure inmany cases  
a little bit of memory use or performance loss isunimportant, but for  
many it will be the decisive factor whichmakes D unusable there.


This is surely an exaggeration.


Why?

There exist platforms and environments where memory and performance are  
concerns, if the D standard library code is not "careful" in it's use of  
both then it will be less suitable than C (for example) and so D will not  
penetrate those platforms.


Manu is using D for games development on modern high-end gaming PCs and he  
is still concerned with memory and performance.


So, there's 2 very different cases where memory and performance are still  
a concern, and .. if they become too much of a concern another solution  
will have to be sought.. and that's bad news for D.


D does not attempt to please everyone out there who is choosing a 
programming language for their next project. There is no suchlanguage,  
nor can one exist. One has to accept that D has anumber of goals, none  
of which are absolute, but merely pointtowards a certain, but not overly  
specific, point in themultidimens

Re: Was: Re: Vote for std.process

[Lars T. Kyllingstad]

On Friday, 12 April 2013 at 11:37:14 UTC, Regan Heath wrote:

I've moved this to another thread to allay complaints.


Thanks!

I completely agree that if code can be made more performant 
without a significant increase in complexity, then we should do 
so.  While it is mostly (but not entirely) irrelevant in the 
context of std.process, it is a problem that should be tackled in 
Phobos as a whole.  Several things could/should be done:


It would be nice to have some sugar on top of alloca(), the use 
of which is usually considered bad practice.  Someone 
(bearophile?) once suggested static arrays whose length is 
determined at runtime, which would be a great addition to the 
language:


void foo(int n)
{
int[n] myArr;
...
}

Furthermore, we need to get the allocator design in place.  In 
SciD, I use David Simcha's region allocator to allocate temporary 
workspace, and it works really well.  The only times I use 'new' 
is when I need persistent memory (e.g. for a return value) and 
the user-supplied buffer is too small.  Phobos would greatly 
benefit from doing this too.


Finally, an example from the new std.process which got some heavy 
criticism in the other thread:


envz[pos++] = (var~'='~val~'\0').ptr;

I have been operating under the assumption that the compiler is 
smart enough to make the above a single allocation.  If it isn't, 
I would consider it a compiler issue.


That said, I am aware that std.process could be improved in some 
places.


Lars

Re: Was: Re: Vote for std.process

[Vladimir Panteleev]

On Friday, 12 April 2013 at 12:30:09 UTC, Lars T. Kyllingstad 
wrote:
Finally, an example from the new std.process which got some 
heavy criticism in the other thread:


envz[pos++] = (var~'='~val~'\0').ptr;

I have been operating under the assumption that the compiler is 
smart enough to make the above a single allocation.  If it 
isn't, I would consider it a compiler issue.


Multiple chained array concatenations are performed at once, 
using the _d_arraycatnT function in Druntime.

Re: Was: Re: Vote for std.process

[Lars T. Kyllingstad]

On Friday, 12 April 2013 at 12:43:57 UTC, Vladimir Panteleev 
wrote:
On Friday, 12 April 2013 at 12:30:09 UTC, Lars T. Kyllingstad 
wrote:
Finally, an example from the new std.process which got some 
heavy criticism in the other thread:


   envz[pos++] = (var~'='~val~'\0').ptr;

I have been operating under the assumption that the compiler 
is smart enough to make the above a single allocation.  If it 
isn't, I would consider it a compiler issue.


Multiple chained array concatenations are performed at once, 
using the _d_arraycatnT function in Druntime.


Good to know.

Lars

Re: Was: Re: Vote for std.process

[Vladimir Panteleev]

On Friday, 12 April 2013 at 11:37:14 UTC, Regan Heath wrote:
The initial point was a vague one, not a specific one.  Manu 
wasn't attempting to block std.process, he had a general 
concern which I share.


OK, but so far my interpretation and replies were mostly in the 
context of std.process - this module being an example where 
performance improvements would have a very small real-life 
benefit. I agree that (generally speaking) improving the 
performance of the code in std.algorithm/array/range would be 
worth the effort and complexity.


It very much matters *who* that 1 user is.  And, the count may 
be higher, and we might never "hear" from these people as they 
find other solutions.  We're lucky that some people who try D 
and have issues tell us about them, they may be 5% of the total 
for all we know.


The same applies to the other side of the argument. A buggy 
standard library probably leaves a worse impression than a slow 
standard library...


In reality the suggested improvements would add only very minor 
complexity and prevent none of the current crop of contributors 
from working with/on std.process.


Well, how do you qualify the amount of optimization that is 
appropriate?


For example, the code in std.process would be even faster, if it 
was completely written in assembler. I hope we'll agree than in 
practice, this would be absurd. Now, what set of well-defined 
arguments would conclude that rewriting it in assembler is 
pointless, but optimizing memory allocations is not? All three 
versions of std.process would perform as well as far as the 
end-user can perceive.


Yes, as well as the users of their applications.  True, none of 
them will even realise they could have been less happy, so none 
of them will realise the effort that went into it, but all of 
them will be better off.


Absolutely - if you ignore the costs. 100%-correct faster code is 
always better than 100%-correct slower code, but the costs are 
the counter-argument.


Add the missing items, without a doubt - which is why no-one is 
suggesting blocking std.process over this issue.


Blocking is one thing, but asking for faster code where it 
doesn't really matter - when there are areas where D could be 
improved at much higher gain per effort - is another.



D is a systems programming language, there is hope that it

Why?

There exist platforms and environments where memory and 
performance are concerns, if the D standard library code is not 
"careful" in it's use of both then it will be less suitable 
than C (for example) and so D will not penetrate those 
platforms.


OK, but once again - how does that line up with the purpose of 
std.process? I can see how std.algorithm can be useful in 
low-spec embedded/gaming systems, but std.process?


Manu is using D for games development on modern high-end gaming 
PCs and he is still concerned with memory and performance.


In Manu's case, every bit of performance counts in the code that 
runs in tight loops, e.g. for every game frame. However, does 
that include std.process?



All true, but performance is one of D's top draw cards:

The D programming language. Modern convenience. Modeling 
power. Native **efficiency**. (**emphasis mine**)


So, it behoves us to make sure the standard library keeps that 
in mind.


Again, I don't (generally) disagree for the general case, however 
I think it pays to mind the context and perspective. When the 
context is std.process and the perspective is the relative cost 
of process creation, it seems like quite a pointless argument.

Re: Was: Re: Vote for std.process

[Lars T. Kyllingstad]

On Friday, 12 April 2013 at 07:04:23 UTC, Manu wrote:


string[string] is used in the main API to receive environment 
variables;
perhaps kinda convenient, but it's impossible to supply 
environment

variables with loads of allocations.


Environment variables are a mapping of strings to strings.  The 
natural way to express such a mapping in D is with a 
string[string].  It shouldn't be necessary to allocate an AA 
literal, though.



toStringz is used liberally; alternatively, alloca() could 
allocate the
c-string's on the stack and zero terminate them there, passing 
a pointer to

the stack string to the OS functions.


It is kind of hard to use alloca() in a safe manner in D, because 
DMD will happily inline functions that use it.  The following 
program will overflow the stack if compiled with -inline:


void doStuff()
{
auto p = alloca(100);
}

void main()
{
foreach (i; 0 .. 1_000_000) doStuff();
}

This is of course fixable, but until that happens, I would 
consider alloca() a no-go for Phobos.

Re: Was: Re: Vote for std.process

[Manu]

On 12 April 2013 22:30, Lars T. Kyllingstad  wrote:

> On Friday, 12 April 2013 at 11:37:14 UTC, Regan Heath wrote:
>
>> I've moved this to another thread to allay complaints.
>>
>
> Thanks!
>
> I completely agree that if code can be made more performant without a
> significant increase in complexity, then we should do so.  While it is
> mostly (but not entirely) irrelevant in the context of std.process, it is a
> problem that should be tackled in Phobos as a whole.  Several things
> could/should be done:
>
> It would be nice to have some sugar on top of alloca(), the use of which
> is usually considered bad practice.  Someone (bearophile?) once suggested
> static arrays whose length is determined at runtime, which would be a great
> addition to the language:
>
> void foo(int n)
> {
> int[n] myArr;
> ...
> }
>

That's beautiful!

Furthermore, we need to get the allocator design in place.  In SciD, I use
> David Simcha's region allocator to allocate temporary workspace, and it
> works really well.  The only times I use 'new' is when I need persistent
> memory (e.g. for a return value) and the user-supplied buffer is too small.
>  Phobos would greatly benefit from doing this too.
>
> Finally, an example from the new std.process which got some heavy
> criticism in the other thread:
>
> envz[pos++] = (var~'='~val~'\0').ptr;
>
> I have been operating under the assumption that the compiler is smart
> enough to make the above a single allocation.  If it isn't, I would
> consider it a compiler issue.
>

Does it? I've not seen the compiler do that, although I'd like to think it
should be possible. 1 allocation is better than 3 I guess, however, I
wonder if that code could be restructured to use the stack aswell.
alloca() is really underrated! I can't imagine why people don't like it.
Perhaps some more helpers built around it might encourage its use? It does
feel a little bit 'raw', like malloc(). It implies some annoying casts.

Re: Was: Re: Vote for std.process

[Manu]

On 12 April 2013 23:08, Vladimir Panteleev wrote:

> On Friday, 12 April 2013 at 11:37:14 UTC, Regan Heath wrote:
>
>> It very much matters *who* that 1 user is.  And, the count may be higher,
>> and we might never "hear" from these people as they find other solutions.
>>  We're lucky that some people who try D and have issues tell us about them,
>> they may be 5% of the total for all we know.
>>
>
> The same applies to the other side of the argument. A buggy standard
> library probably leaves a worse impression than a slow standard library...


If allocating a string on the stack makes it buggy, then there is something
really wrong. It should be no less convenient if appropriate helpers are
available.
With consideration to the string[string] argument, surely instances like
that can be reconsidered? How is string[] going to produce more bugs than
string[string]?
You're being paranoid, or sensationalising the effect of simple
optimisation.

In reality the suggested improvements would add only very minor complexity
>> and prevent none of the current crop of contributors from working with/on
>> std.process.
>>
>
> Well, how do you qualify the amount of optimization that is appropriate?
>

As much is convenient without causing you to start obscuring your code?
That's my personal rule.
But I make it a habit to consider efficiency when designing code, I never
retrofit it. I tend to choose designs that are both simple and efficient at
the start.

For example, the code in std.process would be even faster, if it was
> completely written in assembler. I hope we'll agree than in practice, this
> would be absurd. Now, what set of well-defined arguments would conclude
> that rewriting it in assembler is pointless, but optimizing memory
> allocations is not? All three versions of std.process would perform as well
> as far as the end-user can perceive.


Actually, it would probably be slower if hand-written in assembler. And
again, speed is not my concern here, it's inconsiderate the allocation
policy.

 Yes, as well as the users of their applications.  True, none of them will
>> even realise they could have been less happy, so none of them will realise
>> the effort that went into it, but all of them will be better off.
>>
>
> Absolutely - if you ignore the costs. 100%-correct faster code is always
> better than 100%-correct slower code, but the costs are the
> counter-argument.


Can you describe the 'costs'?

 Add the missing items, without a doubt - which is why no-one is suggesting
>> blocking std.process over this issue.
>>
>
> Blocking is one thing, but asking for faster code where it doesn't really
> matter - when there are areas where D could be improved at much higher gain
> per effort - is another.
>

I'm asking for code that doesn't needlessly allocate, as a policy/habit in
phobos.

 D is a systems programming language, there is hope that it

>>> Why?
>>
>>
>> There exist platforms and environments where memory and performance are
>> concerns, if the D standard library code is not "careful" in it's use of
>> both then it will be less suitable than C (for example) and so D will not
>> penetrate those platforms.
>>
>
> OK, but once again - how does that line up with the purpose of
> std.process? I can see how std.algorithm can be useful in low-spec
> embedded/gaming systems, but std.process?
>
>
>  Manu is using D for games development on modern high-end gaming PCs and
>> he is still concerned with memory and performance.
>>
>
> In Manu's case, every bit of performance counts in the code that runs in
> tight loops, e.g. for every game frame. However, does that include
> std.process?


I'm interested in eliminating allocations. It's just another function that
can't be called in a no-gc area. If it used the stack for its temporaries,
no problem.

 All true, but performance is one of D's top draw cards:
>>
>> The D programming language. Modern convenience. Modeling power.
>> Native **efficiency**. (**emphasis mine**)
>>
>> So, it behoves us to make sure the standard library keeps that in mind.
>>
>
> Again, I don't (generally) disagree for the general case, however I think
> it pays to mind the context and perspective. When the context is
> std.process and the perspective is the relative cost of process creation,
> it seems like quite a pointless argument.
>

It was the first module that appeared for consideration since the recent
discussions about irresponsible GC usage. The argument applies to
everything considered for acceptance into phobos.
I'd like to see it applied as a systematic consideration in the future,
irrespective of the module being considered. Avoiding allocation for
temporaries shouldn't be hard, if some tools are missing, then that is
something that needs further discussion I guess.

Re: Was: Re: Vote for std.process

[Manu]

On 12 April 2013 23:21, Lars T. Kyllingstad  wrote:

> On Friday, 12 April 2013 at 07:04:23 UTC, Manu wrote:
>
>>
>> string[string] is used in the main API to receive environment variables;
>> perhaps kinda convenient, but it's impossible to supply environment
>> variables with loads of allocations.
>>
>
> Environment variables are a mapping of strings to strings.  The natural
> way to express such a mapping in D is with a string[string].  It shouldn't
> be necessary to allocate an AA literal, though.
>

That's a good point, do AA's support literals that don't allocate? You
can't even produce an array literal without it needlessly allocating.

 toStringz is used liberally; alternatively, alloca() could allocate the
>> c-string's on the stack and zero terminate them there, passing a pointer
>> to
>> the stack string to the OS functions.
>>
>
> It is kind of hard to use alloca() in a safe manner in D, because DMD will
> happily inline functions that use it.  The following program will overflow
> the stack if compiled with -inline:
>
> void doStuff()
> {
> auto p = alloca(100);
> }
>
> void main()
> {
> foreach (i; 0 .. 1_000_000) doStuff();
> }
>
> This is of course fixable, but until that happens, I would consider
> alloca() a no-go for Phobos.
>

Very good point. This is a problem.
Hmmm...

Re: Was: Re: Vote for std.process

[Vladimir Panteleev]

On Friday, 12 April 2013 at 13:39:38 UTC, Manu wrote:
If allocating a string on the stack makes it buggy, then there 
is something
really wrong. It should be no less convenient if appropriate 
helpers are

available.


Please see my reply to your other post.

With consideration to the string[string] argument, surely 
instances like
that can be reconsidered? How is string[] going to produce more 
bugs than

string[string]?


env ~= "FOO=BAR";

This will probably not do what you want if there was already a 
line starting with "FOO=" in env.


An array of strings is a less direct representation of the 
environment than a string map. Certain common operations, such as 
finding the value of a variable, or setting / overwriting a 
variable, become more difficult.



You're being paranoid, or sensationalising the effect of simple
optimisation.


Strong words...


And
again, speed is not my concern here, it's inconsiderate the 
allocation

policy.


I'm interested in eliminating allocations. It's just another 
function that
can't be called in a no-gc area. If it used the stack for its 
temporaries,

no problem.


Why allocations, specifically, if not for the performance costs 
of allocation and garbage collection?



Can you describe the 'costs'?


See my previous posts in today's discussions.

As much is convenient without causing you to start obscuring 
your code?

That's my personal rule.
But I make it a habit to consider efficiency when designing 
code, I never
retrofit it. I tend to choose designs that are both simple and 
efficient at

the start.


OK, so if I understand you correctly: you would like Phobos to 
adopt a policy of avoiding heap allocations whenever possible, 
and this argument applies to std.process not because doing so 
would result in a tangible improvement of its performance or 
other metric, but for the purpose of being consistent across 
Phobos. Assuming that the language can provide or allow 
implementing suitably safe abstractions for doing so without 
complicating the code much, I think that's a goal worth looking 
forward, and we have been doing so for some time (hence the 
pending allocator design).

Re: Was: Re: Vote for std.process

[Manu]

On 13 April 2013 00:19, Vladimir Panteleev wrote:

> On Friday, 12 April 2013 at 13:39:38 UTC, Manu wrote:
>
>> If allocating a string on the stack makes it buggy, then there is
>> something
>> really wrong. It should be no less convenient if appropriate helpers are
>> available.
>>
>
> Please see my reply to your other post.
>
>
>  With consideration to the string[string] argument, surely instances like
>> that can be reconsidered? How is string[] going to produce more bugs than
>> string[string]?
>>
>
> env ~= "FOO=BAR";
>
> This will probably not do what you want if there was already a line
> starting with "FOO=" in env.
>
> An array of strings is a less direct representation of the environment
> than a string map. Certain common operations, such as finding the value of
> a variable, or setting / overwriting a variable, become more difficult.


I didn't see any attempt to index the array by key in this case. That's
what an AA is for, and it's not being used here, so it's not a job for an
AA.

I wouldn't use env ~= "FOO=BAR";
I would use env ~= EnvVar("FOO", "BAR");
Or whatever key/value pair structure you like.

 You're being paranoid, or sensationalising the effect of simple
>> optimisation.
>>
>
> Strong words...


Well it seemed appropriate. I can't understand what's so wildly complex
that it would make code utterly unmaintainable, and error prone.

 And
>> again, speed is not my concern here, it's inconsiderate the allocation
>> policy.
>>
>
>  I'm interested in eliminating allocations. It's just another function that
>> can't be called in a no-gc area. If it used the stack for its temporaries,
>> no problem.
>>
>
> Why allocations, specifically, if not for the performance costs of
> allocation and garbage collection?


That's one aspect, but it's also about having control over the allocation
patterns of your program in general. Lots of small allocations fragment the
heap, and they also push the memory barrier.
I couldn't disable the GC and call into phobos functions for very long,
micro-allocations of temporaries not being freed would quickly eat all the
system memory.
Reducing allocations is always better where possible.

As much is convenient without causing you to start obscuring your code?
>>
> That's my personal rule.
>> But I make it a habit to consider efficiency when designing code, I never
>> retrofit it. I tend to choose designs that are both simple and efficient
>> at
>> the start.
>>
>
> OK, so if I understand you correctly: you would like Phobos to adopt a
> policy of avoiding heap allocations whenever possible, and this argument
> applies to std.process not because doing so would result in a tangible
> improvement of its performance or other metric, but for the purpose of
> being consistent across Phobos. Assuming that the language can provide or
> allow implementing suitably safe abstractions for doing so without
> complicating the code much, I think that's a goal worth looking forward,
> and we have been doing so for some time (hence the pending allocator
> design).
>

It starts as soon as the majority agree it's important enough to enforce.
Although I think a tool like: char[len] stackString; would be a
super-useful tool to make this considerably more painless. Some C compilers
support this.

Re: Was: Re: Vote for std.process

[Vladimir Panteleev]

On Friday, 12 April 2013 at 14:58:10 UTC, Manu wrote:
I didn't see any attempt to index the array by key in this 
case. That's
what an AA is for, and it's not being used here, so it's not a 
job for an

AA.


Sorry, not following.

Are you suggesting to use a dynamic array for creating processes, 
but an associative array for examining the current process's 
environment?



I wouldn't use env ~= "FOO=BAR";
I would use env ~= EnvVar("FOO", "BAR");
Or whatever key/value pair structure you like.


OK, but I don't see how it changes your argument. Also, you 
mentioned string[] earlier.


Well it seemed appropriate. I can't understand what's so wildly 
complex

that it would make code utterly unmaintainable, and error prone.


I did not say it would be _utterly_ unmaintainable or error 
prone. Just more so.


It starts as soon as the majority agree it's important enough 
to enforce.

Although I think a tool like: char[len] stackString; would be a
super-useful tool to make this considerably more painless. Some 
C compilers

support this.


I believe this is the feature:

http://en.wikipedia.org/wiki/Variable-length_array

It is part of C99.

-

Earlier today, I wrote:

Please rewrite some part of std.process with performance in 
mind, and post it here for review. This way, we can analyze the 
benefits and drawbacks based on a concrete example, instead of 
vapor and hot air.


I've tried doing this myself, for the bit of code you brought up 
(constructing environment variables). Here's what I ended up with:


-

import std.array;

private struct StaticAppender(size_t SIZE, T)
{
T[SIZE] buffer = void;
Appender!(T[]) appender;
alias appender this;
}

/// Returns a struct containing a fixed-size buffer and an
/// appender. The appender will use the buffer (which has
/// SIZE elements) until it runs out of space, at which
/// point it will reallocate itself on the heap.
auto staticAppender(size_t SIZE, T)()
{
StaticAppender!(SIZE, T) result;
result.appender = appender(result.buffer[]);
return result;
}

/// Allows allocating a T[] given a size.
/// Contains a fized-size buffer of T elements with SIZE
/// length. When asked to allocate an array with
/// length <= than SIZE, the buffer is used instead of
/// the heap.
struct StaticArray(size_t SIZE, T)
{
T[SIZE] buffer = void;
T[] get(size_t size)
{
if (size <= SIZE)
{
buffer[0..size] = T.init;
return buffer[0..size];
}
else
return new T[size];
}
}

// 

void exec(const(char)*[] envz) { /+ ... +/ }

void oldWay(string[string] environment)
{
auto envz = new const(char)*[environment.length + 1];
int pos = 0;
foreach (var, val; environment)
envz[pos++] = (var~'='~val~'\0').ptr;

exec(envz);
}

void newWay(string[string] environment)
{
auto buf = staticAppender!(4096, char)();
StaticArray!(64, size_t) envpBuf;
size_t[] envp = envpBuf.get(environment.length + 1);

size_t pos;
foreach (var, val; environment)
{
envp[pos++] = buf.data.length;
buf.put(var);
buf.put('=');
buf.put(val);
buf.put('\0');
}

// Convert offsets to pointers in-place
auto envz = cast(const(char)*[])envp;
foreach (n; 0..pos)
envz[n] += cast(size_t)buf.data.ptr;

exec(envz);
}

-

As you can see, the code is quite more verbose, even with the 
helper types. It's no longer obvious at a glance what the code is 
doing. Perhaps you can come up with better abstractions?

Re: Was: Re: Vote for std.process

[Manu]

On 13 April 2013 01:29, Vladimir Panteleev wrote:

> On Friday, 12 April 2013 at 14:58:10 UTC, Manu wrote:
>
>> I didn't see any attempt to index the array by key in this case. That's
>> what an AA is for, and it's not being used here, so it's not a job for an
>> AA.
>>
>
> Sorry, not following.
>
> Are you suggesting to use a dynamic array for creating processes, but an
> associative array for examining the current process's environment?


I didn't see anywhere where it was possible to example the current
processed environment? I only saw the mechanism for feeding additional env
vars to the system command.
Linear array of key/value pair struct would be fine.

I wouldn't use env ~= "FOO=BAR";
>> I would use env ~= EnvVar("FOO", "BAR");
>> Or whatever key/value pair structure you like.
>>
>
> OK, but I don't see how it changes your argument. Also, you mentioned
> string[] earlier.


Sorry. I didn't think it through at the time.

It starts as soon as the majority agree it's important enough to enforce.
>>
> Although I think a tool like: char[len] stackString; would be a
>> super-useful tool to make this considerably more painless. Some C
>> compilers
>> support this.
>>
>
> I believe this is the feature:
>
> http://en.wikipedia.org/wiki/**Variable-length_array
>
> It is part of C99.
>
> --**--**-
>
> Earlier today, I wrote:
>
>  Please rewrite some part of std.process with performance in mind, and
>> post it here for review. This way, we can analyze the benefits and
>> drawbacks based on a concrete example, instead of vapor and hot air.
>>
>
> I've tried doing this myself, for the bit of code you brought up
> (constructing environment variables). Here's what I ended up with:
>
> --**--**-
>
> import std.array;
>
> private struct StaticAppender(size_t SIZE, T)
> {
> T[SIZE] buffer = void;
> Appender!(T[]) appender;
> alias appender this;
> }
>
> /// Returns a struct containing a fixed-size buffer and an
> /// appender. The appender will use the buffer (which has
> /// SIZE elements) until it runs out of space, at which
> /// point it will reallocate itself on the heap.
> auto staticAppender(size_t SIZE, T)()
> {
> StaticAppender!(SIZE, T) result;
> result.appender = appender(result.buffer[]);
> return result;
> }
>
> /// Allows allocating a T[] given a size.
> /// Contains a fized-size buffer of T elements with SIZE
> /// length. When asked to allocate an array with
> /// length <= than SIZE, the buffer is used instead of
> /// the heap.
> struct StaticArray(size_t SIZE, T)
> {
> T[SIZE] buffer = void;
> T[] get(size_t size)
> {
> if (size <= SIZE)
> {
> buffer[0..size] = T.init;
> return buffer[0..size];
> }
> else
> return new T[size];
> }
> }
>
> // --**--
>
> void exec(const(char)*[] envz) { /+ ... +/ }
>
> void oldWay(string[string] environment)
> {
> auto envz = new const(char)*[environment.**length + 1];
> int pos = 0;
> foreach (var, val; environment)
>
> envz[pos++] = (var~'='~val~'\0').ptr;
>
> exec(envz);
> }
>
> void newWay(string[string] environment)
> {
> auto buf = staticAppender!(4096, char)();
> StaticArray!(64, size_t) envpBuf;
> size_t[] envp = envpBuf.get(environment.length + 1);
>
> size_t pos;
> foreach (var, val; environment)
> {
> envp[pos++] = buf.data.length;
> buf.put(var);
> buf.put('=');
> buf.put(val);
> buf.put('\0');
> }
>
> // Convert offsets to pointers in-place
> auto envz = cast(const(char)*[])envp;
> foreach (n; 0..pos)
> envz[n] += cast(size_t)buf.data.ptr;
>
> exec(envz);
> }
>
> --**--**-
>
> As you can see, the code is quite more verbose, even with the helper
> types. It's no longer obvious at a glance what the code is doing. Perhaps
> you can come up with better abstractions?
>

Beautiful! Actually, I think if you look a couple of pages below, I think
you'll see something rather like that already there in the windows code.

Thought not sure why you use a size_t array which you just cast to a char*
array?
I think you could also fold that into one pass, rather than 2.
And I'm not sure about this line: envz[n] += cast(size_t)buf.data.ptr;

But those helpers make the problem rather painless.
I wonder if there's opportunity for improvement by having appender support
the ~ operator? Might be able to jig it to use natural concat syntax rather
than put()...

Re: Was: Re: Vote for std.process

[Steven Schveighoffer]

On Fri, 12 Apr 2013 09:42:43 -0400, Manu  wrote:


On 12 April 2013 23:21, Lars T. Kyllingstad  wrote:


On Friday, 12 April 2013 at 07:04:23 UTC, Manu wrote:



string[string] is used in the main API to receive environment  
variables;

perhaps kinda convenient, but it's impossible to supply environment
variables with loads of allocations.



Environment variables are a mapping of strings to strings.  The natural
way to express such a mapping in D is with a string[string].  It  
shouldn't

be necessary to allocate an AA literal, though.



That's a good point, do AA's support literals that don't allocate? You
can't even produce an array literal without it needlessly allocating.


No, because it would have to be COW.

However, a string[string] literal that uses strings only must allocate the  
structure of the AA, not the strings themselves.


The compiler might be able to optimize this by figuring out how much  
memory to allocate in order to contain the entire AA structure, then  
create one block that has all the data in it.  It still needs a new block  
though, otherwise, what happens when you change an AA that was once a  
literal?


I think the best path forward is to replace the functions with a templated  
one that takes a indexable type as the env pointer.  Then one can optimize  
as much as one desires.


-Steve

Re: Was: Re: Vote for std.process

[Vladimir Panteleev]

On Friday, 12 April 2013 at 16:04:09 UTC, Manu wrote:
I didn't see anywhere where it was possible to example the 
current
processed environment? I only saw the mechanism for feeding 
additional env

vars to the system command.
Linear array of key/value pair struct would be fine.


There's the "environment" object. It generally acts like a 
string[string], and has a toAA() method that constructs a real 
string[string].


Beautiful! Actually, I think if you look a couple of pages 
below, I think
you'll see something rather like that already there in the 
windows code.


I see, it also uses appender, although appender's buffer will be 
on the heap, and there is no need for an envz array.


Thought not sure why you use a size_t array which you just cast 
to a char*

array?
I think you could also fold that into one pass, rather than 2.
And I'm not sure about this line: envz[n] += 
cast(size_t)buf.data.ptr;


The problem is that the pointer to the data may change once 
appender reallocates the buffer when it reaches the current 
buffer's capacity. For this reason, we can't store pointers to 
the strings we store, since they can "move" around until the 
point that we're done appending.


I think this is the most common gotcha when writing / working 
with appenders, and it bit be once too. As you can see, the code 
is not completely obvious ;)



But those helpers make the problem rather painless.
I wonder if there's opportunity for improvement by having 
appender support
the ~ operator? Might be able to jig it to use natural concat 
syntax rather

than put()...


Allowing put() take multiple arguments would be an improvement as 
well - not just in usability, but performance as well, since it 
would only need to check for overflow once for all arguments.


I have this in my own appender:
https://github.com/CyberShadow/ae/blob/master/utils/appender.d

Rob Jacques was working on a Phobos appender replacement which 
also had this, I believe:

http://d.puremagic.com/issues/show_bug.cgi?id=5813

Too bad nothing came out of the latter.

Re: Was: Re: Vote for std.process

[Mr. Anonymous]

void oldWay(string[string] environment)
{
auto envz = new const(char)*[environment.length + 1];
int pos = 0;
foreach (var, val; environment)
envz[pos++] = (var~'='~val~'\0').ptr;

exec(envz);
}

void newWay(string[string] environment)
{
auto buf = staticAppender!(4096, char)();
StaticArray!(64, size_t) envpBuf;
size_t[] envp = envpBuf.get(environment.length + 1);

size_t pos;
foreach (var, val; environment)
{
envp[pos++] = buf.data.length;
buf.put(var);
buf.put('=');
buf.put(val);
buf.put('\0');
}

// Convert offsets to pointers in-place
auto envz = cast(const(char)*[])envp;
foreach (n; 0..pos)
envz[n] += cast(size_t)buf.data.ptr;

exec(envz);
}


I just thought of something!
I don't comment here often, but I want to express my opinion:

Currently D, being a system language, offers full control over 
the allocation method, be it the stack or the heap. The helpers 
above show how flexible it is in doing custom optimized stuff if 
one wants to. But there's an obvious drawback, quoting Vladimir:
"the code is quite more verbose, even with the helper types. It's 
no longer obvious at a glance what the code is doing."


But think for a moment - does the programmer usually needs to 
choose the allocation method? Why explicit is default?


It could be the other way around:

void oldWay(string[string] environment)
{
const(char)* envz[environment.length + 1]; // Compiler 
decides whether to use stack or heap.

int pos = 0;
foreach (var, val; environment)
envz[pos++] = (var~'='~val~'\0').ptr; // Use appender. 
Use stack and switch to heap if necessary

exec(envz);
}

How nifty would that be, don't you think?

Another benefit could be that the compiler could adjust the stack 
allocation limit per architecture, and probably it could be 
defined as a command line parameter, e.g. when targeting a low 
end device.


This principle could work on fields other than allocation, e.g. 
parameter passing by ref/value. The programmer needs to only 
specify whether he wants a copy or the actual value, and the 
compiler would decide the optimal way to pass it.

e.g.:

immutable int n;
func(n); // by value

immutable int arr[30];
func(arr); // by ref

But these are probably suggestions for D3 or something... Too 
drastic :)

Re: Was: Re: Vote for std.process

[Mr. Anonymous]

After some Googling, I found out a similar technique already 
exists: auto_buffer in C++.

http://goo.gl/3RLK6

I think it would be perfect to make it the default allocation 
method in D.

Re: Was: Re: Vote for std.process

[Timothee Cour]

> Someone (bearophile?) once suggested static arrays whose length is determined 
> at runtime, which would  > be a great addition to the language:
>void foo(int n){int[n] myArr;}

That would be great. Question about the type: it won't be int[n] as n
is runtime not compile time.
would it be int[] or some other type?

pros of  some other type:
makes some optimizations that benefit from the fact it's stack
allocated possible
cons: we need to be able to reuse existing algos that don't make such
distinction.

maybe another type that would satisfy both type traits:
isDynamicArray!T
isAllocaArray!T

On Sat, Apr 13, 2013 at 4:19 AM, Mr. Anonymous  wrote:
> After some Googling, I found out a similar technique already exists:
> auto_buffer in C++.
> http://goo.gl/3RLK6
>
> I think it would be perfect to make it the default allocation method in D.

'exp' vs 'std'? Forked: Vote for std.process

[Manu]

On 13 April 2013 00:04, Jesse Phillips  wrote:

> On Friday, 12 April 2013 at 06:25:10 UTC, Manu wrote:
>
>> I see this pattern where something is designed, discussed, and then voted
>> into phobos. At this time the design looks good on paper, but there is
>> very
>> little practical experience using the library.
>> The problem then is, once accepted, people start using it, and at some
>> point some issues are found, or ideas for improvement are made based on
>> user experience, but the module can no longer be touched due to the
>> general
>> phobia of making breaking changes...
>>
>
> I think this needs to happen prior to the formal review/voting. I would
> say it should be a precursor to starting the official review, however this
> would raise the bar too high for things like Jacob's Serialization library;
> he has a working library, but it isn't ready for Phobos and it would be
> silly to require the translation prior to approving it for Phobos.
>
> How we choose to add to the exp module would need some consideration.
>

I would say, everything, bar nothing.

The serialisation library is a good example. It's a complicated system, and
it has some history already, offering some confidence from the start.
Experience might suggest that it's more-or-less acceptable into phobos.
It's obviously seen a few projects, but once something is accepted into
phobos, I think it's fair to anticipate its usage to increase
significantly, and in many different kinds of projects.
As an independent library, it might be evaluated by a potential user, and
found not to satisfy the requirements for some reason... they move on and
continue looking at alternatives, nobody's the wiser. Once it's in phobos,
there's a temptation, even an encouragement to use it because it's
'standard'. At this point, it may be found that the project it wasn't
suitable for could be worked with some relatively minor changes, which
should be made, and the library becomes more useful and robust... phobos
doesn't support this pattern at the moment.

I maintain the position that it needs at least a year in the real world
before you can truly be confident in it. New things shouldn't be barred
from post-release tuning on account of "omg it's a breaking change!".

Allocatoin policy in Phobos - Was: Vote for std.process

[Manu]

On 12 April 2013 21:00, Vladimir Panteleev wrote:

> Consider the following hypothetical decisions and outcomes:
>
> 1. std.process is left at is. One user is angry / turned away because it
> performs 0.1% slower than it can be.
>
> 2. std.process is rewritten to minimize allocations. Code complexity goes
> up, new improvements are challenging to add; bugs pop up and go unfixed for
> a while because fewer programmers are qualified or willing to commit the
> effort of making correct fixes. More people are angry / turned away from D
> because its standard library is buggy.
>
> Of course, the above is an exaggerated illustration. But would optimizing
> all code left and right really make more D users happier?
>

Just to be clear, I'm not arguing optimisation for performance here, I'm
arguing intolerance for __unnecessary__ allocations as a policy, or at
least a habit.
There's a whole separate thread on the topic of fighting unnecessary
garbage, and having the ability to use D with strict control over the GC
and/or allocation in general.

If std functions have no reason to allocate, why should they?

There's also the question of priorities. Would you rather than effort is
> spent on optimizing std.process (and dealing with all the fallout from any
> such optimizations), or working on something that is acutely missing and
> hurting D?


If it's somehow hard to put a string on the stack, then there may be a hole
in phobos. I'm not suggesting changes that are somehow hard to implement,
or obscure in some way... they should be utterly trivial.

 D is a systems programming language, there is hope that it will penetrate
>> a wide range of systems and environments - sure in many cases a little bit
>> of memory use or performance loss is unimportant, but for many it will be
>> the decisive factor which makes D unusable there.
>>
>
> This is surely an exaggeration.
>
> D does not attempt to please everyone out there who is choosing a
> programming language for their next project. There is no such language, nor
> can one exist. One has to accept that D has a number of goals, none of
> which are absolute, but merely point towards a certain, but not overly
> specific, point in the multidimensional matrix of trade-offs. D never was
> about achieving maximum performance in all possible cases.
>

And I never suggested we scrap phobos and rewrite it so it maximises
performance at all costs. I highlighted, and suggested trivial changes that
would make a big difference and don't hurt anyone. If it were habit of
phobos devs to generally consider and try and avoid unnecessary allocations
(almost all of which would be approached by using the stack wherever
applicable), the situation would be much better in general. End-users can
write D code however they want, but phobos should strive to be usable in as
many types of software as possible, otherwise what good is a standard
library?

Re: 'exp' vs 'std'? Forked: Vote for std.process

[John Colvin]

On Friday, 12 April 2013 at 14:27:25 UTC, Manu wrote:
On 13 April 2013 00:04, Jesse Phillips 
 wrote:



On Friday, 12 April 2013 at 06:25:10 UTC, Manu wrote:

I see this pattern where something is designed, discussed, 
and then voted
into phobos. At this time the design looks good on paper, but 
there is

very
little practical experience using the library.
The problem then is, once accepted, people start using it, 
and at some
point some issues are found, or ideas for improvement are 
made based on
user experience, but the module can no longer be touched due 
to the

general
phobia of making breaking changes...



I think this needs to happen prior to the formal 
review/voting. I would
say it should be a precursor to starting the official review, 
however this
would raise the bar too high for things like Jacob's 
Serialization library;
he has a working library, but it isn't ready for Phobos and it 
would be
silly to require the translation prior to approving it for 
Phobos.


How we choose to add to the exp module would need some 
consideration.




I would say, everything, bar nothing.

The serialisation library is a good example. It's a complicated 
system, and
it has some history already, offering some confidence from the 
start.
Experience might suggest that it's more-or-less acceptable into 
phobos.
It's obviously seen a few projects, but once something is 
accepted into

phobos, I think it's fair to anticipate its usage to increase
significantly, and in many different kinds of projects.
As an independent library, it might be evaluated by a potential 
user, and
found not to satisfy the requirements for some reason... they 
move on and
continue looking at alternatives, nobody's the wiser. Once it's 
in phobos,
there's a temptation, even an encouragement to use it because 
it's
'standard'. At this point, it may be found that the project it 
wasn't
suitable for could be worked with some relatively minor 
changes, which
should be made, and the library becomes more useful and 
robust... phobos

doesn't support this pattern at the moment.

I maintain the position that it needs at least a year in the 
real world
before you can truly be confident in it. New things shouldn't 
be barred
from post-release tuning on account of "omg it's a breaking 
change!".


I strongly agree with this.

There needs to be a middle step between the wild-west of 
independent libraries and the strictly controlled world of 
standard library.

Re: 'exp' vs 'std'? Forked: Vote for std.process

[Piotr Szturmaj]

http://forum.dlang.org/thread/jcksnp$ah8$1...@digitalmars.com

My idea was to ship exp branch with phobos so all users could look into 
new code. Not all users follow github/newsgroups/etc.

Re: 'exp' vs 'std'? Forked: Vote for std.process

[Manu]

On 13 April 2013 00:41, Piotr Szturmaj  wrote:

> http://forum.dlang.org/thread/**jcksnp$ah8$1...@digitalmars.com
>
> My idea was to ship exp branch with phobos so all users could look into
> new code. Not all users follow github/newsgroups/etc.
>

Why a branch though? That implies that people are git-savvy (not as common
as you think), and that they even have git installed at all...
It should be an exp directory next to std, so anyone can use it.

Re: 'exp' vs 'std'? Forked: Vote for std.process

[John Colvin]

On Friday, 12 April 2013 at 15:05:52 UTC, Manu wrote:
On 13 April 2013 00:41, Piotr Szturmaj  
wrote:



http://forum.dlang.org/thread/**jcksnp$ah8$1...@digitalmars.com

My idea was to ship exp branch with phobos so all users could 
look into

new code. Not all users follow github/newsgroups/etc.



Why a branch though? That implies that people are git-savvy 
(not as common

as you think), and that they even have git installed at all...
It should be an exp directory next to std, so anyone can use it.


Agreed. Git should not be required to *use* phobos, only to 
*contribute* to it.

Re: 'exp' vs 'std'? Forked: Vote for std.process

[Piotr Szturmaj]

W dniu 12.04.2013 17:05, Manu pisze:

On 13 April 2013 00:41, Piotr Szturmaj mailto:bncr...@jadamspam.pl>> wrote:

http://forum.dlang.org/thread/__jcksnp$ah8$1...@digitalmars.com


My idea was to ship exp branch with phobos so all users could look
into new code. Not all users follow github/newsgroups/etc.


Why a branch though? That implies that people are git-savvy (not as
common as you think), and that they even have git installed at all...
It should be an exp directory next to std, so anyone can use it.


I'm sorry, I really meant exp directory, as another "branch" of phobos - 
not related to git (or any VCS) in any way.

Re: 'exp' vs 'std'? Forked: Vote for std.process

[Manu]

On 13 April 2013 01:27, Piotr Szturmaj  wrote:

> W dniu 12.04.2013 17:05, Manu pisze:
>
>> On 13 April 2013 00:41, Piotr Szturmaj > > wrote:
>>
>> 
>> http://forum.dlang.org/thread/**__jcksnp$ah8$1...@digitalmars.com
>>
>> 
>> 
>> >
>>
>> My idea was to ship exp branch with phobos so all users could look
>> into new code. Not all users follow github/newsgroups/etc.
>>
>>
>> Why a branch though? That implies that people are git-savvy (not as
>> common as you think), and that they even have git installed at all...
>> It should be an exp directory next to std, so anyone can use it.
>>
>
> I'm sorry, I really meant exp directory, as another "branch" of phobos -
> not related to git (or any VCS) in any way.
>

Fair enough, that thread above went off on a tangent saying they should use
branches, so I thought that was the idea.

Re: 'exp' vs 'std'? Forked: Vote for std.process

[Steven Schveighoffer]

On Fri, 12 Apr 2013 10:27:11 -0400, Manu  wrote:


I maintain the position that it needs at least a year in the real world
before you can truly be confident in it. New things shouldn't be barred
from post-release tuning on account of "omg it's a breaking change!".


I hate the 'omg it's a breaking change!' mentality as well.  This is an  
UNRELEASED product.


I also hate the idea of creating another javax.

Here is what I would suggest for new modules (std.log, std.serialization,  
etc.):


pragma(msg, module.stringof ~ " is experimental, subject to API changes");

After 1 year, remove the pragma, and freeze the API.

Now, if there were only some way to make pragma(msg...) only print out  
when imported from non-library code...


-Steve

Re: 'exp' vs 'std'? Forked: Vote for std.process

[Manu]

On 13 April 2013 02:07, Steven Schveighoffer  wrote:

> On Fri, 12 Apr 2013 10:27:11 -0400, Manu  wrote:
>
>  I maintain the position that it needs at least a year in the real world
>> before you can truly be confident in it. New things shouldn't be barred
>> from post-release tuning on account of "omg it's a breaking change!".
>>
>
> I hate the 'omg it's a breaking change!' mentality as well.  This is an
> UNRELEASED product.
>
> I also hate the idea of creating another javax.
>

I don't understand this problem? Why is there a problem moving it when it's
ready?
It's already understood to be experimental, and the user has already
accepted the contract that changes can be made (including moving it to std).

Re: 'exp' vs 'std'? Forked: Vote for std.process

[Nick Sabalausky]

On Sat, 13 Apr 2013 00:27:11 +1000
Manu  wrote:

> On 13 April 2013 00:04, Jesse Phillips 
> wrote:
> 
> > On Friday, 12 April 2013 at 06:25:10 UTC, Manu wrote:
> >
> >> I see this pattern where something is designed, discussed, and
> >> then voted into phobos. At this time the design looks good on
> >> paper, but there is very
> >> little practical experience using the library.
> >> The problem then is, once accepted, people start using it, and at
> >> some point some issues are found, or ideas for improvement are
> >> made based on user experience, but the module can no longer be
> >> touched due to the general
> >> phobia of making breaking changes...
> >>

While I think you raise a very good point, I agree with the
possible solutions Vladimir suggested in the original thread: Either
add them straight to 'std' from the start and just mark it with a big
red "EXPERIMENTAL" or perhaps more accurately "NEW MODULE -
STILL SUBJECT TO CHANGE!", or make it super-easy to test such modules
when they're in the review queue (perhaps via DUB?). Actually, I'd like
to see *both*.

The "exp vs std" suggestion *could* work, but I think it his a
somewhat higher potential for unintended problems down the road.

Re: 'exp' vs 'std'? Forked: Vote for std.process

[Steven Schveighoffer]

On Fri, 12 Apr 2013 12:18:30 -0400, Manu  wrote:


On 13 April 2013 02:07, Steven Schveighoffer  wrote:


On Fri, 12 Apr 2013 10:27:11 -0400, Manu  wrote:

 I maintain the position that it needs at least a year in the real world

before you can truly be confident in it. New things shouldn't be barred
from post-release tuning on account of "omg it's a breaking change!".



I hate the 'omg it's a breaking change!' mentality as well.  This is an
UNRELEASED product.

I also hate the idea of creating another javax.



I don't understand this problem? Why is there a problem moving it when  
it's

ready?
It's already understood to be experimental, and the user has already
accepted the contract that changes can be made (including moving it to  
std).


Intentions don't always equal reality.  If enough people make a stink, it  
could result in exp being the "official" release of some modules.  See  
standard location of Java's xml library.


I like the pragma idea because it does not stop compilation, serves as an  
adequate warning that things are in flux, and turning off the pragma  
breaks no code whatsoever.  It's basically a warning that your code may  
break.  But if the API is stable, it will be fine, and no changes are  
needed.


With the exp version, you either keep exp.module around forever as a  
public link to the std.module, or you force people's code to break even  
though no API has changed.


-Steve

Re: 'exp' vs 'std'? Forked: Vote for std.process

[John Colvin]

On Friday, 12 April 2013 at 16:31:50 UTC, Steven Schveighoffer 
wrote:
With the exp version, you either keep exp.module around forever 
as a public link to the std.module, or you force people's code 
to break even though no API has changed.


-Steve


I like the pragma idea better, but I don't see why an exp.module 
link couldn't just be deprecated then removed. Even if you have 
to keep it around for ages, it's hardly going to get in the way 
much.

Re: 'exp' vs 'std'? Forked: Vote for std.process

[Andrej Mitrovic]

On 4/12/13, Steven Schveighoffer  wrote:
> pragma(msg, module.stringof ~ " is experimental, subject to API changes");

I like this better:

version(UseExperimentalProcess) { }
else pragma(msg, module.stringof ~ " is experimental, subject to API changes");

That way I can turn off the warning by passing
-version=UseExperimentalProcess on the command-line. That of course
means I take full responsibility for using experimental code.

Re: 'exp' vs 'std'? Forked: Vote for std.process

[Steven Schveighoffer]

On Fri, 12 Apr 2013 12:40:46 -0400, John Colvin  
 wrote:



I like the pragma idea better, but I don't see why an exp.module link  
couldn't just be deprecated then removed. Even if you have to keep it  
around for ages, it's hardly going to get in the way much.


There's nothing wrong with it, it's just cruft.  And in order to remove  
it, you necessarily must break code.


I just like the no-cruft method better, seems pretty much equivalent to me  
without the code breaking.


-Steve