[Issue 18613] Documentation: recommended construction/destruction patterns for manual memory management

[d-bugmail--- via Digitalmars-d-bugs]

https://issues.dlang.org/show_bug.cgi?id=18613

Iain Buclaw  changed:

   What|Removed |Added

   Priority|P1  |P4

--

[Issue 18613] Documentation: recommended construction/destruction patterns for manual memory management

[d-bugmail--- via Digitalmars-d-bugs]

https://issues.dlang.org/show_bug.cgi?id=18613

--- Comment #2 from Manu  ---
He's interacting with existing C++ code, and doesn't want to have conflicting
allocation strategies. It's gamedev, so custom allocation mechanisms are
already in use.

He did find the cpptod, read it, but he suggested there were still questions.
I'll dig for his thoughts at lunch tomorrow :)

I'll point him to std.experimental.allocator.

Thanks!

--

[Issue 18613] Documentation: recommended construction/destruction patterns for manual memory management

[d-bugmail--- via Digitalmars-d-bugs]

https://issues.dlang.org/show_bug.cgi?id=18613

greenify  changed:

   What|Removed |Added

 CC||greeen...@gmail.com

--- Comment #1 from greenify  ---
Thanks a lot for opening the request on his behalf.
(We might modernize Bugzilla, so stay tuned)

Does your colleague know about the GC
series?https://dlang.org/blog/the-gc-series

Also did your colleague find https://dlang.org/articles/cpptod.html?

Maybe this page could then be expanded.

> Current solution, make a function myDelete that calls dtor and free by hand.

The preferred way is to use make and dispose from std.experimental.allocator -
they are the D analogies to malloc and free though a lot smarter (and dispose
calls the destructor).

--

[Issue 18613] New: Documentation: recommended construction/destruction patterns for manual memory management

[d-bugmail--- via Digitalmars-d-bugs]

https://issues.dlang.org/show_bug.cgi?id=18613

  Issue ID: 18613
   Summary: Documentation: recommended construction/destruction
patterns for manual memory management
   Product: D
   Version: D2
  Hardware: All
OS: All
Status: NEW
  Severity: enhancement
  Priority: P1
 Component: dlang.org
  Assignee: nob...@puremagic.com
  Reporter: turkey...@gmail.com

A colleague couldn't find an article he expected to exist.
I'm posting this on his behalf, because bugzilla doesn't use github login
credentials "yeah I literally clicked twice, was asked for login, closed :P"

```
Coming from C++ I was looking for delete after I did my new. I then found that
there was no delete because of GC.
To stay close to my C++ approach I decided not to use GC which then prompted
the issue, how do I call the dtor + free.
Current solution, make a function myDelete that calls dtor and free by hand.
Could use an article explaining the approach and highlighting pros and cons for
each method. It would also be useful to have a section on what is the preferred
way to do this manual memory management in D and why.
```

--

Re: Example of code with manual memory management

[Kagamin via Digitalmars-d-learn]

https://dlang.org/phobos/std_container.html and corresponding 
code in phobos. Though recently allocators were introduced and 
containers are going to be written with support for allocators.

Example of code with manual memory management

[Dibyendu Majumdar via Digitalmars-d-learn]

Hi,

I am looking for example of types where memory management is 
manual, and the type supports operator overloading, etc. Grateful 
if someone could point me to sample example code.


Thanks and Regards
Dibyendu

Re: associative arrays with manual memory management

[Ilya Yaroshenko via Digitalmars-d-announce]

On Wednesday, 26 August 2015 at 12:10:17 UTC, Per Nordlöw wrote:

On Wednesday, 26 August 2015 at 10:48:11 UTC, Ilya Yaroshenko
>> auto a  = aa!(string, int)(Mallocator.instance); // 3rd CT

highlights

It would be nice to also see an example at
https://github.com/arexeu/aammm

that shows AA-usage in conjunction with some other allocator 
such as FreeList and add a note about the performance 
improvement this gives.


I have not any significant performance improvement with 
FreeList/SharedFreeList comparing with Mallocator. However, the 
difference can be significant for server with 20-30 core CPUs 
because malloc function use internal lock.


Simple benchmark:
https://github.com/arexeu/aammm/commit/260ef4f94618b85463dec5c90e0b894b08750d07#diff-24efeb785f1d7039ab1c9bc29ba99c99R1071

Re: associative arrays with manual memory management

[Dmitry Olshansky via Digitalmars-d-announce]

On 24-Aug-2015 15:01, Ilya Yaroshenko wrote:

http://code.dlang.org/packages/aammm/~master

# aammm
Associative arrays with manual memory management

All enries and buckets would be dealocated and disposed by internal
implementation's destructor.
The destructor is called by garbage collector (by default).



Rox!


 Example
```D
 //std.experimental.allocator is included into `aammm`
 import std.experimental.allocator.mallocator;
 import aammm;

 auto a = AA!(string, int, shared Mallocator)(Mallocator.instance);


Sure hope a factory to do IFTI is available? So that the following works:

auto a  = aa!(string, int)(Mallocator.instance); // 3rd CT param is deduced


 a[foo] = 0;
 a.remove(foo); //dealocates and disposes the entry
 assert(a == null); // should not crash
```




--
Dmitry Olshansky

Re: associative arrays with manual memory management

[via Digitalmars-d-announce]

On Monday, 24 August 2015 at 12:01:52 UTC, Ilya Yaroshenko wrote:

http://code.dlang.org/packages/aammm/~master


It would be nice to have a test example for other allocators. I'm 
especially interested in how much speed we can gain with using a 
non-shared BlockAllocator in combination with aammm.AA.


Can the whole AA be made `@safe pure nothrow` if the allocator is 
stored inside the AA itself (non-shared)?


I'm especially interested in using this for keys and values only 
with no indirections.

Re: associative arrays with manual memory management

[via Digitalmars-d-announce]

On Monday, 24 August 2015 at 12:01:52 UTC, Ilya Yaroshenko wrote:

Nice! I'll try this!


auto a = AA!(string, int, shared Mallocator)


When does the third parameter need to be qualified as `shared`?

Re: associative arrays with manual memory management

[Ilya Yaroshenko via Digitalmars-d-announce]

On Wednesday, 26 August 2015 at 06:50:26 UTC, Per Nordlöw wrote:
On Monday, 24 August 2015 at 12:01:52 UTC, Ilya Yaroshenko 
wrote:

http://code.dlang.org/packages/aammm/~master


It would be nice to have a test example for other allocators. 
I'm especially interested in how much speed we can gain with 
using a non-shared BlockAllocator in combination with aammm.AA.


Can the whole AA be made `@safe pure nothrow` if the allocator 
is stored inside the AA itself (non-shared)?


EDIT: Plus I think to RefCountingAA.

I'm especially interested in using this for keys and values 
only with no indirections.

Re: associative arrays with manual memory management

[Ilya Yaroshenko via Digitalmars-d-announce]

On Wednesday, 26 August 2015 at 06:50:26 UTC, Per Nordlöw wrote:
On Monday, 24 August 2015 at 12:01:52 UTC, Ilya Yaroshenko 
wrote:

http://code.dlang.org/packages/aammm/~master


It would be nice to have a test example for other allocators. 
I'm especially interested in how much speed we can gain with 
using a non-shared BlockAllocator in combination with aammm.AA.


Can the whole AA be made `@safe pure nothrow` if the allocator 
is stored inside the AA itself (non-shared)?



Yes, except constructor. To do not use constructor you can use 
`makeAA` and `disposeAA`.


I'm especially interested in using this for keys and values 
only with no indirections.

Re: associative arrays with manual memory management

[Ilya Yaroshenko via Digitalmars-d-announce]

On Wednesday, 26 August 2015 at 06:41:41 UTC, Per Nordlöw wrote:
On Monday, 24 August 2015 at 12:01:52 UTC, Ilya Yaroshenko 
wrote:


Nice! I'll try this!


auto a = AA!(string, int, shared Mallocator)


When does the third parameter need to be qualified as `shared`?


Only if you would use a shared allocator like Mallocator or 
GCAllocator.


I will add factory template like Dmitry suggested.

auto a  = aa!(string, int)(Mallocator.instance); // 3rd CT param 
is deduced


Ilya

Re: associative arrays with manual memory management

[Ilya Yaroshenko via Digitalmars-d-announce]

On Wednesday, 26 August 2015 at 06:52:01 UTC, Dmitry Olshansky 
wrote:


 auto a = AA!(string, int, shared 
Mallocator)(Mallocator.instance);


Sure hope a factory to do IFTI is available? So that the 
following works:


auto a  = aa!(string, int)(Mallocator.instance); // 3rd CT 
param is deduced




Just added to master branch) Thanks!

Re: associative arrays with manual memory management

[via Digitalmars-d-announce]

On Wednesday, 26 August 2015 at 10:48:11 UTC, Ilya Yaroshenko  
auto a  = aa!(string, int)(Mallocator.instance); // 3rd CT


highlights

It would be nice to also see an example at
https://github.com/arexeu/aammm

that shows AA-usage in conjunction with some other allocator such 
as FreeList and add a note about the performance improvement this 
gives.


In that case, would it be possible to have factory functions for 
AA that automatically derives allocator parameters from key and 
value type for specific allocators?


What do you say?

Re: associative arrays with manual memory management

[via Digitalmars-d-announce]

On Wednesday, 26 August 2015 at 14:21:41 UTC, Ilya Yaroshenko 
wrote:

It is possible but not so useful, thought.


Why is it not so useful?

`AA!(...).Entry.sizeof` and `AA!(...).Entry.alignof` should be 
accessible from user code since v0.0.3 . They can be used to 
construct allocators. I will add example with FreeList.


Great! Thanks!

Re: associative arrays with manual memory management

[Ilya Yaroshenko via Digitalmars-d-announce]

On Wednesday, 26 August 2015 at 12:10:17 UTC, Per Nordlöw wrote:

highlights

It would be nice to also see an example at
https://github.com/arexeu/aammm

that shows AA-usage in conjunction with some other allocator 
such as FreeList and add a note about the performance 
improvement this gives.


In that case, would it be possible to have factory functions 
for AA that automatically derives allocator parameters from key 
and value type for specific allocators?


What do you say?


It is possible but not so useful, thought. 
`AA!(...).Entry.sizeof` and `AA!(...).Entry.alignof` should be 
accessible from user code since v0.0.3 . They can be used to 
construct allocators. I will add example with FreeList.

Re: associative arrays with manual memory management

[Ilya Yaroshenko via Digitalmars-d-announce]

On Wednesday, 26 August 2015 at 14:24:54 UTC, Per Nordlöw wrote:
On Wednesday, 26 August 2015 at 14:21:41 UTC, Ilya Yaroshenko 
wrote:

It is possible but not so useful, thought.


Why is it not so useful?


Because looks like allocators have different additional params, 
that dose not related to AA.

Re: associative arrays with manual memory management

[Ilya Yaroshenko via Digitalmars-d-announce]

On Wednesday, 26 August 2015 at 13:12:38 UTC, Per Nordlöw wrote:
On Wednesday, 26 August 2015 at 09:20:33 UTC, Ilya Yaroshenko 
wrote:
Only if you would use a shared allocator like Mallocator or 
GCAllocator.


Are there cases where a non-shared version of Mallocator or 
GCAllocator is motivated?


Mallocator and GCAllocator are always shared.
Possibly D will have ThreadLocalGCAllocator in the future.



If not could, maybe the shared-ness could be inferred?


`aa` function just uses shared type for shared argument and vise 
versa.

GCAllocator.instance and Mallocator.instance. are always shared.

Re: associative arrays with manual memory management

[via Digitalmars-d-announce]

On Wednesday, 26 August 2015 at 09:20:33 UTC, Ilya Yaroshenko 
wrote:
Only if you would use a shared allocator like Mallocator or 
GCAllocator.


Are there cases where a non-shared version of Mallocator or 
GCAllocator is motivated?


If not could, maybe the shared-ness could be inferred?

Re: associative arrays with manual memory management

[Ilya Yaroshenko via Digitalmars-d]

On Saturday, 22 August 2015 at 04:16:30 UTC, Rikki Cattermole 
wrote:

On 8/22/2015 5:20 AM, Ilya Yaroshenko wrote:

Hi All!

I am going to implement associative arrays with manual memory 
management

based on amazing std.experimental.allocator by Andrei
http://wiki.dlang.org/Review/std.experimental.allocator

I will be happy to receive any advices about algorithms, use 
cases and API.


Best Regards,
Ilya


Will it be language feature fix, or is it an independent 
container?
If the later I already have a simple dumb one which I can share 
(not on this machine). I'll be happy to use what you create. 
Same goes for list and friends ones.


Annonce
http://forum.dlang.org/post/hgawkhhbvkkxbnjzi...@forum.dlang.org

associative arrays with manual memory management

[Ilya Yaroshenko via Digitalmars-d-announce]

http://code.dlang.org/packages/aammm/~master

# aammm
Associative arrays with manual memory management

All enries and buckets would be dealocated and disposed by 
internal implementation's destructor.

The destructor is called by garbage collector (by default).

 Example
```D
//std.experimental.allocator is included into `aammm`
import std.experimental.allocator.mallocator;
import aammm;

auto a = AA!(string, int, shared 
Mallocator)(Mallocator.instance);

a[foo] = 0;
a.remove(foo); //dealocates and disposes the entry
assert(a == null); // should not crash
```
AAMMM is based on Andrei's allocators and Martin's associative 
arrays.


References:
http://erdani.com/d/phobos-prerelease/std_experimental_allocator.html
https://github.com/D-Programming-Language/druntime/pull/1282

Best Regards,
Ilya

Re: associative arrays with manual memory management

[extrawurst via Digitalmars-d-announce]

On Monday, 24 August 2015 at 12:01:52 UTC, Ilya Yaroshenko wrote:

http://code.dlang.org/packages/aammm/~master

# aammm
Associative arrays with manual memory management

[...]


Awesome, I was waiting for something like that. Thank you!

Re: associative arrays with manual memory management

[Dmitry Olshansky via Digitalmars-d]

On 22-Aug-2015 10:46, rsw0x wrote:

On Saturday, 22 August 2015 at 07:37:38 UTC, Dmitry Olshansky wrote:

On 21-Aug-2015 20:20, Ilya Yaroshenko wrote:

Hi All!

I am going to implement associative arrays with manual memory management
based on amazing std.experimental.allocator by Andrei
http://wiki.dlang.org/Review/std.experimental.allocator

I will be happy to receive any advices about algorithms, use cases
and API.

Best Regards,
Ilya


FYI
https://github.com/D-Programming-Language/druntime/pull/1282


Maybe someone who isn't confused by dmd could answer this for me, but
why are the druntime hooks generated by dmd non-templated and rely on
dynamic info(rtti) when all the information is known at compile time?


druntime predates D templates.

--
Dmitry Olshansky

Re: associative arrays with manual memory management

[Ilya Yaroshenko via Digitalmars-d]

On Saturday, 22 August 2015 at 04:16:30 UTC, Rikki Cattermole 
wrote:
Will it be language feature fix, or is it an independent 
container?


Independent container.

If the later I already have a simple dumb one which I can share 
(not on this machine). I'll be happy to use what you create. 
Same goes for list and friends ones.


After AA I will look at RedBlackTree. However looks like Andrei 
have great plans about new std.experemental.collection.


Ilya

Re: associative arrays with manual memory management

[Dmitry Olshansky via Digitalmars-d]

On 21-Aug-2015 20:20, Ilya Yaroshenko wrote:

Hi All!

I am going to implement associative arrays with manual memory management
based on amazing std.experimental.allocator by Andrei
http://wiki.dlang.org/Review/std.experimental.allocator

I will be happy to receive any advices about algorithms, use cases and API.

Best Regards,
Ilya


FYI
https://github.com/D-Programming-Language/druntime/pull/1282

--
Dmitry Olshansky

Re: associative arrays with manual memory management

[rsw0x via Digitalmars-d]

On Saturday, 22 August 2015 at 07:46:22 UTC, rsw0x wrote:
On Saturday, 22 August 2015 at 07:37:38 UTC, Dmitry Olshansky 
wrote:

On 21-Aug-2015 20:20, Ilya Yaroshenko wrote:

Hi All!

I am going to implement associative arrays with manual memory 
management

based on amazing std.experimental.allocator by Andrei
http://wiki.dlang.org/Review/std.experimental.allocator

I will be happy to receive any advices about algorithms, use 
cases and API.


Best Regards,
Ilya


FYI
https://github.com/D-Programming-Language/druntime/pull/1282


Maybe someone who isn't confused by dmd could answer this for 
me, but why are the druntime hooks generated by dmd 
non-templated and rely on dynamic info(rtti) when all the 
information is known at compile time?


I meant that wrt the AA implementation linked, if it did not seem 
obvious by the way. But my question applies to most of the 
druntime hooks. Sorry for doublepost, wanted to clarify.

Re: associative arrays with manual memory management

[rsw0x via Digitalmars-d]

On Saturday, 22 August 2015 at 07:37:38 UTC, Dmitry Olshansky 
wrote:

On 21-Aug-2015 20:20, Ilya Yaroshenko wrote:

Hi All!

I am going to implement associative arrays with manual memory 
management

based on amazing std.experimental.allocator by Andrei
http://wiki.dlang.org/Review/std.experimental.allocator

I will be happy to receive any advices about algorithms, use 
cases and API.


Best Regards,
Ilya


FYI
https://github.com/D-Programming-Language/druntime/pull/1282


Maybe someone who isn't confused by dmd could answer this for me, 
but why are the druntime hooks generated by dmd non-templated and 
rely on dynamic info(rtti) when all the information is known at 
compile time?

Re: associative arrays with manual memory management

[Rikki Cattermole via Digitalmars-d]

On 8/22/2015 9:44 PM, Ilya Yaroshenko wrote:

On Saturday, 22 August 2015 at 04:16:30 UTC, Rikki Cattermole wrote:

Will it be language feature fix, or is it an independent container?


Independent container.


If the later I already have a simple dumb one which I can share (not
on this machine). I'll be happy to use what you create. Same goes for
list and friends ones.


After AA I will look at RedBlackTree. However looks like Andrei have
great plans about new std.experemental.collection.


Yeah he does. I'm quite excited by it.
Although I think right now he is focusing more on a linear 
algerbra/matrix family library. Based upon his response of when I said I 
was trying to get relicense rights to gl3n.

Re: associative arrays with manual memory management

[Ilya Yaroshenko via Digitalmars-d]

On Saturday, 22 August 2015 at 07:37:38 UTC, Dmitry Olshansky 
wrote:


FYI
https://github.com/D-Programming-Language/druntime/pull/1282


Thanks!

Re: associative arrays with manual memory management

[Rikki Cattermole via Digitalmars-d]

On 8/22/2015 5:20 AM, Ilya Yaroshenko wrote:

Hi All!

I am going to implement associative arrays with manual memory management
based on amazing std.experimental.allocator by Andrei
http://wiki.dlang.org/Review/std.experimental.allocator

I will be happy to receive any advices about algorithms, use cases and API.

Best Regards,
Ilya


Will it be language feature fix, or is it an independent container?
If the later I already have a simple dumb one which I can share (not on 
this machine). I'll be happy to use what you create. Same goes for list 
and friends ones.

associative arrays with manual memory management

[Ilya Yaroshenko via Digitalmars-d]

Hi All!

I am going to implement associative arrays with manual memory 
management based on amazing std.experimental.allocator by Andrei 
http://wiki.dlang.org/Review/std.experimental.allocator


I will be happy to receive any advices about algorithms, use 
cases and API.


Best Regards,
Ilya

[Issue 2105] Manual Memory Management for Associative Arrays

[via Digitalmars-d-bugs]

https://issues.dlang.org/show_bug.cgi?id=2105

Andrei Alexandrescu and...@erdani.com changed:

   What|Removed |Added

Version|2.014   |D2

--

Re: Module for manual memory management

[Jacob Carlborg via Digitalmars-d]

On 2015-02-04 21:55, Walter Bright wrote:


No need to reinvent this:

   https://github.com/D-Programming-Language/phobos/blob/master/std/file.d
   line 194


Just for the record, you can get a link to the exact line by clicking on 
the line number in the left margin and the copying the link from the 
address bar.


--
/Jacob Carlborg

Re: Module for manual memory management

[Foo via Digitalmars-d]

On Saturday, 7 February 2015 at 11:29:51 UTC, Jacob Carlborg 
wrote:

On 2015-02-04 21:55, Walter Bright wrote:


No need to reinvent this:

  
https://github.com/D-Programming-Language/phobos/blob/master/std/file.d

  line 194


Just for the record, you can get a link to the exact line by 
clicking on the line number in the left margin and the copying 
the link from the address bar.


The same I've said a page previously. Funny. :)

Re: Module for manual memory management

[Foo via Digitalmars-d]

On Thursday, 5 February 2015 at 07:53:34 UTC, Andrei Alexandrescu 
wrote:

On 2/4/15 11:48 PM, Foo wrote:
I would be glad if you or Walter could review the other parts 
as well.
I'm very sure that your review will be very helpful and I'm 
convinced
that the modules Array or Smart Pointer could be useful for 
phobos.


Your code is a ways from being in reviewable form. It would be 
great if a member of the community could act as a mentor. -- 
Andrei


Oh, I didn't thought that it is so bad. :D It works splendidly so 
far for us. Sorry, that I'm wasting your time. :)

Re: Module for manual memory management

[Foo via Digitalmars-d]

For what it's worth, today I finished the current work. Now we 
will start working with it. If someone has critique, improvement 
suggestions or want to take some ideas, he is free to do so.
To repeat myself: we rewrote some functionality which already 
existed in D, but were improvable. For example the existing 
emplace method is quite long, hard to understand / read and is 
not marked with @nogc. Since we want to avoid the GC wherever 
possible we had to rewrite it. I hope it's useful for someone 
else and that some of you guys take some ideas from it.


https://github.com/Dgame/m3

Re: Module for manual memory management

[Andrei Alexandrescu via Digitalmars-d]

On 2/4/15 9:13 AM, Foo wrote:

For what it's worth, today I finished the current work. Now we will
start working with it. If someone has critique, improvement suggestions
or want to take some ideas, he is free to do so.
To repeat myself: we rewrote some functionality which already existed in
D, but were improvable. For example the existing emplace method is quite
long, hard to understand / read and is not marked with @nogc. Since we
want to avoid the GC wherever possible we had to rewrite it. I hope it's
useful for someone else and that some of you guys take some ideas from it.

https://github.com/Dgame/m3


Opened File.d at random:

@trusted
@nogc
char[] read(const string filename) nothrow {
import std.c.stdio : FILE, SEEK_END, SEEK_SET, fopen, fclose, 
fseek, ftell, fread;


FILE* f = fopen(filename.ptr, READ_BINARY);
scope(exit) fclose(f);

fseek(f, 0, SEEK_END);
immutable size_t fsize = ftell(f);
fseek(f, 0, SEEK_SET);

char[] str = m3.m3.make!(char[])(fsize);
fread(str.ptr, fsize, 1, f);

return str;
}

Then stopped right there. This is nowhere near production quality - 
there is no error checking whatsoever, does more operations than 
necessary, won't work on special files etc. etc. etc.


I applaud the intention but there is a lot more work to be done on this 
before it's in reviewable form.



Andrei

Re: Module for manual memory management

[Foo via Digitalmars-d]

On Wednesday, 4 February 2015 at 20:55:59 UTC, Walter Bright
wrote:

On 2/4/2015 12:42 PM, Foo wrote:
On Wednesday, 4 February 2015 at 20:15:37 UTC, Andrei 
Alexandrescu wrote:

@trusted
@nogc
char[] read(const string filename) nothrow {
Yes that is correct, currently there is no error checking, 
maybe it get that
later. But what do you mean with it use more operations than 
necessary? I
can't see it. But both points are helpful critique. Thanks a 
lot! :)


No need to reinvent this:

  
https://github.com/D-Programming-Language/phobos/blob/master/std/file.d

  line 194

Just use it and change the allocator bit.


It would be much easier for me, if some of you would add such a
allocator to it. :P

Re: Module for manual memory management

[Walter Bright via Digitalmars-d]

On 2/4/2015 12:42 PM, Foo wrote:

On Wednesday, 4 February 2015 at 20:15:37 UTC, Andrei Alexandrescu wrote:

@trusted
@nogc
char[] read(const string filename) nothrow {

Yes that is correct, currently there is no error checking, maybe it get that
later. But what do you mean with it use more operations than necessary? I
can't see it. But both points are helpful critique. Thanks a lot! :)


No need to reinvent this:

  https://github.com/D-Programming-Language/phobos/blob/master/std/file.d
  line 194

Just use it and change the allocator bit.

Re: Module for manual memory management

[Foo via Digitalmars-d]

On Wednesday, 4 February 2015 at 20:15:37 UTC, Andrei 
Alexandrescu wrote:

On 2/4/15 9:13 AM, Foo wrote:
For what it's worth, today I finished the current work. Now we 
will
start working with it. If someone has critique, improvement 
suggestions

or want to take some ideas, he is free to do so.
To repeat myself: we rewrote some functionality which already 
existed in
D, but were improvable. For example the existing emplace 
method is quite
long, hard to understand / read and is not marked with @nogc. 
Since we
want to avoid the GC wherever possible we had to rewrite it. I 
hope it's
useful for someone else and that some of you guys take some 
ideas from it.


https://github.com/Dgame/m3


Opened File.d at random:

@trusted
@nogc
char[] read(const string filename) nothrow {
import std.c.stdio : FILE, SEEK_END, SEEK_SET, fopen, 
fclose, fseek, ftell, fread;


FILE* f = fopen(filename.ptr, READ_BINARY);
scope(exit) fclose(f);

fseek(f, 0, SEEK_END);
immutable size_t fsize = ftell(f);
fseek(f, 0, SEEK_SET);

char[] str = m3.m3.make!(char[])(fsize);
fread(str.ptr, fsize, 1, f);

return str;
}

Then stopped right there. This is nowhere near production 
quality - there is no error checking whatsoever, does more 
operations than necessary, won't work on special files etc. 
etc. etc.


I applaud the intention but there is a lot more work to be done 
on this before it's in reviewable form.



Andrei


Yes that is correct, currently there is no error checking, maybe 
it get that later. But what do you mean with it use more 
operations than necessary? I can't see it. But both points are 
helpful critique. Thanks a lot! :)


As I said above, it's currently only for our work. But I 
presented it here, so that some guys can get some inspiration or 
can reuse our code. :)
Maybe, but really only maybe, I will get some work done, so that 
it's is ready for phobos. But since such a review can take years, 
I see no use to do that work. But if someone else has the will to 
do that, reuse our code!

Re: Module for manual memory management

[Foo via Digitalmars-d]

On Wednesday, 4 February 2015 at 20:55:59 UTC, Walter Bright 
wrote:

On 2/4/2015 12:42 PM, Foo wrote:
On Wednesday, 4 February 2015 at 20:15:37 UTC, Andrei 
Alexandrescu wrote:

@trusted
@nogc
char[] read(const string filename) nothrow {
Yes that is correct, currently there is no error checking, 
maybe it get that
later. But what do you mean with it use more operations than 
necessary? I
can't see it. But both points are helpful critique. Thanks a 
lot! :)


No need to reinvent this:

  
https://github.com/D-Programming-Language/phobos/blob/master/std/file.d

  line 194

Just use it and change the allocator bit.


Yes, I've also looked at that. But I think my code is more 
readable and easier to understand. But I will take a second look. 
;)


BTW: You can redirect to the line by adding #194: 
https://github.com/D-Programming-Language/phobos/blob/master/std/file.d#L194 
just click on the line number on the left.

Re: Module for manual memory management

[Andrei Alexandrescu via Digitalmars-d]

On 2/4/15 11:48 PM, Foo wrote:

I would be glad if you or Walter could review the other parts as well.
I'm very sure that your review will be very helpful and I'm convinced
that the modules Array or Smart Pointer could be useful for phobos.


Your code is a ways from being in reviewable form. It would be great if 
a member of the community could act as a mentor. -- Andrei

Re: Module for manual memory management

[Foo via Digitalmars-d]

On Wednesday, 4 February 2015 at 20:15:37 UTC, Andrei 
Alexandrescu wrote:

On 2/4/15 9:13 AM, Foo wrote:
For what it's worth, today I finished the current work. Now we 
will
start working with it. If someone has critique, improvement 
suggestions

or want to take some ideas, he is free to do so.
To repeat myself: we rewrote some functionality which already 
existed in
D, but were improvable. For example the existing emplace 
method is quite
long, hard to understand / read and is not marked with @nogc. 
Since we
want to avoid the GC wherever possible we had to rewrite it. I 
hope it's
useful for someone else and that some of you guys take some 
ideas from it.


https://github.com/Dgame/m3


Opened File.d at random:

@trusted
@nogc
char[] read(const string filename) nothrow {
import std.c.stdio : FILE, SEEK_END, SEEK_SET, fopen, 
fclose, fseek, ftell, fread;


FILE* f = fopen(filename.ptr, READ_BINARY);
scope(exit) fclose(f);

fseek(f, 0, SEEK_END);
immutable size_t fsize = ftell(f);
fseek(f, 0, SEEK_SET);

char[] str = m3.m3.make!(char[])(fsize);
fread(str.ptr, fsize, 1, f);

return str;
}

Then stopped right there. This is nowhere near production 
quality - there is no error checking whatsoever, does more 
operations than necessary, won't work on special files etc. 
etc. etc.


I applaud the intention but there is a lot more work to be done 
on this before it's in reviewable form.



Andrei


I would be glad if you or Walter could review the other parts as 
well. I'm very sure that your review will be very helpful and I'm 
convinced that the modules Array or Smart Pointer could be useful 
for phobos.

Re: Module for manual memory management

[Foo via Digitalmars-d]

On Saturday, 31 January 2015 at 01:07:21 UTC, Andrei Alexandrescu 
wrote:

On 1/30/15 3:49 PM, Foo wrote:

Is there interest in such a thing?
I'm currently working on something for my own use and I'm 
curious if

anyone else would be interested in something like that.
I'm aware of Unique, RefCounted, Scoped, emplace and so on, 
but I'm not
a big fan of some implementations (in my opinion not much of 
it follows
the KISS principle). Also, as far as I remember, RefCounted 
and Unique

are somewhat outdated.
As said I'm just curious and I'm aware that my code may be 
somewhat

un-phobos like.

My current stand is this: http://dpaste.dzfl.pl/f1423322b7d5
Just to make it clear: m3 = manual memory management = mmm = m3

and here is some test code: http://dpaste.dzfl.pl/026013fbe19f

I repeat: I know that I reimplement some logic like emplace 
and that my

version is shorter but maybe not quite correct/incomplete.


So why come with your own self-admitted incomplete/incorrect 
alternatives instead of adding/improving the standard ones? If 
something's there that shouldn't, remove it. I don't get this 
I didn't get it so I'll redo it incompletely.


And I'm also aware of std.allocator and so on, but that is 
something

different IMO.


Agreed.


Andrei


And I did not literally said 'incorrect'. I said _maybe_ 
incomplete. In my tests it works properly. What I meant with 'not 
quite correct' was that it isn't well tested _yet_. But that can 
be done later.

Re: Module for manual memory management

[Foo via Digitalmars-d]

On Saturday, 31 January 2015 at 01:07:21 UTC, Andrei Alexandrescu 
wrote:

On 1/30/15 3:49 PM, Foo wrote:

Is there interest in such a thing?
I'm currently working on something for my own use and I'm 
curious if

anyone else would be interested in something like that.
I'm aware of Unique, RefCounted, Scoped, emplace and so on, 
but I'm not
a big fan of some implementations (in my opinion not much of 
it follows
the KISS principle). Also, as far as I remember, RefCounted 
and Unique

are somewhat outdated.
As said I'm just curious and I'm aware that my code may be 
somewhat

un-phobos like.

My current stand is this: http://dpaste.dzfl.pl/f1423322b7d5
Just to make it clear: m3 = manual memory management = mmm = m3

and here is some test code: http://dpaste.dzfl.pl/026013fbe19f

I repeat: I know that I reimplement some logic like emplace 
and that my

version is shorter but maybe not quite correct/incomplete.


So why come with your own self-admitted incomplete/incorrect 
alternatives instead of adding/improving the standard ones? If 
something's there that shouldn't, remove it. I don't get this 
I didn't get it so I'll redo it incompletely.

Too much and too long code. ;)
Sometimes it is better to see a new version from scratch to 
rethink old ones. ;) I'm not for removing them, I'm for 
rebuilding them. And a new version can bring new wind into the 
sails. That's why I'm presented my code here: for discussion.
But that is not that important, my main target was the 
implementation of Unique and Shared. You can adapt them with 
minimal effort. But I'm quite sure that my 'make' function is not 
yet present in phobos.
So, what do you think about discussing if and how my make, Unique 
and Shared could improve phobos.


And I'm also aware of std.allocator and so on, but that is 
something

different IMO.


Agreed.


Andrei

Module for manual memory management

[Foo via Digitalmars-d]

Is there interest in such a thing?
I'm currently working on something for my own use and I'm curious 
if anyone else would be interested in something like that.
I'm aware of Unique, RefCounted, Scoped, emplace and so on, but 
I'm not a big fan of some implementations (in my opinion not much 
of it follows the KISS principle). Also, as far as I remember, 
RefCounted and Unique are somewhat outdated.
As said I'm just curious and I'm aware that my code may be 
somewhat un-phobos like.


My current stand is this: http://dpaste.dzfl.pl/f1423322b7d5
Just to make it clear: m3 = manual memory management = mmm = m3

and here is some test code: http://dpaste.dzfl.pl/026013fbe19f

I repeat: I know that I reimplement some logic like emplace and 
that my version is shorter but maybe not quite correct/incomplete.
And I'm also aware of std.allocator and so on, but that is 
something different IMO.

Re: Module for manual memory management

[Andrei Alexandrescu via Digitalmars-d]

On 1/30/15 3:49 PM, Foo wrote:

Is there interest in such a thing?
I'm currently working on something for my own use and I'm curious if
anyone else would be interested in something like that.
I'm aware of Unique, RefCounted, Scoped, emplace and so on, but I'm not
a big fan of some implementations (in my opinion not much of it follows
the KISS principle). Also, as far as I remember, RefCounted and Unique
are somewhat outdated.
As said I'm just curious and I'm aware that my code may be somewhat
un-phobos like.

My current stand is this: http://dpaste.dzfl.pl/f1423322b7d5
Just to make it clear: m3 = manual memory management = mmm = m3

and here is some test code: http://dpaste.dzfl.pl/026013fbe19f

I repeat: I know that I reimplement some logic like emplace and that my
version is shorter but maybe not quite correct/incomplete.


So why come with your own self-admitted incomplete/incorrect 
alternatives instead of adding/improving the standard ones? If 
something's there that shouldn't, remove it. I don't get this I didn't 
get it so I'll redo it incompletely.



And I'm also aware of std.allocator and so on, but that is something
different IMO.


Agreed.


Andrei

Re: D Beginner Trying Manual Memory Management

[Laeeth Isharc via Digitalmars-d-learn]

I don't think you've read h5py source in enough detail :)


You're right - I haven't done more than browsed it.


It's based HEAVILY on duck typing.


There is a question here about what to do in D.  On the one hand, 
the flexibility of being able to open a foreign HDF5 file where 
you don't know beforehand the dataset type is very nice.  On the 
other, the adaptations needed to handle this flexibly get in the 
way when you are dealing with your own data that has a set format 
and where recompilation is acceptable if it changes.  Looking at 
the 'ease' of processing JSON, even using vibed, I think that one 
will need to implement both eventually, but perhaps starting with 
static typing.



In addition, it has way MORE classes than the C++ hierarchy 
does. E.g., the high-level File object actually has these 
parents: File : Group, Group : HLObject, 
MutableMappingWithLock, HLObject : CommonStateObject and 
internally the File also keeps a reference to file id which is 
an instance of FileID which inherits from GroupID which 
inherits from ObjectID, do I need to continue?


Okay - I guess there is a distinction between the interface to 
the outside world (where I think the h5py etc way is superior for 
most uses) and the implementation.  Is not the reason h5py has 
lots of classes primarily because that is how you write good code 
in python, whereas in many cases this is not true in D (not that 
you should ban classes, but often structs + free floating 
functions are more suitable).


PyTables, on the contrary is quite badly written (although it 
works quite well and there are brilliant folks on the dev team 
like francesc alted) and looks like a dump of C code 
interweaved with hackish Python code.


Interesting.  What do you think is low quality about the design?

In h5py you can do things like file[/dataset].write(...) -- 
this just wouldn't work as is in a strictly typed language 
since the indexing operator generally returns you something of 
a Location type (or an interface, rather) which can be a 
group/datatype/dataset which is only known at runtime.


Well, if you don't mind recompiling your code when the data set 
type changes (or you encounter a new data set) then you can do 
that (which is what I posted a link to earlier).


It depends on your use case.  It's hard to think of an 
application more dynamic than web sites, and yet people seem 
happy enough with vibed's use of compiled diet templates as the 
primary implementation.  They would like the option of dynamic 
ones too, and I think this would be useful in this domain too, 
since one does look at foreign data on occasion.  One could of 
course use the quick compilation of D to regenerate parts of the 
code when this happens.  Whether or not this is acceptable 
depends on your use case - for some it might be okay, but 
obviously it is no good if you are writing a generic H5 
browser/charting tool.


So I think if you don't allow static dataset typing it means the 
flexibility  of dynamic typing gets in the way for some uses 
(which might be most of them), but you need to add dynamic typing 
too.


Shall we move this to a different thread and/or email, as I am 
afraid I have hijacked the poor original poster's request.


On the refcounting question, I confess that I do not fully 
understand your concern, which may well reflect a lack of deep 
experience with D on my part.  Adam Ruppe suggests that it's 
generally okay to rely on a struct destructor to call C cleanup 
code.  I can appreciate this may not be true with h5 and, if you 
can spare the time, I would love to understand more precisely why 
not.


Out of all of them, only the dataset supports the write method 
but you don't know it's going to be a dataset. See the problem?


In this case I didn't quite follow.  Where does this fall down ?

void h5write(T)(Dataset x, T data)


I have your email somewhere and will drop you a line.  Or you can 
email me laeeth at laeeth.com.  And let's create a new thread.




Laeeth.

Re: D Beginner Trying Manual Memory Management

[Laeeth Isharc via Digitalmars-d-learn]

In the hierarchy example above (c++ hdf hierarchy link), by 
using UFCS to implement the shared methods (which are achieved 
by multiple inheritance in the c++ counterpart) did you mean 
something like this?


// id.d
struct ID { int id; ... }

// location.d
struct Location { ID _id; alias _id this; ... }

// file.d
public import commonfg; // ugh
struct File { Location _location; alias _location this; ... }

// group.d
public import commonfg;
struct File { Location _location; alias _location this; ... }

// commonfg.d { ... }
enum isContainer(T) = is(T: File) || is(T : Group);
auto method1(T)(T obj, args) if (isContainer!T) { ... }
auto method2(T)(T obj, args) if (isContainer!T) { ... }

I guess two of my gripes with UFCS is (a) you really have to




// another hdf-specific thing here but a good example in 
general is that some functions return you an id for an object 
which is one of the location subtypes (e.g. it could be a File 
or could be a Group depending on run-time conditions), so it 
kind of feels natural to use polymorphism and classes for that, 
but what would you do with the struct approach? The only thing 
that comes to mind is Variant, but it's quite meh to use in 
practice.


Void unlink(File f){}
Void unlink(Group g){}

For simple cases maybe one can keep it simple, and despite the 
Byzantine interface what one is trying to do when using HDF5 is 
not intrinsically so complex.

Re: D Beginner Trying Manual Memory Management

[Laeeth Isharc via Digitalmars-d-learn]

struct File { Location _location; alias _location this; ... }

// group.d
public import commonfg;
struct File { Location _location; alias _location this; ... }

// commonfg.d { ... }
enum isContainer(T) = is(T: File) || is(T : Group);
auto method1(T)(T obj, args) if (isContainer!T) { ... }
auto method2(T)(T obj, args) if (isContainer!T) { ... }

I guess two of my gripes with UFCS is (a) you really have to




// another hdf-specific thing here but a good example in 
general is that some functions return you an id for an object 
which is one of the location subtypes (e.g. it could be a 
File or could be a Group depending on run-time conditions), 
so it kind of feels natural to use polymorphism and classes 
for that, but what would you do with the struct approach? The 
only thing that comes to mind is Variant, but it's quite meh 
to use in practice.


Void unlink(File f){}
Void unlink(Group g){}

For simple cases maybe one can keep it simple, and despite the 
Byzantine interface what one is trying to do when using HDF5 
is not intrinsically so complex.

So your solution is copying and pasting the code?

But now repeat that for 200 other functions and a dozen more 
types that can be polymorphic in weirdest ways possible...


If you are simply have a few lines calling the API and the 
validation is different enough for file and group (I haven't 
written unlink yet) then why not (and move proper shared code out 
into helper functions).  The alternative is a long method with 
lots of conditions, which may be the best in some cases but may 
be harder to follow.


I do like the h5py and pytables approaches.  One doesn't need to 
bother too much with the implementation when using their library. 
 However, what I am doing is quite simple from a data perspective 
- a decent amount of it, but it is not an interesting problem 
from a theoretical perspective - just execution.  Now if you are 
higher octane as a user you may be able to see what I cannot.  
But on the other hand, the Pareto principle applies, and in my 
view a library should make it simple to do simple things.  One 
can't get there if the primary interface is a direct mapping of 
the HDF5 hierarchy, and I also think that is unnecessary with D.


But I very much appreciate your work as the final result is 
better for everyone that way, and you are evidently a much longer 
running user of D than me.  I never used C++ as it just seemed 
too ugly! and I suspect the difference in backgrounds is shaping 
perspectives.


What do you think the trickiest parts are with HDF5?  (You 
mention weird polymorphism).




Laeeth

Re: D Beginner Trying Manual Memory Management

[aldanor via Digitalmars-d-learn]

On Wednesday, 14 January 2015 at 14:54:09 UTC, Laeeth Isharc 
wrote:



In the hierarchy example above (c++ hdf hierarchy link), by 
using UFCS to implement the shared methods (which are achieved 
by multiple inheritance in the c++ counterpart) did you mean 
something like this?


// id.d
struct ID { int id; ... }

// location.d
struct Location { ID _id; alias _id this; ... }

// file.d
public import commonfg; // ugh
struct File { Location _location; alias _location this; ... }

// group.d
public import commonfg;
struct File { Location _location; alias _location this; ... }

// commonfg.d { ... }
enum isContainer(T) = is(T: File) || is(T : Group);
auto method1(T)(T obj, args) if (isContainer!T) { ... }
auto method2(T)(T obj, args) if (isContainer!T) { ... }

I guess two of my gripes with UFCS is (a) you really have to




// another hdf-specific thing here but a good example in 
general is that some functions return you an id for an object 
which is one of the location subtypes (e.g. it could be a File 
or could be a Group depending on run-time conditions), so it 
kind of feels natural to use polymorphism and classes for 
that, but what would you do with the struct approach? The only 
thing that comes to mind is Variant, but it's quite meh to use 
in practice.


Void unlink(File f){}
Void unlink(Group g){}

For simple cases maybe one can keep it simple, and despite the 
Byzantine interface what one is trying to do when using HDF5 is 
not intrinsically so complex.

So your solution is copying and pasting the code?

But now repeat that for 200 other functions and a dozen more 
types that can be polymorphic in weirdest ways possible...

Re: D Beginner Trying Manual Memory Management

[aldanor via Digitalmars-d-learn]

On Wednesday, 14 January 2015 at 16:27:17 UTC, Laeeth Isharc 
wrote:



struct File { Location _location; alias _location this; ... }

// group.d
public import commonfg;
struct File { Location _location; alias _location this; ... }

// commonfg.d { ... }
enum isContainer(T) = is(T: File) || is(T : Group);
auto method1(T)(T obj, args) if (isContainer!T) { ... }
auto method2(T)(T obj, args) if (isContainer!T) { ... }

I guess two of my gripes with UFCS is (a) you really have to




// another hdf-specific thing here but a good example in 
general is that some functions return you an id for an 
object which is one of the location subtypes (e.g. it could 
be a File or could be a Group depending on run-time 
conditions), so it kind of feels natural to use polymorphism 
and classes for that, but what would you do with the struct 
approach? The only thing that comes to mind is Variant, but 
it's quite meh to use in practice.


Void unlink(File f){}
Void unlink(Group g){}

For simple cases maybe one can keep it simple, and despite 
the Byzantine interface what one is trying to do when using 
HDF5 is not intrinsically so complex.

So your solution is copying and pasting the code?

But now repeat that for 200 other functions and a dozen more 
types that can be polymorphic in weirdest ways possible...


If you are simply have a few lines calling the API and the 
validation is different enough for file and group (I haven't 
written unlink yet) then why not (and move proper shared code 
out into helper functions).  The alternative is a long method 
with lots of conditions, which may be the best in some cases 
but may be harder to follow.


I do like the h5py and pytables approaches.  One doesn't need 
to bother too much with the implementation when using their 
library.
 However, what I am doing is quite simple from a data 
perspective - a decent amount of it, but it is not an 
interesting problem from a theoretical perspective - just 
execution.  Now if you are higher octane as a user you may be 
able to see what I cannot.  But on the other hand, the Pareto 
principle applies, and in my view a library should make it 
simple to do simple things.  One can't get there if the primary 
interface is a direct mapping of the HDF5 hierarchy, and I also 
think that is unnecessary with D.


But I very much appreciate your work as the final result is 
better for everyone that way, and you are evidently a much 
longer running user of D than me.  I never used C++ as it just 
seemed too ugly! and I suspect the difference in backgrounds is 
shaping perspectives.


What do you think the trickiest parts are with HDF5?  (You 
mention weird polymorphism).




Laeeth
I don't think you've read h5py source in enough detail :) It's 
based HEAVILY on duck typing. In addition, it has way MORE 
classes than the C++ hierarchy does. E.g., the high-level File 
object actually has these parents: File : Group, Group : 
HLObject, MutableMappingWithLock, HLObject : CommonStateObject 
and internally the File also keeps a reference to file id which 
is an instance of FileID which inherits from GroupID which 
inherits from ObjectID, do I need to continue? :) PyTables, on 
the contrary is quite badly written (although it works quite well 
and there are brilliant folks on the dev team like francesc 
alted) and looks like a dump of C code interweaved with hackish 
Python code.


In h5py you can do things like file[/dataset].write(...) -- 
this just wouldn't work as is in a strictly typed language since 
the indexing operator generally returns you something of a 
Location type (or an interface, rather) which can be a 
group/datatype/dataset which is only known at runtime. Out of all 
of them, only the dataset supports the write method but you don't 
know it's going to be a dataset. See the problem? I don't want 
the user code to deal with any of the HDF5 C API and/or have a 
bunch of if conditions or explicit casts which is outright ugly. 
Ideally, it would work kind of like H5PY, abstracting the user 
away from refcounting, error code checking after each operation, 
object type checking and all that stuff.

Re: D Beginner Trying Manual Memory Management

[Laeeth Isharc via Digitalmars-d-learn]

 I see, thanks! :) I've started liking structs more and more 
 recently as well and been pondering on how to convert a 
 class-based code that looks like this (only the base class 
 has any data):
 it's hard to tell by brief description. but having multiple 
 inheritance
 immediately rings an alarm ring for me. something is 
 very-very-very

 wrong if you need to have a winged whale. ;-)
A real-world example: 
http://www.hdfgroup.org/HDF5/doc/cpplus_RM/hierarchy.html


H5::File is both an H5::Location and H5::CommonFG (but not an 
H5::Object)
H5::Group is both an H5::Object (subclass of H5::Location) and 
H5::CommonFG

H5::Dataset is an H5::Object
i see something named CommonFG here, which seems to good 
thing to

move out of hierarchy altogether.

bwah, i don't even sure that given hierarchy is good for D. C++ 
has no

UFCS, and it's incredibly hard to check if some entity has some
methods/properties in C++, so they have no other choice than to 
work
around that limitations. it may be worthful to redesign the 
whole thing
for D, exploiting D shiny UFCS and metaprogramming features. 
and,

maybe, moving some things to interfaces too.


I just finished reading aldanor's blog, so I know he is slightly 
allergic to naked functions and prefers classes ;)


With Ketmar, I very much agree (predominantly as a user of HDF5 
and less so as an inexperienced D programmr writing a wrapper for 
it).  It's a pain to figure out just how to do simple things 
until you know the H5 library.  You have to create an object for 
file permissions before you even get started, then similarly for 
the data series (datasets) within, another for the dimensions of 
the array, etc etc  - that doesn't fit with the intrinsic nature 
of the domain.


There is a more general question of bindings/wrappers - preserve 
the original structure and naming so existing code can be ported, 
or write a wrapper that makes it easy for the user to accomplish 
his objectives.  It seems like for the bindings preserving the 
library structure is fine, but for the wrapper one might as well 
make things easy.


Eg here https://gist.github.com/Laeeth/9637233db41a11a9d1f4
line 146.  (sorry for duplication and messiness of code, which I 
don't claim to be perfectly written - I wanted to try something 
quickly and have not yet tidied up).


So rather than navigate the Byzantine hierarchy, one can just do 
something like this (which will take a struct of PriceBar - 
date,open,high,low,close - and put it in your desired dataset and 
file, appending or overwriting as you prefer).


dumpDataSpaceVector!PriceBar(file,ticker,array(priceBars[ticker]),DumpMode.truncate);

which is closer to h5py in Python.  (It uses reflection to figure 
out the contents of a non-nested struct, but won't yet cope with 
arrays and nested structs inside).  And of course a full wrapper 
might be a bit more complicated, but I truly think one can do 
better than mapping the HDF5 hierarchy one for one.



Laeeth.

Re: D Beginner Trying Manual Memory Management

[ketmar via Digitalmars-d-learn]

On Tue, 13 Jan 2015 17:08:37 +
Laeeth Isharc via Digitalmars-d-learn
digitalmars-d-learn@puremagic.com wrote:

 I just finished reading aldanor's blog, so I know he is slightly 
 allergic to naked functions and prefers classes ;)
that's due to absense of modules in C/C++. and namespaces aren't of big
help here too. and, of course, due to missing UFCS, that prevents nice
`obj.func()` for free functions. ;-)


signature.asc
Description: PGP signature

Re: D Beginner Trying Manual Memory Management

[ketmar via Digitalmars-d-learn]

On Tue, 13 Jan 2015 18:35:15 +
aldanor via Digitalmars-d-learn digitalmars-d-learn@puremagic.com
wrote:

 I guess two of my gripes with UFCS is (a) you really have to use 
 public imports in the modules where the target types are defined 
 so you bring all the symbols in whether you want it or not (b) 
 you lose access to private members because it's not the same 
 module anymore (correct me if I'm wrong?). Plus, you need to 
 decorate every single free function with a template constraint.
you can make a package and set protection to `package` instead of
`private`, so your function will still be able to access internal
fields, but package users will not. this feature is often missed
by the people who are used to `public`/`protected`/`private` triad.


signature.asc
Description: PGP signature

Re: D Beginner Trying Manual Memory Management

[aldanor via Digitalmars-d-learn]

On Tuesday, 13 January 2015 at 17:08:38 UTC, Laeeth Isharc wrote:
 I see, thanks! :) I've started liking structs more and 
 more recently as well and been pondering on how to convert 
 a class-based code that looks like this (only the base 
 class has any data):
 it's hard to tell by brief description. but having multiple 
 inheritance
 immediately rings an alarm ring for me. something is 
 very-very-very

 wrong if you need to have a winged whale. ;-)
A real-world example: 
http://www.hdfgroup.org/HDF5/doc/cpplus_RM/hierarchy.html


H5::File is both an H5::Location and H5::CommonFG (but not an 
H5::Object)
H5::Group is both an H5::Object (subclass of H5::Location) 
and H5::CommonFG

H5::Dataset is an H5::Object
i see something named CommonFG here, which seems to good 
thing to

move out of hierarchy altogether.

bwah, i don't even sure that given hierarchy is good for D. 
C++ has no

UFCS, and it's incredibly hard to check if some entity has some
methods/properties in C++, so they have no other choice than 
to work
around that limitations. it may be worthful to redesign the 
whole thing
for D, exploiting D shiny UFCS and metaprogramming features. 
and,

maybe, moving some things to interfaces too.


I just finished reading aldanor's blog, so I know he is 
slightly allergic to naked functions and prefers classes ;)


With Ketmar, I very much agree (predominantly as a user of HDF5 
and less so as an inexperienced D programmr writing a wrapper 
for it).  It's a pain to figure out just how to do simple 
things until you know the H5 library.  You have to create an 
object for file permissions before you even get started, then 
similarly for the data series (datasets) within, another for 
the dimensions of the array, etc etc  - that doesn't fit with 
the intrinsic nature of the domain.


There is a more general question of bindings/wrappers - 
preserve the original structure and naming so existing code can 
be ported, or write a wrapper that makes it easy for the user 
to accomplish his objectives.  It seems like for the bindings 
preserving the library structure is fine, but for the wrapper 
one might as well make things easy.


Eg here https://gist.github.com/Laeeth/9637233db41a11a9d1f4
line 146.  (sorry for duplication and messiness of code, which 
I don't claim to be perfectly written - I wanted to try 
something quickly and have not yet tidied up).


So rather than navigate the Byzantine hierarchy, one can just 
do something like this (which will take a struct of PriceBar - 
date,open,high,low,close - and put it in your desired dataset 
and file, appending or overwriting as you prefer).


dumpDataSpaceVector!PriceBar(file,ticker,array(priceBars[ticker]),DumpMode.truncate);

which is closer to h5py in Python.  (It uses reflection to 
figure out the contents of a non-nested struct, but won't yet 
cope with arrays and nested structs inside).  And of course a 
full wrapper might be a bit more complicated, but I truly think 
one can do better than mapping the HDF5 hierarchy one for one.



Laeeth.
In the hierarchy example above (c++ hdf hierarchy link), by using 
UFCS to implement the shared methods (which are achieved by 
multiple inheritance in the c++ counterpart) did you mean 
something like this?


// id.d
struct ID { int id; ... }

// location.d
struct Location { ID _id; alias _id this; ... }

// file.d
public import commonfg; // ugh
struct File { Location _location; alias _location this; ... }

// group.d
public import commonfg;
struct File { Location _location; alias _location this; ... }

// commonfg.d { ... }
enum isContainer(T) = is(T: File) || is(T : Group);
auto method1(T)(T obj, args) if (isContainer!T) { ... }
auto method2(T)(T obj, args) if (isContainer!T) { ... }

I guess two of my gripes with UFCS is (a) you really have to use 
public imports in the modules where the target types are defined 
so you bring all the symbols in whether you want it or not (b) 
you lose access to private members because it's not the same 
module anymore (correct me if I'm wrong?). Plus, you need to 
decorate every single free function with a template constraint.


// another hdf-specific thing here but a good example in general 
is that some functions return you an id for an object which is 
one of the location subtypes (e.g. it could be a File or could be 
a Group depending on run-time conditions), so it kind of feels 
natural to use polymorphism and classes for that, but what would 
you do with the struct approach? The only thing that comes to 
mind is Variant, but it's quite meh to use in practice.

Re: D Beginner Trying Manual Memory Management

[ketmar via Digitalmars-d-learn]

On Mon, 12 Jan 2015 22:07:13 +
aldanor via Digitalmars-d-learn digitalmars-d-learn@puremagic.com
wrote:

 I see, thanks! :) I've started liking structs more and more 
 recently as well and been pondering on how to convert a 
 class-based code that looks like this (only the base class has 
 any data):
it's hard to tell by brief description. but having multiple inheritance
immediately rings an alarm ring for me. something is very-very-very
wrong if you need to have a winged whale. ;-)


signature.asc
Description: PGP signature

Re: D Beginner Trying Manual Memory Management

[ketmar via Digitalmars-d-learn]

On Mon, 12 Jan 2015 23:06:16 +
jmh530 via Digitalmars-d-learn digitalmars-d-learn@puremagic.com
wrote:

 I had seen some stuff on alias thing, but I hadn't bothered to
 try to understand it until now. If I'm understanding the first
 example a href=http://dlang.org/class.html#AliasThis;here/a,
 alias this let's you refer to x in s by writing either s.x (as
 normally) or just s. That didn't seem that interesting, but then
 I found a href=http://3d.benjamin-thaut.de/?p=90;example/a
 where they alias this'ed a struct method. That's pretty
 interesting.
there is nice page by p0nce and it shows nice and simple `alias this`
trick usage:
http://p0nce.github.io/d-idioms/#Extending-a-struct-with-alias-this

and i have stream.d module in my IV package
( http://repo.or.cz/w/iv.d.git/tree ), which works with i/o streams
by testing if passed struct/class has necessary methods (`isReadable!`,
`isWriteable!`, `isSeekable!`, etc.


signature.asc
Description: PGP signature

Re: D Beginner Trying Manual Memory Management

[ketmar via Digitalmars-d-learn]

On Mon, 12 Jan 2015 22:07:13 +
aldanor via Digitalmars-d-learn digitalmars-d-learn@puremagic.com
wrote:

 I see, thanks! :) I've started liking structs more and more 
 recently as well and been pondering on how to convert a 
 class-based code that looks like this (only the base class has 
 any data):
p.s. can't you convert most of that to free functions? thanks to UFCS
you'll be able to use them with `obj.func` notation. and by either
defining `package` protection for class fields, or simply writing that
functions in the same module they will have access to internal object
stuff.

and if you can return `obj` from each function, you can go with
templates and chaining. ;-)


signature.asc
Description: PGP signature

Re: D Beginner Trying Manual Memory Management

[aldanor via Digitalmars-d-learn]

On Tuesday, 13 January 2015 at 08:33:57 UTC, ketmar via 
Digitalmars-d-learn wrote:

On Mon, 12 Jan 2015 22:07:13 +
aldanor via Digitalmars-d-learn 
digitalmars-d-learn@puremagic.com

wrote:

I see, thanks! :) I've started liking structs more and more 
recently as well and been pondering on how to convert a 
class-based code that looks like this (only the base class has 
any data):
it's hard to tell by brief description. but having multiple 
inheritance
immediately rings an alarm ring for me. something is 
very-very-very

wrong if you need to have a winged whale. ;-)
A real-world example: 
http://www.hdfgroup.org/HDF5/doc/cpplus_RM/hierarchy.html


H5::File is both an H5::Location and H5::CommonFG (but not an 
H5::Object)
H5::Group is both an H5::Object (subclass of H5::Location) and 
H5::CommonFG

H5::Dataset is an H5::Object

Re: D Beginner Trying Manual Memory Management

[ketmar via Digitalmars-d-learn]

On Tue, 13 Jan 2015 16:08:15 +
aldanor via Digitalmars-d-learn digitalmars-d-learn@puremagic.com
wrote:

 On Tuesday, 13 January 2015 at 08:33:57 UTC, ketmar via 
 Digitalmars-d-learn wrote:
  On Mon, 12 Jan 2015 22:07:13 +
  aldanor via Digitalmars-d-learn 
  digitalmars-d-learn@puremagic.com
  wrote:
 
  I see, thanks! :) I've started liking structs more and more 
  recently as well and been pondering on how to convert a 
  class-based code that looks like this (only the base class has 
  any data):
  it's hard to tell by brief description. but having multiple 
  inheritance
  immediately rings an alarm ring for me. something is 
  very-very-very
  wrong if you need to have a winged whale. ;-)
 A real-world example: 
 http://www.hdfgroup.org/HDF5/doc/cpplus_RM/hierarchy.html
 
 H5::File is both an H5::Location and H5::CommonFG (but not an 
 H5::Object)
 H5::Group is both an H5::Object (subclass of H5::Location) and 
 H5::CommonFG
 H5::Dataset is an H5::Object
i see something named CommonFG here, which seems to good thing to
move out of hierarchy altogether.

bwah, i don't even sure that given hierarchy is good for D. C++ has no
UFCS, and it's incredibly hard to check if some entity has some
methods/properties in C++, so they have no other choice than to work
around that limitations. it may be worthful to redesign the whole thing
for D, exploiting D shiny UFCS and metaprogramming features. and,
maybe, moving some things to interfaces too.


signature.asc
Description: PGP signature

D Beginner Trying Manual Memory Management

[jmh530 via Digitalmars-d-learn]

I'm new to D. I have some modest knowledge of C++, but am more 
familiar with scripting languages (Matlab, Python, R). D seems so 
much easier than C++ in a lot of ways (and I just learned about 
rdmd today, which is pretty cool). I am concerned about 
performance of D vs. C++, so I wanted to learn a little bit more 
about manual memory management, in case I might ever need it (not 
for any particular application).


The D Language book Section 6.3.4-5 covers the topic. I basically 
copied below and made some small changes.


import core.stdc.stdlib;
import std.stdio;

class Buffer {
private void* data;
// Constructor
this()
{
data = malloc(1024);
}
// Destructor
~this()
{
free(data);
}
}
unittest {
auto b = new Buffer;
auto b1 = b;
destroy(b); //changed from clear in book example
assert(b1.data is null);
writeln(Unit Test Finished);
}

I was thinking that it might be cool to use scope(exit) to handle 
the memory management. It turns out the below unit test works.


unittest {
auto b = new Buffer;
scope(exit) destroy(b);
writeln(Unittest Finished);
}

However, if you leave in the auto b1 and assert, then it fails. I 
suspect this is for the same reason that shared pointers are a 
thing in C++ (it can't handle copies of the pointer).


Alternately, you can use some other scope and something like this
unittest {
{
Buffer b = new Buffer;
scope(exit) destroy(b);
}
destroy(b);
writeln(Unittest Finished);
}

will fail because b has already been destroyed. I thought this 
behavior was pretty cool. If you followed this approach, then you 
wouldn't have to wait until the end of the program to delete the 
pointers. The downside would be if you need to write a lot of 
pointers and there would be a lot of nesting. (likely the 
motivation for the reference counting approach to smart pointers).


I wasn't sure how to figure out a way to combine these two 
components so that I only have to write one line. I thought one 
approach might be to put a scope(exit) within the constructor, 
but that doesn't work. Also, if I try to do it within a template 
function, then the scope(exit) is limited to the function scope, 
which isn't the same thing.


Outside of writing a unique_ptr template class (which I have 
tried to do, but couldn't get it to work) I don't really know how 
to combine them into one line. Any ideas?

Re: D Beginner Trying Manual Memory Management

[ketmar via Digitalmars-d-learn]

On Mon, 12 Jan 2015 19:29:53 +
jmh530 via Digitalmars-d-learn digitalmars-d-learn@puremagic.com
wrote:

the proper answer is too long to write (it will be more an article that
a forum answer ;-), so i'll just give you some directions:

  import std.typecons;

  {
auto b = scoped!B(); // `auto` is important here!
...
  }

`scoped!` allocating class instance *on* *stack*, and automatically
calls destructor when object goes out of scope.

but you'd better consider using struct for such things, as struct are
stack-allocated by default (unlike classes, which are reference type
and should be allocated manually).

there is a big difference between `class` and `struct` in D, much
bigger that in C++ (where it's only about default protection,
actually).


signature.asc
Description: PGP signature

Re: D Beginner Trying Manual Memory Management

[jmh530 via Digitalmars-d-learn]

Thanks for the reply, I wasn't familiar with scoped. I was aware 
that structs are on the stack and classes are on the heap in D, 
but I didn't know it was possible to put a class on the stack. 
Might be interesting to see how this is implemented.


After looking up some more C++, I think what I was trying to do 
is more like make_unique than unique_ptr.


On Monday, 12 January 2015 at 19:42:14 UTC, ketmar via 
Digitalmars-d-learn wrote:

On Mon, 12 Jan 2015 19:29:53 +
jmh530 via Digitalmars-d-learn 
digitalmars-d-learn@puremagic.com

wrote:

the proper answer is too long to write (it will be more an 
article that

a forum answer ;-), so i'll just give you some directions:

  import std.typecons;

  {
auto b = scoped!B(); // `auto` is important here!
...
  }

`scoped!` allocating class instance *on* *stack*, and 
automatically

calls destructor when object goes out of scope.

but you'd better consider using struct for such things, as 
struct are
stack-allocated by default (unlike classes, which are reference 
type

and should be allocated manually).

there is a big difference between `class` and `struct` in D, 
much

bigger that in C++ (where it's only about default protection,
actually).

Re: D Beginner Trying Manual Memory Management

[ketmar via Digitalmars-d-learn]

On Mon, 12 Jan 2015 20:14:19 +
jmh530 via Digitalmars-d-learn digitalmars-d-learn@puremagic.com
wrote:

 Thanks for the reply, I wasn't familiar with scoped. I was aware 
 that structs are on the stack and classes are on the heap in D, 
 but I didn't know it was possible to put a class on the stack. 
 Might be interesting to see how this is implemented.
actually, there is nothing complicated there (if you don't want to
write an universal thing like `emplace!` ;-). it builds a wrapper
struct big enough to hold class instance, copies class `.init` there
and calls class' constructor. the rest of the magic is done by the
compiler: when struct goes out of scope, compiler calls struct
destructor, which in turn calls class destructor. ah, and it forwards
all other requests with `alias this` trick.

 After looking up some more C++, I think what I was trying to do 
 is more like make_unique than unique_ptr.
i don't remember C++ well, but nevertheless i encouraging you to take a
look at `std.typecons`. there are some handy things there, like
`Rebindable!` or `Nullable!`. and some funny things like `BlackHole!`
and `WhiteHole!`. ;-)

it even has `RefCounted!`, but it doesn't play well with classes yet
(AFAIR).


signature.asc
Description: PGP signature

Re: D Beginner Trying Manual Memory Management

[aldanor via Digitalmars-d-learn]

On Monday, 12 January 2015 at 20:30:45 UTC, ketmar via 
Digitalmars-d-learn wrote:
it even has `RefCounted!`, but it doesn't play well with 
classes yet

(AFAIR).
I wonder if it's possible to somehow make a version of refcounted 
that would work with classes (even if limited/restricted in some 
certain ways), or is it just technically impossible because of 
reference semantics?

Re: D Beginner Trying Manual Memory Management

[ketmar via Digitalmars-d-learn]

On Mon, 12 Jan 2015 21:37:27 +
aldanor via Digitalmars-d-learn digitalmars-d-learn@puremagic.com
wrote:

 On Monday, 12 January 2015 at 20:30:45 UTC, ketmar via 
 Digitalmars-d-learn wrote:
  it even has `RefCounted!`, but it doesn't play well with 
  classes yet
  (AFAIR).
 I wonder if it's possible to somehow make a version of refcounted 
 that would work with classes (even if limited/restricted in some 
 certain ways), or is it just technically impossible because of 
 reference semantics?
it's hard. especially hard when you considering inheritance (which is
not playing well with templates) and yes, ref semantics.

on the other side i found myself rarely using classes at all. i mostly
writing templates that checks if a passed thing has all the
neccessary methods and properties in place and just using that. with D
metaprogramming abilities (and `alias this` trick ;-) inheritance
becomes not so important. and so classes too. sometimes i'm using
structs with delegate fields to simulate some sort of virtual methods
'cause i tend to constantly forgetting about that `class` thingy. ;-)

OOP is overrated. at least c++-like (should i say simula-like?)
OOP. ;-)


signature.asc
Description: PGP signature

Re: D Beginner Trying Manual Memory Management

[aldanor via Digitalmars-d-learn]

On Monday, 12 January 2015 at 21:54:51 UTC, ketmar via 
Digitalmars-d-learn wrote:

On Mon, 12 Jan 2015 21:37:27 +
aldanor via Digitalmars-d-learn 
digitalmars-d-learn@puremagic.com

wrote:

On Monday, 12 January 2015 at 20:30:45 UTC, ketmar via 
Digitalmars-d-learn wrote:
 it even has `RefCounted!`, but it doesn't play well with 
 classes yet

 (AFAIR).
I wonder if it's possible to somehow make a version of 
refcounted that would work with classes (even if 
limited/restricted in some certain ways), or is it just 
technically impossible because of reference semantics?
it's hard. especially hard when you considering inheritance 
(which is

not playing well with templates) and yes, ref semantics.

on the other side i found myself rarely using classes at all. i 
mostly

writing templates that checks if a passed thing has all the
neccessary methods and properties in place and just using that. 
with D
metaprogramming abilities (and `alias this` trick ;-) 
inheritance
becomes not so important. and so classes too. sometimes i'm 
using
structs with delegate fields to simulate some sort of virtual 
methods
'cause i tend to constantly forgetting about that `class` 
thingy. ;-)


OOP is overrated. at least c++-like (should i say 
simula-like?)

OOP. ;-)
I see, thanks! :) I've started liking structs more and more 
recently as well and been pondering on how to convert a 
class-based code that looks like this (only the base class has 
any data):


class Base { T m_variable; }
class Common : Base { /* tons of methods; uses m_variable */ }
class Extra : Base { /* another ton of methods; uses m_variable 
*/ }

class A : Extra, Common { ... }
class B : Common { ... }
class C : Extra { ... }

to refcounted structs with alias this but couldn't quite figure 
out how to do it (other than use mixin templates...). Even if the 
multiple alias this DIP was implemented, I don't think it would 
help much here :/

Re: D Beginner Trying Manual Memory Management

[Mike via Digitalmars-d-learn]

On Monday, 12 January 2015 at 19:29:54 UTC, jmh530 wrote:
I'm new to D. I have some modest knowledge of C++, but am more 
familiar with scripting languages (Matlab, Python, R). D seems 
so much easier than C++ in a lot of ways (and I just learned 
about rdmd today, which is pretty cool). I am concerned about 
performance of D vs. C++, so I wanted to learn a little bit 
more about manual memory management, in case I might ever need 
it (not for any particular application).




There is a good article on the D Wiki that covers this topic with 
several different patterns and working examples:


http://wiki.dlang.org/Memory_Management

I hope you'll find it helpful.

Mike

Re: D Beginner Trying Manual Memory Management

[jmh530 via Digitalmars-d-learn]

I had seen some stuff on alias thing, but I hadn't bothered to
try to understand it until now. If I'm understanding the first
example a href=http://dlang.org/class.html#AliasThis;here/a,
alias this let's you refer to x in s by writing either s.x (as
normally) or just s. That didn't seem that interesting, but then
I found a href=http://3d.benjamin-thaut.de/?p=90;example/a
where they alias this'ed a struct method. That's pretty
interesting.

OOP seems like a good idea, but every time I've written a bunch
of classes in C++ or Python, I inevitably wonder to myself why I
just spent 5 times as long doing something I could do with
functions. Then there's endless discussion about pimpl.

On Monday, 12 January 2015 at 21:54:51 UTC, ketmar via
Digitalmars-d-learn wrote:

On Mon, 12 Jan 2015 21:37:27 +
aldanor via Digitalmars-d-learn 
digitalmars-d-learn@puremagic.com

wrote:

On Monday, 12 January 2015 at 20:30:45 UTC, ketmar via 
Digitalmars-d-learn wrote:
 it even has `RefCounted!`, but it doesn't play well with 
 classes yet

 (AFAIR).
I wonder if it's possible to somehow make a version of 
refcounted that would work with classes (even if 
limited/restricted in some certain ways), or is it just 
technically impossible because of reference semantics?
it's hard. especially hard when you considering inheritance 
(which is

not playing well with templates) and yes, ref semantics.

on the other side i found myself rarely using classes at all. i 
mostly

writing templates that checks if a passed thing has all the
neccessary methods and properties in place and just using that. 
with D
metaprogramming abilities (and `alias this` trick ;-) 
inheritance
becomes not so important. and so classes too. sometimes i'm 
using
structs with delegate fields to simulate some sort of virtual 
methods
'cause i tend to constantly forgetting about that `class` 
thingy. ;-)


OOP is overrated. at least c++-like (should i say 
simula-like?)

OOP. ;-)

Re: Manual Memory Management Example for dlang.org Docs (Code Review)

[Sean Kelly]

On Wednesday, 26 February 2014 at 15:00:13 UTC, Dicebot wrote:

On Wednesday, 26 February 2014 at 04:43:30 UTC, Mike wrote:
Please review this (http://dpaste.dzfl.pl/2377217c7870) 
instead.



throw new Exception(Out of memory);


I don't think it is a good thing to do. Of course GC is likely 
to have some memory left in hs internal pools but in general 
once you hit out-of-memory state last thing you want is not 
allocate even more.


For certain standard errors, the proper approach is probably to 
call the appropriate handler in core.exception.

Re: Manual Memory Management Example for dlang.org Docs (Code Review)

[Dicebot]

On Wednesday, 26 February 2014 at 04:43:30 UTC, Mike wrote:

Please review this (http://dpaste.dzfl.pl/2377217c7870) instead.



throw new Exception(Out of memory);


I don't think it is a good thing to do. Of course GC is likely to 
have some memory left in hs internal pools but in general once 
you hit out-of-memory state last thing you want is not allocate 
even more.



// Deallocate test
heapDeallocate(test);


More idiomatic D is to put `scope(exit) heapDeallocate(test);` 
immediately after allocation line.

Re: Manual Memory Management Example for dlang.org Docs (Code Review)

[Mike]

On Wednesday, 26 February 2014 at 15:00:13 UTC, Dicebot wrote:

On Wednesday, 26 February 2014 at 04:43:30 UTC, Mike wrote:
Please review this (http://dpaste.dzfl.pl/2377217c7870) 
instead.



throw new Exception(Out of memory);


I don't think it is a good thing to do. Of course GC is likely 
to have some memory left in hs internal pools but in general 
once you hit out-of-memory state last thing you want is not 
allocate even more.



// Deallocate test
heapDeallocate(test);


More idiomatic D is to put `scope(exit) heapDeallocate(test);` 
immediately after allocation line.


Thanks,

I've made changes and I guess this is the one I'll use for the 
pull.

(http://dpaste.dzfl.pl/31367860c005)

Manual Memory Management Example for dlang.org Docs (Code Review)

[Mike]

As I study D, I am running across quite a few unmaintained 
examples in dlang.org's documentation.  Rather than leaving them 
for the next guy to step in, I'm (one-by-one) submitting pull 
request to fix them.


I need an good manual memory management example, but the one at 
http://dlang.org/memory.html#newdelete uses the deprecated 
allocators.


Daniel Gibson posted an example here 
(http://forum.dlang.org/post/ip76fa$2o7k$3...@digitalmars.com) back 
in 2011.


I've modified that example and posted a new one here 
(http://dpaste.dzfl.pl/f4b09b9e6c46) using destroy.


I'm quite new to D, so I'm asking the community to please review 
this example so I can submit a pull and update the docs.


Thanks,
Mike

Re: Manual Memory Management Example for dlang.org Docs (Code Review)

[Mike]

On Wednesday, 26 February 2014 at 02:23:47 UTC, Mike wrote:


I've modified that example and posted a new one here 
(http://dpaste.dzfl.pl/f4b09b9e6c46) using destroy.




I modified this based on Adam D. Ruppe's example here( 
http://arsdnet.net/dcode/malloc.d).


Please review this (http://dpaste.dzfl.pl/2377217c7870) instead.

Re: Very short article with with manual memory management in C++ and D.

[Paulo Pinto]

On Wednesday, 6 February 2013 at 21:35:12 UTC, MattCoder wrote:
On Wednesday, 6 February 2013 at 13:16:45 UTC, Paulo Pinto 
wrote:
I just wanted to show that the pointer tricks for memory 
management are not an exclusivity from C and C++.


I get it! Just one more note, would you have done any benchmark 
test on between both versions?


Not really, it is very simple code to be worth doing any 
benchmarks.

Very short article with with manual memory management in C++ and D.

[Paulo Pinto]

Hi,

long time ago I wrote a short article about doing manual memory 
management from C++.


http://www.progtools.org/compilers/tutorials/queue/article.html

This was based on a job offer I saw in a games magazine.

Eventually I ported the solution to D, while keeping the code as 
close as possible to the original C++ version.


http://www.progtools.org/compilers/tutorials/queue/queue.cpp.html

http://www.progtools.org/compilers/tutorials/queue/queue.d.html

Since these type of small exercises are the only area where I 
might use D occasionally, are there any issues on the D version 
that could be improved, while keeping the code as close as 
possible to the C++ version?



Thanks,
Paulo

Re: Very short article with with manual memory management in C++ and D.

[MattCoder]

On Wednesday, 6 February 2013 at 09:36:32 UTC, Paulo Pinto wrote:
Eventually I ported the solution to D, while keeping the code 
as close as possible to the original C++ version.


Paulo, can you tell why you chose to keep the D code much as the
same the C++ version?

Re: Very short article with with manual memory management in C++ and D.

[Paulo Pinto]

On Wednesday, 6 February 2013 at 10:13:49 UTC, MattCoder wrote:
On Wednesday, 6 February 2013 at 09:36:32 UTC, Paulo Pinto 
wrote:
Eventually I ported the solution to D, while keeping the code 
as close as possible to the original C++ version.


Paulo, can you tell why you chose to keep the D code much as the
same the C++ version?


Languages with stronger type checking, and GC capabilities are 
usually looked down by some hardcore C++ developers.


I just wanted to show that the pointer tricks for memory 
management are not an exclusivity from C and C++.


Maybe I should present two versions, the current one which mimics 
the C++ version and an improved version.


--
Paulo

Re: manual memory management

[Timon Gehr]

On 01/09/2013 08:33 AM, Rob T wrote:

On Wednesday, 9 January 2013 at 07:23:57 UTC, Mehrdad wrote:

On Wednesday, 9 January 2013 at 07:22:51 UTC, deadalnix wrote:

Well, you CAN indeed, create a dumbed down language that is memory
safe and don't require a GC.



Yeah, that's 1 of my 2 points.


The other one you still ignored: the GC doesn't bring much to the
table. (Re C# Java etc.)


There is a point being made here that is perfectly valid. There is a
form of memory leak that a GC can never catch, such as when when memory
is allocated and simply never deallocated by mistake due to a persistent
in use pointer that should have been nulled but wasn't.
...


This is not entirely accurate. A GC does not necessarily have to assume 
that every reachable pointer will be accessed again. Every memory leak 
can be caught by some GC, but no GC catches all.

Re: manual memory management

[Paulo Pinto]

On Wednesday, 9 January 2013 at 23:53:40 UTC, Andrei Alexandrescu 
wrote:

On 1/9/13 3:38 PM, Paulo Pinto wrote:

My favorite GC book:

http://www.amazon.com/Garbage-Collection-Algorithms-Automatic-Management/dp/0471941484/ref=sr_1_2?s=books-intl-deie=UTF8qid=1357774599sr=1-2


I own the more recent edition and made a pass through it but I 
don't think it discusses safety that much. Any chapter I should 
be looking at?


Andrei


No really. It is my favorite GC book, because it is the only one 
I know where so many GC algorithms get described, but I haven't 
read it in depth to be able to assert how safety is described.


And here is another one about real time GC that I own, although 
it focus on JVMs:


http://www.amazon.com/Realtime-Collection-Oriented-Programming-Languages/dp/3831138931/ref=sr_1_1?ie=UTF8qid=1357806697sr=8-1keywords=Fridtjof+Siebert

Re: manual memory management

[Paulo Pinto]

On Wednesday, 9 January 2013 at 22:29:36 UTC, Jonathan M Davis 
wrote:

On Wednesday, January 09, 2013 14:14:15 Walter Bright wrote:

On 1/9/2013 1:23 PM, Mehrdad wrote:
 And are you considering reference counting to be garbage 
 collection like

 Walter does,

It's not something I came up with. It is generally accepted 
that ref

counting is a form of garbage collection.


I definitely would never have referred to reference counting as 
garbage
collection and find it quite odd that it would be considered to 
be such, but
Wikipedia does indeed list it as being a form of garbage 
collection:


http://en.wikipedia.org/wiki/Garbage_collection_(computer_science)

So, it's backing you up in your usage of the term.

- Jonathan M Davis


This is what I was saying all along, in CS GC books reference 
counting is usually introduced as poor man's GC solution. As the 
most simple way to implement some kind of automatic memory 
management, specially in memory constrained devices at the 
expense of execution speed.


--
Paulo

Re: manual memory management

[Paulo Pinto]

On Thursday, 10 January 2013 at 00:50:30 UTC, Adam D. Ruppe wrote:
On Thursday, 10 January 2013 at 00:18:26 UTC, Walter Bright 
wrote:
And that is not dereferencing null, it is dereferencing 
0x100.


Yes, but it is worth noting that dmd will happily compile that 
code, even if marked @safe - just because the pointer on the 
language level is null doesn't mean it is memory safe at the 
assembly level.


the generated code with @safe is still just what we'd expect 
too:

   3:   31 c0   xoreax,eax
   5:   c7 80 00 00 10 00 0amovDWORD PTR 
[eax+0x10],0xa


That brings us to another point.

It does not matter how safe a language might be, if someone can 
change the assembly then everything is possible.


Or am I going too off-topic?

Re: manual memory management

[eles]

On Tuesday, 8 January 2013 at 22:19:56 UTC, Walter Bright wrote:

On 1/7/2013 3:11 PM, H. S. Teoh wrote:


One thing I'd add is that a GC is *required* if you want to 
have a language that guarantees memory safety, which D aims to 
do. There's an inescapable reason why manual memory management 
in D is only possible in @system code.


I think that, at least for @system code (or invent another @), D 
programs should be able to require only GC-free code (forbiding, 
for example, features requiring GC).


I do some Linux kernel code. For most of it, I find it quite 
difficult to rely on the GC if I would use D instead of C.


One specific question: when handling an interrupt in Linux 
kernel, you are not allowed to sleep. But I think GC (and 
GC-triggering features) will sleep. So their use should be 
forbidden.


The main difference wrt to C programming for kernel: in C, to 
avoid sleeping, you need to stay away of some kernel/library 
functions; in D, you need to stay away of some language features.


Is one thing to avoid using specific functions. Is another thing 
to avoid using specific language constructs.

Re: manual memory management

[eles]

On Tuesday, 8 January 2013 at 23:10:01 UTC, Paulo Pinto wrote:

Am 08.01.2013 17:12, schrieb Benjamin Thaut:

Am 08.01.2013 16:46, schrieb H. S. Teoh:
Without dismissing your experience in game development, I think 
that your experience was spoiled by D's GC quality.


After all, there are Java VMs driving missiles and ship battle 
systems, which have even higher timer requirements.


Yes, but they are relying on specific, very special constructs of 
Java, such as:


http://www.rtsj.org/specjavadoc/javax/realtime/NoHeapRealtimeThread.html
http://www.rtsj.org/specjavadoc/javax/realtime/RealtimeThread.html

which have very little, if any, to do do with regular Java. BTW, 
since when a Java programmer should be concerned about... heap? 
What's that? malloc()?


The fact that it uses the same syntax as Java, simply does not 
mae it Java, at least in the (regular) JVM sense.

Re: manual memory management

[Paulo Pinto]

On Thursday, 10 January 2013 at 09:05:53 UTC, eles wrote:

On Tuesday, 8 January 2013 at 23:10:01 UTC, Paulo Pinto wrote:

Am 08.01.2013 17:12, schrieb Benjamin Thaut:

Am 08.01.2013 16:46, schrieb H. S. Teoh:
Without dismissing your experience in game development, I 
think that your experience was spoiled by D's GC quality.


After all, there are Java VMs driving missiles and ship battle 
systems, which have even higher timer requirements.


Yes, but they are relying on specific, very special constructs 
of Java, such as:


http://www.rtsj.org/specjavadoc/javax/realtime/NoHeapRealtimeThread.html
http://www.rtsj.org/specjavadoc/javax/realtime/RealtimeThread.html

which have very little, if any, to do do with regular Java. 
BTW, since when a Java programmer should be concerned about... 
heap? What's that? malloc()?


Any developer worth its salt should worry how their application 
makes use of the available resources.




The fact that it uses the same syntax as Java, simply does not 
mae it Java, at least in the (regular) JVM sense.


The JVM is just a possible implementation of Java, the language.

It was quite an unfortunate decision for the Sun marketing team 
to call the language and the VM the same name.


This always ends up in lots of confusions when people consider 
the JVM to be the only way to execute Java.


Usually only developers with compiler design background end up 
making the difference, as languages and implementations are quite 
two different issues.



--
Paulo

Re: manual memory management

[Rob T]

On Thursday, 10 January 2013 at 08:38:18 UTC, Paulo Pinto wrote:


This is what I was saying all along, in CS GC books reference 
counting is usually introduced as poor man's GC solution. As 
the most simple way to implement some kind of automatic memory 
management, specially in memory constrained devices at the 
expense of execution speed.


--
Paulo


This is likely a long shot (and may have already been proposed), 
but what the heck: If reference counting is considered to be 
garbage collection, and D is a garbage collected language, then 
can the current form of GC be replaced with a reference counted 
version that is fully automated, or does something like that 
always have to be manually hand crafted because it is not generic 
enough to fit in?


BTW: I did read through the responses concerning by past posts 
about the GG and memory safety, and I agree with those responses. 
I now have a much better understanding of what is meant by 
memory safety, so thanks to everyone who took the time to 
respond to my ramblings, I've learned something new.


--rt

Re: manual memory management

[Walter Bright]

On 1/10/2013 4:05 PM, Rob T wrote:

This is likely a long shot (and may have already been proposed), but what the
heck: If reference counting is considered to be garbage collection, and D is a
garbage collected language, then can the current form of GC be replaced with a
reference counted version that is fully automated,


Yes, but you'd have to rework the semantics of array slices, function closures, 
etc., to add in the reference counts.

Re: manual memory management

[Timon Gehr]

On 01/08/2013 06:30 AM, H. S. Teoh wrote:

...

I don't know if I know them all, but certainly the following are
GC-dependent:

- Slicing/appending arrays (which includes a number of string
   operations), .dup, .idup;


Slicing is not GC-dependent.


- Delegates  anything requiring access to local variables after the
   containing scope has exited;


Yes, but this does not make delegate literals useless without a GC. 
scope delegate literals do not allocate but instead point directly to 
the stack.



- Built-in AA's;
- Classes (though I believe it's possible to manually manage memory for
   classes via Phobos' emplace), including exceptions (IIRC);


Classes are not GC-dependent at all. 'new'-expressions are GC-dependent. 
(though I think DMD still allows overloading them.)



- std.container (IIRC Andrei was supposed to work on an allocator model
   for it so that it's usable without a GC)

AFAIK, the range-related code in Phobos has been under scrutiny to
contain no hidden allocations (hence the use of structs instead of
classes for various range constructs). So unless there are bugs,
std.range and std.algorithm should be safe to use without involving the
GC.



The choice of structs vs classes is kind of necessary. I do not always 
want to write 'save' in order to not consume other copies of the range.

(InputRangeObject notably gets this wrong, this is why it is not used.)


Static arrays are GC-free, and so are array literals (I *think*) as long
as you don't do any memory-related operation on them like appending or
.dup'ing.


Currently, with DMD, array literals always allocate if they are not 
static. (Even if they are directly assigned to a static array variable!)



So strings should be still somewhat usable, though quite
limited. I don't know if std.format (including writefln  friends)
invoke the GC -- I think they do,  under the hood. So writefln may not be
usable, or maybe it's just certain format strings that can't be used,
and if you're careful you may be able to pull it off without touching
the GC.



I think they use output ranges under the hood. The main issue is 
toString(), which inherently requires GC allocation.



AA literals are NOT safe, though -- anything to do with built-in AA's
will involve the GC. (I have an idea that may make AA literals usable
without runtime allocation -- but CTFE is still somewhat limited right
now so my implementation doesn't quite work yet.)



I think the fact that it is not possible to save away arbitrary data 
structures at compile time for runtime use is just a limitation of the 
current implementation. Anyways mutable literals require allocation 
because of aliasing concerns.



But yeah, it would be nice if the official docs can indicate which
features are GC-dependent.


T

Re: manual memory management

[Walter Bright]

On 1/8/2013 11:42 PM, Mehrdad wrote:

(True, it wouldn't give you the power of a systems language, but that's quite
obviouly not my point -- the point is that it's a _perfectly possible_
memory-safe language which we made, so I don't understand Walter's comment about
a GC being required for a memory-safe language.)



The misunderstanding is you are not considering reference counting as a form of 
GC. It is.

Re: manual memory management

[Mehrdad]

On Wednesday, 9 January 2013 at 08:14:35 UTC, Walter Bright wrote:

On 1/8/2013 11:42 PM, Mehrdad wrote:
(True, it wouldn't give you the power of a systems language, 
but that's quite
obviouly not my point -- the point is that it's a _perfectly 
possible_
memory-safe language which we made, so I don't understand 
Walter's comment about

a GC being required for a memory-safe language.)



The misunderstanding is you are not considering reference 
counting as a form of GC. It is.


So you would say that C++ code (which uses reference counting) 
uses garbage collection?

Re: manual memory management

[Paulo Pinto]

On Wednesday, 9 January 2013 at 02:49:34 UTC, Rob T wrote:
On Tuesday, 8 January 2013 at 23:30:34 UTC, David Nadlinger 
wrote:

On Tuesday, 8 January 2013 at 23:12:43 UTC, Rob T wrote:
The only major thing that concerns me is the lack of proper 
shared library support. I hope this omission is resolved soon.


What do you need it for? Runtime loading of D shared objects? 
Or just linking to them (i.e. binding by ld/dyld at load 
time)? I'm trying to collect data on real-world use cases 
resp. expectations right now.


David


I *really* need runtime loading of plug-in code for a server 
application. This allows the server code to remain untouched 
while allowing extensions to be added on by a 3rd party.


Runtime linked shared libs are also nice to have for the simple 
reason that shared libraries can be updated (to a point) 
without having to rebuild and relink all applications that make 
use of the libraries. There are pros and cons to static vs 
dynamic linking, but definitely both are useful to have.


I'm very surprised that not too many people have been screaming 
for dynamic linking and runtime loading. It's very hard for me 
to imagine not having the feature because it's so darn useful 
and an essential feature if your strategy is to allow 3rd 
parties to extend an application without hacking at the source 
code.


If there's another better way, I'd sure like to know about it!

--rt


You can write plugins without dynamic loading, they just are a 
bit more cumbersome to write.


Like I used to do back in the late 90's, by making use of UNIX's 
IPC.


The IPC to use (shared memory, pipes, mailbox, sockets) depends 
on what is required from the plugin. With shared memory being the 
closest to what dynamic loading achieves.


Of course this raises another set of issues like:

- Take care what happens when a plugin dies
- Too many loaded plugins can stress the scheduler
- You might have synchronization issues when using shared memory
- It is a bit more painful to code for

This is the school of thought of the Plan9/Go guys and most 
operating system micro-kernel architectures.


--
Paulo

Re: manual memory management

[Benjamin Thaut]

Am 09.01.2013 02:59, schrieb H. S. Teoh:

On Wed, Jan 09, 2013 at 12:21:05AM +0100, deadalnix wrote:

On Tuesday, 8 January 2013 at 16:12:41 UTC, Benjamin Thaut wrote:

My impression so far: No one who is writing a tripple A gaming
title or engine is only remotly interested in using a GC. Game
engine programmers almost do anything to get better performance on
a certain plattform. There are really elaborate taks beeing done
just to get 1% more performance. And because of that, a GC is the
very first thing every serious game engine programmer will kick.
You have to keep in mind that most games run at 30 FPS. That means
you only have 33 ms to do everything. Rendering, simulating
physics, doing the game logic, handling network input, playing
sounds, streaming data, and so on.
Some games even try to get 60 FPS which makes it even harder as
you only have 16 ms to compute everything. Everything is
performance critical if you try to achive that.



That is a real misrepresentation of the reality. Such people avoid
the GC, but simply because they avoid all kind of allocation
altogether, preferring allocating up-front.


Interesting, that sounds like the heapless programming of the old, old
days where you map out everything beforehand, and limit yourself to use
only what is there. Y'know, with fixed-sized arrays, stacks, etc., fixed
maximum number of objects, etc.. Not a bad approach if you want to
tightly control everything.

Well, except that you do have preallocation, presumably during runtime
at startup (and between game levels, perhaps?), so it's not as rigid,
but during gameplay itself this is pretty much what it amounts to,
right?


T



No its not heapless programming. You just try to avoid allocations 
during gameplay. But they are done if needed (or if there is no time to 
do it the clean way)


Kind Regards
Benjamin Thaut

Re: manual memory management

[Benjamin Thaut]

Am 09.01.2013 00:21, schrieb deadalnix:


That is a real misrepresentation of the reality. Such people avoid the
GC, but simply because they avoid all kind of allocation altogether,
preferring allocating up-front.


But in the end they still don't want a GC, correct?

Kind Regards
Benjamin Thaut

Re: manual memory management

[Paulo Pinto]

On Wednesday, 9 January 2013 at 08:28:44 UTC, Mehrdad wrote:
On Wednesday, 9 January 2013 at 08:14:35 UTC, Walter Bright 
wrote:

On 1/8/2013 11:42 PM, Mehrdad wrote:
(True, it wouldn't give you the power of a systems language, 
but that's quite
obviouly not my point -- the point is that it's a _perfectly 
possible_
memory-safe language which we made, so I don't understand 
Walter's comment about

a GC being required for a memory-safe language.)



The misunderstanding is you are not considering reference 
counting as a form of GC. It is.


So you would say that C++ code (which uses reference counting) 
uses garbage collection?


Yes.

Reference counting is a poor man's form of garbage collection and 
almost every book about garbage collection starts by introducing 
reference counting, before moving on to mark-and-sweep and all 
the remaining algorithms.


Both fall under the umbrella of automatic memory management.

Oh, a bit off topic but are you aware that C++11 has a GC API?

--
Paulo

Re: manual memory management

[Mehrdad]

On Wednesday, 9 January 2013 at 08:51:47 UTC, Paulo Pinto wrote:

On Wednesday, 9 January 2013 at 08:28:44 UTC, Mehrdad wrote:
On Wednesday, 9 January 2013 at 08:14:35 UTC, Walter Bright 
wrote:

On 1/8/2013 11:42 PM, Mehrdad wrote:
(True, it wouldn't give you the power of a systems language, 
but that's quite
obviouly not my point -- the point is that it's a _perfectly 
possible_
memory-safe language which we made, so I don't understand 
Walter's comment about

a GC being required for a memory-safe language.)



The misunderstanding is you are not considering reference 
counting as a form of GC. It is.


So you would say that C++ code (which uses reference counting) 
uses garbage collection?


Yes.


You (or Walter I guess) are the first person I've seen who calls 
C++ garbage collected.





Oh, a bit off topic but are you aware that C++11 has a GC API?


Yes, but I'm not aware of any code which claims to have written a 
GC for it.



--
Paulo

Re: manual memory management

[Jonathan M Davis]

On Wednesday, January 09, 2013 00:30:32 David Nadlinger wrote:
 On Tuesday, 8 January 2013 at 23:12:43 UTC, Rob T wrote:
  The only major thing that concerns me is the lack of proper
  shared library support. I hope this omission is resolved soon.
 
 What do you need it for? Runtime loading of D shared objects? Or
 just linking to them (i.e. binding by ld/dyld at load time)? I'm
 trying to collect data on real-world use cases resp. expectations
 right now.

You pretty much need shared libraries for plugins (so, runtime loading of 
shared libraries), whereas all dynamic linking really does is save disk space. 
So, I'd consider the loading of shared libraries at runtime to be a necessity 
for some types of projects whereas for pretty much anything other than 
embedded systems (which will probably want to use C anyway), the saved disk 
space part is fairly useless.

With the stuff I work on at work, I couldn't use D precisely because we require 
plugins (think of trying to do something like gstreamer but without plugins -
it doesn't work very well). Now, much as I love D, I wouldn't try and get the 
stuff that I'm doing at work moved over to D, because it's a large, existing, 
C++ code base which works quite well, and I don't think that switching to D 
would be worth it, but anyone looking to do similar projects in D would be out 
of luck right now.

- Jonathan M Davis

Re: manual memory management

[Mehrdad]

On Wednesday, 9 January 2013 at 08:54:11 UTC, Mehrdad wrote:
Yes, but I'm not aware of any code which claims to have written 
a GC for it.



A precise* GC that is, not a conservative one.

Re: manual memory management

[Jonathan M Davis]

On Wednesday, January 09, 2013 09:54:10 Mehrdad wrote:
 You (or Walter I guess) are the first person I've seen who calls
 C++ garbage collected.

I sure wouldn't call that garbage collection - not when there's no garbage 
collector. But Walter has certainly called it that from time to time.

I think that the other term that Paulo just used - automatic memory management 
- is far more accurate.

- Jonathan M Davis