Re: An important pull request: accessing shared affix for immutable data

[tsbockman via Digitalmars-d]

On Saturday, 13 February 2016 at 03:05:08 UTC, Andrei 
Alexandrescu wrote:
I was actually suggesting encoding the shared-ness into the 
memory
address itself. (No tricks - just allocate shared data in a 
separate
memory range. This may only be feasible on 64-bit though, with 
its

over-abundance of address space.)

This way, you can determine whether the (meta-)data is shared 
without
accessing it at all, just by asking the allocator about the 
address.


Yah, TypedAllocator can be used for exactly that. -- Andrei


Cool. Then this technique could also be used to address Timon 
Gehr's concerns about violating the type system, if necessary.

Re: An important pull request: accessing shared affix for immutable data

[Andrei Alexandrescu via Digitalmars-d]

On 02/12/2016 10:03 PM, tsbockman wrote:

On Saturday, 13 February 2016 at 01:27:49 UTC, Andrei Alexandrescu wrote:

On 02/12/2016 08:02 PM, tsbockman wrote:

What about providing a way to mark a piece of data as thread-local at
allocation time?


Yah, could be part of the metadata. Then after inspecting that bit,
shared can be casted away. I consider that an optimization that
doesn't add or remove generality. -- Andrei


I was actually suggesting encoding the shared-ness into the memory
address itself. (No tricks - just allocate shared data in a separate
memory range. This may only be feasible on 64-bit though, with its
over-abundance of address space.)

This way, you can determine whether the (meta-)data is shared without
accessing it at all, just by asking the allocator about the address.


Yah, TypedAllocator can be used for exactly that. -- Andrei

Re: An important pull request: accessing shared affix for immutable data

[tsbockman via Digitalmars-d]

On Saturday, 13 February 2016 at 01:27:49 UTC, Andrei 
Alexandrescu wrote:

On 02/12/2016 08:02 PM, tsbockman wrote:
What about providing a way to mark a piece of data as 
thread-local at

allocation time?


Yah, could be part of the metadata. Then after inspecting that 
bit, shared can be casted away. I consider that an optimization 
that doesn't add or remove generality. -- Andrei


I was actually suggesting encoding the shared-ness into the 
memory address itself. (No tricks - just allocate shared data in 
a separate memory range. This may only be feasible on 64-bit 
though, with its over-abundance of address space.)


This way, you can determine whether the (meta-)data is shared 
without accessing it at all, just by asking the allocator about 
the address.

Re: An important pull request: accessing shared affix for immutable data

[Andrei Alexandrescu via Digitalmars-d]

On 02/12/2016 09:21 PM, Timon Gehr wrote:

Const could also mean mutable. This can hence reference the same data as
both shared and unshared, which violates the type system.


If const comes from mutable, then shared is superfluous leading to extra 
synchronization. That's suboptimal, but how does it violate the type 
system? -- Andrei

Re: An important pull request: accessing shared affix for immutable data

[Andrei Alexandrescu via Digitalmars-d]

On 02/12/2016 08:42 PM, Dicebot wrote:

So you
envision this kind of metadata support to be only available via specific
type of allocator, AffixAllocator, and not being supported by most of them?


That is correct. You want metadata, you put an AffixAllocator together. 
-- Andrei

Re: An important pull request: accessing shared affix for immutable data

[Timon Gehr via Digitalmars-d]

On 12.02.2016 20:12, Andrei Alexandrescu wrote:


* If the buffer is mutable, then the allocator assumes it hasn't been
shared across threads, so it returns a reference to a mutable affix.

* If the buffer is const, then the allocator must conservatively assume
it might have been immutable and subsequently shared among threads.
Therefore, several threads may request the affix of the same buffer
simultaneously. So it returns a reference to a shared affix.


Const could also mean mutable. This can hence reference the same data as 
both shared and unshared, which violates the type system.

Re: An important pull request: accessing shared affix for immutable data

[Timon Gehr via Digitalmars-d]

On 13.02.2016 01:30, Andrei Alexandrescu wrote:

On 02/12/2016 06:29 PM, Timon Gehr wrote:

The first thing that comes to mind is that accessing a global
associative array is not 'pure'.


The weird thing is in this case is. The analogy is limited. -- Andrei


Why is it limited? Accessing the affix isn't pure either. I.e. reference 
counting will not work in pure code.

Re: An important pull request: accessing shared affix for immutable data

[Dicebot via Digitalmars-d]

On 02/13/2016 03:35 AM, Andrei Alexandrescu wrote:
> Folks who defined refcounted types use the special AffixAllocator
> internally in an encapsulated manner. They define it, they use it -
> outside intervention is not possible. Those who use their own allocation
> calls and then use assumeUnique cannot use it on the new types. Or maybe
> I'm not understanding something.

Ah probably this is the important point I have totally missed. So you
envision this kind of metadata support to be only available via specific
type of allocator, AffixAllocator, and not being supported by most of them?

Re: An important pull request: accessing shared affix for immutable data

[Andrei Alexandrescu via Digitalmars-d]

On 02/12/2016 08:24 PM, Dicebot wrote:

- considering this API is going to be dangerously @system, possible
corruption of metadata is indeed very scary - though not a blocker
within my own value list


I don't think this is an argument worth minding, either.


- I wonder if would make for a cleaner design to have distinct allocator
instances for thread-local and shared memory instead of branching on
qualifiers of requested type


TypedAllocator does exactly that. Not difficult to integrate here, but 
the key here is removing immutability without subverting the type system 
(which is a distinct matter).



Right now the main concern to me is how such design is going to interact
with existing code like `assumeUnique`. With GC which treats all memory
similarly it is legal (and somewhat common) to allocate mutable
non-shared data first and only later cook it into immutable via
`assumeUnique`. But if allocator has to treat such memory differently
(i.e. add barriers/atomics  for non-TLS one) such code will start
silently introduce major bugs.


Folks who defined refcounted types use the special AffixAllocator 
internally in an encapsulated manner. They define it, they use it - 
outside intervention is not possible. Those who use their own allocation 
calls and then use assumeUnique cannot use it on the new types. Or maybe 
I'm not understanding something.



Andrei

Re: An important pull request: accessing shared affix for immutable data

[Andrei Alexandrescu via Digitalmars-d]

On 02/12/2016 08:02 PM, tsbockman wrote:

What about providing a way to mark a piece of data as thread-local at
allocation time?


Yah, could be part of the metadata. Then after inspecting that bit, 
shared can be casted away. I consider that an optimization that doesn't 
add or remove generality. -- Andrei

Re: An important pull request: accessing shared affix for immutable data

[Dicebot via Digitalmars-d]

On 02/12/2016 09:12 PM, Andrei Alexandrescu wrote:
> So after thinking a bit I managed to convince myself that the affixes in
> an AffixAllocator can be accessed with removing immutable, but without
> actually breaking any guarantee made by the type system. (Affixes are
> extra bytes allocated before and after the actual allocation.) The logic
> goes as follows:
> 
> * If the buffer is mutable, then the allocator assumes it hasn't been
> shared across threads, so it returns a reference to a mutable affix.
> 
> * If the buffer is const, then the allocator must conservatively assume
> it might have been immutable and subsequently shared among threads.
> Therefore, several threads may request the affix of the same buffer
> simultaneously. So it returns a reference to a shared affix.
> 
> * If the buffer is shared, then the allocator assumes again several
> threads may access the affix so it returns a reference to a shared affix.

Thank you for moving forward with this. I haven't yet had chance to play
with the code but it looks like a good start. Some minor notes on
implementation (I have also discussed it a bit with deadalnix in IRC):

- cache thrashing for shared allocations can be reduced by aligning
those on cache line size (for thread-local ones it is not needed)
- considering this API is going to be dangerously @system, possible
corruption of metadata is indeed very scary - though not a blocker
within my own value list
- I wonder if would make for a cleaner design to have distinct allocator
instances for thread-local and shared memory instead of branching on
qualifiers of requested type

Right now the main concern to me is how such design is going to interact
with existing code like `assumeUnique`. With GC which treats all memory
similarly it is legal (and somewhat common) to allocate mutable
non-shared data first and only later cook it into immutable via
`assumeUnique`. But if allocator has to treat such memory differently
(i.e. add barriers/atomics  for non-TLS one) such code will start
silently introduce major bugs.

Sadly, don't have any specific suggeestions about all of this yet.

Re: An important pull request: accessing shared affix for immutable data

[tsbockman via Digitalmars-d]

On Friday, 12 February 2016 at 19:12:48 UTC, Andrei Alexandrescu 
wrote:
* If the buffer is const, then the allocator must 
conservatively assume it might have been immutable and 
subsequently shared among threads. Therefore, several threads 
may request the affix of the same buffer simultaneously. So it 
returns a reference to a shared affix.


What about providing a way to mark a piece of data as 
thread-local at allocation time?


The allocator could then stick it in a separate memory range from 
the (potentially) shared stuff, and just issue a fatal `Error` if 
an attempt was made to access it from the wrong thread.


This would prevent `const` from triggering unnecessary atomic ops 
or locks. The main disadvantage I see is that the logic to find 
the correct reference count address would be a little more 
complicated, but I suspect this would be a good trade-off given 
how expensive thread-safe memory operations can be.

Re: An important pull request: accessing shared affix for immutable data

[tsbockman via Digitalmars-d]

On Friday, 12 February 2016 at 19:12:48 UTC, Andrei Alexandrescu 
wrote:
* If the buffer is const, then the allocator must 
conservatively assume it might have been immutable and 
subsequently shared among threads. Therefore, several threads 
may request the affix of the same buffer simultaneously. So it 
returns a reference to a shared affix.


I understand the reasoning here, but I really dislike the idea of 
`const` as a de-optimization. However, given the way that const 
is used, I guess this wouldn't be much of a problem in practice 
provided that some form of borrowing or inc/dec pair elision is 
implemented.

Re: An important pull request: accessing shared affix for immutable data

[Andrei Alexandrescu via Digitalmars-d]

On 02/12/2016 06:52 PM, deadalnix wrote:

Providing some metadata in the allocate is in itself a good idea.
Locating these data with the object is usually not :
  - Mutating the metadata will create sharing overhead on the whole
cache line. Sharing of immutable would become inefficient.
  - It tends to create allocation size that aren't friendly to
underlying allocators. For instance, an alocation of size 2^n + 8 bumps
you to the next size class, often 2^(n+1) or alike. This creates a lot
of internal fragmentation.
  - Buffer overflow/underflow WILL spill in the alocator metadata. You
don't want that. This pretty much guaranteed that the worse thing that
can happen will happen: corrupting the alocator.

jemalloc moved away from using them for small runs for these reasons.


I think we're good there. -- Andrei

Re: An important pull request: accessing shared affix for immutable data

[Andrei Alexandrescu via Digitalmars-d]

On 02/12/2016 06:29 PM, Timon Gehr wrote:

The first thing that comes to mind is that accessing a global
associative array is not 'pure'.


The weird thing is in this case is. The analogy is limited. -- Andrei

Re: An important pull request: accessing shared affix for immutable data

[deadalnix via Digitalmars-d]

On Friday, 12 February 2016 at 19:12:48 UTC, Andrei Alexandrescu 
wrote:

https://github.com/D-Programming-Language/phobos/pull/3991

A short while ago Dicebot discussed the notion of using the 
allocator to store the reference count of objects (and 
generally metadata). The allocator seems to be a good place 
because in a way it's a source of "ground truth" - no matter 
how data is qualified, it originated as untyped mutable bytes 
from the allocator.


So after thinking a bit I managed to convince myself that the 
affixes in an AffixAllocator can be accessed with removing 
immutable, but without actually breaking any guarantee made by 
the type system. (Affixes are extra bytes allocated before and 
after the actual allocation.) The logic goes as follows:


* If the buffer is mutable, then the allocator assumes it 
hasn't been shared across threads, so it returns a reference to 
a mutable affix.


* If the buffer is const, then the allocator must 
conservatively assume it might have been immutable and 
subsequently shared among threads. Therefore, several threads 
may request the affix of the same buffer simultaneously. So it 
returns a reference to a shared affix.


* If the buffer is shared, then the allocator assumes again 
several threads may access the affix so it returns a reference 
to a shared affix.


One simple way to look at this is: the allocator keeps an 
associative array mapping allocated buffers to metadata (e.g. 
reference counts). The allocated buffers may be immutable, 
which doesn't require the metadata to be immutable as well. The 
only difference between an approach based on an associative 
array and AffixAllocator is that the latter is faster (the 
association is fixed by layout).



Destroy!

Andrei


So if I may, I think we should avoid affix data in the general 
case.


Providing some metadata in the allocate is in itself a good idea. 
Locating these data with the object is usually not :
 - Mutating the metadata will create sharing overhead on the 
whole cache line. Sharing of immutable would become inefficient.
 - It tends to create allocation size that aren't friendly to 
underlying allocators. For instance, an alocation of size 2^n + 8 
bumps you to the next size class, often 2^(n+1) or alike. This 
creates a lot of internal fragmentation.
 - Buffer overflow/underflow WILL spill in the alocator metadata. 
You don't want that. This pretty much guaranteed that the worse 
thing that can happen will happen: corrupting the alocator.


jemalloc moved away from using them for small runs for these 
reasons.

Re: An important pull request: accessing shared affix for immutable data

[Timon Gehr via Digitalmars-d]

On 12.02.2016 20:12, Andrei Alexandrescu wrote:

https://github.com/D-Programming-Language/phobos/pull/3991

A short while ago Dicebot discussed the notion of using the allocator to
store the reference count of objects (and generally metadata). The
allocator seems to be a good place because in a way it's a source of
"ground truth" - no matter how data is qualified, it originated as
untyped mutable bytes from the allocator.

So after thinking a bit I managed to convince myself that the affixes in
an AffixAllocator can be accessed with removing immutable, but without
actually breaking any guarantee made by the type system. (Affixes are
extra bytes allocated before and after the actual allocation.) The logic
goes as follows:

* If the buffer is mutable, then the allocator assumes it hasn't been
shared across threads, so it returns a reference to a mutable affix.

* If the buffer is const, then the allocator must conservatively assume
it might have been immutable and subsequently shared among threads.
Therefore, several threads may request the affix of the same buffer
simultaneously. So it returns a reference to a shared affix.

* If the buffer is shared, then the allocator assumes again several
threads may access the affix so it returns a reference to a shared affix.

One simple way to look at this is: the allocator keeps an associative
array mapping allocated buffers to metadata (e.g. reference counts). The
allocated buffers may be immutable, which doesn't require the metadata
to be immutable as well. The only difference between an approach based
on an associative array and AffixAllocator is that the latter is faster
(the association is fixed by layout).


Destroy!

Andrei



The first thing that comes to mind is that accessing a global 
associative array is not 'pure'.

Re: Safe cast of arrays

[Chris Wright via Digitalmars-d]

On Fri, 12 Feb 2016 14:32:32 -0500, Steven Schveighoffer wrote:

> what I'm trying to say safe shouldn't allow is reinterpret casting.
> i.e.: *cast(T*)(&x)
> 
> So casting IMO shouldn't be allowed unless it invokes some kind of
> handler that ensures the conversion is safe.
> 
> I'd include in this list:
> 
> a) casting between object types
> b) casting builtin types that are not,
> or do not contain, references (that are defined by the compiler)
> c) casting an aggregate that has a matching opCast

Casting an array is basically a backdoor way to make a union, ignoring 
opCast. One of the cases that should be explicitly disallowed here (and 
of course it isn't). Observe:

import std.stdio;
struct A {
void* m;
size_t i;
}
struct B {
size_t i;
A opCast() {
return A(null, i);
}
}

void main() @safe {
A[] aa = [A(new int, 5)];
auto bb = cast(B[])aa;
writeln(bb[0].i);  // prints -22192128
}

If this honored opCast, it would print 5. Instead it prints a pointer 
address. (Also, the length of array bb is 2.)

This corresponds to what the spec says, but that's probably not the 
desired behavior.

learn some d those days, can't understand the code here

[Junfeng via Digitalmars-d]

Hi,

Come here for help. what I doing is setup vim and use dcd for 
goto define, but sometimes dcd-server got crash, issue is here: 
https://github.com/Hackerpilot/DCD/issues/294


Then I build the debug version, and an assert error comes out:

https://github.com/Hackerpilot/libdparse/blob/master/src/dparse/parser.d#L6654

paste here:

T[] ownArray(T)(T[] from)
{
if (allocator is null)
return from;
if (from.length == 0)
return null;
T[] to = cast(T[]) allocator.allocate(T.sizeof * 
from.length);
assert (to.length == from.length, format("from.length = 
%d, to.length = %d", from.length, to.length));

to[] = from[];
return to;
}

For my limited d knowledge, T[] is an array has abi layout:

0: size
size_t: ptr

seems the code try to allocate an array with size from.length, 
but after cast, the length at offset 0 will be 0, so assert 
fail(why dmd allow this cast here? I try use GC.malloc and cast 
in my small test app, dmd will error "Error: cannot cast 
expression malloc(400LU, 0u, null) of type void* to Node[]").


a quick search, seem we should write it like:

t[] to = (cast(T*)allocator.allocate(T.sizeof * from.length))[0 
.. from.length]);


but after this modify, it will crash at
 to[] = from[];

Re: Weird issue with std.range.iota.length

[Meta via Digitalmars-d]

On Friday, 12 February 2016 at 19:20:29 UTC, Andrei Alexandrescu 
wrote:

On 02/12/2016 02:18 PM, Meta wrote:
On Friday, 12 February 2016 at 19:00:29 UTC, Andrei 
Alexandrescu wrote:

Tried that for a while, it quickly became a mess of entangled
intestines. Not worth it. -- Andrei


What do you think of the solution I proposed above? It's not 
pretty but

there has to be some kind of workaround. Not being able to use
`parallel` with an iota-range of ulongs on 32-bit DMD is not 
good. If
you will pre-approve it (comments/suggestions optional) I can 
submit a

PR within the next day or so.


I'd be more partial to an additional template parameter. Or 
even change the type of length in certain cases. -- Andrei


Okay, I will look into other solutions and post some options soon.

Re: Safe cast of arrays

[Steven Schveighoffer via Digitalmars-d]

On 2/12/16 12:15 PM, Chris Wright wrote:


Casting between primitive value types (eg long -> int) is @safe. You
can't get memory errors that way, and the conversions are well-defined.

Casting between object references is @safe (assuming the object
references are valid; @safe doesn't protect you from dereferencing an
invalid pointer you got from @system code). You can dereference null that
way, but that's allowed by design.


All good points.

what I'm trying to say safe shouldn't allow is reinterpret casting. 
i.e.: *cast(T*)(&x)


So casting IMO shouldn't be allowed unless it invokes some kind of 
handler that ensures the conversion is safe.


I'd include in this list:

a) casting between object types
b) casting builtin types that are not, or do not contain, references 
(that are defined by the compiler)

c) casting an aggregate that has a matching opCast


If you wanted to restrict casts between array types, that would be more
reasonable, but some work has already gone into making those casts safe
(eg long[] -> int[]). It would also prevent @safe memory-mapped IO, even
if we provided a wrapper that yielded a ubyte[].


Casting an array involves casting a pointer with a reinterpret style 
cast. IMO, the language is better off requiring a @trusted escape for 
such things.




If you're just talking about casting from void[] in @safe code, that's
reasonable, but a little more restrictive than necessary. Casting *to*
void[] in this scenario is safe, just not generally useful -- you
wouldn't be able to cast back in @safe code.


casting to void[] doesn't require a cast. So I think it should be fine 
in @safe code.


-Steve

Re: Weird issue with std.range.iota.length

[Andrei Alexandrescu via Digitalmars-d]

On 02/12/2016 02:18 PM, Meta wrote:

On Friday, 12 February 2016 at 19:00:29 UTC, Andrei Alexandrescu wrote:

Tried that for a while, it quickly became a mess of entangled
intestines. Not worth it. -- Andrei


What do you think of the solution I proposed above? It's not pretty but
there has to be some kind of workaround. Not being able to use
`parallel` with an iota-range of ulongs on 32-bit DMD is not good. If
you will pre-approve it (comments/suggestions optional) I can submit a
PR within the next day or so.


I'd be more partial to an additional template parameter. Or even change 
the type of length in certain cases. -- Andrei

Re: Weird issue with std.range.iota.length

[Andrei Alexandrescu via Digitalmars-d]

On 02/12/2016 02:10 PM, H. S. Teoh via Digitalmars-d wrote:

The intestines are still visible in iota(), and anything that uses it in
32-bit.


Do we want them everywhere? -- Andrei

Re: Weird issue with std.range.iota.length

[Meta via Digitalmars-d]

On Friday, 12 February 2016 at 19:00:29 UTC, Andrei Alexandrescu 
wrote:
Tried that for a while, it quickly became a mess of entangled 
intestines. Not worth it. -- Andrei


What do you think of the solution I proposed above? It's not 
pretty but there has to be some kind of workaround. Not being 
able to use `parallel` with an iota-range of ulongs on 32-bit DMD 
is not good. If you will pre-approve it (comments/suggestions 
optional) I can submit a PR within the next day or so.

Re: Weird issue with std.range.iota.length

[H. S. Teoh via Digitalmars-d]

On Fri, Feb 12, 2016 at 02:00:29PM -0500, Andrei Alexandrescu via Digitalmars-d 
wrote:
> On 02/12/2016 12:58 PM, H. S. Teoh via Digitalmars-d wrote:
> >IMO, fixing a specific type for length is a mistake.  Truly generic
> >code should treat the length as an opaque type (possibly satisfying
> >some constraints, like <-comparable, ==-comparable, and maybe
> >supporting basic arithmetic operations), and should use generic
> >templates like CommonType!(...) if it needs to work with multiple
> >length types.
> 
> Tried that for a while, it quickly became a mess of entangled
> intestines.  Not worth it. -- Andrei

The intestines are still visible in iota(), and anything that uses it in
32-bit.


T

-- 
This is a tpyo.

An important pull request: accessing shared affix for immutable data

[Andrei Alexandrescu via Digitalmars-d]

https://github.com/D-Programming-Language/phobos/pull/3991

A short while ago Dicebot discussed the notion of using the allocator to 
store the reference count of objects (and generally metadata). The 
allocator seems to be a good place because in a way it's a source of 
"ground truth" - no matter how data is qualified, it originated as 
untyped mutable bytes from the allocator.


So after thinking a bit I managed to convince myself that the affixes in 
an AffixAllocator can be accessed with removing immutable, but without 
actually breaking any guarantee made by the type system. (Affixes are 
extra bytes allocated before and after the actual allocation.) The logic 
goes as follows:


* If the buffer is mutable, then the allocator assumes it hasn't been 
shared across threads, so it returns a reference to a mutable affix.


* If the buffer is const, then the allocator must conservatively assume 
it might have been immutable and subsequently shared among threads. 
Therefore, several threads may request the affix of the same buffer 
simultaneously. So it returns a reference to a shared affix.


* If the buffer is shared, then the allocator assumes again several 
threads may access the affix so it returns a reference to a shared affix.


One simple way to look at this is: the allocator keeps an associative 
array mapping allocated buffers to metadata (e.g. reference counts). The 
allocated buffers may be immutable, which doesn't require the metadata 
to be immutable as well. The only difference between an approach based 
on an associative array and AffixAllocator is that the latter is faster 
(the association is fixed by layout).



Destroy!

Andrei

Re: Weird issue with std.range.iota.length

[Andrei Alexandrescu via Digitalmars-d]

On 02/12/2016 12:58 PM, H. S. Teoh via Digitalmars-d wrote:

IMO, fixing a specific type for length is a mistake.  Truly generic code
should treat the length as an opaque type (possibly satisfying some
constraints, like <-comparable, ==-comparable, and maybe supporting
basic arithmetic operations), and should use generic templates like
CommonType!(...) if it needs to work with multiple length types.


Tried that for a while, it quickly became a mess of entangled 
intestines. Not worth it. -- Andrei

Re: Weird issue with std.range.iota.length

[H. S. Teoh via Digitalmars-d]

On Fri, Feb 12, 2016 at 05:38:40PM +, Jonathan M Davis via Digitalmars-d 
wrote:
> On Friday, 12 February 2016 at 16:52:17 UTC, ixid wrote:
> >On Friday, 12 February 2016 at 15:59:09 UTC, Jonathan M Davis wrote:
> >>It would be far better IMHO to just do a check in iota and throw a
> >>RangeError if the length wouldn't fit in size_t. Having length ever
> >>be anything other than size_t is just going to cause problems with
> >>other ranges. On 32-bit systems, you lose out on the ability to have
> >>a range that covers all values of long or ulong, but that's of very
> >>limited usefulness anyway, and as long as the number of elements is
> >>no greater than size_t.max, it would be fine - which would cover
> >>virtually all use cases. No, it's not perfect, but allowing length
> >>to be anything but size_t just causes bugs - especially in generic
> >>code.
> >>
> >>- Jonathan M Davis
> >
> >What about a template overload where you can set the length type as
> >separate from the type of the range elements?
> 
> No other range has anything but size_t for its length. It's what
> arrays use.  It's what the container types use. Code in general is
> going to assume that length is size_t. I think that having any range
> types with a length type of anything but size_t is a mistake. It
> interacts badly with everything else.  The post that started this
> thread is just one example of that.
[...]

IMO, fixing a specific type for length is a mistake.  Truly generic code
should treat the length as an opaque type (possibly satisfying some
constraints, like <-comparable, ==-comparable, and maybe supporting
basic arithmetic operations), and should use generic templates like
CommonType!(...) if it needs to work with multiple length types.

Restricting length to size_t or ulong or whatever, is an arbitrary
restriction on ranges, because conceptually you can have ranges whose
length exceeds the size of all native integral types (e.g., BigInt
lengths), but certain length computations may still make sense, and the
range is still usable as long as you only ever operate on a small
segment of it.

Of course, working with an opaque length type makes it harder to write
correct code, but such is the price of generic programming.


T

-- 
Debian GNU/Linux: Cray on your desktop.

opApply and opApplyReverse

[Q. Schroll via Digitalmars-d]

One upon time, we decided that we can replace opNeg, opCom, etc. 
and opAdd, opSubtract, etc. by generic names opUnary and opBinary.


Why don't we have a single iteration operator? Can't we just 
provide the "Reverse" information by some bool argument (or 
string if you like maybe more functionality in the future) like 
the other operators?


I'd suppose to make opApply and opApplyReverse aliases to 
!false / !"Forward" and !true / 
!"Reverse" for compatibility.


The workaround I use:
static pure opApp(bool rev)()
{
  import std.format : format;
  immutable code =
  q{
int opApply%s(scope int delegate(ref Idcs) dg)
{
  ... (with %s everywhere something Reverse-generic happens)
}
  }
  return rev ? code.format("Reverse", ...)
 : code.format("",...);
}

mixin(opApp!false);
mixin(opApp!true);

Re: Weird issue with std.range.iota.length

[Jonathan M Davis via Digitalmars-d]

On Friday, 12 February 2016 at 16:52:17 UTC, ixid wrote:
On Friday, 12 February 2016 at 15:59:09 UTC, Jonathan M Davis 
wrote:
It would be far better IMHO to just do a check in iota and 
throw a RangeError if the length wouldn't fit in size_t. 
Having length ever be anything other than size_t is just going 
to cause problems with other ranges. On 32-bit systems, you 
lose out on the ability to have a range that covers all values 
of long or ulong, but that's of very limited usefulness 
anyway, and as long as the number of elements is no greater 
than size_t.max, it would be fine - which would cover 
virtually all use cases. No, it's not perfect, but allowing 
length to be anything but size_t just causes bugs - especially 
in generic code.


- Jonathan M Davis


What about a template overload where you can set the length 
type as separate from the type of the range elements?


No other range has anything but size_t for its length. It's what 
arrays use. It's what the container types use. Code in general is 
going to assume that length is size_t. I think that having any 
range types with a length type of anything but size_t is a 
mistake. It interacts badly with everything else. The post that 
started this thread is just one example of that.


- Jonathan M Davis

Re: [dlang.org] Getting the ddox pages out of limbo

[via Digitalmars-d]

On Fri, Feb 12, 2016 at 06:19:45PM +0100, anonymous via Digitalmars-d wrote:
> And then there's Adam D. Ruppe's . No idea how that
> plays into the whole mess we're in.

It is separate exactly to avoid the trouble of the official site.

I certainly wouldn't mind if we were to switch over, but I'm not quite ready 
for that yet, I am still doing a lot of changes on my site.

[dlang.org] Getting the ddox pages out of limbo

[anonymous via Digitalmars-d]

We currently have two differently generated versions of the library 
documentation on dlang.org:


1) dlang.org/phobos/* (and dlang.org/phobos-prerelease/*) is generated 
by dmd's Ddoc functionality. These are the official docs, part of the 
"Documentation" section.


2) dlang.org/library/* (and dlang.org/library-prerelease/*) is generated 
by ddox (). Once upon a time, 
these docs were supposed to become official, replacing the other ones. 
Currently, they're in an unfortunate state of beta-forever.


ddox has a number of issues that make it currently not feasible to just 
switch over:


* Macro for "path to base of docs"?
https://github.com/rejectedsoftware/ddox/issues/87

* macros from parent scopes are not known in child scopes
https://github.com/rejectedsoftware/ddox/issues/116

* treating of underscore not consistent with dmd
https://github.com/rejectedsoftware/ddox/issues/117

There may be more.

Now, is the plan still to make the switch to ddox for the official 
documentation? Is anyone actually working towards that goal?


If not, maybe we should get rid of the ddox pages, at least for the time 
being. Currently, I think they do more harm than good: They confuse 
people who find them through google, and they drain maintenance resources.


Regarding google, hiding them via robots.txt could be a milder 
alternative to outright deleting them.


And then there's Adam D. Ruppe's . No idea how 
that plays into the whole mess we're in.

Re: Safe cast of arrays

[Chris Wright via Digitalmars-d]

On Fri, 12 Feb 2016 08:45:54 -0500, Steven Schveighoffer wrote:

> A cast to const may be viable.

Touché.

> However, I think casting in safe code is
> probably not something to allow.

*All* casting?

Casting between primitive value types (eg long -> int) is @safe. You 
can't get memory errors that way, and the conversions are well-defined.

Casting between object references is @safe (assuming the object 
references are valid; @safe doesn't protect you from dereferencing an 
invalid pointer you got from @system code). You can dereference null that 
way, but that's allowed by design.

If you wanted to restrict casts between array types, that would be more 
reasonable, but some work has already gone into making those casts safe 
(eg long[] -> int[]). It would also prevent @safe memory-mapped IO, even 
if we provided a wrapper that yielded a ubyte[].

If you're just talking about casting from void[] in @safe code, that's 
reasonable, but a little more restrictive than necessary. Casting *to* 
void[] in this scenario is safe, just not generally useful -- you 
wouldn't be able to cast back in @safe code.

Re: Weird issue with std.range.iota.length

[ixid via Digitalmars-d]

On Friday, 12 February 2016 at 15:59:09 UTC, Jonathan M Davis 
wrote:
It would be far better IMHO to just do a check in iota and 
throw a RangeError if the length wouldn't fit in size_t. Having 
length ever be anything other than size_t is just going to 
cause problems with other ranges. On 32-bit systems, you lose 
out on the ability to have a range that covers all values of 
long or ulong, but that's of very limited usefulness anyway, 
and as long as the number of elements is no greater than 
size_t.max, it would be fine - which would cover virtually all 
use cases. No, it's not perfect, but allowing length to be 
anything but size_t just causes bugs - especially in generic 
code.


- Jonathan M Davis


What about a template overload where you can set the length type 
as separate from the type of the range elements?

Re: Weird issue with std.range.iota.length

[Steven Schveighoffer via Digitalmars-d]

On 2/12/16 10:59 AM, Jonathan M Davis wrote:

On Friday, 12 February 2016 at 14:36:29 UTC, Meta wrote:

On Friday, 12 February 2016 at 08:11:57 UTC, Jonathan M Davis wrote:

What it comes down to is that length should always be size_t. That's
what it is for arrays, and that's what most code expects. Allowing
other types just causes trouble for generic code. However, in the
case of iota with long, if length is size_t, then on 32-bit systems,
it's possible to have a range from iota which is longer than size_t
can represent (much as it would normally be crazy to have a range
that long). So, at some point, someone made it so that iota uses
ulong for length instead of size_t when it's a range of longs or
ulongs. It's the only thing in Phobos that does, and it causes
problems. Changing it back to size_t has been discussed but not
agreed upon. But we're between a rock and a hard place with this one.
There is no clean solution.

Personally, I'd very much like to see iota just always use size_t for
length like every other range (the only ranges which would be
affected would be ludicrously long anyway, and it would only affect
32-bit programs). But that hasn't happened yet, so iota over longs
and ulongs doesn't behave nicely on 32-bit systems.

Regardless of which way we go, the problem will _eventually_ go away
when 32-bit systems finally die out, but that's likely to take a while.

- Jonathan M Davis


What about adding another overload of iota, say, iotaEx or something
along those lines. All it would do is the following:

auto iotaEx(B, E)(B begin, E end)
{
assert(unsigned(end - begin) <= size_t.max);

static struct Result
{
typeof(iota(begin, end)) payload;

@property size_t length()
{
return cast(size_t)payload.length;
}

alias payload this;
}

return Result(iota(begin, end));
}


It would be far better IMHO to just do a check in iota and throw a
RangeError if the length wouldn't fit in size_t. Having length ever be
anything other than size_t is just going to cause problems with other
ranges. On 32-bit systems, you lose out on the ability to have a range
that covers all values of long or ulong, but that's of very limited
usefulness anyway, and as long as the number of elements is no greater
than size_t.max, it would be fine - which would cover virtually all use
cases. No, it's not perfect, but allowing length to be anything but
size_t just causes bugs - especially in generic code.


Just because you have a range with a ulong length doesn't mean you are 
going to iterate the entire thing.


What we need is this:

standardLength(R)(R r) if (isInputRange!R && hasLength!R)
{
   static if(is(typeof(r.length) == size_t)) return r;
   else
   {
  static struct Result
  {
 R wrapped;
 size_t length() { return to!size_t(wrapped.length); }
 alias wrapped this;
  }
  return Result(r);
   }
}

-Steve

Re: Weird issue with std.range.iota.length

[Jonathan M Davis via Digitalmars-d]

On Friday, 12 February 2016 at 14:36:29 UTC, Meta wrote:
On Friday, 12 February 2016 at 08:11:57 UTC, Jonathan M Davis 
wrote:
What it comes down to is that length should always be size_t. 
That's what it is for arrays, and that's what most code 
expects. Allowing other types just causes trouble for generic 
code. However, in the case of iota with long, if length is 
size_t, then on 32-bit systems, it's possible to have a range 
from iota which is longer than size_t can represent (much as 
it would normally be crazy to have a range that long). So, at 
some point, someone made it so that iota uses ulong for length 
instead of size_t when it's a range of longs or ulongs. It's 
the only thing in Phobos that does, and it causes problems. 
Changing it back to size_t has been discussed but not agreed 
upon. But we're between a rock and a hard place with this one. 
There is no clean solution.


Personally, I'd very much like to see iota just always use 
size_t for length like every other range (the only ranges 
which would be affected would be ludicrously long anyway, and 
it would only affect 32-bit programs). But that hasn't 
happened yet, so iota over longs and ulongs doesn't behave 
nicely on 32-bit systems.


Regardless of which way we go, the problem will _eventually_ 
go away when 32-bit systems finally die out, but that's likely 
to take a while.


- Jonathan M Davis


What about adding another overload of iota, say, iotaEx or 
something along those lines. All it would do is the following:


auto iotaEx(B, E)(B begin, E end)
{
assert(unsigned(end - begin) <= size_t.max);

static struct Result
{
typeof(iota(begin, end)) payload;

@property size_t length()
{
return cast(size_t)payload.length;
}

alias payload this;
}

return Result(iota(begin, end));
}


It would be far better IMHO to just do a check in iota and throw 
a RangeError if the length wouldn't fit in size_t. Having length 
ever be anything other than size_t is just going to cause 
problems with other ranges. On 32-bit systems, you lose out on 
the ability to have a range that covers all values of long or 
ulong, but that's of very limited usefulness anyway, and as long 
as the number of elements is no greater than size_t.max, it would 
be fine - which would cover virtually all use cases. No, it's not 
perfect, but allowing length to be anything but size_t just 
causes bugs - especially in generic code.


- Jonathan M Davis

Re: Weird issue with std.range.iota.length

[Meta via Digitalmars-d]

On Friday, 12 February 2016 at 08:11:57 UTC, Jonathan M Davis 
wrote:
What it comes down to is that length should always be size_t. 
That's what it is for arrays, and that's what most code 
expects. Allowing other types just causes trouble for generic 
code. However, in the case of iota with long, if length is 
size_t, then on 32-bit systems, it's possible to have a range 
from iota which is longer than size_t can represent (much as it 
would normally be crazy to have a range that long). So, at some 
point, someone made it so that iota uses ulong for length 
instead of size_t when it's a range of longs or ulongs. It's 
the only thing in Phobos that does, and it causes problems. 
Changing it back to size_t has been discussed but not agreed 
upon. But we're between a rock and a hard place with this one. 
There is no clean solution.


Personally, I'd very much like to see iota just always use 
size_t for length like every other range (the only ranges which 
would be affected would be ludicrously long anyway, and it 
would only affect 32-bit programs). But that hasn't happened 
yet, so iota over longs and ulongs doesn't behave nicely on 
32-bit systems.


Regardless of which way we go, the problem will _eventually_ go 
away when 32-bit systems finally die out, but that's likely to 
take a while.


- Jonathan M Davis


What about adding another overload of iota, say, iotaEx or 
something along those lines. All it would do is the following:


auto iotaEx(B, E)(B begin, E end)
{
assert(unsigned(end - begin) <= size_t.max);

static struct Result
{
typeof(iota(begin, end)) payload;

@property size_t length()
{
return cast(size_t)payload.length;
}

alias payload this;
}

return Result(iota(begin, end));
}

Re: Visual studio official d support

[Steven Schveighoffer via Digitalmars-d]

On 2/11/16 3:30 PM, Tofu Ninja wrote:

On Tuesday, 9 February 2016 at 22:11:12 UTC, Steven Schveighoffer wrote:

A while ago there was a movement to get d included officially in
visual studio. Just got this email:

 An idea you supported has been closed. Thank you for your feedback.

Message:
This is a great candidate for a language service extension. Moved to
newly created wiki for requested extensions.

https://github.com/Microsoft/vscode/wiki/Requested-Extensions


Also vscode != visual studio


OK, my mistake.

-Steve

Re: Safe cast of arrays

[Steven Schveighoffer via Digitalmars-d]

On 2/10/16 11:01 PM, Chris Wright wrote:

On Wed, 10 Feb 2016 22:39:20 -0500, Steven Schveighoffer wrote:


I think casting a mutable array to any array type is a recipe for memory
issues, no matter what is in the elements. Remember that you are casting
a reference that still has a mutable pointer to it.

@safe should start from a very cautious and overtightened state, and
then we loosen it as we find issues.

As it was done, it has holes, and so when we fix things, code breaks.



I agree with the principle, but it's always safe to read a pointer as if
it were not a pointer, and that's what a cast to an immutable array would
do.



A cast to immutable is a guarantee to the compiler that there is no 
other mutable references that you will use again.


This is not the case here.

A cast to const may be viable. However, I think casting in safe code is 
probably not something to allow. If you need to make something work 
outside the compiler's comfort zone, there's always @trusted.


-Steve

Re: Just because it's a slow Thursday on this forum

[Andrei Alexandrescu via Digitalmars-d]

On 2/11/16 4:38 PM, H. S. Teoh via Digitalmars-d wrote:

(Next
thing you know, newbies will be asking why there's both `print` and
`write` that do the same thing except different.)


That sounds unlikely to me. What would be a few examples/precedents? -- 
Andrei

Re: Babylon JS-like game engine or complete port

[Guillaume Piolat via Digitalmars-d]

On Friday, 12 February 2016 at 11:24:27 UTC, Karabuta wrote:



Was referring to assimp. Which one do you recommended for 
newbie(physics, animation, importing of prebuilt asserts)?


I'd recommend to use SDL2 (or any other windowing library) and 
OpenGL directly with Derelict until you feel comfortable with 
them. And after that think about meshes but the need won't 
necessarily come fast. Rendering mesh is rather involved, ASSIMP 
only does loading.

Re: Weird issue with std.range.iota.length

[Jonathan M Davis via Digitalmars-d]

On Friday, 12 February 2016 at 10:37:31 UTC, tsbockman wrote:

On Friday, 12 February 2016 at 10:34:32 UTC, tsbockman wrote:
Bounds checking. If iota's length is not expressible as a 
`size_t`, convert it to one *safely* using `std.conv.to()`.


Just to clarify - the bit about `std.conv.to()` is just an 
example.


A better solution in this specific case would be to check once, 
while the iota() range is being constructed, whether its length 
is expressible as a `size_t` or not.


Yeah. I think that that's the best approach, but it does 
unfortunately have the downside that you pretty much can't have a 
range over all of the longs or ulongs on 32-bit systems (though 
fortunately that should rarely be an issue, and I don't really 
see a fix for that that doesn't cause problems with length and 
other ranges).


- Jonathan M Davis

Re: Babylon JS-like game engine or complete port

[Karabuta via Digitalmars-d]

On Thursday, 11 February 2016 at 21:59:55 UTC, Guillaume Piolat 
wrote:

On Thursday, 11 February 2016 at 18:42:41 UTC, karabuta wrote:
On Thursday, 11 February 2016 at 10:25:00 UTC, Guillaume 
Piolat wrote:
On Wednesday, 10 February 2016 at 20:07:24 UTC, karabuta 
wrote:

[...]


Javascript world beat us easily in things being easy.
The current D offering is not as integrated but each 
component is pretty much better.


[...]


Does it have a loader? I don't think I saw something like that 
in the docs.


I don't see what you have quoted with [...]


If you are talking about bgfx, there is a dynamic loader and a 
static binding:

http://code.dlang.org/packages/bgfx-d
http://code.dlang.org/packages/derelict-bgfx

For ASSIMP there is a dynamic loader:
http://code.dlang.org/packages/derelict-assimp3




Was referring to assimp. Which one do you recommended for 
newbie(physics, animation, importing of prebuilt asserts)?

Re: Weird issue with std.range.iota.length

[tsbockman via Digitalmars-d]

On Friday, 12 February 2016 at 10:34:32 UTC, tsbockman wrote:
Bounds checking. If iota's length is not expressible as a 
`size_t`, convert it to one *safely* using `std.conv.to()`.


Just to clarify - the bit about `std.conv.to()` is just an 
example.


A better solution in this specific case would be to check once, 
while the iota() range is being constructed, whether its length 
is expressible as a `size_t` or not.

Re: Weird issue with std.range.iota.length

[tsbockman via Digitalmars-d]

On Friday, 12 February 2016 at 08:11:57 UTC, Jonathan M Davis 
wrote:
Regardless of which way we go, the problem will _eventually_ go 
away when 32-bit systems finally die out, but that's likely to 
take a while.


Many micro-controllers will probably remain 32-bit indefinitely; 
64-bit is just a waste of power for most applications. Even if 
you want to ignore non-PC/smartphone CPUs, we'll still have the 
same problem whenever someone gets around to implementing 128-bit 
cent/ucent.


Ultimately, the only real solution to this kind of problem is 
some combination of:


1) Select the right type for the job (don't use `ulong` when you 
really mean `size_t`), and


2) Bounds checking. If iota's length is not expressible as a 
`size_t`, convert it to one *safely* using `std.conv.to()`.


Anything less is just asking for bugs.

Re: Just because it's a slow Thursday on this forum

[ixid via Digitalmars-d]

On Friday, 12 February 2016 at 03:18:47 UTC, Nick Sabalausky 
wrote:
I'd be perfectly happy to have it, particularly if it had a 
less confusing name, but can definitely see it being debatable 
whether it really is Phobos-worthy.


Andrei has previously expressed a desire for a big standard 
library and it is a very good introduction to the language for 
newbies to be able to use print. It's simple and does what they 
expect, writefln("%s %s %s %s", a, b, c, d) is anything but 
simple and contains multiple pitfalls for a new user. This is 
likely the second most commonly used function (or thing someone 
wants a function to do) after main.


Why is the resistance to convenience functions so high in this 
community? If you want people to use D it will need to be both 
powerful and feel light and easy to use. It would seem to be 
better for D to allow convenience functions in Phobos more 
broadly where the use case is common.


There are already many print and write functions, it's not a 
thing that has only one way to do it that will be greatly muddied 
by the addition of another function, can you cite threads from 
other languages where people are confused by a function called 
print that prints?

Re: Wannabe contributor frustrations

[Daniel Murphy via Digitalmars-d]

On 12/02/2016 4:10 AM, Jonathan M Davis wrote:


IIRC, it uses 2.067 and will continue to do so until GDC and LDC have
switched to using the D version of the frontend instead of the older,
C++ version.



Most likely we will stick with 2.067 until GDC and LDC both have 2.068 
releases.  There are a couple bugs in 2.067 that affect building DMD, 
but using it as a host compiler has been fairly painless so far and I 
don't think there's any huge motivation to bump it up.

Re: OT: 'conduct unbecoming of a hacker'

[Abdulhaq via Digitalmars-d]

On Friday, 12 February 2016 at 03:19:52 UTC, Nick Sabalausky 
wrote:

On 02/11/2016 04:54 PM, w0rp wrote:
His article is way too long. It seems like an article about 
whining

about how people whine too much.


It's metawhine! :)


These meta whines get on my nerves, everything was much better in 
Usenet days.

Great new website

[SomeDude via Digitalmars-d]

Hey, just a post to say how cool the new website is. I can browse 
it on an iPhone without issue.

Re: Weird issue with std.range.iota.length

[Jonathan M Davis via Digitalmars-d]

On Friday, 12 February 2016 at 05:51:34 UTC, Meta wrote:

If you try to compile this code, it will currently not work:

foreach (n; iota(1UL, 1000).parallel)
{
//...
}


This is because of how the length is calculated by iota:

auto iota(B, E)(B begin, E end)
if (isIntegral!(CommonType!(B, E)) || isPointer!(CommonType!(B, 
E)))

{
import std.conv : unsigned;

//...

//The return type of length. When either begin or end
//is ulong, then IndexType == ulong
alias IndexType = typeof(unsigned(end - begin));

static struct Result
{
private Value current, pastLast;

//...

@property IndexType length() const
{
return unsigned(pastLast - current);
}
}

return Result(begin, end);
}


And because std.parallelism.TaskPool.defaultWorkUnitSize takes 
a size_t, which with a 32-bit DMD is uint.


What I want to know is, is this considered a bug? If so I will 
submit a pull request to fix it.


What it comes down to is that length should always be size_t. 
That's what it is for arrays, and that's what most code expects. 
Allowing other types just causes trouble for generic code. 
However, in the case of iota with long, if length is size_t, then 
on 32-bit systems, it's possible to have a range from iota which 
is longer than size_t can represent (much as it would normally be 
crazy to have a range that long). So, at some point, someone made 
it so that iota uses ulong for length instead of size_t when it's 
a range of longs or ulongs. It's the only thing in Phobos that 
does, and it causes problems. Changing it back to size_t has been 
discussed but not agreed upon. But we're between a rock and a 
hard place with this one. There is no clean solution.


Personally, I'd very much like to see iota just always use size_t 
for length like every other range (the only ranges which would be 
affected would be ludicrously long anyway, and it would only 
affect 32-bit programs). But that hasn't happened yet, so iota 
over longs and ulongs doesn't behave nicely on 32-bit systems.


Regardless of which way we go, the problem will _eventually_ go 
away when 32-bit systems finally die out, but that's likely to 
take a while.


- Jonathan M Davis