Re: wrong isInputRange design
On Saturday, 3 December 2016 at 17:29:47 UTC, rumbu wrote:
On Saturday, 3 December 2016 at 16:37:21 UTC, Jerry wrote:
On Saturday, 3 December 2016 at 11:52:00 UTC, rumbu wrote:
[...]
[...]
[...]
Is that the exact code? isInputRange checks to see if the type
has "front" defined.
https://github.com/dlang/phobos/blob/v2.072.0/std/range/primitives.d#L162
Also "string" is just an alias of an array,
"immutable(char)[]". So an array should have "front" defined.
Can you post more code?
No, an array should not have "front" defined according to D
language specification. "front" for arrays is defined in
std.range.primitives. That's the problem, I cannot have two
specialized functions (one taking arrays and one taking
ranges), because std.range.primitives hijacks any char array
and transforms it in a range). front, empty, popFront must not
be defined for arrays in the same module as isInputRange.
Exact code is irrelevant, but you're welcome:
string constructor:
https://github.com/rumbu13/numerics/blob/master/src/numerics/fixed.d#L978
range constructor
https://github.com/rumbu13/numerics/blob/master/src/numerics/fixed.d#L1104
This works for me when specialising for input ranges, strings and
arrays.
auto f(T)(T val)
if(isInputRange!T && !isSomeString!T && !isArray!T) {}
auto f(T)(T val)
if(isSomeString!T) {}
auto f(T)(T val)
if(isArray!T && !isSomeString!T)
bye,
lobo
Re: Eclipse OMR project provides a reusable Garbage Collector
On Sat, 03 Dec 2016 15:45:00 +, Chris Wright wrote: > Also, OMR assumes there are no unaligned pointers to GC memory. Also no pointers-to-members, but I have a way around both these problems. Obvious in hindsight. The memory overhead is O(#allocations). The time overhead is O(lg #allocations) per pointer reference per scan. So we won't see numbers as impressive as Java's, unfortunately -- not unless we forbid pointers-to-members and rework array slicing. The details: When I allocate a thing, I insert an entry for the thing into an ordered set of allocated ranges. The entry will be a (begin, end) pointer pair for the allocation. I can get this info because allocations tell me how much to allocate, and OMR's GC gives me a pointer. When inserting that allocated range into the set, I will have to erase any overlapping elements. This shouldn't be difficult -- as long as the no-overlapping-ranges invariant was maintained before this call, I only have to look one element to the left, and it's quick to determine how far to the right I need to look. When scanning an object, we find the pointers in that object as we currently do. Then, instead of enqueueing those pointers to be scanned, we look them up in the allocation set and enqueue the allocation's `begin` pointer. Pretty much what we do today. It's just replacing a little less of the GC than we would prefer.
Re: ThinLTO is there. I think that should settle the final/virtual debate
On Sun, 04 Dec 2016 01:36:50 +, deadalnix wrote: > LTO optimization can devirtualize all function that do not need to be > virtual, and even use profile infos to speculatively devirtualize - aka > JVM grade devirtualization. > > I would love to see this leveraged to finally put to rest the final vs > virtual debate. If we use this tech properly, everything that do not > need to be virtual can be finalized - except across shared object, which > shouldn't be too much of an issue in practice. Unless someone can think of a compelling reason to shorten virtual function tables, anyway -- final member functions don't appear in vtables, but the linker can't remove them from the list.
ThinLTO is there. I think that should settle the final/virtual debate
First, presentation: https://www.youtube.com/watch?v=9OIEZAj243g Some of this is available in LLVM today, and everything presented here will be in 4.0 . The long story short: ThinLTO can do most of what LTO does but with a price that is much closer to the one of a regular build than the one of a classic LTO build. LTO optimization can devirtualize all function that do not need to be virtual, and even use profile infos to speculatively devirtualize - aka JVM grade devirtualization. I would love to see this leveraged to finally put to rest the final vs virtual debate. If we use this tech properly, everything that do not need to be virtual can be finalized - except across shared object, which shouldn't be too much of an issue in practice.
Re: [OT] Feedback on Cooperative Threading Project
On 04/12/2016 5:54 AM, Benjamin Ylvisaker wrote: I am a grad school acquaintance of Andrei's. I contacted him a little while ago asking for feedback on a project that I'm working on. He suggested I post it to the D list. So here I am. The link below is to a draft that is currently under review for PLDI 2017. Please do not share it in a way that might get to a committee member for that conference. I am happy to hear any feedback you have, whether it's critical, enthusiastic or indifferent. http://charcoal-lang.org/charcoal_intro_draft.pdf Thanks, Ben ben.ylvisa...@coloradocollege.edu I would suggest if your ABI requires it to move the callback from last argument to either first or second. Not that I read your paper in much depth. You would get problems with e.g. c's var args with it last.
Re: CTFE Status
On Sunday, 4 December 2016 at 00:14:26 UTC, Stefan Koch wrote:
On Saturday, 3 December 2016 at 23:30:58 UTC, Stefan Koch wrote:
Another bug in continue break handling has appeared.
uint fn() {
for (;;)
{
if (true) break;
return 0;
}
return 1;
}
static assert(fn());
The above code is enough to trigger it.
Apparently the fixup jump for the break is never inserted.
I doubt that this is an off-by-one error again.
It's going to be tricky.
I figured out part of the reason for this bug.
baically it's like this.
uint fn() {
for (;;)
beginBlock:
//causes and unconditonal jump (UC1) after the end of the block
{
if (true) break;
// causes an unconditional jump(UC2) here to the end of the
block
UC2: goto endBlock;
return 0;
endBlock:
UC1: goto BeginBlock;
}
return 1;
}
This can be fixed by setting a state-flag whenever we are going
to process the body of an infinite loop.
If anyone tried at any point to break out of the loop we need to
introduce a conditional that becomes false and allows us to exit.
It's ugly but it will work regardless of nesting-level I
hope.
Re: CTFE Status
On Saturday, 3 December 2016 at 23:30:58 UTC, Stefan Koch wrote:
Another bug in continue break handling has appeared.
uint fn() {
for (;;)
{
if (true) break;
return 0;
}
return 1;
}
static assert(fn());
The above code is enough to trigger it.
Apparently the fixup jump for the break is never inserted.
I doubt that this is an off-by-one error again.
It's going to be tricky.
I figured out part of the reason for this bug.
baically it's like this.
uint fn() {
for (;;)
beginBlock:
//causes and unconditonal jump (UC1) after the end of the block
{
if (true) break;
// causes an unconditional jump(UC2) here to the end of the block
UC2: goto endBlock;
return 0;
endBlock:
UC1: goto BeginBlock;
}
return 1;
}
Re: CTFE Status
Another bug in continue break handling has appeared.
uint fn() {
for (;;)
{
if (true) break;
return 0;
}
return 1;
}
static assert(fn());
The above code is enough to trigger it.
Apparently the fixup jump for the break is never inserted.
I doubt that this is an off-by-one error again.
It's going to be tricky.
Re: CTFE Status
On Saturday, 3 December 2016 at 19:28:52 UTC, Stefan Koch wrote:
On Saturday, 3 December 2016 at 18:16:21 UTC, Stefan Koch wrote:
[...]
StructLiteral regressions fixed.
That one was hard to find.
Basically the for Array arguments interfered with this one.
Arguments are technically the same expression as everything
else.
But in order to support targets that do have native function
calls,
and in order to archive more performance for the interpreter
they have to be treated quite differently from non-argument
expressions.
For now they should work, but it's brittle at best.
The following code now compiles with newCTFE:
struct S
{
uint u1;
uint u2;
}
S makeTenTen()
{
return S(10, 10);
}
S makeS(uint a, uint b)
{
return S(a, b);
}
uint getu2(S s)
{
return s.u2;
}
static assert(getu2(S(10, 14)) == 14);
static assert(makeTenTen() == makeS(10, 10));
Re: D with CygWin
So, just another attempt..
=
import std.stdio;
import std.string;
import core.sys.windows.winbase;
extern(C)
{
alias int function(in char*, int, ...) open_t;
open_t _my_open;
extern uint errno;
}
void main()
{
writefln("Open Library");
auto mod = LoadLibrary("C:\\cygwin\\bin\\cygwin1.dll");
if (mod == null)
{
writefln("Failed load cygwin1.dll");
return;
}
writefln("Get Proc");
_my_open = cast(open_t) GetProcAddress(mod, "open");
if (_my_open == null)
{
writefln("Failed open open symbol");
return;
}
writefln("_open");
int res = _my_open(toStringz("/bin/bash"), 0);
writefln("res=%s errno=%s", res, errno);
}
=
The code compiles without problem, BUT without last 2 lines it
runs, and with last 2 lines it said that executable is corrupted.
Why???
Re: CTFE Status
On Saturday, 3 December 2016 at 18:16:21 UTC, Stefan Koch wrote: [...] StructLiteral regressions fixed. That one was hard to find. Basically the for Array arguments interfered with this one. Arguments are technically the same expression as everything else. But in order to support targets that do have native function calls, and in order to archive more performance for the interpreter they have to be treated quite differently from non-argument expressions. For now they should work, but it's brittle at best.
Re: CTFE Status
On Saturday, 3 December 2016 at 18:00:10 UTC, Stefan Koch wrote:
On Saturday, 3 December 2016 at 02:07:06 UTC, Stefan Koch wrote:
[...]
switch (...) { case ... , ... } (unreliable due to differences
in case-sorting :( )
for-each on strings and arrays.
if (...) { ... } else { ... } (if(__ctfe) && if(!__ctfe) are
special cased and non-ctfe code is removed)
assert(..., "...")
Assigning to array length. (currently without copying on
reallocation :( )
Bounds-Checked :
Array-Indexing
Left-Shift Right-Shift. (always unsigned! :( )
[...]
Forgot to add assert, I forget that I implemented it a few days
ago.
Re: CTFE Status
On Saturday, 3 December 2016 at 02:07:06 UTC, Stefan Koch wrote:
Currently I am working on the ability of returning structs.
I will keep you updated,
This is coming along rather nicely.
StructLiterals as Arguments regressed.
But they did rather dubious stuff in the first place.
So supported types are.
StringLiterals.
Integral Array Literals.
Dynamic int[] and uint[] Arrays.
Structs with int or uint members.
and
StructLiterals with int/uint members (regressed currently.)
64bit integrals (long) do work but only if the interpreter
backend is used.
Supported operations.
All basic math. (+ - * / %)
...++ ...-- +++... --...
while(...) {...}
do ... while(...)
label :
goto label;
break;
break label;
continue;
continue label;
for(...;...;...)
... ? ... : ...
switch (...) { case ... , ... } (unreliable due to differences in
case-sorting :( )
for-each on strings and arrays.
Assigning to array length. (currently without copying on
reallocation :( )
Bounds-Checked :
Array-Indexing
Left-Shift Right-Shift. (always unsigned! :( )
Unsupported :
&& ||
classes (unlikely to be added in the near future)
function calls (plain and closure)
slices
pointers
static arrays.
floating point math (unlikely to be added in the near future)
Re: wrong isInputRange design
On Saturday, 3 December 2016 at 16:37:21 UTC, Jerry wrote: Also "string" is just an alias of an array, "immutable(char)[]". So an array should have "front" defined. Can you post more code? You still have to `import std.range.primitives : front`, even for arrays.
Re: wrong isInputRange design
On Saturday, 3 December 2016 at 16:37:21 UTC, Jerry wrote:
On Saturday, 3 December 2016 at 11:52:00 UTC, rumbu wrote:
import std.range.primitives: isInputRange;
void test(R)(ref R range) if (isInputRange!R)
{
auto c = r.front; //Error: no property 'front' for type
'string'
}
[...]
import std.range.primitives: isInputRange, ElementType;
import std.traits: isSomeChar;
void foo(C)(const(C)[] array) if (isSomeChar!C)
{
//this will be never called
}
void foo(R)(ref R range) if (isInputRange!R &&
isSomeChar!(ElementType!R))
{
//this will be always called
//any call to range.front, range.empty and so on will
result in error
}
foo(somestring)
I expect foo!string to be called instead of foo!Range, because
in my context isInputRange!string should return false.
Instead, my context is hijacked by the definitions spread
along std.range.primitives module.
The workaround I found is to define the second overload like
this:
void foo(R)(ref R range) if (isInputRange!R && !isSomeString!R
&& isSomeChar!(ElementType!R))
Is that the exact code? isInputRange checks to see if the type
has "front" defined.
https://github.com/dlang/phobos/blob/v2.072.0/std/range/primitives.d#L162
Also "string" is just an alias of an array,
"immutable(char)[]". So an array should have "front" defined.
Can you post more code?
No, an array should not have "front" defined according to D
language specification. "front" for arrays is defined in
std.range.primitives. That's the problem, I cannot have two
specialized functions (one taking arrays and one taking ranges),
because std.range.primitives hijacks any char array and
transforms it in a range). front, empty, popFront must not be
defined for arrays in the same module as isInputRange.
Exact code is irrelevant, but you're welcome:
string constructor:
https://github.com/rumbu13/numerics/blob/master/src/numerics/fixed.d#L978
range constructor
https://github.com/rumbu13/numerics/blob/master/src/numerics/fixed.d#L1104
[OT] Feedback on Cooperative Threading Project
I am a grad school acquaintance of Andrei's. I contacted him a little while ago asking for feedback on a project that I'm working on. He suggested I post it to the D list. So here I am. The link below is to a draft that is currently under review for PLDI 2017. Please do not share it in a way that might get to a committee member for that conference. I am happy to hear any feedback you have, whether it's critical, enthusiastic or indifferent. http://charcoal-lang.org/charcoal_intro_draft.pdf Thanks, Ben ben.ylvisa...@coloradocollege.edu
Re: wrong isInputRange design
On Saturday, 3 December 2016 at 11:52:00 UTC, rumbu wrote:
import std.range.primitives: isInputRange;
void test(R)(ref R range) if (isInputRange!R)
{
auto c = r.front; //Error: no property 'front' for type
'string'
}
string s = "some string";
test(s);
The problem is that isInputRange will always return true for
string types (because it will check the availability of front,
popFront, empty through the entire std.range.primitives module.
Since the only thing that was imported is std.range.primitives:
isInputRange, front is not available for strings.
OK, the simple resolution is to import entirely
std.range.primitives, but the source of the problem was
another: I tried to define two overloads, one accepting
strings, and one accepting ranges;
import std.range.primitives: isInputRange, ElementType;
import std.traits: isSomeChar;
void foo(C)(const(C)[] array) if (isSomeChar!C)
{
//this will be never called
}
void foo(R)(ref R range) if (isInputRange!R &&
isSomeChar!(ElementType!R))
{
//this will be always called
//any call to range.front, range.empty and so on will
result in error
}
foo(somestring)
I expect foo!string to be called instead of foo!Range, because
in my context isInputRange!string should return false. Instead,
my context is hijacked by the definitions spread along
std.range.primitives module.
The workaround I found is to define the second overload like
this:
void foo(R)(ref R range) if (isInputRange!R && !isSomeString!R
&& isSomeChar!(ElementType!R))
Is that the exact code? isInputRange checks to see if the type
has "front" defined.
https://github.com/dlang/phobos/blob/v2.072.0/std/range/primitives.d#L162
Also "string" is just an alias of an array, "immutable(char)[]".
So an array should have "front" defined. Can you post more code?
Re: Eclipse OMR project provides a reusable Garbage Collector
On Sat, 03 Dec 2016 15:21:02 +, Chris Wright wrote:
> On Sat, 03 Dec 2016 10:29:02 +, Dibyendu Majumdar wrote:
>> I would imagine that some effort is necessary ... but the payoff is
>> huge isn't it ? To have a robust GC would change the game for D.
>> It said that the OMR expects objects to be allocated on 8-byte aligned
>> memory and it requires one GC byte - which is assumed to be at the
>> start of the allocated object but can be changed if required.
>
> One byte for the GC, or one byte of overhead for the glue layer to use?
>
> Assuming that all structs are padded to word boundaries, one byte would
> allow a 256-word struct. I can create a struct with 10,000 fields, and
> each can be more than one word.
Also, OMR assumes there are no unaligned pointers to GC memory. So if I
wrote the following (perfectly valid) D code, it would crash:
class Foo {
bool[4] bools;
}
auto a = &new Foo().bools[1];
Re: Eclipse OMR project provides a reusable Garbage Collector
On Sat, 03 Dec 2016 10:29:02 +, Dibyendu Majumdar wrote: > I would imagine that some effort is necessary ... but the payoff is huge > isn't it ? To have a robust GC would change the game for D. It's possible that ldc and gdc will switch their runtimes to use OMR, but dmd will not. I might be able to provide an alternate runtime with dub. Will have to investigate. If that works, we might end up splitting the runtime in two -- a lot of the runtime won't need to change, and I'd rather not have the maintenance overhead for typeinfo and unicode handling just to provide a better GC. > I think all GCs need to interact with threads so there has to be some > linkage between the thread sub-system and GC. True. If this project were designed to be used a la carte, it would have a pluggable mechanism for identifying thread stacks and pausing threads. That would make it much easier to port a runtime to OMR, but it would make OMR more complex -- and it's already 775kLOC. It adds more work to my plate, but I can deal. > I saw a slide deck or some docs that explained the requirements to use > OMR GC. Unfortunately I did not save a link and cannot locate this now. > I will post a link if I find it again. Thanks! Watching it now. > It said that the OMR expects objects to be allocated on 8-byte aligned > memory and it requires one GC byte - which is assumed to be at the start > of the allocated object but can be changed if required. One byte for the GC, or one byte of overhead for the glue layer to use? Assuming that all structs are padded to word boundaries, one byte would allow a 256-word struct. I can create a struct with 10,000 fields, and each can be more than one word.
wrong isInputRange design
import std.range.primitives: isInputRange;
void test(R)(ref R range) if (isInputRange!R)
{
auto c = r.front; //Error: no property 'front' for type
'string'
}
string s = "some string";
test(s);
The problem is that isInputRange will always return true for
string types (because it will check the availability of front,
popFront, empty through the entire std.range.primitives module.
Since the only thing that was imported is std.range.primitives:
isInputRange, front is not available for strings.
OK, the simple resolution is to import entirely
std.range.primitives, but the source of the problem was another:
I tried to define two overloads, one accepting strings, and one
accepting ranges;
import std.range.primitives: isInputRange, ElementType;
import std.traits: isSomeChar;
void foo(C)(const(C)[] array) if (isSomeChar!C)
{
//this will be never called
}
void foo(R)(ref R range) if (isInputRange!R &&
isSomeChar!(ElementType!R))
{
//this will be always called
//any call to range.front, range.empty and so on will result
in error
}
foo(somestring)
I expect foo!string to be called instead of foo!Range, because in
my context isInputRange!string should return false. Instead, my
context is hijacked by the definitions spread along
std.range.primitives module.
The workaround I found is to define the second overload like this:
void foo(R)(ref R range) if (isInputRange!R && !isSomeString!R &&
isSomeChar!(ElementType!R))
Re: Eclipse OMR project provides a reusable Garbage Collector
On Saturday, 3 December 2016 at 02:23:13 UTC, Dibyendu Majumdar wrote: IBM has open sourced the J9 Garbage Collector as part of Eclipse OMR project. Perhaps D could use this. It is claimed that the GC implementation is relatively straight forward to reuse. https://github.com/eclipse/omr BTW the .Net Core CLR also appears to be reusable - and maybe even easier to integrate. There is a standalone sample - and the integration interface seems well defined. https://github.com/dotnet/coreclr/tree/master/src/gc
Re: Eclipse OMR project provides a reusable Garbage Collector
On Saturday, 3 December 2016 at 10:29:02 UTC, Dibyendu Majumdar wrote: That said, let's look at the API: omrobjectptr_t OMR_GC_Allocate(OMR_VMThread * omrVMThread, uintptr_t allocationCategory, uintptr_t size, uintptr_t allocateFlags); omrobjectptr_t OMR_GC_AllocateNoGC(OMR_VMThread * omrVMThread, uintptr_t allocationCategory, uintptr_t size, uintptr_t allocateFlagss); omr_error_t OMR_GC_SystemCollect(OMR_VMThread* omrVMThread, uint32_t gcCode); First parameter is a pointer to an OMR thread. I'd have to rewrite core.thread to use OMR's thread system, or maybe I could fake OMR threads with enough effort. That's another barrier. I think all GCs need to interact with threads so there has to be some linkage between the thread sub-system and GC. I saw a slide deck or some docs that explained the requirements to use OMR GC. Unfortunately I did not save a link and cannot locate this now. I will post a link if I find it again. It said that the OMR expects objects to be allocated on 8-byte aligned memory and it requires one GC byte - which is assumed to be at the start of the allocated object but can be changed if required. Here is the video that describes how to use the GC. Says initial integration should take <100 lines of code! GC details are about 30 minutes into the talk. https://developer.ibm.com/open/videos/eclipse-omr-tech-talk/
Re: Eclipse OMR project provides a reusable Garbage Collector
On Saturday, 3 December 2016 at 06:26:22 UTC, Chris Wright wrote: On Sat, 03 Dec 2016 02:23:13 +, Dibyendu Majumdar wrote: IBM has open sourced the J9 Garbage Collector as part of Eclipse OMR project. Perhaps D could use this. It is claimed that the GC implementation is relatively straight forward to reuse. That's awesome! Unfortunately, it looks like incorporating it would take a fair bit of effort, and upstreaming it will not happen. I'm going to take a deeper look this weekend, but it's not looking good so far. I would imagine that some effort is necessary ... but the payoff is huge isn't it ? To have a robust GC would change the game for D. That said, let's look at the API: omrobjectptr_t OMR_GC_Allocate(OMR_VMThread * omrVMThread, uintptr_t allocationCategory, uintptr_t size, uintptr_t allocateFlags); omrobjectptr_t OMR_GC_AllocateNoGC(OMR_VMThread * omrVMThread, uintptr_t allocationCategory, uintptr_t size, uintptr_t allocateFlagss); omr_error_t OMR_GC_SystemCollect(OMR_VMThread* omrVMThread, uint32_t gcCode); First parameter is a pointer to an OMR thread. I'd have to rewrite core.thread to use OMR's thread system, or maybe I could fake OMR threads with enough effort. That's another barrier. I think all GCs need to interact with threads so there has to be some linkage between the thread sub-system and GC. I saw a slide deck or some docs that explained the requirements to use OMR GC. Unfortunately I did not save a link and cannot locate this now. I will post a link if I find it again. It said that the OMR expects objects to be allocated on 8-byte aligned memory and it requires one GC byte - which is assumed to be at the start of the allocated object but can be changed if required. Regards Dibyendu
An interesting article on GC interaction with OS scheduler
This article presents an insight into a peculiar interaction between Java GC and Linux scheduler on the background of Linux containerization technology - the CGROUP. IMHO the CGROUP only amplifies the discussed impact, that exists anyway. A short summary: Linux scheduler allocates a time slice for an application to run. When the time slice is depleted (i.e. all the application threads together have eaten up all the time resource), the application is stopped until next time slice is allocated. Many-threaded GC is able to deplete the time slice very quickly, causing Stop-The-World effect, even in generally concurrent GC design. https://engineering.linkedin.com/blog/2016/11/application-pauses-when-running-jvm-inside-linux-control-groups
