Re: Testing array ptr for offset 0...
On Thursday, 26 May 2016 at 21:13:14 UTC, Era Scarecrow wrote: To do what I want currently it's something like... enum Size = 1024, Other = 128; Data[Size][Other] staticarray; //stack allocation Data[][] sliced = staticarray[]; scan(sliced, condition); void scan(ref Data[][] data, Condition cond) { int lengths[Size]; foreach(i; ...) { if (cond) data[i][lengths[i]++] = ... } //cleanup/shrink foreach(i, l; lengths) data[i] = data[i][0 .. l]; } Like this: static void rawlength(ref Data[] r, size_t len) { r = r.ptr[0..len]; } void scan(ref Data[][] data, Condition cond) { foreach(i; ...) { if (cond) data[i].rawlength = ... } }
Re: Testing array ptr for offset 0...
On Friday, 27 May 2016 at 09:18:47 UTC, Marc Schütz wrote: You can do that with arrays, too, without causing allocations: assert(slice.length < static_array.length); slice = slice.ptr[0 .. slice.length+1]; Of course that's unsafe, but your pointer magic certainly is, too. Initial impressions suggest it's actually slower. The multiple dereferences to do it since it's a 2d array...
Re: Testing array ptr for offset 0...
On Thursday, 26 May 2016 at 22:47:02 UTC, Era Scarecrow wrote: On Thursday, 26 May 2016 at 22:15:42 UTC, ag0aep6g wrote: Sorry, I'm still lost. Why can't you do whatever you're doing in opOpAssign directly there, or in a free function? Does the pseudo-array contain any additional data? Would a wrapper around a built-in array not work? Well a few things at work. First the array is originally a static array, so I pass it as a slice; But if you pass it as a slice you get the entire length, and modifying the length requires either changing length or appending, both which may cause allocation (something I want to avoid). By making my own compatible structure I can manage the length and pointer directly and precisely. You can do that with arrays, too, without causing allocations: assert(slice.length < static_array.length); slice = slice.ptr[0 .. slice.length+1]; Of course that's unsafe, but your pointer magic certainly is, too.
Re: Testing array ptr for offset 0...
On Thursday, 26 May 2016 at 22:15:42 UTC, ag0aep6g wrote: On 05/26/2016 11:13 PM, Era Scarecrow wrote: void scan(ref Data[][] data, Condition cond) { foreach(i; ...) { if (cond) data[i] ~= ... } } Sorry, I'm still lost. Why can't you do whatever you're doing in opOpAssign directly there, or in a free function? Does the pseudo-array contain any additional data? Would a wrapper around a built-in array not work? Well a few things at work. First the array is originally a static array, so I pass it as a slice; But if you pass it as a slice you get the entire length, and modifying the length requires either changing length or appending, both which may cause allocation (something I want to avoid). By making my own compatible structure I can manage the length and pointer directly and precisely. Second I would prefer the length to start at 0 and grow UP TO the size of the static array. If it was just a single array this wouldn't be an issue, but I'm working with a couple hundred of these. Plus I don't want to add more data to the memory or structure than necessary to treat it as an array after the fact. I shouldn't have to. Truthfully the same result can be done with shrinking it afterwards, but I was trying to see if I could make it any faster by managing the pointer and length directly and avoiding the large temporaries. So far any speed gains are so minimal it's worth dropping. Oh, I see. Yeah, the answer is garbage. Seeing the spec section that Adam D. Ruppe linked, the length field actually comes first. That CTFE errors out when you move the pointer to a variable screams that something is wrong. The code seems to be seriously misinterpreted in CTFE. I've filed an issue: https://issues.dlang.org/show_bug.cgi?id=16081 Glad I found a bug! :) And all because [].ptr.offsetof wasn't an option... I just wanted to guarantee it was the correct order (in case it gets changed in the future). But ultimately it isn't worth pursuing.
Re: Testing array ptr for offset 0...
On 05/26/2016 11:13 PM, Era Scarecrow wrote: By adding a struct overload for opOpAssign I can shrink it all down to this, and avoid lengths entirely... As such internally the length starts at 0, and checks are in place to ensure the bounds are never exceeded. void scan(ref Data[][] data, Condition cond) { foreach(i; ...) { if (cond) data[i] ~= ... } } Sorry, I'm still lost. Why can't you do whatever you're doing in opOpAssign directly there, or in a free function? Does the pseudo-array contain any additional data? Would a wrapper around a built-in array not work? [...] The answer is wrong _because_ the code blows up. I inverted the check to check if length is at offset 0 since (the length can only be 1 and a better test), however it _still_ gives the wrong answer. Curiously CTFE is fine with this casting as long as the pointer stays in that one spot. Oh, I see. Yeah, the answer is garbage. Seeing the spec section that Adam D. Ruppe linked, the length field actually comes first. That CTFE errors out when you move the pointer to a variable screams that something is wrong. The code seems to be seriously misinterpreted in CTFE. enum x = () { size_t[] arr = [13]; return *(cast(size_t*) &arr); } (); pragma(msg, x); That prints "13LU". Apparently &arr is mistaken for arr here. I've filed an issue: https://issues.dlang.org/show_bug.cgi?id=16081
Re: Testing array ptr for offset 0...
On Thursday, 26 May 2016 at 12:33:31 UTC, Adam D. Ruppe wrote: On Thursday, 26 May 2016 at 12:30:30 UTC, Alex Parrill wrote: The line "not having to make another array to keep track of lengths and then shorten them" is fairly vague. "Shortening" an array via slicing is basically free (it's just some integer arithmetic), but I'm not sure if that's what you meant. Indeed, creating a slice from a static array, another slice, or a pointer is free too... But managing the lengths separately and then re-slicing it afterwards is not free. It does have a low constant cost though; but can I remove that cost entirely (including the foreach loop overhead)? There's also the annoyance that on windows machines the stack size seems to be fairly small. Sometimes in my tests the program will crash before it starts because the requested size of a static array on the stack is too big. This means if I have a big stack allocation already, I want to try and avoid further allocations if I can avoid it. byte[1024*1024] buffer; //on my machine, it just crashes when this is allocated.
Re: Testing array ptr for offset 0...
On Thursday, 26 May 2016 at 11:47:13 UTC, ag0aep6g wrote: I don't follow. Why can't you use a built-in array? What can you do with the stand-in that you can't do with the array itself? I can do the job with the built-in arrays, however the need for another temporary array to keep track of lengths so I can slice them after the fact is what i want to avoid. I'm also slicing static arrays. I'm trying to see if this code would be more efficient or not (if it is, it's not by much, which I'm sure has to do with the cache more than anything else). To do what I want currently it's something like... enum Size = 1024, Other = 128; Data[Size][Other] staticarray; //stack allocation Data[][] sliced = staticarray[]; scan(sliced, condition); void scan(ref Data[][] data, Condition cond) { int lengths[Size]; foreach(i; ...) { if (cond) data[i][lengths[i]++] = ... } //cleanup/shrink foreach(i, l; lengths) data[i] = data[i][0 .. l]; } By adding a struct overload for opOpAssign I can shrink it all down to this, and avoid lengths entirely... As such internally the length starts at 0, and checks are in place to ensure the bounds are never exceeded. void scan(ref Data[][] data, Condition cond) { foreach(i; ...) { if (cond) data[i] ~= ... } } Seems a bit back and forth, but as far as I can see it works as you want. Why do you think the answer it gives is wrong? What answer does it give you? The answer is wrong _because_ the code blows up. I inverted the check to check if length is at offset 0 since (the length can only be 1 and a better test), however it _still_ gives the wrong answer. Curiously CTFE is fine with this casting as long as the pointer stays in that one spot. template isArrayLengthOffsetZero() { bool check() { size_t[] arr = new size_t[1]; return *(cast(size_t*) &arr) == 1; } enum isArrayLengthOffsetZero = check(); //wrong answer // enum isArrayLengthOffsetZero = !check(); //right answer but logically wrong } //test struct struct S(T) { static if (isArrayLengthOffsetZero!()) { size_t length; T* ptr; } else { T* ptr; size_t length; } this(T* p) { ptr = p; } //opOpAssign } unittest { import std.stdio; //union to get our overlapping data types union X { string str; S!(immutable char) s_str;} X x; x.s_str = S!(immutable char)("t".ptr); //only ptr assigned //writeln(x.str); //blows up writeln(x.str.length); //not 0... assert(x.str.length == 0); //fails } Most likely the internal array structure CTFE uses for the array is inverted, and if that's the case the proper answer to how to get the simple offset of ptr (or length) is a mess. Forcing the inverted answer works, but this is a lot of overhead to try and make an algorithm faster. Speedups are minimal at best, so I'm about ready to drop this line of thought.
Re: Testing array ptr for offset 0...
On Thursday, 26 May 2016 at 07:51:46 UTC, Era Scarecrow wrote: If the ptr is at offset 0, we declare it first, otherwise second, simple... Except this fails since "no property 'offsetof' for type 'void*'". SO... I make a template to test for it instead. Built-in arrays are kinda magical and thus their members aren't ordinary members you can inspect like that. return *(cast(size_t*) &arr) != 1; And I'm pretty sure that cast is illegal in CTFE. Perhaps the bug here is that dmd isn't throwing an error when it is supposed to... The value being compared gives 0 when I try it, which wouldn't make sense for either .length or .ptr. Am i going about this the wrong way? I don't think there's a need for a test because the ABI spec says length is always defined to be at offset zero anyway: http://dlang.org/spec/abi.html#arrays If you want to test it, a runtime assert would probably be enough, at least then it won't do the wrong thing and your message can tell the user to recompile. Runtime assert is prolly better anyway just in case ctfe disagrees with the execution platform.
Re: Testing array ptr for offset 0...
On Thursday, 26 May 2016 at 12:30:30 UTC, Alex Parrill wrote: The line "not having to make another array to keep track of lengths and then shorten them" is fairly vague. "Shortening" an array via slicing is basically free (it's just some integer arithmetic), but I'm not sure if that's what you meant. Indeed, creating a slice from a static array, another slice, or a pointer is free too...
Re: Testing array ptr for offset 0...
On Thursday, 26 May 2016 at 07:51:46 UTC, Era Scarecrow wrote: ... This smells like an XY problem [0]. Why exactly do you need the internal layout of the array structure? The line "not having to make another array to keep track of lengths and then shorten them" is fairly vague. "Shortening" an array via slicing is basically free (it's just some integer arithmetic), but I'm not sure if that's what you meant. [0]: http://meta.stackexchange.com/questions/66377/what-is-the-xy-problem
Re: Testing array ptr for offset 0...
On 05/26/2016 09:51 AM, Era Scarecrow wrote: So I'm experimenting and want to override how arrays are managed for a short duration, this is for optimization of not having to make another array to keep track of lengths and then shorten them when i can in fact just manage it myself *very* briefly. So, I need to make sure the structure is compatible with the built-in array structure (since the remainder of the calls are via normal convention and no allocation/resizing is done except by progressively smaller slices, so it should be compatible as long as it's a valid pointer). I don't follow. Why can't you use a built-in array? What can you do with the stand-in that you can't do with the array itself? [...] template isArrayPtrOffsetZero() { auto check() { static assert([].sizeof == (size_t.sizeof*2)); size_t[] arr = new size_t[1]; //length == 1. return *(cast(size_t*) &arr) != 1; } enum isArrayPtrOffsetZero = check(); } [...] Except this blows up and always gives the wrong answer... ==1 length test reverses it but is now the wrong test, and changing it to isArrayLengthOffsetZero will in turn give the wrong answer again... Here's how I understand that code: isArrayPtrOffsetZero is supposed to check if the offset of ptr is 0. You do that by setting the length of some array to 1, and then checking if the first half of the array structure is not 1. If it's 1, then length's offset is 0, so ptr's offset can't be 0. If it's not 1, then length's offset isn't 0, so ptr's offset must be 0. Seems a bit back and forth, but as far as I can see it works as you want. Why do you think the answer it gives is wrong? What answer does it give you?
Re: Testing array ptr for offset 0...
On Thursday, 26 May 2016 at 09:16:48 UTC, Kagamin wrote: try this: struct X { byte[] data; alias data this; } This doesn't help me with what I want, and effectively does nothing as far as I can tell. Although testing is showing managing the pointer & length directly is a little slower for whatever reason.
Re: Testing array ptr for offset 0...
try this: struct X { byte[] data; alias data this; }
Testing array ptr for offset 0...
So I'm experimenting and want to override how arrays are managed for a short duration, this is for optimization of not having to make another array to keep track of lengths and then shorten them when i can in fact just manage it myself *very* briefly. So, I need to make sure the structure is compatible with the built-in array structure (since the remainder of the calls are via normal convention and no allocation/resizing is done except by progressively smaller slices, so it should be compatible as long as it's a valid pointer). But i seem to have an issue trying to test if ptr is at offset 0 or not. Having the wrong order would be catastrophic, and if the source changes (for whatever reason) i don't want to have to fix it. So... the obvious test: static if ([].ptr.offsetof == 0) If the ptr is at offset 0, we declare it first, otherwise second, simple... Except this fails since "no property 'offsetof' for type 'void*'". SO... I make a template to test for it instead. template isArrayPtrOffsetZero() { auto check() { static assert([].sizeof == (size_t.sizeof*2)); size_t[] arr = new size_t[1]; //length == 1. return *(cast(size_t*) &arr) != 1; } enum isArrayPtrOffsetZero = check(); } struct X { static if (isArrayPtrOffsetZero!()) { int* ptr; size_t length; } else { ... } } Except this blows up and always gives the wrong answer... ==1 length test reverses it but is now the wrong test, and changing it to isArrayLengthOffsetZero will in turn give the wrong answer again... Am i going about this the wrong way?