Re: forgetting -betterC means no runtime bounds checking?
On Thursday, 5 January 2023 at 09:10:00 UTC, areYouSureAboutThat wrote: My question is: why is there no bounds checking occurring if I forget to use -betterC? module test; extern(C) void main() Note that whether bounds checking is performed depends on [compiler switches](https://dlang.org/dmd-windows.html#switch-boundscheck), and that there is a mode where `@safe` functions are checked, but `@system` functions (which your `main` appears to be) are not. This mode is the default for `-release` builds: ``` -boundscheck=[on|safeonly|off ] Controls if bounds checking is enabled. on: Bounds checks are enabled for all code. This is the default. safeonly: Bounds checks are enabled only in @safe code. This is the default for -release builds. off: Bounds checks are disabled completely (even in @safe code). This option should be used with caution and as a last resort to improve performance. Confirm turning off @safe bounds checks is worthwhile by benchmarking. ``` So, if you want bounds checking, make sure you either have `-boundscheck=on`, or use an `@safe` function together with `-boundscheck=safeonly`.
Re: How to avoid variable capturing in `foreach` loop with lambdas?
On Thursday, 5 January 2023 at 13:05:46 UTC, thebluepandabear
wrote:
Update some time later: the only way (oof!) around this seems
to be using a `static foreach` with arrays:
```D
Button[3] b;
static foreach (indx, BoardSize boardSize; arr) {
b[indx] = new Button();
b[indx].text = format("%sx%s", boardSize[0], boardSize[1]);
b[indx].onButtonClick = {
eventHandler.settingsWindow_onBoardSizeButtonClick(boardSize);
};
_boardSizeRow.addChild(b[indx]);
}
```
This is semantically equivalent to copying and pasting the loop
body `arr.length` number of times, substituting the expression
`arr[indx]` for `boardSize`, and the literal value of `indx` for
`indx`.
Unlike with a runtime (non-`static`) loop, the event handler
closure does not capture a reference to `boardSize` or `indx`,
because they don't actually exist at runtime and so cannot be
referenced. Instead, `arr.length` different event handler
functions are created at compile time, with each having the
appropriate `indx` literal substituted.
So, while it does work as long as `arr.length` is known at
compile time, it will cause a ton of needless code bloat unless
`arr.length` is very small.
Any other ways of fixing this annoying issue?
Yes, see [my earlier
reply](https://forum.dlang.org/post/fwnccjunvnffmtaho...@forum.dlang.org) for an explanation of how to get your original run time loop working as intended.
Re: How to avoid variable capturing in `foreach` loop with lambdas?
On Thursday, 5 January 2023 at 11:55:33 UTC, thebluepandabear
wrote:
```D
foreach (BoardSize boardSize; arr) {
Button button = new Button();
button.text = format("%sx%s", boardSize[0], boardSize[1]);
button.onButtonClick = {
eventHandler.settingsWindow_onBoardSizeButtonClick(boardSize);
};
button.onButtonClick();
_boardSizeRow.addChild(button);
}
```
Running this code, I had expected that everything would work
fine. Unfortunately upon running the code, tapping each of the
buttons returned only the largest `boardSize` value, the one
which is gets iterated last.
The problem is twofold:
1. Closures in D capture their environment by reference.
2. D (incorrectly, in my opinion) considers loop-local variables
to have the same identity across each iteration of the loop
within a single function call.
So, `boardSize` in your event handler is a reference to a single
variable whose value is overwritten on each iteration of the
loop. As the event handlers are (I presume) never called until
after the loop has terminated, the only value they will ever see
is whichever was set by the final iteration of the loop in that
function call.
There are at least two possible solutions:
1. Use a `struct` to explicitly capture `boardSize` by value,
instead of by reference:
```D
static struct ClickHandler {
// If eventHandler is not a global variable of some sort, add
another field for it:
BoardSize iteration_boardSize;
this(BoardSize iteration_boardSize) {
this.iteration_boardSize = iteration_boardSize;
}
void opCall() {
eventHandler.settingsWindow_onBoardSizeButtonClick(iteration_boardSize);
}
}
foreach (BoardSize loop_boardSize; arr) {
Button button = new Button();
button.text = format("%sx%s", loop_boardSize[0],
loop_boardSize[1]);
button.onButtonClick = &(new
ClickHandler(loop_boardSize)).opCall;
button.onButtonClick();
_boardSizeRow.addChild(button);
}
```
2. Use a nested function call with a `boardSize` parameter to
create a copy of `boardSize` with a unique identity on each
iteration of the loop:
```D
foreach (BoardSize loop_boardSize; arr) {
Button button = new Button();
button.text = format("%sx%s", loop_boardSize[0],
loop_boardSize[1]);
button.onButtonClick = (BoardSize iteration_boardSize) {
return {
eventHandler.settingsWindow_onBoardSizeButtonClick(iteration_boardSize);
};
}(loop_boardSize);
button.onButtonClick();
_boardSizeRow.addChild(button);
}
```
These two solutions should compile to approximately the same
runtime code, with optimizations enabled. So, it's really down to
personal preference; the former is more explicit about what the
computer is to do, while the latter is more concise.
Re: Explicit cast to @system?
On Saturday, 8 October 2022 at 23:06:13 UTC, Anonymouse wrote:
I have some nested templated code that takes function pointers.
In many cases I pass it functions of identical signatures,
except some are `@safe` and others are `@system`. In those
cases the templates end up getting instantiated twice. I don't
care about the `@safe`-ness and I'd really like to just have
them all treated as `@system`, with one instantiation per
unique signature.
...
But surely there has to be a better way?
You might be templating more information than necessary. In your
example `foo` doesn't need to be a template at all:
```D
void foo(void function() @system fun) {
pragma(msg, typeof(fun).stringof);
}
```
If your real code needs to template the return type and
parameters of `fun`, for example, consider just templating those
instead of the whole function pointer type:
```D
void foo(R, P...)(R function(P) @system fun) {
pragma(msg, typeof(fun).stringof);
}
```
(Things do get awkward with `ref` and `out`, though, because D
considers them to be part of the function's type rather than part
of the parameter or return types. `ref` is the bane of my D
meta-programming existence.)
Re: Is `void` the correct way to say "do not initialize this variable"?
On Sunday, 2 October 2022 at 23:30:16 UTC, ryuukk_ wrote:
```D
MyStruct test = void;
```
Does this guarantee that the compiler will not initialize it?
It's more of a request, than a guarantee. For example, `= void`
may be ignored for the fields of `struct`s and `class`es:
```D
struct ABC {
char a = 'a';
char b = void;
char c = 'c';
}
void main() @safe {
import core.lifetime : emplace;
import std.stdio : write, writeln;
ABC abc = { a: 'x', b: 'y', c: 'z' };
emplace();
write(`a: '`, abc.a, `', b: '`);
if(abc.b != 0)
write(abc.b);
else
write(`\0`);
writeln(`', c: '`, abc.c, `'`);
}
```
If the `= void` actually prevented initialization of `b`, the
above would print:
```
a: 'a', b: 'y', c: 'c'
```
However, it actually prints this instead on all D compilers with
which I tested:
```
a: 'a', b: '\0', c: 'c'
```
This is because it is considered needlessly complicated - and, at
least for smallish types, possibly actually slower - to support
uninitialized gaps when blitting `.init` values.
Does it work with static arrays of struct too?
Yes.
Re: Is `void` the correct way to say "do not initialize this variable"?
On Sunday, 2 October 2022 at 23:45:45 UTC, drug007 wrote: It works but not as someone could expect. In case of ```D Foo[2] arr = void; ``` `arr` value is not defined, it is not an initialized array of uninitialized elements like you want, it is just uninitialized array. This is incorrect. It is not possible to declare an uninitialized static array variable in D; only the elements are affected by `= void`. The meta data of a static array like `Foo[2] arr` (`.ptr` and `.length`) is determined statically at compile time and inserted where needed into the generated code. It is not stored in mutable memory the way a dynamic array/slice's meta data is, and does not need to be initialized at run time. By contrast, it **is** possible to declare a completely uninitialized dynamic array, or to just leave its elements uninitialized: ```D // Meta data is not initialized, and no elements are allocated. // This has no static array equivalent: int[] arrA = void; // Meta data is initialized, and elements are allocated but not initialized. // This is the dynamic equivalent of the static: // int[2] arr = void; int[] arrB = uninitializedArray!(int[])(2); ```
Re: What are best practices around toString?
On Saturday, 1 October 2022 at 10:02:34 UTC, Salih Dincer wrote: On Saturday, 1 October 2022 at 08:26:43 UTC, tsbockman wrote: `StringBuilder` is a utility shared across the entire project: Appender not good enough; at least in terms of allocating memory and accumulating a string? `Appender` is a legitimate option, but unless it is provided with a good estimate of the final length at the beginning, it will allocate several times for a longer string, and the final buffer will be, on average, 50% larger than needed. Neither of these things is a major problem, but `StringBuilder` is only a few lines of code to perfectly minimize allocation, so why not?
Re: What are best practices around toString?
On Friday, 30 September 2022 at 13:11:56 UTC, christian.koestlin
wrote:
Dear Dlang experts,
up until now I was perfectly happy with implementing
`(override) string toString() const` or something to get nicely
formatted (mostly debug) output for my structs, classes and
exceptions.
Human beings read extremely slowly compared to how quickly the GC
can allocate and free `string`s as needed, so there is no need to
complicate your code with more text formatting strategies unless
you want to generate this debug output far faster than a human
can actually read it.
But recently I stumbled upon
https://wiki.dlang.org/Defining_custom_print_format_specifiers
and additionally
https://github.com/dlang/dmd/blob/4ff1eec2ce7d990dcd58e5b641ef3d0a1676b9bb/druntime/src/object.d#L2637 which at first sight is great, because it provides the same customization of an objects representation with less memory allocations.
When grepping through phobos, there are a bunch of "different"
signatures implemented for this, e.g.
```d
...
phobos/std/typecons.d:void toString(DG)(scope DG sink)
const
...
phobos/std/typecons.d:void toString(DG, Char)(scope DG
sink, scope const ref FormatSpec!Char fmt) const
...
phobos/std/typecons.d:void toString()(scope void
delegate(const(char)[]) sink, scope const ref FormatSpec!char
fmt)
...
phobos/std/sumtype.d:void toString(this This, Sink,
Char)(ref Sink sink, const ref FormatSpec!Char fmt);
...
```
to just show a few.
The `FormatSpec` parameter only belongs there if you're actually
going to do something useful with it in your `toString`
implementation. Even if you are going to use it, you should
probably still provide a convenience overload with a default
specifier.
Furthermore, when one works with instances of struct, objects
or exceptions a `aInstance.toString()` does not "work" when one
only implements the sink interface (which is to be expected),
whereas a `std.conv.to!string` or a formatted write with `%s`
always works (no matter what was used to implement the
toString).
I generally do something like this:
```D
struct A {
string message;
int enthusiasm;
void toString(DG)(scope DG sink) scope const @safe
if(is(DG : void delegate(scope const(char[])) @safe)
|| is(DG : void function(scope const(char[])) @safe))
{
import std.format : formattedWrite;
sink(message);
sink(" x ");
formattedWrite!"%d"(sink, enthusiasm);
sink("!");
}
string toString() scope const pure @safe {
StringBuilder builder;
toString(&(builder.opCall)); // Find the exact string
length.
builder.allocate();
toString(&(builder.opCall)); // Actually write the chars.
return builder.finish();
}
}
```
So, the first `toString` overload defines how to format the value
to text, while the second overload does memory management and
forwards the formatting work to the first.
`StringBuilder` is a utility shared across the entire project:
```D
struct StringBuilder {
private:
char[] buffer;
size_t next;
public:
void opCall(scope const(char[]) str) scope pure @safe nothrow
@nogc {
const curr = next;
next += str.length;
if(buffer !is null)
buffer[curr .. next] = str[];
}
void allocate() scope pure @safe nothrow {
buffer = new char[next];
next = 0;
}
void allocate(const(size_t) maxLength) scope pure @safe
nothrow {
buffer = new char[maxLength];
next = 0;
}
string finish() pure @trusted nothrow @nogc {
assert(buffer !is null);
string ret = cast(immutable) buffer[0 .. next];
buffer = null;
next = 0;
return ret;
}
}
```
The first formatting pass to find the required buffer length can
be skipped if you can somehow pre-calculate the maximum possible
length, or if you prefer the common strategy of repeatedly
re-allocating the buffer with exponentially increasing size used
by the likes of `std.array.Appender`. Since the API for
`toString` remains the same regardless, you are free to choose
the best strategy for each type.
Re: How to do alligned allocation?
On Saturday, 1 October 2022 at 01:37:00 UTC, Steven Schveighoffer wrote: On 9/30/22 11:57 AM, Quirin Schroll wrote: Also, is the alignment of any type guaranteed to be a power of 2? In practice, it's not necessarily a power of 2, but it's *at least* 16 bytes. **Types** always require some power of 2 alignment (on any sensible platform, anyway), and it is usually *less* than 16 bytes - typically `size_t.sizeof`. The fact that the current GC implementation apparently has a minimum block size of 16 bytes, and that minimum size blocks are always size-aligned, is not guaranteed by the public API and *should not be* when requesting memory for something that the type system says only requires an alignment of `void.alignof == 1`. D and C both have formal ways to communicate alignment requirements to the allocator; people should use them and not constrain all future D GC development to conform to undocumented details of the current implementation. In general there are very few types (maybe vectors?) that need alignment more than 16 bytes. 256 bit SIMD (AVX/AVX2) and 512 bit SIMD (AVX512) `__vector`s should be `.sizeof` aligned (32 and 64 bytes, respectively). Memory used for inter-thread communication (such as mutexes) may perform significantly better if cache line aligned (typically 64 bytes, but CPU dependent). I don't know any other examples off the top of my head. The list of bit sizes is currently here: I'm pretty sure those are in **bytes** not **bits**. https://github.com/dlang/dmd/blob/82870e890f6f0e0dca3e8f0032a7819416319124/druntime/src/core/internal/gc/impl/conservative/gc.d#L1392-L1414 That's not a list of alignments, it is block sizes for some GC memory pools. The alignment of each block depends on the alignment of its pool, not just its size. It's not immediately obvious from the context, but I suspect the pools are actually page aligned, which would mean that the non power of 2 sized blocks are **not** consistently aligned to their own sizes. Regardless, it's not part of the public API, so it could change without warning.
Re: How to do alligned allocation?
On Saturday, 1 October 2022 at 00:32:28 UTC, tsbockman wrote:
alias Chunk = AliasSeq!(ubyte, ushort, uint, ulong,
Chunk16)[shift];
Oops, I forgot that `ulong.alignof` is platform dependent. It's
probably best to just go ahead and explicitly specify the
alignment for all `Chunk` types:
```D
private template Aligned(size_t alignment)
if(1 <= alignment && isPowerOf2(alignment))
{
enum int shift = bsr(alignment);
enum size_t mask = alignment - 1;
static if(alignment <= 16) {
enum chunkShift = shift, chunkMask = mask;
align(alignment) struct Chunk {
void[alignment] data;
}
} else {
enum chunkShift = Aligned!(16).shift, chunkMask =
Aligned!(16).mask;
alias Chunk = Aligned!(16).Chunk;
}
}
```
(This also eliminates the `std.meta : AliasSeq` dependency.)
Re: How to do alligned allocation?
On Friday, 30 September 2022 at 15:57:22 UTC, Quirin Schroll
wrote:
When I do `new void[](n)`, is that buffer allocated with an
alignment of 1 or what are the guarantees?
It is guaranteed an alignment of at least 1 because `void.alignof
== 1` (and because that is the lowest possible integer
alignment). When I last checked, `new T` guaranteed a minimum
alignment of `min(T.alignof, 16)`, meaning that all basic scalar
types (`int`, `double`, pointers, etc.), and SIMD `__vector`s up
to 128 bits will be correctly aligned, while 256 bit (for
example, AVX's `__vector(double[4])`) and 512 bit (AVX512) types
might not be.
Arrays and aggregate types (`struct`s and `class`es) by default
use the maximum alignment required by any of their elements or
fields (including hidden fields, like `__vptr` for `class`es).
This can be overridden manually using the `align` attribute,
which must be applied to the aggregate type as a whole. (Applying
`align` to an individual field does something else.)
How can I set an alignment?
If the desired alignment is `<= 16`, you can specify a type with
that `.alignof`.
However, if you may need higher alignment than the maximum
guaranteed to be available from the allocator, or you are not
writing strongly typed code to begin with, as implied by your use
of `void[]`, you can just align the allocation yourself:
```D
void[] newAligned(const(size_t) alignment)(const(size_t) size)
pure @trusted nothrow
if(1 <= alignment && isPowerOf2(alignment))
{
enum alignMask = alignment - 1;
void[] ret = new void[size + alignMask];
const misalign = (cast(size_t) ret.ptr) & alignMask;
const offset = (alignment - misalign) & alignMask;
ret = ret[offset .. offset + size];
return ret;
}
```
However, aligning memory outside of the allocator itself like
this does waste up to `alignment - 1` bytes per allocation, so
it's best to use as much of the allocator's internal alignment
capability as possible:
```D
import core.bitop : bsr;
import std.math : isPowerOf2;
import std.meta : AliasSeq;
void[] newAligned(const(size_t) alignment)(const(size_t) size)
pure @trusted nothrow
if(1 <= alignment && isPowerOf2(alignment))
{
alias Aligned = .Aligned!alignment;
void[] ret = new Aligned.Chunk[(size + Aligned.mask) >>
Aligned.chunkShift];
static if(Aligned.Chunk.alignof == alignment)
enum size_t offset = 0;
else {
const misalign = (cast(size_t) ret.ptr) & Aligned.mask;
const offset = (alignment - misalign) & Aligned.mask;
}
ret = ret[offset .. offset + size];
return ret;
}
private {
align(16) struct Chunk16 {
void[16] data;
}
template Aligned(size_t alignment)
if(1 <= alignment && isPowerOf2(alignment))
{
enum int shift = bsr(alignment);
enum size_t mask = alignment - 1;
static if(alignment <= 16) {
enum chunkShift = shift, chunkMask = mask;
alias Chunk = AliasSeq!(ubyte, ushort, uint, ulong,
Chunk16)[shift];
} else {
enum chunkShift = Aligned!(16).shift, chunkMask =
Aligned!(16).mask;
alias Chunk = Aligned!(16).Chunk;
}
}
}
@safe unittest {
static immutable(size_t[]) alignments =
[ 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024 ];
static immutable(size_t[]) sizes =
[ 9, 31, 4, 57, 369, 3358 ];
foreach(size; sizes) {
static foreach(alignment; alignments) { {
void[] memory = newAligned!alignment(size);
assert(memory.length == size);
assert((cast(size_t) &(memory[0])) % alignment == 0);
} }
}
}
```
Also, is the alignment of any type guaranteed to be a power of
2?
In practice, yes.
On Friday, 30 September 2022 at 16:23:00 UTC, mw wrote:
https://dlang.org/library/core/stdc/stdlib/aligned_alloc.html
It's the C func, so check C lib doc.
https://en.cppreference.com/w/c/memory/aligned_alloc
Note that common implementations place arbitrary restrictions on
the alignments and sizes accepted by `aligned_alloc`, so to
support the general case you would still need a wrapper function
like the one I provided above.
(If this all seems overly complicated, that's because it is. I
have no idea why allocators don't just build in the logic above;
it's extremely simple compared to the rest of what a good
general-purpose heap allocator does.)
Re: AA and struct with const member
On Tuesday, 28 December 2021 at 07:54:56 UTC, frame wrote:
On Tuesday, 28 December 2021 at 06:38:03 UTC, Tejas wrote:
The workaround is okay, but I think we should file a bug
report for this.
This is very ~~stupid~~ undesirable behaviour
I agree. I'll just wait if somebody can explain why this isn't
a bug or wanted behaviour or a known issue.
[The spec
says](https://dlang.org/spec/hash-map.html#construction_assignment_entries):
```
2. If the assigned value type is equivalent with the AA element
type:
1. If the indexing key does not yet exist in AA, a new AA
entry will be allocated, and it will be initialized with the
assigned value.
2. If the indexing key already exists in the AA, the setting
runs normal assignment.
```
Thus, when the value type is constructable but not assignable:
```d
struct S
{
const(int) a;
}
void test(S[string] aa, string key, int value)
{
// Should be a compile-time error, because it might reassign:
test[key] = S(value);
// Should be accepted, because they can be proven at compile
time to never reassign:
test.require("a", S(value));
test.update("a", () => S(value), (ref const(S)) => { });
}
```
`require` and `update` can be fixed rather easily in `object.d`;
I have submitted [issue
22633](https://issues.dlang.org/show_bug.cgi?id=22633) with
sample code.
Re: Using "strcpy" to assign value to dynamic char array
On Monday, 1 November 2021 at 21:32:21 UTC, Ali Çehreli wrote:
Joking aside, I liked the nested struct and its opAssign to
mimic internal `arr.length = 42` syntax. (I know it involves a
potentially expensive delegate but still...)
The nested struct is not needed. UFCS works for setters, too:
```D
void assumedLength(S)(ref S slice, size_t length) {
if(slice.length >= length)
slice.length = length;
else
assert(false, "Let's not corrupt memory today.");
}
void main() {
auto arr = [ 1, 2, 3 ];
arr.assumedLength = 2;
writeln(arr);
}
```
Re: how much "real-life" code can be marked @safe ?
On Saturday, 3 July 2021 at 16:06:33 UTC, Alexandru Ermicioi
wrote:
3. An edge case. Ex: You need to mutate some data and then
assume it is immutable in a constructor.
Can you give a valid example where that is necessary? The main
examples that I can think of either can be `@safe` with the right
API, or are motivated by a desire to avoid the GC and/or
druntime, thus falling under (1).
4. Functionality that doesn't account for @safe/immutable or
any other features when it can in standard library.
True, although it's just another example of my point (2). The
standard library and druntime are dependencies, too...
Take for example array.dup, there is no inout alternative for
it,
and you're pretty much stuck with trusted code, when you'd like
to dup an array that is inout.
`inout` is usually just a convenient way to use one
implementation to handle mutable, `const` and `immutable` cases.
In those rare cases where `inout` itself won't work, it is almost
always possible to accomplish the same thing using `template
this` or separate overloads:
```D
import std.traits : Unqual, CopyConstness;
struct A {
int*[] arr;
this(this This, Arr)(Arr arr) scope pure @safe nothrow
if(is(Arr : E[], E) && is(E : CopyConstness!(This,
Unqual!E)))
{
this.arr = arr.dup;
}
}
void main()
{
A ma = new int*[5];
const(A) ca = new const(int*[3]);
const(A) ia = new immutable(int*[4]);
}
```
Again, it's certainly not obvious how to do this, or why it is
necessary, but it is *possible*.
The one exception here is when the array is already typed `inout`
before it is passed to the constructor. But, that's an example of
(2) since this logic applies transitively throughout the call
stack: if you need to call `dup` anywhere, don't erase the
constness with `inout`.
Re: how much "real-life" code can be marked @safe ?
(Responding out of order:) On Friday, 2 July 2021 at 00:26:52 UTC, someone wrote: But when you start attempting to declare @safe chunks of code that actually DO things ... well, it seems end-of-the-story. If you find yourself unable to get real work done in `@safe` code, this is almost certainly a sign of one of the following problems: 0) You don't fully understand the purpose and valid use of any or all of the `@trusted`, `inout`, `scope`, and `return` annotations. 1) Your code is avoiding use of the garbage collector, and/or does not have `-dip1000` enabled. (`@safe` is still quite useful without the garbage collector, but even with `-dip1000` you'll still need a lot of `@trusted` code.) 2) You have at least one dependency that isn't correctly designed for use with `@safe`. As long as you're willing to use the garbage collector, almost all algorithms can be expressed in an efficient `@safe` way, **but** this sometimes requires knowledge of several advanced features of D, and how and when to combine them. but no castings are allowed with @safe That is simply not true. Many explicit casts are `@safe`, as are nearly all implicit casts. Casting away `const` or `immutable` is `@system`, but should hardly ever be necessary if you understand how to write constructors correctly (see below), and use `inout` appropriately. Many reinterpret casts are also illegal, but `union` and `class` provide `@safe` ways of achieving the same goals for the common cases. - almost all this() constructors are a no-go providing you do something in-between with the parameters until you assign them back to the target variables; Constructors can be `@safe`, but you have to understand how they work in D: The key difficulty is that what appears to be the first "assignment" to each field in a constructor actually constructs that field, instead. Subsequent assignments really are assignments. So, when constructing a `const` object each field can only be "assigned" once, because the fields are also `const`, even in the object's constructor. If you need to do complex calculations to determine field values, use temporary variables and `static` helper functions until you get the final value, and then unconditionally assign that value to a field *once*. **TLDR**; You probably don't understand how to use `@safe` correctly. (Most people don't; the rules are complicated and non-obvious.) Post some example code that you think can't be `@safe`, and I can probably show you how to fix it, unless it involves manual memory management or an incompatible dependency. Even then, the non-`@safe` code can often be isolated behind an `@trusted` API.
Re: Shift operator, unexpected result
On Wednesday, 9 June 2021 at 19:13:10 UTC, JG wrote:
I found the following behaviour, as part of a more complicated
algorithm, unexpected. The program:
import std;
void main()
{
int n = 64;
writeln(123uL>>n);
}
produces:
123
I would expect 0.
What is the rationale for this behaviour or is it a bug?
Because it is a high-performance systems programming language,
the designers of D decided to make the arithmetic operations of
basic types map directly to the arithmetic operations built in to
the CPU; most operations are a single instruction.
The benefit of this is higher performance and smaller binaries.
The disadvantage is that the behaviour of the built in CPU
operations sometimes differs from ordinary arithmetic in
surprising and frustrating ways.
If you want to trade a some speed for correctness/predictability,
try my `checkedint` Dub package. Either way, take a glance at the
[introduction to the
documentation](https://checkedint.dpldocs.info/checkedint.html),
where I list some of the quirks of CPU integer behaviour.
For bit shifts, specifically, many CPUs ignore all but the bottom
`log2(T.sizeof * 8)` bits of the right-hand operand.
(`core.bitop.bsr` can be used to do very fast integer `log2`
operations, and works in CTFE.) Thus, `a >> b` behaves like `a >>
(b & c)`, where `c` is `(T.sizeof * 8) - 1`.
For unsigned types, the behaviour that you very reasonably expect
requires two additional instruction on x86, which looks like
this: `(b <= c)? (a >> b) : 0`. (This should be branchless thanks
to the `cmov` instruction.)
For signed types, some additional work is required to handle
negative shifts; see my `checkedint` package.
Re: Why std.file.append() new lind "\n" not work in Windows?
On Wednesday, 9 June 2021 at 21:10:58 UTC, Marcone wrote:
std.file.append("file; \nApple");
std.file.append("file; \nBanana");
Result:
AppleBanana
Not all systems use the same char sequence for line breaks. In
particular, Microsoft Windows uses "\r\n".
You can use
[`std.ascii.newline`](https://dlang.org/phobos/std_ascii.html#newline) to write platform-independent code. Functions like `std.stdio.writeln` that write a newline at the end of the message will also do the right thing.
Re: How to work with one very large text table but not in memory
On Tuesday, 6 April 2021 at 19:55:03 UTC, Alain De Vos wrote: I have one very large text table I want to work with. But I don't want to keep de table in memory, what do I use ? Using an sql database is overkill in my setting. There are 10 colums but millions of rows. You might find memory mapped files useful: http://phobos.dpldocs.info/std.mmfile.MmFile.html This allows D code to access the entire contents of the file as though it were a giant byte array in RAM, without requiring that there actually be enough physical RAM available to really do that. The OS is responsible for paging data to and from the disk as it is accessed.
Re: templated overload of opAssign
On Monday, 5 April 2021 at 05:22:22 UTC, frame wrote: On Sunday, 4 April 2021 at 18:05:04 UTC, tsbockman wrote: Thus, the solution is to use an explicit `delegate` instead of `lazy`: Yes, I forgot to mention that. Could you please explain why you set 'scope' here? Isn't it wanted to keep references here? `scope` here indicates that no references to the `f` delegate itself will be escaped from `opAssign`, giving the caller the option of allocating the closure on the stack. `opAssign` may retain a reference to an `Exception` thrown by the delegate, but that's OK because the `Exception` is not part of the `f` delegate data structure, not even indirectly or transitively. (I can explain/justify this in more detail if that still doesn't make sense.)
Re: templated overload of opAssign
On Monday, 5 April 2021 at 15:05:24 UTC, kdevel wrote:
You changed the definition of ``bar`` while the exception
collector (``EC``) is meant to catch and collect an exception
thrown from the *unmodified* function.
My point was that the code will work if you do explicitly what
`lazy` does implicitly. If you don't want to modify `bar`, then
it looks like this:
```D
import std.stdio,std.typecons;
struct EC {
Exception [] ex;
auto opAssign (X) (scope X f)
if(is(X : void delegate()) || is(X : void function()))
{
writeln (__PRETTY_FUNCTION__);
try return f (); catch (Exception e) ex ~= e;
}
}
class E : Exception { this (string s) { super (s); } }
void bar (int i) { if (i == 1) throw new E ("E"); }
void main ()
{
EC ec;
ec = () { bar (1); }; // okay
ec.writeln;
}
```
It seems that the operator ``+=`` or ``~=`` may be better
suited to express that intent. Rewriting this in terms of
``opOpAssign`` works as expected:
Given that you define the operation to append `~` to `ex` rather
than overwriting it, `~=` is the best fit, I think.
However, `=` and `~=` should not treat `lazy void` parameters
differently. They should either both work, or neither. I checked
and this is actually a very old regression; both worked way back
in DMD 2.061. So, I've filed a front-end bug report:
https://issues.dlang.org/show_bug.cgi?id=21802
Re: templated overload of opAssign
On Sunday, 4 April 2021 at 16:38:10 UTC, frame wrote:
On Saturday, 3 April 2021 at 13:46:17 UTC, kdevel wrote:
Why does this code
ec.opAssign (bar (1)); // okay
// ec = bar (1); // Error: expression bar(1) is void and has
no value
compile with the abovementioned error?
You cannot assign void returned from bar() as parameter to
opAssign(). The lazy keyword creates some internal delegate,
thus opAssign() works instead.
Thus, the solution is to use an explicit `delegate` instead of
`lazy`:
```D
import std.stdio,std.typecons;
struct EC {
Exception [] ex;
auto opAssign (X: void delegate()) (scope X f)
{
writeln (__PRETTY_FUNCTION__);
try return f (); catch (Exception e) ex ~= e;
}
}
class E : Exception { this (string s) { super (s); } }
auto bar (int i) {
return () {
if (i == 1)
throw new E ("E");
};
}
void main ()
{
EC ec;
ec = bar (1); // okay
ec.writeln;
}
```
Re: Best way to make a template function conditionally @trusted
On Friday, 2 April 2021 at 19:49:30 UTC, ikod wrote:
On Thursday, 1 April 2021 at 22:35:01 UTC, tsbockman wrote:
Suppose I have a templated struct member function for which I
can compute at compile-time when the function is memory safe,
and when it is not. But, the compiler cannot correctly
determine this automatically.
Compiler should be able to derive safety of templated
functions. You may just omit `@safe`/`@trusted`.
The compiler's approach to verifying memory safety is very
simplistic: it declares a function non-`@safe` if any potentially
unsafe operation is found in its implementation, without regard
for the context. It can only ever infer `@safe` or `@system`,
never `@trusted`.
The reason `@trusted` is in the language at all is to allow the
programmer to manually mark as memory safe those functions which
contain operations that would be unsafe in some other context,
but which the programmer has manually analyzed and verified to be
incapable of violating memory safety, no matter what inputs it
receives.
Consider the following program:
```D
import std.stdio;
void writeSafe()(int[2] stuff ...) {
foreach(n; 0 .. stuff.length)
writeln(stuff[n]);
}
void writeTrusted()(int[2] stuff ...) {
foreach(n; 0 .. stuff.length)
writeln(stuff.ptr[n]);
}
void main() @safe {
writeSafe(3, 5);
writeTrusted(3, 5);
}
```
`writeSafe` and `writeTrusted` generate identical code, and are
equally memory safe in reality. The compiler even knows this on
some level, because it correctly deduces that the bounds check
for `stuff[n]` can never fail, and omits it even in `writeSafe`.
Nevertheless, because `.ptr` can be used to violate memory safety
in other contexts, `writeSafe` is inferred as `@safe`, while
`writeTrusted` is inferred as `@system`. And so, the program
above will not compile as it stands.
This is, of course, a trivial example where there is no benefit
to using the non-`@safe` version, but there are more complex
cases where the desired algorithm is memory safe as a whole, but
it cannot be expressed in D without using some operations that
are forbidden in `@safe` code.
Re: Casting lvalues
On Friday, 2 April 2021 at 12:47:35 UTC, z wrote: Even if the function is changed to only accept `shared` parameters, `.reserve` does not appear to support `shared` so the function is impossible to use without somehow changing its type or using `__gshared`. There is no way that `.reserve` can correctly support `shared`, because it cannot know how to correctly synchronize the slice and its contents in the larger context of the whole program. In general, just casting away `shared` to make the compiler stop complaining completely defeats the purpose of `shared`, and makes it just as unsafe as `__gshared`. Instead, you must do any necessary synchronization yourself using, for example, `core.sync.mutex`. Only within properly synchronized regions should you cast away `shared`. This is the purpose of `shared`, and the only difference from `__gshared`: to remind you to synchronize by requiring the use of a cast to do much of anything. Typically, you should synchronize an entire region of code across which the `shared` data under protection will begin and end with all invariants satisfied. Synchronizing individual operations (like just `.reserve` by itself) is usually wrong - or rather, insufficient. The synchronization must guarantee that whenever a thread is writing to the data, it has exclusive access. But, it is safe and fast to have multiple threads read data simultaneously, as long as it is not written to during that time. `core.sync.rwmutex` can be used to implement this optimization. (It is possible to design algorithms that support multiple simultaneous writers using lock-free algorithms (see `core.atomic`) or other more complicated schemes, but this is much harder to do correctly and usually not necessary.) Finally, if you only need to write the data once, after which it will only be read, then you can skip all of this confusion and complexity by just preparing the data from a single thread in a non-`shared` container and then using `cast(immutable)` (if there is only one extant reference to the data) or `.idup` (otherwise). No synchronization is necessary for `immutable` data. `*(cast(Unqual!TT*))`, not ideal That is the correct way to perform an lvalue reinterpret cast (provided that the resulting type is actually compatible with the source type). (direct casting seems to create an rvalue because compilation fails.) Yes, direct casting should result in an rvalue. (Although, I think it is possible to subvert this with a custom `opCast`.)
Re: Best way to make a template function conditionally @trusted
On Friday, 2 April 2021 at 00:03:32 UTC, Paul Backus wrote: On Thursday, 1 April 2021 at 22:35:01 UTC, tsbockman wrote: Is there a better way? Here's a technique I've used: ... As long as the compile-time check is correct, this is sound: the @trusted lambda can be called from @safe code if and only if `shouldBeSystem == false`. Thanks. That is somewhat better than my version.
Best way to make a template function conditionally @trusted
Suppose I have a templated struct member function for which I can compute at compile-time when the function is memory safe, and when it is not. But, the compiler cannot correctly determine this automatically. What is the best way to express this in code? Other than straight-up duplicating the implementation, the only answer I've come up with so far is to create a `private @system` implementation function, and then separate `public @system` and `public @trusted` wrappers with appropriate template constraints. Is there a better way?
Re: Storing interfaces as void[]
On Saturday, 13 March 2021 at 15:44:53 UTC, frame wrote: Maybe I don't get this right but why you don't just use the void[] as it is? void[] mem = [i]; //... return (cast(T[]) mem)[0]; This is how I exchange variable data between DLLs. That works, and is more elegant than the OP's solution. (And, I didn't know you could do it that way, so thanks for sharing!) However, it still adds an unnecessary second level of indirection for interfaces and classes. The simplest and most efficient solution is actually this: auto indirect = cast(void*) i; // i is of type I. // ... return cast(I) indirect; Where `is(I : T*, T) || is(I == interface) || is(I == class)`. The constraint enforces that all types should be accessed through exactly one level of indirection, allowing the same code to handle both reference types and value types with maximum efficiency.
Re: Storing interfaces as void[]
On Saturday, 13 March 2021 at 00:36:37 UTC, David Zhang wrote:
On Friday, 12 March 2021 at 22:18:59 UTC, tsbockman wrote:
You can use TypeInfo references as the keys for the struct
types, too. It's better than hashes because the TypeInfo is
already being generated anyway, and is guaranteed to be
unique, unlike your hashes which could theoretically collide.
Makes sense. Using TypeInfo never occurred to me. I assume they
are generated for COM classes as well?
I'm not sure about that; you should test it yourself. I know that
runtime type information support is incomplete for some
non-`extern(D)` types - for example:
https://issues.dlang.org/show_bug.cgi?id=21690
You can always fall back to fully qualified names for
non-`extern(D)` stuff if you have to.
But you should protect against hash collisions somehow, if you go
that route. Here's a hybrid approach that is immune to hash
collisions and works across DLL boundaries, but can almost always
verify equality with a single pointer comparison:
///
struct TypeKey {
private:
const(void)* ptr;
size_t length;
this(const(TypeInfo) typeInfo) const pure @trusted nothrow
@nogc {
ptr = cast(const(void)*) typeInfo;
length = 0u;
}
this(string fqn) immutable pure @trusted nothrow {
/* We need to allocate a block of size_t to ensure proper
alignment
for the first chunk, which is the hash code of the fqn
string: */
size_t[] chunks = new size_t[1 + (fqn.length +
(size_t.sizeof - 1)) / size_t.sizeof];
chunks[0] = hashOf(fqn);
(cast(char*) chunks.ptr)[size_t.sizeof .. size_t.sizeof +
fqn.length] = fqn;
ptr = cast(immutable(void)*) chunks.ptr;
length = fqn.length;
}
@property const(TypeInfo) typeInfo() const pure @trusted
nothrow @nogc
in(length == 0u)
{
return cast(const(TypeInfo)) ptr;
}
@property size_t fqnHash() const pure @trusted nothrow @nogc
in(length != 0u)
{
return *cast(const(size_t)*) ptr;
}
@property string fqn() const pure @trusted nothrow @nogc
in(length != 0u)
{
const fqnPtr = cast(immutable(char)*) (this.ptr +
size_t.sizeof);
return fqnPtr[0 .. length];
}
public:
string toString() const @safe {
return (length == 0u)? typeInfo.toString() : fqn; }
size_t toHash() const @safe nothrow {
return (length == 0u)? typeInfo.toHash : fqnHash; }
bool opEquals(TypeKey that) const @trusted {
if(this.ptr is that.ptr)
return true;
if(this.length != that.length)
return false;
if(length == 0u)
return (this.typeInfo == that.typeInfo);
if(this.fqnHash != that.fqnHash)
return false;
return (this.fqn == that.fqn);
}
}
template typeKeyOf(Indirect)
if(is(Indirect : T*, T) // Support structs, static arrays,
etc.
|| is(Indirect == interface) || is(Indirect == class))
{
static if(is(Indirect : T*, T) || (__traits(getLinkage,
Indirect) == "D")) {
@property const(TypeKey) typeKeyOf() pure @safe nothrow
@nogc {
return const(TypeKey)(typeid(Indirect)); }
} else {
/* For FQN-based keys, ideally the whole process should
share a single
copy of the heap allocated fqn and its hash, so we'll use
a global. No
synchronization is necessary post construction, since it
is immutable: */
private immutable TypeKey masterKey;
shared static this() {
// With some bit-twiddling, this could be done at
compile time, if needed:
import std.traits : fullyQualifiedName;
masterKey =
immutable(TypeKey)(fullyQualifiedName!Indirect);
}
@property immutable(TypeKey) typeKeyOf() @safe nothrow
@nogc {
assert(masterKey.ptr !is null);
return masterKey;
}
}
}
@safe unittest {
static extern(C++) class X { }
static extern(C++) class Y { }
assert(typeKeyOf!(int*) == typeKeyOf!(int*));
assert(typeKeyOf!(int*) != typeKeyOf!(float*));
assert(typeKeyOf!X == typeKeyOf!X);
assert(typeKeyOf!X != typeKeyOf!Y);
assert(typeKeyOf!X != typeKeyOf!(int*));
assert(typeKeyOf!(float*) != typeKeyOf!Y);
}
///
The lowering is something like this:
...
Makes sense, I always thought of them as existing in separate
places.
Yeah, there are multiple reasonable ways of supporting interfaces
in a language, each with their own trade-offs. But, this is the
one used by D at the moment.
So much head-bashing, and it's over. Thanks, tsbockman,
Imperatorn.
You're very welcome!
Re: Storing interfaces as void[]
On Friday, 12 March 2021 at 19:24:17 UTC, David Zhang wrote:
On Friday, 12 March 2021 at 18:50:26 UTC, tsbockman wrote:
The idea is to implement a service locator s.t. code like this
is possible:
...
I don't really need to copy or move the class instances here,
just be able to read, assign, and replace references to them in
the same place that I read, assign, and replace void[]'s of
structs.
Why do you think you need a `void[]` slice? I think `void*`
pointers are sufficient. This handles all normal data types, as
long as they are allocated on the GC heap:
/* NOTE: It might be wiser to make this a `final class`
instead of a `struct` to get reference semantics: */
struct Registry {
private void*[TypeInfo] map;
/// Return true if the registry was updated, false if not.
bool put(bool overwrite = true, Indirect)(Indirect i) @trusted
if(is(Indirect : T*, T) // Support structs, static arrays,
etc.
|| is(Indirect == interface) || is(Indirect == class))
{
auto key = typeid(Indirect), value = cast(void*) i;
bool updated;
if(value is null)
updated = map.remove(key);
else {
static if(overwrite) {
updated = true;
map[key] = value;
} else
updated = (map.require(key, value) is value);
}
return updated;
}
alias put(Indirect) = put!(true, Indirect);
bool remove(Indirect)() @trusted
if(is(Indirect : T*, T) // Support structs, static arrays,
etc.
|| is(Indirect == interface) || is(Indirect == class))
{
return map.remove(typeid(Indirect));
}
/** Returns a reference (for interfaces and classes) or
a pointer (for everything else) if the type has been
registered,
and null otherwise. **/
Indirect get(Indirect)() @trusted
if(is(Indirect : T*, T) // Support structs, static
arrays, etc.
|| is(Indirect == interface) || is(Indirect == class))
{
return cast(Indirect) map.get(typeid(Indirect), null);
}
}
@safe unittest {
static interface I {
char c() const pure @safe nothrow @nogc;
}
static class C : I {
private char _c;
this(char c) inout pure @safe nothrow @nogc {
this._c = c; }
override char c() const pure @safe nothrow @nogc {
return _c; }
}
static struct S {
char c = 'S';
this(char c) inout pure @safe nothrow @nogc {
this.c = c; }
}
Registry registry;
assert( registry.put!false(new C('1')));
assert( registry.put!I(new C('2')));
assert( registry.put(new S('$')));
assert(!registry.put!false(new C('3'))); // Optionally
protect existing entries.
assert(registry.get!(I).c == '2');
assert(registry.get!(C).c == '1');
assert(registry.remove!I);
assert(registry.get!I is null);
assert(registry.get!C !is null);
assert(registry.get!(S*).c == '$');
assert(registry.get!(int*) is null);
}
NOTE: If you only need one Registry instance in per thread, then
you probably don't need the associative array at all; instead
just use a template like this:
template registered(Indirect)
if(is(Indirect : T*, T) // Support structs, static arrays,
etc.
|| is(Indirect == interface) || is(Indirect == class))
{
/* This uses thread local storage (TLS). Sharing across the
entire
process is possible too, but would require synchronization of
the
registered objects, not just this reference/pointer: */
private Indirect indirect = null;
/// Return true if the registry was updated, false if not.
bool put(bool overwrite = true)(Indirect i) @safe
{
bool updated = (indirect !is i);
static if(overwrite) {
indirect = i;
} else {
updated &= (indirect is null);
if(updated)
indirect = i;
}
return updated;
}
bool remove() @safe {
bool updated = (indirect !is null);
indirect = null;
return updated;
}
/** Returns a reference (for interfaces and classes) or
a pointer (for everything else) if the type has been
registered,
and null otherwise. **/
Indirect get() @safe {
return indirect; }
}
@safe unittest {
static interface I {
char c() const pure @safe nothrow @nogc;
}
static class C : I {
private char _c;
this(char c) inout pure @safe nothrow @nogc {
this._c = c; }
override char c() const pure @safe nothrow @nogc {
return _c; }
}
static struct S {
char c = 'S';
this(char c) inout pure @safe nothrow @nogc {
this.c = c; }
}
assert( registered!(C).put!false(new C('1')));
assert( registered!(I).put(new C('2')));
assert(
Re: Storing interfaces as void[]
On Friday, 12 March 2021 at 18:50:26 UTC, tsbockman wrote: /* This will return `null` if the value isn't really a reference to an instance of a subtype of `I`, or if the key isn't in the map yet. If you don't care about the latter case, it can be shortened to just `cast(I) map[I.stringof]`. */ Oops, that last bit should say `cast(I) map[typeid(I)]`.
Re: Storing interfaces as void[]
On Friday, 12 March 2021 at 17:37:43 UTC, David Zhang wrote:
I want to store interfaces as untyped void[], then cast them
back to the interface at a later time.
Assuming these interfaces are actual D `interface`s, declared
with the keyword and using the default `extern(D)` linkage,
you're way over-complicating things. All `extern(D)` `class`es
are sub-types of `object.Object`, and have type information
accessible through the implicit virtual function table pointer
`__vptr` that can be used to perform safe dynamic casts:
/* Fully qualified names can be *very* long because of templates,
so it can be wasteful to store and compare them at runtime. Let's
use `TypeInfo` instead: */
Object[TypeInfo] map;
void register(I)(I i)
if(is(I == interface)) // This wouldn't work right with value
types like structs.
{
/* The garbage collector will keep the instance referenced by
`i` alive for us
as long as necessary. So, there is no need to copy the
instance unless we want
to be able to mutate it elsewhere without affecting the
instance refrenced by
`map`. */
/* `typeid(i)` will compile, but probably doesn't do what you
want: you need to
retrieve the value using the same key you put it in with, and
below that has
to be `typeid(I)`: */
map[typeid(I)] = cast(Object) i;
}
I get(I)()
if(is(I == interface)) // This wouldn't work right with value
types like structs.
{
/* This will return `null` if the value isn't really a
reference to an instance
of a subtype of `I`, or if the key isn't in the map yet. If
you don't care
about the latter case, it can be shortened to just `cast(I)
map[I.stringof]`. */
auto valuePtr = (typeid(I) in map);
return (valuePtr is null)? null : cast(I) *valuePtr;
}
Many variations on this are possible, depending on exactly what
you're really trying to accomplish. This is my best guess as to
what you're after, though, without seeing the context for the
code you posted.
However, it appears to produce garbage values on get().
Is this even possible, and if so, what is happening here? The
alternative would be a struct { CheckedPtr self; api_fns }
e.g.
void register(I)(I i) {
auto mem = new void[](I.sizeof);
memcpy(mem.ptr, cast(void*) i, I.sizeof);
// CheckedPtr includes a hash of fullyQualifiedName
map[i.get_name()] = CheckedPtr!I(mem.ptr);
}
I get(I)() {
// basically cast(I) p
return map[I.get_name()].as!I();
}
Your code confuses two levels of indirection (because D makes
this super confusing); that is why it doesn't do what you want.
`cast(void*) i` is an untyped pointer to some class instance,
while `I.sizeof` is the size of *the pointer itself*, NOT the
instance it points to.
`I` is *not* the type of an interface instance, it is the type of
a reference to an instance of the interface. (A reference is a
pointer that sometimes automatically dereferences itself.) So,
I.sizeof is always (void*).sizeof, the size of a pointer.
The types of class and interface instances cannot be named or
referenced in D, but `__traits(classInstanceSize, C)` will get
you the size of an instance of class `C`. Getting the size of an
interface instance isn't really a sensible thing to do, since the
whole point of interfaces is that you don't need to know about
the implementing type to work with them. Regardless, copying or
moving the memory of a class instances directly is generally a
bad idea, for various reasons.
Re: Optimizing for SIMD: best practices?(i.e. what features are allowed?)
On Sunday, 7 March 2021 at 22:54:32 UTC, tsbockman wrote:
...
result = diffSq[0];
static foreach(i; 0 .. 3)
result += diffSq[i];
...
Oops, that's supposed to say `i; 1 .. 3`. Fixed:
import std.meta : Repeat;
void euclideanDistanceFixedSizeArray(V)(ref Repeat!(3, const(V))
a, ref Repeat!(3, const(V)) b, ref V result)
if(is(V : __vector(float[length]), size_t length))
{
Repeat!(3, V) diffSq = a;
static foreach(i; 0 .. 3) {
diffSq[i] -= b[i];
diffSq[i] *= diffSq[i];
}
result = diffSq[0];
static foreach(i; 1 .. 3)
result += diffSq[i];
version(LDC) { version(X86_64) {
enum isSupportedPlatform = true;
import ldc.llvmasm : __asm;
result = __asm!V(`vsqrtps $1, $0`, `=x, x`, result);
} }
static assert(isSupportedPlatform);
}
Fixed asm:
pure nothrow @nogc void
app.euclideanDistanceFixedSizeArray!(__vector(float[16])).euclideanDistanceFixedSizeArray(ref const(__vector(float[16])), ref const(__vector(float[16])), ref const(__vector(float[16])), ref const(__vector(float[16])), ref const(__vector(float[16])), ref const(__vector(float[16])), ref __vector(float[16])):
mov rax, qword ptr [rsp + 8]
vmovaps zmm0, zmmword ptr [rax]
vmovaps zmm1, zmmword ptr [r9]
vmovaps zmm2, zmmword ptr [r8]
vsubps zmm0, zmm0, zmmword ptr [rcx]
vsubps zmm1, zmm1, zmmword ptr [rdx]
vmulps zmm1, zmm1, zmm1
vsubps zmm2, zmm2, zmmword ptr [rsi]
vfmadd231ps zmm1, zmm0, zmm0
vfmadd231ps zmm1, zmm2, zmm2
vmovaps zmmword ptr [rdi], zmm1
vsqrtps zmm0, zmm1
vmovaps zmmword ptr [rdi], zmm0
vzeroupper
ret
(I really wish I could just edit my posts here...)
Re: Optimizing for SIMD: best practices?(i.e. what features are allowed?)
On Sunday, 7 March 2021 at 22:54:32 UTC, tsbockman wrote: import std.meta : Repeat; void euclideanDistanceFixedSizeArray(V)(ref Repeat!(3, const(V)) a, ref Repeat!(3, const(V)) b, out V result) if(is(V : __vector(float[length]), size_t length)) ... Resulting asm with is(V == __vector(float[16])): .LCPI1_0: .long 0x7fc0 pure nothrow @nogc void app.euclideanDistanceFixedSizeArray!(__vector(float[16])).euclideanDistanceFixedSizeArray(ref const(__vector(float[16])), ref const(__vector(float[16])), ref const(__vector(float[16])), ref const(__vector(float[16])), ref const(__vector(float[16])), ref const(__vector(float[16])), out __vector(float[16])): mov rax, qword ptr [rsp + 8] vbroadcastsszmm0, dword ptr [rip + .LCPI1_0] ... Apparently the optimizer is too stupid to skip the redundant float.nan broadcast when result is an `out` parameter, so just make it `ref V result` instead for better code gen: pure nothrow @nogc void app.euclideanDistanceFixedSizeArray!(__vector(float[16])).euclideanDistanceFixedSizeArray(ref const(__vector(float[16])), ref const(__vector(float[16])), ref const(__vector(float[16])), ref const(__vector(float[16])), ref const(__vector(float[16])), ref const(__vector(float[16])), ref __vector(float[16])): mov rax, qword ptr [rsp + 8] vmovaps zmm0, zmmword ptr [rax] vmovaps zmm1, zmmword ptr [r9] vmovaps zmm2, zmmword ptr [r8] vsubps zmm0, zmm0, zmmword ptr [rcx] vmulps zmm0, zmm0, zmm0 vsubps zmm1, zmm1, zmmword ptr [rdx] vsubps zmm2, zmm2, zmmword ptr [rsi] vaddps zmm0, zmm0, zmm0 vfmadd231ps zmm0, zmm1, zmm1 vfmadd231ps zmm0, zmm2, zmm2 vmovaps zmmword ptr [rdi], zmm0 vsqrtps zmm0, zmm0 vmovaps zmmword ptr [rdi], zmm0 vzeroupper ret
Re: Optimizing for SIMD: best practices?(i.e. what features are allowed?)
On Sunday, 7 March 2021 at 18:00:57 UTC, z wrote:
On Friday, 26 February 2021 at 03:57:12 UTC, tsbockman wrote:
static foreach(size_t i; 0 .. 3/+typeof(a).length+/){
distance += a[i].abs;//abs required by the caller
(a * a) above is always positive for real numbers. You don't
need the call to abs unless you're trying to guarantee that
even nan values will have a clear sign bit.
I do not know why but the caller's performance nosedives
My way is definitely (slightly) better; something is going wrong
in either the caller, or the optimizer. Show me the code for the
caller and maybe I can figure it out.
whenever there is no .abs at this particular line.(there's a 3x
difference, no joke.)
Perhaps the compiler is performing a value range propagation
(VRP) based optimization in the caller, but isn't smart enough to
figure out that the value is already always positive without the
`abs` call? I've run into that specific problem before.
Alternatively, sometimes trivial changes to the code that
*shouldn't* matter make the difference between hitting a smart
path in the optimizer, and a dumb one. Automatic SIMD
optimization is quite sensitive and temperamental.
Either way, the problem can be fixed by figuring out what
optimization the compiler is doing when it knows that distance is
positive, and just doing it manually.
Same for assignment instead of addition, but with a 2x
difference instead.
Did you fix the nan bug I pointed out earlier? More generally,
are you actually verifying the correctness of the results in any
way for each alternative implementation? Because you can get big
speedups sometimes from buggy code when the compiler realizes
that some later logic error makes earlier code irrelevant, but
that doesn't mean the buggy code is better...
Re: Optimizing for SIMD: best practices?(i.e. what features are allowed?)
On Sunday, 7 March 2021 at 13:26:37 UTC, z wrote:
On Thursday, 25 February 2021 at 11:28:14 UTC, z wrote:
However, AVX512 support seems limited to being able to use the
16 other YMM registers, rather than using the same code
template but changed to use ZMM registers and double the
offsets to take advantage of the new size.
Compiled with «-g -enable-unsafe-fp-math
-enable-no-infs-fp-math -ffast-math -O -release -mcpu=skylake» :
You're not compiling with AVX512 enabled. You would need to use
-mcpu=skylake-avx512.
However, LLVM's code generation for AVX512 seems to be pretty
terrible still, so you'll need to either use some inline ASM, or
stick with AVX2. Here's a structure of arrays style example:
import std.meta : Repeat;
void euclideanDistanceFixedSizeArray(V)(ref Repeat!(3, const(V))
a, ref Repeat!(3, const(V)) b, out V result)
if(is(V : __vector(float[length]), size_t length))
{
Repeat!(3, V) diffSq = a;
static foreach(i; 0 .. 3) {
diffSq[i] -= b[i];
diffSq[i] *= diffSq[i];
}
result = diffSq[0];
static foreach(i; 0 .. 3)
result += diffSq[i];
version(LDC) { version(X86_64) {
enum isSupportedPlatform = true;
import ldc.llvmasm : __asm;
result = __asm!V(`vsqrtps $1, $0`, `=x, x`, result);
} }
static assert(isSupportedPlatform);
}
Resulting asm with is(V == __vector(float[16])):
.LCPI1_0:
.long 0x7fc0
pure nothrow @nogc void
app.euclideanDistanceFixedSizeArray!(__vector(float[16])).euclideanDistanceFixedSizeArray(ref const(__vector(float[16])), ref const(__vector(float[16])), ref const(__vector(float[16])), ref const(__vector(float[16])), ref const(__vector(float[16])), ref const(__vector(float[16])), out __vector(float[16])):
mov rax, qword ptr [rsp + 8]
vbroadcastsszmm0, dword ptr [rip + .LCPI1_0]
vmovaps zmmword ptr [rdi], zmm0
vmovaps zmm0, zmmword ptr [rax]
vmovaps zmm1, zmmword ptr [r9]
vmovaps zmm2, zmmword ptr [r8]
vsubps zmm0, zmm0, zmmword ptr [rcx]
vmulps zmm0, zmm0, zmm0
vsubps zmm1, zmm1, zmmword ptr [rdx]
vsubps zmm2, zmm2, zmmword ptr [rsi]
vaddps zmm0, zmm0, zmm0
vfmadd231ps zmm0, zmm1, zmm1
vfmadd231ps zmm0, zmm2, zmm2
vmovaps zmmword ptr [rdi], zmm0
vsqrtps zmm0, zmm0
vmovaps zmmword ptr [rdi], zmm0
vzeroupper
ret
Re: Can't I allocate at descontructor?
On Friday, 5 March 2021 at 21:17:24 UTC, tsbockman wrote:
On Friday, 5 March 2021 at 21:02:08 UTC, H. S. Teoh wrote:
class C {...}
import core.memory : GC;
C c = cast(C) GC.malloc(C.sizeof);
...
...
import core.memory : GC;
C c = cast(C) GC.malloc(C.sizeof);
...
Also, that's not the correct way to manually allocate a class on
the heap. C.sizeof is the size of a reference to C, not an
instance of C, and we need to blit and construct the instance
before it is safe to use:
import core.memory : GC;
C c = cast(C) GC.malloc(__traits(classInstanceSize, C));
import core.lifetime : emplace;
emplace(c, anyConstructorArgsGoHere);
...
Re: Can't I allocate at descontructor?
On Friday, 5 March 2021 at 21:02:08 UTC, H. S. Teoh wrote:
If you know when you can deallocate something, that means you
don't need the GC to collect it, so you could just allocate it
on the malloc heap instead, and call destroy/free once you're
done. You could use the C version of malloc/free. You can
also optionally use GC.malloc/GC.free.
E.g.:
class C {...}
import core.memory : GC;
C c = cast(C) GC.malloc(C.sizeof);
... // use c
// We're done with c, destroy it
destroy(c); // this will call the dtor
GC.free(cast(void*) c);
c = null; // optional, just to ensure we don't accidentally
use it again
Unless the function is nothrow, that should really be:
import core.memory : GC;
C c = cast(C) GC.malloc(C.sizeof);
scope(exit) {
// We're done with c, destroy it
destroy(c); // this will call the dtor
GC.free(cast(void*) c);
c = null; // optional, just to ensure we don't
accidentally use it again
}
... // use c
Or,
import core.memory : GC;
C c = cast(C) GC.malloc(C.sizeof);
try {
... // use c
} finally {
// We're done with c, destroy it
destroy(c); // this will call the dtor
GC.free(cast(void*) c);
c = null; // optional, just to ensure we don't
accidentally use it again
}
Re: Optimizing for SIMD: best practices?(i.e. what features are allowed?)
On Thursday, 25 February 2021 at 11:28:14 UTC, z wrote:
float euclideanDistanceFixedSizeArray(float[3] a, float[3] b) {
Use __vector(float[4]), not float[3].
float distance;
The default value for float is float.nan. You need to explicitly
initialize it to 0.0f or something if you want this function to
actually do anything useful.
a[] -= b[];
a[] *= a[];
With __vector types, this can be simplified (not optimized) to
just:
a -= b;
a *= a;
static foreach(size_t i; 0 .. 3/+typeof(a).length+/){
distance += a[i].abs;//abs required by the caller
(a * a) above is always positive for real numbers. You don't need
the call to abs unless you're trying to guarantee that even nan
values will have a clear sign bit.
Also, there is no point to adding the first component to zero,
and copying element [0] from a SIMD register into a scalar is
free, so this can become:
float distance = a[0];
static foreach(size_t i; 1 .. 3)
distance += a[i];
}
return sqrt(distance);
}
Final assembly output (ldc 1.24.0 with -release -O3
-preview=intpromote -preview=dip1000 -m64 -mcpu=haswell
-fp-contract=fast -enable-cross-module-inlining):
vsubps xmm0, xmm1, xmm0
vmulps xmm0, xmm0, xmm0
vmovshdup xmm1, xmm0
vaddss xmm1, xmm0, xmm1
vpermilpd xmm0, xmm0, 1
vaddss xmm0, xmm0, xmm1
vsqrtss xmm0, xmm0, xmm0
ret
Re: Optimizing for SIMD: best practices?(i.e. what features are allowed?)
On Thursday, 25 February 2021 at 11:28:14 UTC, z wrote:
Is there any way to guarantee that "packed" versions of SIMD
instructions will be used?(e.g. vmulps, vsqrtps, etc...)
To give some context, this is a sample of one of the functions
that could benefit from better SIMD usage :
float euclideanDistanceFixedSizeArray(float[3] a, float[3] b) {
You need to use __vector(float[4]) instead of float[3] to tell
the compiler to pack multiple elements per SIMD register. Right
now your data lacks proper alignment for SIMD load/stores.
Beyond that, SIMD code is rather difficult to optimize. Code
written in ignorance or in a rush is unlikely to be meaningfully
faster than ordinary scalar code, unless the data flow is very
simple. You will probably get a bigger speedup for less effort
and pain by first minimizing heap allocations, maximizing
locality of reference, minimizing indirections, and minimizing
memory use. (And, of course, it should go without saying that
choosing an asymptotically efficient high-level algorithm is more
important than any micro-optimization for large data sets.)
Nevertheless, if you are up to the challenge, SIMD can sometimes
provide a final 2-3x speed boost.
Your algorithms will need to be designed to minimize mixing of
data between SIMD channels, as this forces the generation of lots
of extra instructions to swizzle the data, or worse to unpack and
repack it. Something like a Cartesian dot product or cross
product will benefit much less from SIMD than vector addition,
for example. Sometimes the amount of swizzling can be greatly
reduced with a little algebra, other times you might need to
refactor an array of structures into a structure of arrays.
Per-element conditional branches are very bad, and often
completely defeat the benefits of SIMD. For very short segments
of code (like conditional assignment), replace them with a SIMD
conditional move (vcmp and vblend). Bit-twiddling is your friend.
Finally, do not trust the compiler or the optimizer. People love
to make the claim that "The Compiler" is always better than
humans at micro-optimizations, but this is not at all the case
for SIMD code with current systems. I have found even LLVM to
produce quite bad SIMD code for complex algorithms, unless I
carefully structure my code to make it as easy as possible for
the optimizer to get to the final assembly I want. A sprinkling
of manual assembly code (directly, or via a library) is also
necessary to fill in certain instructions that the compiler
doesn't know when to use at all.
Resources I have found very helpful:
Matt Godbolt's Compiler Explorer online visual disassembler
(supports D):
https://godbolt.org/
Felix Cloutier's x86 and amd64 instruction reference:
https://www.felixcloutier.com/x86/
Agner Fog's optimization guide (especially the instruction
tables):
https://agner.org/optimize/
Re: Class instance alignment
On Tuesday, 23 February 2021 at 03:53:00 UTC, tsbockman wrote:
size_t alignedSize(size_t typeSize, size_t typeAlignment) pure
@safe nothrow @nogc {
version(assert) {
import core.bitop : bsr;
assert(typeAlignment == (size_t(1) <<
bsr(typeAlignment)));
}
size_t ret = typeSize & ~(typeAlignment - 1);
ret += (ret < typeSize)? typeAlignment : 0;
return ret;
}
Better:
size_t alignedSize(size_t typeSize, size_t typeAlignment) pure
@safe nothrow @nogc {
version(assert) {
import core.bitop : bsr;
assert(typeAlignment == (size_t(1) <<
bsr(typeAlignment)));
}
const alignMask = typeAlignment - 1;
return (typeSize + alignMask) & ~alignMask;
}
Re: Class instance alignment
On Monday, 22 February 2021 at 02:23:27 UTC, Steven Schveighoffer
wrote:
Hm... but does TypeInfo detail alignment? If so, we can make
this work anyway, just bump up the size needed to a power-of-2
pool.
It doesn't even need to be a power-of-2, assuming the pools
themselves are properly aligned - just a multiple of the
alignment:
size_t alignedSize(size_t typeSize, size_t typeAlignment) pure
@safe nothrow @nogc {
version(assert) {
import core.bitop : bsr;
assert(typeAlignment == (size_t(1) <<
bsr(typeAlignment)));
}
size_t ret = typeSize & ~(typeAlignment - 1);
ret += (ret < typeSize)? typeAlignment : 0;
return ret;
}
(This CTFE-able and can be memoized with a template, if desired.
It's also just a few branchless instructions at runtime, if it's
needed then for some reason.)
Re: Trying to reduce memory usage
On Friday, 19 February 2021 at 00:13:19 UTC, Jon Degenhardt wrote: On Wednesday, 17 February 2021 at 04:10:24 UTC, tsbockman wrote: I spent some time experimenting with this problem, and here is the best solution I found, assuming that perfect de-duplication is required. (I'll put the code up on GitHub / dub if anyone wants to have a look.) It would be interesting to see how the performance compares to tsv-uniq (https://github.com/eBay/tsv-utils/tree/master/tsv-uniq). The prebuilt binaries turn on all the optimizations (https://github.com/eBay/tsv-utils/releases). My program (called line-dedup below) is modestly faster than yours, with the gap gradually widening as files get bigger. Similarly, when not using a memory-mapped scratch file, my program is modestly less memory hungry than yours, with the gap gradually widening as files get bigger. In neither case is the difference very exciting though; the real benefit of my algorithm is that it can process files too large for physical memory. It might also handle frequent hash collisions better, and could be upgraded to handle huge numbers of very short lines efficiently. Test 0 (139 MiB, 1537300 unique lines of 204): sort source | uniq > dest Wall time: 2.04 s User time: 9.34 s System time: 0.23 s Max resident set: 385 MiB tsv-uniq source > dest Wall time: 0.82 s User time: 0.73 s System time: 0.13 s Max resident set: 229 MiB line-dedup source dest Wall time: 0.58 s User time: 0.52 s System time 0.05 s Max resident set: 206 MiB Test 1 (1.4 GiB, 14338280 unique lines of 2004): sort source | uniq > dest Wall time: 22.9 s User time: 107 s System time: 2.16 s Max resident set: 3.76 GiB tsv-uniq source > dest Wall time: 9.94 s User time: 9.25 s System time: 1.3 s Max resident set: 2.34 GiB line-dedup source dest Wall time: 6.34 s User time: 5.88 s System time 0.44 s Max resident set: 1.98 GiB Test 2 (4.2 GiB, 44379959 unique lines of 6004): sort source | uniq > dest Wall time: 87.1 s User time: 352.3 s System time: 7.41 s Max resident set: 7.84 GiB tsv-uniq source > dest Wall time: 42.3 s User time: 40.7 s System time: 4.82 s Max resident set: 7.86 GiB line-dedup source dest Wall time: 23 s User time: 19.6 s System time 1.4 s Max resident set: 5.96 GiB The test files have a fractal distribution, with many lines having a few duplicates, and a few lines having many duplicates.
Re: Class instance alignment
On Saturday, 20 February 2021 at 05:44:33 UTC, kinke wrote: There's https://github.com/dlang/druntime/blob/728f1d9c3b7a37eba4d59ee2637fb924053cba6d/src/core/internal/traits.d#L261. Thanks! That's helpful. But AFAIK, the GC only guarantees an alignment of 16 and doesn't respect any overaligned members or alignment spec for the class. Well, that's just another reason not to use the GC for my current project, then: I'm using 256-bit AVX vectors extensively. That alignment limit really *needs* to be raised to at least 32 bytes, given that even DMD has some support for AVX. 64 bytes would be better, since AVX512 is going mainstream soon. And, 128 bytes is the largest common cache line size, I think? If raising the limit is considered unacceptable for some reason, then trying to allocate something with an unsupported alignment should be an error instead of just silently doing the wrong thing.
Class instance alignment
How can I get the alignment of a class instance? I know how to get the size: __traits(classInstanceSize, T) But, there doesn't appear to be any equivalent trait for the alignment. (Knowing the alignment of a class instance is required to correctly use core.lifetime.emplace, or do any sort of manual allocation of class instances. I know that I can conservatively estimate the alignment as nextPow2(__traits(classInstanceSize, T)), but that is nearly always a wasteful over-estimate. I've come across others' code online that simply assumes the alignment is size_t.alignof, but that's not right either because a class instance may contain SIMD vectors with higher alignments, or it may be aligned to a cache line size for efficient multi-threaded access.)
Re: Struct delegate access corruption
On Thursday, 18 February 2021 at 08:29:48 UTC, kinke wrote:
Nope, Paul is right, the copy ctors don't solve anything in
this regard. Simplest example I could come up with:
https://run.dlang.io/is/TgxyU3
I found that example very confusing, as it does not contain an
explicit copy constructor, violate memory safety, or output
incorrect results.
However, I experimented with it and I think figured out what
you're getting at? Copy constructors (and postblits) may not be
called for moves. I guess that makes sense...
///
import std.stdio : write, writeln;
struct S {
private const(S*) self, old = null;
this(bool refSelf) inout pure @trusted {
if(refSelf)
self =
}
@disable this(this) inout pure @safe;
this(scope ref inout(typeof(this)) that) inout pure @trusted {
self =
old =
}
void print() const @safe {
write();
if(old is null)
write(" (never copied)");
else
write(" (last copied from ", old, " to ", self, ")");
writeln(" in sync: ", self is );
}
}
S makeS() @safe {
S s = true;
s.print();
return s; // NRVO
}
void takeS(S s) @safe {
s.print();
}
void main() @safe {
S s = true;
s.print();
takeS(s);
takeS(makeS());
}
///
This works on LDC, but fails on DMD with output:
7FFF765249A0 (never copied) in sync: true
7FFF765249C0 (last copied from 7FFF765249A0 to 7FFF765249D0) in
sync: false
7FFF76524A00 (never copied) in sync: true
7FFF765249F0 (never copied) in sync: false
(It's weird that DMD runs the copy constructor with a destination
address that isn't even the real destination.)
Anyway, thanks for helping me understand, everyone.
Re: Struct delegate access corruption
On Wednesday, 17 February 2021 at 20:18:53 UTC, Paul Backus wrote: On Wednesday, 17 February 2021 at 19:42:00 UTC, tsbockman wrote: A copy constructor and opAssign can be used to update pointers that are relative to : https://dlang.org/spec/struct.html#struct-copy-constructor Unfortunately this is not enough, because the compiler is free to implicitly move struct instances in memory any time it wants to. See the bug report below for more details: https://issues.dlang.org/show_bug.cgi?id=17448 Until D gets move constructors, structs with interior pointers should be avoided. That bug is about postblits, this(this), not copy constructors: this(ref typeof(this)). Copy constructors were added to the language specifically to fix those sort of problems. From the spec: https://dlang.org/spec/struct.html#struct-postblit WARNING: The postblit is considered legacy and is not recommended for new code. Code should use copy constructors defined in the previous section. For backward compatibility reasons, a struct that explicitly defines both a copy constructor and a postblit will only use the postblit for implicit copying. However, if the postblit is disabled, the copy constructor will be used. If a struct defines a copy constructor (user-defined or generated) and has fields that define postblits, a deprecation will be issued, informing that the postblit will have priority over the copy constructor. However, since issue #17448 is still open I have posted a question to the issue report asking for clarification: https://issues.dlang.org/show_bug.cgi?id=17448#c39
Re: Struct delegate access corruption
On Wednesday, 17 February 2021 at 17:45:01 UTC, H. S. Teoh wrote:
I.e., the following is NOT a good idea:
struct S {
void delegate() member;
this() {
member =
}
private void impl();
}
because a pointer to S is stored inside S itself, so as soon as
S gets copied or moved, the delegate context pointer is no
longer valid (or else points to a different copy of the struct
than the one it will be called from).
A copy constructor and opAssign can be used to update pointers
that are relative to :
https://dlang.org/spec/struct.html#struct-copy-constructor
// The following program prints 9 with the copy constructor, or 7
if it is commented out:
module app;
import std.stdio;
struct Foo {
int[2] things;
private int* ptr;
this(const(int[2]) things...) inout pure @trusted nothrow
@nogc {
this.things = things;
ptr = &(this.things[0]);
}
// Copy constructor:
this(scope ref const(typeof(this)) that) inout pure @trusted
nothrow @nogc {
this.things = that.things;
this.ptr = this.things.ptr + (that.ptr - that.things.ptr);
}
// Defining a matching opAssign is a good idea, as well:
ref typeof(this) opAssign(scope ref const(typeof(this)) that)
return pure @trusted nothrow @nogc {
__ctor(that);
return this;
}
void choose(const(int) x) pure @trusted nothrow @nogc {
ptr = &(things[x]); }
@property ref inout(int) chosen() return inout pure @safe
nothrow @nogc {
return *ptr; }
}
void main() {
auto foo = Foo(3, 7);
foo.choose(1);
Foo bar = foo;
bar.things[1] = 9;
writeln(bar.chosen);
}
Re: Trying to reduce memory usage
On Wednesday, 17 February 2021 at 04:10:24 UTC, tsbockman wrote: On files small enough to fit in RAM, it is similar in speed to the other solutions posted, but less memory hungry. Memory consumption in this case is around (sourceFile.length + 32 * lineCount * 3 / 2) bytes. Run time is similar to other posted solutions: about 3 seconds per GiB on my desktop. Oops, I think the memory consumption should be (sourceFile.length + 32 * (lineCount + largestBucket.lineCount / 2)) bytes. (In the limit where everything ends up in one bucket, it's the same, but that shouldn't normally happen unless the entire file has only one unique line in it.)
Re: Trying to reduce memory usage
On Friday, 12 February 2021 at 01:23:14 UTC, Josh wrote: I'm trying to read in a text file that has many duplicated lines and output a file with all the duplicates removed. By the end of this code snippet, the memory usage is ~5x the size of the infile (which can be multiple GB each), and when this is in a loop the memory usage becomes unmanageable and often results in an OutOfMemory error or just a complete lock up of the system. Is there a way to reduce the memory usage of this code without sacrificing speed to any noticeable extent? My assumption is the .sort.uniq needs improving, but I can't think of an easier/not much slower way of doing it. I spent some time experimenting with this problem, and here is the best solution I found, assuming that perfect de-duplication is required. (I'll put the code up on GitHub / dub if anyone wants to have a look.) -- 0) Memory map the input file, so that the program can pass around slices to it directly without making copies. This also allows the OS to page it in and out of physical memory for us, even if it is too large to fit all at once. 1) Pre-compute the required space for all large data structures, even if an additional pass is required to do so. This makes the rest of the algorithm significantly more efficient with memory, time, and lines of code. 2) Do a top-level bucket sort of the file using a small (8-16 bit) hash into some scratch space. The target can be either in RAM, or in another memory-mapped file if we really need to minimize physical memory use. The small hash can be a few bits taken off the top of a larger hash (I used std.digest.murmurhash). The larger hash is cached for use later on, to accelerate string comparisons, avoid unnecessary I/O, and perhaps do another level of bucket sort. If there is too much data to put in physical memory all at once, be sure to copy the full text of each line into a region of the scratch file where it will be together with the other lines that share the same small hash. This is critical, as otherwise the string comparisons in the next step turn into slow random I/O. 3) For each bucket, sort, filter out duplicates, and write to the output file. Any sorting algorithm(s) may be used if all associated data fits in physical memory. If not, use a merge sort, whose access patterns won't thrash the disk too badly. 4) Manually release all large data structures, and delete the scratch file, if one was used. This is not difficult to do, since their life times are well-defined, and ensures that the program won't hang on to GiB of space any longer than necessary. -- I wrote an optimized implementation of this algorithm. It's fast, efficient, and really does work on files too large for physical memory. However, it is complicated at almost 800 lines. On files small enough to fit in RAM, it is similar in speed to the other solutions posted, but less memory hungry. Memory consumption in this case is around (sourceFile.length + 32 * lineCount * 3 / 2) bytes. Run time is similar to other posted solutions: about 3 seconds per GiB on my desktop. When using a memory-mapped scratch file to accommodate huge files, the physical memory required is around max(largestBucket.data.length + 32 * largestBucket.lineCount * 3 / 2, bucketCount * writeBufferSize) bytes. (Virtual address space consumption is far higher, and the OS will commit however much physical memory is available and not needed by other tasks.) The run time is however long it takes the disk to read the source file twice, write a (sourceFile.length + 32 * lineCount * 3 / 2) byte scratch file, read back the scratch file, and write the destination file. I tried it with a 38.8 GiB, 380_000_000 line file on a magnetic hard drive. It needed a 50.2 GiB scratch file and took about an hour (after much optimization and many bug fixes).
Re: std.typecons rebindable + tuple with const class gives warning
On Thursday, 4 February 2021 at 08:16:06 UTC, Saurabh Das wrote:
This code:
void main()
{
import std.typecons : rebindable, tuple;
const c = new C();
auto t = tuple(c.rebindable);
}
class C
{
}
When compiled with DMD 2.095.0 gives a warning:
Warning: struct Rebindable has method toHash, however it cannot
be called with const(Rebindable!(const(C))) this.
What is causing this? How can this warning be resolved?
`Rebindable!(C).toHash` forwards to `C.toHash`, which is
inherited from `Object.toHash`, which has the type: `nothrow
@trusted ulong()` according to pragma(msg,
typeof(Object.toHash));` That type signature forbids calling
`Object.toHash` on a mutable object. (An example of why this
might be a valid design choice, would be if computing the hash is
expensive and so the result is cached, which requires the freedom
to mutate the instance.)
To fix this problem, you need to do at least one of the following:
1) Make `c` mutable by declaring it with `auto` or `C` instead of
`const`. This is the only option if you cannot change the
definition of `C`.
2) If you can change `C`, you can override `toHash` in `C` with a
signature and implementation that do not require a mutable
object. Examples:
A) If you don't need associative array support from `C` or
its descendants at all, simply define `C.toHash` with more
permissive (to the caller) attributes, and crash if it gets
called unexpectedly:
class C
{
override size_t toHash() scope const pure @safe nothrow @nogc
{
assert(0, "Not implemented!");
}
}
B) If you want to support associative arrays by treating
every instance of `C` as a unique value:
class C
{
override size_t toHash() scope const pure @safe nothrow @nogc
{
static assert(C.sizeof == size_t.sizeof);
union Bits
{
const(C) self;
const(size_t) hash;
}
return Bits(this).hash;
}
// opEquals must always be defined such that is consistent
with toHash, such that this passes:
// if(this.opEquals(that))
// assert(this.toHash() == that.toHash());
override bool opEquals(Object that) scope const pure @safe
nothrow @nogc
{
return (this is that);
}
bool opEquals(scope const(C) that) scope const pure @safe
nothrow @nogc
{
return (this is that);
}
}
C) If you want to support associative arrays by treating
separate instances of `C` as equal based on the contents of their
data fields, then you will need to define appropriate `toHash`
and `opEquals` implementations:
class Point
{
int x, y;
override size_t toHash() scope const pure @safe nothrow @nogc
{
return size_t(x) * size_t(y);
}
override bool opEquals(Object that) scope const pure @safe
nothrow @nogc
{
if(auto thatPoint = cast(Point) that)
return opEquals(thatPoint);
else
return false;
}
bool opEquals(scope const(Point) that) scope const pure @safe
nothrow @nogc
{
return (this.x == that.x) && (this.y == that.y);
}
}
TLDR; Either make `c` mutable, or override/overload the `C`
associative array support methods `toHash` and `opEquals` to
support `const(C)` objects.
Re: F*cked by memory corruption after assiging value to associative array
On Thursday, 28 January 2021 at 20:17:09 UTC, frame wrote: On Thursday, 28 January 2021 at 19:22:16 UTC, tsbockman wrote: It is possible to get things sort of working with on Windows, anyway. I'm ok with it as long as the memory is not re-used by the GC. It seems that it can be prevented with addRoot() successfully. GC.addRoot is not enough by itself. Each GC needs to know about every single thread that may own or mutate any pointer to memory managed by that particular GC. If a GC doesn't know, memory may be prematurely freed, and therefore wrongly re-used. This is because when it scans memory for pointers to find out which memory is still in use, an untracked thread may be hiding a pointer on its stack or in registers, or it might move a pointer value from a location late in the scanning order to a location early in the scanning order while the GC is scanning the memory in between, such that the pointer value is not in either location *at the time the GC checks it*. You won't be able to test for this problem easily, because it is non-deterministic and depends upon the precise timing with which each thread is scheduled and memory is synchronized. But, it will probably still bite you later. If you were just manually creating additional threads unknown to the GC, you could tell the GC about them with core.thread.osthread.thread_attachThis and thread_detachThis. But, I don't know if those work right when there are multiple disconnected copies of D runtime running at the same time like this. The official solution is to get the GC proxy connected properly from each DLL to the EXE. This is still very broken on Windows in other ways (again, explained at my link), but it should at least prevent the race condition I described above, as well as being more efficient than running multiple GCs in parallel. Alternatively, you can design your APIs so that no pointer to GC memory is ever owned or mutated by any thread unknown to that GC. (This is the only option when working across language boundaries.)
Re: F*cked by memory corruption after assiging value to associative array
On Thursday, 28 January 2021 at 07:50:43 UTC, frame wrote: Under Linux everything is shared. Under Windows each DLL seems to run in its own thread, has its own rt_options and do not see any __gshared variable value. Its completely isolated and so I assume that also GC is. This stuff works correctly under Linux, and is quite broken in Windows. This has been known for years, but hasn't been fixed yet. This link for my other reply gives more details: https://forum.dlang.org/post/veeksndchoppftluj...@forum.dlang.org Also https://wiki.dlang.org/Win32_DLLs_in_D says: Each EXE and DLL will have their own gc instance. They each have their own GC instance because no one has fully fixed the problems discussed at my link, above, not because it's actually a good idea for them each to have their own GC instance. It is possible to get things sort of working with on Windows, anyway. But, this requires either: A) Following all the same rules that you would need to follow if you wanted to share D GCed memory with another thread written in C. (Just adding GC roots is not enough.) B) Ensuring that the GC proxy connections are properly established before doing anything else. This doesn't actually work correctly or reliably, but it might work well enough for your use case. Maybe.
Re: Why am I getting a dividing by zero error message
On Thursday, 28 January 2021 at 18:37:37 UTC, Ruby The Roobster
wrote:
Here is the output/input of the program:
Type in data for an egg:
Width: 3
Hight: 2
object.Error@(0): Integer Divide by Zero
...
Here is the source code:
import std.stdio;
import std.string;
void main(){
egg[1000] data;
data[0] = (new egg(0,0,"a"));
for(int i = 1;i<1000;i++)
{
...
data[i] = new egg(tempx,tempy,tempz);
for(int x = 0;x < i;x++)
{
int tempa;
int tempb;
double temp;
tempa = data[x].x;
if(tempa < 0)
tempa-=tempa;
tempb = data[x].y;
if(tempb < 0)
tempb-=tempb;
temp = tempa / tempb;
if(temp > highs)
{
highs = temp;
high = x;
}
}
...
Why is this happening? Does anybody know?
1) Outside the loops, you set `data[0] = (new Egg(0, 0, "a"))`
2) The outer `for(int i` loop, starts at `i = 1`, and so it never
overwrites the work of step (1).
3) The inner `for(int x` loop starts at `x = 0`, so its first
pass divides by `data[1].y`. Since you set that field equal to 0
in step (1), and both operands have type `uint`, the result is a
divide by zero error.
4) Also, `if(tempb < 0) tempb-=tempb;` will convert any negative
values into zeroes. Maybe you meant to write `if(tempb < 0) tempb
= -tempb;` to get the absolute value, instead?
There are many ways you could fix the program. (I won't try to
guess what the best way is, though, because it's not entirely
clear to me what this program is actually intended to do. As is,
it does some calculations, but doesn't try to output their
results in any way.)
Re: Initializing D runtime and executing module and TLS ctors for D libraries
On Thursday, 28 January 2021 at 00:58:17 UTC, rikki cattermole wrote: On 28/01/2021 1:16 PM, tsbockman wrote: The documentation build on dlang.org is broken. Check the source code or Adam D. Ruppe's dpldocs.info for the complete documentation: http://dpldocs.info/experimental-docs/core.thread.osthread.html Fixed: https://issues.dlang.org/show_bug.cgi?id=21309 I still don't see thread_setThis and thread_detachThis anywhere on the dlang.org copy.
Re: Initializing D runtime and executing module and TLS ctors for D libraries
On Sunday, 24 January 2021 at 03:59:26 UTC, Ali Çehreli wrote: I am surprised how much I had learned at that time and how much I've already forgotten. :/ For example, my PR involves thread_setThis, which seems to be history now: https://docarchives.dlang.io/v2.068.0/phobos/core_thread.html#.thread_setThis And thread_detachThis seems to be missing now: https://dlang.org/phobos/core_thread.html https://dlang.org/phobos/core_thread_osthread.html Documentation issue or is it not needed anymore? The documentation build on dlang.org is broken. Check the source code or Adam D. Ruppe's dpldocs.info for the complete documentation: http://dpldocs.info/experimental-docs/core.thread.osthread.html You'll find thread_setThis and thread_detachThis are still there.
Re: F*cked by memory corruption after assiging value to associative array
On Wednesday, 27 January 2021 at 18:09:39 UTC, frame wrote: I have no idea if there are multiple runtimes. I just use the mixin SimpleDllMain. But there must be multiple instances of GCs running because Another thread is running right now which I think is touching upon these same issues. Adam D. Ruppe explains some of what's going on: https://forum.dlang.org/post/veeksndchoppftluj...@forum.dlang.org Sadly, it looks like shared D DLLs are just kind of broken on Windows, unless you want to go the betterC route...
Re: F*cked by memory corruption after assiging value to associative array
On Wednesday, 27 January 2021 at 18:09:39 UTC, frame wrote: there must be multiple instances of GCs running because Sharing data between multiple threads that each use a different instance of the D GC will definitely not work right, because each GC will only know to pause the threads and scan the roots that it has been directly informed of. There is supposed to only be one instance of the D GC running per process. If you have more than one running then either you aren't linking and loading the DLLs correctly, or you have run into a serious bug in the D tooling. Or my debugger lied to me. I have found the gdb debugger on Linux often breaks horribly on my D code, especially when it is multi-threaded, and the debugger is only semi-usable. Maybe the Windows debugger is better now? (I wouldn't know, since I haven't used it in a while.) I think skepticism is warranted here.
Re: Deduct and return class type
On Saturday, 23 January 2021 at 05:54:09 UTC, Виталий Фадеев
wrote:
auto listFactory( T )( T data )
{
return new List!( T )( data );
}
This is the correct way to do it in D; it's annoying, I know.
(Personally, I usually shorten `listFactory` to just `list` to
make the name as similar as possible to the type name, without
actually causing a collision.)
Re: isCallable fails
On Friday, 22 January 2021 at 03:08:23 UTC, frame wrote: It's like asking your mechanican if he can guess what's causing the weird sound in the car and he replies with: why didn't you disassemble your car already? Cars are mass-produced copies of a relatively small number of rather similar designs, and mechanics' customers *pay them* to understand and fix their cars so that the customers don't have to. By contrast, computer programs are tremendously more diverse than car designs, you are not paying anyone here, and we participate here to pass our skills and knowledge on to others, not just to do their work for them. Nevertheless, I will use your analogy: You: My car shakes when I turn left. What do you think is wrong with it? Mechanic: Hard to say. Bring it in to the shop and I'll take a look. You: It's a simple question! I shouldn't have to bring it in to the shop. Mechanic: Fine. If it was my car, here's how I would go about finding the problem... (Detailed explanation follows.) Do that, and if you still can't tell what's wrong, come back and talk to me again. You: That's too much work. This is ridiculous. It's a simple question: just tell me what's wrong with my car! Mechanic: Why do you think I can diagnose the problem with *even less information than you have*? If it's so simple, why do you need help? You: Rude! I'm outa here...
Re: isCallable fails
On Friday, 22 January 2021 at 02:18:11 UTC, frame wrote: Anyway. It was a simple question not a prove that there is a bug in the compiler. isCallable is intended to either evaluate to a boolean, or give a clear compile-time error message. Since it did neither of those things for you, I suspect that something is wrong in the D project's code in addition to, or instead of, your code. You guys confuse a forum post question with a public examination. No, we're just used to helping each other get to the bottom of things like this. No one asked you for anything different from what we generally expect of each other in this context. This specific expectation mostly exists because, often, without a self-contained example we *truly don't know how to help you*. You are of course free to decide that our methods aren't worth your time, but please don't take it personally.
Re: isCallable fails
On Friday, 22 January 2021 at 00:59:32 UTC, tsbockman wrote: 2) Remove, simplify, or mock up/stub out (like for unit tests) some large chunk of the remaining code in the program, from a part of the program that I *think* is unrelated to the error. 3) Re-test the program. a) If the failure goes away, goto (2) and try again by removing a different part of the program. I should clarify here that when I say "try again" I mean from a backup of the last version of the reduction-in-progress that *did* demonstrate the failure.
Re: isCallable fails
On Thursday, 21 January 2021 at 05:20:27 UTC, frame wrote: I was not asking here to re-produce my code or debug since it cannot provide a snippet if I do not know what could cause the error. Generally, I don't need to know what causes an error in order to produce an MCVE. I have successfully done so on many occasions. It's annoying and slow, but usually very possible. If it seems like it's going to be really difficult for a particular bug, it's fine to try asking others without an MCVE first. But, you should think of MCVE discovery as a valuable skill, not an impossibility. My technique works like this: 0) Make note of the symptoms/message for the specific failure that I am trying to isolate. 1) Make a separate copy of the program's source code which I can freely destroy. 2) Remove, simplify, or mock up/stub out (like for unit tests) some large chunk of the remaining code in the program, from a part of the program that I *think* is unrelated to the error. 3) Re-test the program. a) If the failure goes away, goto (2) and try again by removing a different part of the program. b) If the failure is still present and I think there is more that could be removed or simplified, goto (2). c) If the failure is still present, and I don't see anything else to remove or simplify, then I'm done. Whatever is left of the source code (hopefully something short and non-proprietary) is my MCVE. It's sort of a guess-and-check binary search for the minimum code that triggers the error. This process can be quite tedious, but it often does not require already knowing the cause of the error. (dustmite is a tool that partially automates this.) Sometimes it's worth going through, because having an MCVE is *really* useful, especially for other people who don't know the rest of your project's mostly-unrelated code base and dependencies well.
Re: opIndex negative index?
On Thursday, 21 January 2021 at 14:00:28 UTC, cerjones wrote: I'm a bit unsure if this is reasonable or not. Thoughts? Signed indexes are fine, if there is a principled reason to use them. Just make sure that you are consistent by making `length`, `opDollar`, etc. use signed types, as well. Otherwise, you're laying a trap for users of the API who may accidentally write code with mixed signed-unsigned integer operations. That's a big problem, because mixed signed-unsigned integer operations are done incorrectly by default, often with no warnings, in D and some other C-like languages. Consistently using signed types for lengths and indexes will make the API incompatible with some Phobos (D's `std`) range code, but you should get a compile-time error message. Mixing signed and unsigned, on the other hand, is likely to result in subtly wrong code that compiles without error, but sometimes does the wrong thing.
Re: To switch GC from FIFO to LIFO paradigm.
On Friday, 15 January 2021 at 12:39:30 UTC, MGW wrote: GC cleans memory using the FIFO paradigm. Is it possible to switch GC to work using the LIFO paradigm? As others already said, the current GC isn't FIFO; it just scans everything once in a while a frees whatever it can, new or old. However, generational GCs are somewhat closer to LIFO than what we have now, which does provide some performance gains under common usage patterns. People have discussed adding a generational GC to D in the past, and I think the conclusion was that it requires pervasive write barriers (not the atomics kind), which leadership considers inappropriate for D for other reasons.
Re: writeln and write at CTFE
On Wednesday, 13 January 2021 at 23:38:54 UTC, oddp wrote:
Just two langs I use from time to time:
1) nim via forced ctfe; way faster than d by running through a
vm:
const foo = fib(42)
static:
echo "foobar"
2) crystal via macros:
{{ puts "foobar" }}
Another one would be zig via comptime, but never tried that.
Good to know... doesn't sound like it's some kind of standard
"2021" feature, though. Those are all lesser-known languages
(like D), not industry standard stuff like C++, Java, etc.
Re: Why many programmers don't like GC?
On Wednesday, 13 January 2021 at 21:56:58 UTC, mw wrote: I think this flexibility to mix GC & manual memory management is very unique in D. Actually I'm not sure if it can be done in other languages at all. Yes, this is one of the great things about D. There are miscellaneous problems with the D runtime and the D standard library that make it harder than it needs to be, though. Improvements I would like to see in the future: 1) Finalize std.experimental.allocator 2) Good, safe, flexible reference counting module in the standard library (this requires further development of dip1000 and the like, I think). 3) Upgrade core.thread to fully support @nogc. I shouldn't lose access to Thread.sleep and the like just because a thread isn't being monitored by the GC. 4) Single-threaded versions of various components related to memory management that are more efficient because they don't need to be thread-safe. For example, people say that reference counting is slow because incrementing and decrementing the count is an atomic operation, but most references will never be shared between threads so it is just a waste to use atomics. Still, all of these issues can be worked around today; D lacks high quality standards in this area more than it lacks necessary features.
Re: writeln and write at CTFE
On Wednesday, 13 January 2021 at 11:50:26 UTC, Andrey wrote: Today is 2021. Dlang still doesn't have ctfe write functions? You make it sound like D is behind the times. Is CTFE I/O a standard feature in other languages? How many other languages even have a CTFE feature comparable to D's?
Re: Why many programmers don't like GC?
On Wednesday, 13 January 2021 at 18:58:56 UTC, Marcone wrote: I've always heard programmers complain about Garbage Collector GC. But I never understood why they complain. What's bad about GC? SHORT VERSION: While garbage collection is great for many applications, GC also has some significant disadvantages. As a systems programming language, D attracts users who care about the disadvantages of GC, but D's design prevents it from mitigating the downsides of GC to the extent that less powerful languages like Java can. LONG VERSION: There are many different garbage collector designs, so not all of these criticisms apply to all garbage collector designs. Nevertheless, most every design suffers from at least some of these problems: 1) Freezes/pauses: many garbage collectors have to pause all other threads in the program for some significant time in order to collect. This takes somewhere between a few ms and a few s, depending on how much data needs to be scanned. For interactive applications, these pauses can cripple the program's performance. 60 FPS, or 16ms per frame, is a typical target rate for modern user interfaces, video playback, and games. When any given frame may be subject to a 5ms pause by the GC, the processing for *every* frame must be limited to what can be accomplished in 11ms, effectively wasting 30% of the available CPU performance of *all cores* on most frames, when no collection was necessary. But, 5ms is on the low end for GC pauses. Pauses longer than 16ms are common with some GC designs, guaranteeing dropped frames which are distracting and unpleasant to the user. For real-time applications such as hardware control systems, a pause that is too long could actually break something or injure someone. 2) Much higher memory consumption: all practical heap memory management schemes have some overhead - additional memory that is consumed by the manager itself, rather than the rest of the program's allocations. But, garbage collectors typically require three to ten times the size of the data for good performance, as opposed to two times or less for reference counting or manual management. 3) RAII doesn't work properly because the GC usually doesn't guarantee that destructors will be run for objects that it frees. I don't know why this is such a common limitation, but my guess is that it is due to one or both of: a) Collection often happens on a different thread from an object's allocation and construction. So, either all destructors must be thread-safe, or they just can't be run. b) A collection may occur at some awkward point where the program invariants depended on by non thread-safe destructors are violated, like inside of another destructor. It is certainly possible to retrofit correct resource management on top of a GC scheme, but the logic required to determine when to close file handles (for example) is often the same as the logic required to determine when memory can safely be freed, so why not just combine the two and skip the GC? GC has significant advantages, of course: simplicity (outside the GC itself), safety, and, for better designs, high throughput. But, the D users are more critical of GC than most for good reason: I) D is a systems programming language. While a good GC is the best choice for many, many applications, the kinds of applications for which GC is inappropriate usually require, or at least greatly benefit from, the full power of a systems programming language. So, the minority of programmers who have good reason to avoid GC tend to leave other languages like Java, C#, python, and JavaScript and come to us (or C, C++, Zig, or Rust). II) Historically, D's GC was embarrassingly bad compared to the state-of-the-art designs used by the JVM and .NET platforms. D's GC has improved quite a lot over the years, but it is not expected to ever catch up to the really good ones, because it is limited by other design decisions in the D language that prioritize efficient, easy-to-understand interoperability with C-style code over having the best possible GC. In particular, D's GC is a stop-the-world design that must pause *all* threads that may own any GC memory whenever it collects, and thus it fully suffers from problem (1) which I described earlier. Also, it used to have the additional problem that it leaked memory by design (not a bug). This is mostly fixed now, but for some reason the fix is not enabled by default?! https://dlang.org/spec/garbage.html#precise_gc (In before someone answers with, "Nothing. It works for me, so people who say it's bad are just stupid and don't profile.")
Re: Member variables in method are null when called as delegate from thread
On Wednesday, 13 January 2021 at 02:15:49 UTC, Tim wrote:
Basically, the program calls a function which modifies a
document in the database. If it is called form it's own class'
constructor, it works fine. If it is called by a thread, it
never returns.
...
class Caller : Thread{
void delegate() mFunc;
this(void delegate() func){
mFunc = func;
super();
start();
}
void loop(){
while(true){
mFunc();
}
}
}
class Callable{
MongoClient db;
Caller caller;
this(){
db = connectMongoDB("127.0.0.1");
foo();
caller = new Caller();
}
~this(){
db.cleanupConnections();
}
void foo(){
writeln("Started");
auto result =
db.getCollection("test.collection").findAndModify([
"state": "running"],
["$set": ["state": "stopped"]
]);
writeln(result);
writeln("Finished");
}
}
Note that if you are trying to debug a crash or hang of a program
by printing messages to the console, you need to flush stdout
multiple times in the vicinity of the problem, otherwise stdio's
buffering may make it appear as though the program crashed or
hung significantly earlier than it really did. (This is not a
merely theoretical problem; I trip over it frequently myself.)
Anyway, I think your real problem is that MongoClient is not
thread-safe. From the official vibe.d documentation
(https://vibed.org/api/vibe.db.mongo.mongo/connectMongoDB):
Note that the returned MongoClient uses a
vibe.core.connectionpool.ConnectionPool internally to create
and reuse connections as necessary. Thus, the MongoClient
instance can - and should - be shared among all fibers in a
thread by storing in in a thread local variable.
Note the "in a thread" part; you may only use a connection from
the same thread that opened it.
(Why? I'm not familiar with vibe.d's API or code base, so I don't
really know. But, I'd guess that the connection reuse mechanism
mentioned in the docs requires some of the information that you
might expect to be stored in the MongoClient instance itself to
instead end up in thread-local storage (whether native or
emulated). Or, there may simply be a manual "same thread" check
built into the DB operations to prevent data races, and the error
message isn't reaching you for some reason. `assert` messages
don't print in release mode, and I've found the gdb debugger for
D quite unreliable when trying to inspect multi-threaded code.)
Try moving the calls to `connectMongoDB` and `cleanupConnections`
into the same thread as `foo`. (I can't think of a good reason to
be doing these in the original thread, other than convenience.)
If you want to loop in multiple threads simultaneously, just open
a separate connection per thread, like the vibe.d docs suggest.
Re: Member variables in method are null when called as delegate from thread
On Tuesday, 12 January 2021 at 14:00:11 UTC, Steven Schveighoffer wrote: On 1/11/21 8:49 PM, tsbockman wrote: However, this re-ordering IS permitted to freely alter the behavior of your code from the perspective of OTHER threads. A likely cause of your bug is that the write to db by the constructor's thread is being committed to memory after the read of db by the MessageService thread. I don't think this is valid. You might be right, but your analysis below assumes the answers to a number of questions which aren't answered in the source code provided by the OP. Perhaps you are familiar with the implementations of the APIs in question, but I'm not and thought it unwise to assume too much, given that the whole reason we're having this discussion is that the code doesn't actually work... Regardless, the simple way to find out if I'm on the right track or not is just to protect access to Foo's fields with a mutex and see if that fixes the problem. If it does, then either it's a memory ordering issue like I suggested (or a code gen bug), and the mutex can be replaced with something more efficient if necessary. 1. the compiler MUST NOT reorder the storage of db to after you pass a delegate into an opaque function (array allocation). Is the function actually opaque, though? If the source code is available to the compiler for inlining (or maybe if it's marked `pure`?) then reordering is still allowed. 2. The CPU is not going to reorder, because the memory allocation is going to take a global lock anyway (mutex locks should ensure memory consistency). This is not a safe assumption. It is quite easy to design a thread-safe allocator that does not take a global lock for every allocation, and indeed *necessary* if you want it to scale well to heavy loads on high core count systems. Even if that's how it works today, I wouldn't write code that depends on this behavior, unless the language standard formally guaranteed it, because someone will change it sooner or later as core counts continue to climb. I can't ever imagine creating a thread (which is likely what MessageService ctor is doing) to not have a consistent memory with the creating thread on construction. It seems reasonable to assume that thread creation includes a write barrier somewhere, but what if MessageService is using an existing thread pool? The CPU would have to go out of its way to make it inconsistent. No, there are many levels of caching involved in the system, most of which are not shared by all cores. The CPU has to go out of its way to make memory appear consistent between cores, and this is expensive enough that it doesn't do so by default. That's why atomics and memory barriers exist, to tell the CPU to go out of its way to make things consistent. You often don't have to deal with these issues directly when using higher-level multi-threading APIs, but that's because they try to include the appropriate atomics/barriers internally, not because the CPU has to "go out of its way" to make things inconsistent.
Re: Member variables in method are null when called as delegate from thread
On Monday, 11 January 2021 at 00:43:00 UTC, Tim wrote: When MessageService calls the delegate for start, db is null. If I call start() in the Foo constructor it works just fine. Am I missing something here? Do delegates get called outside of their class context? I know I could just pass the db into start but I want to work out exactly why this is happening The compiler and the physical CPU are both allowed to change the order in which instructions are executed to something different from what your code specifies, as long as the visible, "official" results and effects of the chosen order of execution are the same as those of your specified code, FROM THE PERSPECTIVE OF THE EXECUTING THREAD. This is allowed so that the compiler can optimize to minimize negative "unofficial" effects such as the passage of time and memory consumption. However, this re-ordering IS permitted to freely alter the behavior of your code from the perspective of OTHER threads. A likely cause of your bug is that the write to db by the constructor's thread is being committed to memory after the read of db by the MessageService thread. In order to RELIABLY fix this kind of problem, you must correctly use the only commands which the compiler and CPU are NOT allowed to reorder with respect to other threads, namely atomic operations, memory barriers and synchronization primitives. A wide selection of these tools may be found in these D runtime library modules: core.sync: http://dpldocs.info/experimental-docs/core.sync.html core.atomic: http://dpldocs.info/experimental-docs/core.atomic.html core.thread: http://dpldocs.info/experimental-docs/core.thread.html (I recommend using Adam D. Ruppe's unofficial but superior rendering of the D runtime documentation at dpldocs.info rather than the official dlang.org rendering, as I found some necessary pieces of the documentation are just mysteriously missing from the offical version.) Be warned that most models of multi-threaded programming are difficult to implement correctly, as opposed to ALMOST correctly with subtle heisen-bugs. You should either stick to one of the known simple models like immutable message passing with GC, or do some studying before writing too much code. Here are some resources which I have found very helpful in learning to understand this topic, and to avoid its pitfalls: Short educational game: https://deadlockempire.github.io/ Tech talk by C++ expert Herb Sutter (D's core.atomic uses the C++ memory model): https://channel9.msdn.com/Shows/Going+Deep/Cpp-and-Beyond-2012-Herb-Sutter-atomic-Weapons-1-of-2 https://channel9.msdn.com/Shows/Going+Deep/Cpp-and-Beyond-2012-Herb-Sutter-atomic-Weapons-2-of-2 If you want to seriously dig into this, I suggest reviewing some or all of the content at the links above. If you're still confused about how to apply it in D, feel free to come back and ask for examples or code reviews. I'd rather not start with examples, though, because if you don't understand the rules and principles behind them, it's really easy to unknowingly introduce bugs into otherwise correct examples with seemingly innocent changes.
Re: Cannot implicitly convert expression of type const(string[]) to string[]
On Saturday, 9 January 2021 at 02:07:50 UTC, Ali Çehreli wrote:
The destination is immutable(char)[].
No, it's not. string[] means immutable(char)[][] - note the
second set of brackets.
Even though the source is 'const ref', other.text is a copy of
the slice object (the pointer and the length). Because the
elements are immutable, other.text cannot mutate value.text.
The elements are not immutable. Each element is an mutable slice
of immutable characters.
> Your code has a logical inconsistency
I don't see where. I think this is one of those too strict
cases of D where I unhappily slap a cast and move on.
The cast you propose breaks the type system. For example:
module app;
import std.stdio;
struct Value
{
int value;
string data;
string[] text;
}
void test(const ref Value value) @trusted // Don't do this!
{
Value other = void;
other.text = cast(string[]) value.text; // This cast is
@system for good reason.
other.text[0] = "Oops";
}
void main() @safe
{
immutable(Value) row = {
value: 10,
data: "ggg",
text: [ "Don't change me, bro!" ]
};
writeln(row.text);
test(row);
writeln(row.text);
}
Re: Cannot implicitly convert expression of type const(string[]) to string[]
On Friday, 8 January 2021 at 23:10:13 UTC, tsbockman wrote: 5) Tell the compiler that `other.text` may be used to mutate `row.text` by `const` from the `value` parameter of `test`. Do not do this unless that's really what you want! Oops, that should say: 5) Tell the compiler that `other.text` may be used to mutate `row.text` by *removing* `const` from the `value` parameter of `test`. Do not do this unless that's really what you want!
Re: Cannot implicitly convert expression of type const(string[]) to string[]
On Friday, 8 January 2021 at 20:43:37 UTC, Andrey wrote:
Hello,
struct Value
{
int value;
string data;
string[] text;
}
void test(const ref Value value)
{
Value other = void;
other.text = value.text;
}
void main()
{
Value row;
row.value = 10;
row.data = "ggg";
test(row);
}
I want to pass variable "row" inside function "test" as a read
only parameter.
Inside I create another variable and try to assign field "text".
On that line I get:
Error: cannot implicitly convert expression value.text of type
const(string[]) to string[].
1. How to assign correctly (and without dup/ugly cast())?
2. Or how to pass "row" correctly?
Your code has a logical inconsistency which the compiler has
rightly detected and flagged for you. You should actually fix the
inconsistency, not just shut the compiler up.
If the compiler accepted your code, it would be possible to later
modify the array pointed at by `row.text` through `other.text`,
because `other.text` is not `const`. This would violate your
stated desire to pass `row` as read-only.
There are various ways you can fix this, depending on your needs:
1) Change the type of `other` to `const(Value)` so that
`other.text` cannot be used to modify `row.text`.
2) Change the type of `Value.text` to `const(string[])` so
that it cannot be used to modify `row.text`.
3) Assign a copy (`dup`) of `row.text` to `other.text`
instead, so that mutating the array pointed to by `other.text`
will not modify the original read-only `row`.
4) Implement a copy-on-write scheme for `Value.text`.
5) Tell the compiler that `other.text` may be used to mutate
`row.text` by `const` from the `value` parameter of `test`. Do
not do this unless that's really what you want!
(1) and (2) are good options, if their limitations happen to fit
whatever you're really doing. (Note that either requires you to
write a constructor for `Value` since you won't be able to
directly write to `Value.text` anymore.)
Unless you are managing huge quantities of data with this `Value`
type, (3) is a perfectly reasonable thing to do and you shouldn't
feel guilty about the small "waste" of memory and CPU time for
the extra allocation. (4) is an option of last resort - not a
crazy one, but not worth the trouble if the amount of data is
small relative to system resources and/or the other activities of
the program.
(5) is simple but WRONG unless you are trying to create multiple
references to the same mutable data.
Re: D string to C struct fixed-size array
On Monday, 4 January 2021 at 02:17:33 UTC, bdh wrote: I'm pretty sure it's correct? Here's the D version: https://repo.or.cz/magickd.git/blob/e5d06e939:/source/magickd/core/c/image.d#l751 Here's the C version: http://hg.code.sf.net/p/graphicsmagick/code/file/a622095da492/magick/image.h#l683 /* XXX: haven't done private members yet, wonder if they're needed to * complete this */ Leaving out the private members of a struct changes its size, and sometimes its alignment. This will definitely break things if you ever allocate an instance yourself (including via the stack). If you only EVER access that type via pointers to instances allocated by an API which internally uses the complete, correct definition, and ALL of the private members are at the end of the struct, it should work - but those are two big "if"s, and too easy to unthinkingly violate later by accident. So, it is best to ensure that the struct definition matches precisely on both the D and C sides of the API, or at least enough to ensure that the size and alignment are the same.
Re: converting D's string to use with C API with unicode
On Sunday, 6 December 2020 at 02:07:10 UTC, Jack wrote:
On Saturday, 5 December 2020 at 23:31:31 UTC, tsbockman wrote:
On Saturday, 5 December 2020 at 21:55:13 UTC, Jack wrote:
wstring ws;
transcode(output[i], ws);
auto s = malloc(ws.length + 1);
if(!s) {
onOutOfMemoryError();
}
memcpy(s, ws.ptr, ws.length);
`ws.length` is the length in `wchar`s, but `memcpy` expects
the size in bytes. (This is because it takes `void*` pointers
as inputs, and so does not know the element type or its size.)
How do I get this size in bytes from wstring?
`ws.length * wchar.sizeof` should do it. `wstring` is just an
alias for `immutable(wchar[])`, and the `length` property is the
number of `wchar` elements in the slice.
Re: converting D's string to use with C API with unicode
On Saturday, 5 December 2020 at 21:55:13 UTC, Jack wrote:
my code now look like this, still there's a memory corrupt.
Could anyone help point out where is it?
...
foreach(i; 0..output.length) {
wstring ws;
transcode(output[i], ws);
auto s = malloc(ws.length + 1);
if(!s) {
onOutOfMemoryError();
}
memcpy(s, ws.ptr, ws.length);
`ws.length` is the length in `wchar`s, but `memcpy` expects the
size in bytes. (This is because it takes `void*` pointers as
inputs, and so does not know the element type or its size.)
Also, I think you need to manually zero-terminate `s`. You
allocate space to do so, but don't actually use it. (I believe
that transcode will only zero-terminate the destination if the
source argument is already zero-terminated.)
r.output[i] = cast(wchar*)s;
}
Re: converting D's string to use with C API with unicode
On Saturday, 5 December 2020 at 19:51:14 UTC, Jack wrote:
version(Windows) extern(C) export
struct C_ProcessResult
{
wchar*[] output;
In D, `T[]` (where T is some element type, `wchar*` in this case)
is a slice structure that bundles a length and a pointer
together. It is NOT the same thing as `T[]` in C. You will get
memory corruption if you try to use `T[]` directly when
interfacing with C.
Instead, you must use a bare pointer, plus a separate length/size
if the C API accepts one. I'm guessing that
`C_ProcessResult.output` should have type `wchar**`, but I can't
say for sure without seeing the Windows API documentation or C
header file in which the C structure is detailed.
bool ok;
}
struct ProcessResult
{
string[] output;
bool ok;
C_ProcessResult toCResult()
{
auto r = C_ProcessResult();
r.ok = this.ok; // just copy, no conversion needed
foreach(s; this.output)
r.output ~= cast(wchar*)s.ptr;
This is incorrect, and will corrupt memory. `cast(wchar*)` is a
reinterpret cast, and an invalid one at that. It says, "just take
my word for it, the data at the address stored in `s.ptr` is
UTF16 encoded." But, that's not true: the data is UTF8 encoded,
because `s` is a `string`, so this will thoroughly confuse things
and not do what you want at all. The text will be garbled and you
will likely trigger a buffer overrun on the C side of things.
What you need to do instead is allocate a separate array of
`wchar[]`, and then use the UTF8 to UTF16 conversion algorithm to
fill the new `wchar[]` array based on the `char` elements in `s`.
The conversion algorithm is non-trivial, but the `std.encoding`
module can do it for you.
return r;
}
}
Note also that when exchanging heap-allocated data (such as most
strings or arrays) with a C API, you must figure out who is
responsible for de-allocating the memory at the proper time - and
NOT BEFORE. If you allocate memory with D's GC (using `new` or
the slice concatenation operators `~` and `~=`), watch out that
you keep a reference to it alive on the D side until after the C
API is completely done with it. Otherwise, D's GC may not realize
it's still in use, and may de-allocate it early, causing memory
corruption in a way that is very difficult to debug.
Re: Is core.thread.osthread.Thread thread-safe?
On Saturday, 5 December 2020 at 18:58:13 UTC, Adam D. Ruppe wrote: On Saturday, 5 December 2020 at 18:48:19 UTC, tsbockman wrote: (The documentation for core.thread is broken right now, with dead links and at least one reference to an example that doesn't actually appear anywhere on the page.) im not sure about your other question but use my docs they actually work (...for the most part lol) http://dpldocs.info/core.thread Yes, yours are much better - they have a bunch of content which is mysteriously missing from the official docs build. Thanks!
Is core.thread.osthread.Thread thread-safe?
I want to do this:
import core.thread.osthread : Thread;
void main() {
shared(Thread) thread1 = new Thread({
// Do stuff for a while...
});
auto thread2 = new Thread({
// ...
if(thread1.isRunning)
// Do a thing.
else
// Do a different thing.
// ...
});
auto thread3 = new Thread({
// ...
if(thread1.isRunning)
// Do a thing.
else
// Do a different thing.
// ...
});
}
However, I get various compile-time errors when I try to write
code like this because nothing in the Thread API is annotated
`shared`. Are Thread objects really not thread-safe, and if so
what is the correct way to synchronize access to one?
(The documentation for core.thread is broken right now, with dead
links and at least one reference to an example that doesn't
actually appear anywhere on the page.)
Re: Why is this allowed
On Wednesday, 1 July 2020 at 20:05:51 UTC, tsbockman wrote:
If you want the compiler to stop you from accidentally keeping
references to stack variables past the end of their scope, you
need to annotate your functions @safe and compile with
-preview=dip1000: https://run.dlang.io/is/3VdDaN
Furthermore, the problem your example shows has nothing to do
with implicit static to dynamic array conversion, as without
@safe the same error can easily be committed with non-array
types: https://run.dlang.io/is/nBjibd
Hmm. Those run.dlang.io short links seem to allow editing of the
code, so I'd better paste it here for permanence:
// Compile with -preview=dip1000
struct Database {
int[] data;
void set(int[] _data) @safe {
data = _data;
}
}
void myFunc(ref Database db) @safe {
int[3] x;
db.set(x); // This is a compile-time error, as it should be.
}
Database theDB;
void main() {
myFunc(theDB);
}
// This version shows that the problem is not using @safe and
dip1000,
// not anything to do with arrays:
struct Database {
int* data;
void set(ref int _data) {
data = &_data;
}
}
void myFunc(ref Database db) {
int x;
db.set(x); // oops
}
Database theDB;
void main() {
myFunc(theDB);
}
Re: Why is this allowed
On Tuesday, 30 June 2020 at 16:36:45 UTC, H. S. Teoh wrote:
And on that note, this implicit static -> dynamic array
conversion is seriously a nasty misfeature that ought to be
killed with fire. It leads to bugs like this:
struct Database {
int[] data;
void set(int[] _data) {
data = _data;
}
}
void myFunc(ref Database db) {
int[3] x;
db.set(x); // oops
}
If you want the compiler to stop you from accidentally keeping
references to stack variables past the end of their scope, you
need to annotate your functions @safe and compile with
-preview=dip1000: https://run.dlang.io/is/3VdDaN
Furthermore, the problem your example shows has nothing to do
with implicit static to dynamic array conversion, as without
@safe the same error can easily be committed with non-array
types: https://run.dlang.io/is/nBjibd
Re: Python eval() equivalent in Dlang working in Runtime?
On Friday, 1 May 2020 at 18:07:54 UTC, H. S. Teoh wrote: On Fri, May 01, 2020 at 05:44:27PM +, tsbockman via Digitalmars-d-learn wrote: On Friday, 1 May 2020 at 15:42:54 UTC, Baby Beaker wrote: > There is a Python eval() equivalent in Dlang working in > Runtime? No, and there almost certainly never will be due to fundamental differences between the languages. Depending on your goal, the closest alternatives are using the string mixin language feature, writing a parser (std.conv or certain DUB packages can help), or embedding a scripting engine such as AngelScript or Squirrel into your program. [...] Actually, if you're willing to ship a working copy of dmd with your program, you *could* compile D code on-the-fly and dynamically load it as a dll/shared library. Good to know, but that's quite different from the eval() of interpreted languages with respect to both semantics and performance, so it's really another item for the "D alternatives to eval()" list.
Re: Python eval() equivalent in Dlang working in Runtime?
On Friday, 1 May 2020 at 15:42:54 UTC, Baby Beaker wrote: There is a Python eval() equivalent in Dlang working in Runtime? No, and there almost certainly never will be due to fundamental differences between the languages. Depending on your goal, the closest alternatives are using the string mixin language feature, writing a parser (std.conv or certain DUB packages can help), or embedding a scripting engine such as AngelScript or Squirrel into your program. What are you really trying to do?
Re: Enum conversion
On Tuesday, 21 April 2020 at 16:03:20 UTC, Russel Winder wrote: then which of these is the right way of accessing the value? cast(ubyte)ZoneNumber.One to!ubyte(ZoneNumber.One) Either is acceptable because there is no way that this operation can fail. Using a naked `cast` makes less work for the compiler and performs optimally with or without inlining, though. conversely what is the right way of going the other way: cast(ZoneNumber)1 to!ZoneNumber(1) Use `to` except where you can gaurantee that the input value maps to a valid enum member, because `cast` does not check your work: writeln(cast(ZoneNumber)17); // breaks the type system writeln(to!ZoneNumber(17)); // throws a ConvException: Value (17) does not match any member value of enum 'ZoneNumber' So, `cast` is faster, but unsafe. `to` is slower, but protects the enum's invariant.
Re: Checked!({short, ushort, byte, ubyte}, Throw): compilation fails
On Saturday, 18 April 2020 at 15:20:42 UTC, kdevel wrote: On Saturday, 18 April 2020 at 08:39:52 UTC, tsbockman wrote: https://code.dlang.org/packages/checkedint Hm. $ dub test Generating test runner configuration 'checkedint-test-library' for 'library' (library). Excluding package.d file from test due to https://issues.dlang.org/show_bug.cgi?id=11847 Performing "unittest" build using [...]/linux/bin64/dmd for x86_64. checkedint ~master: building configuration "checkedint-test-library"... Linking... Running ./checkedint-test-library source/checkedint/package.d(672): [unittest] unittest failure core.exception.AssertError@source/checkedint/package.d(672): unittest failure This should be fixed now. (It actually appears there was a bug in two of my unit tests *and* in *old* versions of the compiler, where the compiler bug allowed my bad unit tests to wrongly pass. Weird...)
Re: Checked!({short, ushort, byte, ubyte}, Throw): compilation fails
On Saturday, 18 April 2020 at 15:20:42 UTC, kdevel wrote: On Saturday, 18 April 2020 at 08:39:52 UTC, tsbockman wrote: https://code.dlang.org/packages/checkedint Hm. $ dub test Generating test runner configuration 'checkedint-test-library' for 'library' (library). Excluding package.d file from test due to https://issues.dlang.org/show_bug.cgi?id=11847 Performing "unittest" build using [...]/linux/bin64/dmd for x86_64. checkedint ~master: building configuration "checkedint-test-library"... Linking... Running ./checkedint-test-library source/checkedint/package.d(672): [unittest] unittest failure core.exception.AssertError@source/checkedint/package.d(672): unittest failure I think that's a compiler bug, not a bug in my checkedint package, as the unit tests pass on earlier D compiler versions (including GDC 2.075). Regardless, I'll try to track it down and either fix it or file a compiler bug report for it today.
Re: Checked!({short, ushort, byte, ubyte}, Throw): compilation fails
On Friday, 17 April 2020 at 21:25:34 UTC, kdevel wrote: A curiosity. Usually you cast into the type on the left. But Checked!(short, Throw) b = cast (Checked!(short, Throw)) a; does not compile: Error: template std.experimental.checkedint.Checked!(int, Throw).Checked.opCast cannot deduce function from argument types !(Checked!(short, Throw))(), candidates are: [...] One has to go over the underlying type? The author of std.experimental.checkedint hasn't (yet) written the opCast overload required to make that code work. It's just a feature missing from the code for that specific module in the standard library, not a language issue. The above code will throw when casting (before the assignment), because 65535 can't fit in a short. It's remarkable that the cast to the /underlying type/ throws. I would have expected that cast(short) a is equivalent to what actually must be written as cast(short) a.get You also get a deprecation message, about an integral promotion not being performed. I believe the result is correct and the warning can be ignored. So the warning is a bug? The deprecation message is a consequence of a (very annoying) language change made after std.experimental.checkedint was written. It's not really a bug, although someone should probably fix it anyway. Anyway, you might want to give my checkedint package on DUB a try, instead: https://code.dlang.org/packages/checkedint (It's the original candidate for std.experimental.checkedint. Alexandrescu didn't like it, so he rejected it and wrote his own. But, I still think mine has a better API, especially if you use the SmartInt types.)
Re: Checked!({short, ushort, byte, ubyte}, Throw): compilation fails
On Friday, 17 April 2020 at 21:25:34 UTC, kdevel wrote: On Friday, 17 April 2020 at 12:59:20 UTC, Simen Kjærås wrote: A curiosity. Usually you cast into the type on the left. But Checked!(short, Throw) b = cast (Checked!(short, Throw)) a; does not compile: Error: template std.experimental.checkedint.Checked!(int, Throw).Checked.opCast cannot deduce function from argument types !(Checked!(short, Throw))(), candidates are: [...] One has to go over the underlying type? The author of std.experimental.checkedint hasn't implemented the overload of opCast required to make your code work. It's just a missing feature. The above code will throw when casting (before the assignment), because 65535 can't fit in a short. It's remarkable that the cast to the /underlying type/ throws. I would have expected that cast(short) a is equivalent to what actually must be written as cast(short) a.get You also get a deprecation message, about an integral promotion not being performed. I believe the result is correct and the warning can be ignored. So the warning is a bug? The warning is caused by some (very annoying, but intentional) changes made to the compiler after std.experimental.checkedint was written. That module needs to be updated accordingly, but no one has done it yet. Anyway, you might want to take a look at my checkedint package on Dub: https://code.dlang.org/packages/checkedint In my opinion, the API is better - especially if you use SmartInt. (I submitted it for inclusion in the standard library originally, but Alexandrescu did not agree with my approach and so wrote his own version, which became std.experimental.checkedint.)
Re: Why D isn't the next "big thing" already
On Tuesday, 26 July 2016 at 15:11:00 UTC, llaine wrote: I'm using D since a few month now and I was wondering why people don't jump onto it that much and why it isn't the "big thing" already. D2 is under active development. Bugs get fixed, bottlenecks get optimized, and features get added or improved constantly. These changes don't usually seem like a big deal from one release to the next, but they add up quickly. Compared to what we have now, D2 was (in my opinion) unfinished junk a few years ago. The quality has improved a lot since then, but it will take time for the bad taste left in many people's mouthes by the unstable, incomplete early builds to be forgotten. This is a common problem for open source projects: the dev team is naturally more enthusiastic about the project than others, and also feels pressure to market it in order to attract testers, contributors, and donors. The result is that the product is declared "ready" before it really is by the standards of outsiders. People get fooled by the hype, try a half-baked build, and sour on the project. As long as the dev team continues to solve D2's problems faster than they're adding new ones, I expect that adoption will continue to increase.
Re: Creating a "fixed-range int" with opDispatch and/or alias this?
On Wednesday, 1 June 2016 at 19:59:51 UTC, Mark Isaacson wrote: I'm trying to create a type that for all intents and purposes behaves exactly like an int except that it limits its values to be within a certain range [a,b]. Theoretically, I would think this looks something like: ... It looks like opDispatch doesn't participate in resolution of operator overloads. Is there any way I can achieve my desired result? I know alias this forwards operations like +=, but with alias this I cannot wrap the operation to do the bounds checking. I think you would need to implement all of: * this(...) * opAssign(...) * opOpAssign(...) * opBinary(...) * opBinaryRight(...) * opUnary(...) FWIW, the fixed range int part of this question is just an example, I'm mostly just interested in whether this idea is possible without a lot of bloat/duplication. For a single type, I think the bloat is required. If you want to generate a lot of similar types, though, you could probably write a mixin template to generate the methods for you.
Re: union alignment
On Wednesday, 18 May 2016 at 01:46:37 UTC, tsbockman wrote: Shouldn't a union type always have an `alignof` at least as great as the `alignof` for its largest member? Apparently not; it's actually DMD and LDC that are wrong here: http://bugzilla.gdcproject.org/show_bug.cgi?id=226
Re: Enum that can be 0 or null
On Saturday, 21 May 2016 at 01:09:42 UTC, Alex Parrill wrote:
On Friday, 20 May 2016 at 22:10:51 UTC, tsbockman wrote:
If that's not a satisfactory answer, please show some specific
examples of code that you don't know how to make work without
VK_NULL_HANDLE so that I can propose a workaround.
Because, as I mentioned in the OP, for VK_NULL_HANDLE to work
like it does in C on 32-bit systems, it needs to be compatible
with both pointer types (VkDevice, VkInstance, etc) and 64-bit
integer types (VkFence, VkSemaphore, many others).
That is not code. As I said, there are many possible workarounds,
but it is hard to say which is best for you without actually
seeing your code.
As an example, if VK_NULL_HANDLE only ever needs to be assigned
to opaque types on the D side (that is, types that serve only as
an ID or address for communicating with the C side), you could do
this:
private struct VkNullHandle { }
enum VK_NULL_HANDLE = VkNullHandle.init;
mixin template VkHandle(bool dispatchable) {
static if (dispatchable || (size_t.sizeof == 8))
void* bits = null;
else
ulong bits = 0;
this(typeof(this) that) {
this.bits = that.bits; }
this(VkNullHandle that) {
this.bits = typeof(this.bits).init; }
ref typeof(this) opAssign(typeof(this) that) {
this.bits = that.bits;
return this;
}
ref typeof(this) opAssign(VkNullHandle that) {
this.bits = typeof(this.bits).init;
return this;
}
}
struct VkDevice { mixin VkHandle!true; }
struct VkInstance { mixin VkHandle!true; }
struct VkFence { mixin VkHandle!false; }
struct VkSemaphore { mixin VkHandle!false; }
void main() {
VkInstance a = VK_NULL_HANDLE;
VkFence b = VK_NULL_HANDLE;
}
(DPaste: https://dpaste.dzfl.pl/8f4ce39a907f )
The above is typesafe, and can easily be made compatible with a
typical C API, since C does no parameter-related name mangling.
Re: Enum that can be 0 or null
On Saturday, 21 May 2016 at 01:53:21 UTC, Mike Parker wrote: When binding to a C library, it's desirable to have as close to the original C API as possible so that you *can* drop C snippets and examples into D code and have them just work. Obviously, 100% compatibility is not possible, but it is possible to get pretty close most of the time. The reason that VK_NULL_HANDLE cannot (currently) be directly implemented in D it is not type safe. D's type safety is usually considered to be a positive selling point for the language; that's what I mean when I say that "you aren't *supposed* to be able to just copy-paste any random C snippet into D".
Re: Enum that can be 0 or null
On Friday, 20 May 2016 at 13:39:50 UTC, Alex Parrill wrote: (How) can I make a constant that is either zero or null depending on how it is used? In the general case, I don't think this is possible in D currently. (It will become possible if the proposed "multiple alias this" ever gets implemented, though: http://wiki.dlang.org/DIP66 ) However, various workarounds are possible depending on the context. Anyone have an idea on how to make this work? Why do you need to? Just use null for pointer types, and 0 for integers. D is not C; you aren't *supposed* to be able to just copy-paste any random C snippet into D and expect it to work without modification. If that's not a satisfactory answer, please show some specific examples of code that you don't know how to make work without VK_NULL_HANDLE so that I can propose a workaround.
Re: template instantiation
On Thursday, 19 May 2016 at 20:14:48 UTC, vit wrote: Is this a bug? Yes. I've filed a report: https://issues.dlang.org/show_bug.cgi?id=16050
union alignment
Is this a bug?
---
module app;
union Foo(A, B) {
A a;
B b;
}
void main() {
alias F = Foo!(double, ulong);
import std.stdio, std.algorithm;
writefln("sizeof: (%s >= %s) == %s",
F.sizeof, max(double.sizeof, ulong.sizeof),
F.sizeof >= max(double.sizeof, ulong.sizeof));
writefln("alignof: (%s >= %s) == %s",
F.alignof, max(double.alignof, ulong.alignof),
F.alignof >= max(double.alignof, ulong.alignof));
}
---
Shouldn't a union type always have an `alignof` at least as great
as the `alignof` for its largest member?
DMD 64:
sizeof: (8 >= 8) == true
alignof: (8 >= 8) == true
DMD 32:
sizeof: (8 >= 8) == true
alignof: (4 >= 4) == true
GDC 64:
sizeof: (8 >= 8) == true
alignof: (8 >= 8) == true
GDC 32:
sizeof: (8 >= 8) == true
alignof: (4 >= 8) == false <<= Bug?
LDC 64:
sizeof: (8 >= 8) == true
alignof: (8 >= 8) == true
LDC 32:
sizeof: (8 >= 8) == true
alignof: (4 >= 4) == true
Re: Request assistance converting C's #ifndef to D
On Friday, 13 May 2016 at 06:05:14 UTC, Andrew Edwards wrote:
Additionally, what's the best way to handle nested #ifdef's?
Those that appear inside structs, functions and the like... I
know that global #ifdef's are turned to version blocks but
versions blocks cannot be used inside classes, stucts,
functions, etc.
`static if` and `version()` can be nested, and both work just
fine inside classes, structs, functions, etc.:
module app;
version = withPrint;
struct A {
version(withPrint) {
class B {
static if(size_t.sizeof == 4) {
static void print() {
import std.stdio : writeln;
version(unittest) {
writeln("Hello, 32-bit world of unit tests!");
} else {
writeln("Hello, 32-bit world!");
}
}
} else {
static void print() {
import std.stdio : writeln;
version(unittest) {
writeln("Hello, presumably 64-bit world of unit
tests!");
} else {
writeln("Hello, presumably 64-bit world!");
}
}
}
}
}
}
void main() {
A.B.print();
}
(Try it on DPaste: https://dpaste.dzfl.pl/0fafe316f739)
Re: imports && -run [Bug?]
On Friday, 13 May 2016 at 01:16:36 UTC, Andrew Edwards wrote: command: dmd -run mod inc output: Undefined symbols for architecture x86_64: "_D3inc5printFZv", referenced from: __Dmain in mod.o ld: symbol(s) not found for architecture x86_64 clang: error: linker command failed with exit code 1 (use -v to see invocation) --- errorlevel 1 None of the variations of imports work when compiled with the -run switch but all work perfectly well without it. According to the DMD compiler manual, the -run switch only accepts a single source file: -run srcfile args... After the first source file, any further arguments passed to DMD will be interpreted as arguments to be passed to the program being run. Have you tried using DUB? It has lots of convenient features, including a `run` command that supports multiple source files: http://code.dlang.org/docs/commandline#run
Re: Request assistance converting C's #ifndef to D
On Friday, 13 May 2016 at 04:59:23 UTC, Andrew Edwards wrote:
Is there a way to reproduce the same behavior? Are there
reason's for not allowing this functionality or am I just
misunderstanding and going about things the wrong way?
[1] same result whether placed before or after the
#include/import statement.
though I'm not sure if such a thing will actually work, since
order-dependent declarations in D are a kind of dangerous
territory to tread on.
C's #include directive and D's import statement have
fundamentally different semantics:
C's is basically equivalent to copy-pasting the contents of the
included file at the location of the #include directive. Combined
with C's other pre-processor features, this means that a header
file can (potentially) do something *completely* different in
each file that includes it, because it is being recompiled from
scratch, in a new context.
D's import statement, on the other hand, merely makes one
module's symbols visible from another. Each module is only
compiled once, regardless of how many times it is imported. As
such, a module's semantics cannot be changed by each importer,
unlike in C.
So what is the current best practice when encountering such
statements during porting?
The affected code should be re-factored to use some other means
of configuration, such as:
1) Adding configuration parameters individually to each public
symbol as needed:
minA.d
=
import minB;
void main()
{
print!10();
}
minB.d
=
module minB;
void print(long MIN)()
{
import std.stdio: writeln;
writeln(MIN);
}
2) Enclosing the module in a `struct` or `class` which can be
instantiated and configured individually by each user, either at
run time or compile time:
minA.d
=
import minB;
immutable minB = MinB(10);
void main()
{
minB.print();
}
minB.d
=
module minB;
struct MinB {
long MIN;
void print()
{
import std.stdio: writeln;
writeln(MIN);
}
}
3) Enclosing the module in a `template` that can be instantiated
and configured by each user at compile time:
minA.d
=
import minB;
alias minB = MinB!10;
// Or: alias print = MinB!(10).print;
void main()
{
minB.print();
// Or: print();
}
minB.d
=
module minB;
template MinB(long MIN) {
void print()
{
import std.stdio: writeln;
writeln(MIN);
}
}
4) Passing `version` identifiers at compile time (using the
-version command line switch):
minA.d
=
import minB;
void main()
{
print!10();
}
minB.d
=
module minB;
version(Min10) {
enum MIN = 10;
} else {
enum MIN = 100;
}
void print()
{
import std.stdio: writeln;
writeln(MIN);
}
5) As a last resort, enclosing the module in a `mixin template`
that can be mixed in to each importing module with the required
configuration options at compile time:
minA.d
=
import minB;
mixin minB!10;
void main()
{
print();
}
minB.d
=
module minB;
mixin template minB(long MIN) {
void print()
{
import std.stdio: writeln;
writeln(MIN);
}
}
(5) is the most similar to how the C example works, but most of
the time it is a very inefficient solution, causing tons of code
bloat and redundant instantiations of the included functions and
data structures. Don't do this unless you can't find any other
reasonable way to do it.
Which of (1) to (4) is best just depends on exactly what the
pre-processor logic was being used for.
Re: How the heck is onInvalidMemoryOperationError() nothrow?
On Friday, 6 May 2016 at 02:57:59 UTC, Jeremy DeHaan wrote: In core.exception, we have a lovely function called onInvalidMemoryOperationError(). This function is marked as nothrow (plus other annotations). This function literally does nothing except throwing an error. How can it be marked as nothrow? From the spec (https://dlang.org/spec/function.html#nothrow-functions): "Nothrow functions can only throw exceptions derived from class Error." Throwing an Error is, for many purposes, considered fundamentally different than throwing an Exception because Error objects aren't meant to be caught by user code. Throwing an Error is supposed to just be a way of crashing the program with a nice message and stack trace. A key benefit of this distinction, is that it enables the use of `assert()` statements in `nothrow` code.
