Re: GC memory fragmentation
On Tuesday, 13 April 2021 at 12:30:13 UTC, tchaloupka wrote: I'm not so sure if pages of small objects (or large) that are not completely empty can be reused as a new bucket or only free pages can be reused. Does anyone has some insight of this? Some kind of GC memory dump and analyzer tool as mentioned `Diamond` would be of tremendous help to diagnose this.. As now it's possible to inject own GC to runtime, I've tried to compile debug version of the GC implementation and use that to check some debugging printouts with regard to small allocations. The code I've used to simulate the behavior (with `--DRT-gcopt=incPoolSize:0`): ```D // fill 1st pool (minsize is 1MB so it's 256 pages, two small objects in each ubyte[][512] pool1; foreach (i; 0..pool1.length) pool1[i] = new ubyte[2042]; // no more space in pool1 so new one is allocated and also filled ubyte[][512] pool2; foreach (i; 0..pool2.length) pool2[i] = new ubyte[2042]; // free up first small object from first pool first page pool1[0] = null; GC.collect(); // allocate another small object pool1[0] = new ubyte[2042]; ``` And shortened result is: ``` // first pool allocations GC::malloc(gcx = 0x83f6a0, size = 2044 bits = a, ti = TypeInfo_h) Gcx::allocPage(bin = 13) Gcx::allocPage(bin = 13) Gcx::newPool(npages = 1, minPages = 256) Gcx::allocPage(bin = 13) => p = 0x7f709e75c000 // second pool allocations GC::malloc(gcx = 0x83f6a0, size = 2044 bits = a, ti = TypeInfo_h) Gcx::allocPage(bin = 13) Gcx.fullcollect() preparing mark. scan stacks. scan roots[] scan ranges[] 0x51e4e0 .. 0x53d580 free'ing free'd 0 bytes, 0 pages from 1 pools recovered small pages = 0 Gcx::allocPage(bin = 13) Gcx::newPool(npages = 1, minPages = 256) Gcx::allocPage(bin = 13) => p = 0x7f709e65c000 // collection of first item of first pool first bin GC.fullCollect() Gcx.fullcollect() preparing mark. scan stacks. scan roots[] scan ranges[] 0x51e4e0 .. 0x53d580 free'ing collecting 0x7f709e75c000 free'd 0 bytes, 0 pages from 2 pools recovered small pages = 0 // and allocation to test where it'll allocate GC::malloc(gcx = 0x83f6a0, size = 2044 bits = a, ti = TypeInfo_h) recover pool => 0x83d1a0 => p = 0x7f709e75c000 ``` So if I'm not mistaken, bins in pools are reused when there is a free space (ast last allocation reuses address from first item in first bin in first pool). I've also tested big allocations with same result. But if this is so, I don't understand why in our case there is 95% free space in the GC and it still grows from time to time. At least rate of grow is minimized with `--DRT-gcopt=incPoolSize:0` Would have to log and analyze the behavior more with a custom GC build..k
Re: GC memory fragmentation
On Sunday, 11 April 2021 at 09:10:22 UTC, tchaloupka wrote: Hi, we're using vibe-d (on Linux) for a long running REST API server and have problem with constantly growing memory until system kills it with OOM killer. The Hunt Framework is also suffering from this. We are trying to make a simple example to illustrate it.
What are virtual functions?
I'm trying to understand, what virtual functions are. I found the [specs](https://dlang.org/spec/function.html#virtual-functions), but I can't make head or tail of it. - What is a `vtbl[]`? Obviously a function pointer table. But where to find this? The examples don't use it. Maybe something inside of the compiler? - Which of the eight functions in the example are virtual and and which not? OK B.abc is said to be virtual, as the comment states. But it seems never to be used. And why is it considered to be virtual? - There is also the term "covariant function", which is not explained. What is this? I'm asking, because I'm currently writing new docs for `std.format`. The [current (stable) docs of `formatValue`](https://dlang.org/phobos/std_format.html#formatValue) tell, that some `toString` versions are discouraged, but not for virtual functions. I would like to understand the reason for that, so I can put that properly into the new docs. The reasons, why it's better to not use these versions in normal functions is explained in the [changelog](https://dlang.org/changelog/2.079.0.html#toString). But I miss the reason, why virtual functions should still use them; probably, because I did not understand, what that is. Can you help me?
Re: What are virtual functions?
On Wednesday, 14 April 2021 at 13:43:20 UTC, Berni44 wrote: I'm asking, because I'm currently writing new docs for `std.format`. The [current (stable) docs of `formatValue`](https://dlang.org/phobos/std_format.html#formatValue) tell, that some `toString` versions are discouraged, but not for virtual functions. I would like to understand the reason for that, so I can put that properly into the new docs. The reasons, why it's better to not use these versions in normal functions is explained in the [changelog](https://dlang.org/changelog/2.079.0.html#toString). But I miss the reason, why virtual functions should still use them; probably, because I did not understand, what that is. The recommended `toString` versions are templates, but virtual functions can't be templates (because you can't have a function pointer that points to a template). So, the non-template versions are still considered acceptable for cases when `toString` has to be virtual--i.e., when you're overriding `Object.toString`.
Re: What are virtual functions?
On Wednesday, 14 April 2021 at 13:43:20 UTC, Berni44 wrote:
I'm trying to understand, what virtual functions are. I found
the
[specs](https://dlang.org/spec/function.html#virtual-functions), but I can't make head or tail of it.
- What is a `vtbl[]`? Obviously a function pointer table. But
where to find this? The examples don't use it. Maybe something
inside of the compiler?
- Which of the eight functions in the example are virtual and
and which not? OK B.abc is said to be virtual, as the comment
states. But it seems never to be used. And why is it considered
to be virtual?
- There is also the term "covariant function", which is not
explained. What is this?
Recommended reading:
https://en.wikipedia.org/wiki/Liskov_substitution_principle
This is all related to object-oriented programming and class
inheritance. Because we can put a subclass object into a
superclass variable (`class Child : Parent { }; Parent parent =
new Child;`), we cannot look at the *type* of an object variable
to decide which methods to call, because the object itself may be
of a subtype. As such, when we call a method `foo` on `Parent`,
the compiler looks up the class info in a pointer in the first 8
bytes of the object, finds the method pointer for `foo`, and
calls it with the object as a hidden parameter. (This is the
`this` field.)
So a virtual method is a method that is called "virtually", as
compared to directly by name, by turning the method name into a
function pointer call via the classinfo.
The list of function pointers for methods in the class info is
called the virtual method table, or vtable.
Covariance is related to the Liskov principle, and just means
that because `Child` can be treated as a `Parent`, a method that
returns `Parent` in the superclass can be overridden (its vtable
pointer replaced with a new one) by one that returns a `Child` in
the subclass. In other words, as "Child class replaces Parent
class", the "return type `Child`" can replace the "return type
`Parent`"; ie. in the child class you can use a child class of
the return type, ie. they "vary together" - covariance.
The opposite (contravariance) happens for parameters: if a
superclass method takes `Child`, the subclass can take `Parent`
instead - again, because `Child` can turn into `Parent` per
Liskov.
A different way to think about this is that method parameter and
return types form a contract that is defined by the superclass
and fulfilled by the subclass, and the subclass can relax the
call contract ("I demand from my caller") and restrict the return
contract ("I promise my caller"). Since the `Child`, by Liskov,
can do everything the `Parent` can do, demanding less - ie. a
`Parent` instead of a `Child` - keeps the superclass's call
contract valid, and promising more - ie. returning a `Child`
instead of a `Parent`, which may have additional capabilities -
keeps the superclass's return contract valid.
Re: GC memory fragmentation
On Wednesday, 14 April 2021 at 12:47:22 UTC, Heromyth wrote: On Sunday, 11 April 2021 at 09:10:22 UTC, tchaloupka wrote: Hi, we're using vibe-d (on Linux) for a long running REST API server and have problem with constantly growing memory until system kills it with OOM killer. The Hunt Framework is also suffering from this. We are trying to make a simple example to illustrate it. I think this is a rather serious problem for a language saying "GC is fine" 🌈. Is there an issue for this?
How do I create classes dynamically?
Maybe I am just too short in D, but I wanted to find out if it is
possible to create classes dynamically. My problem is, I just
don't know where to start reading. Maybe at mixin templates?
CreateClassWithName!("MyDynamicClassName");
should create the following class to work with dynamically:
class MyDynamicClassName {
this() { writeln("I was not written, but still I exist!"); }
}
So that I in the end by means of
MyDynamicClassName cls = new MyDynamicClassName;
can work with it.
Normally I would think of a macro first, but as far as I
understood D doesn't know macros (which I'm not really sad
about), but maybe I'm just thinking too complicated yet. I would
appreciate any hints, because as I said, I don't even know where
to start reading.
Re: How do I create classes dynamically?
On 4/14/21 4:38 PM, Mario wrote:
Maybe I am just too short in D, but I wanted to find out if it is
possible to create classes dynamically. My problem is, I just don't know
where to start reading. Maybe at mixin templates?
CreateClassWithName!("MyDynamicClassName");
should create the following class to work with dynamically:
class MyDynamicClassName {
  this() { writeln("I was not written, but still I exist!"); }
}
So that I in the end by means of
MyDynamicClassName cls = new MyDynamicClassName;
can work with it.
Normally I would think of a macro first, but as far as I understood D
doesn't know macros (which I'm not really sad about), but maybe I'm just
thinking too complicated yet. I would appreciate any hints, because as I
said, I don't even know where to start reading.
There is no good supported way to do this with the current runtime.
The way this is normally done is to use your own reflection system, and
build a mechanism to create classes based on their name.
There was an old way, using Object.factory, but I would not recommend
that, it's brittle and doesn't always work.
-Steve
Re: How do I create classes dynamically?
On Wednesday, 14 April 2021 at 20:47:57 UTC, Steven Schveighoffer wrote: There is no good supported way to do this with the current runtime. The way this is normally done is to use your own reflection system, and build a mechanism to create classes based on their name. There was an old way, using Object.factory, but I would not recommend that, it's brittle and doesn't always work. -Steve Too bad. Nevertheless, thank you for your quick answer to my question. -Mario
Re: How do I create classes dynamically?
On Wednesday, 14 April 2021 at 20:38:16 UTC, Mario wrote: Maybe I am just too short in D, but I wanted to find out if it is possible to create classes dynamically. My problem is, I just don't know where to start reading. Maybe at mixin templates? What exactly do you mean? Your goal is probably achievable through some means, like I can kinda make D classes with a scripting language even.
