Re: Livestreaming DConf?
On 05/11/2014 11:15 PM, John Carter via Digitalmars-d-announce wrote: The timezones being so different, I'm not sure livestreaming will help me... But I do plan to watch any videos of it as soon as they are up. It's the same problem for all Europeans (and many other), including me. I'm also looking forward to the videos, but I think livestreaming is a good thing for those who are able to follow them.
Re: Livestreaming DConf?
On Friday, 9 May 2014 at 19:48:20 UTC, Andrei Alexandrescu wrote: Hi folks, We at Facebook are very excited about the upcoming DConf 2014. In fact, so excited we're considering livestreaming the event for the benefit of the many of us who can't make it to Menlo Park, CA. Livestreaming entails additional costs so we're trying to assess the size of the online audience. Please follow up here and on twitter: https://twitter.com/D_Programming/status/464854296001933312 Thanks, Andrei Will that also mean that we won't need to wait several months for all videos to be published? :)
Re: Livestreaming DConf?
If you use a Google hangout on air with a laptop that is pointing at the presenter, I don't see much extra cost, except perhaps the cost of the bandwidth(free?). You probably even put a link on the dlang website that opens the hangout, with a different hangout for each session. If you are wanting to delay access to the videos you would have to go onto your youtube channel and make the videos private after the live streaming. On Fri, May 9, 2014 at 9:48 PM, Andrei Alexandrescu via Digitalmars-d-announce digitalmars-d-announce@puremagic.com wrote: Hi folks, We at Facebook are very excited about the upcoming DConf 2014. In fact, so excited we're considering livestreaming the event for the benefit of the many of us who can't make it to Menlo Park, CA. Livestreaming entails additional costs so we're trying to assess the size of the online audience. Please follow up here and on twitter: https://twitter.com/D_ Programming/status/464854296001933312 Thanks, Andrei
Re: Livestreaming DConf?
On Monday, 12 May 2014 at 07:21:39 UTC, Dicebot wrote: Will that also mean that we won't need to wait several months for all videos to be published? :) This is an interesting point. If memory serves last time one video was released per week, meaning that reddit/hn readers could properly digest and discuss them in turn. You'll have to assume that someone will rip the livestream and publish it all the very same day.
Re: Livestreaming DConf?
On 5/12/14, Dicebot via Digitalmars-d-announce digitalmars-d-announce@puremagic.com wrote: Will that also mean that we won't need to wait several months for all videos to be published? :) Better record the livestream then!
Re: More radical ideas about gc and reference counting
On 12 May 2014 10:50, Walter Bright via Digitalmars-d digitalmars-d@puremagic.com wrote: They are using Rust to write a safe and performant web browser while developing the language. Sure. But that browser hasn't been released yet. Consider that I've written safe and performant code in D, but others tell me I am unique and that I cannot expect average programmers to get it right. I repeatedly point out to Manu that he can write performant code in D that does not suffer from GC stalls, and he repeatedly replies that he has to work with average programmers who are not capable of doing this. What? You've never offered me a practical solution. You tell me I have to sacrifice any dependency on libraries (ridiculous), and all the modern conveniences and safety of automatic memory management to do it! Indeed, average programmers are a real-life practical problem, and it's not just them, I also appreciate the convenience offered personally. I only have one life, I *really* appreciate saving time on mundane and otherwise inconsequential tasks. Tell me some other reasons why I would be attracted to D? Take that all away, and what's the point? Automating some boilerplate is nice, but it's not the motivating reason for a wholesale adoption. You haven't told me how I can use the GC (or whatever memory management scheme, I really don't care) in the low frequency code (again, read: almost all code ever), and not have it interfere with the high frequency code. This is the fundamental problem with the GC. I can't use it ***ANYWHERE***, including any libraries I link. You can't isolate a GC, it's effects are not localised. If you could, maybe it'd be more workable... but even if it were properly concurrent and didn't halt the realtime threads when it collected, it's still totally impractical for any background thread to freeze for 10s-100s of ms while it runs a collect because I received a network packet which needs to be sent somewhere for processing or whatever. What do I do? So while I have no doubt that the Mozilla team may be very effective at using Rust and making it shine, that may not be transferable to the larger community.
Re: More radical ideas about gc and reference counting
On 12.05.2014 06:57, Ola Fosheim Grøstad ola.fosheim.grostad+dl...@gmail.com wrote: On Sunday, 11 May 2014 at 20:45:08 UTC, Rainer Schuetze wrote: On 11.05.2014 22:33, Walter Bright wrote: The Boehm collector cannot move objects around, the D one can. Oh it can? Really? Yes. D, for example, requires that objects not be self-referential for this reason. I don't think the GC would have problems with fixing up internal pointers to the object itself. self-referential is prohibited to allow moving structures by memcpy, e.g. as return value. Does this mean that you cannot safely implement something as basic as a circular linked list? Depends on your implementation, but I would not expect that the nodes of your linked list could be subject to being moved (mostly happens on the stack AFAICT), as pointers to the node from other nodes would become invalid, too.
Re: More radical ideas about gc and reference counting
On Monday, 12 May 2014 at 00:44:54 UTC, Andrei Alexandrescu wrote: On 5/11/14, 2:49 PM, ponce wrote: On Sunday, 11 May 2014 at 21:43:06 UTC, sclytrack wrote: There is very little use of @, it's mostly and ~. Heck I didn't find any @ while casually browsing the code. It's like they are not using it at all. Similarly in current C++ std::shared_ptr is barely there while std::unique_ptr is all over the place. The right tally here would include bald pointers as well. Also, I'm quite surprised by the confidence of this assertion. -- Andrei Well I wa'm probably much biased indeed. I have no data.
Re: [OT] Go officially won't get generics
On Wednesday, 7 May 2014 at 15:54:42 UTC, Paulo Pinto wrote: So the videos of the Gophercon 2014 are being made available. Rob Pike did the keynote. At the expected question about generics, his answer was There are no plans for generics. I said we're going to leave the language; we're done.. Discussion ongoing on HN, https://news.ycombinator.com/item?id=7708904 -- Paulo When Rob Pike was asked at the end of his keynote speech whether Go will support generics one day the majority of the audience was immediately laughing. If I were into Go this would make me think. I don't think parameterized types are necessary for a typed language to be good. But in Go it would be helpful to allow workarounds for modelling problems. As I like the way concurrency is done in Go I tried several times to find programming solutions that can manage without violating visibility rules or having duplicate variables or some other compromise. But there was simply no way.
Re: More radical ideas about gc and reference counting
On Monday, 12 May 2014 at 07:12:29 UTC, Manu via Digitalmars-d wrote: You haven't told me how I can use the GC (or whatever memory management scheme, I really don't care) in the low frequency code (again, read: almost all code ever), and not have it interfere with the high frequency code. You will like Don's talk this year ;) Game dev code in not just any low level code though.
Re: More radical ideas about gc and reference counting
On Monday, 12 May 2014 at 00:44:54 UTC, Andrei Alexandrescu wrote: On 5/11/14, 2:49 PM, ponce wrote: On Sunday, 11 May 2014 at 21:43:06 UTC, sclytrack wrote: There is very little use of @, it's mostly and ~. Heck I didn't find any @ while casually browsing the code. It's like they are not using it at all. Similarly in current C++ std::shared_ptr is barely there while std::unique_ptr is all over the place. The right tally here would include bald pointers as well. Also, I'm quite surprised by the confidence of this assertion. -- Andrei Well I was talking with no data indeed.
Re: More radical ideas about gc and reference counting
On 12 May 2014 17:24, Dicebot via Digitalmars-d digitalmars-d@puremagic.com wrote: On Monday, 12 May 2014 at 07:12:29 UTC, Manu via Digitalmars-d wrote: You haven't told me how I can use the GC (or whatever memory management scheme, I really don't care) in the low frequency code (again, read: almost all code ever), and not have it interfere with the high frequency code. You will like Don's talk this year ;) I'm super-disappointed I can't make it this year! We were evicted from our house, have to move, and I can't bail for a week and leave that all on my mrs while she kicks along the fulltime job :( Game dev code in not just any low level code though. It's not just low-level code. It's also very high level code. Gamedev is a unique (and extremely interesting) field which combines and tightly integrates almost every imaginable field of software engineering. I can't think of any that are left out, or any other industries that also share this property. I argue, if you can make a modern high-end game in a language, it can do anything.
Re: More radical ideas about gc and reference counting
On Monday, 12 May 2014 at 00:50:24 UTC, Walter Bright wrote:
On 5/11/2014 1:59 PM, Timon Gehr wrote:
Borrowed pointers are not even superficially similar to near*.
They are
compatible with everything else, because they can store data
that was borrowed
from anywhere else.
As long as those pointers don't escape. Am I right in that one
cannot store a borrowed pointer into a global data structure?
Perhaps:
struct Test {
n: 'static int, // [1]
m: int
}
static val: int = 123;
static mut t: Test = Test { n: 'static val, m: 0 };
fn main() {
unsafe { // [2]
let p = mut t.m;
*p = 456;
println!({} {}, *t.n, t.m); // prints: 123 456
}
}
[1]: In order to create a static instance of 'Test', the 'n'
field (which is a borrowed pointer) must be specified as to be
pointing at a static immutable (int) variable.
[2]: Any use of static mutable data requires the use of an
'unsafe' block (similar to @trusted in D)
Re: More radical ideas about gc and reference counting
On Monday, 12 May 2014 at 08:10:43 UTC, Tommi wrote:
Perhaps: [..]
Somewhat surprisingly to me, you can later on change the borrowed
pointer in the mutable static 'Test' to point at a mutable static
int:
struct Test {
n: 'static int
}
static old: int = 111;
static mut new: int = 222;
static mut t: Test = Test { n: 'static old };
fn main() {
unsafe {
println!({}, *t.n); // prints: 111
t.n = new; // Can point to a different static
println!({}, *t.n); // prints: 222
// ...but can't point to a local, e.g:
// let v = 123;
// t.n = v; // error: `v` does not live long enough
new = 333;
println!({}, *t.n); // prints: 333
}
}
Re: radical ideas about GC and ARC : need to be time driven?
On Monday, 12 May 2014 at 04:22:21 UTC, Marco Leise wrote:
On the positive side the talk about Rust, in particular how
reference counted pointers decay to borrowed pointers made me
think the same could be done for our scope args. A reference
counted slice with 3 machine words could decay to a 2 machine
word scoped slice. Most of my code at least just works on the
slices and doesn't keep a reference to them. A counter example
is when you have something like an XML parser - a use case
that D traditionally (see Tango) excelled in. The GC
environment and slices make it possible to replace string
copies with cheap slices into the original XML string.
Rust also has a solution for this: They have lifetime
annotations. D's scope could be extended to support something
similar:
scope(input) string getSlice(scope string input);
or with methods:
struct Xml {
scope(this) string getSlice();
}
scope(symbol) means, this value references/aliases (parts of)
the value referred to by symbol. The compiler can then make
sure it is never assigned to variables with longer lifetimes than
symbol.
Re: More radical ideas about gc and reference counting
On 5/12/2014 12:12 AM, Manu via Digitalmars-d wrote: What? You've never offered me a practical solution. I have, you've just rejected them. What do I do? 1. you can simply do C++ style memory management. shared_ptr, etc. 2. you can have the non-pausible code running in a thread that is not registered with the gc, so the gc won't pause it. This requires that this thread not allocate gc memory, but it can use gc memory allocated by other threads, as long as those other threads retain a root to it. 3. D allows you to create and use any memory management scheme you want. You are simply not locked into GC. For example, I rewrote my Empire game into D and it did not do any allocation at all - no GC, not even malloc. I know that you'll need to do allocation, I'm just pointing out that GC allocations and pauses are hardly inevitable. 4. for my part, I have implemented @nogc so you can track down gc usage in code. I have also been working towards refactoring Phobos to eliminate unnecessary GC allocations and provide alternatives that do not allocate GC memory. Unfortunately, these PR's just sit there. 5. you can divide your app into multiple processes that communicate via interprocess communication. One of them pausing will not pause the others. You can even do things like turn off the GC collections in those processes, and when they run out of memory just kill them and restart them. (This is not an absurd idea, I've heard of people doing that effectively.) 6. If you call C++ libs, they won't be allocating memory with the D GC. D code can call C++ code. If you run those C++ libs in separate threads, they won't get paused, either (see (2)). 7. The Warp program I wrote avoids GC pauses by allocating ephemeral memory with malloc/free, and (ironically) only using GC for persistent data structures that should never be free'd. Then, I just turned off GC collections, because they'd never free anything anyway. 8. you can disable and enable collections, and you can cause collections to be run at times when nothing is happening (like when the user has not input anything for a while). The point is, the fact that D has 'new' that allocates GC memory simply does not mean you are obliged to use it. The GC is not going to pause your program if you don't allocate with it. Nor will it ever run a collection at uncontrollable, random, asynchronous times.
Re: More radical ideas about gc and reference counting
On 5/11/2014 10:57 PM, Marco Leise wrote: Am Sun, 11 May 2014 17:50:25 -0700 schrieb Walter Bright newshou...@digitalmars.com: As long as those pointers don't escape. Am I right in that one cannot store a borrowed pointer into a global data structure? Right, and that's the point and entirely positive-to-do™. This means that a global data structure in Rust has to decide what memory allocation scheme its contents must use, and cannot (without tagging) mix memory allocation schemes. For example, let's say a compiler has internally a single hash table of strings. With a GC, those strings can be statically allocated, or on the GC heap, or anything with a lifetime longer than the table's. But I don't see how this could work in Rust.
Re: More radical ideas about gc and reference counting
Walter Bright: But I don't see how this could work in Rust. Ask it to competent Rust developers/programmers. Bye, bearophile
Re: More radical ideas about gc and reference counting
On Monday, 12 May 2014 at 08:45:56 UTC, Walter Bright wrote: On 5/12/2014 12:12 AM, Manu via Digitalmars-d wrote: What? You've never offered me a practical solution. I have, you've just rejected them. What do I do? 1. you can simply do C++ style memory management. shared_ptr, etc. 2. you can have the non-pausible code running in a thread that is not registered with the gc, so the gc won't pause it. This requires that this thread not allocate gc memory, but it can use gc memory allocated by other threads, as long as those other threads retain a root to it. 3. D allows you to create and use any memory management scheme you want. You are simply not locked into GC. For example, I rewrote my Empire game into D and it did not do any allocation at all - no GC, not even malloc. I know that you'll need to do allocation, I'm just pointing out that GC allocations and pauses are hardly inevitable. 4. for my part, I have implemented @nogc so you can track down gc usage in code. I have also been working towards refactoring Phobos to eliminate unnecessary GC allocations and provide alternatives that do not allocate GC memory. Unfortunately, these PR's just sit there. 5. you can divide your app into multiple processes that communicate via interprocess communication. One of them pausing will not pause the others. You can even do things like turn off the GC collections in those processes, and when they run out of memory just kill them and restart them. (This is not an absurd idea, I've heard of people doing that effectively.) 6. If you call C++ libs, they won't be allocating memory with the D GC. D code can call C++ code. If you run those C++ libs in separate threads, they won't get paused, either (see (2)). 7. The Warp program I wrote avoids GC pauses by allocating ephemeral memory with malloc/free, and (ironically) only using GC for persistent data structures that should never be free'd. Then, I just turned off GC collections, because they'd never free anything anyway. 8. you can disable and enable collections, and you can cause collections to be run at times when nothing is happening (like when the user has not input anything for a while). The point is, the fact that D has 'new' that allocates GC memory simply does not mean you are obliged to use it. The GC is not going to pause your program if you don't allocate with it. Nor will it ever run a collection at uncontrollable, random, asynchronous times. The only solutions to the libraries problem that I can see here require drastic separation of calls to said libraries from any even vaguely time critical code. This is quite restrictive. Yes, calling badly optimised libraries from a hot loop is a bad idea anyway, but the GC changes this from well it might take a little more time than usual, but we can spare a few nano-seconds and it'll show up easily in the profiler to it might, sometimes, cause the GC to run a full collection on our 3.96 / 4.00 GB heap with an associated half-second pause. And here we go again, I can't use that library, it's memory management scheme is incompatible with my needs, I'll have to rewrite it myself...
Re: More radical ideas about gc and reference counting
On Monday, 12 May 2014 at 09:05:39 UTC, John Colvin wrote: On Monday, 12 May 2014 at 08:45:56 UTC, Walter Bright wrote: The only solutions to the libraries problem that I can see here require drastic separation of calls to said libraries from any even vaguely time critical code. This is quite restrictive. I think this is more of library writing culture problem than engineering problem. High quality library shouldn't rely on any internal allocations at all, deferring this decision to user code. Otherwise you will eventually have problems, GC or not.
Re: More radical ideas about gc and reference counting
On Monday, 12 May 2014 at 09:05:39 UTC, John Colvin wrote: On Monday, 12 May 2014 at 08:45:56 UTC, Walter Bright wrote: On 5/12/2014 12:12 AM, Manu via Digitalmars-d wrote: What? You've never offered me a practical solution. I have, you've just rejected them. What do I do? 1. you can simply do C++ style memory management. shared_ptr, etc. 2. you can have the non-pausible code running in a thread that is not registered with the gc, so the gc won't pause it. This requires that this thread not allocate gc memory, but it can use gc memory allocated by other threads, as long as those other threads retain a root to it. 3. D allows you to create and use any memory management scheme you want. You are simply not locked into GC. For example, I rewrote my Empire game into D and it did not do any allocation at all - no GC, not even malloc. I know that you'll need to do allocation, I'm just pointing out that GC allocations and pauses are hardly inevitable. 4. for my part, I have implemented @nogc so you can track down gc usage in code. I have also been working towards refactoring Phobos to eliminate unnecessary GC allocations and provide alternatives that do not allocate GC memory. Unfortunately, these PR's just sit there. 5. you can divide your app into multiple processes that communicate via interprocess communication. One of them pausing will not pause the others. You can even do things like turn off the GC collections in those processes, and when they run out of memory just kill them and restart them. (This is not an absurd idea, I've heard of people doing that effectively.) 6. If you call C++ libs, they won't be allocating memory with the D GC. D code can call C++ code. If you run those C++ libs in separate threads, they won't get paused, either (see (2)). 7. The Warp program I wrote avoids GC pauses by allocating ephemeral memory with malloc/free, and (ironically) only using GC for persistent data structures that should never be free'd. Then, I just turned off GC collections, because they'd never free anything anyway. 8. you can disable and enable collections, and you can cause collections to be run at times when nothing is happening (like when the user has not input anything for a while). The point is, the fact that D has 'new' that allocates GC memory simply does not mean you are obliged to use it. The GC is not going to pause your program if you don't allocate with it. Nor will it ever run a collection at uncontrollable, random, asynchronous times. The only solutions to the libraries problem that I can see here require drastic separation of calls to said libraries from any even vaguely time critical code. This is quite restrictive. Yes, calling badly optimised libraries from a hot loop is a bad idea anyway, but the GC changes this from well it might take a little more time than usual, but we can spare a few nano-seconds and it'll show up easily in the profiler to it might, sometimes, cause the GC to run a full collection on our 3.96 / 4.00 GB heap with an associated half-second pause. And here we go again, I can't use that library, it's memory management scheme is incompatible with my needs, I'll have to rewrite it myself... A badly placed malloc() in library code can also trigger OS virtualization mechanisms and make processes being swapped out to disk, with the respective overhead in disk access and time spent on kernel code. So it is just not the we can spare a few nano-seconds. -- Paulo
Re: The Current Status of DQt
On Fri, 09 May 2014 09:56:09 + Kagamin via Digitalmars-d digitalmars-d@puremagic.com wrote: Please see this public service announcement: http://xkcd.com/1179/ Though it lists 20130227 as discouraged format, but it's a valid ISO 8601 format, and phobos Date.toISOString generates string in that format: http://dlang.org/phobos/std_datetime.html#.Date.toISOString Yes, it's supported, because it's standard, but it's preferred that toISOExtString be used precisely because the non-extended format is not only discouraged, but it's harder to read (which is probably why it's discouraged). - Jonathan M Davis
Re: More radical ideas about gc and reference counting
Am Sun, 11 May 2014 22:11:28 -0700 schrieb Walter Bright newshou...@digitalmars.com: But I thought ARC cannot be designed without GC to resolve cycles. It can be, there are various schemes to deal with that, including don't create cycles. GC is just one of them. http://en.wikipedia.org/wiki/Reference_counting#Dealing_with_reference_cycles Yes that article mentions: a) avoid creating them b) explicitly forbid reference cycles c) Judicious use of weak references d) manually track that data structure's lifetime e) tracing garbage collector f) adding to a root list all objects whose reference count is decremented to a non-zero value and periodically searching all objects reachable from those roots. To pick up your statement again: »Your proposal still relies on a GC to provide the memory safety, […] it is a hybrid ARC/GC system.« a) and b) let's assume never creating cycles is not a feasible option in a systems programming language c) and d) don't provide said memory safety e) and f) ARE tracing garbage collectors ergo: »But I thought ARC cannot be designed without GC to resolve cycles.« Your were arguing against Michel Fortin's proposal on the surface, when your requirement cannot even be fulfilled theoretically it seems. Which could mean that you don't like the idea of replacing D's GC with an ARC solution. »This is the best horse I could find for the price. It is pretty fast and ...« »No, it still has four legs.« -- Marco
Re: More radical ideas about gc and reference counting
Am Mon, 12 May 2014 01:54:58 -0700 schrieb Walter Bright newshou...@digitalmars.com: On 5/11/2014 10:57 PM, Marco Leise wrote: Am Sun, 11 May 2014 17:50:25 -0700 schrieb Walter Bright newshou...@digitalmars.com: As long as those pointers don't escape. Am I right in that one cannot store a borrowed pointer into a global data structure? Right, and that's the point and entirely positive-to-do™. This means that a global data structure in Rust has to decide what memory allocation scheme its contents must use, and cannot (without tagging) mix memory allocation schemes. For example, let's say a compiler has internally a single hash table of strings. With a GC, those strings can be statically allocated, or on the GC heap, or anything with a lifetime longer than the table's. But I don't see how this could work in Rust. :( Good question. I have no idea. -- Marco
Re: More radical ideas about gc and reference counting
On Sunday, 11 May 2014 at 21:43:06 UTC, sclytrack wrote: I like this owner/unique, borrow thing. @ is managed (currently reference counted) ~ is owner is borrow I like it too. But a few notes: 1) The managed pointer @T has been deprecated and you should use the standard library types GcT and RcT instead. 2) The owned pointer ~T has been largely removed from the language and you should use the standard library type BoxT instead. The basic idea is that if a function needs to have ownership of its argument, the function should take its argument by value. And if the function doesn't need the ownership, it should take its argument either by a mutable or immutable reference (they don't like to call it borrowed pointer anymore, it's called simply a reference now). Owned types get moved by default when you pass them to a function that takes its argument by value. You call the 'clone' method to make a copy of a variable of an owned type.
Re: More radical ideas about gc and reference counting
Am Mon, 12 May 2014 09:32:58 + schrieb Paulo Pinto pj...@progtools.org: On Monday, 12 May 2014 at 09:05:39 UTC, John Colvin wrote: On Monday, 12 May 2014 at 08:45:56 UTC, Walter Bright wrote: On 5/12/2014 12:12 AM, Manu via Digitalmars-d wrote: What? You've never offered me a practical solution. I have, you've just rejected them. What do I do? 1. you can simply do C++ style memory management. shared_ptr, etc. 2. you can have the non-pausible code running in a thread that is not registered with the gc, so the gc won't pause it. This requires that this thread not allocate gc memory, but it can use gc memory allocated by other threads, as long as those other threads retain a root to it. 3. D allows you to create and use any memory management scheme you want. You are simply not locked into GC. For example, I rewrote my Empire game into D and it did not do any allocation at all - no GC, not even malloc. I know that you'll need to do allocation, I'm just pointing out that GC allocations and pauses are hardly inevitable. 4. for my part, I have implemented @nogc so you can track down gc usage in code. I have also been working towards refactoring Phobos to eliminate unnecessary GC allocations and provide alternatives that do not allocate GC memory. Unfortunately, these PR's just sit there. 5. you can divide your app into multiple processes that communicate via interprocess communication. One of them pausing will not pause the others. You can even do things like turn off the GC collections in those processes, and when they run out of memory just kill them and restart them. (This is not an absurd idea, I've heard of people doing that effectively.) 6. If you call C++ libs, they won't be allocating memory with the D GC. D code can call C++ code. If you run those C++ libs in separate threads, they won't get paused, either (see (2)). 7. The Warp program I wrote avoids GC pauses by allocating ephemeral memory with malloc/free, and (ironically) only using GC for persistent data structures that should never be free'd. Then, I just turned off GC collections, because they'd never free anything anyway. 8. you can disable and enable collections, and you can cause collections to be run at times when nothing is happening (like when the user has not input anything for a while). The point is, the fact that D has 'new' that allocates GC memory simply does not mean you are obliged to use it. The GC is not going to pause your program if you don't allocate with it. Nor will it ever run a collection at uncontrollable, random, asynchronous times. The only solutions to the libraries problem that I can see here require drastic separation of calls to said libraries from any even vaguely time critical code. This is quite restrictive. Yes, calling badly optimised libraries from a hot loop is a bad idea anyway, but the GC changes this from well it might take a little more time than usual, but we can spare a few nano-seconds and it'll show up easily in the profiler to it might, sometimes, cause the GC to run a full collection on our 3.96 / 4.00 GB heap with an associated half-second pause. And here we go again, I can't use that library, it's memory management scheme is incompatible with my needs, I'll have to rewrite it myself... A badly placed malloc() in library code can also trigger OS virtualization mechanisms and make processes being swapped out to disk, with the respective overhead in disk access and time spent on kernel code. So it is just not the we can spare a few nano-seconds. -- Paulo Yes, it could easily extend to a longer wait. I think we all know programs that hang while the system is swapping out. Don't let it get to that! A PC game would typically reduce caches or texture resolutions before running out of RAM. Linux has a threshold of free pages it tries to keep available at any time to satisfy occasional small allocations. http://www.science.unitn.it/~fiorella/guidelinux/tlk/node39.html All-in-all malloc is less likely to cause long pauses. It just allocates and doesn't ask itself if there might be dead memory to salvage to satisfy a request. Time will tell if all well written D libraries will be @nogc to move the question of allocations to the user. -- Marco
Re: More radical ideas about gc and reference counting
On Monday, 12 May 2014 at 10:51:33 UTC, Marco Leise wrote:
Time will tell if all well written D libraries will be @nogc
to move the question of allocations to the user.
If there was such a thing as weakly nogc then we would could
do this
//some library function
void foo(OR, IR)(OR o, IR i) @weak-nogc
{
//take things from i and put them in o
}
allocations would be possible during the execution of foo, but
*only* in the implementations of OR and IR, which means that the
developer gets the control and guarantees they need, but doesn't
have to explicitly pre-allocate (which might not even be
possible).
I don't see how it would work with UFCS though...
Re: More radical ideas about gc and reference counting
On 05/12/2014 02:50 AM, Walter Bright wrote: On 5/11/2014 1:59 PM, Timon Gehr wrote: On 05/11/2014 10:05 PM, Walter Bright wrote: That's clearly an additional benefit of the borrowed pointer notion. But have you examined generated Rust code for the cost of inc/dec? I haven't, but I don't see any way they could avoid this (very expensive) cost without borrowed pointers. Sure, but performance is the additional benefit. One constant theme in this thread, one I find baffling, is the regular dismissal of the performance implications of inc/dec. Irrelevant, I'm not doing that. (And again, reference counting is not the only allocation mechanism in Rust. AFAICT, most allocations use owned pointers.) Borrowed pointers are not necessary to support raw pointers - this can be (and is in some systems) supported by simply wrapping the raw pointer with a dummy reference count. ... I have no idea what this part is trying to bring across. The reason for borrowed pointers is performance. No, it is safety. Raw pointers give you all of the performance. Rust would be non-viable without them. True in that it would fail to meet its design goals. Rust provides a tracing garbage collector as well, so it is at least as viable as D regarding performance of safe memory management. (Probably more, actually, because it does not provide precision-unfriendly constructs such as undiscriminated unions.) I strongly suggest writing a snippet in [[insert your favorite proven technology RC language here]] and disassembling the result, and have a look at what inc/dec entails. ... I don't have trouble seeing the cost of reference counting. (And it's you who claimed that this is going to be the only memory allocation scheme in use in Rust code.) The thing is, if the compiler is capable of figuring out these lifetimes by examining the code, There are explicit lifetime annotations in function signatures. Yes, because the compiler cannot figure it out itself, so the programmer has to annotate. ... You are saying 'if the compiler is capable of figuring out these lifetimes by examining the code, then ...' and then you are saying that the compiler is incapable of figuring them out itself. What is it that we are arguing here? Are you saying the Rust compiler should infer all memory management automatically or that it cannot possibly do that, or something else? It is simply not true that type systems are inherently restricted to checking trivial properties. They can be made as strong as mathematical logic without much fuss. Again, Rust would not need borrowed pointers nor the annotations for them if this knowledge could be deduced by the compiler. Heck, if the compiler can deduce lifetimes accurately, It does not deduce anything. It checks that borrowed pointers do not outlive their source. Lifetime parameters are used to transport the required information across function signatures. you can get rid of GC and RC, and just have the compiler insert malloc/free in the right spots. ... That's a form of GC, and I already acknowledged that global region inference exists, but noted that this is not what is used. Note that there is a Java version that does this partway, sometimes it will replace a GC object with a stack allocated one if it is successful in deducing that the object lifetime does not exceed the lifetime of the function. ... I know. Also, inference is harder to control and less efficient than simply making the relevant information part of type signatures. http://en.wikipedia.org/wiki/Region-based_memory_management#Region_inference This work was completed in 1995[9] and integrated into the ML Kit, a version of ML based on region allocation in place of garbage collection. This permitted a direct comparison between the two on medium-sized test programs, yielding widely varying results (between 10 times faster and four times slower) depending on how region-friendly the program was; compile times, however, were on the order of minutes. Yes, one is and the other is @. No, actually currently one is and the other is RCT AFAIK. Then Rust changed again. The document I read on borrowed pointers was likely out of date, though it had no date on it. ... Yes, most documents on Rust are at least slightly out of date. RCT is not more general. It cannot refer to stack-allocated data, for instance. So there is no general pointer type that has an unbounded lifetime? ... How can it be general and have an unbounded lifetime and be safe? Sure, borrowing is very lightweight, but ultimately what is most important is that it solves the problem of multiple incompatible pointer types and makes the type system more expressive as well. Adding more pointer types makes a type system more expressive, by definition. ... No. Also, this is irrelevant, because I was highlighting the _importance_ of the fact that it does in this case. A function that uses none of the
Re: More radical ideas about gc and reference counting
Timon Gehr: (Probably more, actually, because it does not provide precision-unfriendly constructs such as undiscriminated unions.) I suggested to add an optional method named onGC to unions that if present is called at run-time by the GC to know what's the real type of stored data, to make tracing more precise. Bye, bearophile
Re: More radical ideas about gc and reference counting
On Sunday, 11 May 2014 at 18:18:41 UTC, Rainer Schuetze wrote: For a reasonable GC I currently see 2 possible directions: 1. Use a scheme that takes a snapshot of the heap, stack and registers at the moment of collection and do the actual collection in another thread/process while the application can continue to run. This is the way Leandro Lucarellas concurrent GC works (http://dconf.org/2013/talks/lucarella.html), but it relies on fork that doesn't exist on every OS/architecture. A manual copy of the memory won't scale to very large memory, though it might be compressed to possible pointers. Worst case it will need twice as much memory as the current heap. It would be very interesting how far we can push this model on the supported platforms. 2. Change the compiler to emit (library defined) write barriers for modifications of (possible) pointers. This will allow to experiment with more sophisticated GC algorithms (if the write barrier is written in D, we might also need pointers without barriers to implement it). I know Walter is against this, and I am also not sure if this adds acceptable overhead, but we don't have proof of the opposite, too. As we all know, the usual eager reference counting with atomic operations is not memory-safe, so my current favorite is concurrent buffered reference counting (see chapter 18.2/3 The garbage collection handbook by Richard Jones et al): reference count modifications are not performed directly by the write barrier, but it just logs the operation into a thread local buffer. This is then processed by a collector thread which also detects cycles (only on candidates which had their reference count decreased during the last cycle). Except for very large reference chains this scales with the number of executed pointer modifications and not with the heap size. I'm surprised that you didn't include: 3. Thread-local GC, isolated zones (restricting where references to objects of a particular heap can be placed), exempting certain threads from GC completely, ...
Kitchen Sales Ikea
This Forum is probably the best forum that i have ever used and i would just like to say how proud i am to be a member of this forum a href=http://www.cheapexdisplaykitchensale.co.uk; Kitchen Sales Ikea/a
Re: More radical ideas about gc and reference counting
On 05/12/2014 10:54 AM, Walter Bright wrote: On 5/11/2014 10:57 PM, Marco Leise wrote: Am Sun, 11 May 2014 17:50:25 -0700 schrieb Walter Bright newshou...@digitalmars.com: As long as those pointers don't escape. Am I right in that one cannot store a borrowed pointer into a global data structure? Right, and that's the point and entirely positive-to-do™. This means that a global data structure in Rust has to decide what memory allocation scheme its contents must use, Global variables are banned in Rust code outside of unsafe blocks. and cannot (without tagging) mix memory allocation schemes. ... Tagging won't help with all memory allocation schemes. For example, let's say a compiler has internally a single hash table of strings. With a GC, those strings can be statically allocated, or on the GC heap, or anything with a lifetime longer than the table's. But I don't see how this could work in Rust. It's possible if you don't make the table global. (OTOH in D this is not going to work at all.)
Re: More radical ideas about gc and reference counting
On Mon, 12 May 2014 03:39:12 -0400, Manu via Digitalmars-d digitalmars-d@puremagic.com wrote: On 12 May 2014 17:24, Dicebot via Digitalmars-d You will like Don's talk this year ;) I'm super-disappointed I can't make it this year! ?!! http://dconf.org/2014/talks/evans.html We were evicted from our house, have to move, and I can't bail for a week and leave that all on my mrs while she kicks along the fulltime job :( Oh that sucks... -Steve
Re: More radical ideas about gc and reference counting
On Sun, 11 May 2014 16:33:04 -0400, Walter Bright newshou...@digitalmars.com wrote: On 5/11/2014 2:48 AM, Benjamin Thaut wrote: Mostly percise doesn't help. Its either fully percise or beeing stuck with a impercise mark sweep. This is not correct. It helps because most of the false pointers will be in the heap, and the heap will be accurately scanned, nearly eliminating false references to garbage. It doesn't matter where the false pointers are. The largest issue with false pointers is not how many false pointers there are. It only matters how large the block is that it points at. The larger your blocks get, the more likely they are pointed at by the stack. On 32-bit systems, allocate 1/256th of your memory space (i.e. 16.5MB), and the likelihood of random data on the stack pointing at it is roughly 1/256. This problem is just about eliminated with 64-bit pointers. Yes. D, for example, requires that objects not be self-referential for this reason. As previously stated, self referencing does not preclude GC moving. This statement is simply false, you can self reference in D for objects. You cannot for structs, but not because of a possibility for the moving GC, but because of the requirement to be able to move a struct instance. And in fact, even if it's forbidden, requires is too strong a word -- there is no static or runtime prevention of this. -Steve
Re: More radical ideas about gc and reference counting
On 12 May 2014 18:45, Walter Bright via Digitalmars-d digitalmars-d@puremagic.com wrote: On 5/12/2014 12:12 AM, Manu via Digitalmars-d wrote: What? You've never offered me a practical solution. I have, you've just rejected them. What do I do? 1. you can simply do C++ style memory management. shared_ptr, etc. I already have C++. I don't want another one. 2. you can have the non-pausible code running in a thread that is not registered with the gc, so the gc won't pause it. This requires that this thread not allocate gc memory, but it can use gc memory allocated by other threads, as long as those other threads retain a root to it. It still sounds the same as manual memory management though in practise, like you say, the other thread must maintain a root to it, which means I need to manually retain it somehow, and when the worker thread finishes with it, it needs to send a signal or something back to say it's done so it can be released... it sounds more inconvenient than direct manual memory management in practise. Sounds slow too. Dec-ing a ref is certainly faster than inter-thread communication. This also makes library calls into effective RPC's if I can't call into them from the active threads. How long is a collect liable to take in the event the GC threads need to collect? Am I likely to lose my service threads for 100s of milliseconds at a time? I'll think on it, but I don't think there's anything practically applicable here, and it really sounds like it creates a lot more trouble and complexity than it addresses. 3. D allows you to create and use any memory management scheme you want. You are simply not locked into GC. For example, I rewrote my Empire game into D and it did not do any allocation at all - no GC, not even malloc. I know that you'll need to do allocation, I'm just pointing out that GC allocations and pauses are hardly inevitable. C++ lets me create any memory management scheme I like by the same argument. I lose all the parts of the language that implicitly depend on the GC, and 3rd party libs (that don't care about me and my project). Why isn't it a reasonable argument to say that not having access to libraries is completely unrealistic? You can't write modern software without extensive access to libraries. Period. I've said before, I don't want to be a second class citizen with access to only a subset of the language. 4. for my part, I have implemented @nogc so you can track down gc usage in code. I have also been working towards refactoring Phobos to eliminate unnecessary GC allocations and provide alternatives that do not allocate GC memory. Unfortunately, these PR's just sit there. The effort is appreciated, but it was never a solution. I said @nogc was the exact wrong approach to my situation right from the start, and I predicted that would be used as an argument the moment it appeared. Tracking down GC usage isn't helpful when it leads you to a lib call that you can't change. And again, eliminating useful and productive parts of the language is not a goal we should be shooting for. I'll find it useful in the high-performance realtime bits; ie, the bits that I typically disassemble and scrutinise after every compile. But that's not what we're discussing here. I'm happy with D for my realtime code, I have the low-level tools I need to make the real-time code run fast. @nogc is a little bonus that will allow to guarantee no sneaky allocations are finding their way into the fast code, and that might save a little time, but I never really saw that as a significant problem in the first place. What we're talking about is productivity, convenience and safety in the non-realtime code. The vast majority of code, that programmers spend most of their days working on. Consider it this way... why do you have all these features in D that cause implicit allocation if you don't feel they're useful and important parts of the language? Assuming you do feel they're important parts of the language, why do you feel it's okay to tell me I don't deserve access to them? Surely I'm *exactly* the target market for D...? High-pressure, intensive production environments, still depending exclusively on native code, with code teams often in the realm of 50-100, containing many juniors, aggressive schedules which can't afford to waste engineering hours... this is a code environment that's prone to MANY bugs, and countless wasted hours as a consequence. Convenience and safety are important to me... I don't know what you think I'm interested in D for if you think I should be happy to abandon a whole chunk of the language, just because I have a couple of realtime threads :/ 5. you can divide your app into multiple processes that communicate via interprocess communication. One of them pausing will not pause the others. You can even do things like turn off the GC collections in those processes, and when they run out of memory just kill them and restart them. (This is not
Re: More radical ideas about gc and reference counting
Manu: we are an industry in desperate need of salvation, it's LONG overdue, and I want something that actually works well for us, not a crappy set of compromises because the language has a fundamental incompatibility with my industry :/ Perhaps the game industry has to start the creation of a language designed for its needs, like the scientific people have done (Julia), the browser ones (Rust), the Web ones have done, etc. With lot of work in less than ten years you can have an usable language. Bye, bearophile
Re: More radical ideas about gc and reference counting
On Monday, 12 May 2014 at 08:45:56 UTC, Walter Bright wrote: 2. you can have the non-pausible code running in a thread that is not registered with the gc, so the gc won't pause it. This requires that this thread not allocate gc memory, but it can use gc memory allocated by other threads, as long as those other threads retain a root to it. This and @nogc is a very promising trend, but you should still be able to partion the GC search space. The key to controlled real time performance is to partition the search space, that goes for anything algorithmic; memory management inclusive. That applies to any scheme like owned, ARC, GC etc. It makes little sense having to trace everything if only the physics engine is the one churning memory like crazy. And fork() is not a solution without extensive structuring of allocations. Stuff like physics touch all pages the physics objects are onto like 100+ times per second, so you need to group allocations to pages based on usage patterns. (No point in forking if you get write traps on 50.000 pages the next time the physics engine run :-). 3. D allows you to create and use any memory management scheme you want. You are simply not locked into GC. For example, I rewrote my Empire game into D and it did not do any allocation at all - no GC, not even malloc. I know that you'll need to do allocation, I'm just pointing out that GC allocations and pauses are hardly inevitable. This is no doubt the best approach for a MMO client. You have a window on the world and cache as much as possible both to memory and disk. Basically get as much memory from the OS as you can hoard (with headroom set by heuristics) when your application has focus and release caches that are outside the window when you loose focus to another application. This means you need a dedicated runtime for games that can delay GC collection and eat into the caches when you are low on computational resources. You also need to distinguish between memory that is locked to RAM and memory that can swap. You should always lock memory for real time threads. So if you want to GC this, you need a GC that support multiple heaps. (Some hardware might also distinguish between RAM that is accessible by the GPU or that has different characteristics in areas such as persistence or performance.) 5. you can divide your app into multiple processes that communicate via interprocess communication. One of them pausing will not pause the others. You can even do things like turn off Why processes and not threads with their own local GC? 6. If you call C++ libs, they won't be allocating memory with the D GC. D code can call C++ code. If you run those C++ libs But what happens if that C++ code does new HeapStuff(D_allocated_memory) and then calls back to D? You cannot presume that C++ coders have the discipline to always allocate local memory from the stack, so basically you cannot GC collect while there are C++ functions on the stack. In order to get there the GC collector needs to understand the malloc heap and trace that one too. Auditing all C++ libraries I want to use is too much work, and tracing the malloc heap is too time consuming, so at the end of the day you'll get a more robust environment by only scanning (tracing) the stacks when there is only D function calls on the stack, with a precise collector. That means you need to partition the search space otherwise the collector might not run in time. Freezing the world is really ugly. Most applications are actually soft real time. Games are part hard real time, part soft real time. The difference between games and other applications is that there is less headroom so you have to do more work to make the glitches and stuttering occur sufficiently seldom to be perceived as acceptable by the end user. But games are not special.
Re: More radical ideas about gc and reference counting
On 5/12/2014 3:18 AM, Marco Leise wrote: Your were arguing against Michel Fortin's proposal on the surface, when your requirement cannot even be fulfilled theoretically it seems. Lots of people use ARC without a GC. Which could mean that you don't like the idea of replacing D's GC with an ARC solution. I don't like the idea of replacing D's GC with ARC. But for different reasons.
Re: More radical ideas about gc and reference counting
On 5/12/2014 5:15 AM, Timon Gehr wrote: On 05/12/2014 10:54 AM, Walter Bright wrote: On 5/11/2014 10:57 PM, Marco Leise wrote: Am Sun, 11 May 2014 17:50:25 -0700 schrieb Walter Bright newshou...@digitalmars.com: As long as those pointers don't escape. Am I right in that one cannot store a borrowed pointer into a global data structure? Right, and that's the point and entirely positive-to-do™. This means that a global data structure in Rust has to decide what memory allocation scheme its contents must use, Global variables are banned in Rust code outside of unsafe blocks. Global can also mean assigning through a reference passed as a parameter.
Re: 64-bit DMD for windows?
On Thursday, 15 December 2011 at 21:05:05 UTC, captaindet wrote: On 2011-12-15 04:47, torhu wrote: On 14.12.2011 12:54, dmd.20.browse...@xoxy.net wrote: Hi, Is there a 64-bit version of DMD for windows? The download page offers only an x86 version. Or am I reading too much into that? Cheers, buk There's not much you would need a 64-bit compiler for on Windows. What are you going to use it for? now what is this for a strange comment? you need 64bit for windows for the same reasons than for any other platform: accessing loads of mem. yes, for some this is really important! for me it is actually a dealbreaker - i'd love to use D for my scientific programming, but my datasets often reach several GB... my computer has 16GB and i intend to make use of them. det It is NOT just for memory addressing , wchi is very simple under 64 bit, but also 64 bit registers, 16 of them , not just 8 32 bit. I think there should be a 64 bit version.
Druntime regression
The latest git HEAD druntime has broken existing code: https://issues.dlang.org/show_bug.cgi?id=12738 :-( I'm no longer so sure it's a *regression*, strictly speaking, but it is definitely a breakage that's going to cause users pain. I.e., it better be in large bold print in the changelog for the next release if this change is going to stay. T -- The best way to destroy a cause is to defend it poorly.
Re: More radical ideas about gc and reference counting
On 13 May 2014 02:16, bearophile via Digitalmars-d digitalmars-d@puremagic.com wrote: Manu: we are an industry in desperate need of salvation, it's LONG overdue, and I want something that actually works well for us, not a crappy set of compromises because the language has a fundamental incompatibility with my industry :/ Perhaps the game industry has to start the creation of a language designed for its needs, like the scientific people have done (Julia), the browser ones (Rust), the Web ones have done, etc. With lot of work in less than ten years you can have an usable language. But D is *so close*... and I like it! _ I have to say that this discussion has certainly left me somewhat intrigued by Rust though. I've never given it a fair go because I find the syntax so distasteful and deterring. I wonder if there's a market for a rust fork that re-skin's the language ;)
Re: More radical ideas about gc and reference counting
On 5/12/2014 2:12 AM, Dicebot wrote: I think this is more of library writing culture problem than engineering problem. High quality library shouldn't rely on any internal allocations at all, deferring this decision to user code. Otherwise you will eventually have problems, GC or not. Consider my PR: https://github.com/D-Programming-Language/phobos/pull/2149 This is exactly what it does - it 'pushes' the decisions about allocating memory up out of the library to the user. I suspect a great deal of storage allocation can be removed from Phobos with this technique, without sacrificing performance, flexibility, or memory safety. (In fact, it improves on performance and flexibility!) I also agree with your larger point that if you are relying on an unknown library for time critical code, and that library was not designed with time criticality guarantees in mind, you're going to have nothing but trouble. Regardless of GC or RC.
Re: More radical ideas about gc and reference counting
On Monday, 12 May 2014 at 16:16:06 UTC, bearophile wrote: Perhaps the game industry has to start the creation of a language designed for its needs, like the scientific people have done (Julia), the browser ones (Rust), the Web ones have done, etc. With lot of work in less than ten years you can have an usable language. I don't think games are unique or special. Most games are even in the easy space by having mostly static data. Meaning the amount of unexpected dynamic data is pretty low. Games also have the luxury of redefining the requirements spec to match available technology. The games industry does however have its own culture and paradigms and fashions… With subcultures. However, most interactive applications will suffer from the same issues if you increase the load so that they run out of headroom. Even unix commands like find and grep have latency requirements if the interaction is to be pleasant. By good fortune find and grep haven't changed their interface for 40+ years, so they were designed for low performance CPUs. That does not mean that you cannot design a better find-like application today that will run into runtime related usability issues if you freeze the world. At the end of the day, a system level language should support key strategies used for writing performant system level code in a reliable manner. It should also not lock you to a specific runtime that you couldn't easily write yourself. It should also not lock you to a specific model of how to structure your code (like monitors). I am not even sure it should provide OS abstractions, because that is not really system level programming. That is unixy (Posix) programming. A system level programming language should be free of OS and modelling related legacy.
Re: More radical ideas about gc and reference counting
On Monday, 12 May 2014 at 17:03:18 UTC, Walter Bright wrote: On 5/12/2014 2:12 AM, Dicebot wrote: I think this is more of library writing culture problem than engineering problem. High quality library shouldn't rely on any internal allocations at all, deferring this decision to user code. Otherwise you will eventually have problems, GC or not. Consider my PR: https://github.com/D-Programming-Language/phobos/pull/2149 This is exactly what it does - it 'pushes' the decisions about allocating memory up out of the library to the user. I suspect a great deal of storage allocation can be removed from Phobos with this technique, without sacrificing performance, flexibility, or memory safety. (In fact, it improves on performance and flexibility!) We have already had discussion where I did state that current @nogc implementation is not robust enough and failed to explain the use case for weaker @nogc clearly. Conclusion was that we should return to this topic after Don's DConf talk ;)
Re: More radical ideas about gc and reference counting
On Monday, 12 May 2014 at 17:03:41 UTC, Manu via Digitalmars-d wrote: But D is *so close*... and I like it! _ I have to say that this discussion has certainly left me somewhat intrigued by Rust though. I've never given it a fair go because I find the syntax so distasteful and deterring. I wonder if there's a market for a rust fork that re-skin's the language ;) Right now D has practical benefit of being more stable and library rich. But switching to Rust eventually does seem tempting as I find foundations of their type system much closer to my beliefs about good coding practices. It lacks any good static reflection though. And this stuff is damn addictive when you try it of D caliber.
Re: More radical ideas about gc and reference counting
On 5/12/2014 4:35 AM, bearophile wrote: I suggested to add an optional method named onGC to unions that if present is called at run-time by the GC to know what's the real type of stored data, to make tracing more precise. Unions of pointers are so rare in actual code that treating them conservatively is not a big problem.
Re: More radical ideas about gc and reference counting
Walter Bright: Unions of pointers are so rare in actual code that treating them conservatively is not a big problem. std.variant.Algebraic is based on on std.variant.VariantN, and on std.variant.VariantN is based on an union, and often you use algebraic data types to represent trees and similar data structures that contain many references/pointers. Adding Adding an onGC() method to std.variant.VariantN you allow the GC to manage Algebraic well enough. Bye, bearophile
Re: More radical ideas about gc and reference counting
On 13 May 2014 03:17, Walter Bright via Digitalmars-d digitalmars-d@puremagic.com wrote: On 5/12/2014 4:35 AM, bearophile wrote: I suggested to add an optional method named onGC to unions that if present is called at run-time by the GC to know what's the real type of stored data, to make tracing more precise. Unions of pointers are so rare in actual code that treating them conservatively is not a big problem. I find it fairly common. I just searched through my code, and 7 out of 12 unions had pointers.
Re: More radical ideas about gc and reference counting
On 13 May 2014 03:14, Dicebot via Digitalmars-d digitalmars-d@puremagic.com wrote: On Monday, 12 May 2014 at 17:03:41 UTC, Manu via Digitalmars-d wrote: But D is *so close*... and I like it! _ I have to say that this discussion has certainly left me somewhat intrigued by Rust though. I've never given it a fair go because I find the syntax so distasteful and deterring. I wonder if there's a market for a rust fork that re-skin's the language ;) Right now D has practical benefit of being more stable and library rich. But switching to Rust eventually does seem tempting as I find foundations of their type system much closer to my beliefs about good coding practices. It lacks any good static reflection though. And this stuff is damn addictive when you try it of D caliber. They have a lot more work to do. There doesn't seem to be a useful windows compiler for a start... _
Re: More radical ideas about gc and reference counting
On 5/12/2014 10:31 AM, Manu via Digitalmars-d wrote: I just searched through my code, and 7 out of 12 unions had pointers. Relative number of objects with unions, not declarations with unions!
Re: More radical ideas about gc and reference counting
On 5/12/2014 10:07 AM, Dicebot wrote: We have already had discussion where I did state that current @nogc implementation is not robust enough and failed to explain the use case for weaker @nogc clearly. Conclusion was that we should return to this topic after Don's DConf talk ;) Sure - next week!
Re: More radical ideas about gc and reference counting
On 5/12/2014 10:25 AM, bearophile wrote: Walter Bright: Unions of pointers are so rare in actual code that treating them conservatively is not a big problem. std.variant.Algebraic is based on on std.variant.VariantN, and on std.variant.VariantN is based on an union, and often you use algebraic data types to represent trees and similar data structures that contain many references/pointers. Adding Adding an onGC() method to std.variant.VariantN you allow the GC to manage Algebraic well enough. BTW, the RTinfo can be used to discriminate unions.
Re: 64-bit DMD for windows?
On Monday, 12 May 2014 at 16:47:21 UTC, steven kladitis wrote: It is NOT just for memory addressing , wchi is very simple under 64 bit, but also 64 bit registers, 16 of them , not just 8 32 bit. I think there should be a 64 bit version. This topic is 3 years old, DMD can already generate 64-bit programs on Windows (although I don't think DMD itself is 64-bit).
Re: More radical ideas about gc and reference counting
Walter Bright: BTW, the RTinfo can be used to discriminate unions. I don't know if std.variant.VariantN is already using such RTinfo. I don't know much about RTinfo. Bye, bearophile
Re: More radical ideas about gc and reference counting
On 5/12/2014 7:46 AM, Steven Schveighoffer wrote: It doesn't matter where the false pointers are. The largest issue with false pointers is not how many false pointers there are. It only matters how large the block is that it points at. The larger your blocks get, the more likely they are pointed at by the stack. On 32-bit systems, allocate 1/256th of your memory space (i.e. 16.5MB), and the likelihood of random data on the stack pointing at it is roughly 1/256. This problem is just about eliminated with 64-bit pointers. Generally, it is a bad idea to allocate such large blocks on the GC heap. GC's work best when the size of the objects being allocated is very small relative to the size of the heap space. Fortunately, it's a mathematical inevitability that large allocations relative to the GC size are rare, and so it isn't much of a pain to handle them manually. And in fact, even if it's forbidden, requires is too strong a word -- there is no static or runtime prevention of this. It's still forbidden. Andrei wrote a template that will verify this at runtime, but I don't recall its name.
Re: More radical ideas about gc and reference counting
On 13 May 2014 03:44, Walter Bright via Digitalmars-d digitalmars-d@puremagic.com wrote: On 5/12/2014 10:31 AM, Manu via Digitalmars-d wrote: I just searched through my code, and 7 out of 12 unions had pointers. Relative number of objects with unions, not declarations with unions! Ah, well I have 3 different tree/graph structures with unions, and tree/graph nodes have a tendency to accumulate many instances.
dmd and pkg-config
Since dmd.conf has specific flags depending on word size generated by dmd, is there any way to know this state i.e. like an environment variable? pkg-config command is an easy way to give the compiler the right flags for an specific library. Now these flags are for both, 32 and 64 bit. If pkg-config was able to know this information, then will first search on the right library path by setting PKG_CONFIG_PATH environment variable, avoiding the use of --no-warn-search-mismatch flag and becoming in a faster linking. Another solution should be include the pkg-config functionality into the compiler itself. Regards, -- Jordi Sayol
Re: 64-bit DMD for windows?
On Monday, 12 May 2014 at 17:46:21 UTC, Kapps wrote: On Monday, 12 May 2014 at 16:47:21 UTC, steven kladitis wrote: It is NOT just for memory addressing , wchi is very simple under 64 bit, but also 64 bit registers, 16 of them , not just 8 32 bit. I think there should be a 64 bit version. This topic is 3 years old, DMD can already generate 64-bit programs on Windows (although I don't think DMD itself is 64-bit). I still only see 32 bit version for Windows. I admit I have a 32 bit laptop , over 10 years old :) . All other laptops and pcs I have are 64 bit processors. If anyone out there has a 64 bit; version 2.065 , for windows let me know.
Re: More radical ideas about gc and reference counting
On Monday, 12 May 2014 at 16:03:28 UTC, Manu via Digitalmars-d wrote: How long is a collect liable to take in the event the GC threads need to collect? Am I likely to lose my service threads for 100s of milliseconds at a time? I'll think on it, but I don't think there's anything practically applicable here, and it really sounds like it creates a lot more trouble and complexity than it addresses. Your concerns stem not as much from the speed concern of the GC, but from the freeze-the-world aspect of it. Would a concurrent collector not solve these issues? As http://msdn.microsoft.com/en-us/library/ee787088%28v=vs.110%29.aspx#concurrent_garbage_collection explains a little bit, the actual time your threads spend frozen should be little (but I admit I don't know exactly how little), and so long as you don't allocate too much during the collection itself (which you say you don't), you should be able to keep running your code during the collection. If it's not possible to implement concurrent collection in D (and it's already been shown it is possible), then I'd agree that ARC is very important. But depending on how little the stop-the-world time from a concurrent GC can get, perhaps this could work around some issues that you're desiring ARC for. A generational collector could help in theory with your high memory usage situations. I doubt you allocate a gigabyte each frame, so the actual generation 0 content should be fairly low. Much of your memory usage should be allocations that will not be freed for long periods of time, while the per-frame and other short allocations should be fast to collect as there aren't many of them. Depending on how tunable the GC is, I feel like it should be possible to get away with a GC even for soft real-time programs like games. The problem is it's hard to tell until we get a proper concurrent collector in D2, just like it's hard to tell how significant the impact of ARC is until we get an optimized implementation of it in the compiler. Neither of these is simple. I do quite like the idea of ARC, it's just something that someone would have to actually implement (well) in order to see how much of an impact it really has in D. For the truly low frequency situations, you could get away with a library type for ARC as well, and as you mentioned, for high frequency you would get around ARC regardless.
Re: More radical ideas about gc and reference counting
On Monday, 12 May 2014 at 17:52:18 UTC, Walter Bright wrote: On 5/12/2014 7:46 AM, Steven Schveighoffer wrote: pointing at it is roughly 1/256. This problem is just about eliminated with 64-bit pointers. Not generally true. This presumes that the heap is not in the lower region of the address space and that you don't use 64 bit ints on the stack. Generally, it is a bad idea to allocate such large blocks on the GC heap. GC's work best when the size of the objects being allocated is very small relative to the size of the heap space. Generally not true. This is a deficiency of not having a smart allocator / precise scanning that use available meta information properly (obtained statically or by profiling). Fortunately, it's a mathematical inevitability that large allocations relative to the GC size are rare, and so it isn't much of a pain to handle them manually. Programmer pain is not measured in number of instances, but in terms of model complexity. Ola.
Re: More radical ideas about gc and reference counting
On 5/12/14, 10:25 AM, bearophile wrote: Walter Bright: Unions of pointers are so rare in actual code that treating them conservatively is not a big problem. std.variant.Algebraic is based on on std.variant.VariantN, and on std.variant.VariantN is based on an union, and often you use algebraic data types to represent trees and similar data structures that contain many references/pointers. Adding Adding an onGC() method to std.variant.VariantN you allow the GC to manage Algebraic well enough. I, too, felt the need of onGC() - actually preGC() - in my allocators implementation. Specifically, a thread-local freelist would save a pointer to the root in thread-local storage (i.e. a traditional D global variable). That would thread through a number of free nodes available for allocation. When a GC cycle occurs, it's okay if the list stays referenced; the GC will consider it used and won't do anything in particular about it. However, the GC cycle is a good opportunity to clean these freelists and offer the memory for other size classes, seeing as the freelists may grow unreasonably large and then just hold memory for no good reason. A hook that nulls all freelist heads just as the collection process starts would be helpful. Andrei
Re: isUniformRNG
On 5/11/2014 8:16 AM, Joseph Rushton Wakeling via Digitalmars-d wrote:
On 11/05/14 05:58, Nick Sabalausky via Digitalmars-d wrote:
The seed doesn't need to be compromised for synchronized RNGs to fubar
security.
Before we go onto the detail of discussion, thanks very much for the
extensive explanation. I was slightly worried that my previous email
might have come across as dismissive of your (completely understandable)
concerns.
Oh, not at all. I've been finding the discussion rather interesting. :)
I'm actually quite keen that we can find a mutually agreeable
solution :-)
I agree. And I think our stances aren't quite as opposed as they may seem.
Sure, but this is a consequence of two things (i) CryptoRng is a value
type and (ii) it gets passed by value, not by reference.
In your example, obviously one can fix the problem by having the
function declared instead as
ubyte[] getRandomJunk(ref CryptoRng rand) { ... }
but I suspect you'd say (and I would agree) that this is inadequate:
it's relying on programmer virtue to ensure the correct behaviour, and
sooner or later someone will forget that ref. (It will also not
handle other cases, such as an entity that needs to internally store the
RNG, as we've discussed many times on the list with reference to e.g.
RandomSample or RandomCover.)
Right. Agreed on all points.
Obviously one _can_ solve the problem by the internal state variables of
the RNG being static, but I'd suggest to you that RNG-as-reference-type
(which doesn't necessarily need to mean class)
Yea, doesn't necessarily mean class, but if it is made a reference type
then class is likely the best option. For example, I'd typically regard
struct* in a D API as a code smell.
solves that particular
problem without the constraints that static internal variables have.
Pretty much agreed, but the only question is whether those constraints
of using static internals are good/bad/inconsequential:
For non-crypto RNGs:
While I've tended to think the usefulness of a library-provided RNG that
permits independent-but-identically-seeded instances is small and
debatable, through this discussion I have become sufficiently convinced
that they're worth permitting. Besides, even if there weren't need for
it, the downsides of permitting such a feature (as long as it's not
accident-prone) are minimal, if not zero.
So I'm fine going the class route (or otherwise reference-based) and
making internal state per-instance. Or even having a duplicate this RNG
with identical state function, if people want it.
I think we're pretty well agreed on non-crypto RNGs. Your stance is
convincing here.
For crypto-RNGs:
A crypto-RNG exists for exactly one reason only: To stretch the useful
lifetime of a limited source of truely non-deterministic
randomness/entropy (or nearly-true randomness, if that's the best
available). Because of this, any and all determinism is a concession,
not a feature (unlike for non-crypto deterministic-RNGs). Even *how* you
use it deliberately affects the internal state, not just how many times
you asked for a value. These things go all-out to throw any wrenches
they can into any sources of determinism. I was actually quite impressed :)
In fact, the seeding/reseeding is specifically defined to be completely
internal to the crypto-RNG (at least with Hash_DRBG anyway, probably
others) - the user *never* provides a seed - which intentionally makes
it that much harder for an application to use deterministic seeds (which
would compromise the security implications of the crypto-RNG, and
therefore defeat the whole point of using a crypto-RNG instead of a
normal RNG).
All this is because determinism is NOT what a crypto-RNG is for, it's
exactly what a crypto-RNG is specifically designed to fight against.
What all that implies: A crypto-RNG shouldn't *explicitly* provide a way
to get different instances with identical internal states, and
definitely shouldn't let it happen by accident. It's also under
absolutely no obligation whatsoever for relying on determinism to even
be possible at all (it would carry certain difficulties anyway).
Luckily though, that doesn't imply anything particularly profound. *If*
it's even possible to get identical crypto-RNGs at all, then as long you
have to work to do it (memcopying raw class data, providing a custom
entropy source that's written to be deterministic, or even using muliple
threads/processes, etc), then everything's all good.
Therefore, a class or otherwise reference-based approach is fine for
crypt-RNGs, too.
I think my preference would still be to keep the internal state static
here though (again, just speaking for crypto-RNGs only). As I've argued,
the determinism is a non-feature for crypto-RNGs (they deliberately
fight it every step of the way), and the shared state carries a couple
entropy-related benefits (Reseeding one, ie accumulating additional
fresh entropy,
Re: More radical ideas about gc and reference counting
12-May-2014 22:08, Andrei Alexandrescu пишет: On 5/12/14, 10:25 AM, bearophile wrote: A hook that nulls all freelist heads just as the collection process starts would be helpful. One word - weak pointers. Then head of freelist is weak and can be collected at whim. Andrei -- Dmitry Olshansky
Re: 64-bit DMD for windows?
On 5/12/2014 2:04 PM, steven kladitis wrote: On Monday, 12 May 2014 at 17:46:21 UTC, Kapps wrote: On Monday, 12 May 2014 at 16:47:21 UTC, steven kladitis wrote: It is NOT just for memory addressing , wchi is very simple under 64 bit, but also 64 bit registers, 16 of them , not just 8 32 bit. I think there should be a 64 bit version. This topic is 3 years old, DMD can already generate 64-bit programs on Windows (although I don't think DMD itself is 64-bit). I still only see 32 bit version for Windows. I admit I have a 32 bit laptop , over 10 years old :) . All other laptops and pcs I have are 64 bit processors. If anyone out there has a 64 bit; version 2.065 , for windows let me know. You don't need a 64-bit version: Compiling 64-bit programs doesn't require a 64-bit compiler. Just install VC++, use the DMD 2.065 Win installer, and then toss in the -m64 flag when compiling. Works fine. Doesn't matter if DMD itself is 32-bit.
Re: radical ideas about GC and ARC : need to be time driven?
On Sunday, 11 May 2014 at 05:16:26 UTC, Paulo Pinto wrote: This is what java.lang.ref.ReferenceQueue are for in Java, but one needs to be a GC expert on how to use it, otherwise it will hinder the GCs work. I think all memory-partitioning-related performance requires expert knowledge. If people care about performance and reliability they have to accept that they cannot blindly use abstractions or throw everything into the same bag. Java is probably a good example of how unrealistic it is to have a general programming language that does reasonable well in most domains. The outcome has not been everybody under the Sun umbrella, but a wide variety of Java runtime-solutions and special systems.
Re: More radical ideas about gc and reference counting
On 5/12/14, 11:17 AM, Dmitry Olshansky wrote: 12-May-2014 22:08, Andrei Alexandrescu пишет: On 5/12/14, 10:25 AM, bearophile wrote: A hook that nulls all freelist heads just as the collection process starts would be helpful. One word - weak pointers. Then head of freelist is weak and can be collected at whim. Of course. My point here is that here you need simpler support than full-blown weak pointers. -- Andrei
Re: More radical ideas about gc and reference counting
On 13 May 2014 04:07, Kapps via Digitalmars-d digitalmars-d@puremagic.com wrote: On Monday, 12 May 2014 at 16:03:28 UTC, Manu via Digitalmars-d wrote: How long is a collect liable to take in the event the GC threads need to collect? Am I likely to lose my service threads for 100s of milliseconds at a time? I'll think on it, but I don't think there's anything practically applicable here, and it really sounds like it creates a lot more trouble and complexity than it addresses. Your concerns stem not as much from the speed concern of the GC, but from the freeze-the-world aspect of it. Would a concurrent collector not solve these issues? I originally thought it would... but the more I think on it, I don't think it would make an awful lot of difference in practise. If the stalls were 'short' (like 1-5ms on the background threads, 500µs-1ms on the realtime threads), then maybe it would be workable, but I don't know that it would be even close to that? Also, I think it would be very difficult to implement on a machine without virtual memory, or much of an operating system in general? The problem remains that with no free memory, frequency of collection becomes so high, that it's extremely unlikely full collection so often would be better than ARC. As http://msdn.microsoft.com/en-us/library/ee787088%28v=vs.110%29.aspx#concurrent_garbage_collection explains a little bit, the actual time your threads spend frozen should be little (but I admit I don't know exactly how little), and so long as you don't allocate too much during the collection itself (which you say you don't), you should be able to keep running your code during the collection. If it's not possible to implement concurrent collection in D (and it's already been shown it is possible), then I'd agree that ARC is very important. But depending on how little the stop-the-world time from a concurrent GC can get, perhaps this could work around some issues that you're desiring ARC for. A generational collector could help in theory with your high memory usage situations. I doubt you allocate a gigabyte each frame, so the actual generation 0 content should be fairly low. Much of your memory usage should be allocations that will not be freed for long periods of time, while the per-frame and other short allocations should be fast to collect as there aren't many of them. Yeah, it would probably be better, if it's possible. Implementation needs to be considered from the perspective of embedded systems with no OS or MMU, and as little as 64mb of ram (the smallest modern systems). Mid-range systems are 512mb and no MMU. 'next-gen' systems are basically like little PC's with crappy OS's, so more likely a decent GC is possible on a ps4/xbone... but very few have the luxury of developing for just one system. It occurred to me earlier that things like strings might enjoy their own separate heap. And maybe some special logic for strings that outlived their scope to be actively returned to their heap rather than waiting for collection. If the heap were successfully broken down into a suite of sub-heaps, I have absolutely no idea how to make estimates about the performance of this system, and if it would approach an acceptable level. I'm skeptical it would, and it still won't decrease collection frequency. But I'd be happy to be surprised. Depending on how tunable the GC is, I feel like it should be possible to get away with a GC even for soft real-time programs like games. The problem is it's hard to tell until we get a proper concurrent collector in D2, just like it's hard to tell how significant the impact of ARC is until we get an optimized implementation of it in the compiler. Neither of these is simple. I do quite like the idea of ARC, it's just something that someone would have to actually implement (well) in order to see how much of an impact it really has in D. I understand the problem. The first hurdle is overcoming the hostility against it though. There is a severe prejudice. For the truly low frequency situations, you could get away with a library type for ARC as well, and as you mentioned, for high frequency you would get around ARC regardless. Yup.
Next step on reference counting topics
There's been a lot of talk lately regarding improving resource management for D, and I'd like to figure the next logical step to take. It seems clear that we have reached a collective impasse on a few fundamentals, and that more just talk about it all is at the point of diminishing returns. One action item that is hopefully useful to people of all viewpoints is to double down on library support, and see how far we can get and what insights we collect from the experience. For that I'm proposing we start real work toward a state-of-the-art std.refcounted module. It would include adapters for class, array, and pointer types, and should inform language improvements for qualifiers (i.e. the tail-const problem), copy elision, literals, operators, and such. Who wants to champion this effort? Thanks, Andrei
Re: More radical ideas about gc and reference counting
On Monday, 12 May 2014 at 18:07:51 UTC, Kapps wrote: Depending on how tunable the GC is, I feel like it should be possible to get away with a GC even for soft real-time programs like games. Even if you manage to make it work for game clients you also should make it work for low latency game servers, as code sharing is an important advantage. What a game/world server requires differs a lot, but highly dynamic and flexible worlds have to keep the physics to a single node (or tight cluster) for a region. That means you want to have as many players as possible tied to that node. In essence you want both performance, low latency, reliability, and little overhead in an evolutionary context (it has to support heavy modification over time). My gut feeling is that a runtime satisfying one game design will not satisfy another one as long as one insists on one global GC. In essence, it will never really work well. IMO, the same goes for ARC since RC does not perform well with multi-threading even when you use near optimal patterns and strategies. If ARC is only to be used where speed does not matter then you might as well use shared_ptr.
Re: More radical ideas about gc and reference counting
On 05/12/2014 06:37 PM, Walter Bright wrote: On 5/12/2014 5:15 AM, Timon Gehr wrote: On 05/12/2014 10:54 AM, Walter Bright wrote: On 5/11/2014 10:57 PM, Marco Leise wrote: Am Sun, 11 May 2014 17:50:25 -0700 schrieb Walter Bright newshou...@digitalmars.com: As long as those pointers don't escape. Am I right in that one cannot store a borrowed pointer into a global data structure? Right, and that's the point and entirely positive-to-do™. This means that a global data structure in Rust has to decide what memory allocation scheme its contents must use, Global variables are banned in Rust code outside of unsafe blocks. Global can also mean assigning through a reference passed as a parameter. Do you mean the table is not actually global but passed by parameter, or that the global table is accessed in unsafe code and then passed by parameter or something else?
Re: More radical ideas about gc and reference counting
On 2014-05-12 19:14, Dicebot wrote: It lacks any good static reflection though. And this stuff is damn addictive when you try it of D caliber. It has macros, that basically requires great support for static reflection to be usable. -- /Jacob Carlborg
Re: More radical ideas about gc and reference counting
Andrei Alexandrescu: I, too, felt the need of onGC() - actually preGC() - in my allocators implementation. ... A hook that nulls all freelist heads just as the collection process starts would be helpful. How is this going to help increase tracing precision of unions (and Algebraic built on top of unions)? Bye, bearophile
Re: More radical ideas about gc and reference counting
On 5/12/14, 12:59 PM, bearophile wrote: Andrei Alexandrescu: I, too, felt the need of onGC() - actually preGC() - in my allocators implementation. ... A hook that nulls all freelist heads just as the collection process starts would be helpful. How is this going to help increase tracing precision of unions (and Algebraic built on top of unions)? How did I give the impression it has anything to do with unions? -- Andrei
Re: More radical ideas about gc and reference counting
Andrei Alexandrescu: How did I give the impression it has anything to do with unions? -- Andrei OK, so yours is not an answer to my proposal, nor related to it. Bye, bearophile
Re: More radical ideas about gc and reference counting
On 5/12/2014 12:36 PM, Timon Gehr wrote: Do you mean the table is not actually global but passed by parameter, Yes. But note that the distinction between the two is often blurry. Under the hood on some systems, global data is accessed via the equivalent of a hidden parameter.
borrowed pointers vs ref
It's been brought up more than once that the 'scope' storage class is an unimplemented borrowed pointer. But thinking a bit more along those lines, actually 'ref' fills the role of a borrowed pointer. One particularly apropos behavior is that struct member functions pass 'this' by ref, meaning that members can be called without the inc/dec millstone. ref is still incomplete as far as this goes, but we can go the extra distance with it, and then it will be of great help in supporting any ref counting solution. What it doesn't work very well with are class references. But Andrei suggested that we can focus the use of 'scope' to deal with that in an analogous way. What do you think? Anyone want to enumerate a list of the current deficiencies of 'ref' in regards to this, so we can think about solving it?
Re: borrowed pointers vs ref
How would you assign a borrowed pointer?
Re: 64-bit DMD for windows?
On 5/12/14, Nick Sabalausky via Digitalmars-d digitalmars-d@puremagic.com wrote: You don't need a 64-bit version: Compiling 64-bit programs doesn't require a 64-bit compiler. Just install VC++, use the DMD 2.065 Win installer, and then toss in the -m64 flag when compiling. Works fine. Doesn't matter if DMD itself is 32-bit. As Vladimir in IRC reminded me, there is one use-case: You may need it for some intensive CTFE stuff (excessive memory allocations and no freeing by the compiler). That is, if you need more than 3/4 gigs.
Re: borrowed pointers vs ref
On 5/12/2014 1:49 PM, Kagamin wrote: How would you assign a borrowed pointer? A ref could only be assigned to another ref.
Re: borrowed pointers vs ref
On 5/12/2014 2:13 PM, Walter Bright wrote: On 5/12/2014 1:49 PM, Kagamin wrote: How would you assign a borrowed pointer? A ref could only be assigned to another ref. I mean to a ref of the same or smaller scope.
Re: borrowed pointers vs ref
The first thing that comes to my mind is applying this somehow to
the (T) vs (ref T) function problem. (const ref, scope ref,
references to r-values, you know the problem.) At the moment I
just follow this pattern.
void foo(ref const T bar) {
/* ... */
}
// Second overload to make r-values just work.
void foo(const T bar) {
foo(bar);
}
auto ref sometimes works, sometimes it's more trouble than its
worth.
Re: borrowed pointers vs ref
On Mon, 12 May 2014 16:36:12 -0400, Walter Bright newshou...@digitalmars.com wrote: It's been brought up more than once that the 'scope' storage class is an unimplemented borrowed pointer. But thinking a bit more along those lines, actually 'ref' fills the role of a borrowed pointer. One particularly apropos behavior is that struct member functions pass 'this' by ref, meaning that members can be called without the inc/dec millstone. ref is still incomplete as far as this goes, but we can go the extra distance with it, and then it will be of great help in supporting any ref counting solution. Hm... the one piece that I think would be damaging is to not be able to take an address of the 'this' reference. It's probably OK to just use pointers and static functions in some cases, but member functions do not have that luxury. In other words, operators. Big example would be a doubly linked list used with ~=. -Steve
Re: More radical ideas about gc and reference counting
On Mon, 12 May 2014 14:14:28 -0400, Ola Fosheim Grøstad ola.fosheim.grostad+dl...@gmail.com wrote: On Monday, 12 May 2014 at 17:52:18 UTC, Walter Bright wrote: On 5/12/2014 7:46 AM, Steven Schveighoffer wrote: pointing at it is roughly 1/256. This problem is just about eliminated with 64-bit pointers. Not generally true. This presumes that the heap is not in the lower region of the address space and that you don't use 64 bit ints on the stack. I was thinking in terms of purely a random number happening to point at heap data. Practically speaking, I don't know the true likelihood based on the heap address scheme of 64-bit OSes, but I know that we always have a complainer who will try and do an array-append test on 32-bit code, and end up exhausting memory unexpectedly. -Steve
Re: borrowed pointers vs ref
On 05/12/2014 10:36 PM, Walter Bright wrote: It's been brought up more than once that the 'scope' storage class is an unimplemented borrowed pointer. But thinking a bit more along those lines, actually 'ref' fills the role of a borrowed pointer. One particularly apropos behavior is that struct member functions pass 'this' by ref, meaning that members can be called without the inc/dec millstone. ref is still incomplete as far as this goes, but we can go the extra distance with it, and then it will be of great help in supporting any ref counting solution. What it doesn't work very well with are class references. But Andrei suggested that we can focus the use of 'scope' to deal with that in an analogous way. What do you think? I think everything should be treated uniformly. But a storage class is not sufficient. Anyone want to enumerate a list of the current deficiencies of 'ref' in regards to this, so we can think about solving it? Eg: - Cannot make tail const. / Cannot be reassigned. - Cannot store in data structures. - Cannot borrow slices of memory. - Closures? - (Probably more)
Re: borrowed pointers vs ref
On Monday, 12 May 2014 at 20:36:10 UTC, Walter Bright wrote:
It's been brought up more than once that the 'scope' storage
class is an unimplemented borrowed pointer. But thinking a bit
more along those lines, actually 'ref' fills the role of a
borrowed pointer.
One particularly apropos behavior is that struct member
functions pass 'this' by ref, meaning that members can be
called without the inc/dec millstone.
ref is still incomplete as far as this goes, but we can go the
extra distance with it, and then it will be of great help in
supporting any ref counting solution.
What it doesn't work very well with are class references. But
Andrei suggested that we can focus the use of 'scope' to deal
with that in an analogous way.
What do you think?
Anyone want to enumerate a list of the current deficiencies of
'ref' in regards to this, so we can think about solving it?
I would prefer 'scope ref' that would allow the solution for
classes and everything else to be unified, i.e. everything uses
scope.
When it comes to the implicit 'this' by ref, it could be
redefined to pass by scope ref.
Another reason is: I know this doesn't(and might never) work in
D, but based on the intuitive meaning of 'ref' I fully expected
the below example to work when I first started learning the
language.
struct A
{
ref int a_m;
this(ref int a)
{
a_m = a;
}
}
Whereas 'scope' on the other hand is self documenting imho.
Re: More radical ideas about gc and reference counting
Le 12/05/2014 19:14, Dicebot a écrit : On Monday, 12 May 2014 at 17:03:41 UTC, Manu via Digitalmars-d wrote: But D is *so close*... and I like it! _ I have to say that this discussion has certainly left me somewhat intrigued by Rust though. I've never given it a fair go because I find the syntax so distasteful and deterring. I wonder if there's a market for a rust fork that re-skin's the language ;) Right now D has practical benefit of being more stable and library rich. But switching to Rust eventually does seem tempting as I find foundations of their type system much closer to my beliefs about good coding practices. It lacks any good static reflection though. And this stuff is damn addictive when you try it of D caliber. All compile time things of D are marvelous. This with the compile time and the language less error prone make me want D. I am not sure I need safety so much. It's nice but not mandatory for any of my projects. The only one which has to be safe is DQuick.
Re: More radical ideas about gc and reference counting
On Mon, 12 May 2014 13:52:20 -0400, Walter Bright newshou...@digitalmars.com wrote: On 5/12/2014 7:46 AM, Steven Schveighoffer wrote: It doesn't matter where the false pointers are. The largest issue with false pointers is not how many false pointers there are. It only matters how large the block is that it points at. The larger your blocks get, the more likely they are pointed at by the stack. On 32-bit systems, allocate 1/256th of your memory space (i.e. 16.5MB), and the likelihood of random data on the stack pointing at it is roughly 1/256. This problem is just about eliminated with 64-bit pointers. Generally, it is a bad idea to allocate such large blocks on the GC heap. GC's work best when the size of the objects being allocated is very small relative to the size of the heap space. Fortunately, it's a mathematical inevitability that large allocations relative to the GC size are rare, and so it isn't much of a pain to handle them manually. The issue arises when one allocates such a large block for temporary use repeatedly, but expects it to be collected between allocations. The consequences are extremely disastrous. The workaround is simply to keep it around, but that's not always a scalable solution. And in fact, even if it's forbidden, requires is too strong a word -- there is no static or runtime prevention of this. It's still forbidden. Andrei wrote a template that will verify this at runtime, but I don't recall its name. Can you cite the spec where it says it's forbidden? Forgotten templates are not a convincing argument. Regardless, Java can use a moving GC, and allows self references. The idea that self references prevent a moving GC is simply false. If you think about it a bit, you will understand why. -Steve
Re: More radical ideas about gc and reference counting
On Mon, 12 May 2014 17:32:09 -0400, Steven Schveighoffer schvei...@yahoo.com wrote: The workaround is simply to keep it around, but that's not always a scalable solution. Sorry, actually you can free it. That's the correct workaround. -Steve
Re: More radical ideas about gc and reference counting
On Monday, 12 May 2014 at 21:22:09 UTC, Steven Schveighoffer wrote: On Mon, 12 May 2014 14:14:28 -0400, Ola Fosheim Grøstad ola.fosheim.grostad+dl...@gmail.com wrote: On Monday, 12 May 2014 at 17:52:18 UTC, Walter Bright wrote: On 5/12/2014 7:46 AM, Steven Schveighoffer wrote: pointing at it is roughly 1/256. This problem is just about eliminated with 64-bit pointers. Not generally true. This presumes that the heap is not in the lower region of the address space and that you don't use 64 bit ints on the stack. I was thinking in terms of purely a random number happening to point at heap data. Practically speaking, I don't know the true likelihood based on the heap address scheme of 64-bit OSes, but Wicked topic. In AMD64 mode hi-mem is usually reserved for kernel etc. Traditionally the unixy heap grew from low towards high addresses: http://en.wikipedia.org/wiki/Sbrk But that is legacy. I think mmap is it… :-P And layout is randomized to reduce the effect of buffer overflow etc. :-( I know that we always have a complainer who will try and do an array-append test on 32-bit code, and end up exhausting memory unexpectedly. Uhuh. Not focusing on precise collection gets ugly.
Re: radical ideas about GC and ARC : need to be time driven?
Le 12/05/2014 06:26, Marco Leise a écrit : Am Mon, 12 May 2014 03:36:34 +1000 schrieb Manu via Digitalmars-d digitalmars-d@puremagic.com: On 12 May 2014 02:38, Marco Leise via Digitalmars-d digitalmars-d@puremagic.com wrote: Am Sun, 11 May 2014 14:52:50 +1000 schrieb Manu via Digitalmars-d digitalmars-d@puremagic.com: On 11 May 2014 05:39, H. S. Teoh via Digitalmars-d digitalmars-d@puremagic.com wrote: On Sat, May 10, 2014 at 09:16:54PM +0200, Xavier Bigand via Digitalmars-d wrote: - Same question if D migrate to ARC? I highly doubt D will migrate to ARC. ARC will probably become *possible*, but some language features fundamentally rely on the GC, and I can't see how that will ever be changed. Which ones are incompatible with ARC? Pass-by-value slices as 2 machine words 64bit pointers are only 40-48 bits, so there's 32bits waste for an offset... and if the base pointer is 32byte aligned (all allocated memory is aligned), then you can reclaim another 5 bits there... I think saving an arg register would probably be worth a shift. 32bit pointers... not so luck :/ video games consoles though have bugger all memory, so heaps of spare bits in the pointers! :P And remember how people abused the high bit in 32-bit until kernels were modified to support the full address space and the Windows world got that LARGE_ADDRESS_AWARE flag to mark executables that do not gamble with the high bit. On the positive side the talk about Rust, in particular how reference counted pointers decay to borrowed pointers made me think the same could be done for our scope args. A reference counted slice with 3 machine words could decay to a 2 machine word scoped slice. Most of my code at least just works on the slices and doesn't keep a reference to them. A counter example is when you have something like an XML parser - a use case that D traditionally (see Tango) excelled in. The GC environment and slices make it possible to replace string copies with cheap slices into the original XML string. I don't really understand why there is no parser with something like slices in a language without GC. It's not possible to put the array to a more globally place, then the parser API will use 2 indexes instead of the buffer as parameter?
Re: More radical ideas about gc and reference counting
On 05/11/2014 08:18 PM, Rainer Schuetze wrote: 1. Use a scheme that takes a snapshot of the heap, stack and registers at the moment of collection and do the actual collection in another thread/process while the application can continue to run. This is the way Leandro Lucarellas concurrent GC works (http://dconf.org/2013/talks/lucarella.html), but it relies on fork that doesn't exist on every OS/architecture. A manual copy of the memory won't scale to very large memory, though it might be compressed to possible pointers. Worst case it will need twice as much memory as the current heap. There is a problem with this scheme, copy-on-write is extremely expensive when a mutation happens. That's one page fault (context switch) + copying a whole page + mapping the new page. It's much worse with huge pages (2MB page size).
Re: More radical ideas about gc and reference counting
On Monday, 12 May 2014 at 19:13:50 UTC, Ola Fosheim Grøstad wrote: On Monday, 12 May 2014 at 18:07:51 UTC, Kapps wrote: Depending on how tunable the GC is, I feel like it should be possible to get away with a GC even for soft real-time programs like games. Even if you manage to make it work for game clients you also should make it work for low latency game servers, as code sharing is an important advantage. What a game/world server requires differs a lot, but highly dynamic and flexible worlds have to keep the physics to a single node (or tight cluster) for a region. That means you want to have as many players as possible tied to that node. In essence you want both performance, low latency, reliability, and little overhead in an evolutionary context (it has to support heavy modification over time). My gut feeling is that a runtime satisfying one game design will not satisfy another one as long as one insists on one global GC. In essence, it will never really work well. IMO, the same goes for ARC since RC does not perform well with multi-threading even when you use near optimal patterns and strategies. If ARC is only to be used where speed does not matter then you might as well use shared_ptr. .NET allows configuring the garbage collector by specifying workstation (concurrent, background [allow generation 0/1 collection while a generation 2 collection is going], one primary heap and a large object heap) or server (not certain if concurrent/background, but multiple heaps that get handled in parallel during collections). Or in situations where you have many processes running at once, disabling concurrent collection to reduce context switching overhead. In reality, most people leave the default concurrent collector, which is what I'd hope the default for D would be, but if it was sufficiently tunable something like vibe.d could decide to go with something more similar to what .NET uses for servers (which ASP.NET uses by default). I haven't been able to find good concrete numbers online, but the few sources I've found say that generation 0/1 collection tends to take 1 to 2-3 milliseconds and is not run concurrently because it's so short. This is quite sufficient for most projects, but perhaps could be tweaked a bit more for certain aspects like gaming, possibly even enabling concurrent collection for generation 0/1, but I'm not sure if this works well or is feasible. Still, the important thing is to get a good general one to use first, like the default one .NET uses for workstation applications.
Re: borrowed pointers vs ref
On 5/12/2014 2:15 PM, Steven Schveighoffer wrote: Hm... the one piece that I think would be damaging is to not be able to take an address of the 'this' reference. It's probably OK to just use pointers and static functions in some cases, but member functions do not have that luxury. In other words, operators. Big example would be a doubly linked list used with ~=. @trusted/@system code will be able to take the address of a ref.
Re: More radical ideas about gc and reference counting
On 5/12/2014 2:32 PM, Steven Schveighoffer wrote: It's still forbidden. Andrei wrote a template that will verify this at runtime, but I don't recall its name. Can you cite the spec where it says it's forbidden? Forgotten templates are not a convincing argument. Regardless, Java can use a moving GC, and allows self references. The idea that self references prevent a moving GC is simply false. If you think about it a bit, you will understand why. I see this is not specified in the documentation. Not sure what happened here, but I'll have to think about it.
Re: More radical ideas about gc and reference counting
On Monday, 12 May 2014 at 22:27:06 UTC, Kapps wrote: because it's so short. This is quite sufficient for most projects, but perhaps could be tweaked a bit more for certain aspects like gaming, possibly even enabling concurrent collection for generation 0/1, but I'm not sure if this works well or is feasible. Still, the important thing is to get a good general one to use first, like the default one .NET uses for workstation applications. I agree that getting a good (100% precise) GC is an important first step. I am not so sure about generation based GC when you have a window on a world map that you move around which roughly is FIFO (first in, first out). But to get good speed I think you are better off having multiple pools that can be released with no collection when a network-connection drops (if you have one conceptual pool per connection), and optimized allocators that give you pre-initialized objects etc. In the ideal world all of this is transparent once you have specified your memory model (in detail), so you only have to issue a new PlayerConnection in the main logic of your program and can tweak the memory handling elsewhere. That is not the D way, from what I can tell from the forum posts so far, because new is going to stay tied to one global GC heap. So you have to write utility functions… which makes programs less legible.
Re: FYI - mo' work on std.allocator
On Sunday, 27 April 2014 at 05:43:07 UTC, Andrei Alexandrescu wrote: Added SbrkRegion, SimpleBlocklist, and Blocklist. http://erdani.com/d/phobos-prerelease/std_allocator.html#.SbrkRegion http://erdani.com/d/phobos-prerelease/std_allocator.html#.SimpleBlocklist http://erdani.com/d/phobos-prerelease/std_allocator.html#.Blocklist https://github.com/andralex/phobos/blob/allocator/std/allocator.d Destruction is as always welcome. I plan to get into tracing tomorrow morning. Andrei Some comments on the version currently checked in (916032a0b6a76b6e37169121ee5cc680bb40b4c4): Line 3173: b2 is unused Line 3177: b3 is unused Line 3496: tids is unused Line 3512: b is unused Line 4235: r2 is never used. This one is probably a bug. Line 5426: b2 is never used Line 5958: b is unused Line 6337: alloc2 is unused. Looks like another bug
Some simple ideas about GC
I'd like to share some thoughts on improving D's GC, nothing radically different though. A few observations - Pause times seem to be a much bigger problem than CPU usage or memory bandwith. Focus on reducing the pause times. - The GC code is already fairly optimized, so there is a very low profitability in small-scale code optimizations. Improve the algorithms not the code. - The current GC code is not hackable. First rewrite then improve. and corresponding ideas. - Marking could be parallelized, sweeping should be done in the background, GC could serve allocations during sweep from a separate pool (e.g. thread-local). - The current GC does a lot of bookkeeping work due to how the pools are organized (heterogeneous bin sizes). I suspect (but don't know) that there are big gains in organizing this differently. - A testable and more modular rewrite (using recent D practices) would encourage more contribution and is necessary for experimentation. -Martin
Re: 64-bit DMD for windows?
On 5/12/2014 5:01 PM, Andrej Mitrovic via Digitalmars-d wrote: On 5/12/14, Nick Sabalausky via Digitalmars-d digitalmars-d@puremagic.com wrote: You don't need a 64-bit version: Compiling 64-bit programs doesn't require a 64-bit compiler. Just install VC++, use the DMD 2.065 Win installer, and then toss in the -m64 flag when compiling. Works fine. Doesn't matter if DMD itself is 32-bit. As Vladimir in IRC reminded me, there is one use-case: You may need it for some intensive CTFE stuff (excessive memory allocations and no freeing by the compiler). That is, if you need more than 3/4 gigs. Right, there's certainly that. But that has nothing to do with whether you're trying to build a 64-bit or 32-bit program, and (at least for Windows) it isn't even an issue at all unless you actually are hitting that limit (unlikely for a newcomer to D). It sounded like steven kladitis was worried about just being able to create 64-bit programs. For that, it makes no difference if the compiler itself is a 32- or 64-bit build.
Re: FYI - mo' work on std.allocator
On 5/12/14, 3:58 PM, Brian Schott wrote: On Sunday, 27 April 2014 at 05:43:07 UTC, Andrei Alexandrescu wrote: Added SbrkRegion, SimpleBlocklist, and Blocklist. http://erdani.com/d/phobos-prerelease/std_allocator.html#.SbrkRegion http://erdani.com/d/phobos-prerelease/std_allocator.html#.SimpleBlocklist http://erdani.com/d/phobos-prerelease/std_allocator.html#.Blocklist https://github.com/andralex/phobos/blob/allocator/std/allocator.d Destruction is as always welcome. I plan to get into tracing tomorrow morning. Andrei Some comments on the version currently checked in (916032a0b6a76b6e37169121ee5cc680bb40b4c4): Line 3173: b2 is unused Line 3177: b3 is unused Line 3496: tids is unused Line 3512: b is unused Line 4235: r2 is never used. This one is probably a bug. Line 5426: b2 is never used Line 5958: b is unused Line 6337: alloc2 is unused. Looks like another bug Will fix. Awesome, thanks! -- Andrei
