Re: My wish for 2015...
On Fri, 26 Dec 2014 00:13:57 + Tobias Pankrath via Digitalmars-d wrote: > On Friday, 26 December 2014 at 00:05:49 UTC, Xinok wrote: > > On Thursday, 25 December 2014 at 20:00:43 UTC, Martin Nowak > > wrote: > >> On Thursday, 25 December 2014 at 14:55:32 UTC, Xinok wrote: > >> Ah OK, thought you had problems with stale stack values. > >> False pointers were discussed here > >> https://youtu.be/LQY1m_eT37c?t=23m0s, but it's an issue that > >> only affects few programs (32-bit with floats or random data > >> in GC scanned memory). So unless we can use precise GC info to > >> speed up marking, it will be a lot of work with little impact, > >> even more since we have Win64 support. > > > > Any program that processes significant amounts of information, > > such as scientific data, images, or videos, is likely to > > experience issues with the conservative GC. > > Would a well supported RC scheme be better than a precise GC for > these use cases? it depends of many things, including data allocation and access patterns. for some of them precise generational GC can be a much better choice, as it can discards the whole pages of "newdead" objects without scanning. yet we need some read or write barriers for this, preferably with support from the compiler. signature.asc Description: PGP signature
Re: My wish for 2015...
On Fri, 26 Dec 2014 00:05:47 + Xinok via Digitalmars-d wrote: > Other languages that have transitioned from conservative to > precise have experienced gains and losses in different suites of > applications. In particular, programs that perform a lot of > allocations actually saw better performance from the precise GC. > Some languages benefit more from a precise GC more than others, > so it's hard to say how successful it would be in D. besides, it's always easy to use conservative GC instead of precise GC. the reverse is not true. signature.asc Description: PGP signature
Re: My wish for 2015...
On Friday, 26 December 2014 at 00:05:49 UTC, Xinok wrote: On Thursday, 25 December 2014 at 20:00:43 UTC, Martin Nowak wrote: On Thursday, 25 December 2014 at 14:55:32 UTC, Xinok wrote: Ah OK, thought you had problems with stale stack values. False pointers were discussed here https://youtu.be/LQY1m_eT37c?t=23m0s, but it's an issue that only affects few programs (32-bit with floats or random data in GC scanned memory). So unless we can use precise GC info to speed up marking, it will be a lot of work with little impact, even more since we have Win64 support. Any program that processes significant amounts of information, such as scientific data, images, or videos, is likely to experience issues with the conservative GC. Would a well supported RC scheme be better than a precise GC for these use cases?
Re: My wish for 2015...
On Thursday, 25 December 2014 at 20:00:43 UTC, Martin Nowak wrote: On Thursday, 25 December 2014 at 14:55:32 UTC, Xinok wrote: Ah OK, thought you had problems with stale stack values. False pointers were discussed here https://youtu.be/LQY1m_eT37c?t=23m0s, but it's an issue that only affects few programs (32-bit with floats or random data in GC scanned memory). So unless we can use precise GC info to speed up marking, it will be a lot of work with little impact, even more since we have Win64 support. Any program that processes significant amounts of information, such as scientific data, images, or videos, is likely to experience issues with the conservative GC. While it would be nice to say "just use 64-bit" to everybody, we can't ignore the fact that 32-bit is still widely used and it's not going away anytime soon. As for Win64 support, I've found it iffy at best; sometimes it works, sometimes it doesn't. I think implementing a precise GC would be worth it in the long run. The key is realizing that there are many optimizations that can be applied which are impossible with a conservative GC. It may make programs a bit slower initially, but it's something that we can optimize over time (well beyond the 2015 goal). Other languages that have transitioned from conservative to precise have experienced gains and losses in different suites of applications. In particular, programs that perform a lot of allocations actually saw better performance from the precise GC. Some languages benefit more from a precise GC more than others, so it's hard to say how successful it would be in D.
Re: My wish for 2015...
On Thursday, 25 December 2014 at 14:55:32 UTC, Xinok wrote: On Thursday, 25 December 2014 at 10:52:07 UTC, Martin Nowak wrote: On Saturday, 20 December 2014 at 22:11:35 UTC, Xinok wrote: (1) We need a precise garbage collector. The fact that a garbage-collected language experiences memory leaks truly reflects poorly on on D. What's the reason for leakage? I'm a bit skeptical about the precise GC work. https://github.com/D-Programming-Language/druntime/pull/1057#issuecomment-65918407 Perhaps "leakage" is the wrong term to use here. I'm referring to the fact that a conservative garbage collector fails to distinguish between pointers and non-pointers, so it happens that dead objects fail to get collected. It's a much greater problem on 32-bit with the limited address space, especially if you perform many large allocations. Ah OK, thought you had problems with stale stack values. False pointers were discussed here https://youtu.be/LQY1m_eT37c?t=23m0s, but it's an issue that only affects few programs (32-bit with floats or random data in GC scanned memory). So unless we can use precise GC info to speed up marking, it will be a lot of work with little impact, even more since we have Win64 support.
Re: My wish for 2015...
On Thursday, 25 December 2014 at 10:52:07 UTC, Martin Nowak wrote: On Saturday, 20 December 2014 at 22:11:35 UTC, Xinok wrote: (1) We need a precise garbage collector. The fact that a garbage-collected language experiences memory leaks truly reflects poorly on on D. What's the reason for leakage? I'm a bit skeptical about the precise GC work. https://github.com/D-Programming-Language/druntime/pull/1057#issuecomment-65918407 Perhaps "leakage" is the wrong term to use here. I'm referring to the fact that a conservative garbage collector fails to distinguish between pointers and non-pointers, so it happens that dead objects fail to get collected. It's a much greater problem on 32-bit with the limited address space, especially if you perform many large allocations.
Re: My wish for 2015...
On Saturday, 20 December 2014 at 22:11:35 UTC, Xinok wrote: I think the problem of memory management can be reduced to two points: (1) The garbage collector for D is sub-par. I want to try out a few obvious things in the next couple of weeks. - better predictable memory access during marking - parallel marking - parallel finalization (nice to have) Those are fairly simple to implement, but I'm hoping for huge improvements. (1) We need a precise garbage collector. The fact that a garbage-collected language experiences memory leaks truly reflects poorly on on D. What's the reason for leakage? I'm a bit skeptical about the precise GC work. https://github.com/D-Programming-Language/druntime/pull/1057#issuecomment-65918407
Re: My wish for 2015...
On Wednesday, 24 December 2014 at 05:17:09 UTC, Jonathan Marler wrote: This may only be a personal opinion but I also agree that memory management should be made the #1 priority. That being said, in order to make this a priority it would be great to see more organization when it comes to D development. A simple prioritized list of features and developers currently working on them would be a great start:) I like the summary of features and completely agree. I agree 100% with this, as someone who doesn't have a lot of time to devote to using D I have to dig through the wiki to even find out what is being currently worked on in a new revision, if any DIPs have been approved/implemented, etc. A D news feed would be nice, unless there is one and I've missed it. I know there's an official D twitter but its had 2 posts in 2 months.
Re: My wish for 2015...
On Wednesday, 24 December 2014 at 23:14:53 UTC, Delirious wrote: On Saturday, 20 December 2014 at 22:11:35 UTC, Xinok wrote: (2) Furthermore, we need to improve the performance of the garbage collector. "Performance" is a complex issue in the case of GCs. There is not "highest performance GC", only GCs optimally tuned for certain scenarios. While a simple, imprecise, mark-and-sweep collector like the one used by D right now is not that complicated, modern, industrial GCs are very complex. There are many decisions to be made and there is no single "right" answer. A simple philosophy to follow is to "optimize for the common case". As I stated, I'm not expecting an optimal solution within the next year, just something that's "good enough". There's lots of details to work out, but that's not a problem. It's not uncommon for discussions here to go beyond 200+ posts, so I have no doubt decisions will be made. At this point I would like to quote a post about the evolution of the Go garbage collector because Go originally had a primitive GC too but is now far head of D's by any standard: ... All the more reason why this issue needs resolution. D was first released 13 years ago whereas Go is only five years old and already has a far superior garbage collector. I know it's to be expected as Google has billions of dollars at it's disposal, but all most people see is what they're given. That's possibly the greatest challenge of D, the fact that it's a community-developed language. If you can't get the community to agree on specific goals and deadlines, then people will work on whatever interests them and it becomes a disorganized mess. Replacing the GC with a different one is controversial, for example: except for very large heaps, it is unclear whether a generational GC would be faster overall package "unsafe" makes it hard to implement fully precise GC and compacting GC I'm not saying we need to replace the GC altogether, and that's probably an unrealistic goal to achieve within the next year anyways. For example, making the GC precise would add a bit of overhead but it would also benefit by scanning less memory overall (no need to scan ubyte[] for pointers). Furthermore, if we stick to the stop-the-world model we have now, there's no reason we can't implement a concurrent sweep like Go has. ;-)
Re: My wish for 2015...
On Saturday, 20 December 2014 at 22:11:35 UTC, Xinok wrote: (2) Furthermore, we need to improve the performance of the garbage collector. "Performance" is a complex issue in the case of GCs. There is not "highest performance GC", only GCs optimally tuned for certain scenarios. While a simple, imprecise, mark-and-sweep collector like the one used by D right now is not that complicated, modern, industrial GCs are very complex. There are many decisions to be made and there is no single "right" answer. At this point I would like to quote a post about the evolution of the Go garbage collector because Go originally had a primitive GC too but is now far head of D's by any standard: Plans for Go 1.4+ garbage collector: hybrid stop-the-world/concurrent collector stop-the-world part limited by a 10ms deadline CPU cores dedicated to running the concurrent collector tri-color mark-and-sweep algorithm non-generational non-compacting fully precise incurs a small cost if the program is moving pointers around lower latency, but most likely also lower throughput, than Go 1.3 GC Go 1.3 garbage collector updates on top of Go 1.1: concurrent sweep (results in smaller pause times) fully precise Go 1.1 garbage collector: mark-and-sweep (parallel implementation) non-generational non-compacting mostly precise (except stack frames) stop-the-world bitmap-based representation zero-cost when the program is not allocating memory (that is: shuffling pointers around is as fast as in C, although in practice this runs somewhat slower than C because the Go compiler is not as advanced as C compilers such as GCC) supports finalizers on objects there is no support for weak references Go 1.0 garbage collector: same as Go 1.1, but instead of being mostly precise the garbage collector is conservative. The conservative GC is able to ignore objects such as []byte. Replacing the GC with a different one is controversial, for example: except for very large heaps, it is unclear whether a generational GC would be faster overall package "unsafe" makes it hard to implement fully precise GC and compacting GC
Re: My wish for 2015...
On 24 December 2014 at 04:49, Vic via Digitalmars-d wrote: > On Saturday, 20 December 2014 at 22:11:35 UTC, Xinok wrote: >> >> I'm going to make a stark proposal to the you all, the community and all D >> users as whole. I wish for us to set an ultimate goal to be made top >> priority and complete by the end of next year. My wish is to resolve the >> issue of memory management for D by the end of 2015. This is a significant >> issue that has affected most of us at one point or another. I think this >> gives D a bad rap more than anything else and is a point of contention for >> many, especially those with a background in C/C++. >> >> I think the problem of memory management can be reduced to two points: >> (1) The garbage collector for D is sub-par. >> (2) There are too many implicit allocations in Phobos. >> >> I think three goals need to be met for the problem of memory management to >> be satisfied: >> (1) We need a precise garbage collector. The fact that a garbage-collected >> language experiences memory leaks truly reflects poorly on on D. >> (2) Furthermore, we need to improve the performance of the garbage >> collector. There are some things the developer can do to reduce the time and >> frequency collection cycles, but the current situation is far from optimal. >> (3) We need a viable alternative to the garbage collection. Whether that >> be allocators, ref counting, or full-fledged manual memory management, there >> is great demand for the ability to use D without the GC with little hassle. >> >> I sincerely believe that this is the greatest issue facing D today and >> something that should have been resolved a long time ago. The fact that >> relatively simple programs can crash with out-of-memory errors (especially >> 32-bit executables) and high-performance code experiences frequent or >> lengthy collection cycles means we have a bad situation on our hands. >> >> Things like @nogc are a start but much more needs to be done. I'm not >> hoping for an optimal solution, nor am I expecting a state-of-the-art >> garbage collector. I think we should simply aim for "good enough". Then once >> we have a better memory management scheme, we can begin incorporating these >> changes into Phobos. >> >> What do you all think? Can we make improving memory management the top >> priority for 2015 with the goal of developing an adequate solution by the >> end of next year? > > > +1. Seems like a reasonable compromise. > > (but to be a negative Nancy: to be realistically achieved: it must be the > only goal and w/ a laser focus. D community still has to demonstrate > discipline, here is an inspiration: http://youtube.com/watch?v=iYWzMvlj2RQ > ). > > Still cheers and hope, Vic You should see me talk about changes that neglect/break cross-platform compatibility. (ie: anything non-x86 :o)
Re: My wish for 2015...
On Saturday, 20 December 2014 at 22:11:35 UTC, Xinok wrote: I'm going to make a stark proposal to the you all, the community and all D users as whole. I wish for us to set an ultimate goal to be made top priority and complete by the end of next year. My wish is to resolve the issue of memory management for D by the end of 2015. This is a significant issue that has affected most of us at one point or another. I think this gives D a bad rap more than anything else and is a point of contention for many, especially those with a background in C/C++. I think the problem of memory management can be reduced to two points: (1) The garbage collector for D is sub-par. (2) There are too many implicit allocations in Phobos. I think three goals need to be met for the problem of memory management to be satisfied: (1) We need a precise garbage collector. The fact that a garbage-collected language experiences memory leaks truly reflects poorly on on D. (2) Furthermore, we need to improve the performance of the garbage collector. There are some things the developer can do to reduce the time and frequency collection cycles, but the current situation is far from optimal. (3) We need a viable alternative to the garbage collection. Whether that be allocators, ref counting, or full-fledged manual memory management, there is great demand for the ability to use D without the GC with little hassle. I sincerely believe that this is the greatest issue facing D today and something that should have been resolved a long time ago. The fact that relatively simple programs can crash with out-of-memory errors (especially 32-bit executables) and high-performance code experiences frequent or lengthy collection cycles means we have a bad situation on our hands. Things like @nogc are a start but much more needs to be done. I'm not hoping for an optimal solution, nor am I expecting a state-of-the-art garbage collector. I think we should simply aim for "good enough". Then once we have a better memory management scheme, we can begin incorporating these changes into Phobos. What do you all think? Can we make improving memory management the top priority for 2015 with the goal of developing an adequate solution by the end of next year? This may only be a personal opinion but I also agree that memory management should be made the #1 priority. That being said, in order to make this a priority it would be great to see more organization when it comes to D development. A simple prioritized list of features and developers currently working on them would be a great start:) I like the summary of features and completely agree.
Re: My wish for 2015...
On Saturday, 20 December 2014 at 22:11:35 UTC, Xinok wrote: I'm going to make a stark proposal to the you all, the community and all D users as whole. I wish for us to set an ultimate goal to be made top priority and complete by the end of next year. My wish is to resolve the issue of memory management for D by the end of 2015. This is a significant issue that has affected most of us at one point or another. I think this gives D a bad rap more than anything else and is a point of contention for many, especially those with a background in C/C++. I think the problem of memory management can be reduced to two points: (1) The garbage collector for D is sub-par. (2) There are too many implicit allocations in Phobos. I think three goals need to be met for the problem of memory management to be satisfied: (1) We need a precise garbage collector. The fact that a garbage-collected language experiences memory leaks truly reflects poorly on on D. (2) Furthermore, we need to improve the performance of the garbage collector. There are some things the developer can do to reduce the time and frequency collection cycles, but the current situation is far from optimal. (3) We need a viable alternative to the garbage collection. Whether that be allocators, ref counting, or full-fledged manual memory management, there is great demand for the ability to use D without the GC with little hassle. I sincerely believe that this is the greatest issue facing D today and something that should have been resolved a long time ago. The fact that relatively simple programs can crash with out-of-memory errors (especially 32-bit executables) and high-performance code experiences frequent or lengthy collection cycles means we have a bad situation on our hands. Things like @nogc are a start but much more needs to be done. I'm not hoping for an optimal solution, nor am I expecting a state-of-the-art garbage collector. I think we should simply aim for "good enough". Then once we have a better memory management scheme, we can begin incorporating these changes into Phobos. What do you all think? Can we make improving memory management the top priority for 2015 with the goal of developing an adequate solution by the end of next year? +1. Seems like a reasonable compromise. (but to be a negative Nancy: to be realistically achieved: it must be the only goal and w/ a laser focus. D community still has to demonstrate discipline, here is an inspiration: http://youtube.com/watch?v=iYWzMvlj2RQ ). Still cheers and hope, Vic
Re: My wish for 2015...
On 12/20/14 2:11 PM, Xinok wrote: I'm going to make a stark proposal to the you all, the community and all D users as whole. I wish for us to set an ultimate goal to be made top priority and complete by the end of next year. My wish is to resolve the issue of memory management for D by the end of 2015. This is a significant issue that has affected most of us at one point or another. I think this gives D a bad rap more than anything else and is a point of contention for many, especially those with a background in C/C++. I think the problem of memory management can be reduced to two points: (1) The garbage collector for D is sub-par. (2) There are too many implicit allocations in Phobos. I think three goals need to be met for the problem of memory management to be satisfied: (1) We need a precise garbage collector. The fact that a garbage-collected language experiences memory leaks truly reflects poorly on on D. (2) Furthermore, we need to improve the performance of the garbage collector. There are some things the developer can do to reduce the time and frequency collection cycles, but the current situation is far from optimal. (3) We need a viable alternative to the garbage collection. Whether that be allocators, ref counting, or full-fledged manual memory management, there is great demand for the ability to use D without the GC with little hassle. [snip] Great. I'm up for this. -- Andrei
My wish for 2015...
I'm going to make a stark proposal to the you all, the community and all D users as whole. I wish for us to set an ultimate goal to be made top priority and complete by the end of next year. My wish is to resolve the issue of memory management for D by the end of 2015. This is a significant issue that has affected most of us at one point or another. I think this gives D a bad rap more than anything else and is a point of contention for many, especially those with a background in C/C++. I think the problem of memory management can be reduced to two points: (1) The garbage collector for D is sub-par. (2) There are too many implicit allocations in Phobos. I think three goals need to be met for the problem of memory management to be satisfied: (1) We need a precise garbage collector. The fact that a garbage-collected language experiences memory leaks truly reflects poorly on on D. (2) Furthermore, we need to improve the performance of the garbage collector. There are some things the developer can do to reduce the time and frequency collection cycles, but the current situation is far from optimal. (3) We need a viable alternative to the garbage collection. Whether that be allocators, ref counting, or full-fledged manual memory management, there is great demand for the ability to use D without the GC with little hassle. I sincerely believe that this is the greatest issue facing D today and something that should have been resolved a long time ago. The fact that relatively simple programs can crash with out-of-memory errors (especially 32-bit executables) and high-performance code experiences frequent or lengthy collection cycles means we have a bad situation on our hands. Things like @nogc are a start but much more needs to be done. I'm not hoping for an optimal solution, nor am I expecting a state-of-the-art garbage collector. I think we should simply aim for "good enough". Then once we have a better memory management scheme, we can begin incorporating these changes into Phobos. What do you all think? Can we make improving memory management the top priority for 2015 with the goal of developing an adequate solution by the end of next year?
