Re: Thread local and memory allocation

2011-10-05 Thread Walter Bright
On 10/5/2011 6:25 AM, deadalnix wrote: I see two solutions to deal with this : Something allocated on a Thread local heap can be seen from other threads, and this is safe as long as a reference is kept in the allocating thread. So, if you cast something TL and mutable as immutable, you have to

Re: Thread local and memory allocation

2011-10-05 Thread Sean Kelly
Maybe the correct approach is simply to try and eliminate the mutex protecting GC operations so allocations can be performed concurrently by multiple threads? Sent from my iPhone On Oct 4, 2011, at 8:55 PM, Andrew Wiley wrote: > On Tue, Oct 4, 2011 at 8:59 PM, Robert Jacques wrote: >> On Tue,

Re: Thread local and memory allocation

2011-10-05 Thread deadalnix
Le 04/10/2011 20:30, Walter Bright a écrit : On 10/4/2011 2:32 AM, deadalnix wrote: With casts to immutable, it is perfectly correct if you, the user, ensure that there are no other mutable references to the same data. It's just that the compiler itself cannot make this guarantee, hence it's "uns

Re: Thread local and memory allocation

2011-10-04 Thread Robert Jacques
On Tue, 04 Oct 2011 23:56:52 -0400, Andrew Wiley wrote: On Tue, Oct 4, 2011 at 10:55 PM, Andrew Wiley wrote: On Tue, Oct 4, 2011 at 8:59 PM, Robert Jacques wrote: On Tue, 04 Oct 2011 10:54:58 -0400, Andrew Wiley wrote: On Tue, Oct 4, 2011 at 3:52 AM, Walter Bright wrote: On 10/4/2011

Re: Thread local and memory allocation

2011-10-04 Thread Robert Jacques
On Tue, 04 Oct 2011 23:55:19 -0400, Andrew Wiley wrote: On Tue, Oct 4, 2011 at 8:59 PM, Robert Jacques wrote: On Tue, 04 Oct 2011 10:54:58 -0400, Andrew Wiley wrote: On Tue, Oct 4, 2011 at 3:52 AM, Walter Bright wrote: On 10/4/2011 1:22 AM, deadalnix wrote: Do you mean manage the mem

Re: Thread local and memory allocation

2011-10-04 Thread bearophile
deadalnix: > This is why the type system is made for isn't it ? Casts are often the points where type systems fail :-) Bye, bearophile

Re: Thread local and memory allocation

2011-10-04 Thread Andrew Wiley
On Tue, Oct 4, 2011 at 10:55 PM, Andrew Wiley wrote: > On Tue, Oct 4, 2011 at 8:59 PM, Robert Jacques wrote: >> On Tue, 04 Oct 2011 10:54:58 -0400, Andrew Wiley >> wrote: >> >>> On Tue, Oct 4, 2011 at 3:52 AM, Walter Bright >>> wrote: On 10/4/2011 1:22 AM, deadalnix wrote: > >

Re: Thread local and memory allocation

2011-10-04 Thread Andrew Wiley
On Tue, Oct 4, 2011 at 8:59 PM, Robert Jacques wrote: > On Tue, 04 Oct 2011 10:54:58 -0400, Andrew Wiley > wrote: > >> On Tue, Oct 4, 2011 at 3:52 AM, Walter Bright >> wrote: >>> >>> On 10/4/2011 1:22 AM, deadalnix wrote: Do you mean manage the memory that way : Shared heap -> TL

Re: Thread local and memory allocation

2011-10-04 Thread Robert Jacques
On Tue, 04 Oct 2011 10:54:58 -0400, Andrew Wiley wrote: On Tue, Oct 4, 2011 at 3:52 AM, Walter Bright wrote: On 10/4/2011 1:22 AM, deadalnix wrote: Do you mean manage the memory that way : Shared heap -> TL pool within the shared heap -> allocation in thread from TL pool. And complete GC

Re: Thread local and memory allocation

2011-10-04 Thread Robert Jacques
On Tue, 04 Oct 2011 17:50:03 -0400, deadalnix wrote: Le 04/10/2011 08:02, Robert Jacques a écrit : On Mon, 03 Oct 2011 15:48:57 -0400, deadalnix wrote: [snip] What I suggest in add a flag SHARED in BlkAttr and store it as an attribute of the block. Later modification could be made according

Re: Thread local and memory allocation

2011-10-04 Thread deadalnix
Le 04/10/2011 08:02, Robert Jacques a écrit : On Mon, 03 Oct 2011 15:48:57 -0400, deadalnix wrote: [snip] What I suggest in add a flag SHARED in BlkAttr and store it as an attribute of the block. Later modification could be made according to this flag. This attribute shouldn't be modifiable l

Re: Thread local and memory allocation

2011-10-04 Thread Walter Bright
On 10/4/2011 2:32 AM, deadalnix wrote: Le 04/10/2011 10:52, Walter Bright a écrit : Allocate an object, then cast it to immutable, and pass it to another thread. That is explicitly said to be unsafe on D's website. As long as a reference exist in the creating thread, this should work, but if

Re: Thread local and memory allocation

2011-10-04 Thread Andrew Wiley
On Tue, Oct 4, 2011 at 3:52 AM, Walter Bright wrote: > On 10/4/2011 1:22 AM, deadalnix wrote: >> >> Do you mean manage the memory that way : >> Shared heap -> TL pool within the shared heap -> allocation in thread from >> TL pool. >> >> And complete GC collect. > > Yes. > > >> This is a good solut

Re: Thread local and memory allocation

2011-10-04 Thread deadalnix
Le 04/10/2011 10:52, Walter Bright a écrit : Allocate an object, then cast it to immutable, and pass it to another thread. That is explicitly said to be unsafe on D's website. As long as a reference exist in the creating thread, this should work, but if thoses references disapears, you'll en

Re: Thread local and memory allocation

2011-10-04 Thread Walter Bright
On 10/4/2011 1:22 AM, deadalnix wrote: Do you mean manage the memory that way : Shared heap -> TL pool within the shared heap -> allocation in thread from TL pool. And complete GC collect. Yes. This is a good solution do reduce contention on allocation. But a very different thing than I wa

Re: Thread local and memory allocation

2011-10-04 Thread deadalnix
Le 04/10/2011 02:15, Walter Bright a écrit : On 10/3/2011 4:20 PM, Sean Kelly wrote: Immutable data would have to be allocated on the shared heap as well, which means the contention for the shared heap may actually be fairly significant. But the alternatives are all too complex (migrating immuta

Re: Thread local and memory allocation

2011-10-03 Thread Robert Jacques
On Mon, 03 Oct 2011 15:48:57 -0400, deadalnix wrote: [snip] What I suggest in add a flag SHARED in BlkAttr and store it as an attribute of the block. Later modification could be made according to this flag. This attribute shouldn't be modifiable later on. What do you think ? Is it something i

Re: Thread local and memory allocation

2011-10-03 Thread Jason House
Sean Kelly Wrote: > On Oct 3, 2011, at 3:27 PM, Jason House wrote: > > > Sean Kelly Wrote: > >> There's another important issue that hasn't yet been addressed, which is > >> that when the GC collects memory, the thread that finalizes non-shared > >> data should be the one that created it. So t

Re: Thread local and memory allocation

2011-10-03 Thread Walter Bright
On 10/3/2011 4:20 PM, Sean Kelly wrote: Immutable data would have to be allocated on the shared heap as well, which means the contention for the shared heap may actually be fairly significant. But the alternatives are all too complex (migrating immutable data from local pools to a common pool whe

Re: Thread local and memory allocation

2011-10-03 Thread Walter Bright
On 10/3/2011 12:48 PM, deadalnix wrote: What do you think ? Is it something it worth working on ? If it is, how can I help ? It is a great idea, and it has been discussed before. The difficulties are when thread local allocated data gets shared with other threads, like for instance immutable

Re: Thread local and memory allocation

2011-10-03 Thread Sean Kelly
On Oct 3, 2011, at 3:55 PM, Walter Bright wrote: > On 10/3/2011 12:48 PM, deadalnix wrote: >> What do you think ? Is it something it worth working on ? If it is, how can I >> help ? > > It is a great idea, and it has been discussed before. The difficulties are > when thread local allocated data

Re: Thread local and memory allocation

2011-10-03 Thread Sean Kelly
On Oct 3, 2011, at 3:27 PM, Jason House wrote: > Sean Kelly Wrote: >> There's another important issue that hasn't yet been addressed, which is >> that when the GC collects memory, the thread that finalizes non-shared data >> should be the one that created it. So that SHARED flag should really b

Re: Thread local and memory allocation

2011-10-03 Thread Jason House
Sean Kelly Wrote: > There's another important issue that hasn't yet been addressed, which is that > when the GC collects memory, the thread that finalizes non-shared data should > be the one that created it. So that SHARED flag should really be a thread-id > of some sort. Alternately, each thr

Re: Thread local and memory allocation

2011-10-03 Thread deadalnix
Yes, I was thinking is such a thing. Each thread has a local heap and you have a common shared heap too, with shared data in it. Is such a case, the flag is suffiscient because then the GC could handle that and trigger thread local heap allocation instead of shared one. This is consistent wit

Re: Thread local and memory allocation

2011-10-03 Thread bearophile
Sean Kelly: > I'd favor per-thread heaps but am open to suggestions and/or help. (I am ignorant still about such issues, so I usually keep myself quiet about them. Please forgive me if I am saying stupid things.) The memory today is organized like a tree, the larger memories are slower, and the

Re: Thread local and memory allocation

2011-10-03 Thread Sean Kelly
On Oct 3, 2011, at 12:48 PM, deadalnix wrote: > D's uses thread local storage for most of its data. And it's a good thing. > > However, the allocation mecanism isn't aware of it. In addition, it has no > way to handle it in the future as things are specified. > > As long as you don't have any p

Thread local and memory allocation

2011-10-03 Thread deadalnix
D's uses thread local storage for most of its data. And it's a good thing. However, the allocation mecanism isn't aware of it. In addition, it has no way to handle it in the future as things are specified. As long as you don't have any pointer in shared memory to thread local data (thank to t