Re: [go-nuts] Huge map[string]*Object and GC where *Object is from sync.Pool

2018-09-29 Thread Peter Mogensen
On 09/28/2018 03:14 PM, Peter Mogensen wrote: > In other words.. using a map as index into another datastructure without > the map values being considered by the garbage collector as references. I just quickly read about the java.util.WeakHashMap and it says that the keys are the weak reference

Re: [go-nuts] Huge map[string]*Object and GC where *Object is from sync.Pool

2018-09-29 Thread Dan Kortschak
Please re-read what I was answering: "I know what a uintptr is but what would you put in it if not a pointer to another object?". If you have done that and still believe what I wrote is not true, please explain what I have got wrong. I am afraid I do not understand at all what you think is not tr

Re: [go-nuts] Huge map[string]*Object and GC where *Object is from sync.Pool

2018-09-28 Thread Dan Kortschak
You put in it a number that has the same bit pattern as a pointer to a value. There is nothing more to it than that. It does not refer to anything. They exist so you can do uncomfortable pointer arithmetic. On Fri, 2018-09-28 at 15:14 +0200, Henrik Johansson wrote: > I know what a uintptr is but w

Re: [go-nuts] Huge map[string]*Object and GC where *Object is from sync.Pool

2018-09-28 Thread Peter Mogensen
On 09/28/2018 03:43 PM, Ian Lance Taylor wrote: > That statement is not subject to change. Note that that statement > does not say that if p1 and p2 are pointer values, and that p1 == p2, > that uintptr(unsafe.Pointer(p1)) == uintptr(unsafe.Pointer(p2)). You > must not take intuitions from com

Re: [go-nuts] Huge map[string]*Object and GC where *Object is from sync.Pool

2018-09-28 Thread Ian Lance Taylor
On Thu, Sep 27, 2018 at 11:14 PM, Peter Mogensen wrote: > > Are you saying this statement from the Go spec is subject to change: > "Pointer values are comparable. Two pointer values are equal if they > point to the same variable or if both have value nil. Pointers to > distinct zero-size variables

Re: [go-nuts] Huge map[string]*Object and GC where *Object is from sync.Pool

2018-09-28 Thread Peter Mogensen
On 09/28/2018 03:29 PM, Robert Engels wrote: > Use the GC as is unless you have a large static object set, it could work for > that... That was a part of the whole premise (see subject). A huge number of same type/size objects. /Peter -- You received this message because you are subscribed

Re: [go-nuts] Huge map[string]*Object and GC where *Object is from sync.Pool

2018-09-28 Thread Peter Mogensen
On 09/28/2018 03:24 PM, Robert Engels wrote: > If you have a object reference then the object is going to tracked by the GC > anyway, no matter where it is referenced from, so you are not saving > anything, as that reference still must be somewhere... ... yes. I haven't really done measuremen

Re: [go-nuts] Huge map[string]*Object and GC where *Object is from sync.Pool

2018-09-28 Thread Peter Mogensen
On 09/28/2018 03:18 PM, Tamás Gulácsi wrote: > > This is not that. > My idea could maybe be restated simpler as having a huge double-linked > list of sync.Pool objects and using wanting to supplement it by a map > index, but avoiding storing pointers in the map. > > > There's n

Re: [go-nuts] Huge map[string]*Object and GC where *Object is from sync.Pool

2018-09-28 Thread Tamás Gulácsi
> This is not that. > My idea could maybe be restated simpler as having a huge double-linked > list of sync.Pool objects and using wanting to supplement it by a map > index, but avoiding storing pointers in the map. > > There's no such thing as a "list of sync.Pool objects" (if you refer the

Re: [go-nuts] Huge map[string]*Object and GC where *Object is from sync.Pool

2018-09-28 Thread Henrik Johansson
I know what a uintptr is but what would you put in it if not a pointer to another object? Isn't this very analogous to what you said: "a weak hashmap uses weak references to refer to the contained objects so that they will be collected if nothing else refers to them". Maybe I am missing something.

Re: [go-nuts] Huge map[string]*Object and GC where *Object is from sync.Pool

2018-09-28 Thread Peter Mogensen
On 09/28/2018 03:11 PM, Robert Engels wrote: > His statement is correct. First of all, a weak reference in java is not > like a weak pointer in C++, at least they are not needed to break > cycles, as the GC is immune to that issue. The difference is that a weak > hashmap uses weak references to

Re: [go-nuts] Huge map[string]*Object and GC where *Object is from sync.Pool

2018-09-28 Thread Henrik Johansson
That's clever but irrelevant for this discussion. fre 28 sep. 2018 kl 14:57 skrev Jan Mercl <0xj...@gmail.com>: > On Fri, Sep 28, 2018 at 2:53 PM Henrik Johansson > wrote: > > > The data pointed to by the uintptrs ... > > Uintptrs are integers. They do not point to anything. > -- > > -j > -- Y

Re: [go-nuts] Huge map[string]*Object and GC where *Object is from sync.Pool

2018-09-28 Thread Jan Mercl
On Fri, Sep 28, 2018 at 2:53 PM Henrik Johansson wrote: > The data pointed to by the uintptrs ... Uintptrs are integers. They do not point to anything. -- -j -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and

Re: [go-nuts] Huge map[string]*Object and GC where *Object is from sync.Pool

2018-09-28 Thread Peter Mogensen
On 09/28/2018 02:53 PM, Henrik Johansson wrote: > How is storing unintptrs in a map different from say java.util.WeakHashMap? > The data pointed to by the uintptrs can at any given time have been > reclaimed by the GC much the same as weak references in Java. > > I am not saying you are using i

Re: [go-nuts] Huge map[string]*Object and GC where *Object is from sync.Pool

2018-09-28 Thread Henrik Johansson
How is storing unintptrs in a map different from say java.util.WeakHashMap? The data pointed to by the uintptrs can at any given time have been reclaimed by the GC much the same as weak references in Java. I am not saying you are using it the same way as one would normally use weak references in o

Re: [go-nuts] Huge map[string]*Object and GC where *Object is from sync.Pool

2018-09-28 Thread Peter Mogensen
On 09/28/2018 02:04 PM, Robert Engels wrote: > This is in no way similar to weak references in Java. Weak references > are safe and appropriate for many caching data structures. The code idea > presented here is not related.  Correct. I've used weak references in other languages (not Java) to,

Re: [go-nuts] Huge map[string]*Object and GC where *Object is from sync.Pool

2018-09-28 Thread Peter Mogensen
On 09/28/2018 09:39 AM, Dan Kortschak wrote: > > I don't understand what you are asking. First, uintptr is not a pointer > type, it's an integer type that is the same size as a pointer and that > can be interconverted with an unsafe.Pointer. If you hold a uintptr > that was converted from an un

Re: [go-nuts] Huge map[string]*Object and GC where *Object is from sync.Pool

2018-09-28 Thread Dan Kortschak
On Fri, 2018-09-28 at 08:14 +0200, Peter Mogensen wrote: > > On 09/27/2018 11:12 PM, Dan Kortschak wrote: > > > > Unless you are holding a real pointer to the value, they will have > > no > > root to hold them and so will by GC'd. > Also if the pointer is returned by Pool.Get() ? > That's probabl

Re: [go-nuts] Huge map[string]*Object and GC where *Object is from sync.Pool

2018-09-28 Thread Peter Mogensen
On 09/28/2018 09:13 AM, Henrik Johansson wrote: > This sounds much like the trickery people have used both successfully > but also disastrously using weak references in Java. > Is that where you got the idea from? No. I think I mentioned in my first post where I got the idea from. I haven't pro

Re: [go-nuts] Huge map[string]*Object and GC where *Object is from sync.Pool

2018-09-28 Thread Henrik Johansson
This sounds much like the trickery people have used both successfully but also disastrously using weak references in Java. Is that where you got the idea from? fre 28 sep. 2018 kl 08:36 skrev Keith Randall : > On Thu, Sep 27, 2018 at 11:26 PM Peter Mogensen wrote: > >> >> >> On 09/28/2018 08:17

Re: [go-nuts] Huge map[string]*Object and GC where *Object is from sync.Pool

2018-09-27 Thread Keith Randall
On Thu, Sep 27, 2018 at 11:26 PM Peter Mogensen wrote: > > > On 09/28/2018 08:17 AM, Peter Mogensen wrote: > > > > > > On 09/28/2018 03:04 AM, keith.rand...@gmail.com wrote: > >> Objects returned by Get() are not special in any way. They will be GCd > >> just like an object returned by new(). In

Re: [go-nuts] Huge map[string]*Object and GC where *Object is from sync.Pool

2018-09-27 Thread Peter Mogensen
On 09/28/2018 08:17 AM, Peter Mogensen wrote: > > > On 09/28/2018 03:04 AM, keith.rand...@gmail.com wrote: >> Objects returned by Get() are not special in any way. They will be GCd >> just like an object returned by new(). In fact, they will often be just >> a new()'d object. >> There is no as

Re: [go-nuts] Huge map[string]*Object and GC where *Object is from sync.Pool

2018-09-27 Thread Peter Mogensen
On 09/28/2018 03:04 AM, keith.rand...@gmail.com wrote: > Objects returned by Get() are not special in any way. They will be GCd > just like an object returned by new(). In fact, they will often be just > a new()'d object. > There is no association of such an object with the pool.  A pool is just

Re: [go-nuts] Huge map[string]*Object and GC where *Object is from sync.Pool

2018-09-27 Thread Peter Mogensen
On 09/27/2018 11:12 PM, Dan Kortschak wrote: > Unless you are holding a real pointer to the value, they will have no > root to hold them and so will by GC'd. Also if the pointer is returned by Pool.Get() ? That's probably one of the main questions here. Will an object from sync.Pool which have

Re: [go-nuts] Huge map[string]*Object and GC where *Object is from sync.Pool

2018-09-27 Thread keith . randall
Objects returned by Get() are not special in any way. They will be GCd just like an object returned by new(). In fact, they will often be just a new()'d object. There is no association of such an object with the pool. A pool is just a fancy free list. Get might return a previously Put'd object

Re: [go-nuts] Huge map[string]*Object and GC where *Object is from sync.Pool

2018-09-27 Thread Francis
Ah, ok. In that case keeping a reference to a *Object in a uintptr will not prevent it from being garbage collected. For each garbage collection cycle the runtime visits every memory reference that is still 'alive'. Any references which were not visited will be reclaimed as free memory. So if

Re: [go-nuts] Huge map[string]*Object and GC where *Object is from sync.Pool

2018-09-27 Thread Dan Kortschak
Unless you are holding a real pointer to the value, they will have no root to hold them and so will by GC'd. Further, the uintptr in the map will not be updated if the value is moved (does not happen now but may in the future). What you're doing here is unsafe. On Thu, 2018-09-27 at 19:52 +0200,

Re: [go-nuts] Huge map[string]*Object and GC where *Object is from sync.Pool

2018-09-27 Thread Peter Mogensen
On 09/27/2018 07:44 PM, Francis wrote: > I believe the pool does not track the objects that it returns from a > call to `Get()`. I would be surprised if it did track these objects. > There is no real need for it to track these objects. Exactly, that was my point. Else there would be no reason t

Re: [go-nuts] Huge map[string]*Object and GC where *Object is from sync.Pool

2018-09-27 Thread Francis
I believe the pool does not track the objects that it returns from a call to `Get()`. I would be surprised if it did track these objects. There is no real need for it to track these objects. The role of a sync pool is to give you an instance of *Object when you need it. As an optimisation if it

Re: [go-nuts] Huge map[string]*Object and GC where *Object is from sync.Pool

2018-09-27 Thread Peter Mogensen
On 09/27/2018 07:19 PM, Francis wrote: > It is true that uintptr is not looked at by the garbage collector. This > will definitely reduce your GC costs. But there is no way (that I know > of) to convert your uintptr back into a *Object that is safe. I assume of course that we know that all uint

Re: [go-nuts] Huge map[string]*Object and GC where *Object is from sync.Pool

2018-09-27 Thread Francis
It's unclear to me how this could be useful. It is true that uintptr is not looked at by the garbage collector. This will definitely reduce your GC costs. But there is no way (that I know of) to convert your uintptr back into a *Object that is safe. Because uintptr is ignored by the GC it won't

Re: [go-nuts] Huge map[string]*Object and GC where *Object is from sync.Pool

2018-09-27 Thread Robert Engels
No offense, sorry for high jacking, I was trying to counter point their usefulness, as a way to limit additional resources being spent to improve GC when using them, since I think their application is error prone in many most? cases. > On Sep 27, 2018, at 9:30 AM, Peter Mogensen wrote: > >

Re: [go-nuts] Huge map[string]*Object and GC where *Object is from sync.Pool

2018-09-27 Thread Peter Mogensen
On 09/27/2018 04:20 PM, Ian Davis wrote: > > https://github.com/golang/go/issues/23199 describes another gotcha when using > pools: pinning of memory buffers. > Yes... that's a good point for people considering to use sync.Pool. For the purpose of this question however, let's assume the dec

Re: [go-nuts] Huge map[string]*Object and GC where *Object is from sync.Pool

2018-09-27 Thread Ian Davis
On Thu, 27 Sep 2018, at 3:06 PM, Peter Mogensen wrote: > > > On 09/27/2018 03:58 PM, Robert Engels wrote: > > It wasn’t necessarily a warning to you :) > > > > It comes from the days of GC bashing in Java and so everyone tried to > > manually write garbage free programs using pools and it had a

Re: [go-nuts] Huge map[string]*Object and GC where *Object is from sync.Pool

2018-09-27 Thread Robert Engels
In a highly concurrent system it can be difficult to know when to call Put if the object reference is shared among threads/routines. Similar to why C++ has shared pointers. It might be easy for the original author to know when a Put should be called but as a system grows and new functionality is

Re: [go-nuts] Huge map[string]*Object and GC where *Object is from sync.Pool

2018-09-27 Thread Peter Mogensen
On 09/27/2018 03:58 PM, Robert Engels wrote: > It wasn’t necessarily a warning to you :) > > It comes from the days of GC bashing in Java and so everyone tried to > manually write garbage free programs using pools and it had a bad effect on > both performance and reliability. GC is there for

Re: [go-nuts] Huge map[string]*Object and GC where *Object is from sync.Pool

2018-09-27 Thread Robert Engels
It wasn’t necessarily a warning to you :) It comes from the days of GC bashing in Java and so everyone tried to manually write garbage free programs using pools and it had a bad effect on both performance and reliability. GC is there for a reason, use it... :) I would just hate to see Go develo

Re: [go-nuts] Huge map[string]*Object and GC where *Object is from sync.Pool

2018-09-27 Thread Robert Engels
Based on my experience and I believe many others, I would caution against the use of pools. Although it can be useful for objects that are very expensive to create/initialize using it in a more general case just to improve GC can be fraught with issues in a highly concurrent and/or complex syste

Re: [go-nuts] Huge map[string]*Object and GC where *Object is from sync.Pool

2018-09-27 Thread Peter Mogensen
On 09/27/2018 03:25 PM, Robert Engels wrote: > Based on my experience and I believe many others, I would caution against the > use of pools. Although it can be useful for objects that are very expensive > to create/initialize using it in a more general case just to improve GC can > be fraught

[go-nuts] Huge map[string]*Object and GC where *Object is from sync.Pool

2018-09-27 Thread Peter Mogensen
On 09/27/2018 03:12 PM, Ian Davis wrote: > > > On Thu, 27 Sep 2018, at 2:04 PM, Peter Mogensen wrote: >> >> Of course... it requires that you handle any collisions in hashing the >> string key to an int yourself, but wrt. the value I curious if anyone >> can see issued with just storing a uint

Re: [go-nuts] Huge map[string]*Object and GC where *Object is from sync.Pool

2018-09-27 Thread Ian Davis
On Thu, 27 Sep 2018, at 2:04 PM, Peter Mogensen wrote: > > Of course... it requires that you handle any collisions in hashing the > string key to an int yourself, but wrt. the value I curious if anyone > can see issued with just storing a uintptr instead of the pointer for > sync.Pool managed o

[go-nuts] Huge map[string]*Object and GC where *Object is from sync.Pool

2018-09-27 Thread Peter Mogensen
Hi, I noticed the GC fix which avoids scanning maps if both key and value are not pointer types. [1] and see it used [2] and discussed [3] So... Initially, I figured this was not worth thinking too much about, but then I came to thing about the relatively common use case where your map value is a