Re: [go-nuts] Why golang allocated memory space increases?

[Henrik Johansson]

Isn't this because of the GC tracking these and treating then as
effectively weak references to borrow a Java term?
If they are not pointers they are not tracked by the GC and I guess they
could all be removed at every scan?
Just guessing though, I haven't in any way checked it.

On Thu, Mar 26, 2020 at 11:34 AM Jakob Borg  wrote:

> On 26 Mar 2020, at 09:51, Tamás Gulácsi  wrote:
>
>
> sync.Pool MUST use pointers (*[]byte)
>
>
> I've seen this a lot but I confess I don't understand it. A []byte is
> essentially a fat pointer, what does it matter if we put that or a *[]byte
> into the pool? I understand that putting a []byte will result in an
> allocation of a new []byte, but probably the purpose is to preserve and
> avoid reallocating the backing array, which might be large.
>
> Is there something fundamental about the sync.Pool that means that a put
> *must* be zero-allocation for it to function? Likely the calling code
> around the puts/gets are not zero-allocation either.
>
> //jb
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/DB9E8E73-10B0-451B-BE66-13AF2930BB74%40kastelo.net
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAKOF6942VjqZ21b5ztW6wn6FTuk_MZDNz%2B3p1v2Njr_RAwRO4w%40mail.gmail.com.

Re: [go-nuts] Why golang allocated memory space increases?

[Jakob Borg]

On 26 Mar 2020, at 09:51, Tamás Gulácsi 
mailto:tgulacs...@gmail.com>> wrote:

sync.Pool MUST use pointers (*[]byte)

I've seen this a lot but I confess I don't understand it. A []byte is 
essentially a fat pointer, what does it matter if we put that or a *[]byte into 
the pool? I understand that putting a []byte will result in an allocation of a 
new []byte, but probably the purpose is to preserve and avoid reallocating the 
backing array, which might be large.

Is there something fundamental about the sync.Pool that means that a put *must* 
be zero-allocation for it to function? Likely the calling code around the 
puts/gets are not zero-allocation either.

//jb

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/DB9E8E73-10B0-451B-BE66-13AF2930BB74%40kastelo.net.

Re: [go-nuts] Why golang allocated memory space increases?

[Tamás Gulácsi]

2020. március 26., csütörtök 7:58:59 UTC+1 időpontban steve tang a 
következőt írta:
>
> Thanks,  I used sync.Pool to allocate a byte.Buffer,  and response.Body  
> isn't constant size in program, I find golang will grow the byte.Buffer 
> space against to initial space. I changed code as follows:
>
> type SyncPool struct{ Pool sync.Pool }
>
> func (s *SyncPool) Get(n int) []byte {
>//比n大，不需要slice增长时，使用buffer pool
>if b, ok := s.Pool.Get().([]byte); ok && cap(b) >= n {
>   return b[:n]
>} else {
>   //miss size
>   s.Put(b)
>   return make([]byte, n)
>}
> }
>
> func (s *SyncPool) Put(b []byte) {
>s.Pool.Put(b)
> }
>
>
>
> const FourMB = 4 * 1024 * 1024
>
> var pool = {
>sync.Pool{
>   New: func() interface{} {
>  return make([]byte, FourMB)
>   },
>},
> }
>
>
>
>
> On Thursday, March 26, 2020 at 12:22:01 PM UTC+8, robert engels wrote:
>>
>> If the pool is a sync.Pool:
>>
>> Any item stored in the Pool may be removed automatically at any time without 
>>18  // notification. If the Pool holds the only reference when this 
>> happens, the19  // item might be deallocated.
>>
>> So placing an object in the pool does not guarantee it won’t be collected 
>> - causing a future allocation.
>>
>> On Mar 25, 2020, at 10:22 PM, steve tang  wrote:
>>
>> chunkCrc32Hash := crc32.NewIEEE()
>> chunkBuf := bufferpool.GetInstance().Get()
>> //回收buffer
>> defer bufferpool.GetInstance().Put(chunkBuf)
>> writer := io.MultiWriter(chunkBuf, chunkCrc32Hash)
>> _, copyErr := io.Copy(writer, chunkResp.RawResponse.Body)
>> if copyErr != nil && copyErr != io.EOF {
>>logmgr.ErrorLogger(ctx, "fail to write chunk data to crc32 or buffer, 
>> req: %s, err: %s", originReq.PrettyPrint(), copyErr)
>>return false, exception.OEFException(nil, 
>> http.StatusInternalServerError, CopyChunkError, "internal server exception")
>> }
>>
>> In above codes,  it writes chunk data to buffer pool, and the chunk data 
>> size is 4MB.  When I used golang tool pprof to export memory pprof, there 
>> about 8MB space has been allocated.  Could somebody help me why golang 
>> allocated 
>> memory space increases?
>>
>>
>> <20200326-111844.png>
>>
>>
>>
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golan...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/6dbe93e6-9bb7-4112-87cb-085e68d9d393%40googlegroups.com
>>  
>> 
>> .
>> <20200326-111844.png>
>>
>>
>>
1. sync.Pool MUST use pointers (*[]byte)
2. what is `chunkBuf := bufferpool.GetInstance().Get()` ? it must be an 
io.Writer, but you should Reset/Truncate it when you Put it back to the 
pool!

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/e6e65488-d105-4880-bc61-90dd0ee9c7e2%40googlegroups.com.

Re: [go-nuts] Why golang allocated memory space increases?

[steve tang]

Thanks,  I used sync.Pool to allocate a byte.Buffer,  and response.Body  
isn't constant size in program, I find golang will grow the byte.Buffer 
space against to initial space. I changed code as follows:

type SyncPool struct{ Pool sync.Pool }

func (s *SyncPool) Get(n int) []byte {
   //比n大，不需要slice增长时，使用buffer pool
   if b, ok := s.Pool.Get().([]byte); ok && cap(b) >= n {
  return b[:n]
   } else {
  //miss size
  s.Put(b)
  return make([]byte, n)
   }
}

func (s *SyncPool) Put(b []byte) {
   s.Pool.Put(b)
}



const FourMB = 4 * 1024 * 1024

var pool = {
   sync.Pool{
  New: func() interface{} {
 return make([]byte, FourMB)
  },
   },
}




On Thursday, March 26, 2020 at 12:22:01 PM UTC+8, robert engels wrote:
>
> If the pool is a sync.Pool:
>
> Any item stored in the Pool may be removed automatically at any time without  
>   18  // notification. If the Pool holds the only reference when this 
> happens, the19  // item might be deallocated.
>
> So placing an object in the pool does not guarantee it won’t be collected 
> - causing a future allocation.
>
> On Mar 25, 2020, at 10:22 PM, steve tang > 
> wrote:
>
> chunkCrc32Hash := crc32.NewIEEE()
> chunkBuf := bufferpool.GetInstance().Get()
> //回收buffer
> defer bufferpool.GetInstance().Put(chunkBuf)
> writer := io.MultiWriter(chunkBuf, chunkCrc32Hash)
> _, copyErr := io.Copy(writer, chunkResp.RawResponse.Body)
> if copyErr != nil && copyErr != io.EOF {
>logmgr.ErrorLogger(ctx, "fail to write chunk data to crc32 or buffer, 
> req: %s, err: %s", originReq.PrettyPrint(), copyErr)
>return false, exception.OEFException(nil, 
> http.StatusInternalServerError, CopyChunkError, "internal server exception")
> }
>
> In above codes,  it writes chunk data to buffer pool, and the chunk data 
> size is 4MB.  When I used golang tool pprof to export memory pprof, there 
> about 8MB space has been allocated.  Could somebody help me why golang 
> allocated 
> memory space increases?
>
>
> <20200326-111844.png>
>
>
>
>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golan...@googlegroups.com .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/6dbe93e6-9bb7-4112-87cb-085e68d9d393%40googlegroups.com
>  
> 
> .
> <20200326-111844.png>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/e3c0f5b2-700a-49eb-9593-b4f936bb93e1%40googlegroups.com.

Re: [go-nuts] Why golang allocated memory space increases?

[robert engels]

If the pool is a sync.Pool:
Any item stored in the Pool may be removed automatically at any time without
18  // notification. If the Pool holds the only reference when this 
happens, the
19  // item might be deallocated.
So placing an object in the pool does not guarantee it won’t be collected - 
causing a future allocation.

> On Mar 25, 2020, at 10:22 PM, steve tang  wrote:
> 
> chunkCrc32Hash := crc32.NewIEEE()
> chunkBuf := bufferpool.GetInstance().Get()
> //回收buffer
> defer bufferpool.GetInstance().Put(chunkBuf)
> writer := io.MultiWriter(chunkBuf, chunkCrc32Hash)
> _, copyErr := io.Copy(writer, chunkResp.RawResponse.Body)
> if copyErr != nil && copyErr != io.EOF {
>logmgr.ErrorLogger(ctx, "fail to write chunk data to crc32 or buffer, req: 
> %s, err: %s", originReq.PrettyPrint(), copyErr)
>return false, exception.OEFException(nil, http.StatusInternalServerError, 
> CopyChunkError, "internal server exception")
> }
> 
> In above codes,  it writes chunk data to buffer pool, and the chunk data size 
> is 4MB.  When I used golang tool pprof to export memory pprof, there about 
> 8MB space has been allocated.  Could somebody help me why golang allocated 
> memory space increases?
> 
> 
> <20200326-111844.png>
> 
> 
> 
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/6dbe93e6-9bb7-4112-87cb-085e68d9d393%40googlegroups.com
>  
> .
> <20200326-111844.png>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/D42FB691-68B9-4925-BF33-38CE78089673%40ix.netcom.com.

[go-nuts] Why golang allocated memory space increases?

[steve tang]

chunkCrc32Hash := crc32.NewIEEE()
chunkBuf := bufferpool.GetInstance().Get()
//回收buffer
defer bufferpool.GetInstance().Put(chunkBuf)
writer := io.MultiWriter(chunkBuf, chunkCrc32Hash)
_, copyErr := io.Copy(writer, chunkResp.RawResponse.Body)
if copyErr != nil && copyErr != io.EOF {
   logmgr.ErrorLogger(ctx, "fail to write chunk data to crc32 or buffer, 
req: %s, err: %s", originReq.PrettyPrint(), copyErr)
   return false, exception.OEFException(nil, 
http.StatusInternalServerError, CopyChunkError, "internal server exception")
}

In above codes,  it writes chunk data to buffer pool, and the chunk data 
size is 4MB.  When I used golang tool pprof to export memory pprof, there 
about 8MB space has been allocated.  Could somebody help me why golang 
allocated 
memory space increases?


[image: 20200326-111844.png]



-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/6dbe93e6-9bb7-4112-87cb-085e68d9d393%40googlegroups.com.