Re: [go-nuts] Race detector compatible ring buffer Part II

['Axel Wagner' via golang-nuts]

I found a lock-free ringbuffer implementation written in C, which seems to
do what you want:
https://github.com/QuantumLeaps/lock-free-ring-buffer/blob/main/src/
This is a relatively direct translation to Go, using generics from go 1.18
(to be released very soon):
https://go.dev/play/p/nNEt66r71Yf?v=gotip
I tried running it with -race and got no complaints.

On Fri, Mar 11, 2022 at 7:52 AM Axel Wagner 
wrote:

> Uhm, I just actually looked at the code.
> You still use `r.start %= N`, which is a non-atomic access to r.start.
> AIUI, SPSC stands for "single producer, single consumer", i.e. you know
> that Get and Put will only be called from a single goroutine, respectively?
> In that case, you wouldn't even need atomics to manipulate r.start/r.end.
> Of course, you can't have a guarantee that your ringbuffer does not
> overflow, that way.
> But ISTM to make the conceptual code work with the race detector, you
> should use atomic.Values for the elements of the slice and then can use
> non-atomic accesses to r.start/r.end.
>
> On Fri, Mar 11, 2022 at 7:45 AM Axel Wagner 
> wrote:
>
>> You probably want to make the element type of the slice an atomic.Value,
>> instead of an interface{}. You shouldn't need a mutex then.
>>
>> On Fri, Mar 11, 2022 at 7:31 AM Cameron Elliott 
>> wrote:
>>
>>>
>>>
>>> Ian, thank you very much for the suggestion to use atomics.
>>> Unfortunately, for a standard SPSC ring buffer, I don't think it does
>>> the trick.
>>>
>>>
>>> I am attaching a simple ring buffer program at the end.
>>>
>>> If you run 'go run -race main.go' on the example,
>>> a race will occur.
>>> The race that occurs is a write, then read to the
>>> same element of a slice on two different goroutines.
>>>
>>> Of course a race-detected is expected.
>>>
>>> This can be fixed by mutexing Put() and Get(),
>>> because through some magic, mutexs affect the tables/tags
>>> the race detector maintains in order to catch races.
>>>
>>> Using sync.atomic on the ring buffer indexes doesn't
>>> affect the race-detector state for read and writes.
>>>
>>> I spent more time investigating, it seems there are
>>> two ways to make a traditional ring buffer compatible
>>> with the race detector:
>>>
>>> 1. Use build tags to conditionally Lock/Unlock mutexes
>>> where you would not actually need them, in order to reset
>>> the race detector on the object crossing goroutines.
>>>
>>> 2. Use the pragma //go:linkname to get access to the 'runtime.race'
>>> functions, in order to call Enable()/Disable/ReleaseMerge/Aquire
>>> as sync.Pool does in order to make the race detector happy.
>>>
>>> If there are other methods, please let me know!
>>> Thanks for any feedback!
>>>
>>> Cameron/Seattle
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> package main
>>>
>>> import (
>>> "sync/atomic"
>>> "time"
>>> )
>>>
>>> type RB struct {
>>> //mu sync.Mutex
>>> buf[]interface{}
>>> start, end int64
>>> }
>>>
>>>
>>> const N = 10
>>>
>>> func NewRB() *RB {
>>> return {buf: make([]interface{}, N)}
>>> }
>>>
>>> func (r *RB) Put(x interface{}) {
>>> //mu.Lock()
>>> //defer mu.Unlock()
>>>
>>> r.buf[r.end] = x
>>> atomic.AddInt64(,1)
>>> //r.end++
>>> r.end %= N
>>>
>>> }
>>>
>>> func (r *RB) Get() interface{} {
>>> //mu.Lock()
>>> //defer mu.Unlock()
>>>
>>> v := r.buf[r.start]
>>> atomic.AddInt64(,1)
>>> //r.start++
>>> r.start %= N
>>>
>>> return v
>>> }
>>>
>>> func main() {
>>>
>>> r := NewRB()
>>>
>>> go func() {
>>> r.Put(12345)
>>> }()
>>>
>>> time.Sleep(time.Millisecond)
>>>
>>> a := r.Get().(int)
>>> println(a)
>>> }
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to golang-nuts+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/golang-nuts/d7cc3410-c0bb-40d6-bf75-5e655ba3136en%40googlegroups.com
>>> 
>>> .
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAEkBMfEePeUQHqUYUxH_FrPxMGbUdLbsXXGLv2sq%2BwHWx97agg%40mail.gmail.com.

Re: [go-nuts] Race detector compatible ring buffer Part II

['Axel Wagner' via golang-nuts]

Uhm, I just actually looked at the code.
You still use `r.start %= N`, which is a non-atomic access to r.start.
AIUI, SPSC stands for "single producer, single consumer", i.e. you know
that Get and Put will only be called from a single goroutine, respectively?
In that case, you wouldn't even need atomics to manipulate r.start/r.end.
Of course, you can't have a guarantee that your ringbuffer does not
overflow, that way.
But ISTM to make the conceptual code work with the race detector, you
should use atomic.Values for the elements of the slice and then can use
non-atomic accesses to r.start/r.end.

On Fri, Mar 11, 2022 at 7:45 AM Axel Wagner 
wrote:

> You probably want to make the element type of the slice an atomic.Value,
> instead of an interface{}. You shouldn't need a mutex then.
>
> On Fri, Mar 11, 2022 at 7:31 AM Cameron Elliott  wrote:
>
>>
>>
>> Ian, thank you very much for the suggestion to use atomics.
>> Unfortunately, for a standard SPSC ring buffer, I don't think it does the
>> trick.
>>
>>
>> I am attaching a simple ring buffer program at the end.
>>
>> If you run 'go run -race main.go' on the example,
>> a race will occur.
>> The race that occurs is a write, then read to the
>> same element of a slice on two different goroutines.
>>
>> Of course a race-detected is expected.
>>
>> This can be fixed by mutexing Put() and Get(),
>> because through some magic, mutexs affect the tables/tags
>> the race detector maintains in order to catch races.
>>
>> Using sync.atomic on the ring buffer indexes doesn't
>> affect the race-detector state for read and writes.
>>
>> I spent more time investigating, it seems there are
>> two ways to make a traditional ring buffer compatible
>> with the race detector:
>>
>> 1. Use build tags to conditionally Lock/Unlock mutexes
>> where you would not actually need them, in order to reset
>> the race detector on the object crossing goroutines.
>>
>> 2. Use the pragma //go:linkname to get access to the 'runtime.race'
>> functions, in order to call Enable()/Disable/ReleaseMerge/Aquire
>> as sync.Pool does in order to make the race detector happy.
>>
>> If there are other methods, please let me know!
>> Thanks for any feedback!
>>
>> Cameron/Seattle
>>
>>
>>
>>
>>
>>
>>
>> package main
>>
>> import (
>> "sync/atomic"
>> "time"
>> )
>>
>> type RB struct {
>> //mu sync.Mutex
>> buf[]interface{}
>> start, end int64
>> }
>>
>>
>> const N = 10
>>
>> func NewRB() *RB {
>> return {buf: make([]interface{}, N)}
>> }
>>
>> func (r *RB) Put(x interface{}) {
>> //mu.Lock()
>> //defer mu.Unlock()
>>
>> r.buf[r.end] = x
>> atomic.AddInt64(,1)
>> //r.end++
>> r.end %= N
>>
>> }
>>
>> func (r *RB) Get() interface{} {
>> //mu.Lock()
>> //defer mu.Unlock()
>>
>> v := r.buf[r.start]
>> atomic.AddInt64(,1)
>> //r.start++
>> r.start %= N
>>
>> return v
>> }
>>
>> func main() {
>>
>> r := NewRB()
>>
>> go func() {
>> r.Put(12345)
>> }()
>>
>> time.Sleep(time.Millisecond)
>>
>> a := r.Get().(int)
>> println(a)
>> }
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/golang-nuts/d7cc3410-c0bb-40d6-bf75-5e655ba3136en%40googlegroups.com
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAEkBMfGQU3E%3DaAO71%3DjaauEY_S8JFCLUAucc3RaaB3PB2XeRrg%40mail.gmail.com.

Re: [go-nuts] Race detector compatible ring buffer Part II

['Axel Wagner' via golang-nuts]

You probably want to make the element type of the slice an atomic.Value,
instead of an interface{}. You shouldn't need a mutex then.

On Fri, Mar 11, 2022 at 7:31 AM Cameron Elliott  wrote:

>
>
> Ian, thank you very much for the suggestion to use atomics.
> Unfortunately, for a standard SPSC ring buffer, I don't think it does the
> trick.
>
>
> I am attaching a simple ring buffer program at the end.
>
> If you run 'go run -race main.go' on the example,
> a race will occur.
> The race that occurs is a write, then read to the
> same element of a slice on two different goroutines.
>
> Of course a race-detected is expected.
>
> This can be fixed by mutexing Put() and Get(),
> because through some magic, mutexs affect the tables/tags
> the race detector maintains in order to catch races.
>
> Using sync.atomic on the ring buffer indexes doesn't
> affect the race-detector state for read and writes.
>
> I spent more time investigating, it seems there are
> two ways to make a traditional ring buffer compatible
> with the race detector:
>
> 1. Use build tags to conditionally Lock/Unlock mutexes
> where you would not actually need them, in order to reset
> the race detector on the object crossing goroutines.
>
> 2. Use the pragma //go:linkname to get access to the 'runtime.race'
> functions, in order to call Enable()/Disable/ReleaseMerge/Aquire
> as sync.Pool does in order to make the race detector happy.
>
> If there are other methods, please let me know!
> Thanks for any feedback!
>
> Cameron/Seattle
>
>
>
>
>
>
>
> package main
>
> import (
> "sync/atomic"
> "time"
> )
>
> type RB struct {
> //mu sync.Mutex
> buf[]interface{}
> start, end int64
> }
>
>
> const N = 10
>
> func NewRB() *RB {
> return {buf: make([]interface{}, N)}
> }
>
> func (r *RB) Put(x interface{}) {
> //mu.Lock()
> //defer mu.Unlock()
>
> r.buf[r.end] = x
> atomic.AddInt64(,1)
> //r.end++
> r.end %= N
>
> }
>
> func (r *RB) Get() interface{} {
> //mu.Lock()
> //defer mu.Unlock()
>
> v := r.buf[r.start]
> atomic.AddInt64(,1)
> //r.start++
> r.start %= N
>
> return v
> }
>
> func main() {
>
> r := NewRB()
>
> go func() {
> r.Put(12345)
> }()
>
> time.Sleep(time.Millisecond)
>
> a := r.Get().(int)
> println(a)
> }
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/d7cc3410-c0bb-40d6-bf75-5e655ba3136en%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAEkBMfGDrre6Rc4E-npy33-W8CZy7_ZT-w%3DCc4LskR7N015j6Q%40mail.gmail.com.

[go-nuts] Race detector compatible ring buffer Part II

[Cameron Elliott]

Ian, thank you very much for the suggestion to use atomics.
Unfortunately, for a standard SPSC ring buffer, I don't think it does the 
trick.


I am attaching a simple ring buffer program at the end.

If you run 'go run -race main.go' on the example,
a race will occur.
The race that occurs is a write, then read to the
same element of a slice on two different goroutines.

Of course a race-detected is expected.

This can be fixed by mutexing Put() and Get(),
because through some magic, mutexs affect the tables/tags
the race detector maintains in order to catch races.

Using sync.atomic on the ring buffer indexes doesn't
affect the race-detector state for read and writes.

I spent more time investigating, it seems there are
two ways to make a traditional ring buffer compatible
with the race detector:

1. Use build tags to conditionally Lock/Unlock mutexes
where you would not actually need them, in order to reset
the race detector on the object crossing goroutines.

2. Use the pragma //go:linkname to get access to the 'runtime.race'
functions, in order to call Enable()/Disable/ReleaseMerge/Aquire
as sync.Pool does in order to make the race detector happy.

If there are other methods, please let me know!
Thanks for any feedback!

Cameron/Seattle







package main

import (
"sync/atomic"
"time"
)

type RB struct {
//mu sync.Mutex
buf[]interface{}
start, end int64
}


const N = 10

func NewRB() *RB {
return {buf: make([]interface{}, N)}
}

func (r *RB) Put(x interface{}) {
//mu.Lock()
//defer mu.Unlock()

r.buf[r.end] = x
atomic.AddInt64(,1)
//r.end++
r.end %= N

}

func (r *RB) Get() interface{} {
//mu.Lock()
//defer mu.Unlock()

v := r.buf[r.start]
atomic.AddInt64(,1)
//r.start++
r.start %= N

return v
}

func main() {

r := NewRB()

go func() {
r.Put(12345)
}()

time.Sleep(time.Millisecond)

a := r.Get().(int)
println(a)
}


-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/d7cc3410-c0bb-40d6-bf75-5e655ba3136en%40googlegroups.com.

Re: [go-nuts] Race detector compatible ring buffer implementation

[Ian Lance Taylor]

On Thu, Mar 10, 2022 at 9:32 PM Cameron Elliott  wrote:
>
> Is there a way to implement a standard no-Mutex ring buffer in
> a manner that is compatible with using the Go Race Detector?
>
> Of course, a standard ring buffer (no-mutex) as presented in Wikipedia
> will throw race detection errors even with "correct" usage.
>
> I suppose what I really need is access to "internal/race", as sync.Pool does.
> Is there a way to get this somehow?
>
> I know I can add a mutex to my ring buffer, to make the race detector happy.
> In fact, I can use the race build flags "// +build !race" to switch
> between sync.Mutex, and a no-operation sync.Locker implementation.
> Sadly, having compared the optimized assembly (objdump),
> the NOP-Locker doesn't entirely get eliminated, so,
> for performance reasons, I'd like to avoid this approach.
>
> Please don't say "don't use a ring buffer", there are applications where
> even in user-space, and in Go, usage is justified, IMHO.

Use the sync/atomic package.  https://pkg.go.dev/sync/atomic

Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAOyqgcUL1icFgauW8L16O5Zc1DApd3MwkAR3LcPrKPziRq5mOA%40mail.gmail.com.

[go-nuts] Race detector compatible ring buffer implementation

[Cameron Elliott]

Is there a way to implement a standard no-Mutex ring buffer in
a manner that is compatible with using the Go Race Detector?

Of course, a standard ring buffer (no-mutex) as presented in Wikipedia
will throw race detection errors even with "correct" usage.

I suppose what I really need is access to "internal/race", as sync.Pool 
does.
Is there a way to get this somehow?

I know I can add a mutex to my ring buffer, to make the race detector happy.
In fact, I can use the race build flags "// +build !race" to switch
between sync.Mutex, and a no-operation sync.Locker implementation.
Sadly, having compared the optimized assembly (objdump),
the NOP-Locker doesn't entirely get eliminated, so, 
for performance reasons, I'd like to avoid this approach.

Please don't say "don't use a ring buffer", there are applications where
even in user-space, and in Go, usage is justified, IMHO.

Thanks,
Cameron

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/7e0037a0-6b89-4be5-9902-0ab00d719f80n%40googlegroups.com.

Re: [go-nuts] Re: Pod memory keeps on increasing and restart with error OOMKilled

[Rakesh K R]

Hi,
Thank you. I know its kafka related question but thread started with issue 
in golang but later we are suspecting issue in go kafka library 
configuration.
FYI, I am interested in kafka client or producer properties. 
log.retention.hours or log.retention.bytes are kafka broker related 
configuration. so I am confused now

On Friday, March 11, 2022 at 1:34:23 AM UTC+5:30 ren...@ix.netcom.com wrote:

> Look at log.retention.hours and log.retention.bytes
>
> You should post this in the Kafka forums not the Go ones. 
>
> On Mar 10, 2022, at 11:04 AM, Rakesh K R  wrote:
>
> Hi,
>
> Sorry I am not sure which kafka configuration are you referring here. Can 
> you please point me to the right configuration responsible for retaining 
> the message for replay.
> I see following properties which might be related but not sure:
> queued.min.messages
> queued.max.messages.kbytes
> queue.buffering.max.messages
> queue.buffering.max.kbytes
> linger.ms ---> this is currently set to 1000
> message.timeout.ms
>
> Thank you
>
> On Thursday, March 10, 2022 at 9:50:50 PM UTC+5:30 ren...@ix.netcom.com 
> wrote:
>
>> You need to configure Kafka for how long it retains messages for replay - 
>> or some other option to store on disk. 
>>
>> On Mar 10, 2022, at 10:07 AM, Rakesh K R  wrote:
>>
>> Tamas,
>>
>> Thanks you. So any suggestion on how to make application release this 
>> 900MiB memory back to OS so that pod will not end up in OOMKilled state?
>>
>> On Thursday, March 10, 2022 at 1:45:18 PM UTC+5:30 Tamás Gulácsi wrote:
>>
>>> gopkg.in/confluentinc/confluent-kafka-go.v1/kafka._Cfunc_GoBytes
>>>
>>> says it uses cgo, hiding it's memory usage from Go. I bet that 900MiB of 
>>> memory is there...
>>>
>>>
>>> Rakesh K R a következőt írta (2022. március 10., csütörtök, 7:26:57 
>>> UTC+1):
>>>
 HI,
 I have a micro service application deployed on kubernetes cluster(with 
 1gb of pod memory limit). This app receive continuous messages(500 message 
 per second) from producer app over kafka interface(these messages are 
 encoded in protobuf format.)

 *Basic application flow:*
 1. Get the message one by one from kafka
 2. unmarshal proto message
 3. apply business logic
 4. write the message to redis cache(in []byte format)

 When pod starts memory will be around 50mb and memory starts increasing 
 as traffic flows into the application. It is never released back to OS. As 
 a result pod restarts with error code *OOMKilled*.
 I have integrated grafana to see memory usage like RSS, heap, stack.
 During this traffic flow, in-use heap size is 80mb, idle heap is 80mb 
 where as process resident memory is at 800-1000MB. Stopping the traffic 
 completely for hours did not help and RSS continue to remain in 1000mb.
 Tried to analyze this with pprof and it reports only 80mb are in in-use 
 section. So I am wondering where these remaining 800-1000mb of pods memory 
 went. Also application allocates memory like slices/maps/strings to 
 perform 
 business logic(see alloc_space pprof output below)

 I tried couple of experiments:
 1. Calling FreeOsMemory() in the app but that did not help
 2. invoking my app with GODEBUG=madvdontneed=1 my_app_executable and 
 did not help
 3. Leaving the application for 5-6hrs without any traffic to see 
 whether memory comes down. It did not help
 4. pprof shows only 80mb of heap in use
 5. Tried upgrading golang version from 1.13 to 1.16 as there were some 
 improvements in runtime. It did not help

 pprof output for *alloc_space*:

 (pprof) top20
 Showing nodes accounting for 481.98GB, 91.57% of 526.37GB total
 Dropped 566 nodes (cum <= 2.63GB)
 Showing top 20 nodes out of 114
   flat  flat%   sum%cum   cum%
78.89GB 14.99% 14.99%78.89GB 14.99%  
 github.com/go-redis/redis/v7/internal/proto.(*Reader).readStringReply
67.01GB 12.73% 27.72%   285.33GB 54.21% 
  airgroup/internal/wrapper/agrediswrapper.GetAllConfigurationForGroups
58.75GB 11.16% 38.88%58.75GB 11.16%  
 google.golang.org/protobuf/internal/impl.(*MessageInfo).MessageOf
52.26GB  9.93% 48.81%52.26GB  9.93%  reflect.unsafe_NewArray
45.78GB  8.70% 57.50%46.38GB  8.81% 
  encoding/json.(*decodeState).literalStore
36.98GB  7.02% 64.53%36.98GB  7.02%  reflect.New
28.20GB  5.36% 69.89%28.20GB  5.36%  
 gopkg.in/confluentinc/confluent-kafka-go.v1/kafka._Cfunc_GoBytes
25.60GB  4.86% 74.75%63.62GB 12.09%  
 google.golang.org/protobuf/proto.MarshalOptions.marshal
12.79GB  2.43% 77.18%   165.56GB 31.45% 
  encoding/json.(*decodeState).object
12.73GB  2.42% 79.60%12.73GB  2.42%  reflect.mapassign
11.05GB  2.10% 81.70%63.31GB 12.03%  reflect.MakeSlice
10.06GB  1.91% 83.61%12.36GB  2.35% 
  

Re: [go-nuts] Pointer to a pointer

[Rob Pike]

This topic has come up before. The scope of the identifier is set up
to allow, unlike in C, constructs such as

type stateFunction func() stateFunction

as is used in my talk about a scanner, https://talks.golang.org/2011/lex.slide

As I mentioned above, recursive type definitions appear even in an
early (and very interesting) test.

-rob



On Fri, Mar 11, 2022 at 1:01 AM Manlio Perillo  wrote:
>
> On Thursday, March 10, 2022 at 2:48:27 PM UTC+1 axel.wa...@googlemail.com 
> wrote:
>>
>> On Thu, Mar 10, 2022 at 2:38 PM Manlio Perillo  wrote:
>>>
>>> On Thursday, March 10, 2022 at 2:04:44 PM UTC+1 Jan Mercl wrote:

 On Thu, Mar 10, 2022 at 1:40 PM 'wagner riffel' via golang-nuts 
  wrote:

 > I don't think it's mentioned in the specification, my bet is that
 > unless your type requires inifnity amout of memory (eg: `type t struct
 > {t}`) or the type is an interface and break its rules, (eg: `type
 > iface interface{ iface }`) you can use self-reference.

 The validity of `type T *T` in Go is based on two things: 1) The 
 visibility of the identifier in `type T ...` is specified to start right 
 after the identifier, 2) It's possible, in this case, to compute the size 
 of type T. So no problem here.

>>>
>>> The only reference I found in the spec (after a quick search) is:
>>> 8. The scope of a type identifier declared inside a function begins at 
>>> the identifier in the TypeSpec and ends at the end of the innermost 
>>> containing block.
>>
>>
>> Also:
>>>
>>> The scope of an identifier denoting a constant, type, variable, or function 
>>> (but not method) declared at top level (outside any function) is the 
>>> package block.
>>
>>
>
> But this seems different from "The scope of a type identifier declared inside 
> a function **begins** at the identifier in the ...".
> My interpretation of the text you mentioned is: the identifier is **not** in 
> scope **until** the type definition is complete.
>
> Thanks
> Manlio
>
>>
>>
>>>
>>>
>>> > [...]
>>>
>>> Thanks
>>> Manlio
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups 
>>> "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to golang-nuts...@googlegroups.com.
>>>
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/golang-nuts/aba04b1a-29a3-4950-8ba8-c4d6a4d9aa3en%40googlegroups.com.
>
> --
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/43992fed-8c73-444d-aedc-19d549726896n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAOXNBZTid05AnogEeWGbCCRQrxD3AuC4gx0GmfEChLXKBf_G9g%40mail.gmail.com.

Re: [go-nuts] Re: Pod memory keeps on increasing and restart with error OOMKilled

[Robert Engels]

Look at log.retention.hours and log.retention.bytes

You should post this in the Kafka forums not the Go ones. 

> On Mar 10, 2022, at 11:04 AM, Rakesh K R  wrote:
> 
> Hi,
> Sorry I am not sure which kafka configuration are you referring here. Can you 
> please point me to the right configuration responsible for retaining the 
> message for replay.
> I see following properties which might be related but not sure:
> queued.min.messages
> queued.max.messages.kbytes
> queue.buffering.max.messages
> queue.buffering.max.kbytes
> linger.ms ---> this is currently set to 1000
> message.timeout.ms
> 
> Thank you
> 
>> On Thursday, March 10, 2022 at 9:50:50 PM UTC+5:30 ren...@ix.netcom.com 
>> wrote:
>> You need to configure Kafka for how long it retains messages for replay - or 
>> some other option to store on disk. 
>> 
 On Mar 10, 2022, at 10:07 AM, Rakesh K R  wrote:
 
>>> Tamas,
>> 
>>> Thanks you. So any suggestion on how to make application release this 
>>> 900MiB memory back to OS so that pod will not end up in OOMKilled state?
>>> 
> On Thursday, March 10, 2022 at 1:45:18 PM UTC+5:30 Tamás Gulácsi wrote:
> gopkg.in/confluentinc/confluent-kafka-go.v1/kafka._Cfunc_GoBytes
> 
> says it uses cgo, hiding it's memory usage from Go. I bet that 900MiB of 
> memory is there...
> 
> 
> Rakesh K R a következőt írta (2022. március 10., csütörtök, 7:26:57 
> UTC+1):
>> HI,
>> I have a micro service application deployed on kubernetes cluster(with 
>> 1gb of pod memory limit). This app receive continuous messages(500 
>> message per second) from producer app over kafka interface(these 
>> messages are encoded in protobuf format.)
>> 
>> Basic application flow:
>> 1. Get the message one by one from kafka
>> 2. unmarshal proto message
>> 3. apply business logic
>> 4. write the message to redis cache(in []byte format)
>> 
>> When pod starts memory will be around 50mb and memory starts increasing 
>> as traffic flows into the application. It is never released back to OS. 
>> As a result pod restarts with error code OOMKilled.
>> I have integrated grafana to see memory usage like RSS, heap, stack.
>> During this traffic flow, in-use heap size is 80mb, idle heap is 80mb 
>> where as process resident memory is at 800-1000MB. Stopping the traffic 
>> completely for hours did not help and RSS continue to remain in 1000mb.
>> Tried to analyze this with pprof and it reports only 80mb are in in-use 
>> section. So I am wondering where these remaining 800-1000mb of pods 
>> memory went. Also application allocates memory like slices/maps/strings 
>> to perform business logic(see alloc_space pprof output below)
>> 
>> I tried couple of experiments:
>> 1. Calling FreeOsMemory() in the app but that did not help
>> 2. invoking my app with GODEBUG=madvdontneed=1 my_app_executable and did 
>> not help
>> 3. Leaving the application for 5-6hrs without any traffic to see whether 
>> memory comes down. It  did not help
>> 4. pprof shows only 80mb of heap in use
>> 5. Tried upgrading golang version from 1.13 to 1.16 as there were some 
>> improvements in runtime. It did not help
>> 
>> pprof output for alloc_space:
>> 
>> (pprof) top20
>> Showing nodes accounting for 481.98GB, 91.57% of 526.37GB total
>> Dropped 566 nodes (cum <= 2.63GB)
>> Showing top 20 nodes out of 114
>>   flat  flat%   sum%cum   cum%
>>78.89GB 14.99% 14.99%78.89GB 14.99%  
>> github.com/go-redis/redis/v7/internal/proto.(*Reader).readStringReply
>>67.01GB 12.73% 27.72%   285.33GB 54.21%  
>> airgroup/internal/wrapper/agrediswrapper.GetAllConfigurationForGroups
>>58.75GB 11.16% 38.88%58.75GB 11.16%  
>> google.golang.org/protobuf/internal/impl.(*MessageInfo).MessageOf
>>52.26GB  9.93% 48.81%52.26GB  9.93%  reflect.unsafe_NewArray
>>45.78GB  8.70% 57.50%46.38GB  8.81%  
>> encoding/json.(*decodeState).literalStore
>>36.98GB  7.02% 64.53%36.98GB  7.02%  reflect.New
>>28.20GB  5.36% 69.89%28.20GB  5.36%  
>> gopkg.in/confluentinc/confluent-kafka-go.v1/kafka._Cfunc_GoBytes
>>25.60GB  4.86% 74.75%63.62GB 12.09%  
>> google.golang.org/protobuf/proto.MarshalOptions.marshal
>>12.79GB  2.43% 77.18%   165.56GB 31.45%  
>> encoding/json.(*decodeState).object
>>12.73GB  2.42% 79.60%12.73GB  2.42%  reflect.mapassign
>>11.05GB  2.10% 81.70%63.31GB 12.03%  reflect.MakeSlice
>>10.06GB  1.91% 83.61%12.36GB  2.35%  
>> filterServersForDestinationDevicesAndSendToDistributionChan
>> 6.92GB  1.32% 84.92%   309.45GB 58.79%  
>> groupAndSendToConfigPolicyChannel
>> 6.79GB  1.29% 86.21%48.85GB  9.28%  
>> publishInternalMsgToDistributionService
>> 6.79GB  1.29% 87.50%   174.81GB 

[go-nuts] Re: Possible float64 precision problem

[Wojciech Muła]

On Wednesday, March 9, 2022 at 12:37:10 PM UTC+1 christoph...@gmail.com 
wrote:

> This value is converted to float64/double, and divided by 2^64.
> The resulting number is multiplied by 1e8. 
>
> Could it be that the C program is performing the computation with long 
> double (80 bit) precision and that Go is doing it with 64bit precision ? 
>

Unlikely. Currently C/C++ compilers and Go use SSE instructions (on x86) 
for floating-point calculations. In GCC/clang you have to either explicitly 
set compiler flags to use an FPU (thus 80-bit precision) or use the `long 
double` type in your C code.

w.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/59525e27-52b7-41d3-8519-7a4caed72a5an%40googlegroups.com.

Re: [go-nuts] Re: Pod memory keeps on increasing and restart with error OOMKilled

[Rakesh K R]

Hi,
Sorry I am not sure which kafka configuration are you referring here. Can 
you please point me to the right configuration responsible for retaining 
the message for replay.
I see following properties which might be related but not sure:
queued.min.messages
queued.max.messages.kbytes
queue.buffering.max.messages
queue.buffering.max.kbytes
linger.ms ---> this is currently set to 1000
message.timeout.ms

Thank you

On Thursday, March 10, 2022 at 9:50:50 PM UTC+5:30 ren...@ix.netcom.com 
wrote:

> You need to configure Kafka for how long it retains messages for replay - 
> or some other option to store on disk. 
>
> On Mar 10, 2022, at 10:07 AM, Rakesh K R  wrote:
>
> Tamas,
>
> Thanks you. So any suggestion on how to make application release this 
> 900MiB memory back to OS so that pod will not end up in OOMKilled state?
>
> On Thursday, March 10, 2022 at 1:45:18 PM UTC+5:30 Tamás Gulácsi wrote:
>
>> gopkg.in/confluentinc/confluent-kafka-go.v1/kafka._Cfunc_GoBytes
>>
>> says it uses cgo, hiding it's memory usage from Go. I bet that 900MiB of 
>> memory is there...
>>
>>
>> Rakesh K R a következőt írta (2022. március 10., csütörtök, 7:26:57 
>> UTC+1):
>>
>>> HI,
>>> I have a micro service application deployed on kubernetes cluster(with 
>>> 1gb of pod memory limit). This app receive continuous messages(500 message 
>>> per second) from producer app over kafka interface(these messages are 
>>> encoded in protobuf format.)
>>>
>>> *Basic application flow:*
>>> 1. Get the message one by one from kafka
>>> 2. unmarshal proto message
>>> 3. apply business logic
>>> 4. write the message to redis cache(in []byte format)
>>>
>>> When pod starts memory will be around 50mb and memory starts increasing 
>>> as traffic flows into the application. It is never released back to OS. As 
>>> a result pod restarts with error code *OOMKilled*.
>>> I have integrated grafana to see memory usage like RSS, heap, stack.
>>> During this traffic flow, in-use heap size is 80mb, idle heap is 80mb 
>>> where as process resident memory is at 800-1000MB. Stopping the traffic 
>>> completely for hours did not help and RSS continue to remain in 1000mb.
>>> Tried to analyze this with pprof and it reports only 80mb are in in-use 
>>> section. So I am wondering where these remaining 800-1000mb of pods memory 
>>> went. Also application allocates memory like slices/maps/strings to perform 
>>> business logic(see alloc_space pprof output below)
>>>
>>> I tried couple of experiments:
>>> 1. Calling FreeOsMemory() in the app but that did not help
>>> 2. invoking my app with GODEBUG=madvdontneed=1 my_app_executable and 
>>> did not help
>>> 3. Leaving the application for 5-6hrs without any traffic to see whether 
>>> memory comes down. It did not help
>>> 4. pprof shows only 80mb of heap in use
>>> 5. Tried upgrading golang version from 1.13 to 1.16 as there were some 
>>> improvements in runtime. It did not help
>>>
>>> pprof output for *alloc_space*:
>>>
>>> (pprof) top20
>>> Showing nodes accounting for 481.98GB, 91.57% of 526.37GB total
>>> Dropped 566 nodes (cum <= 2.63GB)
>>> Showing top 20 nodes out of 114
>>>   flat  flat%   sum%cum   cum%
>>>78.89GB 14.99% 14.99%78.89GB 14.99%  
>>> github.com/go-redis/redis/v7/internal/proto.(*Reader).readStringReply
>>>67.01GB 12.73% 27.72%   285.33GB 54.21% 
>>>  airgroup/internal/wrapper/agrediswrapper.GetAllConfigurationForGroups
>>>58.75GB 11.16% 38.88%58.75GB 11.16%  
>>> google.golang.org/protobuf/internal/impl.(*MessageInfo).MessageOf
>>>52.26GB  9.93% 48.81%52.26GB  9.93%  reflect.unsafe_NewArray
>>>45.78GB  8.70% 57.50%46.38GB  8.81% 
>>>  encoding/json.(*decodeState).literalStore
>>>36.98GB  7.02% 64.53%36.98GB  7.02%  reflect.New
>>>28.20GB  5.36% 69.89%28.20GB  5.36%  
>>> gopkg.in/confluentinc/confluent-kafka-go.v1/kafka._Cfunc_GoBytes
>>>25.60GB  4.86% 74.75%63.62GB 12.09%  
>>> google.golang.org/protobuf/proto.MarshalOptions.marshal
>>>12.79GB  2.43% 77.18%   165.56GB 31.45% 
>>>  encoding/json.(*decodeState).object
>>>12.73GB  2.42% 79.60%12.73GB  2.42%  reflect.mapassign
>>>11.05GB  2.10% 81.70%63.31GB 12.03%  reflect.MakeSlice
>>>10.06GB  1.91% 83.61%12.36GB  2.35% 
>>>  filterServersForDestinationDevicesAndSendToDistributionChan
>>> 6.92GB  1.32% 84.92%   309.45GB 58.79% 
>>>  groupAndSendToConfigPolicyChannel
>>> 6.79GB  1.29% 86.21%48.85GB  9.28% 
>>>  publishInternalMsgToDistributionService
>>> 6.79GB  1.29% 87.50%   174.81GB 33.21%  encoding/json.Unmarshal
>>> 6.14GB  1.17% 88.67% 6.14GB  1.17%  
>>> google.golang.org/protobuf/internal/impl.consumeBytes
>>> 4.64GB  0.88% 89.55%14.39GB  2.73% 
>>>  GetAllDevDataFromGlobalDevDataDb
>>> 4.11GB  0.78% 90.33%18.47GB  3.51% 
>>>  GetAllServersFromServerRecordDb
>>> 3.27GB  0.62% 90.95% 3.27GB  0.62%  net.HardwareAddr.String
>>> 3.23GB  0.61% 91.57% 3.23GB  0.61%  reflect.makemap

Re: [go-nuts] Re: Pod memory keeps on increasing and restart with error OOMKilled

[Robert Engels]

You need to configure Kafka for how long it retains messages for replay - or 
some other option to store on disk. 

> On Mar 10, 2022, at 10:07 AM, Rakesh K R  wrote:
> 
> Tamas,
> Thanks you. So any suggestion on how to make application release this 900MiB 
> memory back to OS so that pod will not end up in OOMKilled state?
> 
>> On Thursday, March 10, 2022 at 1:45:18 PM UTC+5:30 Tamás Gulácsi wrote:
>> gopkg.in/confluentinc/confluent-kafka-go.v1/kafka._Cfunc_GoBytes
>> 
>> says it uses cgo, hiding it's memory usage from Go. I bet that 900MiB of 
>> memory is there...
>> 
>> 
>> Rakesh K R a következőt írta (2022. március 10., csütörtök, 7:26:57 UTC+1):
>>> HI,
>>> I have a micro service application deployed on kubernetes cluster(with 1gb 
>>> of pod memory limit). This app receive continuous messages(500 message per 
>>> second) from producer app over kafka interface(these messages are encoded 
>>> in protobuf format.)
>>> 
>>> Basic application flow:
>>> 1. Get the message one by one from kafka
>>> 2. unmarshal proto message
>>> 3. apply business logic
>>> 4. write the message to redis cache(in []byte format)
>>> 
>>> When pod starts memory will be around 50mb and memory starts increasing as 
>>> traffic flows into the application. It is never released back to OS. As a 
>>> result pod restarts with error code OOMKilled.
>>> I have integrated grafana to see memory usage like RSS, heap, stack.
>>> During this traffic flow, in-use heap size is 80mb, idle heap is 80mb where 
>>> as process resident memory is at 800-1000MB. Stopping the traffic 
>>> completely for hours did not help and RSS continue to remain in 1000mb.
>>> Tried to analyze this with pprof and it reports only 80mb are in in-use 
>>> section. So I am wondering where these remaining 800-1000mb of pods memory 
>>> went. Also application allocates memory like slices/maps/strings to perform 
>>> business logic(see alloc_space pprof output below)
>>> 
>>> I tried couple of experiments:
>>> 1. Calling FreeOsMemory() in the app but that did not help
>>> 2. invoking my app with GODEBUG=madvdontneed=1 my_app_executable and did 
>>> not help
>>> 3. Leaving the application for 5-6hrs without any traffic to see whether 
>>> memory comes down. It  did not help
>>> 4. pprof shows only 80mb of heap in use
>>> 5. Tried upgrading golang version from 1.13 to 1.16 as there were some 
>>> improvements in runtime. It did not help
>>> 
>>> pprof output for alloc_space:
>>> 
>>> (pprof) top20
>>> Showing nodes accounting for 481.98GB, 91.57% of 526.37GB total
>>> Dropped 566 nodes (cum <= 2.63GB)
>>> Showing top 20 nodes out of 114
>>>   flat  flat%   sum%cum   cum%
>>>78.89GB 14.99% 14.99%78.89GB 14.99%  
>>> github.com/go-redis/redis/v7/internal/proto.(*Reader).readStringReply
>>>67.01GB 12.73% 27.72%   285.33GB 54.21%  
>>> airgroup/internal/wrapper/agrediswrapper.GetAllConfigurationForGroups
>>>58.75GB 11.16% 38.88%58.75GB 11.16%  
>>> google.golang.org/protobuf/internal/impl.(*MessageInfo).MessageOf
>>>52.26GB  9.93% 48.81%52.26GB  9.93%  reflect.unsafe_NewArray
>>>45.78GB  8.70% 57.50%46.38GB  8.81%  
>>> encoding/json.(*decodeState).literalStore
>>>36.98GB  7.02% 64.53%36.98GB  7.02%  reflect.New
>>>28.20GB  5.36% 69.89%28.20GB  5.36%  
>>> gopkg.in/confluentinc/confluent-kafka-go.v1/kafka._Cfunc_GoBytes
>>>25.60GB  4.86% 74.75%63.62GB 12.09%  
>>> google.golang.org/protobuf/proto.MarshalOptions.marshal
>>>12.79GB  2.43% 77.18%   165.56GB 31.45%  
>>> encoding/json.(*decodeState).object
>>>12.73GB  2.42% 79.60%12.73GB  2.42%  reflect.mapassign
>>>11.05GB  2.10% 81.70%63.31GB 12.03%  reflect.MakeSlice
>>>10.06GB  1.91% 83.61%12.36GB  2.35%  
>>> filterServersForDestinationDevicesAndSendToDistributionChan
>>> 6.92GB  1.32% 84.92%   309.45GB 58.79%  
>>> groupAndSendToConfigPolicyChannel
>>> 6.79GB  1.29% 86.21%48.85GB  9.28%  
>>> publishInternalMsgToDistributionService
>>> 6.79GB  1.29% 87.50%   174.81GB 33.21%  encoding/json.Unmarshal
>>> 6.14GB  1.17% 88.67% 6.14GB  1.17%  
>>> google.golang.org/protobuf/internal/impl.consumeBytes
>>> 4.64GB  0.88% 89.55%14.39GB  2.73%  GetAllDevDataFromGlobalDevDataDb
>>> 4.11GB  0.78% 90.33%18.47GB  3.51%  GetAllServersFromServerRecordDb
>>> 3.27GB  0.62% 90.95% 3.27GB  0.62%  net.HardwareAddr.String
>>> 3.23GB  0.61% 91.57% 3.23GB  0.61%  reflect.makemap
>>> (pprof)
>>> 
>>> 
>>> Need experts help in analyzing this issue.
>>> 
>>> Thanks in advance!!
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/e9b91937-7bf5-4526-940f-2a60b2989ddfn%40googlegroups.com.

-- 
You received this message 

[go-nuts] Re: Pod memory keeps on increasing and restart with error OOMKilled

[Rakesh K R]

Tamas,
Thanks you. So any suggestion on how to make application release this 
900MiB memory back to OS so that pod will not end up in OOMKilled state?

On Thursday, March 10, 2022 at 1:45:18 PM UTC+5:30 Tamás Gulácsi wrote:

> gopkg.in/confluentinc/confluent-kafka-go.v1/kafka._Cfunc_GoBytes
>
> says it uses cgo, hiding it's memory usage from Go. I bet that 900MiB of 
> memory is there...
>
>
> Rakesh K R a következőt írta (2022. március 10., csütörtök, 7:26:57 UTC+1):
>
>> HI,
>> I have a micro service application deployed on kubernetes cluster(with 
>> 1gb of pod memory limit). This app receive continuous messages(500 message 
>> per second) from producer app over kafka interface(these messages are 
>> encoded in protobuf format.)
>>
>> *Basic application flow:*
>> 1. Get the message one by one from kafka
>> 2. unmarshal proto message
>> 3. apply business logic
>> 4. write the message to redis cache(in []byte format)
>>
>> When pod starts memory will be around 50mb and memory starts increasing 
>> as traffic flows into the application. It is never released back to OS. As 
>> a result pod restarts with error code *OOMKilled*.
>> I have integrated grafana to see memory usage like RSS, heap, stack.
>> During this traffic flow, in-use heap size is 80mb, idle heap is 80mb 
>> where as process resident memory is at 800-1000MB. Stopping the traffic 
>> completely for hours did not help and RSS continue to remain in 1000mb.
>> Tried to analyze this with pprof and it reports only 80mb are in in-use 
>> section. So I am wondering where these remaining 800-1000mb of pods memory 
>> went. Also application allocates memory like slices/maps/strings to perform 
>> business logic(see alloc_space pprof output below)
>>
>> I tried couple of experiments:
>> 1. Calling FreeOsMemory() in the app but that did not help
>> 2. invoking my app with GODEBUG=madvdontneed=1 my_app_executable and did 
>> not help
>> 3. Leaving the application for 5-6hrs without any traffic to see whether 
>> memory comes down. It did not help
>> 4. pprof shows only 80mb of heap in use
>> 5. Tried upgrading golang version from 1.13 to 1.16 as there were some 
>> improvements in runtime. It did not help
>>
>> pprof output for *alloc_space*:
>>
>> (pprof) top20
>> Showing nodes accounting for 481.98GB, 91.57% of 526.37GB total
>> Dropped 566 nodes (cum <= 2.63GB)
>> Showing top 20 nodes out of 114
>>   flat  flat%   sum%cum   cum%
>>78.89GB 14.99% 14.99%78.89GB 14.99%  
>> github.com/go-redis/redis/v7/internal/proto.(*Reader).readStringReply
>>67.01GB 12.73% 27.72%   285.33GB 54.21% 
>>  airgroup/internal/wrapper/agrediswrapper.GetAllConfigurationForGroups
>>58.75GB 11.16% 38.88%58.75GB 11.16%  
>> google.golang.org/protobuf/internal/impl.(*MessageInfo).MessageOf
>>52.26GB  9.93% 48.81%52.26GB  9.93%  reflect.unsafe_NewArray
>>45.78GB  8.70% 57.50%46.38GB  8.81% 
>>  encoding/json.(*decodeState).literalStore
>>36.98GB  7.02% 64.53%36.98GB  7.02%  reflect.New
>>28.20GB  5.36% 69.89%28.20GB  5.36%  
>> gopkg.in/confluentinc/confluent-kafka-go.v1/kafka._Cfunc_GoBytes
>>25.60GB  4.86% 74.75%63.62GB 12.09%  
>> google.golang.org/protobuf/proto.MarshalOptions.marshal
>>12.79GB  2.43% 77.18%   165.56GB 31.45% 
>>  encoding/json.(*decodeState).object
>>12.73GB  2.42% 79.60%12.73GB  2.42%  reflect.mapassign
>>11.05GB  2.10% 81.70%63.31GB 12.03%  reflect.MakeSlice
>>10.06GB  1.91% 83.61%12.36GB  2.35% 
>>  filterServersForDestinationDevicesAndSendToDistributionChan
>> 6.92GB  1.32% 84.92%   309.45GB 58.79% 
>>  groupAndSendToConfigPolicyChannel
>> 6.79GB  1.29% 86.21%48.85GB  9.28% 
>>  publishInternalMsgToDistributionService
>> 6.79GB  1.29% 87.50%   174.81GB 33.21%  encoding/json.Unmarshal
>> 6.14GB  1.17% 88.67% 6.14GB  1.17%  
>> google.golang.org/protobuf/internal/impl.consumeBytes
>> 4.64GB  0.88% 89.55%14.39GB  2.73% 
>>  GetAllDevDataFromGlobalDevDataDb
>> 4.11GB  0.78% 90.33%18.47GB  3.51% 
>>  GetAllServersFromServerRecordDb
>> 3.27GB  0.62% 90.95% 3.27GB  0.62%  net.HardwareAddr.String
>> 3.23GB  0.61% 91.57% 3.23GB  0.61%  reflect.makemap
>> (pprof)
>>
>>
>> Need experts help in analyzing this issue.
>>
>> Thanks in advance!!
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/e9b91937-7bf5-4526-940f-2a60b2989ddfn%40googlegroups.com.

Re: [go-nuts] Pointer to a pointer

[Manlio Perillo]

On Thursday, March 10, 2022 at 2:48:27 PM UTC+1 axel.wa...@googlemail.com 
wrote:

> On Thu, Mar 10, 2022 at 2:38 PM Manlio Perillo  
> wrote:
>
>> On Thursday, March 10, 2022 at 2:04:44 PM UTC+1 Jan Mercl wrote:
>>
>>> On Thu, Mar 10, 2022 at 1:40 PM 'wagner riffel' via golang-nuts <
>>> golan...@googlegroups.com> wrote:
>>>
>>> > I don't think it's mentioned in the specification, my bet is that
>>> > unless your type requires inifnity amout of memory (eg: `type t struct
>>> > {t}`) or the type is an interface and break its rules, (eg: `type
>>> > iface interface{ iface }`) you can use self-reference.
>>>
>>> The validity of `type T *T` in Go is based on two things: 1) The 
>>> visibility of the identifier in `type T ...` is specified to start right 
>>> after the identifier, 2) It's possible, in this case, to compute the size 
>>> of type T. So no problem here.
>>>
>>>
>> The only reference I found in the spec (after a quick search) is:
>> 8. The scope of a type identifier declared inside a function begins 
>> at the identifier in the TypeSpec and ends at the end of the innermost 
>> containing block.
>>
>
> Also:
>
>> The scope of an identifier denoting a constant, type, variable, or 
>> function (but not method) declared at top level (outside any function) is 
>> the package block.
>
>
>
But this seems different from "The scope of a type identifier declared 
inside a function **begins** at the identifier in the ...".
My interpretation of the text you mentioned is: the identifier is **not** 
in scope **until** the type definition is complete.

Thanks
Manlio


>  
>
>>
>> > [...]
>>
>> Thanks 
>> Manlio
>>
> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts...@googlegroups.com.
>>
> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/aba04b1a-29a3-4950-8ba8-c4d6a4d9aa3en%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/43992fed-8c73-444d-aedc-19d549726896n%40googlegroups.com.

Re: [go-nuts] Pointer to a pointer

[Holloway Kean Ho]

HI all, 

> Does anyone know what the upper bound on this could be?

Interesting question. I have a thought experiment theory: "LIMIT_MAX = 
(TOTAL_MEMORY - OTHER_SERVICES - DATA) / MBIT"

Explanation:
1. Assuming you're using 64-bit OS and memory structure is aligned using 
64-bit memory, your MBIT should be `uint64` for each memory address as a 
value.
2. Each level of pointer (e.g. pointer of pointer ) shall only consume 1 
`uint64` memory space.
3. Iterate the thinking recursively and you eventually ran out of memory 
available on a given system.
4. Hence, the upper limit, LIMIT_MAX shall be the remaining free memory 
after subtracting other services consuming the memory space 
(OTHER_SERVICES)  and the initial data size, the value of the first degree 
pointer (DATA).

Note:
1. For 32-bit memory addressing, it's at max 4GB at `uint32` per iteration 
due to the addressing capacity limit, which is why 64-bit CPU is created at 
the first place.


I may be wrong but I'm curious to know. As for how to prove it, I have no 
idea and I don't intend to crash my laptop at the moment. Besides, I never 
use pointer of pointer beyond 3 degree deep. =p

Regards,
Holloway

On Thursday, March 10, 2022 at 2:53:05 PM UTC+8 kortschak wrote:

> On Wed, 2022-03-09 at 18:58 -0800, shan...@gmail.com wrote:
> > This morning someone asked about dereferincing a pointer to a pointer
> > to a pointer
> >
> > At first nobody had ever thought about, let alone knew the answer,
> > but some example code was shown, and sure enough ***val is possible
> > ```
> > package main
> >
> > import "fmt"
> >
> > func main() {
> > a := 0
> > b := 
> > c := 
> > UltimatePointOne()
> > fmt.Println(a)
> > }
> >
> > func UltimatePointOne(n ***int) {
> > ***n = 1
> > }
> > ```
> >
> >
> > On a lark a go playground example was tried to find what the maximum
> > * is in Go
> >
> > https://go.dev/play/p/YhibY3p7TSD
> >
> > There's 28 there, but it's not the limit
> >
> > Does anyone know what the upper bound on this could be?
> >
> > 256 * ?
> >
> > 32k * ?
>
> I think aspects of this thread are sad. None of us know the background
> of the OP and this kind of thinking illustrates a joyful level of
> curiosity that could have been answered in a way that helps the
> questioner and build a positive community (for Shane, Rob did answer it
> and in way that is really quite deep, and thinking about how he
> answered it will teach you something that is worth learning).
>
> Calling a question "silly" demeans the questioner without understanding
> where they are coming from.
>
> Dan
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/50570864-f5f2-4835-b22b-6852bfb49949n%40googlegroups.com.

Re: [go-nuts] Pointer to a pointer

[Manlio Perillo]

On Thursday, March 10, 2022 at 2:04:44 PM UTC+1 Jan Mercl wrote:

> On Thu, Mar 10, 2022 at 1:40 PM 'wagner riffel' via golang-nuts <
> golan...@googlegroups.com> wrote:
>
> > I don't think it's mentioned in the specification, my bet is that
> > unless your type requires inifnity amout of memory (eg: `type t struct
> > {t}`) or the type is an interface and break its rules, (eg: `type
> > iface interface{ iface }`) you can use self-reference.
>
> The validity of `type T *T` in Go is based on two things: 1) The 
> visibility of the identifier in `type T ...` is specified to start right 
> after the identifier, 2) It's possible, in this case, to compute the size 
> of type T. So no problem here.
>
>
The only reference I found in the spec (after a quick search) is:
8. The scope of a type identifier declared inside a function begins at 
the identifier in the TypeSpec and ends at the end of the innermost 
containing block.

> [...]

Thanks 
Manlio

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/aba04b1a-29a3-4950-8ba8-c4d6a4d9aa3en%40googlegroups.com.

Re: [go-nuts] Pointer to a pointer

[Jan Mercl]

On Thu, Mar 10, 2022 at 2:18 PM 'Axel Wagner' via golang-nuts
 wrote:

> TBH I find it rather surprising that the spec does not mention why `type X X` 
> is *not* allowed.

The specs guarantee unsafe.Sizeof() to produce the size. I think that
gives a Go compiler the right to reject that type definition.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAA40n-Vct2AQLZnqAR3mGnwK7tLB9_crz6Nfj%3DkrFYn4OgXk%3DA%40mail.gmail.com.

Re: [go-nuts] Pointer to a pointer

['Axel Wagner' via golang-nuts]

TBH I find it rather surprising that the spec does not mention why `type X
X` is *not* allowed.
It's obvious that it can't be, but from what I can tell, the spec doesn't
forbid it.
Otherwise I would've chalked up the validity of `type X *X` for "anything
that's not forbidden, is allowed". A type definition
 is

TypeDef = identifier Type .
> Type = TypeName | TypeLit | "(" Type ")" .
> TypeName  = identifier | QualifiedIdent .
> TypeLit   = … | PointerType | … .

PointerType = "*" BaseType .

BaseType= Type .


So `type X *X` is obviously valid (after taking scope into account
, as Jam mentions). But so
would `type X X` be.

On Thu, Mar 10, 2022 at 2:04 PM Jan Mercl <0xj...@gmail.com> wrote:

> On Thu, Mar 10, 2022 at 1:40 PM 'wagner riffel' via golang-nuts <
> golang-nuts@googlegroups.com> wrote:
>
> > I don't think it's mentioned in the specification, my bet is that
> > unless your type requires inifnity amout of memory (eg: `type t struct
> > {t}`) or the type is an interface and break its rules, (eg: `type
> > iface interface{ iface }`) you can use self-reference.
>
> The validity of `type T *T` in Go is based on two things: 1) The
> visibility of the identifier in `type T ...` is specified to start right
> after the identifier, 2) It's possible, in this case, to compute the size
> of type T. So no problem here.
>
> > You're correct, C doesn't allow self reference in type decl, more ...
>
> In `typedef self *self;` the visibility of the second instance of
> identifier `self` starts only after it, ie. preceding the final `;'. So the
> first identifier `self`, the one after `typedef` is undefined and that's
> the real reason it does not work. However:
>
> 
> jnml@e5-1650:~/tmp$ cat main.c
> typedef int T;
> typedef T T;
>
> int main() {}
> jnml@e5-1650:~/tmp$ gcc -Wall main.c
> jnml@e5-1650:~/tmp$
> 
>
> This works because typedef is equal to Go type aliases. But the full Go
> thing cannot work:
>
> 
> jnml@e5-1650:~/tmp$ cat main.c
> typedef int T;
> typedef T *T;
>
> int main() {}
> jnml@e5-1650:~/tmp$ gcc -Wall main.c
> main.c:2:12: error: conflicting types for ‘T’
> 2 | typedef T *T;
>   |^
> main.c:1:13: note: previous declaration of ‘T’ was here
> 1 | typedef int T;
>   |
> 
>
> Here the problem is that we define T to be two different types, the first
> it's an int and the second is a pointer to int.
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/CAA40n-VotEGgVupaJ6S1zRF%3D3hDHDKAz0V5_enaD9EfYfL7t-Q%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAEkBMfFDr%2BJ5Z79%2BTY_-Yjj5cNniPWRH7ccyK0vSyaHsBcayKA%40mail.gmail.com.

Re: [go-nuts] Pointer to a pointer

[Jan Mercl]

On Thu, Mar 10, 2022 at 1:40 PM 'wagner riffel' via golang-nuts <
golang-nuts@googlegroups.com> wrote:

> I don't think it's mentioned in the specification, my bet is that
> unless your type requires inifnity amout of memory (eg: `type t struct
> {t}`) or the type is an interface and break its rules, (eg: `type
> iface interface{ iface }`) you can use self-reference.

The validity of `type T *T` in Go is based on two things: 1) The visibility
of the identifier in `type T ...` is specified to start right after the
identifier, 2) It's possible, in this case, to compute the size of type T.
So no problem here.

> You're correct, C doesn't allow self reference in type decl, more ...

In `typedef self *self;` the visibility of the second instance of
identifier `self` starts only after it, ie. preceding the final `;'. So the
first identifier `self`, the one after `typedef` is undefined and that's
the real reason it does not work. However:


jnml@e5-1650:~/tmp$ cat main.c
typedef int T;
typedef T T;

int main() {}
jnml@e5-1650:~/tmp$ gcc -Wall main.c
jnml@e5-1650:~/tmp$


This works because typedef is equal to Go type aliases. But the full Go
thing cannot work:


jnml@e5-1650:~/tmp$ cat main.c
typedef int T;
typedef T *T;

int main() {}
jnml@e5-1650:~/tmp$ gcc -Wall main.c
main.c:2:12: error: conflicting types for ‘T’
2 | typedef T *T;
  |^
main.c:1:13: note: previous declaration of ‘T’ was here
1 | typedef int T;
  |


Here the problem is that we define T to be two different types, the first
it's an int and the second is a pointer to int.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAA40n-VotEGgVupaJ6S1zRF%3D3hDHDKAz0V5_enaD9EfYfL7t-Q%40mail.gmail.com.

[go-nuts] Strange symbol size in symtab

[Nikolay Dubina]

Hello,

I am getting strange byte sizes for Undefined symbols in symtabs.

How to reproduce:
1. build and go binary (e.g. https://github.com/gohugoio/hugo)
2. make symbtab file (go tool nm -size )

In file you will see entries like this
```
 4294971392 U _symlink
 4294971392 U _sysconf
 4294971392 U _sysctl
 4294971392 U _sysctlbyname
 4294971392 U _unlink
 4294971392 U _unlinkat
 4294971392 U _unsetenv
 4294971392 U _usleep
 4294971392 U _utimes
 4294971392 U _wait4
 4294971392 U _write
```

What is strange here:
- 4294971392 is number of bytes per documentation: https://pkg.go.dev/cmd/nm
- this is around 4GB
- binary of hugo is much smaller 62MB
- the size of symbol has to be different, but it is same for all these 
undefined symbols.

Would be nice if anyone seen this problem or tells me what is happening 
here.

I am also planning to dig dipper into go core, but would like to check with 
you guys first.

Thank you and hope everyone is doing well,

-- Nikolay

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/fa9aa8e9-bc23-416c-8c3e-9604b31f25cfn%40googlegroups.com.

Re: [go-nuts] Pointer to a pointer

['wagner riffel' via golang-nuts]

On Thu Mar 10, 2022 at 12:41 PM CET, Manlio Perillo wrote:
> Interesting example, thanks.
>
> But how does `type self *self` works?  If I remember correctly, it is not
> discussed in the Language Specification and in The Go Programming Language
> book.
>

I don't think it's mentioned in the specification, my bet is that
unless your type requires inifnity amout of memory (eg: `type t struct
{t}`) or the type is an interface and break its rules, (eg: `type
iface interface{ iface }`) you can use self-reference.

> By the way, in C `typedef self *self` is invalid (assuming my code is
> written correctly).

You're correct, C doesn't allow self reference in type decl, more
interesting types beside pointers:
  `type fn func() fn`
  `type s []s`
  `type ch chan ch`
  `type m map[*m]m`

-w

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CIG6Z0BDF5P1.1O1MIIQNEHEZY%40pampas.

Re: [go-nuts] Pointer to a pointer

[Manlio Perillo]

Interesting example, thanks.

But how does `type self *self` works?  If I remember correctly, it is not 
discussed in the Language Specification and in The Go Programming Language 
book.

By the way, in C `typedef self *self` is invalid (assuming my code is 
written correctly).

Manlio

On Thursday, March 10, 2022 at 6:44:52 AM UTC+1 Rob 'Commander' Pike wrote:

> Here's a program with 1000 *s. You can see the pattern, make it any
> number you like.
>
> https://go.dev/play/p/FZXWcQTutEG
>
> // You can edit this code!
> // Click here and start typing.
> package main
>
> import "fmt"
>
> type self *self
>
> func main() {
> var p self
> p = 
>
> fmt.Println(p)
> }
>
>
> -rob
>
> On Thu, Mar 10, 2022 at 4:29 PM Kurtis Rader  wrote:
> >
> > On Wed, Mar 9, 2022 at 9:12 PM shan...@gmail.com  
> wrote:
> >>
> >> Um
> >>
> >> Really?
> >>
> >> Which of these things are you specifically trying to prevent happening
> >> - Curiousity
> >> - Creativity
> >> - Asking questions
> >> - Some combination of the above
> >>
> >> I mean, I appreciate that you think that people should *know* whatever 
> it is you think you know, but that's a really *really* poor response
> >
> >
> > Yes, your question was silly. The limit is going to be both platform 
> dependent and dependent on the resources (e.g., memory) available on the 
> platform. Your question is silly because regardless of the fundamental 
> limits imposed by the Go language or the platform it runs on absolutely no 
> one will ever write a function that gets within many orders of magnitude of 
> the limit. So your question is interesting in a hypothetical sense but not 
> in a practical sense. For the former I suggest you start a research project 
> and write a paper for review that explains why, or why not, the existing 
> limit is a problem.
> >
> >>
> >> On Thursday, March 10, 2022 at 4:08:02 PM UTC+11 Kurtis Rader wrote:
> >>>
> >>> On Wed, Mar 9, 2022 at 8:38 PM Jan Mercl <0xj...@gmail.com> wrote:
> 
>  A linked list, for example, consists of pointers to pointers to 
> pointers...
> 
>  Why should any limit exist to the length of the list except resources 
> available?
> >>>
> >>>
> >>> Yes, but the O.P. was asking about a silly example. Specifically, when 
> defining a function that receives pointers how many levels of indirection 
> are allowed in the declaration. In practice 99.9% of the time a single 
> level of indirection is specified and 0.09% of the time two levels are 
> specified. Etcetera. For example, if
> >>>
> >>> func wtf(i int) {
> >>> }
> >>>
> >>> is supported, which has eight levels of indirection, why isn't 16? 32? 
> 64? Etcetera levels of indirection supported when defining a function. It's 
> a silly question that shows the O.P. doesn't understand how compilers work. 
> Let alone how people use languages like Go in real life.
> >>>
> 
>  On Thu, Mar 10, 2022, 03:59 shan...@gmail.com  
> wrote:
> >
> > This morning someone asked about dereferincing a pointer to a 
> pointer to a pointer
> >
> > At first nobody had ever thought about, let alone knew the answer, 
> but some example code was shown, and sure enough ***val is possible
> > ```
> > package main
> >
> > import "fmt"
> >
> > func main() {
> > a := 0
> > b := 
> > c := 
> > UltimatePointOne()
> > fmt.Println(a)
> > }
> >
> > func UltimatePointOne(n ***int) {
> > ***n = 1
> > }
> > ```
> >
> >
> > On a lark a go playground example was tried to find what the maximum 
> * is in Go
> >
> > https://go.dev/play/p/YhibY3p7TSD
> >
> > There's 28 there, but it's not the limit
> >
> > Does anyone know what the upper bound on this could be?
> >
> > 256 * ?
> >
> > 32k * ?
> >
> > --
> > You received this message because you are subscribed to the Google 
> Groups "golang-nuts" group.
> > To unsubscribe from this group and stop 

Re: [go-nuts] Pointer to a pointer

[shan...@gmail.com]

Apologies; I had misunderstood your code and was thinking that - because of 
earlier responses - this wasn't going to be a nice day.

> for Shane, Rob did answer it and in way that is really quite deep, and 
thinking about how he answered it will teach you something that is worth 
learning)

Yeah it's definitely above my paygrade and will take some time for me to 
get my head around
On Thursday, March 10, 2022 at 8:31:58 PM UTC+11 Rob 'Commander' Pike wrote:

> On Thu, Mar 10, 2022 at 5:08 PM shan...@gmail.com  
> wrote:
> >
> > Is this really how you want to be known?
>
> Sure, why not? It's a more interesting program than one might think.
>
> For a richer example of the foundational idea here, see the peano.go
> program in the test directory in the repo.
>
> -rob
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/fd44a6d2-4764-4d90-9637-2808683fcedfn%40googlegroups.com.

Re: [go-nuts] Pointer to a pointer

[Rob Pike]

On Thu, Mar 10, 2022 at 5:08 PM shan...@gmail.com  wrote:
>
> Is this really how you want to be known?

Sure, why not? It's a more interesting program than one might think.

For a richer example of the foundational idea here, see the peano.go
program in the test directory in the repo.

-rob

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAOXNBZQpnc%3DCQDYM9Vgk-bPU%2BaevXG4SA15OEGmMtC1b1tXDLg%40mail.gmail.com.

Re: [go-nuts] Possible float64 precision problem

[christoph...@gmail.com]

Thank you for offering your help. 

I can now definitely confirm that there is no computation difference 
between Go and C in my program. There is thus no red flag to use Go in our 
scientific application. What a relief. 

The problem is that our C and Go programs don't store the values in the 
same location. I thus compared different values, but they were close enough 
to be confusing. 

It is impressive that the C and Go computation yields the exact same values 
and output when using %g. I could locate my values with a simple text 
search and diagnose their misplacement. I still need to determine which of 
the C or Go program is doing it wrong. This code translation process may 
have uncover a bug in the C program. 
Le jeudi 10 mars 2022 à 00:31:25 UTC+1, pdc...@gmail.com a écrit :

> Hi Christopher, what input int64 is leading to this result (difference in 
> behavior between Go & C). Does it happen with any input?
> I'm asking because I'm interested in playing around with this a bit, maybe 
> writing two trivial programs (one on each language) and comparing the 
> machine code generated from each one. I'm trying to venture into low level 
> debugging/reverse engineering and I think it will be a good exercise.
>
> Thank you!
>
> Best
>
> Pablo
>
> On Wed, Mar 9, 2022, 4:39 PM 'Dan Kortschak' via golang-nuts <
> golan...@googlegroups.com> wrote:
>
>> On Wed, 2022-03-09 at 03:37 -0800, christoph...@gmail.com wrote:
>> > I'm translating a scientific C program into Go that is doing some
>> > 64bit floating point operations.
>> >
>> > In this process I check that the same input yields the same output.
>> > Unfortunately they don't yield the same result, though the
>> > computation is simple. It is as follow. I receive a 64bit integer
>> > value.
>> >
>> > This value is converted to float64/double, and divided by 2^64.
>> > The resulting number is multiplied by 1e8.
>> >
>> > With C I get 41 6E 84 FD 00 09 90 D7, with Go I get 41 6E 84 FD 00 09
>> > E6 8E. The last 15bits are different. The computation is performed
>> > with the same computer.
>> >
>> > Could it be that the C program is performing the computation with
>> > long double (80 bit) precision and that Go is doing it with 64bit
>> > precision ?
>> >
>> > Is there something I could do about it because that might be a red
>> > flag for replacing the C program with a Go program.
>>
>> This is not very surprising depending on the algorithms that are being
>> used/the problem that is being solved. Some problems are fundamentally
>> difficult to solve exactly and the nature of floating point makes them
>> sensitive to the precise set of operations used, intermediate rounding
>> and the order of operations (even for operations that are commutative
>> in theory). As Robert said, knowing the C compiler will be important,
>> and I'd go further, knowing which platform you are building the Go
>> program on can be important due to differences in how floating point
>> operations are rendered into machine code by the compiler, or even how
>> the processor orders apparently commutative operations.
>>
>> Assuming the values that you've pasted above are big endian, then the
>> Go value is within a 1e12th of the value calculate by C (
>> https://go.dev/play/p/dn7G2LI75RC). This is not terrible, and maybe
>> that level of precision is all that can be promised by the algorithm
>> (and believing digits smaller that 1e-12 is dangerous). Alternatively
>> there is no fundamental limit at this point and there is a better more
>> stable algorithm that you can use (though you are only four orders of
>> magnitude from the machine epsilon
>> https://en.wikipedia.org/wiki/Machine_epsilon, so be aware).
>>
>> Floats are tricky beasts and can easily trip people up. I would suggest
>> that you read
>> https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html (and
>> the more friendly https://floating-point-gui.de/).
>>
>>
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts...@googlegroups.com.
>>
> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/ee58b6e5ad163014b256d249e21c875307fecddb.camel%40kortschak.io
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/027222f6-8f42-4db8-ab5c-b34020cd42a6n%40googlegroups.com.

[go-nuts] Re: Pod memory keeps on increasing and restart with error OOMKilled

[Tamás Gulácsi]

gopkg.in/confluentinc/confluent-kafka-go.v1/kafka._Cfunc_GoBytes

says it uses cgo, hiding it's memory usage from Go. I bet that 900MiB of 
memory is there...


Rakesh K R a következőt írta (2022. március 10., csütörtök, 7:26:57 UTC+1):

> HI,
> I have a micro service application deployed on kubernetes cluster(with 1gb 
> of pod memory limit). This app receive continuous messages(500 message per 
> second) from producer app over kafka interface(these messages are encoded 
> in protobuf format.)
>
> *Basic application flow:*
> 1. Get the message one by one from kafka
> 2. unmarshal proto message
> 3. apply business logic
> 4. write the message to redis cache(in []byte format)
>
> When pod starts memory will be around 50mb and memory starts increasing as 
> traffic flows into the application. It is never released back to OS. As a 
> result pod restarts with error code *OOMKilled*.
> I have integrated grafana to see memory usage like RSS, heap, stack.
> During this traffic flow, in-use heap size is 80mb, idle heap is 80mb 
> where as process resident memory is at 800-1000MB. Stopping the traffic 
> completely for hours did not help and RSS continue to remain in 1000mb.
> Tried to analyze this with pprof and it reports only 80mb are in in-use 
> section. So I am wondering where these remaining 800-1000mb of pods memory 
> went. Also application allocates memory like slices/maps/strings to perform 
> business logic(see alloc_space pprof output below)
>
> I tried couple of experiments:
> 1. Calling FreeOsMemory() in the app but that did not help
> 2. invoking my app with GODEBUG=madvdontneed=1 my_app_executable and did 
> not help
> 3. Leaving the application for 5-6hrs without any traffic to see whether 
> memory comes down. It did not help
> 4. pprof shows only 80mb of heap in use
> 5. Tried upgrading golang version from 1.13 to 1.16 as there were some 
> improvements in runtime. It did not help
>
> pprof output for *alloc_space*:
>
> (pprof) top20
> Showing nodes accounting for 481.98GB, 91.57% of 526.37GB total
> Dropped 566 nodes (cum <= 2.63GB)
> Showing top 20 nodes out of 114
>   flat  flat%   sum%cum   cum%
>78.89GB 14.99% 14.99%78.89GB 14.99%  
> github.com/go-redis/redis/v7/internal/proto.(*Reader).readStringReply
>67.01GB 12.73% 27.72%   285.33GB 54.21% 
>  airgroup/internal/wrapper/agrediswrapper.GetAllConfigurationForGroups
>58.75GB 11.16% 38.88%58.75GB 11.16%  
> google.golang.org/protobuf/internal/impl.(*MessageInfo).MessageOf
>52.26GB  9.93% 48.81%52.26GB  9.93%  reflect.unsafe_NewArray
>45.78GB  8.70% 57.50%46.38GB  8.81% 
>  encoding/json.(*decodeState).literalStore
>36.98GB  7.02% 64.53%36.98GB  7.02%  reflect.New
>28.20GB  5.36% 69.89%28.20GB  5.36%  
> gopkg.in/confluentinc/confluent-kafka-go.v1/kafka._Cfunc_GoBytes
>25.60GB  4.86% 74.75%63.62GB 12.09%  
> google.golang.org/protobuf/proto.MarshalOptions.marshal
>12.79GB  2.43% 77.18%   165.56GB 31.45% 
>  encoding/json.(*decodeState).object
>12.73GB  2.42% 79.60%12.73GB  2.42%  reflect.mapassign
>11.05GB  2.10% 81.70%63.31GB 12.03%  reflect.MakeSlice
>10.06GB  1.91% 83.61%12.36GB  2.35% 
>  filterServersForDestinationDevicesAndSendToDistributionChan
> 6.92GB  1.32% 84.92%   309.45GB 58.79% 
>  groupAndSendToConfigPolicyChannel
> 6.79GB  1.29% 86.21%48.85GB  9.28% 
>  publishInternalMsgToDistributionService
> 6.79GB  1.29% 87.50%   174.81GB 33.21%  encoding/json.Unmarshal
> 6.14GB  1.17% 88.67% 6.14GB  1.17%  
> google.golang.org/protobuf/internal/impl.consumeBytes
> 4.64GB  0.88% 89.55%14.39GB  2.73% 
>  GetAllDevDataFromGlobalDevDataDb
> 4.11GB  0.78% 90.33%18.47GB  3.51%  GetAllServersFromServerRecordDb
> 3.27GB  0.62% 90.95% 3.27GB  0.62%  net.HardwareAddr.String
> 3.23GB  0.61% 91.57% 3.23GB  0.61%  reflect.makemap
> (pprof)
>
>
> Need experts help in analyzing this issue.
>
> Thanks in advance!!
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/c578632b-008b-4b42-883c-615bd2f9244dn%40googlegroups.com.

[go-nuts] Re: Download a zip file from a URL

[Tamás Gulácsi]

You miss the error returned by out.Close(). So, finish DownloadFile with 
"return out.Close()"
And os.Create will truncate the file, no need to Remove it - though if you 
want to remove, just "_ = os.Remove(filepath) "- don't check the error.


Gaurav a következőt írta (2022. március 10., csütörtök, 6:54:30 UTC+1):

> I am trying to download a zip file using a simple http Get. This program 
> downloads the file but the downloaded file is corrupted.
>
> package main
> import (
> "fmt"
> "io"
> "net/http"
> "os"
> )
>
> func DownloadFile(url string) error {
> resp, err := http.Get(url)
> if err != nil {
> return err
> }
> defer resp.Body.Close()
>
> filepath := "test.zip"
>
> _, err = os.Stat(filepath)
> if err == nil {
> err := os.Remove(filepath)
> if err != nil {
> return err
> }
> }
>
> out, err := os.Create(filepath)
> if err != nil {
> return err
> }
> defer out.Close()
>
> _, err = io.Copy(out, resp.Body)
> if err != nil {
> return err
> }
>
> return nil
> }
>
> func main() {
> err := DownloadFile("
> https://store1.gofile.io/download/78f1898f-2b02-465a-a39c-4cf7e35822a2/test.zip
> ")
> if err != nil {
> fmt.Println(err)
> }
> }
>
> What am I missing here? Regards
> Gaurav
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/c3d2ea0a-5fb3-4791-8715-07978a3dd27fn%40googlegroups.com.