Re: [go-nuts] Concurrent io.Copy(tcpConn, file) and tcpConn.Write(...)

[Liam]

On Monday, April 27, 2020 at 10:00:52 PM UTC-7, Ian Lance Taylor wrote:
>
> On Mon, Apr 27, 2020 at 6:59 PM Liam > 
> wrote: 
> > 
> > On Monday, April 27, 2020 at 5:56:52 PM UTC-7, Ian Lance Taylor wrote: 
> >> 
> >> On Mon, Apr 27, 2020 at 5:10 PM Liam  wrote: 
> >> > 
> >> > On Monday, April 27, 2020 at 4:22:41 PM UTC-7, Ian Lance Taylor 
> wrote: 
> >> >> 
> >> >> On Sun, Apr 26, 2020 at 4:55 PM Liam  wrote: 
> >> >> > 
> >> >> > During an io.Copy() where the Writer is a TCPConn and the Reader 
> is a 200K disk file, my code may concurrently Write() on the same TCPConn. 
> >> >> > 
> >> >> > I see the result of the Write() inserted into the result of the 
> io.Copy(). I had the impression that was impossible, but I must be 
> mistaken, as the sendfile(2) docs read: 
> >> >> > 
> >> >> > Note that a successful call to sendfile() may write fewer bytes 
> than requested; the caller should be prepared to retry the call if there 
> were unsent bytes. 
> >> >> > 
> >> >> > Could someone confirm that one must indeed synchronize concurrent 
> use of tcpConn.Write() and io.Copy(tcpConn, file)? 
> >> >> 
> >> >> Synchronization should not be required.  internal/poll.Sendfile 
> >> >> acquires a write lock on dstFD, which is the TCP socket.  That 
> should 
> >> >> ensure that the contents of an ordinary Write (which also acquires a 
> >> >> write lock) should not interleave with the sendfile data. 
> >> >> 
> >> >> That said, if the sendfile system call cannot be used for whatever 
> >> >> reason, the net package will fall back on doing ordinary Read and 
> >> >> Write calls.  And those Write calls can be interleaved with other 
> >> >> Write calls done by a different goroutine.  I think that is probably 
> >> >> permitted, in that io.Copy doesn't promise to not interleave with 
> >> >> simultaneous Write calls on the destination. 
> >> >> 
> >> >> So in the general case you should indeed use your own locking to 
> avoid 
> >> >> interleaving between io.Copy and a concurrent Write. 
> >> > 
> >> > 
> >> > Thanks for the details. Where could I add a Println() to reveal why 
> it doesn't call poll.Sendfile()? 
> >> > 
> >> > I expect this system to use sendfile(2). The file is a normal file on 
> a local partition (running on a Digital Ocean Droplet). 
> >> > 
> >> > 
> >> > /etc/fstab has: 
> >> > UUID=[omitted] /   ext4defaults1 1 
> >> > 
> >> > 
> >> > $ df -h 
> >> > Filesystem  Size  Used Avail Use% Mounted on 
> >> > devtmpfs981M 0  981M   0% /dev 
> >> > tmpfs   996M 0  996M   0% /dev/shm 
> >> > tmpfs   996M  436K  995M   1% /run 
> >> > tmpfs   996M 0  996M   0% /sys/fs/cgroup 
> >> > /dev/vda159G  5.7G   51G  11% / 
> >> > tmpfs   200M 0  200M   0% /run/user/0 
>
 
Well this is a surprise... Added some println()s

// my network setup
   aCfgTcp := net.ListenConfig{KeepAlive: -1}
   aListener, err := aCfgTcp.Listen(nil, iConf.Listen.Net, 
iConf.Listen.Laddr)
   if err != nil { return err }
   aCert, err := tls.LoadX509KeyPair(iConf.Listen.CertPath, 
iConf.Listen.KeyPath)
   if err != nil { return err }
   aCfgTls := tls.Config{Certificates: []tls.Certificate{aCert}}
   aListener = tls.NewListener(aListener, &aCfgTls)

// conn writer goroutine, before io.Copy
   println(".. io.Copy to net") 
// conn reader goroutine, before io.CopyN
   println(".. io.CopyN to file") 

// package io
func copyBuffer(dst Writer, src Reader, buf []byte) (written int64, err 
error) {
// If the reader has a WriteTo method, use it to do the copy.
// Avoids an allocation and a copy.
if wt, ok := src.(WriterTo); ok {
println(".. WriteTo")
return wt.WriteTo(dst)
}
// Similarly, if the writer has a ReadFrom method, use it to do the 
copy.
if rt, ok := dst.(ReaderFrom); ok {
println(".. ReadFrom")
return rt.ReadFrom(src)
}
println(".. manual")
[etc]

// package net
func (c *TCPConn) readFrom(r io.Reader) (int64, error) {
if n, err, handled := splice(c.fd, r); handled {
println(".. spliced")
return n, err
}
if n, err, handled := sendFile(c.fd, r); handled {
println(".. sendFiled")
return n, err
}
println(".. generic")
return genericReadFrom(c, r)

LOG:
.. io.CopyN to file
.. manual
.. io.Copy to net
.. manual
.. io.CopyN to file
.. manual
.. io.Copy to net
.. manual
.. io.CopyN to file
.. manual
.. io.Copy to net
.. manual
.. io.Copy to net
.. manual
.. io.Copy to net
.. manual
.. io.Copy to net
.. manual

Not a sendfile(2) or splice(2) in sight :-(

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/45642e1e-a9d6-4cd2-

Re: [go-nuts] Concurrent io.Copy(tcpConn, file) and tcpConn.Write(...)

[Liam]

On Tuesday, April 28, 2020 at 12:05:00 AM UTC-7, Liam wrote:
>
>
>
> On Monday, April 27, 2020 at 10:00:52 PM UTC-7, Ian Lance Taylor wrote:
>>
>> On Mon, Apr 27, 2020 at 6:59 PM Liam  wrote: 
>> > 
>> > On Monday, April 27, 2020 at 5:56:52 PM UTC-7, Ian Lance Taylor wrote: 
>> >> 
>> >> On Mon, Apr 27, 2020 at 5:10 PM Liam  wrote: 
>> >> > 
>> >> > On Monday, April 27, 2020 at 4:22:41 PM UTC-7, Ian Lance Taylor 
>> wrote: 
>> >> >> 
>> >> >> On Sun, Apr 26, 2020 at 4:55 PM Liam  wrote: 
>> >> >> > 
>> >> >> > During an io.Copy() where the Writer is a TCPConn and the Reader 
>> is a 200K disk file, my code may concurrently Write() on the same TCPConn. 
>> >> >> > 
>> >> >> > I see the result of the Write() inserted into the result of the 
>> io.Copy(). I had the impression that was impossible, but I must be 
>> mistaken, as the sendfile(2) docs read: 
>> >> >> > 
>> >> >> > Note that a successful call to sendfile() may write fewer bytes 
>> than requested; the caller should be prepared to retry the call if there 
>> were unsent bytes. 
>> >> >> > 
>> >> >> > Could someone confirm that one must indeed synchronize concurrent 
>> use of tcpConn.Write() and io.Copy(tcpConn, file)? 
>> >> >> 
>> >> >> Synchronization should not be required.  internal/poll.Sendfile 
>> >> >> acquires a write lock on dstFD, which is the TCP socket.  That 
>> should 
>> >> >> ensure that the contents of an ordinary Write (which also acquires 
>> a 
>> >> >> write lock) should not interleave with the sendfile data. 
>> >> >> 
>> >> >> That said, if the sendfile system call cannot be used for whatever 
>> >> >> reason, the net package will fall back on doing ordinary Read and 
>> >> >> Write calls.  And those Write calls can be interleaved with other 
>> >> >> Write calls done by a different goroutine.  I think that is 
>> probably 
>> >> >> permitted, in that io.Copy doesn't promise to not interleave with 
>> >> >> simultaneous Write calls on the destination. 
>> >> >> 
>> >> >> So in the general case you should indeed use your own locking to 
>> avoid 
>> >> >> interleaving between io.Copy and a concurrent Write. 
>> >> > 
>> >> > 
>> >> > Thanks for the details. Where could I add a Println() to reveal why 
>> it doesn't call poll.Sendfile()? 
>> >> > 
>> >> > I expect this system to use sendfile(2). The file is a normal file 
>> on a local partition (running on a Digital Ocean Droplet). 
>> >> > 
>> >> > 
>> >> > /etc/fstab has: 
>> >> > UUID=[omitted] /   ext4defaults1 1 
>> >> > 
>> >> > 
>> >> > $ df -h 
>> >> > Filesystem  Size  Used Avail Use% Mounted on 
>> >> > devtmpfs981M 0  981M   0% /dev 
>> >> > tmpfs   996M 0  996M   0% /dev/shm 
>> >> > tmpfs   996M  436K  995M   1% /run 
>> >> > tmpfs   996M 0  996M   0% /sys/fs/cgroup 
>> >> > /dev/vda159G  5.7G   51G  11% / 
>> >> > tmpfs   200M 0  200M   0% /run/user/0 
>>
>  
> Well this is a surprise... Added some println()s
>
> // my network setup
>aCfgTcp := net.ListenConfig{KeepAlive: -1}
>aListener, err := aCfgTcp.Listen(nil, iConf.Listen.Net, 
> iConf.Listen.Laddr)
>if err != nil { return err }
>aCert, err := tls.LoadX509KeyPair(iConf.Listen.CertPath, 
> iConf.Listen.KeyPath)
>if err != nil { return err }
>aCfgTls := tls.Config{Certificates: []tls.Certificate{aCert}}
>aListener = tls.NewListener(aListener, &aCfgTls)
>

I left out my accept loop:
   var aConn net.Conn
   for {
  aConn, err = aListener.Accept()
  ...

Is this the root cause, as net.Listener only provides Accept() (Conn, 
error), and net.Conn doesn't provide ReadFrom()?

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/2e4aba99-fa0a-4898-94f2-5af8c57e39af%40googlegroups.com.

[go-nuts] marshal multiple json documents which may have different format versions into the latest struct version

[Chris Burkert]

Dear all,

my application users shall be able to provide multiple json documents
(files and urls) which I'd like to marshall into one structure.
Additionally these json documents may have different versions. I know how
to marshal a document into a version specific struct if I know the format
version before (for simplicity of this example I don't handle errors):
https://play.golang.org/p/ixVI5CzPqFP

What I would like (in the example the village field was renamed to cities )
is a struct of type ModelV2 with all four values merged in Cities.

Is there a best practice for a backwards compatible behavior which:

   - identifies the json format version of each document
   - skips that document if it is higher than the supported format version
   in my application
   - merges supported format versions into ONE struct

Of course I have to implement the semantics on my own but how can I
approach the topic?

thanks

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CALWqRZog1iVW3ovJKVwqmXEuKFLCN9b5FTT-s36TAcMaHrtiwA%40mail.gmail.com.

[go-nuts] Json Parse Data [unexecpted non whitespace 1 to 28]

[Ali Hassan]

[image: Capture.JPG]

Data : TYPE OF Json



var member Member // struct where json 
data, err :=json.Marshall(member); if err != nil{ fmt.Printf("Error %s",err
)} 
fmt.Printf("Data", data)

err = json.NewDecoder(request.Body).Decode(&member); if err != nil {fmt.
Printf("Error %s",err) // this is print "EOF"}

unexpected non whitespaceing json parse data 1 to  colummn 28 



-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/950a3753-88bf-4748-ab58-e59707914b67%40googlegroups.com.

[go-nuts] Re: func before function

[Volker Dobler]

On Monday, 27 April 2020 18:46:20 UTC+2, valen...@gmail.com wrote:
>
> Why is it necessary to write func in go before declaring a function; if in 
> C, when parsing a function, there is no such need?
> Why "func sum(a, b int) int {...}" can't be "sum(a, b int) int {...}"
>

Of course it could be like that. But why should it be
like that? Is there any benefit? Did you think it through?
How about methods and function literals? Is the grammer
still non-ambiguous?

V.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/c9a21f34-3adf-4313-9f3c-64b49536609f%40googlegroups.com.

[go-nuts] Re: func before function

[Ali Hassan]

may be that's the way compiler understand where new function start 

On Monday, April 27, 2020 at 9:46:20 PM UTC+5, valen...@gmail.com wrote:
>
> Why is it necessary to write func in go before declaring a function; if in 
> C, when parsing a function, there is no such need?
> Why "func sum(a, b int) int {...}" can't be "sum(a, b int) int {...}"
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/d8781ced-a875-428b-8f5c-ddfd4f3a0984%40googlegroups.com.

[go-nuts] Re: Json Parse Data [unexecpted non whitespace 1 to 28]

[Brian Candler]

Can you make this into a standalone reproducing test case on 
play.golang.org, then share the URL with us?

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/d214b174-9950-40d1-86c9-ea260310c1b9%40googlegroups.com.

Re: [go-nuts] Why do we use xchg rather than lock mov to inplement atomic.StoreX?

[Cholerae Hu]

But on gcc 9.3, atomic store with seq_cst order, will be compiled to 
mov+fence rather than xchg, see https://gcc.godbolt.org/z/ucbQt6 . Why do 
we use xchg rather than mov+fence in Go?

在 2020年4月28日星期二 UTC+8上午7:26:15，Ian Lance Taylor写道：
>
> On Sun, Apr 26, 2020 at 1:31 AM Cholerae Hu  > wrote: 
> > 
> > Atomic.StoreX doesn't return the old value of the given pointer, so lock 
> mov would work. Why do we use a xchg instead? It there any performance 
> issue? 
>
> I assume that you are talking about Intel processors.  Intel 
> processors do not have a lock mov instruction. 
>
> From the Intel architecture manual: 
>
> The LOCK prefix can be prepended only to the following 
> instructions and only to those forms 
> of the instructions where the destination operand is a memory 
> operand: ADD, ADC, AND, 
> BTC, BTR, BTS, CMPXCHG, CMPXCH8B, DEC, INC, NEG, NOT, OR, SBB, SUB, 
> XOR, 
> XADD, and XCHG. 
>
> Ian 
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/1103e78c-b7e9-489c-8d22-14569ac12992%40googlegroups.com.

[go-nuts] Re: marshal multiple json documents which may have different format versions into the latest struct version

[Manlio Perillo]

On Tuesday, April 28, 2020 at 10:52:56 AM UTC+2, Chris Burkert wrote:
>
> Dear all,
>
> my application users shall be able to provide multiple json documents 
> (files and urls) which I'd like to marshall into one structure. 
> Additionally these json documents may have different versions. I know how 
> to marshal a document into a version specific struct if I know the format 
> version before (for simplicity of this example I don't handle errors): 
> https://play.golang.org/p/ixVI5CzPqFP
>
> What I would like (in the example the village field was renamed to cities 
> ) is a struct of type ModelV2 with all four values merged in Cities.
>
> Is there a best practice for a backwards compatible behavior which:
>
>- identifies the json format version of each document
>- skips that document if it is higher than the supported format 
>version in my application
>- merges supported format versions into ONE struct
>
> Of course I have to implement the semantics on my own but how can I 
> approach the topic?
>
>
You can first unmarshal a struct containing only the Version field.  As an 
example:
https://play.golang.org/p/1oDzdWlTCfC 



Manlio 

> thanks
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/274e489d-afdb-4ec9-a5b3-26440364c489%40googlegroups.com.

[go-nuts] Re: [ANN] Gio: portable immediate mode GUI programs in Go for iOS/tvOS, Android, macOS, Linux, Windows

[Fino]

hello Elias,

as far as I understand, GIO is a 2D GUI lib, maybe similar with Flutter, 

is it possible to embed one/multi 3D widget inside a view/window? 

any architecture thinking on such requirement?

BR fino

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/8d73d454-ca1f-40f5-bf37-83fa0c79c308%40googlegroups.com.

Re: [go-nuts] Re: [ANN] Gio: portable immediate mode GUI programs in Go for iOS/tvOS, Android, macOS, Linux, Windows

[Elias Naur]

On Tue, Apr 28, 2020 at 3:23 PM Fino  wrote:
>
> hello Elias,
>
> as far as I understand, GIO is a 2D GUI lib, maybe similar with Flutter,
>
> is it possible to embed one/multi 3D widget inside a view/window?
>
> any architecture thinking on such requirement?
>

The other way around is possible: embedding Gio into a program that uses
the GPU. See the "glfw" example:

https://git.sr.ht/~eliasnaur/gio/tree/master/example/glfw/main.go

-- elias

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAMAFT9Ud%3DLy9Gr8ES%3DiWx5w7ap-wEkWWmKQAfomn-RXAsPvy4A%40mail.gmail.com.

[go-nuts] Re: Json Parse Data [unexecpted non whitespace 1 to 28]

[Ali Hassan]

I try

On Tuesday, April 28, 2020 at 1:53:38 PM UTC+5, Ali Hassan wrote:
>
> [image: Capture.JPG]
>
> Data : TYPE OF Json
>
>
>
> var member Member // struct where json 
> data, err :=json.Marshall(member); if err != nil{ fmt.Printf("Error %s",
> err)} 
> fmt.Printf("Data", data)
>
> err = json.NewDecoder(request.Body).Decode(&member); if err != nil {fmt.
> Printf("Error %s",err) // this is print "EOF"}
>
> unexpected non whitespaceing json parse data 1 to  colummn 28 
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/6f4910f9-3005-4128-95e3-69573a6aef67%40googlegroups.com.

[go-nuts] Re: Json Parse Data [unexecpted non whitespace 1 to 28]

[Ali Hassan]

https://play.golang.org/p/cRBdSyGcGfp
Demo Version 

On Tuesday, April 28, 2020 at 1:53:38 PM UTC+5, Ali Hassan wrote:
>
> [image: Capture.JPG]
>
> Data : TYPE OF Json
>
>
>
> var member Member // struct where json 
> data, err :=json.Marshall(member); if err != nil{ fmt.Printf("Error %s",
> err)} 
> fmt.Printf("Data", data)
>
> err = json.NewDecoder(request.Body).Decode(&member); if err != nil {fmt.
> Printf("Error %s",err) // this is print "EOF"}
>
> unexpected non whitespaceing json parse data 1 to  colummn 28 
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/1cedee73-bbe3-471a-a3ea-927735b85cbd%40googlegroups.com.

[go-nuts] json decode is very slow

[Sarath Prabath Redlapalli Jaya]

req := &mystruct{}
json.NewDecoder(r.Body).Decode(req)


We've instrument the above code block and reading request body bytes is 
taking very long i.e., upto 5 secs sometimes reaching upto 20 seconds at 
the throughput of ~4-5K RPM

The Request Body Size Metrics are as follows

Average: 73190 Bytes
90th percentile: 170862 Bytes
99th percentile: 467638 Bytes

Tried ioutil.ReadAll too. But, the read time is same. We're not sure why 
it's so very slow on production with HIGH BANDWIDTH Network line.

Also, there are 5% of errors in this API with following errors while 
decoding json data

   1. unexpected EOF
   2. Error in decoding request: read tcp 127.0.0.1:8080->127.0.0.1:48896: 
   i/o timeout

We're using Kubernetes with Istio 1.0.5 on GCP. These errors are probably 
from Envoy Sidecar, but since the Envoy acts as a reverse proxy, not sure 
why we can get EOF or Timeout Errors since end user connection terminates 
at proxy itself.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/871e3230-a4f3-4469-8aaf-a8e3b6f49e73%40googlegroups.com.

Re: [go-nuts] Re: Json Parse Data [unexecpted non whitespace 1 to 28]

[Matthew Zimmerman]

Ali, that demo doesn't run given there's no http environment in the
playground.  I assume this issue is not HTTP specific.

Something more like this (obviously change your data where it doesn't parse
what you want):
https://play.golang.org/p/RACYspxInyA

Also, please don't store or recreate password schemes.  If you must, at
least use https://godoc.org/golang.org/x/crypto/bcrypt

On Tue, Apr 28, 2020 at 12:34 PM Ali Hassan 
wrote:

> https://play.golang.org/p/cRBdSyGcGfp
> Demo Version
>
> On Tuesday, April 28, 2020 at 1:53:38 PM UTC+5, Ali Hassan wrote:
>>
>> [image: Capture.JPG]
>>
>> Data : TYPE OF Json
>>
>>
>>
>> var member Member // struct where json
>> data, err :=json.Marshall(member); if err != nil{ fmt.Printf("Error %s",
>> err)}
>> fmt.Printf("Data", data)
>>
>> err = json.NewDecoder(request.Body).Decode(&member); if err != nil {fmt.
>> Printf("Error %s",err) // this is print "EOF"}
>>
>> unexpected non whitespaceing json parse data 1 to  colummn 28
>>
>>
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/1cedee73-bbe3-471a-a3ea-927735b85cbd%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAD53Lr4s1HRtFNk%3D6q9KK5br-A%3DqGBre7KMMVCOzZpGXQh2Z3g%40mail.gmail.com.

Re: [go-nuts] Need a way to check only key exist or not? without value?

[adithyasashasai]

is it mentioned anywhere such that "map[string]struct{}" is efficeient?

On Tuesday, April 28, 2020 at 10:23:08 AM UTC+5:30, Randall O'Reilly wrote:
>
> I think map[string]struct{} takes no storage for the value and is the most 
> efficient way to do this. 
>
> - Randy 
>
> > On Apr 27, 2020, at 7:20 PM, Shishir Verma  > wrote: 
> > 
> > I think the idiomatic way to implement a set in golang is to use a map 
> with bool values. Here is an example from effective go documentation: 
> > 
> > 
> > attended := map[string]bool{ 
> > "Ann": true, 
> > "Joe": true, 
> > ... 
> > } 
> > 
> > if attended[person] { // will be false if person is not in the map 
> > fmt.Println(person, "was at the meeting") 
> > } 
> > 
> > 
> > 
> > On Monday, 27 April 2020 22:16:20 UTC+5:30, adithya...@gmail.com wrote: 
> > Basically i need a slice with indexed values, so that i can check only 
> existence. 
> > or a map with only keys? 
> > How it can be done? 
> > 
> > -- 
> > You received this message because you are subscribed to the Google 
> Groups "golang-nuts" group. 
> > To unsubscribe from this group and stop receiving emails from it, send 
> an email to golan...@googlegroups.com . 
> > To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/1201e6f3-621e-4875-9374-d7713fa7d8aa%40googlegroups.com.
>  
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/267feb65-6068-444d-82fb-fa7976bfaace%40googlegroups.com.

Re: [go-nuts] Json Parse Data [unexecpted non whitespace 1 to 28]

[Marcin Romaszewicz]

Ali, your example has several problems.

First, you do this:

var p Person
data, err := json.Marshal(p); if err != nil{
fmt.Println("Error", err)
}

What this does is encode an empty object, that's fine.

Next, you read the HTTP body from the request, and try to unmarshal that.

fmt.Println("Data", data)
err = json.NewDecoder(r.Body).Decode(&p);if err != nil{
fmt.Println("Error2", err)
return
}

We have no idea what you're sending. If you are issuing a GET, there will
be no reasonable body, and the JSON parser will correctly tell you that
there is no data. Did you mean to initialize NewDecoder with "data"? If you
do so, your example works.

In order for your code to work properly, you need to do an HTTP POST to
/data on your handler with a JSON payload conforming to Person's schema.



On Tue, Apr 28, 2020 at 1:54 AM Ali Hassan  wrote:

> [image: Capture.JPG]
>
> Data : TYPE OF Json
>
>
>
> var member Member // struct where json
> data, err :=json.Marshall(member); if err != nil{ fmt.Printf("Error %s",
> err)}
> fmt.Printf("Data", data)
>
> err = json.NewDecoder(request.Body).Decode(&member); if err != nil {fmt.
> Printf("Error %s",err) // this is print "EOF"}
>
> unexpected non whitespaceing json parse data 1 to  colummn 28
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/950a3753-88bf-4748-ab58-e59707914b67%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CA%2Bv29LvhTSvLtH7emnVpPjS%3Ddfzs8o1CiybzU4QgZ2bVYeUg8A%40mail.gmail.com.

Re: [go-nuts] Concurrent io.Copy(tcpConn, file) and tcpConn.Write(...)

[Liam]

On Tuesday, April 28, 2020 at 1:42:15 AM UTC-7, Liam wrote:
>
>
>
> On Tuesday, April 28, 2020 at 12:05:00 AM UTC-7, Liam wrote:
>>
>>
>>
>> On Monday, April 27, 2020 at 10:00:52 PM UTC-7, Ian Lance Taylor wrote:
>>>
>>> On Mon, Apr 27, 2020 at 6:59 PM Liam  wrote: 
>>> > 
>>> > On Monday, April 27, 2020 at 5:56:52 PM UTC-7, Ian Lance Taylor wrote: 
>>> >> 
>>> >> On Mon, Apr 27, 2020 at 5:10 PM Liam  wrote: 
>>> >> > 
>>> >> > On Monday, April 27, 2020 at 4:22:41 PM UTC-7, Ian Lance Taylor 
>>> wrote: 
>>> >> >> 
>>> >> >> On Sun, Apr 26, 2020 at 4:55 PM Liam  wrote: 
>>> >> >> > 
>>> >> >> > During an io.Copy() where the Writer is a TCPConn and the Reader 
>>> is a 200K disk file, my code may concurrently Write() on the same TCPConn. 
>>> >> >> > 
>>> >> >> > I see the result of the Write() inserted into the result of the 
>>> io.Copy(). I had the impression that was impossible, but I must be 
>>> mistaken, as the sendfile(2) docs read: 
>>> >> >> > 
>>> >> >> > Note that a successful call to sendfile() may write fewer bytes 
>>> than requested; the caller should be prepared to retry the call if there 
>>> were unsent bytes. 
>>> >> >> > 
>>> >> >> > Could someone confirm that one must indeed synchronize 
>>> concurrent use of tcpConn.Write() and io.Copy(tcpConn, file)? 
>>> >> >> 
>>> >> >> Synchronization should not be required.  internal/poll.Sendfile 
>>> >> >> acquires a write lock on dstFD, which is the TCP socket.  That 
>>> should 
>>> >> >> ensure that the contents of an ordinary Write (which also acquires 
>>> a 
>>> >> >> write lock) should not interleave with the sendfile data. 
>>> >> >> 
>>> >> >> That said, if the sendfile system call cannot be used for whatever 
>>> >> >> reason, the net package will fall back on doing ordinary Read and 
>>> >> >> Write calls.  And those Write calls can be interleaved with other 
>>> >> >> Write calls done by a different goroutine.  I think that is 
>>> probably 
>>> >> >> permitted, in that io.Copy doesn't promise to not interleave with 
>>> >> >> simultaneous Write calls on the destination. 
>>> >> >> 
>>> >> >> So in the general case you should indeed use your own locking to 
>>> avoid 
>>> >> >> interleaving between io.Copy and a concurrent Write. 
>>> >> > 
>>> >> > 
>>> >> > Thanks for the details. Where could I add a Println() to reveal why 
>>> it doesn't call poll.Sendfile()? 
>>> >> > 
>>> >> > I expect this system to use sendfile(2). The file is a normal file 
>>> on a local partition (running on a Digital Ocean Droplet). 
>>> >> > 
>>> >> > 
>>> >> > /etc/fstab has: 
>>> >> > UUID=[omitted] /   ext4defaults1 1 
>>> >> > 
>>> >> > 
>>> >> > $ df -h 
>>> >> > Filesystem  Size  Used Avail Use% Mounted on 
>>> >> > devtmpfs981M 0  981M   0% /dev 
>>> >> > tmpfs   996M 0  996M   0% /dev/shm 
>>> >> > tmpfs   996M  436K  995M   1% /run 
>>> >> > tmpfs   996M 0  996M   0% /sys/fs/cgroup 
>>> >> > /dev/vda159G  5.7G   51G  11% / 
>>> >> > tmpfs   200M 0  200M   0% /run/user/0 
>>>
>>  
>> Well this is a surprise... Added some println()s
>>
>> // my network setup
>>aCfgTcp := net.ListenConfig{KeepAlive: -1}
>>aListener, err := aCfgTcp.Listen(nil, iConf.Listen.Net, 
>> iConf.Listen.Laddr)
>>if err != nil { return err }
>>aCert, err := tls.LoadX509KeyPair(iConf.Listen.CertPath, 
>> iConf.Listen.KeyPath)
>>if err != nil { return err }
>>aCfgTls := tls.Config{Certificates: []tls.Certificate{aCert}}
>>aListener = tls.NewListener(aListener, &aCfgTls)
>>
>
> I left out my accept loop:
>var aConn net.Conn
>for {
>   aConn, err = aListener.Accept()
>   ...
>

Adding these to the Accept() loop shows that I see a tls.Conn but not a 
net.TCPConn:

if _, ok := aConn.(*net.TCPConn); ok { fmt.Println(".. have tcpconn") }
if _, ok := aConn.(*tls.Conn); ok { fmt.Println(".. have tlsconn") }

Why doesn't tls.Conn either implement ReadFrom() or provide a way to obtain 
the underlying TCPConn?

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/8f00e2ac-1b10-40d8-8510-a6d796f637ad%40googlegroups.com.

Re: [go-nuts] Concurrent io.Copy(tcpConn, file) and tcpConn.Write(...)

[Tamás Gulácsi]

TLS needs encyption, not jost "shoveling the bytes" to the underlying 
connection.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/3721e316-bfa1-433e-a3a6-0ad90339f5a6%40googlegroups.com.

Re: [go-nuts] Concurrent io.Copy(tcpConn, file) and tcpConn.Write(...)

[Robert Engels]

Depends on how the file descriptor is implemented. But the end result probably 
has the same performance unless the network card is doing the TLS - which is 
possible. 

> On Apr 28, 2020, at 2:21 PM, Tamás Gulácsi  wrote:
> 
> 
> TLS needs encyption, not jost "shoveling the bytes" to the underlying 
> connection.
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/3721e316-bfa1-433e-a3a6-0ad90339f5a6%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/A8FC08DC-3E85-45FB-BFBF-BC47AF6E65AF%40ix.netcom.com.

Re: [go-nuts] Deleting map entry from the top level of a nested map doesn't clean the underlying memory

[Naveen Kak]

Basically using the Top command at end of test.
Let me give a quick summary of the scenario.

Basically we have an application which keeps getting data on a TCP
connection, we allocate a global slice to temporarily hold the data and
then store in nested maps.
For every TCP data transaction, we store the data in a  global slice
temporarily and then operate on the maps.
For a certain amount of data, i know what should be the total allocation,
but when i check using top command at end of test its almost always double
or slightly more.

if I call runtime.GC() on every tcp transaction, memory looks good and end
of test it is as expected ( everything reclaimed)
Is it because the system is heavily loaded ( per sec we may be getting huge
no of transactions) so garbage collector cant keep track of the objects to
be freed.
As a test I even tried to block the delete operations on Map, just addition
to maps on a smaller scale though, even then memory is almost double of
what is expected.
only solution is calling runtime.GC() on every tcp transaction, then
everything is fine.
I am really not sure whether the map deletions don't work properly or is it
just the GC which is not able to work efficiently enough on its own.

Calling GC so often is an overhead and would cause CPU spikes.







On Tue, Apr 28, 2020 at 3:21 AM Ian Lance Taylor  wrote:

> On Sun, Apr 26, 2020 at 9:57 PM Naveen Kak  wrote:
>
>> I have my system up and running for let's say 10 hours or so where these
>> operations on map ( continuous add/delete) keep happening. Memory keeps
>> growing and goes very high.
>> Eventually at end of test we clear all map entries, memory is reclaimed
>> and is good.
>> It's basically kind of load test.
>> Appreciate your engagement.
>>
>
> Thanks for the information, but you didn't really answer my question.  How
> exactly are you measuring whether the memory has been garbage collected?
>
> Ian
>
>
>
>> On Mon, 27 Apr, 2020, 3:41 AM Ian Lance Taylor,  wrote:
>>
>>> On Sun, Apr 26, 2020 at 2:48 PM  wrote:
>>>
 https://play.golang.org/p/e22ufH-T2M1

 This is my sample data structure.

 package main

 import (
 "fmt"
 )

 type MicroChkpt struct {
 comprtype uint32
 MicroChkptInfoMap map[uint32][]byte
 }

 type CallChkpt struct {
 FullChkptData []byte
 MicroChkptMap map[uint32]*MicroChkpt
 ckey uint32
 comprtype uint32
 AuditInProgress bool
 }

 var CallChkptMap map[uint32]*CallChkpt

 func main() {
 fmt.Println("Hello, playground")
 }

 So its a nested map structure,
 CallChkptMap->MicroChkptMap->MicroChkptInfoMap
 So i was expecting on deleting an entry from the top level map
 CallChkptMap, whole underlying memory used by nested maps would be
 reclaimed.

 Its not happening till all entries are removed from the top level map(
 then only i see memory dipping), map has ongoing insert and delete
 operations and grows pretty big.
 Any workarounds to reclaim the memory on deleting the specific entry
 please?


 Should i go to the nested maps first, set them to nil and then delete
 the entry from the top level map?
 Appreciate all your time and inputs.

>>>
>>>
>>> How exactly are you measuring whether the memory has been garbage
>>> collected?
>>>
>>> Ian
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAHB4VaAHgKv_FyB7FnHGkLtgpc9K_E9cte6gYBDLThfpVCy8jQ%40mail.gmail.com.

Re: [go-nuts] Concurrent io.Copy(tcpConn, file) and tcpConn.Write(...)

[Liam]

The Linux kernel has TLS; one reason is to allow sendfile(2) with TLS. But 
I guess Go doesn't enable that yet?

https://www.kernel.org/doc/html/latest/networking/tls.html


On Tuesday, April 28, 2020 at 12:39:04 PM UTC-7, Robert Engels wrote:
>
> Depends on how the file descriptor is implemented. But the end result 
> probably has the same performance unless the network card is doing the TLS 
> - which is possible. 
>
> On Apr 28, 2020, at 2:21 PM, Tamás Gulácsi  > wrote:
>
> TLS needs encyption, not jost "shoveling the bytes" to the underlying 
> connection.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/dfda5a47-e174-42d8-8e94-f18f75c3e2ca%40googlegroups.com.

Re: [go-nuts] Concurrent io.Copy(tcpConn, file) and tcpConn.Write(...)

[Brian Candler]

On Tuesday, 28 April 2020 21:09:38 UTC+1, Liam wrote:
>
> The Linux kernel has TLS; one reason is to allow sendfile(2) with TLS. But 
> I guess Go doesn't enable that yet?
>
> https://www.kernel.org/doc/html/latest/networking/tls.html
>
>
https://blog.filippo.io/playing-with-kernel-tls-in-linux-4-13-and-go/

"So yeah, not a huge fan . But of course, it 
wouldn't be me if I didn't fork the Go crypto/tls package to work with it."

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/d4fdf252-8777-44d9-a753-f87195d22204%40googlegroups.com.

Re: [go-nuts] Deleting map entry from the top level of a nested map doesn't clean the underlying memory

['Kevin Chowski' via golang-nuts]

Guessing based on your latest description: are you aware that there is no 
partial slice collection in GC? That is:


 

> bigSlice := make([]int, 1000*1000)

subSlice := bigSlice[0:1:1]

bigSlice = nil

runtime.GC()

// At this point, bigSlice is still allocated! It cannot be freed by the GC 
> (in the current implementation)

useTheSlice(subSlice) 
>

You have to explicitly reallocate smaller slices (and copy over the data 
from the big slice into each smaller slice) in order for the GC to be able 
to collect each smaller slice individually.
 
As suggested earlier in the thread, if you can create and share a small, 
self-contained program which exhibits the behavior you're speaking of, 
someone on this forum is much more likely to be able to help you. Trying to 
guess how a memory leak is manifesting is just... guesswork without looking 
at full code. There are lots of reasons why memory may not be collected by 
the GC, including random bugs in your program :)


On Tuesday, April 28, 2020 at 2:00:55 PM UTC-6, Naveen Kak wrote:
>
> Basically using the Top command at end of test.
> Let me give a quick summary of the scenario.
>
> Basically we have an application which keeps getting data on a TCP 
> connection, we allocate a global slice to temporarily hold the data and 
> then store in nested maps.
> For every TCP data transaction, we store the data in a  global slice 
> temporarily and then operate on the maps.
> For a certain amount of data, i know what should be the total allocation, 
> but when i check using top command at end of test its almost always double 
> or slightly more.
>
> if I call runtime.GC() on every tcp transaction, memory looks good and end 
> of test it is as expected ( everything reclaimed)
> Is it because the system is heavily loaded ( per sec we may be getting 
> huge no of transactions) so garbage collector cant keep track of 
> the objects to be freed.
> As a test I even tried to block the delete operations on Map, just 
> addition to maps on a smaller scale though, even then memory is almost 
> double of what is expected.
> only solution is calling runtime.GC() on every tcp transaction, then 
> everything is fine.
> I am really not sure whether the map deletions don't work properly or is 
> it just the GC which is not able to work efficiently enough on its own.
>
> Calling GC so often is an overhead and would cause CPU spikes.
>
>
>
>
>
>
>
> On Tue, Apr 28, 2020 at 3:21 AM Ian Lance Taylor  > wrote:
>
>> On Sun, Apr 26, 2020 at 9:57 PM Naveen Kak > > wrote:
>>
>>> I have my system up and running for let's say 10 hours or so where these 
>>> operations on map ( continuous add/delete) keep happening. Memory keeps 
>>> growing and goes very high. 
>>> Eventually at end of test we clear all map entries, memory is reclaimed 
>>> and is good. 
>>> It's basically kind of load test. 
>>> Appreciate your engagement. 
>>>
>>
>> Thanks for the information, but you didn't really answer my question.  
>> How exactly are you measuring whether the memory has been garbage collected?
>>
>> Ian
>>
>>  
>>
>>> On Mon, 27 Apr, 2020, 3:41 AM Ian Lance Taylor, >> > wrote:
>>>
 On Sun, Apr 26, 2020 at 2:48 PM > 
 wrote:

> https://play.golang.org/p/e22ufH-T2M1
>
> This is my sample data structure.
>
> package main
>
> import (
> "fmt"
> )
>
> type MicroChkpt struct {
> comprtype uint32
> MicroChkptInfoMap map[uint32][]byte
> }
>
> type CallChkpt struct {
> FullChkptData []byte
> MicroChkptMap map[uint32]*MicroChkpt
> ckey uint32
> comprtype uint32
> AuditInProgress bool
> }
>
> var CallChkptMap map[uint32]*CallChkpt
>
> func main() {
> fmt.Println("Hello, playground")
> }
>
> So its a nested map structure, 
> CallChkptMap->MicroChkptMap->MicroChkptInfoMap
> So i was expecting on deleting an entry from the top level map 
> CallChkptMap, whole underlying memory used by nested maps would be 
> reclaimed.
>
> Its not happening till all entries are removed from the top level map( 
> then only i see memory dipping), map has ongoing insert and delete 
> operations and grows pretty big.
> Any workarounds to reclaim the memory on deleting the specific entry 
> please?
>
>
> Should i go to the nested maps first, set them to nil and then delete 
> the entry from the top level map?
> Appreciate all your time and inputs.
>


 How exactly are you measuring whether the memory has been garbage 
 collected?

 Ian

>>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/f4b8f164-085c-4d3e-8f51-c16138c3ac83%40googlegroups.com.

Re: [go-nuts] Deleting map entry from the top level of a nested map doesn't clean the underlying memory

[Robert Engels]

Also, it may just be that the runtime is better off allocating more and not 
doing a GC based on available memory and CPU usage. The “max heap” feature in 
development may help here. 

> On Apr 28, 2020, at 3:18 PM, 'Kevin Chowski' via golang-nuts 
>  wrote:
> 
> 
> Guessing based on your latest description: are you aware that there is no 
> partial slice collection in GC? That is:
> 
> 
>  
>> bigSlice := make([]int, 1000*1000)
>> subSlice := bigSlice[0:1:1]
>> bigSlice = nil
>> runtime.GC()
>> // At this point, bigSlice is still allocated! It cannot be freed by the GC 
>> (in the current implementation)
>> useTheSlice(subSlice) 
> 
> You have to explicitly reallocate smaller slices (and copy over the data from 
> the big slice into each smaller slice) in order for the GC to be able to 
> collect each smaller slice individually.
>  
> As suggested earlier in the thread, if you can create and share a small, 
> self-contained program which exhibits the behavior you're speaking of, 
> someone on this forum is much more likely to be able to help you. Trying to 
> guess how a memory leak is manifesting is just... guesswork without looking 
> at full code. There are lots of reasons why memory may not be collected by 
> the GC, including random bugs in your program :)
> 
> 
>> On Tuesday, April 28, 2020 at 2:00:55 PM UTC-6, Naveen Kak wrote:
>> Basically using the Top command at end of test.
>> Let me give a quick summary of the scenario.
>> 
>> Basically we have an application which keeps getting data on a TCP 
>> connection, we allocate a global slice to temporarily hold the data and then 
>> store in nested maps.
>> For every TCP data transaction, we store the data in a  global slice 
>> temporarily and then operate on the maps.
>> For a certain amount of data, i know what should be the total allocation, 
>> but when i check using top command at end of test its almost always double 
>> or slightly more.
>> 
>> if I call runtime.GC() on every tcp transaction, memory looks good and end 
>> of test it is as expected ( everything reclaimed)
>> Is it because the system is heavily loaded ( per sec we may be getting huge 
>> no of transactions) so garbage collector cant keep track of the objects to 
>> be freed.
>> As a test I even tried to block the delete operations on Map, just addition 
>> to maps on a smaller scale though, even then memory is almost double of what 
>> is expected.
>> only solution is calling runtime.GC() on every tcp transaction, then 
>> everything is fine.
>> I am really not sure whether the map deletions don't work properly or is it 
>> just the GC which is not able to work efficiently enough on its own.
>> 
>> Calling GC so often is an overhead and would cause CPU spikes.
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>>> On Tue, Apr 28, 2020 at 3:21 AM Ian Lance Taylor  wrote:
 On Sun, Apr 26, 2020 at 9:57 PM Naveen Kak  wrote:
>>> 
 I have my system up and running for let's say 10 hours or so where these 
 operations on map ( continuous add/delete) keep happening. Memory keeps 
 growing and goes very high. 
 Eventually at end of test we clear all map entries, memory is reclaimed 
 and is good. 
 It's basically kind of load test. 
 Appreciate your engagement. 
>>> 
>>> Thanks for the information, but you didn't really answer my question.  How 
>>> exactly are you measuring whether the memory has been garbage collected?
>>> 
>>> Ian
>>> 
>>>  
> On Mon, 27 Apr, 2020, 3:41 AM Ian Lance Taylor,  wrote:
>> On Sun, Apr 26, 2020 at 2:48 PM  wrote:
> 
>> https://play.golang.org/p/e22ufH-T2M1
>> 
>> This is my sample data structure.
>> 
>> package main
>> 
>> import (
>> "fmt"
>> )
>> 
>> type MicroChkpt struct {
>> comprtype uint32
>> MicroChkptInfoMap map[uint32][]byte
>> }
>> 
>> type CallChkpt struct {
>> FullChkptData []byte
>> MicroChkptMap map[uint32]*MicroChkpt
>> ckey uint32
>> comprtype uint32
>> AuditInProgress bool
>> }
>> 
>> var CallChkptMap map[uint32]*CallChkpt
>> 
>> func main() {
>> fmt.Println("Hello, playground")
>> }
>> 
>> So its a nested map structure, 
>> CallChkptMap->MicroChkptMap->MicroChkptInfoMap
>> So i was expecting on deleting an entry from the top level map 
>> CallChkptMap, whole underlying memory used by nested maps would be 
>> reclaimed.
>> 
>> Its not happening till all entries are removed from the top level map( 
>> then only i see memory dipping), map has ongoing insert and delete 
>> operations and grows pretty big.
>> Any workarounds to reclaim the memory on deleting the specific entry 
>> please?
>> 
>> 
>> 
>> Should i go to the nested maps first, set them to nil and then delete 
>> the entry from the top level map?
>> Appreciate all your time and inputs.
>> 
> 
> 
> How exactly are you measuring whether the memory has been garbage 

Re: [go-nuts] Deleting map entry from the top level of a nested map doesn't clean the underlying memory

[Ian Lance Taylor]

On Tue, Apr 28, 2020 at 1:00 PM Naveen Kak  wrote:

> Basically using the Top command at end of test.
>

The top command will show you the memory that the program has requested
from the operating system and has not returned to the operating system.
 The Go memory allocator works by requesting memory from the operating
system as it needs it.  The Go garbage collector works by looking at that
memory and marking it as available for future allocations by the Go memory
allocator.  The Go garbage collector does not immediately return memory to
the operating system.  That is because requesting from and returning to the
operating system are relatively slow operations, and a program that has
needed memory once is likely to need it again.

So the top program is not a good way to judge what the garbage collector is
doing.  It is an OK way to judge the maximum memory use of the program,
which will include memory that has been allocated and memory that has been
garbage collected.

If a Go program runs for a while with excess memory, it will slowly return
it to the operating system.  You can encourage that process by using
runtime/debug.FreeOSMemory.

In general, though, if you want to examine the garbage collector, I
recommend that you use runtime.ReadMemStats rather than top.

Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAOyqgcUmtxu7Qe4njXbtaYD%2BZ4J8Cpqd4R-jwWit1ib8tu7AkA%40mail.gmail.com.

Re: [go-nuts] Why do we use xchg rather than lock mov to inplement atomic.StoreX?

[keith . randall]

It looks like the mechanism used by C's std::atomic would not be useful for 
us.

We require release semantics on atomic stores.  That is, if one thread does:

.. some other writes ...
atomic.StoreInt32(p, 1)

and another thread does

if atomic.LoadInt32(p) == 1 {
   .. some other reads ...
}

If the load sees the store, then the "other reads" must see all of the 
"other writes". For the C atomic you cited, it does:

regular write
mfence

That doesn't provide the guarantee we need. A write before the atomic could 
be reordered with the regular write, causing the reader to not see one of 
the writes it was required to.

For our use case, it would have to be

mfence
regular write

and the semantics of mfence would need to prevent write-write reorderings 
(does it do that? Not sure.)

We'd need some indication that changing it would be faster, as well.

On Tuesday, April 28, 2020 at 4:03:00 AM UTC-7, Cholerae Hu wrote:
>
> But on gcc 9.3, atomic store with seq_cst order, will be compiled to 
> mov+fence rather than xchg, see https://gcc.godbolt.org/z/ucbQt6 . Why do 
> we use xchg rather than mov+fence in Go?
>
> 在 2020年4月28日星期二 UTC+8上午7:26:15，Ian Lance Taylor写道：
>>
>> On Sun, Apr 26, 2020 at 1:31 AM Cholerae Hu  wrote: 
>> > 
>> > Atomic.StoreX doesn't return the old value of the given pointer, so 
>> lock mov would work. Why do we use a xchg instead? It there any performance 
>> issue? 
>>
>> I assume that you are talking about Intel processors.  Intel 
>> processors do not have a lock mov instruction. 
>>
>> From the Intel architecture manual: 
>>
>> The LOCK prefix can be prepended only to the following 
>> instructions and only to those forms 
>> of the instructions where the destination operand is a memory 
>> operand: ADD, ADC, AND, 
>> BTC, BTR, BTS, CMPXCHG, CMPXCH8B, DEC, INC, NEG, NOT, OR, SBB, SUB, 
>> XOR, 
>> XADD, and XCHG. 
>>
>> Ian 
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/72e75767-44fb-44df-a688-b5ea9b2f0a0c%40googlegroups.com.

Re: [go-nuts] Need a way to check only key exist or not? without value?

[Shishir Verma]

I think it is kind of intuitive that empty struct takes 0 bytes whereas a 
bool variable takes 1 byte in memory and hence a map with struct{} values 
would consume lesser memory than the other. I tried checking this using 
code and Randall's point proves:

mapmem.go:

package main

import (
   "fmt"
   _ "net/http/pprof"
   "runtime"
   "unsafe"
)

const (
   entries = 101
)

func main() {
   printAllocs()
   //Empty struct takes 0 bytes in memory whereas a boolean takes 1
   s := struct{}{}
   b := true
   fmt.Printf("size of empty struct: %T, %d\n", s, unsafe.Sizeof(s))
   fmt.Printf("size of a boolean: %T, %d\n", b, unsafe.Sizeof(b))

printAllocs()

//Map with int keys and bool values
   hashset := make(map[int]bool, entries)

for index := 0; index < entries-1; index++ {
   hashset[index] = true
   }

fmt.Printf("Number of elements in map with bool values: %d \n", len(
hashset))
   printAllocs()

//Map with int keys and empty struct values
   hashmap := make(map[int]struct{}, entries)

for index := 0; index < entries-1; index++ {
   hashmap[index] = struct{}{}
   }

fmt.Printf("Number of elements in map with empty struct values: %d \n", 
len(hashmap))
   printAllocs()
}

func printAllocs() {
   var m runtime.MemStats
   runtime.ReadMemStats(&m)
   fmt.Printf("Heap size: %6d \n", m.Alloc/1e6)
}


And here is the output: (Please note that the GC runs and collects memory 
in between most of the times and hence the total heap size shows 22MB 
rather than 47MB)

$ GODEBUG=gctrace=1 ./mapmem
Heap size:  0 
size of empty struct: struct {}, 0
size of a boolean: bool, 1
Heap size:  0 
gc 1 @0.002s 1%: 0.002+0.25+0.019 ms clock, 0.009+0.11/0.045/0.37+0.079 ms 
cpu, 23->23->23 MB, 24 MB goal, 4 P
Number of elements in map with bool values: 100 
Heap size: 24 
gc 2 @0.129s 0%: 0.003+0.25+0.018 ms clock, 0.012+0.076/0.066/0.22+0.072 ms 
cpu, 44->44->21 MB, 47 MB goal, 4 P
Number of elements in map with empty struct values: 100 
Heap size: 22 

As you can see, the map with struct{} values grows the heap size to ~22 MB, 
whereas the map with bool values takes it close to ~24 MB for a million 
entries. Hence, if you are concerned about memory usage, you would rather 
use empty struct values.

I still feel a map with bool values is easier to read though, but that's 
just my opinion. :)


On Wednesday, 29 April 2020 00:18:12 UTC+5:30, adithya...@gmail.com wrote:
>
>
> is it mentioned anywhere such that "map[string]struct{}" is efficeient?
>
> On Tuesday, April 28, 2020 at 10:23:08 AM UTC+5:30, Randall O'Reilly wrote:
>>
>> I think map[string]struct{} takes no storage for the value and is the 
>> most efficient way to do this. 
>>
>> - Randy 
>>
>> > On Apr 27, 2020, at 7:20 PM, Shishir Verma  wrote: 
>> > 
>> > I think the idiomatic way to implement a set in golang is to use a map 
>> with bool values. Here is an example from effective go documentation: 
>> > 
>> > 
>> > attended := map[string]bool{ 
>> > "Ann": true, 
>> > "Joe": true, 
>> > ... 
>> > } 
>> > 
>> > if attended[person] { // will be false if person is not in the map 
>> > fmt.Println(person, "was at the meeting") 
>> > } 
>> > 
>> > 
>> > 
>> > On Monday, 27 April 2020 22:16:20 UTC+5:30, adithya...@gmail.com 
>> wrote: 
>> > Basically i need a slice with indexed values, so that i can check only 
>> existence. 
>> > or a map with only keys? 
>> > How it can be done? 
>> > 
>> > -- 
>> > You received this message because you are subscribed to the Google 
>> Groups "golang-nuts" group. 
>> > To unsubscribe from this group and stop receiving emails from it, send 
>> an email to golan...@googlegroups.com. 
>> > To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/1201e6f3-621e-4875-9374-d7713fa7d8aa%40googlegroups.com.
>>  
>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/afa734a9-edab-4358-aed9-de4a80a1c3a5%40googlegroups.com.

Re: [go-nuts] Re: [ANN] Gio: portable immediate mode GUI programs in Go for iOS/tvOS, Android, macOS, Linux, Windows

[Fino]

OK, more complex than I thought.  need more learning ~ 

BR fino

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/c6340169-18f1-46b2-9c38-f26ff2d1cdca%40googlegroups.com.

[go-nuts] Is this AMQP 0.9.1 library updated?

['Wesley Peng' via golang-nuts]

Hello,
I tried to access RabbitMQ using this AMQP 
library:https://github.com/streadway/amqp

I am not sure if this library get updated following the recent RMQ 
version.Please suggest, thanks.

Wesley Peng
wesleyp...@aol.com

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/1077505155.1304617.1588125421267%40mail.yahoo.com.

Re: [go-nuts] Is this AMQP 0.9.1 library updated?

[Kurtis Rader]

On Tue, Apr 28, 2020 at 7:23 PM 'Wesley Peng' via golang-nuts <
golang-nuts@googlegroups.com> wrote:

> I tried to access RabbitMQ using this AMQP library:
> https://github.com/streadway/amqp
>
> I am not sure if this library get updated following the recent RMQ version.
> Please suggest, thanks.
>

My recommendation is to open a GitHub issue at the link you provided. It
looks like the project is active. Recently opened issues have been closed.

-- 
Kurtis Rader
Caretaker of the exceptional canines Junior and Hank

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CABx2%3DD-nxztC9h1t23yOMDuzdU0PhHdRTbSZJdhfgQ4tb97zYA%40mail.gmail.com.

[go-nuts] http middleware and request context

[Anuj Agrawal]

Hi,

I have a situation where I have a chain of middlewares with a logger
middleware that comes before an authentication middleware. Logger
middleware logs about the incoming request, calls the next middleware
in the chain and then logs about the response. Authentication
middleware authenticates the incoming request and adds the
authenticated user to the request context (which ends up creating a
new request object to be passed downstream) before invoking the next
middleware.

Problem statement is that I want to log the authenticated user id in
the logger middleware. But because the request object known to the
logger middleware is not the same request object that the
authentication middleware set the user context with, I do not have the
user information available in the logger middleware.

I am deliberating on the ways of achieving this. Right now, I am
contemplating to add a pointer to a zero value struct with agreed upon
attributes (or with just a context attribute) or a map to the context
of the request object in logger middleware or an earlier middleware.
Then, in any downstream middleware I can set values of struct
attributes (or manipulate the context attribute) or set key value
pairs to the map. This makes the "context" being set by downstream
middlewares available to upstream middlewares after a downstream
middleware returns.

Are there better ways of doing it? Hopefully, others in the community
may have also faced similar problems and I would like to know about
any alternate ways that others may have used.

Thanks,
Anuj

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CABnk5p0H0YUm2vkL9Nb_CXVpgRy%2BJ87O6NH8FmgR5ewL4%3DSnsg%40mail.gmail.com.

Re: [go-nuts] Why do we use xchg rather than lock mov to inplement atomic.StoreX?

[Cholerae Hu]

On x86-TSO model, it seems that we don't need any mfence to archive 
acquire-release semantics. Acquire-release semantics only need compiler 
barrier to prevent compiler reordering, see https://godbolt.org/z/7JcX-d .

在 2020年4月29日星期三 UTC+8上午7:42:26，keith@gmail.com写道：
>
> It looks like the mechanism used by C's std::atomic would not be useful 
> for us.
>
> We require release semantics on atomic stores.  That is, if one thread 
> does:
>
> .. some other writes ...
> atomic.StoreInt32(p, 1)
>
> and another thread does
>
> if atomic.LoadInt32(p) == 1 {
>.. some other reads ...
> }
>
> If the load sees the store, then the "other reads" must see all of the 
> "other writes". For the C atomic you cited, it does:
>
> regular write
> mfence
>
> That doesn't provide the guarantee we need. A write before the atomic 
> could be reordered with the regular write, causing the reader to not see 
> one of the writes it was required to.
>
> For our use case, it would have to be
>
> mfence
> regular write
>
> and the semantics of mfence would need to prevent write-write reorderings 
> (does it do that? Not sure.)
>
> We'd need some indication that changing it would be faster, as well.
>
> On Tuesday, April 28, 2020 at 4:03:00 AM UTC-7, Cholerae Hu wrote:
>>
>> But on gcc 9.3, atomic store with seq_cst order, will be compiled to 
>> mov+fence rather than xchg, see https://gcc.godbolt.org/z/ucbQt6 . Why 
>> do we use xchg rather than mov+fence in Go?
>>
>> 在 2020年4月28日星期二 UTC+8上午7:26:15，Ian Lance Taylor写道：
>>>
>>> On Sun, Apr 26, 2020 at 1:31 AM Cholerae Hu  wrote: 
>>> > 
>>> > Atomic.StoreX doesn't return the old value of the given pointer, so 
>>> lock mov would work. Why do we use a xchg instead? It there any performance 
>>> issue? 
>>>
>>> I assume that you are talking about Intel processors.  Intel 
>>> processors do not have a lock mov instruction. 
>>>
>>> From the Intel architecture manual: 
>>>
>>> The LOCK prefix can be prepended only to the following 
>>> instructions and only to those forms 
>>> of the instructions where the destination operand is a memory 
>>> operand: ADD, ADC, AND, 
>>> BTC, BTR, BTS, CMPXCHG, CMPXCH8B, DEC, INC, NEG, NOT, OR, SBB, SUB, 
>>> XOR, 
>>> XADD, and XCHG. 
>>>
>>> Ian 
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/80d2c494-809b-47d0-bb9b-549b32068c1c%40googlegroups.com.

[go-nuts] External/internal linker and cgo packages

[Vincent Blanchon]

Hello,

I'm building a simple program that has a dependency to 
github.com/DataDog/zstd, a wrapper of a C code.
So by default, Go will use the external linker. When debugging with, I can 
see

host link: "clang" "-m64" "-Wl,-headerpad,1144" "-Wl,-no_pie" 
"-Wl,-pagezero_size,400" "-o" 
"/var/folders/bb/w1lgkjdx47qd7695hb5r400hgn/T/go-build122622791/b001/exe/a.out"
 "-Qunused-arguments" 
"/var/folders/bb/w1lgkjdx47qd7695hb5r400hgn/T/go-link-023224426/go.o" 
"/var/folders/bb/w1lgkjdx47qd7695hb5r400hgn/T/go-link-023224426/00.o" 
[...] 
"/var/folders/bb/w1lgkjdx47qd7695hb5r400hgn/T/go-link-023224426/53.o" 
"/var/folders/bb/w1lgkjdx47qd7695hb5r400hgn/T/go-link-023224426/54.o" 
"/var/folders/bb/w1lgkjdx47qd7695hb5r400hgn/T/go-link-023224426/55.o" 
"-g" "-O2" "-g" "-O2" "-g" "-O2" "-lpthread" "-g" "-O2" "-framework" 
"CoreFoundation" "-framework" "Security" "-no-pie"

It links 55 files, from 00.o to 55.oIf I had more custom debug, I 
can see than those files are mostly duplication:

04.o is 
/path/to/go-build/67/678a3a394ba4feae32a2ddb829ea65c1cf24be64248dcbcc3dae709b8e3b826d-d
 github.com/DataDog/zstd 
/path/to/go-build/67/678a3a394ba4feae32a2ddb829ea65c1cf24be64248dcbcc3dae709b8e3b826d-d(_x005.o)
05.o is 
/path/to/go-build/67/678a3a394ba4feae32a2ddb829ea65c1cf24be64248dcbcc3dae709b8e3b826d-d
 github.com/DataDog/zstd 
/path/to/go-build/67/678a3a394ba4feae32a2ddb829ea65c1cf24be64248dcbcc3dae709b8e3b826d-d(_x006.o)

I can see 40 times this zstd package among the 55 files. Is it expected? 
What is the difference between them?

Also, what would happen to the non known cgo packages (not net, os/user or 
runtime/cgo) if cgo is not enabled? How the internal linker will manage 
this cgo package?

Thank you

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/db2d65de-8fc5-460e-b309-57f27cdf86cf%40googlegroups.com.

[go-nuts] Re: Json Parse Data [unexecpted non whitespace 1 to 28]

[Ali Hassan]

[Unexcepted NonWhitespacing json data parse after line 1 to 28 ] 

You can checkout my repo
https://github.com/ali2210/HealthyTickets/blob/master/main.go



On Tuesday, April 28, 2020 at 1:53:38 PM UTC+5, Ali Hassan wrote:
>
> [image: Capture.JPG]
>
> Data : TYPE OF Json
>
>
>
> var member Member // struct where json 
> data, err :=json.Marshall(member); if err != nil{ fmt.Printf("Error %s",
> err)} 
> fmt.Printf("Data", data)
>
> err = json.NewDecoder(request.Body).Decode(&member); if err != nil {fmt.
> Printf("Error %s",err) // this is print "EOF"}
>
> unexpected non whitespaceing json parse data 1 to  colummn 28 
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/d25dfa3b-0c4f-4ea3-b2a7-f91b38b5eee9%40googlegroups.com.