Re: [go-nuts] Is there a way to syncronise map reads and writes?

[robert engels]

That is not correct.

See https://go.dev/play/p/aZ7LostRuKR 

You can run this under the race detector without failure.

Still, the race detector cannot detect all races so better to look at the 
source for sync.Map at 105 


You can see that it reads the underlying shared maps while not under 
lock/exclusive access.

Starting a go routine creates a “happens before” relationship which allows this 
to work.

> On Feb 23, 2022, at 12:10 AM, Jason E. Aten  wrote:
> 
> Unfortunately, that's not how Go maps work. You'll still get races unless you 
> synchronize even "read-only" maps.
> 
> On Tuesday, February 22, 2022 at 11:59:37 PM UTC-6 ren...@ix.netcom.com wrote:
> The top level map locks doesn’t matter - it is write once at init and read 
> only after. 
> 
>> On Feb 22, 2022, at 11:51 PM, Jason E. Aten > > wrote:
>> 
>> You have not synchronized access to the top level mapLocks map itself.  
>> Thus you will get races. You need to have a mutex that you lock before you 
>> access mapLocks; not just the maps that are inside it.
> 
>> 
>> I would also recommend being a little object oriented about this design, 
>> putting each map inside its own struct, and make a set of accessor methods 
>> that Lock() and defer Unlock() on a mutex that protects the maps inside.  
>> Read code like this for an idea of what I mean. 
>> https://github.com/glycerine/atomicmap/blob/master/amap.go 
>> 
>> 
>> On Tuesday, February 22, 2022 at 9:55:04 PM UTC-6 ren...@ix.netcom.com 
>>  wrote:
>> Something else is wrong - because marketMaps is read-only after init… so 
>> unless you have some other code - that is not the map causing the panic.
>> 
>> My guess is because you are returning the map from ReadMaps (or RWMaps) that 
>> you are using THAT map by multiple threads outside the lock (the map is a 
>> reference not a copy) and that is causing the panic.
>> 
>> 
>>> On Feb 22, 2022, at 9:39 PM, Zhaoxun Yan > wrote:
>>> 
>> 
>>> Hi guys!
>>> I know this is quite challenging, but it makes sense in io-heavy 
>>> applications.
>>> 
>>> I need to test the responsiveness of different financial future quotation 
>>> sources. They return the quotations spontaneously, and my program respond 
>>> to each one of the CGO call backs from the quotation library and save the 
>>> current prices. Meanwhile, a timer ensures to check the current quotations 
>>> of each available sources. And obviously, it raised up a panic due to 
>>> synchronization of maps on the call of ReadMaps function. Unfortunately 
>>> this came up occasionally and `go build -race` cannot raise the problem.
>>> 
>>> Here is a piece of the code, only the read/write part:
>>> 
>>> --
>>> 
>>> // Please Ignore SyChan to alert the timer that all available sources are 
>>> collected
>>> var SyChan chan int = make(chan int, 3)
>>> 
>>> // Make up a map of 5 mutex locks and a map of 5 data structs, indexed by 
>>> 1-5
>>> var marketMaps = make(map[int]map[string]Market)
>>> var mapLocks = make(map[int]*sync.Mutex)
>>> 
>>> // quotation data
>>> type Market struct {
>>> Price float64 `json:"price"`
>>> Timestamp string  `json:"timestamp"`
>>> }
>>> 
>>> func init() {
>>> for i := 1; i <= 5; i++ {
>>> mapLocks[i] = new(sync.Mutex)
>>> marketMaps[i] = make(map[string]Market)
>>> }
>>> }
>>> 
>>> //Make sure that for each source, R/W has no race, only took place as 
>>> acquiring the particular Mutex inside the map of Mutex.
>>> func RWMaps(targetnum int, purpose, future_id string, market Market) 
>>> map[string]Market {
>>> 
>>> mapLocks[targetnum].Lock()
>>> defer mapLocks[targetnum].Unlock()
>>> 
>>> if purpose == "update" {//The original Write part
>>> marketMaps[targetnum][future_id] = market
>>>return nil
>>> } else { //The read part, has been extracted to ReadMaps
>>> SyChan <- 1
>>> return marketMaps[targetnum]
>>> }
>>> }
>>> 
>>> //Here is why I use map: not all 5 sources must be available, some may have 
>>> connection failure and would be marked as false in Usable[i] , i being its 
>>> source No.
>>> func ReadMaps(targetnum, checkTime int) map[string]Market {
>>> mapLocks[targetnum].Lock()
>>> defer mapLocks[targetnum].Unlock()
>>> if Usable[targetnum] {
>>> fmt.Printf("%d-th time to read source %d \n", checkTime, 
>>> targetnum)
>>> }
>>> SyChan <- 1
>>> return marketMaps[targetnum]
>>> }
>>> 
>>> --
>>> 
>>> My problem is :
>>> 
>>> I still want to keep the map structure, rather 

Re: [go-nuts] Is there a way to syncronise map reads and writes?

[Robert Engels]

The top level map locks doesn’t matter - it is write once at init and read only 
after. 

> On Feb 22, 2022, at 11:51 PM, Jason E. Aten  wrote:
> 
> You have not synchronized access to the top level mapLocks map itself.  Thus 
> you will get races. You need to have a mutex that you lock before you access 
> mapLocks; not just the maps that are inside it.
> 
> I would also recommend being a little object oriented about this design, 
> putting each map inside its own struct, and make a set of accessor methods 
> that Lock() and defer Unlock() on a mutex that protects the maps inside.  
> Read code like this for an idea of what I mean. 
> https://github.com/glycerine/atomicmap/blob/master/amap.go
> 
>> On Tuesday, February 22, 2022 at 9:55:04 PM UTC-6 ren...@ix.netcom.com wrote:
>> Something else is wrong - because marketMaps is read-only after init… so 
>> unless you have some other code - that is not the map causing the panic.
>> 
>> My guess is because you are returning the map from ReadMaps (or RWMaps) that 
>> you are using THAT map by multiple threads outside the lock (the map is a 
>> reference not a copy) and that is causing the panic.
>> 
 On Feb 22, 2022, at 9:39 PM, Zhaoxun Yan  wrote:
 
>>> Hi guys!
>>> I know this is quite challenging, but it makes sense in io-heavy 
>>> applications.
>>> 
>>> I need to test the responsiveness of different financial future quotation 
>>> sources. They return the quotations spontaneously, and my program respond 
>>> to each one of the CGO call backs from the quotation library and save the 
>>> current prices. Meanwhile, a timer ensures to check the current quotations 
>>> of each available sources. And obviously, it raised up a panic due to 
>>> synchronization of maps on the call of ReadMaps function. Unfortunately 
>>> this came up occasionally and `go build -race` cannot raise the problem.
>>> 
>>> Here is a piece of the code, only the read/write part:
>>> 
>>> --
>>> 
>>> // Please Ignore SyChan to alert the timer that all available sources are 
>>> collected
>>> var SyChan chan int = make(chan int, 3)
>>> 
>>> // Make up a map of 5 mutex locks and a map of 5 data structs, indexed by 
>>> 1-5
>>> var marketMaps = make(map[int]map[string]Market)
>>> var mapLocks = make(map[int]*sync.Mutex)
>>> 
>>> // quotation data
>>> type Market struct {
>>> Price float64 `json:"price"`
>>> Timestamp string  `json:"timestamp"`
>>> }
>>> 
>>> func init() {
>>> for i := 1; i <= 5; i++ {
>>> mapLocks[i] = new(sync.Mutex)
>>> marketMaps[i] = make(map[string]Market)
>>> }
>>> }
>>> 
>>> //Make sure that for each source, R/W has no race, only took place as 
>>> acquiring the particular Mutex inside the map of Mutex.
>>> func RWMaps(targetnum int, purpose, future_id string, market Market) 
>>> map[string]Market {
>>> 
>>> mapLocks[targetnum].Lock()
>>> defer mapLocks[targetnum].Unlock()
>>> 
>>> if purpose == "update" {//The original Write part
>>> marketMaps[targetnum][future_id] = market
>>>return nil
>>> } else { //The read part, has been extracted to ReadMaps
>>> SyChan <- 1
>>> return marketMaps[targetnum]
>>> }
>>> }
>>> 
>>> //Here is why I use map: not all 5 sources must be available, some may have 
>>> connection failure and would be marked as false in Usable[i] , i being its 
>>> source No.
>>> func ReadMaps(targetnum, checkTime int) map[string]Market {
>>> mapLocks[targetnum].Lock()
>>> defer mapLocks[targetnum].Unlock()
>>> if Usable[targetnum] {
>>> fmt.Printf("%d-th time to read source %d \n", checkTime, 
>>> targetnum)
>>> }
>>> SyChan <- 1
>>> return marketMaps[targetnum]
>>> }
>>> 
>>> --
>>> 
>>> My problem is :
>>> 
>>> I still want to keep the map structure, rather than naming mutex1, mutex2, 
>>> mutex3,... , marketmsg1, marketmsg2,  And obviously if the map 
>>> prohibits reading or writing spontaneously, the writing by each usable 
>>> sources is not fair - They must line up as one queue (like using one Mutex 
>>> for all instead) hence the checking of quotation snap on a time-spot is 
>>> defected.  I have also checked the sync.Map and it seems to allow 
>>> spontaneous reading but still prohibits spontaneous writing.
>>> 
>>> My instinct is to make map on the pointer of structs. On that way, if I 
>>> could use sync.Map, both the read and write function only read the map to 
>>> find the address of data structs, then writing several data structs on the 
>>> same time won't violate the rule on sync.Map.
>>> 
>>> Is the proposal above legitimate? And anyone could come up a test case to 
>>> mimic the quotation sources? 

Re: [go-nuts] Is there a way to syncronise map reads and writes?

[robert engels]

Something else is wrong - because marketMaps is read-only after init… so unless 
you have some other code - that is not the map causing the panic.

My guess is because you are returning the map from ReadMaps (or RWMaps) that 
you are using THAT map by multiple threads outside the lock (the map is a 
reference not a copy) and that is causing the panic.

> On Feb 22, 2022, at 9:39 PM, Zhaoxun Yan  wrote:
> 
> Hi guys!
> I know this is quite challenging, but it makes sense in io-heavy applications.
> 
> I need to test the responsiveness of different financial future quotation 
> sources. They return the quotations spontaneously, and my program respond to 
> each one of the CGO call backs from the quotation library and save the 
> current prices. Meanwhile, a timer ensures to check the current quotations of 
> each available sources. And obviously, it raised up a panic due to 
> synchronization of maps on the call of ReadMaps function. Unfortunately this 
> came up occasionally and `go build -race` cannot raise the problem.
> 
> Here is a piece of the code, only the read/write part:
> 
> --
> 
> // Please Ignore SyChan to alert the timer that all available sources are 
> collected
> var SyChan chan int = make(chan int, 3)
> 
> // Make up a map of 5 mutex locks and a map of 5 data structs, indexed by 1-5
> var marketMaps = make(map[int]map[string]Market)
> var mapLocks = make(map[int]*sync.Mutex)
> 
> // quotation data
> type Market struct {
> Price float64 `json:"price"`
> Timestamp string  `json:"timestamp"`
> }
> 
> func init() {
> for i := 1; i <= 5; i++ {
> mapLocks[i] = new(sync.Mutex)
> marketMaps[i] = make(map[string]Market)
> }
> }
> 
> //Make sure that for each source, R/W has no race, only took place as 
> acquiring the particular Mutex inside the map of Mutex.
> func RWMaps(targetnum int, purpose, future_id string, market Market) 
> map[string]Market {
> 
> mapLocks[targetnum].Lock()
> defer mapLocks[targetnum].Unlock()
> 
> if purpose == "update" {//The original Write part
> marketMaps[targetnum][future_id] = market
>return nil
> } else { //The read part, has been extracted to ReadMaps
> SyChan <- 1
> return marketMaps[targetnum]
> }
> }
> 
> //Here is why I use map: not all 5 sources must be available, some may have 
> connection failure and would be marked as false in Usable[i] , i being its 
> source No.
> func ReadMaps(targetnum, checkTime int) map[string]Market {
> mapLocks[targetnum].Lock()
> defer mapLocks[targetnum].Unlock()
> if Usable[targetnum] {
> fmt.Printf("%d-th time to read source %d \n", checkTime, 
> targetnum)
> }
> SyChan <- 1
> return marketMaps[targetnum]
> }
> 
> --
> 
> My problem is :
> 
> I still want to keep the map structure, rather than naming mutex1, mutex2, 
> mutex3,... , marketmsg1, marketmsg2,  And obviously if the map prohibits 
> reading or writing spontaneously, the writing by each usable sources is not 
> fair - They must line up as one queue (like using one Mutex for all instead) 
> hence the checking of quotation snap on a time-spot is defected.  I have also 
> checked the sync.Map and it seems to allow spontaneous reading but still 
> prohibits spontaneous writing.
> 
> My instinct is to make map on the pointer of structs. On that way, if I could 
> use sync.Map, both the read and write function only read the map to find the 
> address of data structs, then writing several data structs on the same time 
> won't violate the rule on sync.Map.
> 
> Is the proposal above legitimate? And anyone could come up a test case to 
> mimic the quotation sources? Because with CGO to complicate the program, go 
> cannot raise the race problem on -race check, and the successful build can 
> panic occasionally.
> 
> Thanks in advance! 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/9a981200-23f8-4dae-8c20-7acfdcd3f2fcn%40googlegroups.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this 

Re: [go-nuts] go log.Println(http.ListenAndServe(...)) blocks

[Robert Engels]

Because the arguments are evaluated when the go keyword is encountered - and 
listen and serve is blocking on port 8082

> On Feb 18, 2022, at 10:38 AM, Christoph Berger  
> wrote:
> 
> 
> Hi gophers, 
> 
> I hope someone can help finding out why a particular goroutine call can block 
> the main goroutine.
> 
> In the following code, all "go" calls spawn off a goroutine and return as 
> expected, except for the last one that blocks the main goroutine. 
> 
> Is this a bug, or am I missing something subtle or even obvious (obvious to 
> all others only of course)?
> 
> package main
> 
> import (
> "log"
> "net/http"
> )
> 
> func main() {
> // all of these work as expected
> go http.ListenAndServe("localhost:8080", nil)
> go log.Println("goroutine")
> go func() {
> log.Println(http.ListenAndServe("localhost:8081", nil))
> }()
> 
> // The following line blocks the main goroutine.
> go log.Println(http.ListenAndServe("localhost:8082", nil))
> 
> log.Println("after go log.Println(http.ListenAndServe())") // never 
> prints
> select {} // remove this, and the code still never finishes
> }
> 
> All three servers eventually run (try curl localhost:8080; curl 
> localhost:8081; curl localhost:8082),
> 
> In the playground, the code even deadlocks. 
> 
> Any idea?
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/fd72b5b3-f01e-44d5-a1a0-f784fe7b884fn%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/A941EEE2-51E6-45DD-868F-478798B474A0%40ix.netcom.com.

Re: [go-nuts] Re: Yey another Error Handling proposal

[Robert Engels]

I don’t disagree. It is used in C because they don’t have exceptions so you 
need centralized handling.  Go can be viewed as similar in that respect - but 
with some extra limitations and additions. 

> On Feb 16, 2022, at 4:53 PM, 'Axel Wagner' via golang-nuts 
>  wrote:
> 
> 
> This is golang-nuts, though. Not C-nuts. Go's goto statement is significantly 
> different from C's (see Ian's earlier response). And using it for error 
> handling in this manner is extremely uncommon.
> 
>> On Wed, Feb 16, 2022 at 11:12 PM Robert Engels  wrote:
>> Using goto for error handling in C is very common. See the Linux kernel. 
>> 
>>>> On Feb 16, 2022, at 3:07 PM, Corin Lawson  wrote:
>>>> 
>>> Hi Vojta,
>>> 
>>> Can you please provide some real world examples (e.g. link to open source 
>>> project) or a code style guideline that promotes the use of that pattern of 
>>> using a goto?  I don't believe that it is idiomatic Go.  Personally, I can 
>>> count on one hand the number of times I've seen the usage of goto in Go; be 
>>> it 'in the wild' or otherwise.
>>> 
>>> I appriciate the leg work that you've done to get to this point, I can't 
>>> honestly say I've reviewed the existing error handling proposals.  I 
>>> imagine it's a hot topic!  I am not the gatekeeper of what is and is not 
>>> idiomatic Go (I'm not sure anyone is!)  But can't say I share your 
>>> experience; when I read and write code in my workplace, a lot of the error 
>>> handling involves logic specific to the call the produced the error (e.g. 
>>> wrapping the error) or a simple naked return.  I just don't see the value 
>>> proposition at this time.
>>> 
>>> Cheers,
>>> Corin
>>> 
>>>> On Wednesday, 16 February 2022 at 12:50:16 am UTC+11 bargl@gmail.com 
>>>> wrote:
>>>> Hi, 
>>>> my name is Vojta and I would like to join a error handling proposals 
>>>> discussion and because I don't know what else to say I guess I will jump 
>>>> right to it.
>>>> 
>>>> I know everyone has his/her own opinion about this topic, which has its 
>>>> pros and cons. 
>>>> And even though I think current solution is well-done I found myself 
>>>> smiling when I browse through my or someone else's source code because of 
>>>> that very well known reoccurring pattern:
>>>> ```
>>>> if err != nil { ... } 
>>>> ```
>>>> Do not get me wrong but I think it is pretty ironic when you see 
>>>> reoccurring pattern in context where you try to minimize these into more 
>>>> generalized form.
>>>> 
>>>> I tried to read most issues on Github with error-handling label, but there 
>>>> are just so many that in this point I am glad I found link to Error 
>>>> Handling meta issue which summarize all important issues about this topic. 
>>>> I would like to get your opinion about solution that I did not find in 
>>>> this summarized list.
>>>> 
>>>> I would like to get opinion of people that know little more about golang 
>>>> itself and are able to provide "holes" in this solution. Feel free to 
>>>> point them out, but please keep in mind that I may not be able to solve 
>>>> them alone. Like I said, I just wanted to discuss this solution before I 
>>>> file official issue proposal.
>>>> 
>>>> Solution
>>>> I got inspired with golangs `:=` operator that handles declaration and 
>>>> assignment of variable. It's basically two operations in one. So what 
>>>> about using something similar, like `?=`, that would assign variables and 
>>>> additionally check if last variable (if error) is not nil?
>>>> 
>>>> What I'm talking about is replace these lines:
>>>> ```
>>>> if value, err = ReturnValueAndErr(); err != nil { 
>>>>   goto Err 
>>>> }
>>>> if otherValue, err = DeriveAnotherValueAndErr(value); err != nil { 
>>>>   goto Err 
>>>> } 
>>>> 
>>>> Err: 
>>>>   // handle errors
>>>> ```
>>>> 
>>>> with these lines:
>>>> ```
>>>> value, err ?= ReturnValueAndErr()
>>>> otherValue, err ?= DeriveAnotherValueAndErr(value)
>>>> 
>>>> error:
>>>> // handle error
>>>> ```
>>>> 
>>>> It's very short and seems idiomatic 

Re: [go-nuts] Re: Yey another Error Handling proposal

[Robert Engels]

Using goto for error handling in C is very common. See the Linux kernel. 

> On Feb 16, 2022, at 3:07 PM, Corin Lawson  wrote:
> 
> Hi Vojta,
> 
> Can you please provide some real world examples (e.g. link to open source 
> project) or a code style guideline that promotes the use of that pattern of 
> using a goto?  I don't believe that it is idiomatic Go.  Personally, I can 
> count on one hand the number of times I've seen the usage of goto in Go; be 
> it 'in the wild' or otherwise.
> 
> I appriciate the leg work that you've done to get to this point, I can't 
> honestly say I've reviewed the existing error handling proposals.  I imagine 
> it's a hot topic!  I am not the gatekeeper of what is and is not idiomatic Go 
> (I'm not sure anyone is!)  But can't say I share your experience; when I read 
> and write code in my workplace, a lot of the error handling involves logic 
> specific to the call the produced the error (e.g. wrapping the error) or a 
> simple naked return.  I just don't see the value proposition at this time.
> 
> Cheers,
> Corin
> 
>> On Wednesday, 16 February 2022 at 12:50:16 am UTC+11 bargl@gmail.com 
>> wrote:
>> Hi, 
>> my name is Vojta and I would like to join a error handling proposals 
>> discussion and because I don't know what else to say I guess I will jump 
>> right to it.
>> 
>> I know everyone has his/her own opinion about this topic, which has its pros 
>> and cons. 
>> And even though I think current solution is well-done I found myself smiling 
>> when I browse through my or someone else's source code because of that very 
>> well known reoccurring pattern:
>> ```
>> if err != nil { ... } 
>> ```
>> Do not get me wrong but I think it is pretty ironic when you see reoccurring 
>> pattern in context where you try to minimize these into more generalized 
>> form.
>> 
>> I tried to read most issues on Github with error-handling label, but there 
>> are just so many that in this point I am glad I found link to Error Handling 
>> meta issue which summarize all important issues about this topic. I would 
>> like to get your opinion about solution that I did not find in this 
>> summarized list.
>> 
>> I would like to get opinion of people that know little more about golang 
>> itself and are able to provide "holes" in this solution. Feel free to point 
>> them out, but please keep in mind that I may not be able to solve them 
>> alone. Like I said, I just wanted to discuss this solution before I file 
>> official issue proposal.
>> 
>> Solution
>> I got inspired with golangs `:=` operator that handles declaration and 
>> assignment of variable. It's basically two operations in one. So what about 
>> using something similar, like `?=`, that would assign variables and 
>> additionally check if last variable (if error) is not nil?
>> 
>> What I'm talking about is replace these lines:
>> ```
>> if value, err = ReturnValueAndErr(); err != nil { 
>>   goto Err 
>> }
>> if otherValue, err = DeriveAnotherValueAndErr(value); err != nil { 
>>   goto Err 
>> } 
>> 
>> Err: 
>>   // handle errors
>> ```
>> 
>> with these lines:
>> ```
>> value, err ?= ReturnValueAndErr()
>> otherValue, err ?= DeriveAnotherValueAndErr(value)
>> 
>> error:
>> // handle error
>> ```
>> 
>> It's very short and seems idiomatic to golang and it's main feature is it 
>> does not break the flow of thought that author tried to express. Error 
>> handling itself is already defined (and used) feature - labels and name of 
>> label is intentionally already known keyword to get the link between ?= 
>> operator and error handling. 
>> 
>> There are few limitations though:
>> variables needs to be declared before
>> (I mean not really, but idea is to access assigned variables in label.
>> so value, otherValue and err should be declared)
>> label error must exists and serve only this purpose
>> (compiler option could change the name for backward compatibility)
>> So what do you say?
>> Can you make it better?
>> 
>> Cheers,
>> Vojta
>> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/be02eec5-34f6-429f-965f-30fe6b39893fn%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/14F50F22-CC7A-41B9-9D10-F2A664F6C85B%40ix.netcom.com.

Re: [go-nuts] Re: C function strtod() crashes with cgo on Windows, with gcc 64 bits

[Robert Engels]

My guess is that you need to set the compiler options to ensure a double is 8 
bytes to match Go. 

My point still applies - the stack is being corrupted due to a size mismatch in 
the parameters/return values. 

The fact that the original bug said it worked on Linux and crashed on windows 
and now it is crashing on two different versions of Linux is additional 
evidence. 

I think the exact compiler version and command line options are needed to 
diagnose. 

> On Feb 16, 2022, at 8:27 AM, Robert Engels  wrote:
> 
> 
> You might want to share a small reproducible test case - it is almost 
> certainly an issue on your end. Most likely stack corruption creating invalid 
> pointer/return addresses. When you return a double rather than a float you 
> are changing the stack size and alignment. 
> 
>>> On Feb 16, 2022, at 8:19 AM, stryker2k2  wrote:
>>> 
>> Folks of golang-nuts,
>> 
>> It is now February of 2022 and it seems that "strtod" is still and issue. 
>> After hours of searching for a solution, I found this Google Group dated 8 
>> years earlier (technically 7 1/2 years) and found my answer! So, although I 
>> am raising an old post from the grave, I feel that I must add in my findings 
>> for the next poor chap who sees this post in 8 years and also show my 
>> appreciation nearly a decade afterwards.
>> 
>> We have a very complex program that we compile for Windows 7/10 using the 
>> developers choice of either Ubuntu 18.04 ( gcc version 7.3-win32) and/or 
>> Ubuntu 20.04 (gcc version 9.3-win32). My developers using Ubuntu 18.04 (gcc 
>> 7.3) noticed that a portion of our program would crash when being accessed. 
>> I did some digging and this is what I saw...
>> 
>> `strtod` compiled on Ubuntu 18.04 (gcc 7.3) = CRASH
>> `strtod` compiled on Ubuntu 20.04 (gcc 9.3) = No Issue
>> 
>> That lead me to this post. 
>> 
>> We are now using `strtold` instead. `strtold` works on both version of 
>> Ubuntu/gcc.
>> 
>> Thank you all a ton for having this conversation 8+ years ago! You all rock!
>>> On Wednesday, October 15, 2014 at 8:34:50 PM UTC-5 brainman wrote:
>>> I don't know what the problem is, but I have created new issue for it 
>>> https://code.google.com/p/go/issues/detail?id=8941.
>>> 
>>> Alex
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/55152b22-0581-4780-82ce-e29f82310579n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/5DA39BAD-9FF4-4A71-8EFB-75BBCDFC5A68%40ix.netcom.com.

Re: [go-nuts] Re: C function strtod() crashes with cgo on Windows, with gcc 64 bits

[Robert Engels]

Apologies. I see the issue has a sample case attached. 

> On Feb 16, 2022, at 8:27 AM, Robert Engels  wrote:
> 
> 
> You might want to share a small reproducible test case - it is almost 
> certainly an issue on your end. Most likely stack corruption creating invalid 
> pointer/return addresses. When you return a double rather than a float you 
> are changing the stack size and alignment. 
> 
>>> On Feb 16, 2022, at 8:19 AM, stryker2k2  wrote:
>>> 
>> Folks of golang-nuts,
>> 
>> It is now February of 2022 and it seems that "strtod" is still and issue. 
>> After hours of searching for a solution, I found this Google Group dated 8 
>> years earlier (technically 7 1/2 years) and found my answer! So, although I 
>> am raising an old post from the grave, I feel that I must add in my findings 
>> for the next poor chap who sees this post in 8 years and also show my 
>> appreciation nearly a decade afterwards.
>> 
>> We have a very complex program that we compile for Windows 7/10 using the 
>> developers choice of either Ubuntu 18.04 ( gcc version 7.3-win32) and/or 
>> Ubuntu 20.04 (gcc version 9.3-win32). My developers using Ubuntu 18.04 (gcc 
>> 7.3) noticed that a portion of our program would crash when being accessed. 
>> I did some digging and this is what I saw...
>> 
>> `strtod` compiled on Ubuntu 18.04 (gcc 7.3) = CRASH
>> `strtod` compiled on Ubuntu 20.04 (gcc 9.3) = No Issue
>> 
>> That lead me to this post. 
>> 
>> We are now using `strtold` instead. `strtold` works on both version of 
>> Ubuntu/gcc.
>> 
>> Thank you all a ton for having this conversation 8+ years ago! You all rock!
>>> On Wednesday, October 15, 2014 at 8:34:50 PM UTC-5 brainman wrote:
>>> I don't know what the problem is, but I have created new issue for it 
>>> https://code.google.com/p/go/issues/detail?id=8941.
>>> 
>>> Alex
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/55152b22-0581-4780-82ce-e29f82310579n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/172BABD6-6037-422F-B1D5-22A8652FA9FB%40ix.netcom.com.

Re: [go-nuts] Re: C function strtod() crashes with cgo on Windows, with gcc 64 bits

[Robert Engels]

You might want to share a small reproducible test case - it is almost certainly 
an issue on your end. Most likely stack corruption creating invalid 
pointer/return addresses. When you return a double rather than a float you are 
changing the stack size and alignment. 

> On Feb 16, 2022, at 8:19 AM, stryker2k2  wrote:
> 
> Folks of golang-nuts,
> 
> It is now February of 2022 and it seems that "strtod" is still and issue. 
> After hours of searching for a solution, I found this Google Group dated 8 
> years earlier (technically 7 1/2 years) and found my answer! So, although I 
> am raising an old post from the grave, I feel that I must add in my findings 
> for the next poor chap who sees this post in 8 years and also show my 
> appreciation nearly a decade afterwards.
> 
> We have a very complex program that we compile for Windows 7/10 using the 
> developers choice of either Ubuntu 18.04 ( gcc version 7.3-win32) and/or 
> Ubuntu 20.04 (gcc version 9.3-win32). My developers using Ubuntu 18.04 (gcc 
> 7.3) noticed that a portion of our program would crash when being accessed. I 
> did some digging and this is what I saw...
> 
> `strtod` compiled on Ubuntu 18.04 (gcc 7.3) = CRASH
> `strtod` compiled on Ubuntu 20.04 (gcc 9.3) = No Issue
> 
> That lead me to this post. 
> 
> We are now using `strtold` instead. `strtold` works on both version of 
> Ubuntu/gcc.
> 
> Thank you all a ton for having this conversation 8+ years ago! You all rock!
>> On Wednesday, October 15, 2014 at 8:34:50 PM UTC-5 brainman wrote:
>> I don't know what the problem is, but I have created new issue for it 
>> https://code.google.com/p/go/issues/detail?id=8941.
>> 
>> Alex
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/55152b22-0581-4780-82ce-e29f82310579n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/483D32A3-6FE0-4D69-8F03-CE20E7A91FB2%40ix.netcom.com.

Re: [go-nuts] cmd/trace: Indirect reporting of OS Latency?

[Robert Engels]

A windows machine won’t have ‘perf sched’. 

I don’t think Windows has similar lowlevel scheduling tracing - at least not 
that I remember. 

I suggest installing Linux on a separate partition and direct booting. 

Linux has a LOT of scheduling tracing/tuning options - some require a custom 
kernel. 

Have fun!


> On Feb 12, 2022, at 8:51 PM, Rich Moyse  wrote:
> 
> 
> Robert, thanks for your reply - especially the link to perf sched!  I plan to 
> run the go program on Windows physical machine.  I'll post my results once 
> done.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/18ebe8c8-4741-4e0f-93a1-0c316444588an%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/F1F3C1AC-37B8-4D98-8BC2-7165C1E7F9C0%40ix.netcom.com.

Re: [go-nuts] cmd/trace: Indirect reporting of OS Latency?

[robert engels]

I suggest you use the linux scheduler tracing facilities, e.g. perf shed to 
diagnose (see https://www.brendangregg.com/perf.html#SchedulerAnalysis 
)

But you may have to also do this outside the VM - to schedule trace the VM.

In general, timing at 1ms intervals with a 2% error is probably all you can 
expect running in a VM - especially if you don’t know (or manage) the load on 
the bare metal. 2% on 1ms is 20 us - and a raw bare metal scheduler will 
probably only get you to 5-6 us.

You can simplify your analysis by testing on bare metal first.



> On Feb 11, 2022, at 10:05 PM, Rich Moyse  wrote:
> 
> TLDR
> I’d be grateful if someone could either confirm or dismiss the effect of OS, 
> go runtime, and/or Hypervisor latency on the output of a goroutine trace.  
> The trace report displays empty gaps of 3-14ms where expected periodic 
> goroutines should execute, as there’s a goroutine triggered by each 1ms timer 
> tick.  Attempts to select items within these gaps return only five “counter” 
> items - nothing else.  Those counter items suggest that nothing is running.  
> Perhaps external preemptive behavior interrupts the running go process, 
> thereby suspending the go runtime, causing these gaps?  If so, can these 
> trace gaps be attributed to OS and/or Hypervisor scheduler preemption?
> 
> 
> Although a recent (2/4/2022) go-nuts summary mentions a post to the thread: 
> Issues when using time.Ticker microsecond duration 
> , whose 
> other posts discuss a range of causes, including OS induced latency, this 
> thread attempts to understand how external latency sources affect the trace 
> report.
> 
> In addition to the problem description below, the go code and trace screen 
> images are available via this gist 
>  
> while the trace file is available here 
> .
>   The go program was compiled first using 1.18beta1 and later with 1.17.  
> It’s running in a 16.04 Ubuntu Desktop VM that’s managed by Hyper-V on a 
> physical machine running Windows 10 Pro.  The go program limits itself to the 
> interplay between two goroutines: a single sending goroutine and a single 
> receiving one.
> 
> These goroutines share an unbuffered channel.  The sender is only throttled 
> by the capacity of the receiver to consume its messages while the receiver is 
> gated by a timer that ticks in 1ms 
>  intervals.  The sending code 
> executes in ~600ns  then blocks 
> waiting for the receiver.  The receiver executes in ~8us 
> , starting after the ticker 
> expires until it's blocked waiting for the timer’s next tick.  Due to their 
> speedy execution, the go runtime scheduler never intervenes to preemptively 
> interrupt either goroutine.  Moreover, the garbage collector doesn’t run 
> because only this same pair of goroutines are active and the message 
> communicated between them is an empty structure.
> 
> In general the regular rhythm of the receiver’s timer appears in the 
> “Goroutine analysis” view of the trace report as evenly spaced markings at 
> nearly one 1ms intervals.  However, these ruler-like markings are 
> sporadically absent resulting in gaps between 3-14ms.  During these gaps 
> neither the receiving goroutine nor its coupled sender execute.  When 
> selecting the rectangular gap area only “5 items'' are reported as selected. 
> These 5 items displayed by the “Counter Samples (5)” tab indicate that no 
> goroutines are “Running”, nor are there “Runnable” ones.  Here are all the 
> counters reported by the selected gap appearing at time: ~502ms into the 
> trace report: 
>   
> Counter  Series Time   
> Value
> 
> Goroutines GCWaiting 502.3296180004 0
> 
> GoroutinesRunnable  502.32961800040
> 
> GoroutinesRunning   502.32961800040
> 
> Threads   InSyscall 501.2523010
> 
>  Threads  Running   501.2523011
> 
> 
> Since the program was running trace, to collect the data needed to produce 
> its report, perhaps the trace itself induced a measure of go runtime latency? 
>  To eliminate this potential latency source, the trace package and its 
> associated http server goroutine were eliminated.  Instead, latency events 
> were directly detected and reported by the receiving goroutine.  This 
> goroutine computed the latency between timer ticks by remembering the prior 
> timer value and subtracting it from the newly provided one.  It outputted, 
> via fmt.Print, a message whenever the computed latency exceeded 5ms. 

Re: [go-nuts] Does anyone gets automatic response from 944677...@qq.com?

[Robert Engels]

Same issue. Some one set up auto reply to message threads. 

> On Feb 11, 2022, at 6:36 AM, Kamil Ziemian  wrote:
> 
> 
> Hello,
> 
> I try to don't care about this response, but I see them one time to much. 
> After most if not all my post here, I get email from 944677...@qq.com with 
> title
> "自动回复: [go-nuts]: title of the thread"
> and content
> "这是来自QQ邮箱的假期自动回复邮件。
>  
> 您好，我暂时无法亲自回复您的邮件。我将在最短的时间内尽快给您回复。谢谢。"
> Gmail translate it to
> "This is a holiday auto-reply email from QQ mailbox.
> 
> Hello, I am temporarily unable to respond to your email in person. I will 
> reply to you as soon as possible in the shortest possible time. thanks."
> 
> It happens even if I gust open new thread, so no one other than me is 
> involved in it. Does anyone else has this problem? And how I can solve it?
> 
> Best,
> Kamil
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/271ada26-1629-41e9-8bed-ac0b0fa2d5a9n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/84EB7275-CB7D-4B8A-9C50-2AA601FE10FE%40ix.netcom.com.

Re: [go-nuts] Re: DataRace with slice, should i fix it or ignore it?

[Robert Engels]

See issue 5045

> On Feb 9, 2022, at 8:42 AM, Robert Engels  wrote:
> 
> 
> The problem with this reasoning is that Go still does not have an explicit 
> memory model with respect to happens before relationships - falling back on 
> “whatever the current implementation behavior is”. This makes it more likely 
> that any change in the compiler will need to continue to support the benign 
> data races. 
> 
>>> On Feb 9, 2022, at 8:25 AM, peterGo  wrote:
>>> 
>> 
>> Pelen Li,
>> 
>> Boehm covers your specific case: "there is no reason to believe that a 
>> currently working program with “benign races” will continue to work when it 
>> is recompiled. Perhaps most surprisingly, this includes even the case of 
>> potentially concurrent writes of the same value by different threads."
>> 
>> Peter
>> 
>>> On Wed, Feb 9, 2022 at 9:17 AM peterGo  wrote:
>>> Pelen Li,
>>> 
>>> Always fix your data races. You should consider the results of data races 
>>> as undefined.
>>> 
>>> Dmitry Vyukov, who implemented the Go race detector, once wrote an 
>>> interesting article with the title: "Benign data races: what could possibly 
>>> go wrong?" 
>>> 
>>> https://twitter.com/dvyukov/status/288858957827682304
>>> 
>>> The Go Blog: Introducing the Go Race Detector
>>> Dmitry Vyukov and Andrew Gerrand
>>> 26 June 2013
>>> 
>>> https://go.dev/blog/race-detector
>>> 
>>> Hans-J. Boehm wrote: "there is no reason to believe that a currently 
>>> working program with “benign races” will continue to work when it is 
>>> recompiled."
>>> 
>>> How to miscompile programs with “benign” data races
>>> Hans-J. Boehm
>>> HP Laboratories
>>> 
>>> https://www.usenix.org/legacy/event/hotpar11/tech/final_files/Boehm.pdf
>>> 
>>> Nonetheless many programmers clearly believe, along with [15] that certain 
>>> kinds of data races can be safely ignored in practice because they will 
>>> produce expected results with all reasonable implementations. Here we show 
>>> that all kinds of C or C++ source-level “benign” races discussed in the 
>>> literature can in fact lead to incorrect execution as a result of perfectly 
>>> reasonable compiler transformations, or when the program is moved to a 
>>> different hardware platform. Thus there is no reason to believe that a 
>>> currently working program with “benign races” will continue to work when it 
>>> is recompiled. Perhaps most surprisingly, this includes even the case of 
>>> potentially concurrent writes of the same value by different threads.
>>> 
>>> And so on.
>>> 
>>> Peter
>>> 
>>>> On Wednesday, February 9, 2022 at 3:21:26 AM UTC-5 penglo...@gmail.com 
>>>> wrote:
>>>> I want to set a value with the index of the slice. I don't really care if 
>>>> there are multiple goroutines cover the value with each other, because the 
>>>> value is same.
>>>> 
>>>> Can i just ignore this DataRace Warning? I don't know if this will cause 
>>>> panic.
>>>> 
>>>> Here's my example:
>>>> I defined a structure with slice, and a Add() function for it. sample like 
>>>> this:
>>>> ```go
>>>> package test_slice
>>>> 
>>>> type SliceObj struct {
>>>> set []uint
>>>> }
>>>> 
>>>> func New(length int64) *SliceObj {
>>>> return {
>>>> set: make([]uint, length),
>>>> }
>>>> }
>>>> 
>>>> func (b *SliceObj) Add(i uint) {
>>>> b.set[i] = i
>>>> }
>>>> ```
>>>> 
>>>> And then i make a main file to test it, like this:
>>>> ```go
>>>> package main
>>>> 
>>>> import (
>>>> "time"
>>>> 
>>>> "test_slice"
>>>> )
>>>> 
>>>> func main() {
>>>> s := test_slice.New(100)
>>>> go func() {
>>>> s.Add(10)
>>>> }()
>>>> s.Add(10)
>>>> 
>>>> time.Sleep(3 * time.Second)
>>>> }
>>>> ```
>>>> 
>>>> And data race is detected:
>>>> (I know the reason of this warning, but I don't know if I can ignore it)
>>>

Re: [go-nuts] Re: DataRace with slice, should i fix it or ignore it?

[Robert Engels]

The problem with this reasoning is that Go still does not have an explicit 
memory model with respect to happens before relationships - falling back on 
“whatever the current implementation behavior is”. This makes it more likely 
that any change in the compiler will need to continue to support the benign 
data races. 

> On Feb 9, 2022, at 8:25 AM, peterGo  wrote:
> 
> 
> Pelen Li,
> 
> Boehm covers your specific case: "there is no reason to believe that a 
> currently working program with “benign races” will continue to work when it 
> is recompiled. Perhaps most surprisingly, this includes even the case of 
> potentially concurrent writes of the same value by different threads."
> 
> Peter
> 
>> On Wed, Feb 9, 2022 at 9:17 AM peterGo  wrote:
>> Pelen Li,
>> 
>> Always fix your data races. You should consider the results of data races as 
>> undefined.
>> 
>> Dmitry Vyukov, who implemented the Go race detector, once wrote an 
>> interesting article with the title: "Benign data races: what could possibly 
>> go wrong?" 
>> 
>> https://twitter.com/dvyukov/status/288858957827682304
>> 
>> The Go Blog: Introducing the Go Race Detector
>> Dmitry Vyukov and Andrew Gerrand
>> 26 June 2013
>> 
>> https://go.dev/blog/race-detector
>> 
>> Hans-J. Boehm wrote: "there is no reason to believe that a currently working 
>> program with “benign races” will continue to work when it is recompiled."
>> 
>> How to miscompile programs with “benign” data races
>> Hans-J. Boehm
>> HP Laboratories
>> 
>> https://www.usenix.org/legacy/event/hotpar11/tech/final_files/Boehm.pdf
>> 
>> Nonetheless many programmers clearly believe, along with [15] that certain 
>> kinds of data races can be safely ignored in practice because they will 
>> produce expected results with all reasonable implementations. Here we show 
>> that all kinds of C or C++ source-level “benign” races discussed in the 
>> literature can in fact lead to incorrect execution as a result of perfectly 
>> reasonable compiler transformations, or when the program is moved to a 
>> different hardware platform. Thus there is no reason to believe that a 
>> currently working program with “benign races” will continue to work when it 
>> is recompiled. Perhaps most surprisingly, this includes even the case of 
>> potentially concurrent writes of the same value by different threads.
>> 
>> And so on.
>> 
>> Peter
>> 
>>> On Wednesday, February 9, 2022 at 3:21:26 AM UTC-5 penglo...@gmail.com 
>>> wrote:
>>> I want to set a value with the index of the slice. I don't really care if 
>>> there are multiple goroutines cover the value with each other, because the 
>>> value is same.
>>> 
>>> Can i just ignore this DataRace Warning? I don't know if this will cause 
>>> panic.
>>> 
>>> Here's my example:
>>> I defined a structure with slice, and a Add() function for it. sample like 
>>> this:
>>> ```go
>>> package test_slice
>>> 
>>> type SliceObj struct {
>>> set []uint
>>> }
>>> 
>>> func New(length int64) *SliceObj {
>>> return {
>>> set: make([]uint, length),
>>> }
>>> }
>>> 
>>> func (b *SliceObj) Add(i uint) {
>>> b.set[i] = i
>>> }
>>> ```
>>> 
>>> And then i make a main file to test it, like this:
>>> ```go
>>> package main
>>> 
>>> import (
>>> "time"
>>> 
>>> "test_slice"
>>> )
>>> 
>>> func main() {
>>> s := test_slice.New(100)
>>> go func() {
>>> s.Add(10)
>>> }()
>>> s.Add(10)
>>> 
>>> time.Sleep(3 * time.Second)
>>> }
>>> ```
>>> 
>>> And data race is detected:
>>> (I know the reason of this warning, but I don't know if I can ignore it)
>>> ==
>>> WARNING: DATA RACE
>>> Write at 0x00c000180050 by goroutine 18:
>>>   test_slice.(*SliceObj).Add()
>>>   test_slice.go:27 +0x68
>>>   main.main.func1()
>>>   test.go:25 +0x36
>>> 
>>> Previous write at 0x00c000180050 by main goroutine:
>>>   test_slice.(*SliceObj).Add()
>>>   test_slice.go:27 +0xfd
>>>   main.main()
>>>   test.go:27 +0xcb
>>> 
>>> Goroutine 18 (running) created at:
>>>   main.main()
>>>   test.go:24 +0xca
>>> ==
>>> Found 1 data race(s)
>>> exit status 66
>> 
>> -- 
>> You received this message because you are subscribed to a topic in the 
>> Google Groups "golang-nuts" group.
>> To unsubscribe from this topic, visit 
>> https://groups.google.com/d/topic/golang-nuts/ai0eQtnas7A/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to 
>> golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/2594a018-872a-4d54-8ede-c769042e99b5n%40googlegroups.com.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> 

Re: [go-nuts] Unstable Performance of Go Program

[Robert Engels]

You can use taskset when starting the Go program to set the cpu affinity. 

> On Feb 9, 2022, at 12:56 AM, Fadhil Kurnia  wrote:
> 
> Hi all,
> 
> I am doing some latency measurement of a subprocess in Go. However I found 
> that Go program is unstable, especially when there is delay/gap between 
> process that I want to measure. Please see the example at 
> https://github.com/klauspost/reedsolomon/issues/180. It also happened in 
> other subprocess besides that reed-solomon encoding.
> 
> I suspect, it was caused by the golang runtime scheduler. Other people, who 
> do research with prototype written in Go, also found the same issues. Here 
> are what they wrote in EPaxos Revisited paper, from NSDI'21:
> "When investigating the factors that limit server throughput, we observed 
> several anomalies as servers neared saturation. Seemingly innocuous changes 
> could have a large impact on throughput. For example, increasing the num ber 
> of available cores caused throughput to drop. We believe that some of the 
> issues may come from unexpected behavior of Go’s thread scheduler.  We 
> suspect that Go occasionally deschedules that thread in order to run 
> less-critical threads that listen for incoming messages."
> 
> Do you have any recommendation on how to make the runtime scheduler more 
> deterministic? 
> 
> I tried to run the program with GOMAXPRCS=1 to limit the CPU usage, 
> minimizing golang scheduler to switch between CPU. However the documentation 
> says, it limit the number of thread used, thus the scheduler can still move 
> the thread from one CPU to another CPU. Is there any way to pin a go program 
> to a specific CPU?
> 
> Thank you.
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/6f4f2d1f-c22e-4402-a887-3dc354d5dcd3n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/F87AD613-6A86-41A2-8462-1F0519178F62%40ix.netcom.com.

Re: [go-nuts] Slices of pointers or not?

[Robert Engels]

I think the OPs question was specifically about the cost of returning from a 
function - it is the same. 

> On Feb 3, 2022, at 8:03 PM, Connor Kuehl  wrote:
> 
> On Thu, Feb 3, 2022 at 7:07 PM Paulo Júnior  wrote:
>> 
>> Hi all.
>> 
>> I hope you are well.
>> 
>> Is there a big difference, in terms of performance or functionality, between 
>> declaring []*Person or []Person as a return type of a function?
> 
> If you find yourself iterating over a group of structs like this a lot
> (especially if there's a lot of them to iterate over), you might want
> to consider measuring the performance differences between []*Person or
> []Person. With pointers, the actual structs might be far away from
> each other in memory, which may prevent you from taking full advantage
> of your processor's cache since it won't be able to benefit from the
> spatial locality that an array (or slice) offers.
> 
> Connor
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAC_QXZS-QEKTgvsK3TKk-yUyc1jP81HngHz6dcDV8bZwqLBT6Q%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/EF34CFE5-9A6A-4CF7-80C4-83F730518260%40ix.netcom.com.

Re: [go-nuts] Register-based ABI benchmarks

[Robert Engels]

+1. Sometimes the compiler optimizations are even worse if they change the 
behavior the chip was typically expecting. 

> On Feb 3, 2022, at 2:23 PM, Ian Lance Taylor  wrote:
> 
> On Thu, Feb 3, 2022 at 7:21 AM Didier Spezia  wrote:
>> 
>> It seems Aarch64 benefits more from the register-based ABI than x86_64.
>> I don''t see really why. Does anyone have a clue?
> 
> My view is that the x86 architecture has fewer registers and has had a
> massive decades-long investment in performance, so stack operations
> are highly optimized in hardware, including things like forwarding
> values stored in the stack by the caller to the retrieval from the
> stack by the callee without waiting even for the memory cache.  The
> ARM architecture has more registers and has historically focused more
> on power savings than on raw performance, so it has less optimization
> on stack handling and benefits more from a smarter compiler.
> 
> In my experience testing compiler optimizations can be frustrating on
> x86 because the hardware is just so good.  Almost every other
> processor architecture shows bigger benefits from compiler
> optimizations.
> 
> Ian
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAOyqgcVBg%2BWkrT636M-VuBjnaSOjUiAd_Einso_%3DBWFWMKRttA%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/32CF3C8F-EE52-4A60-8EB0-53B8CB2E164D%40ix.netcom.com.

Re: [go-nuts] Slices of pointers or not?

[Robert Engels]

No. Because a slice is a slice. But there are several performance differences 
in the usage. 

> On Feb 3, 2022, at 7:09 PM, Paulo Júnior  wrote:
> 
> Hi all.
> 
> I hope you are well. 
> 
> Is there a big difference, in terms of performance or functionality, between 
> declaring []*Person or []Person as a return type of a function?
> 
> Code sample https://go.dev/play/p/tBAod1hZvYu
> 
> Thank you and best regards.
> Paulo.
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/964eb09d-cd5f-4da1-b1ca-cac3e9b5bb69n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/5C7F5D64-0B5D-41D6-9DFE-0A9CB6AC9A67%40ix.netcom.com.

Re: [go-nuts] Register-based ABI benchmarks

[Robert Engels]

Usually Arm cpus have a lot more registers to pass values in. 

> On Feb 3, 2022, at 9:21 AM, Didier Spezia  wrote:
> 
> We are using our own benchmark to evaluate the performance of different CPU 
> models of cloud providers.
> https://github.com/AmadeusITGroup/cpubench1A
> 
> One point we have realized is the results of such benchmark can be biased 
> depending on the version of the Go compiler. 
> 
> For instance, the register-based ABI has a measurable positive impact on 
> performance, but it does not come with the same version of Go depending on 
> the CPU architecture. When we run different versions of Go against the same 
> code base for recent Intel and ARM CPUs, we get: 
> https://github.com/AmadeusITGroup/cpubench1A/issues/8
> 
> It is about +10% throughput for x86_86 (from go 1.16.13 -> 1.17.6) and +17% 
> for Aarch64 (from go 1.17.6 -> 1.18beta1). Yay!
> 
> It seems Aarch64 benefits more from the register-based ABI than x86_64.
> I don''t see really why. Does anyone have a clue?
> Thanks.
> 
> Best regards,
> Didier.
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/0dae635a-768a-4cf4-ae05-84e294ca8745n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/FDD03561-166B-436B-9475-B0E653FA5F3C%40ix.netcom.com.

Re: [go-nuts] Issues when using time.Ticker microsecond duration

[Robert Engels]

I am unclear why you need to use an N of M. I would make sure the hardest case 
is handled and you can use a variety of techniques to partition the work. 

> On Feb 2, 2022, at 6:58 PM, envee  wrote:
> 
> And I forgot to mention that approach I mentioned talks about waking up N 
> goroutines at a time. The way I plan to do is to select a range of N 
> goroutines from my list of goroutines and only allow those goroutines to send 
> HTTP requests.
> I could use this approach to select the N goroutines or even use a Semaphore. 
> I presume using a Semaphore, will allow a good amount of random N goroutines 
> out of M goroutines to execute.
> 
>> On Thursday, 3 February 2022 at 10:26:28 UTC+11 envee wrote:
>> Thanks Robert, Uli and Ian for your suggestions.
>> 
>> I think what I will probably do is use a Ticker with a duration of 1ms. At 
>> every 1ms, I will "wake-up" N number of goroutines to trigger HTTP requests.
>> That number N = (request rate per second / 1000)  = requests per ms.
>> So, if I need to ensure a rate of 1 requests per second, I believe it 
>> should be possible for the Ticker to return after every 1ms and then fire 
>> about 10 requests at every such interval.
>> From the tests that I have performed, I can see that a Ticker pretty 
>> accurately fires at every 1ms interval.
>> I think it's only when the Ticker duration falls below 1ms, that I see 
>> issues.
>> 
>> If my desired rate is less than 1000 per second, then I will create a Ticker 
>> to return every 1000/request rate milliseconds, which will be a number 
>> greater than 1ms. 
>> 
>> This approach is closely based on Robert's suggestion about using a higher 
>> duration for Ticker time and waking up a small subset of goroutines.
>> 
>> I think it should be ok for a client to be accurate at the level of 
>> granularity of 1ms.
>> 
>> 
>>> On Thursday, 3 February 2022 at 01:14:20 UTC+11 ren...@ix.netcom.com wrote:
>>> Because 2 is a magnitude larger than 2000. 
>>> 
>>> > On Feb 1, 2022, at 1:44 PM, Uli Kunitz  wrote: 
>>> > 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/209b18aa-1317-4d72-80a1-222f10a26013n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/F912EE7E-4F80-4C16-ABC1-943572FD576C%40ix.netcom.com.

Re: [go-nuts] Issues when using time.Ticker microsecond duration

[Robert Engels]

Because 2 is a magnitude larger than 2000. 

> On Feb 1, 2022, at 1:44 PM, Uli Kunitz  wrote:
> 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/A56C50B7-D136-4141-A81C-A28081023A79%40ix.netcom.com.

Re: [go-nuts] Re: Why, oh why, do people do this? Things that annoy me in Go code.

[Robert Engels]

If I hired on to a company that enforced 3-4 line function calls I would be 
looking for a job the next day. 

> On Jan 31, 2022, at 6:44 PM, Rick  wrote:
> 
> Really? The idea that functions should never be longer than 2-3 lines is 
> absurd. Functions should take an input, do one thing (without side-effect) 
> and return a result. And their name should indicate what function they 
> compute. Whether that is 2-3 lines or 20-30 lines depends on the function.
> 
>> On Friday, 28 January 2022 at 10:12:48 UTC-8 Rudolf Martincsek wrote:
>> > 2) Long variable names.
>> 
>> Where I work (not in Go), writing comments is frowned upon. That includes 
>> "docblock" style comments. If a function needs to be documented, it means 
>> the implementation is too complex and must be broken apart to reduce 
>> cyclomatic or whatever perceived complexity. Also uncle bob told us that 
>> functions should never be longer than 2-3 lines of code, so it should be 
>> enough to look at the source code to see what it does. That's the general 
>> sentiment in my team.
>> Comments are considered sign of "un"clean code.
>> 
>> So we use long variable and function names to make the code self 
>> documenting. (Hint: it doesn't)
>> Points 3,4,5 have similar roots, because in dynamic languages it was a trend 
>> many years ago. (ie: hungarian notation)
>> 
>>> On Thursday, December 9, 2021 at 2:10:18 PM UTC+2 Amnon wrote:
>>> 1) Long functions that go on forever and contain long lambdas and 8 levels 
>>> of indentation.
>>> 
>>> 2) Long variable names.
>>> 
>>> 3) Variable names which include the type of the variable.
>>> 
>>> 4) Packages whose name contain the word '/pkg/'
>>> 
>>> 5) Repos which contain the prefix go-
>>> 
>>> 6) Code where almost every line prefixed by `_, _ =`
>>> and the underscores won't go away when you wipe your screen
>>> 
>>> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/4b77acfa-2d30-4d1f-8c4d-34a9c7aa6b40n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/B1592541-BBF1-4DE0-A33A-D09C75481D93%40ix.netcom.com.

Re: [go-nuts] Issues when using time.Ticker microsecond duration

[Robert Engels]

Sorry for typos. Early morning on small phone with no coffee and old person 
eyes. :)

> On Jan 31, 2022, at 7:13 AM, Robert Engels  wrote:
> 
> 
> If you need a “guarantee” in these efforts it is much more complex. You must 
> likely need a real-time OS and interrupt driven code. A general purpose 
> language, os and hardware probably won’t cut it. 
> 
> This of it this way though, if you had 20 cpus to spread the work on, you 
> only need to interrupt every 10 ms - far more achievable. As long as the work 
> is independent. 
> 
>>> On Jan 30, 2022, at 11:10 PM, envee  wrote:
>>> 
>> Thanks Kurtis and Robert.
>> 
>> My use-case is for a telecommunications application (HTTP/2 client, 5G 
>> client to be precise) which needs to send 5G (HTTP/2) requests to a server 
>> at a configurable rate (TPS). 
>> While the telecom industry very commonly use the word TPS (Transactions Per 
>> Second), I would like to control the rate within the second i.e. if the TPS 
>> is to be 2000, then I would like to spread these 2000 transactions (HTTP/2 
>> requests) over 1 second uniformly i.e. 1 request every 500 mico-seconds 
>> which will then result in 2000 requests in that 1 second.
>> I believed that by using a time.Ticker to fire every 500 micro-seconds, I 
>> would be able to achieve this outcome.
>> 
>> Is there any other way you could suggest I do this using Go ? 
>> 
>>> On Monday, 31 January 2022 at 15:17:23 UTC+11 ren...@ix.netcom.com wrote:
>>> Pretty much what Kurtis said. Low interval timers usually require 
>>> specialized constructs if not a real time OS to efficiently (or even at 
>>> all). 
>>> 
>>>>> On Jan 30, 2022, at 9:16 PM, envee  wrote:
>>>>> 
>>>> Thanks, but I am not sure how that reference solves the issue ?
>>> 
>>>> Or are you suggesting that the only way is to use Cgo and invoke usleep to 
>>>> get very close to the Ticker duration specified ?
>>>> 
>>>>> On Monday, 31 January 2022 at 11:25:58 UTC+11 ren...@ix.netcom.com wrote:
>>>>> See https://github.com/golang/go/issues/27707
>>>>> 
>>>>>>> On Jan 30, 2022, at 5:50 PM, envee  wrote:
>>>>>>> 
>>>>>> Hi All,
>>>>> 
>>>>>> I understand this issue has been discussed in the past, and I have seen 
>>>>>> a few comments from Ian and Russ Cox about this topic, but I could not 
>>>>>> arrive at a conclusion about how I can make the time.Ticker send me 
>>>>>> ticks at atleast close to the Ticker duration I specify. At the moment, 
>>>>>> I am seeing ticks being sent at way over the specified duration 
>>>>>> especially when I have sub-millisecond durations. 
>>>>>> With 1ms duration, the ticker seems to be quite close to the duration 
>>>>>> (maybe within +/-5%). I would be happier if it were within 1%, but I can 
>>>>>> handle that.
>>>>>> 
>>>>>> With 1 micro-second duration, the ticker sends ticks nearly 4 times 
>>>>>> slower than expected.
>>>>>> 
>>>>>> Here is my sample code
>>>>>> "
>>>>>> ti := time.NewTicker(1 * time.Millisecond)
>>>>>> start := time.Now()
>>>>>> for i := 0; i < 1000; i++ {
>>>>>> <-ti.C
>>>>>> }
>>>>>> fmt.Printf("elapsed time = %v\n", time.Since(start))
>>>>>> "
>>>>>> 
>>>>>> The output is usually close to 1 second as expected (close to) when 
>>>>>> using 1ms duration.
>>>>>> elapsed time = 1.000159338s
>>>>>> 
>>>>>> 
>>>>>> My code for the microsecond test is :
>>>>>> "
>>>>>> ti := time.NewTicker(1 * time.Microsecond)
>>>>>> start := time.Now()
>>>>>> for i := 0; i < 100; i++ {
>>>>>> <-ti.C
>>>>>> }
>>>>>> fmt.Printf("elapsed time = %v\n", time.Since(start))
>>>>>> "
>>>>>> 
>>>>>> With this, I get the following output which shows the elapsed time close 
>>>>>> to 4 seconds :
>>>>>> elapsed time = 4.796662856s
>>>>>> 
>>>>>> Is there anyway, I can ensure t

Re: [go-nuts] Issues when using time.Ticker microsecond duration

[Robert Engels]

If you need a “guarantee” in these efforts it is much more complex. You must 
likely need a real-time OS and interrupt driven code. A general purpose 
language, os and hardware probably won’t cut it. 

This of it this way though, if you had 20 cpus to spread the work on, you only 
need to interrupt every 10 ms - far more achievable. As long as the work is 
independent. 

> On Jan 30, 2022, at 11:10 PM, envee  wrote:
> 
> Thanks Kurtis and Robert.
> 
> My use-case is for a telecommunications application (HTTP/2 client, 5G client 
> to be precise) which needs to send 5G (HTTP/2) requests to a server at a 
> configurable rate (TPS). 
> While the telecom industry very commonly use the word TPS (Transactions Per 
> Second), I would like to control the rate within the second i.e. if the TPS 
> is to be 2000, then I would like to spread these 2000 transactions (HTTP/2 
> requests) over 1 second uniformly i.e. 1 request every 500 mico-seconds which 
> will then result in 2000 requests in that 1 second.
> I believed that by using a time.Ticker to fire every 500 micro-seconds, I 
> would be able to achieve this outcome.
> 
> Is there any other way you could suggest I do this using Go ? 
> 
>> On Monday, 31 January 2022 at 15:17:23 UTC+11 ren...@ix.netcom.com wrote:
>> Pretty much what Kurtis said. Low interval timers usually require 
>> specialized constructs if not a real time OS to efficiently (or even at 
>> all). 
>> 
 On Jan 30, 2022, at 9:16 PM, envee  wrote:
 
>>> Thanks, but I am not sure how that reference solves the issue ?
>> 
>>> Or are you suggesting that the only way is to use Cgo and invoke usleep to 
>>> get very close to the Ticker duration specified ?
>>> 
 On Monday, 31 January 2022 at 11:25:58 UTC+11 ren...@ix.netcom.com wrote:
 See https://github.com/golang/go/issues/27707
 
>> On Jan 30, 2022, at 5:50 PM, envee  wrote:
>> 
> Hi All,
 
> I understand this issue has been discussed in the past, and I have seen a 
> few comments from Ian and Russ Cox about this topic, but I could not 
> arrive at a conclusion about how I can make the time.Ticker send me ticks 
> at atleast close to the Ticker duration I specify. At the moment, I am 
> seeing ticks being sent at way over the specified duration especially 
> when I have sub-millisecond durations. 
> With 1ms duration, the ticker seems to be quite close to the duration 
> (maybe within +/-5%). I would be happier if it were within 1%, but I can 
> handle that.
> 
> With 1 micro-second duration, the ticker sends ticks nearly 4 times 
> slower than expected.
> 
> Here is my sample code
> "
> ti := time.NewTicker(1 * time.Millisecond)
> start := time.Now()
> for i := 0; i < 1000; i++ {
> <-ti.C
> }
> fmt.Printf("elapsed time = %v\n", time.Since(start))
> "
> 
> The output is usually close to 1 second as expected (close to) when using 
> 1ms duration.
> elapsed time = 1.000159338s
> 
> 
> My code for the microsecond test is :
> "
> ti := time.NewTicker(1 * time.Microsecond)
> start := time.Now()
> for i := 0; i < 100; i++ {
> <-ti.C
> }
> fmt.Printf("elapsed time = %v\n", time.Since(start))
> "
> 
> With this, I get the following output which shows the elapsed time close 
> to 4 seconds :
> elapsed time = 4.796662856s
> 
> Is there anyway, I can ensure ticks are sent at approximately within 5% 
> of my specified duration ?
> 
> I understand from time.Ticker docs that ticks will be lost if the ticker 
> handler cannot keep up with the ticker time.
> 
> In the simple code above, I am literally doing very minimal work in each 
> loop iteration and that is about checking the loop condition.
> So I am not sure why ticks are being lost ?
 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/5f55ccf9-1478-4490-a8f4-a35a5764721dn%40googlegroups.com.
>>> 
>>> -- 
>>> You received this message because you are subscribed to the Google Groups 
>>> "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to golang-nuts...@googlegroups.com.
>> 
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/golang-nuts/efa204c2-092f-46e1-abe4-8aa5c502d986n%40googlegroups.com.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to 

Re: [go-nuts] Issues when using time.Ticker microsecond duration

[Robert Engels]

Pretty much what Kurtis said. Low interval timers usually require specialized 
constructs if not a real time OS to efficiently (or even at all). 

> On Jan 30, 2022, at 9:16 PM, envee  wrote:
> 
> Thanks, but I am not sure how that reference solves the issue ?
> Or are you suggesting that the only way is to use Cgo and invoke usleep to 
> get very close to the Ticker duration specified ?
> 
>> On Monday, 31 January 2022 at 11:25:58 UTC+11 ren...@ix.netcom.com wrote:
>> See https://github.com/golang/go/issues/27707
>> 
 On Jan 30, 2022, at 5:50 PM, envee  wrote:
 
>>> Hi All,
>> 
>>> I understand this issue has been discussed in the past, and I have seen a 
>>> few comments from Ian and Russ Cox about this topic, but I could not arrive 
>>> at a conclusion about how I can make the time.Ticker send me ticks at 
>>> atleast close to the Ticker duration I specify. At the moment, I am seeing 
>>> ticks being sent at way over the specified duration especially when I have 
>>> sub-millisecond durations. 
>>> With 1ms duration, the ticker seems to be quite close to the duration 
>>> (maybe within +/-5%). I would be happier if it were within 1%, but I can 
>>> handle that.
>>> 
>>> With 1 micro-second duration, the ticker sends ticks nearly 4 times slower 
>>> than expected.
>>> 
>>> Here is my sample code
>>> "
>>> ti := time.NewTicker(1 * time.Millisecond)
>>> start := time.Now()
>>> for i := 0; i < 1000; i++ {
>>> <-ti.C
>>> }
>>> fmt.Printf("elapsed time = %v\n", time.Since(start))
>>> "
>>> 
>>> The output is usually close to 1 second as expected (close to) when using 
>>> 1ms duration.
>>> elapsed time = 1.000159338s
>>> 
>>> 
>>> My code for the microsecond test is :
>>> "
>>> ti := time.NewTicker(1 * time.Microsecond)
>>> start := time.Now()
>>> for i := 0; i < 100; i++ {
>>> <-ti.C
>>> }
>>> fmt.Printf("elapsed time = %v\n", time.Since(start))
>>> "
>>> 
>>> With this, I get the following output which shows the elapsed time close to 
>>> 4 seconds :
>>> elapsed time = 4.796662856s
>>> 
>>> Is there anyway, I can ensure ticks are sent at approximately within 5% of 
>>> my specified duration ?
>>> 
>>> I understand from time.Ticker docs that ticks will be lost if the ticker 
>>> handler cannot keep up with the ticker time.
>>> 
>>> In the simple code above, I am literally doing very minimal work in each 
>>> loop iteration and that is about checking the loop condition.
>>> So I am not sure why ticks are being lost ?
>> 
>>> -- 
>>> You received this message because you are subscribed to the Google Groups 
>>> "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to golang-nuts...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/golang-nuts/5f55ccf9-1478-4490-a8f4-a35a5764721dn%40googlegroups.com.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/efa204c2-092f-46e1-abe4-8aa5c502d986n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/9070D322-8F84-4033-8205-DAF6CE0A9D18%40ix.netcom.com.

Re: [go-nuts] Issues when using time.Ticker microsecond duration

[Robert Engels]

See https://github.com/golang/go/issues/27707

> On Jan 30, 2022, at 5:50 PM, envee  wrote:
> 
> Hi All,
> I understand this issue has been discussed in the past, and I have seen a few 
> comments from Ian and Russ Cox about this topic, but I could not arrive at a 
> conclusion about how I can make the time.Ticker send me ticks at atleast 
> close to the Ticker duration I specify. At the moment, I am seeing ticks 
> being sent at way over the specified duration especially when I have 
> sub-millisecond durations. 
> With 1ms duration, the ticker seems to be quite close to the duration (maybe 
> within +/-5%). I would be happier if it were within 1%, but I can handle that.
> 
> With 1 micro-second duration, the ticker sends ticks nearly 4 times slower 
> than expected.
> 
> Here is my sample code
> "
> ti := time.NewTicker(1 * time.Millisecond)
> start := time.Now()
> for i := 0; i < 1000; i++ {
> <-ti.C
> }
> fmt.Printf("elapsed time = %v\n", time.Since(start))
> "
> 
> The output is usually close to 1 second as expected (close to) when using 1ms 
> duration.
> elapsed time = 1.000159338s
> 
> 
> My code for the microsecond test is :
> "
> ti := time.NewTicker(1 * time.Microsecond)
> start := time.Now()
> for i := 0; i < 100; i++ {
> <-ti.C
> }
> fmt.Printf("elapsed time = %v\n", time.Since(start))
> "
> 
> With this, I get the following output which shows the elapsed time close to 4 
> seconds :
> elapsed time = 4.796662856s
> 
> Is there anyway, I can ensure ticks are sent at approximately within 5% of my 
> specified duration ?
> 
> I understand from time.Ticker docs that ticks will be lost if the ticker 
> handler cannot keep up with the ticker time.
> 
> In the simple code above, I am literally doing very minimal work in each loop 
> iteration and that is about checking the loop condition.
> So I am not sure why ticks are being lost ?
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/5f55ccf9-1478-4490-a8f4-a35a5764721dn%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/D5DF77AD-1E68-48AE-883D-9423AA6ABF2F%40ix.netcom.com.

Re: [go-nuts] Why Go allow function with unused arguments?

[Robert Engels]

To the ops point, wouldn’t it be better to cause an error for unused parameters 
unless they use the blank identifier?

> On Jan 30, 2022, at 2:09 PM, 'Dan Kortschak' via golang-nuts 
>  wrote:
> 
> On Sun, 2022-01-30 at 12:01 -0800, Kamil Ziemian wrote:
>> Hello,
>> 
>> This is a question from ignorant in the meters of compilers and
>> mediocre Go users at best, so it may be stupid.
>> 
>> I really like that in Go unused variable or import is compiler time
>> error. As such I wonder why function like
>>> func funTest(x int) int {
>>>return 3
>>> }
>> is allowed? I would guess that it is possible to check if function
>> argument is used in function body or not, so it seem plausible to
>> forbid it.
>> 
>> Again, it maybe a stupid question from ignorant person. I have a lot
>> things to learn about Go, but I'm stuck in learning about Unicode and
>> UFT-8, so I guess it will be a long time before I can go back to
>> learning proper Go. I read FAQ and I didn't remember answer to this
>> question from it. This is my excuse for asking this question.
>> 
>> Best regards,
>> Kamil Ziemian
> 
> It is necessary for methods to sometimes have parameters that are not
> used in order for the type to satisfy an interface. Similarly, if
> function values are being passed around the signatures will need to
> match even if the parameters are not being used.
> 
> It is possible to use the blank identifier in these cases to signal
> that a parameter is not being used and that can be linted for.
> 
> See https://github.com/golang/go/issues/39118 for discussion on a
> proposal relating to this.
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/78c6d5db02f0f35230b60b8324189eb367cee209.camel%40kortschak.io.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/BE2F9AE9-AA7B-4A6C-996C-DB88D5E8C956%40ix.netcom.com.

Re: [go-nuts] Re: Drawing millions of images on a bigger canvas

[Robert Engels]

Many image formats support tiling so you don’t need to have the entire image in 
memory (since a very large image is usually scaled or only a portion used at a 
time). 

> On Jan 24, 2022, at 10:07 AM, Howard C. Shaw III  
> wrote:
> 
> 
> One more options to add to  Tamás' suggestions - due to Go's interface 
> design, implementing a draw.Image, which is an image.Image with an additional 
> Set(x, y int, c color.Color) method, is sufficient to use draw.Draw for 
> compositing.
> 
> As such, you can pre-allocate a flat-file with sufficient space to store 4 x 
> 8 x width x height bytes 
> (4 for RGBA, 8 for float64 in bytes), and have your implementation of 
> draw.Image reading and writing 
> to the file by offset. This would minimize memory usage in exchange for a 
> massive slowdown, as 
> every *pixel* operation would now be a read or write to a file, but only one 
> of the small files needs to be open
> at a time. MMapping this file can get some of the performance back, as Tamás 
> suggested, or you can have the file be on a RAMdisk - this alleviates less of 
> the performance issue, as it still requires filesystem operations per-pixel, 
> but is not as slow as dealing with actual spinning media. 
> 
> You would still at some point need to pull in the entire large file to write 
> it into a usable format, so whether there is any significant advantage 
> depends on whether your large file has an area equal to or larger than the 
> sum of the areas of the individual images - that is, to what degree the 
> images are being composited with overlap.
> 
> 
>  
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/2710cbd7-ed44-4343-830b-1b661d0b70d7n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/C07F4399-2169-4A4B-9856-1D1BCCE926B7%40ix.netcom.com.

Re: [go-nuts] profiling help

[robert engels]

Are you certain that the requests aren’t super short - so all you are measuring 
is the runtime startup time / and the overhead of parsing the CGI “protocol". 
This is not insignificant. CGI really only works well for long requests IMO.

> On Jan 12, 2022, at 5:29 PM, Steve Roth  wrote:
> 
> Thank you, Ian.  From the function names, I suspected that the 70% chain was 
> the garbage collector.  But in this run, garbage collection was turned off 
> with GOGC=off, and that was verified with GODEBUG=gctrace=1.  So I thought 
> perhaps they must be something else.
> 
> I will proceed with heap profiling as you suggest.  (And of course I've 
> reviewed pprof blog post many times.)
> 
> Regards,
> Steve
> 
> 
> On Wed, Jan 12, 2022 at 2:58 PM Ian Lance Taylor  > wrote:
> On Wed, Jan 12, 2022 at 1:54 PM Steve Roth  > wrote:
> >
> > I am attempting to profile my code, and I'm unable to interpret the 
> > results.  So first, the meta-question is, what's the best forum to use to 
> > request help with such things?
> 
> This is a good place.  See also https://golang.org/wiki/Questions 
> .
> 
> 
> > The chain that is taking 70% of the time is
> >
> > gcBgMarkWorker
> >
> > systemstack
> >
> > gcBgMarkWorker.func2
> >
> > gcDrain
> >
> > markroot
> >
> > markroot.func1
> >
> > suspendG
> 
> This is the concurrent garbage collector.
> 
> 
> > The chain that is taking 17% of the time is
> >
> > mcall
> >
> > park_m
> >
> > schedule
> >
> > findrunnable
> >
> > stopm
> >
> > mPark
> >
> > notesleep
> >
> > futexsleep
> >
> > futex
> >
> > Again I have no clue what this is doing.  Any guidance is welcome, 
> > including pointers to other forums better suited for such questions.
> 
> This is the goroutine scheduler.
> 
> If your program is CPU bound (as it should be based on your
> description) then 70% in the garbage collector is a lot.  The next
> step is to collect a heap profile to see where the memory is being
> allocated.  You may want to look at https://go.dev/blog/pprof 
> , old but
> still valid.
> 
> Ian
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAAnpqKGzr4MNfnC4Tqam2NoOt9AJ4NKP_Hiarv5usaYQYG1iNw%40mail.gmail.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CA4E339F-DAA4-4BEC-99B1-AAE34059A6F7%40ix.netcom.com.

Re: [go-nuts] Generic member of recursive struct

[Robert Engels]

This is a task for a parser - and the parser can be type safe -but the 
constraints for something as complex as html aren’t typically  expressed in 
types but rather bnf or similar. 

You can look at the Java html renderer/parser and it uses type safe nodes but 
it still has a resilient parser layer to produce the nodes. 



> On Dec 20, 2021, at 3:06 PM, Michael Ellis  wrote:
> 
> Thanks, Axel & Robert.  As I said in the first post, the package already 
> works well for my purposes (and apparently for the few others who use it).  
> It allows defining web content more concisely than raw HTML and makes the 
> full power of Go available for composing repetitive page structures.
> 
> I was mostly just curious about the possibility of detecting bad content at 
> compile time instead of at run time. In practice, that's turned out not to be 
> a significant problem so I'm ok with learning generics don't support a fix.
> 
>> On Monday, December 20, 2021 at 2:31:32 PM UTC-5 axel.wa...@googlemail.com 
>> wrote:
>> Oh, forgot the footnote:
>> 
>> [1] Note that even that doesn't *actually* work, as `*HtmlNode` would become 
>> a parametric type, so it would have to be instantiated to be used in the 
>> constraint - you'd have to write `type HtmlNode[T string | 
>> *HtmlNode[Something]]`. And the fact that there is no actual answer to what 
>> "Something" would be should be a strong indicator for how much generics are 
>> not what you want for this.
>> 
>>> On Mon, Dec 20, 2021 at 8:29 PM Axel Wagner  
>>> wrote:
>>> 
>>> 
 On Mon, Dec 20, 2021 at 7:07 PM Michael Ellis  wrote:
 >Just to be clear, the way I understood you is that you want HtmlTree.C to 
 >be a slice which has elements which can each either be a string or an 
 >*HtmlTree - i.e. you want these to be mixed in a single slice. Correct?
 
 Actually, no.  An HtmlTree instance whose content is a string is a 
 terminal node.  The recursion in the Render() func looks like:
 
   for _, c := range h.C {
 switch c := c.(type) {
 case string:
 b.WriteString(c)
 case *HtmlTree:
 err = Render(c, b, rindent)
 if err != nil {
 return fmt.Errorf("%s : %v", h.T, err)
 }
 default:
 return fmt.Errorf("Bad content %v. Can't render 
 type %T! ", h.C, c)
 }
 }
>>> 
>>> That's pretty much what I meant, yes. You want a sum type (and in lieu of 
>>> sum types, an interface), not generics. 
>>> The "union" aspect of Go generics (the `a | b` syntax) only applies to 
>>> using interfaces as *constraints*, not as *types* - you want to do the 
>>> latter. You want to have a field which is either a string or an *HtmlNode - 
>>> but Go only gives you a convenient way to write the two distinct types 
>>> `type HtmlNodeString struct { /*…*/ C string }` and `type HtmlNodeNode 
>>> struct { /* … */ C *HtmlNode }` into one¹ type declaration and write 
>>> function which can work on either. It's just not what you want.
>>> 
>>> The best option for you is (as Robert mentions) to use an interface, which 
>>> is an "open sum" - meaning you can't have the guarantee that its dynamic 
>>> type is of a limited set of options, but you *can* have a value whose 
>>> actual type is dynamic. You can look at e.g. ast.Node, which is de-facto 
>>> the same thing - it's supposed to be a limited set of options, but in lieu 
>>> of sum types, it's an interface.
>>> 
>>> Go might, at some point, allow the union-elements of constraint interfaces 
>>> to be used as actual types, which *would* be a form of sum types. But 
>>> probably not for a while - it's more complicated than it may seem.
>>> 
 
 
 I suppose it's still a problem, though, since the compiler doesn't have 
 any way of knowing that's how trees of HtmlTree meant to be constructed. I 
 should have expressed the intended constraint on HtmlTree.C as `string | 
 []*HtmlTree`.  Does that make a difference?
> On Monday, December 20, 2021 at 10:25:26 AM UTC-5 
> axel.wa...@googlemail.com wrote:
> Just to be clear, the way I understood you is that you want HtmlTree.C to 
> be a slice which has elements which can each either be a string or an 
> *HtmlTree - i.e. you wan these to be mixed in a single slice. Correct?
> Because that is not a use case for generics, it's a use case for sum 
> types (which Go does not have).
> 
>> On Mon, Dec 20, 2021 at 4:11 PM Michael Ellis  
>> wrote:
>> > They can't, sorry.
>> Ok. Thanks, Axel. 
>> Saves me wasting more time.  In the past 3 years of using Go, this is 
>> the only use case where I've  really wanted generics (other cases I've 
>> encountered so far are easily handled with code generation). 
> 

Re: [go-nuts] Generic member of recursive struct

[Robert Engels]

You create structs like StringNode and HtmlNode that implement the common 
needed operations. 

Imagine a Render() method for a string you convert to html, for an html node 
you render recursively. 

Or similar. If an HtmlNode is not a leaf node it needs recursive operations. 

> On Dec 20, 2021, at 12:52 PM, Michael Ellis  wrote:
> 
> 
> 
>> On Monday, December 20, 2021 at 1:33:49 PM UTC-5 ren...@ix.netcom.com wrote:
>> You should use interfaces and a “node” type.
> 
> Hmm, I tried 
> 
> type Node interface {
> string | []*HtmlTree
> }
> 
> type HtmlTree struct {
> T string // html tagname, e.g. 'head'
> A string // zero or more html attributes, e.g 'id=1 class="foo"'
> C Node   // a slice of content whose elements may be strings or 
> *HtmlTree
> empty bool   // set to true for empty tags like 
> }
> 
> and the compiler said: 
> ./prog.go:21:8: interface contains type constraints.
> 
> (line 21 is the declaration of HtmlTree.C as type Node)
> 
> What's the syntax you have in mind?
> 
> 
>  
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/c619d54d-5a69-4828-b10b-771722f8f646n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/88DB1537-6360-4D34-96BC-C9F2CEB01C0A%40ix.netcom.com.

Re: [go-nuts] Generic member of recursive struct

[Robert Engels]

You should use interfaces and a “node” type. 

> On Dec 20, 2021, at 12:07 PM, Michael Ellis  wrote:
> 
> >Just to be clear, the way I understood you is that you want HtmlTree.C to 
> be a slice which has elements which can each either be a string or an 
> *HtmlTree - i.e. you want these to be mixed in a single slice. Correct?
> 
> Actually, no.  An HtmlTree instance whose content is a string is a terminal 
> node.  The recursion in the Render() func looks like:
> 
>   for _, c := range h.C {
> switch c := c.(type) {
> case string:
> b.WriteString(c)
> case *HtmlTree:
> err = Render(c, b, rindent)
> if err != nil {
> return fmt.Errorf("%s : %v", h.T, err)
> }
> default:
> return fmt.Errorf("Bad content %v. Can't render type 
> %T! ", h.C, c)
> }
> }
> 
> I suppose it's still a problem, though, since the compiler doesn't have any 
> way of knowing that's how trees of HtmlTree meant to be constructed. I should 
> have expressed the intended constraint on HtmlTree.C as `string | 
> []*HtmlTree`.  Does that make a difference?
>> On Monday, December 20, 2021 at 10:25:26 AM UTC-5 axel.wa...@googlemail.com 
>> wrote:
>> Just to be clear, the way I understood you is that you want HtmlTree.C to be 
>> a slice which has elements which can each either be a string or an *HtmlTree 
>> - i.e. you wan these to be mixed in a single slice. Correct?
>> Because that is not a use case for generics, it's a use case for sum types 
>> (which Go does not have).
>> 
>>> On Mon, Dec 20, 2021 at 4:11 PM Michael Ellis  wrote:
>>> > They can't, sorry.
>>> Ok. Thanks, Axel. 
>>> Saves me wasting more time.  In the past 3 years of using Go, this is the 
>>> only use case where I've  really wanted generics (other cases I've 
>>> encountered so far are easily handled with code generation). 
>> 
>>> -- 
>> 
>>> You received this message because you are subscribed to the Google Groups 
>>> "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to golang-nuts...@googlegroups.com.
>> 
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/golang-nuts/ac75dc32-733d-4a73-9735-619c33cb4cd4n%40googlegroups.com.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/2de22684-0263-465c-8ac0-ff288c535fb7n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/9ABD0D98-6041-4A45-A2C3-D924E298BF16%40ix.netcom.com.

Re: [go-nuts] Why, oh why, do people do this? Things that annoy me in Go code.

[Robert Engels]

5 is a best practice when you having multiple implementations written in 
different languages - especially when developed independently. 

> On Dec 9, 2021, at 6:10 AM, Amnon  wrote:
> 
> 1) Long functions that go on forever and contain long lambdas and 8 levels 
> of indentation.
> 
> 2) Long variable names.
> 
> 3) Variable names which include the type of the variable.
> 
> 4) Packages whose name contain the word '/pkg/'
> 
> 5) Repos which contain the prefix go-
> 
> 6) Code where almost every line prefixed by `_, _ =`
> and the underscores won't go away when you wipe your screen
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/d367f41a-53e8-415e-a44c-1ace051359d3n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/4F4B75FD-3266-4239-851A-01BEC1F98C82%40ix.netcom.com.

Re: [go-nuts] Go http/2 implementation is x5 less per performant than http/1.1

[robert engels]

Correct. If you refer to the code in the OP you will see that different 
transports are created in order to test http1 vs http2.

If you wish to use the default behavior to use http2 you need to rebuild the 
stdlib (at least as far as I understand the issue with h2_bundle).

My patches only affect the client not the server, BUT, I would assume similar 
changes are needed on the server side to support highly efficient large uploads.

As an aside, the whole issue the circular dependency and http1/http2 highlights 
a the need to use more interfaced based design. If that was done here and the 
interfaces segregated from the implementations - there would of been no need 
for h2_bundle. There is a reason Java allows both circular dependencies (to a 
point) and uses factories - it is often a far similar pattern.

> On Nov 19, 2021, at 12:04 PM, Jean-François Giorgi  wrote:
> 
> I already use the replace directive for but this doesn't for the net/http 
> h2_bundle thing. see https://www.youtube.com/watch?v=FARQMJndUn0=814s 
> <https://www.youtube.com/watch?v=FARQMJndUn0=814s>
> 
> 
> Le ven. 19 nov. 2021 à 18:33, Robert Engels  <mailto:reng...@ix.netcom.com>> a écrit :
> 
> Use the replace directive to point the net package to your local copy. 
> 
> Much easier to test X changes than stdlib changes. 
> 
>> On Nov 19, 2021, at 10:58 AM, Kirth Gersen > <mailto:kirthal...@gmail.com>> wrote:
>> 
>> Your CL works well with the POC.
>> 
>> Side question not specific to this issue: how to test changes to 
>> golang.org/x/net <http://golang.org/x/net> with net/http ?
>> The 'h2_bundle' trick with go generate & bundle requires to fork the std lib 
>> too ? 
>> I have a hard time figuring how to do this. I tried with gotip but I get an 
>> error with "gotip generate" (bundle: internal error: package "strings" 
>> without types was imported from "golang.org/x/net/http2 
>> <http://golang.org/x/net/http2>")
>> Any doc/tutorial on how to deal with this 'bundle' trick ?
>> 
>> thx
>> Le lundi 15 novembre 2021 à 17:32:48 UTC+1, ren...@ix.netcom.com 
>> <mailto:ren...@ix.netcom.com> a écrit :
>> Since http2 multiplexes streams it will delicately affect latency on other 
>> streams. This is why I suggested using multiple transports - one for high 
>> throughput transfers and another for lower latency “interactive” sessions. 
>> 
>>> On Nov 15, 2021, at 9:23 AM, Kevin Chowski > wrote:
>>> 
>>> These are interesting results, thanks for investigating and sharing 
>>> results!
>> 
>>> 
>>> I see that you have mostly been focusing on throughput in your posts, have 
>>> you done testing for latency differences too?
>>> 
>>> On Saturday, November 13, 2021 at 6:11:40 PM UTC-7 ren...@ix.netcom.com <> 
>>> wrote:
>>> As another data point, I decided to test a few implementations of http2 
>>> downloads on OSX.
>>> 
>>> Using a Go server with default frame size (16k):
>>> 
>>> Go client:  900 MB/s
>>> Java client: 1300 MB/s
>>> curl: 1500 MB/s
>>> 
>>> Using a Java server with default frame size (16k):
>>> 
>>> Go client: 670 MB/s
>>> Java client: 720 MB/s
>>> curl: 800 M/s
>>> 
>>> Using Go server using 256k client max frame size:
>>> 
>>> Go client: 2350 MB/s
>>> Java client: 2800 MB/s
>>> h2load: 4300 MB/s
>>> 
>>> Using Java server using 256k client max frame size:
>>> 
>>> Go client: 2900 MB/s
>>> Java client: 2800 MB/s
>>> h2load: 3750 MB/s
>>> 
>>> For h2load, I needed to create a PR to allow the frame size to be set, see 
>>> https://github.com/nghttp2/nghttp2/pull/1640 
>>> <https://github.com/nghttp2/nghttp2/pull/1640>
>>> 
>>> 
>>>> On Nov 10, 2021, at 7:04 PM, robert engels > wrote:
>>>> 
>>>> No worries. I updated the issue and the CL. I will comment in the CL with 
>>>> a few more details.
>>>> 
>>>>> On Nov 10, 2021, at 2:30 PM, Andrey T. > 
>>>>> wrote:
>>>>> 
>>>>> Thank you Robert, 
>>>>> I somehow missed the reference to the ticket in the first message, sorry 
>>>>> about that.
>>>>> 
>>>>> As for the CL - I think adding link to the github issue, and add a bit of 
>>>>> explanation in a commit message would help.
>>>>> I added link to your CL to the github issue's discussion, 

Re: [go-nuts] Go http/2 implementation is x5 less per performant than http/1.1

[Robert Engels]

Use the replace directive to point the net package to your local copy. 

Much easier to test X changes than stdlib changes. 

> On Nov 19, 2021, at 10:58 AM, Kirth Gersen  wrote:
> 
> Your CL works well with the POC.
> 
> Side question not specific to this issue: how to test changes to 
> golang.org/x/net with net/http ?
> The 'h2_bundle' trick with go generate & bundle requires to fork the std lib 
> too ? 
> I have a hard time figuring how to do this. I tried with gotip but I get an 
> error with "gotip generate" (bundle: internal error: package "strings" 
> without types was imported from "golang.org/x/net/http2")
> Any doc/tutorial on how to deal with this 'bundle' trick ?
> 
> thx
>> Le lundi 15 novembre 2021 à 17:32:48 UTC+1, ren...@ix.netcom.com a écrit :
>> Since http2 multiplexes streams it will delicately affect latency on other 
>> streams. This is why I suggested using multiple transports - one for high 
>> throughput transfers and another for lower latency “interactive” sessions. 
>> 
>>>> On Nov 15, 2021, at 9:23 AM, Kevin Chowski  wrote:
>>>> 
>>> These are interesting results, thanks for investigating and sharing 
>>> results!
>> 
>>> 
>>> I see that you have mostly been focusing on throughput in your posts, have 
>>> you done testing for latency differences too?
>>> 
>>>> On Saturday, November 13, 2021 at 6:11:40 PM UTC-7 ren...@ix.netcom.com 
>>>> wrote:
>>>> As another data point, I decided to test a few implementations of http2 
>>>> downloads on OSX.
>>>> 
>>>> Using a Go server with default frame size (16k):
>>>> 
>>>> Go client:  900 MB/s
>>>> Java client: 1300 MB/s
>>>> curl: 1500 MB/s
>>>> 
>>>> Using a Java server with default frame size (16k):
>>>> 
>>>> Go client: 670 MB/s
>>>> Java client: 720 MB/s
>>>> curl: 800 M/s
>>>> 
>>>> Using Go server using 256k client max frame size:
>>>> 
>>>> Go client: 2350 MB/s
>>>> Java client: 2800 MB/s
>>>> h2load: 4300 MB/s
>>>> 
>>>> Using Java server using 256k client max frame size:
>>>> 
>>>> Go client: 2900 MB/s
>>>> Java client: 2800 MB/s
>>>> h2load: 3750 MB/s
>>>> 
>>>> For h2load, I needed to create a PR to allow the frame size to be set, see 
>>>> https://github.com/nghttp2/nghttp2/pull/1640
>>>> 
>>>> 
>>>>> On Nov 10, 2021, at 7:04 PM, robert engels  wrote:
>>>>> 
>>>>> No worries. I updated the issue and the CL. I will comment in the CL with 
>>>>> a few more details.
>>>>> 
>>>>>> On Nov 10, 2021, at 2:30 PM, Andrey T.  wrote:
>>>>>> 
>>>>>> Thank you Robert, 
>>>>>> I somehow missed the reference to the ticket in the first message, sorry 
>>>>>> about that.
>>>>>> 
>>>>>> As for the CL - I think adding link to the github issue, and add a bit 
>>>>>> of explanation in a commit message would help.
>>>>>> I added link to your CL to the github issue's discussion, hopefully it 
>>>>>> will bring more attention to it. 
>>>>>> 
>>>>>> A.
>>>>>> 
>>>>>>> On Wednesday, November 10, 2021 at 1:22:42 PM UTC-7 
>>>>>>> ren...@ix.netcom.com wrote:
>>>>>>> As reported in the OP, the issue was filed long ago 
>>>>>>> https://github.com/golang/go/issues/47840
>>>>>>> 
>>>>>>> My CL https://go-review.googlesource.com/c/net/+/362834 is a viable fix 
>>>>>>> (and should of been supported originally).
>>>>>>> 
>>>>>>>>> On Nov 10, 2021, at 12:59 PM, Andrey T.  
>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>> Fellas, 
>>>>>>>> I would say the 5x throughput difference is a serious problem.Would 
>>>>>>>> you be kind and open an issue on github about it? 
>>>>>>>> Also, the PR that you have might benefit from explanation about what 
>>>>>>>> you are trying to solve (and probably link to an issue on github), so 
>>>>>>>> it would get more attention. 
>>>>>>>> 
>

Re: [go-nuts] atomic.Value : inconsistent types

[Robert Engels]

I believe your code is broken. You need to use a lock when checking and making 
the switch (see double locking problem) or there would be no need to use 
atomics. 

If you run it under the race detector I am guessing it will fail. 

> On Nov 16, 2021, at 6:25 AM, Stephen Illingworth 
>  wrote:
> 
> 
> When using atomic.Values it is important that the type being stored is 
> consistent. Trying to store a different type in an atomic.Value will cause a 
> panic
> 
> panic: sync/atomic: store of inconsistently typed value into Value
> 
> I've found that the way around this is to create a new instance of 
> atomic.Value whenever I have reason to believe that the type to be stored has 
> changed. In my program this can happen because I am storing an interface in 
> the atomic.Value and the implementation of the interface may change.
> 
> Choosing the moment to create the new atomic.Value however can be tricky so I 
> am now trying the following method:
> 
> 1) Check the type of the existing stored value and the type of the new value
> 2) If they are the same then atomic.Value.Store() can be used
> 3) If they are not the same, then create a new instance of atomic.Value and 
> store new value in that
> 4) Give the new atomic.Value the name of the old atomic.Value
> 
> This method means I no longer have to worry about when I create a new 
> instance of atomic.Value - the new instance is created when the type check 
> fails.
> 
> Example code.
> 
> https://play.golang.org/p/1gmI5eMdOcl
> 
> Now, this obviously requires knowledge of how interface{} is represented 
> internally, which isn't great, but in principle is this a correctly working 
> solution? (It seems to be to me but that doesn't really mean anything)
> 
> Does returning an atomic.Value from a function count as "copying" after first 
> use?
> 
> Is this going to bite me in the future?
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/8893475a-ec61-4f78-9dc0-b80dda9e675an%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/3B7F9D8A-7C7F-43BD-93A9-60A9BE9A2E27%40ix.netcom.com.

Re: [go-nuts] Amateur question: when you should use runes?

[Robert Engels]

When your string contains Unicode characters dealing with it as individual 
bytes is difficult. When using runes with range - each index is a “Unicode 
character” (which may be multiple bytes) - which is easy to use. 

See go.dev/blog/strings



> On Nov 15, 2021, at 12:00 PM, Kamil Ziemian  wrote:
> 
> Hello,
> 
> I read quite a few blog posts, articles, listen to nice number to talks about 
> strings, runes and encoding in Go. I now reading Go Language Spec and I just 
> stuck in the section about runes. I mean, it isn't hard as itself, but it 
> raises to much questions to me. I decided that I need to learn more about 
> Unicode and UTF-8, so from today I'm reading Unicode Technical Site (?), 
> currently the Glossary (https://www.unicode.org/glossary/). But I can't 
> understand one thing: when in practice you should use runes?
> 
> My understanding at this moment is like that. Unicode assign every symbol a 
> number (at this moment I disregard normalization and any other more advance 
> stuff), rune is alias for int32 that stores integer representation of this 
> number. UTF-8 is variable size encoding using one or more bytes to encode 
> symbol and shouldn't and DOESN'T represent integer value of symbols Unicode 
> number. Virtues of UTF-8 are clear as how it allows to save a space is clear 
> to me, but I can't find a reason why I should transform my text to runes? In 
> Go stdlib there is a RuneReader interface (?) so this reason must exists, but 
> I just can't find anything. Maybe it have something to do with sending 
> information using internet? I don't know, this is totally outside my humble 
> knowledge.
> 
> You can say, that since I don't see a reason to use runes, I probably 
> shouldn't care about it. This is a valid point, but I want to know Go 
> reasonable well and constantly find code with runes which reason of existence 
> I don't understand (e.g. functions in stdlib that operates on runes) is quite 
> demoralising to me.
> 
> Best
> Kamil
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/f3dad0e1-cd25-4e33-a7f2-34e0118bf68an%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/649D31F8-C9DB-4C03-A306-D18FF2C8E954%40ix.netcom.com.

Re: [go-nuts] Go http/2 implementation is x5 less per performant than http/1.1

[Robert Engels]

Since http2 multiplexes streams it will delicately affect latency on other 
streams. This is why I suggested using multiple transports - one for high 
throughput transfers and another for lower latency “interactive” sessions. 

> On Nov 15, 2021, at 9:23 AM, Kevin Chowski  wrote:
> 
> These are interesting results, thanks for investigating and sharing results!
> 
> I see that you have mostly been focusing on throughput in your posts, have 
> you done testing for latency differences too?
> 
>> On Saturday, November 13, 2021 at 6:11:40 PM UTC-7 ren...@ix.netcom.com 
>> wrote:
>> As another data point, I decided to test a few implementations of http2 
>> downloads on OSX.
>> 
>> Using a Go server with default frame size (16k):
>> 
>> Go client:  900 MB/s
>> Java client: 1300 MB/s
>> curl: 1500 MB/s
>> 
>> Using a Java server with default frame size (16k):
>> 
>> Go client: 670 MB/s
>> Java client: 720 MB/s
>> curl: 800 M/s
>> 
>> Using Go server using 256k client max frame size:
>> 
>> Go client: 2350 MB/s
>> Java client: 2800 MB/s
>> h2load: 4300 MB/s
>> 
>> Using Java server using 256k client max frame size:
>> 
>> Go client: 2900 MB/s
>> Java client: 2800 MB/s
>> h2load: 3750 MB/s
>> 
>> For h2load, I needed to create a PR to allow the frame size to be set, see 
>> https://github.com/nghttp2/nghttp2/pull/1640
>> 
>> 
>>> On Nov 10, 2021, at 7:04 PM, robert engels  wrote:
>>> 
>>> No worries. I updated the issue and the CL. I will comment in the CL with a 
>>> few more details.
>>> 
>>>> On Nov 10, 2021, at 2:30 PM, Andrey T.  wrote:
>>>> 
>>>> Thank you Robert, 
>>>> I somehow missed the reference to the ticket in the first message, sorry 
>>>> about that.
>>>> 
>>>> As for the CL - I think adding link to the github issue, and add a bit of 
>>>> explanation in a commit message would help.
>>>> I added link to your CL to the github issue's discussion, hopefully it 
>>>> will bring more attention to it. 
>>>> 
>>>> A.
>>>> 
>>>>> On Wednesday, November 10, 2021 at 1:22:42 PM UTC-7 ren...@ix.netcom.com 
>>>>> wrote:
>>>>> As reported in the OP, the issue was filed long ago 
>>>>> https://github.com/golang/go/issues/47840
>>>>> 
>>>>> My CL https://go-review.googlesource.com/c/net/+/362834 is a viable fix 
>>>>> (and should of been supported originally).
>>>>> 
>>>>>>> On Nov 10, 2021, at 12:59 PM, Andrey T.  
>>>>>>> wrote:
>>>>>>> 
>>>>>> Fellas, 
>>>>>> I would say the 5x throughput difference is a serious problem.Would you 
>>>>>> be kind and open an issue on github about it? 
>>>>>> Also, the PR that you have might benefit from explanation about what you 
>>>>>> are trying to solve (and probably link to an issue on github), so it 
>>>>>> would get more attention. 
>>>>>> 
>>>>>> Thanks!
>>>>>> 
>>>>>> Andrey
>>>>>> 
>>>>>> 
>>>>>>> On Tuesday, November 9, 2021 at 4:50:34 PM UTC-7 ren...@ix.netcom.com 
>>>>>>> wrote:
>>>>>>> Well, I figured out a way to do it simply. The CL is here 
>>>>>>> https://go-review.googlesource.com/c/net/+/362834
>>>>>>> 
>>>>>>> The frame size will be used for all connections using that transport, 
>>>>>>> so it is probably better to create a transport specifically for the 
>>>>>>> high-throughput transfers.
>>>>>>> 
>>>>>>> You can also create perform single shot requests like:
>>>>>>> 
>>>>>>> if useH2C {
>>>>>>>rt = {
>>>>>>>   AllowHTTP: true,
>>>>>>>   DialTLS: func(network, addr string, cfg *tls.Config) (net.Conn, 
>>>>>>> error) {
>>>>>>>  return dialer.Dial(network, addr)
>>>>>>>   },
>>>>>>>   MaxFrameSize: 1024*256,
>>>>>>>}
>>>>>>> }
>>>>>>> 
>>>>>>> var body io.ReadCloser = http.NoBody
>>>>>>> 
>>>>>>

Re: [go-nuts] Go http/2 implementation is x5 less per performant than http/1.1

[robert engels]

As another data point, I decided to test a few implementations of http2 
downloads on OSX.

Using a Go server with default frame size (16k):

Go client:  900 MB/s
Java client: 1300 MB/s
curl: 1500 MB/s

Using a Java server with default frame size (16k):

Go client: 670 MB/s
Java client: 720 MB/s
curl: 800 M/s

Using Go server using 256k client max frame size:

Go client: 2350 MB/s
Java client: 2800 MB/s
h2load: 4300 MB/s

Using Java server using 256k client max frame size:

Go client: 2900 MB/s
Java client: 2800 MB/s
h2load: 3750 MB/s

For h2load, I needed to create a PR to allow the frame size to be set, see 
https://github.com/nghttp2/nghttp2/pull/1640


> On Nov 10, 2021, at 7:04 PM, robert engels  wrote:
> 
> No worries. I updated the issue and the CL. I will comment in the CL with a 
> few more details.
> 
>> On Nov 10, 2021, at 2:30 PM, Andrey T. > <mailto:xnow4fippy...@sneakemail.com>> wrote:
>> 
>> Thank you Robert, 
>> I somehow missed the reference to the ticket in the first message, sorry 
>> about that.
>> 
>> As for the CL - I think adding link to the github issue, and add a bit of 
>> explanation in a commit message would help.
>> I added link to your CL to the github issue's discussion, hopefully it will 
>> bring more attention to it. 
>> 
>> A.
>> 
>> On Wednesday, November 10, 2021 at 1:22:42 PM UTC-7 ren...@ix.netcom.com 
>> <http://ix.netcom.com/> wrote:
>> As reported in the OP, the issue was filed long ago 
>> https://github.com/golang/go/issues/47840 
>> <https://github.com/golang/go/issues/47840>
>> 
>> My CL https://go-review.googlesource.com/c/net/+/362834 
>> <https://go-review.googlesource.com/c/net/+/362834> is a viable fix (and 
>> should of been supported originally).
>> 
>> 
>>> On Nov 10, 2021, at 12:59 PM, Andrey T. >> > wrote:
>>> 
>> 
>>> Fellas, 
>>> I would say the 5x throughput difference is a serious problem.Would you be 
>>> kind and open an issue on github about it? 
>>> Also, the PR that you have might benefit from explanation about what you 
>>> are trying to solve (and probably link to an issue on github), so it would 
>>> get more attention. 
>>> 
>>> Thanks!
>>> 
>>> Andrey
>>> 
>>> 
>>> On Tuesday, November 9, 2021 at 4:50:34 PM UTC-7 ren...@ix.netcom.com 
>>> <http://ix.netcom.com/> wrote:
>>> Well, I figured out a way to do it simply. The CL is here 
>>> https://go-review.googlesource.com/c/net/+/362834 
>>> <https://go-review.googlesource.com/c/net/+/362834>
>>> 
>>> The frame size will be used for all connections using that transport, so it 
>>> is probably better to create a transport specifically for the 
>>> high-throughput transfers.
>>> 
>>> You can also create perform single shot requests like:
>>> 
>>> if useH2C {
>>>rt = {
>>>   AllowHTTP: true,
>>>   DialTLS: func(network, addr string, cfg *tls.Config) (net.Conn, 
>>> error) {
>>>  return dialer.Dial(network, addr)
>>>   },
>>>   MaxFrameSize: 1024*256,
>>>}
>>> }
>>> 
>>> var body io.ReadCloser = http.NoBody
>>> 
>>> req, err := http.NewRequestWithContext(ctx, "GET", url, body)
>>> if err != nil {
>>>return err
>>> }
>>> 
>>> resp, err := rt.RoundTrip(req)
>>> 
>>> 
>>>> On Nov 9, 2021, at 3:31 PM, Robert Engels > wrote:
>>>> 
>>>> To be clear, I have no plans to submit a Cl to improve this at this time. 
>>>> 
>>>> It would require some api changes to implement properly. 
>>>> 
>>>>> On Nov 9, 2021, at 12:19 PM, Kirth Gersen > wrote:
>>>>> 
>>>>> Great !
>>>>> 
>>>>> > I made some local mods to the net library, increasing the frame size to 
>>>>> > 256k, and the http2 performance went from 8Gbps to 38Gbps.
>>>>> That is already enormous for us. thx for finding this.
>>>>> 
>>>>> 4 -> Indeed  a lot of WINDOW_UPDATE messages are visible when using 
>>>>> GODEBUG=http2debug=1 
>>>>> 
>>>>> 
>>>>> On Tuesday, November 9, 2021 at 6:28:16 PM UTC+1 ren...@ix.netcom.com 
>>>>> <http://ix.netcom.com/> wrote:
>>>>> I did a review of the codebase.
>>>>> 
>>>>> Http2 is a multiplexed 

Re: [go-nuts] darwin syscall performance?

[robert engels]

Sorry, I meant to send this on the http2 thread - and I also sent prematurely :(

> On Nov 13, 2021, at 2:11 PM, robert engels  wrote:
> 
> As another data point, I decided to test a few implementations of http2 
> downloads on OSX.
> 
> Using a Go server with default frame size (16k):
> 
> Go client:  900 M/s
> Java client: 1300 M/s
> curl: 1500 M/s
> 
> Using a Java server with default frame size (16k):
> Go client: 670 M/s
> Java client: 720 M/s
> curl: 800 M/s
> 
> Using Go server with 256k client max frame size:
> Go client: 2350 M/s
> Java client: 2800 M/s
> curl: unable to set option
> 
> Using Java server with 256k client max frame size:
> Go client: 2900 M/s
> Java client: 2800 M/s
> curl: unable to set option
> 
> 
> Clearly using a Go based server is the way to go :)
> 
> But there also seems to be an opportunity to improve the Go http2 client 
> performance.
> 
> 
>> On Nov 10, 2021, at 8:33 PM, Ian Lance Taylor > <mailto:i...@golang.org>> wrote:
>> 
>> On Wed, Nov 10, 2021 at 5:29 PM robert engels > <mailto:reng...@ix.netcom.com>> wrote:
>> Hi,
>> 
>> While investigating the http2 issue, I saw this in the profiler: (sorry for 
>> the screenshot)
>> 
>> 
>> 
>> and looking at the code:
>> 
>> func syscall_syscall(fn, a1, a2, a3 uintptr) (r1, r2, err uintptr) {
>>entersyscall()
>>libcCall(unsafe.Pointer(abi.FuncPCABI0(syscall)), unsafe.Pointer())
>>exitsyscall()
>>return
>> }
>> 
>> This is implies that the runtime overhead to make the system call - in 
>> entersyscall() and exitsyscal() - is more than 10x the cost of doing the 
>> actual call.
>> 
>> Am I reading this right? If so, is there any outstanding issues to try and 
>> optimize this?
>> 
>> Btw, the call being performed is a simple write to a socket.
>> 
>> I'm not sure I trust the profiling here: it's fairly likely that the time 
>> required by the system call itself is not being correctly attributed.  But 
>> the general idea may be true: it does take time to record in the scheduler 
>> that we are entering a system call.  This is also why cgo calls are slow.  
>> And, in fact, on macOS system calls are essentially cgo calls, because macOS 
>> requires us to go through the C based libSystem rather than simply executing 
>> a SYSCALL or SVC instruction.
>> 
>> I don't know offhand whether there are any open issues for speeding up these 
>> calls, and I couldn't find any in a quick search.  People who work on the Go 
>> runtime are well aware of the problem.
>> 
>> Ian
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts+unsubscr...@googlegroups.com 
>> <mailto:golang-nuts+unsubscr...@googlegroups.com>.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/CAOyqgcVJqwMttzrE3GXeQf%2BC%3DKVtGFz%3DaXv%3DNXxaunWngE9L6Q%40mail.gmail.com
>>  
>> <https://groups.google.com/d/msgid/golang-nuts/CAOyqgcVJqwMttzrE3GXeQf%2BC%3DKVtGFz%3DaXv%3DNXxaunWngE9L6Q%40mail.gmail.com?utm_medium=email_source=footer>.
> 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/5A864FCE-9769-44C4-BE1C-E40D33BE8467%40ix.netcom.com.

Re: [go-nuts] darwin syscall performance?

[robert engels]

As another data point, I decided to test a few implementations of http2 
downloads on OSX.

Using a Go server with default frame size (16k):

Go client:  900 M/s
Java client: 1300 M/s
curl: 1500 M/s

Using a Java server with default frame size (16k):
Go client: 670 M/s
Java client: 720 M/s
curl: 800 M/s

Using Go server with 256k client max frame size:
Go client: 2350 M/s
Java client: 2800 M/s
curl: unable to set option

Using Java server with 256k client max frame size:
Go client: 2900 M/s
Java client: 2800 M/s
curl: unable to set option


Clearly using a Go based server is the way to go :)

But there also seems to be an opportunity to improve the Go http2 client 
performance.


> On Nov 10, 2021, at 8:33 PM, Ian Lance Taylor  wrote:
> 
> On Wed, Nov 10, 2021 at 5:29 PM robert engels  <mailto:reng...@ix.netcom.com>> wrote:
> Hi,
> 
> While investigating the http2 issue, I saw this in the profiler: (sorry for 
> the screenshot)
> 
> 
> 
> and looking at the code:
> 
> func syscall_syscall(fn, a1, a2, a3 uintptr) (r1, r2, err uintptr) {
>entersyscall()
>libcCall(unsafe.Pointer(abi.FuncPCABI0(syscall)), unsafe.Pointer())
>exitsyscall()
>return
> }
> 
> This is implies that the runtime overhead to make the system call - in 
> entersyscall() and exitsyscal() - is more than 10x the cost of doing the 
> actual call.
> 
> Am I reading this right? If so, is there any outstanding issues to try and 
> optimize this?
> 
> Btw, the call being performed is a simple write to a socket.
> 
> I'm not sure I trust the profiling here: it's fairly likely that the time 
> required by the system call itself is not being correctly attributed.  But 
> the general idea may be true: it does take time to record in the scheduler 
> that we are entering a system call.  This is also why cgo calls are slow.  
> And, in fact, on macOS system calls are essentially cgo calls, because macOS 
> requires us to go through the C based libSystem rather than simply executing 
> a SYSCALL or SVC instruction.
> 
> I don't know offhand whether there are any open issues for speeding up these 
> calls, and I couldn't find any in a quick search.  People who work on the Go 
> runtime are well aware of the problem.
> 
> Ian
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com 
> <mailto:golang-nuts+unsubscr...@googlegroups.com>.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAOyqgcVJqwMttzrE3GXeQf%2BC%3DKVtGFz%3DaXv%3DNXxaunWngE9L6Q%40mail.gmail.com
>  
> <https://groups.google.com/d/msgid/golang-nuts/CAOyqgcVJqwMttzrE3GXeQf%2BC%3DKVtGFz%3DaXv%3DNXxaunWngE9L6Q%40mail.gmail.com?utm_medium=email_source=footer>.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/73CB6AC5-B9B7-4421-AD07-E54DB3E4B003%40ix.netcom.com.

Re: [go-nuts] Is this a known problem?

[Robert Engels]

I think all of this will probably be handled with wrapper collections types and 
the iterator pattern. 

Generics should make this easy. It would be great is range supported this but 
it seems doubtful. 

> On Nov 12, 2021, at 10:37 AM, 'Axel Wagner' via golang-nuts 
>  wrote:
> 
> 
> It is still a different operation. For example, a range over a map is not 
> necessarily deterministic.
> 
> If you continue to ask detail-questions like this, it would be useful if you 
> could provide an actual program you'd want to write using them. Otherwise, it 
> makes a lot more sense to just wait for the spec-changes to be known. At that 
> point, it'll be easier to talk about if a specific example does or does not 
> conform to the spec.
> 
>> On Fri, Nov 12, 2021 at 5:08 PM T L  wrote:
>> 
>> 
>>> On Fri, Nov 12, 2021 at 11:54 PM Axel Wagner 
>>>  wrote:
>>> Oh, sorry, I just re-read. Your example still requires `range` to return 
>>> distinct types, so that probably shouldn't work.
>> 
>> Not this reason:
>> 
>> func Bar[T []byte|map[int]byte](s T) {
>> for i, v := range s { _, _ = i, v} // cannot range over s (variable of 
>> type T constrained by []byte|map[int]byte) (type set has no single 
>> underlying type)
>> }
>>  
>>> 
 On Fri, Nov 12, 2021 at 4:52 PM Axel Wagner 
  wrote:
 The error message gives you a reason - there is no single underlying type. 
 This works:
 
 type B []byte
 func Bar[T []byte|B](s T) {
 for range s {}
 }
 
 But yes, your example arguably should be made to work eventually. I would 
 suggest filing an issue about that.
 
> On Fri, Nov 12, 2021 at 4:47 PM tapi...@gmail.com  
> wrote:
> This one doesn't compile either.
> So the range operation doesn't support values of parameter types?
> 
> type MyByte byte
> 
> func Bar[T []byte|[]MyByte](s T) {
> for range s {} // cannot range over s (variable of type T constrained 
> by []byte|[]MyByte) (type set has no single underlying 
> } 
> 
>> On Friday, November 12, 2021 at 11:40:53 PM UTC+8 
>> axel.wa...@googlemail.com wrote:
>>> On Fri, Nov 12, 2021 at 4:29 PM tapi...@gmail.com  
>>> wrote:
 On Friday, November 12, 2021 at 11:10:17 PM UTC+8 
 axel.wa...@googlemail.com wrote:
 `range s` works differently for string and []byte. In the former case 
 it iterates over utf8-encoded unicode codepoints, in the latter it 
 iterates over bytes.
>>> 
>>> It is true, but why it matters here?
>> 
>> Because it makes `range` different operations for the purposes of 
>> determining the allowed operations on a type-parameter.
>>  
>> 
>>>  
 
> On Fri, Nov 12, 2021 at 3:18 PM tapi...@gmail.com  
> wrote:
> 
> func Bar[T []byte|string](s T) bool {
> for _, v := range s { // cannot range over s (variable of type T 
> constrained by []byte|string) (T has no structural type)
> if v > 100 {
> return true
> }
> }
> return false
> }
> 
> The message is quite confusing.
 
> -- 
> You received this message because you are subscribed to the Google 
> Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, 
> send an email to golang-nuts...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/44b5f535-adc8-4049-ba41-638f90becc3cn%40googlegroups.com.
>>> 
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to golang-nuts...@googlegroups.com.
>> 
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/golang-nuts/abd83194-8198-40e8-ad12-82953f85fba5n%40googlegroups.com.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/42054eee-a9df-40d1-b148-4e5ead89ee0cn%40googlegroups.com.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAEkBMfHFJavSL%3DtrB8ZJgXRmyzqm3PE1ae139RLy1KkkCKkRDQ%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google 

Re: [go-nuts] Go http/2 implementation is x5 less per performant than http/1.1

[robert engels]

No worries. I updated the issue and the CL. I will comment in the CL with a few 
more details.

> On Nov 10, 2021, at 2:30 PM, Andrey T.  wrote:
> 
> Thank you Robert, 
> I somehow missed the reference to the ticket in the first message, sorry 
> about that.
> 
> As for the CL - I think adding link to the github issue, and add a bit of 
> explanation in a commit message would help.
> I added link to your CL to the github issue's discussion, hopefully it will 
> bring more attention to it. 
> 
> A.
> 
> On Wednesday, November 10, 2021 at 1:22:42 PM UTC-7 ren...@ix.netcom.com 
> wrote:
> As reported in the OP, the issue was filed long ago 
> https://github.com/golang/go/issues/47840 
> <https://github.com/golang/go/issues/47840>
> 
> My CL https://go-review.googlesource.com/c/net/+/362834 
> <https://go-review.googlesource.com/c/net/+/362834> is a viable fix (and 
> should of been supported originally).
> 
> 
>> On Nov 10, 2021, at 12:59 PM, Andrey T. > > wrote:
>> 
> 
>> Fellas, 
>> I would say the 5x throughput difference is a serious problem.Would you be 
>> kind and open an issue on github about it? 
>> Also, the PR that you have might benefit from explanation about what you are 
>> trying to solve (and probably link to an issue on github), so it would get 
>> more attention. 
>> 
>> Thanks!
>> 
>> Andrey
>> 
>> 
>> On Tuesday, November 9, 2021 at 4:50:34 PM UTC-7 ren...@ix.netcom.com 
>> <http://ix.netcom.com/> wrote:
>> Well, I figured out a way to do it simply. The CL is here 
>> https://go-review.googlesource.com/c/net/+/362834 
>> <https://go-review.googlesource.com/c/net/+/362834>
>> 
>> The frame size will be used for all connections using that transport, so it 
>> is probably better to create a transport specifically for the 
>> high-throughput transfers.
>> 
>> You can also create perform single shot requests like:
>> 
>> if useH2C {
>>rt = {
>>   AllowHTTP: true,
>>   DialTLS: func(network, addr string, cfg *tls.Config) (net.Conn, error) 
>> {
>>  return dialer.Dial(network, addr)
>>   },
>>   MaxFrameSize: 1024*256,
>>}
>> }
>> 
>> var body io.ReadCloser = http.NoBody
>> 
>> req, err := http.NewRequestWithContext(ctx, "GET", url, body)
>> if err != nil {
>>return err
>> }
>> 
>> resp, err := rt.RoundTrip(req)
>> 
>> 
>>> On Nov 9, 2021, at 3:31 PM, Robert Engels > wrote:
>>> 
>>> To be clear, I have no plans to submit a Cl to improve this at this time. 
>>> 
>>> It would require some api changes to implement properly. 
>>> 
>>>> On Nov 9, 2021, at 12:19 PM, Kirth Gersen > wrote:
>>>> 
>>>> Great !
>>>> 
>>>> > I made some local mods to the net library, increasing the frame size to 
>>>> > 256k, and the http2 performance went from 8Gbps to 38Gbps.
>>>> That is already enormous for us. thx for finding this.
>>>> 
>>>> 4 -> Indeed  a lot of WINDOW_UPDATE messages are visible when using 
>>>> GODEBUG=http2debug=1 
>>>> 
>>>> 
>>>> On Tuesday, November 9, 2021 at 6:28:16 PM UTC+1 ren...@ix.netcom.com 
>>>> <http://ix.netcom.com/> wrote:
>>>> I did a review of the codebase.
>>>> 
>>>> Http2 is a multiplexed protocol with independent streams. The Go 
>>>> implementation uses a common reader thread/routine to read all of the 
>>>> connection content, and then demuxes the streams and passes the data via 
>>>> pipes to the stream readers. This multithreaded nature requires the use of 
>>>> locks to coordinate. By managing the window size, the connection reader 
>>>> should never block writing to a steam buffer - but a stream reader may 
>>>> stall waiting for data to arrive - get descheduled - only to be quickly 
>>>> rescheduled when reader places more data in the buffer - which is 
>>>> inefficient.
>>>> 
>>>> Out of the box on my machine, http1 is about 37 Gbps, and http2 is about 7 
>>>> Gbps on my system.
>>>> 
>>>> Some things that jump out:
>>>> 
>>>> 1. The chunk size is too small. Using 1MB pushed http1 from 37 Gbs to 50 
>>>> Gbps, and http2 to 8 Gbps.
>>>> 
>>>> 2. The default buffer in io.Copy() is too small. Use io.CopyBuffer() with 
>>>> a larger buffer - I change

Re: [go-nuts] Go http/2 implementation is x5 less per performant than http/1.1

[robert engels]

As reported in the OP, the issue was filed long ago 
https://github.com/golang/go/issues/47840 
<https://github.com/golang/go/issues/47840>

My CL https://go-review.googlesource.com/c/net/+/362834 
<https://go-review.googlesource.com/c/net/+/362834> is a viable fix (and should 
of been supported originally).

> On Nov 10, 2021, at 12:59 PM, Andrey T.  wrote:
> 
> Fellas, 
> I would say the 5x throughput difference is a serious problem.Would you be 
> kind and open an issue on github about it? 
> Also, the PR that you have might benefit from explanation about what you are 
> trying to solve (and probably link to an issue on github), so it would get 
> more attention. 
> 
> Thanks!
> 
> Andrey
> 
> 
> On Tuesday, November 9, 2021 at 4:50:34 PM UTC-7 ren...@ix.netcom.com 
> <http://ix.netcom.com/> wrote:
> Well, I figured out a way to do it simply. The CL is here 
> https://go-review.googlesource.com/c/net/+/362834 
> <https://go-review.googlesource.com/c/net/+/362834>
> 
> The frame size will be used for all connections using that transport, so it 
> is probably better to create a transport specifically for the high-throughput 
> transfers.
> 
> You can also create perform single shot requests like:
> 
> if useH2C {
>rt = {
>   AllowHTTP: true,
>   DialTLS: func(network, addr string, cfg *tls.Config) (net.Conn, error) {
>  return dialer.Dial(network, addr)
>   },
>   MaxFrameSize: 1024*256,
>}
> }
> 
> var body io.ReadCloser = http.NoBody
> 
> req, err := http.NewRequestWithContext(ctx, "GET", url, body)
> if err != nil {
>return err
> }
> 
> resp, err := rt.RoundTrip(req)
> 
> 
>> On Nov 9, 2021, at 3:31 PM, Robert Engels > > wrote:
>> 
>> To be clear, I have no plans to submit a Cl to improve this at this time. 
>> 
>> It would require some api changes to implement properly. 
>> 
>>> On Nov 9, 2021, at 12:19 PM, Kirth Gersen >> > wrote:
>>> 
>>> Great !
>>> 
>>> > I made some local mods to the net library, increasing the frame size to 
>>> > 256k, and the http2 performance went from 8Gbps to 38Gbps.
>>> That is already enormous for us. thx for finding this.
>>> 
>>> 4 -> Indeed  a lot of WINDOW_UPDATE messages are visible when using 
>>> GODEBUG=http2debug=1 
>>> 
>>> 
>>> On Tuesday, November 9, 2021 at 6:28:16 PM UTC+1 ren...@ix.netcom.com 
>>> <http://ix.netcom.com/> wrote:
>>> I did a review of the codebase.
>>> 
>>> Http2 is a multiplexed protocol with independent streams. The Go 
>>> implementation uses a common reader thread/routine to read all of the 
>>> connection content, and then demuxes the streams and passes the data via 
>>> pipes to the stream readers. This multithreaded nature requires the use of 
>>> locks to coordinate. By managing the window size, the connection reader 
>>> should never block writing to a steam buffer - but a stream reader may 
>>> stall waiting for data to arrive - get descheduled - only to be quickly 
>>> rescheduled when reader places more data in the buffer - which is 
>>> inefficient.
>>> 
>>> Out of the box on my machine, http1 is about 37 Gbps, and http2 is about 7 
>>> Gbps on my system.
>>> 
>>> Some things that jump out:
>>> 
>>> 1. The chunk size is too small. Using 1MB pushed http1 from 37 Gbs to 50 
>>> Gbps, and http2 to 8 Gbps.
>>> 
>>> 2. The default buffer in io.Copy() is too small. Use io.CopyBuffer() with a 
>>> larger buffer - I changed to 4MB. This pushed http1 to 55 Gbs, and http2 to 
>>> 8.2. Not a big difference but needed for later.
>>> 
>>> 3. The http2 receiver frame size of 16k is way too small. There is overhead 
>>> on every frame - the most costly is updating the window.
>>> 
>>> I made some local mods to the net library, increasing the frame size to 
>>> 256k, and the http2 performance went from 8Gbps to 38Gbps.
>>> 
>>> 4. I haven’t tracked it down yet, but I don’t think the window size update 
>>> code is not working as intended - it seems to be sending window updates 
>>> (which are expensive due to locks) far too frequently. I think this is the 
>>> area that could use the most improvement - using some heuristics there is 
>>> the possibility to detect the sender rate, and adjust the refresh rate 
>>> (using high/low water marks).
>>> 
>>> 5. The implementation might need improvements using lock-fr

Re: [go-nuts] Go http/2 implementation is x5 less per performant than http/1.1

[robert engels]

Well, I figured out a way to do it simply. The CL is here 
https://go-review.googlesource.com/c/net/+/362834 
<https://go-review.googlesource.com/c/net/+/362834>

The frame size will be used for all connections using that transport, so it is 
probably better to create a transport specifically for the high-throughput 
transfers.

You can also create perform single shot requests like:

if useH2C {
   rt = {
  AllowHTTP: true,
  DialTLS: func(network, addr string, cfg *tls.Config) (net.Conn, error) {
 return dialer.Dial(network, addr)
  },
  MaxFrameSize: 1024*256,
   }
}

var body io.ReadCloser = http.NoBody

req, err := http.NewRequestWithContext(ctx, "GET", url, body)
if err != nil {
   return err
}

resp, err := rt.RoundTrip(req)

> On Nov 9, 2021, at 3:31 PM, Robert Engels  wrote:
> 
> To be clear, I have no plans to submit a Cl to improve this at this time. 
> 
> It would require some api changes to implement properly. 
> 
>> On Nov 9, 2021, at 12:19 PM, Kirth Gersen  wrote:
>> 
>> Great !
>> 
>> > I made some local mods to the net library, increasing the frame size to 
>> > 256k, and the http2 performance went from 8Gbps to 38Gbps.
>> That is already enormous for us. thx for finding this.
>> 
>> 4 -> Indeed  a lot of WINDOW_UPDATE messages are visible when using 
>> GODEBUG=http2debug=1 
>> 
>> 
>> On Tuesday, November 9, 2021 at 6:28:16 PM UTC+1 ren...@ix.netcom.com wrote:
>> I did a review of the codebase.
>> 
>> Http2 is a multiplexed protocol with independent streams. The Go 
>> implementation uses a common reader thread/routine to read all of the 
>> connection content, and then demuxes the streams and passes the data via 
>> pipes to the stream readers. This multithreaded nature requires the use of 
>> locks to coordinate. By managing the window size, the connection reader 
>> should never block writing to a steam buffer - but a stream reader may stall 
>> waiting for data to arrive - get descheduled - only to be quickly 
>> rescheduled when reader places more data in the buffer - which is 
>> inefficient.
>> 
>> Out of the box on my machine, http1 is about 37 Gbps, and http2 is about 7 
>> Gbps on my system.
>> 
>> Some things that jump out:
>> 
>> 1. The chunk size is too small. Using 1MB pushed http1 from 37 Gbs to 50 
>> Gbps, and http2 to 8 Gbps.
>> 
>> 2. The default buffer in io.Copy() is too small. Use io.CopyBuffer() with a 
>> larger buffer - I changed to 4MB. This pushed http1 to 55 Gbs, and http2 to 
>> 8.2. Not a big difference but needed for later.
>> 
>> 3. The http2 receiver frame size of 16k is way too small. There is overhead 
>> on every frame - the most costly is updating the window.
>> 
>> I made some local mods to the net library, increasing the frame size to 
>> 256k, and the http2 performance went from 8Gbps to 38Gbps.
>> 
>> 4. I haven’t tracked it down yet, but I don’t think the window size update 
>> code is not working as intended - it seems to be sending window updates 
>> (which are expensive due to locks) far too frequently. I think this is the 
>> area that could use the most improvement - using some heuristics there is 
>> the possibility to detect the sender rate, and adjust the refresh rate 
>> (using high/low water marks).
>> 
>> 5. The implementation might need improvements using lock-free structures, 
>> atomic counters, and busy-waits in order to achieve maximum performance.
>> 
>> So 38Gbps for http2 vs 55 Gbps for http1. Better but still not great. Still, 
>> with some minor changes, the net package could allow setting of a large 
>> frame size on a per stream basis - which would enable much higher 
>> throughput. The gRPC library allows this.
>> 
>> 
>>> On Nov 8, 2021, at 10:58 AM, Kirth Gersen >> > wrote:
>>> 
>> 
>>> http/2 implementation seems ~5x slower in bytes per seconds (when transfer 
>>> is cpu capped).
>>> 
>>> POC: https://github.com/nspeed-app/http2issue 
>>> <https://github.com/nspeed-app/http2issue>
>>> 
>>> I submitted an issue about this 3 months ago in the Go Github ( 
>>> https://github.com/golang/go/issues/47840 
>>> <https://github.com/golang/go/issues/47840> ) but first commenter 
>>> misunderstood it and it got buried (they're probably just swamped with too 
>>> many open issues (5k+...)).
>>> 
>>> Everything using Golang net/http is impacted, the Caddy web server for 
>>> instance.
>>> 
>>> I know it probably doesn't matter fo

Re: [go-nuts] Go http/2 implementation is x5 less per performant than http/1.1

[Robert Engels]

To be clear, I have no plans to submit a Cl to improve this at this time. 

It would require some api changes to implement properly. 

> On Nov 9, 2021, at 12:19 PM, Kirth Gersen  wrote:
> 
> Great !
> 
> > I made some local mods to the net library, increasing the frame size to 
> > 256k, and the http2 performance went from 8Gbps to 38Gbps.
> That is already enormous for us. thx for finding this.
> 
> 4 -> Indeed  a lot of WINDOW_UPDATE messages are visible when using 
> GODEBUG=http2debug=1 
> 
> 
>> On Tuesday, November 9, 2021 at 6:28:16 PM UTC+1 ren...@ix.netcom.com wrote:
>> I did a review of the codebase.
>> 
>> Http2 is a multiplexed protocol with independent streams. The Go 
>> implementation uses a common reader thread/routine to read all of the 
>> connection content, and then demuxes the streams and passes the data via 
>> pipes to the stream readers. This multithreaded nature requires the use of 
>> locks to coordinate. By managing the window size, the connection reader 
>> should never block writing to a steam buffer - but a stream reader may stall 
>> waiting for data to arrive - get descheduled - only to be quickly 
>> rescheduled when reader places more data in the buffer - which is 
>> inefficient.
>> 
>> Out of the box on my machine, http1 is about 37 Gbps, and http2 is about 7 
>> Gbps on my system.
>> 
>> Some things that jump out:
>> 
>> 1. The chunk size is too small. Using 1MB pushed http1 from 37 Gbs to 50 
>> Gbps, and http2 to 8 Gbps.
>> 
>> 2. The default buffer in io.Copy() is too small. Use io.CopyBuffer() with a 
>> larger buffer - I changed to 4MB. This pushed http1 to 55 Gbs, and http2 to 
>> 8.2. Not a big difference but needed for later.
>> 
>> 3. The http2 receiver frame size of 16k is way too small. There is overhead 
>> on every frame - the most costly is updating the window.
>> 
>> I made some local mods to the net library, increasing the frame size to 
>> 256k, and the http2 performance went from 8Gbps to 38Gbps.
>> 
>> 4. I haven’t tracked it down yet, but I don’t think the window size update 
>> code is not working as intended - it seems to be sending window updates 
>> (which are expensive due to locks) far too frequently. I think this is the 
>> area that could use the most improvement - using some heuristics there is 
>> the possibility to detect the sender rate, and adjust the refresh rate 
>> (using high/low water marks).
>> 
>> 5. The implementation might need improvements using lock-free structures, 
>> atomic counters, and busy-waits in order to achieve maximum performance.
>> 
>> So 38Gbps for http2 vs 55 Gbps for http1. Better but still not great. Still, 
>> with some minor changes, the net package could allow setting of a large 
>> frame size on a per stream basis - which would enable much higher 
>> throughput. The gRPC library allows this.
>> 
 On Nov 8, 2021, at 10:58 AM, Kirth Gersen  wrote:
 
>>> http/2 implementation seems ~5x slower in bytes per seconds (when transfer 
>>> is cpu capped).
>>> 
>>> POC: https://github.com/nspeed-app/http2issue
>>> 
>>> I submitted an issue about this 3 months ago in the Go Github ( 
>>> https://github.com/golang/go/issues/47840 ) but first commenter 
>>> misunderstood it and it got buried (they're probably just swamped with too 
>>> many open issues (5k+...)).
>>> 
>>> Everything using Golang net/http is impacted, the Caddy web server for 
>>> instance.
>>> 
>>> I know it probably doesn't matter for most use cases because it's only 
>>> noticeable with high throughput transfers (>1 Gbps). 
>>> Most http benchmarks focus on "requests per second" and not "bits per 
>>> seconds" but this performance matters too sometimes.
>>> 
>>> If anyone with expertise in profiling Go code and good knowledge of the 
>>> net/http lib internal could take a look. It would be nice to optimize it or 
>>> at least have an explanation.
>>> 
>>> thx (sorry if wrong  group to post this).
>>> 
>> 
>>> -- 
>>> You received this message because you are subscribed to the Google Groups 
>>> "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to golang-nuts...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/golang-nuts/89926c2f-ec73-43ad-be49-a8bc76a18345n%40googlegroups.com.
>> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/7332f727-6716-4c4d-85c5-a86cacd0c89fn%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 

Re: [go-nuts] Go http/2 implementation is x5 less per performant than http/1.1

[robert engels]

I did a review of the codebase.

Http2 is a multiplexed protocol with independent streams. The Go implementation 
uses a common reader thread/routine to read all of the connection content, and 
then demuxes the streams and passes the data via pipes to the stream readers. 
This multithreaded nature requires the use of locks to coordinate. By managing 
the window size, the connection reader should never block writing to a steam 
buffer - but a stream reader may stall waiting for data to arrive - get 
descheduled - only to be quickly rescheduled when reader places more data in 
the buffer - which is inefficient.

Out of the box on my machine, http1 is about 37 Gbps, and http2 is about 7 Gbps 
on my system.

Some things that jump out:

1. The chunk size is too small. Using 1MB pushed http1 from 37 Gbs to 50 Gbps, 
and http2 to 8 Gbps.

2. The default buffer in io.Copy() is too small. Use io.CopyBuffer() with a 
larger buffer - I changed to 4MB. This pushed http1 to 55 Gbs, and http2 to 
8.2. Not a big difference but needed for later.

3. The http2 receiver frame size of 16k is way too small. There is overhead on 
every frame - the most costly is updating the window.

I made some local mods to the net library, increasing the frame size to 256k, 
and the http2 performance went from 8Gbps to 38Gbps.

4. I haven’t tracked it down yet, but I don’t think the window size update code 
is not working as intended - it seems to be sending window updates (which are 
expensive due to locks) far too frequently. I think this is the area that could 
use the most improvement - using some heuristics there is the possibility to 
detect the sender rate, and adjust the refresh rate (using high/low water 
marks).

5. The implementation might need improvements using lock-free structures, 
atomic counters, and busy-waits in order to achieve maximum performance.

So 38Gbps for http2 vs 55 Gbps for http1. Better but still not great. Still, 
with some minor changes, the net package could allow setting of a large frame 
size on a per stream basis - which would enable much higher throughput. The 
gRPC library allows this.

> On Nov 8, 2021, at 10:58 AM, Kirth Gersen  wrote:
> 
> http/2 implementation seems ~5x slower in bytes per seconds (when transfer is 
> cpu capped).
> 
> POC: https://github.com/nspeed-app/http2issue 
> 
> 
> I submitted an issue about this 3 months ago in the Go Github ( 
> https://github.com/golang/go/issues/47840 
>  ) but first commenter 
> misunderstood it and it got buried (they're probably just swamped with too 
> many open issues (5k+...)).
> 
> Everything using Golang net/http is impacted, the Caddy web server for 
> instance.
> 
> I know it probably doesn't matter for most use cases because it's only 
> noticeable with high throughput transfers (>1 Gbps). 
> Most http benchmarks focus on "requests per second" and not "bits per 
> seconds" but this performance matters too sometimes.
> 
> If anyone with expertise in profiling Go code and good knowledge of the 
> net/http lib internal could take a look. It would be nice to optimize it or 
> at least have an explanation.
> 
> thx (sorry if wrong  group to post this).
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/89926c2f-ec73-43ad-be49-a8bc76a18345n%40googlegroups.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/AB3F7933-45CC-4C83-8B7B-D1DF6D609546%40ix.netcom.com.

Re: [go-nuts] Golang JPEG2000 implementation / Gauging community interest in supporting it?

[Robert Engels]

There are several open source Java encoder/decoder libraries. These can be 
translated pretty easily to Go. 

> On Nov 5, 2021, at 3:19 PM, 'Dan Kortschak' via golang-nuts 
>  wrote:
> 
> On Wed, 2021-11-03 at 18:50 -0700, Adam Koszek wrote:
>> Hello,
>> 
>> We (Segmed.ai) are processing a lot of medical imaging data. It comes
>> to us in the form of PNG/JPG/DICOM files. 90% of it is uncompressed
>> or using a normal JPEG encoding, but around ~7% of it is encoded with
>> lossless JPEG 2000 format.
>> 
>> We use Suyash Kumar's library: https://github.com/suyashkumar/dicom
>> which depends on the Golang "image/jpeg" library. We'd love to get 
>> "image/jpeg" to support JPEG 2000 lossless encoding format.
>> 
>> I know it's rather a rare format, yet it's "the standard". I wonder
>> if there are any other users interested in getting JPEG 2000
>> supported natively in Go? Or maybe someone out there has its
>> implementation written, and would need hand open-sourcing it?
>> 
>> I was thinking that maybe there are companies that would benefit from
>> it, and we could co-sponsor someone to whip a nice implementation for
>> us. We could probably help with organizing this, pitch in some $$$
>> too, and manage the work. In case you're a domain expert in search of
>> a cool project, feel free to reply to this post.
>> 
>> Thanks,
>> 
>> Wojciech Adam Koszek
>> 
> 
> 
> Part of the issue here is the availability of the spec. The ITU are
> unpleasantly closed (at least for free copies) with spec documents
> which reduces the likelihood of open source project implementations.
> 
> (Not putting myself forward in the case that a spec is provided).
> 
> Dan
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/69f9b44981f8fd9134d9e1ceb3e19f4a6486b75d.camel%40kortschak.io.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/DB5DC230-8336-4C5E-AD76-C50797A826B0%40ix.netcom.com.

Re: [go-nuts] Sleep causes a tenfold difference in time interval

[Robert Engels]

Because when you sleep you deschedule the process/thread by the OS - and so the 
cached code and data needs to be reloaded so operations aren’t slower until 
then. 

> On Nov 4, 2021, at 11:19 AM, 尚义龙  wrote:
> 
> 
> I was doing pressure testing in my project and found a large gap in the time 
> interval due to time.Sleep, which I abstracted to the following code:
> code 1:
> ```go
> func TestSleep(t *testing.T) {
>   for i := 0; i < 8; i++ {
>   //time.Sleep(time.Second)
>   t1 := time.Now()
>   fmt.Println( time.Since(t1))
>   }
> }
> ```
> output:
> ```shell
> 229ns
> 37ns
> 36ns
> 37ns
> 36ns
> 38ns
> 37ns
> 39ns
> ```
> code 2(add time.Sleep):
> ```
> func TestSleep(t *testing.T) {
>   for i := 0; i < 8; i++ {
>   time.Sleep(time.Second)
>   t1 := time.Now()
>   fmt.Println( time.Since(t1))
>   }
> }
> ```
> ```shell
> 471ns
> 267ns
> 855ns
> 484ns
> 359ns
> 296ns
> 324ns
> 302ns
> ```
> I have two questions:
> 1. Why is there such a big difference between the two results (only the 
> time.Sleep difference)?
> 2. Why is the first output in code1 much larger than the next?
> please.
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/3908bb81-d57f-414b-b514-40016b549789n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/294FD8BF-4E74-41C1-924D-D5F57C903726%40ix.netcom.com.

Re: [go-nuts] Does GC pause affect non-go threads spawned by CGo?

[Robert Engels]

No, unless they call back into Go. 

> On Oct 25, 2021, at 10:43 AM, Amir DER  wrote:
> 
> Does GC pause affect non-go threads spawned by CGo?
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/7d165522-f261-4391-9708-eb3ace3fb7a2n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/E9FA7E6B-CF4B-4CB3-A711-11DB0C9FC029%40ix.netcom.com.

Re: [go-nuts] testing if strconv.Quote() would do change a string, without calling it

[Robert Engels]

I was thinking the other way. If !IsPrint() then strconv.Quote()

> On Oct 13, 2021, at 6:24 PM, Tim Hockin  wrote:
> 
> 
> ` IsPrint(r) || r == '\\' || r == '"' ` passes tests.  I need to build 
> confidence in that, though :)  Thanks.
> 
>> On Wed, Oct 13, 2021 at 4:16 PM Robert Engels  wrote:
>> A simple loop calling IsPrint is your best bet. You could then have a custom 
>> implementation of Quote that started at a specified index. 
>> 
>>> On Oct 13, 2021, at 5:46 PM, 'Tim Hockin' via golang-nuts 
>>>  wrote:
>>> 
>>> Is there any ready-built function that can tell me whether 
>>> `strconv.Quote()` would produce a different string than its input, without 
>>> actually running it?  Or is there a clearly documented set of rules one 
>>> could use to test each rune in a string?
>>> 
>>> I am trying to avoid allocations, and MOST of the inputs will be safe (but 
>>> not all).  Calling strconv.Quote() has a measurable impact, so I'd avoid it 
>>> if I could...
>>> 
>>> Does such an animal exist?
>>> 
>>> Tim
>>> -- 
>>> You received this message because you are subscribed to the Google Groups 
>>> "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to golang-nuts+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/golang-nuts/1bb3f806-5930-4866-8249-0bbc0ee383b8n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/A27DF12D-BFB7-4841-8113-69AF70C4BDB1%40ix.netcom.com.

Re: [go-nuts] testing if strconv.Quote() would do change a string, without calling it

[Robert Engels]

A simple loop calling IsPrint is your best bet. You could then have a custom 
implementation of Quote that started at a specified index. 

> On Oct 13, 2021, at 5:46 PM, 'Tim Hockin' via golang-nuts 
>  wrote:
> 
> Is there any ready-built function that can tell me whether `strconv.Quote()` 
> would produce a different string than its input, without actually running it? 
>  Or is there a clearly documented set of rules one could use to test each 
> rune in a string?
> 
> I am trying to avoid allocations, and MOST of the inputs will be safe (but 
> not all).  Calling strconv.Quote() has a measurable impact, so I'd avoid it 
> if I could...
> 
> Does such an animal exist?
> 
> Tim
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/1bb3f806-5930-4866-8249-0bbc0ee383b8n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/8E01D846-1995-4937-B6D6-C91B6B72DF43%40ix.netcom.com.

Re: [go-nuts] Can generics help me make my library safer?

[Robert Engels]

I think you can only do that if you make sql parsing a first class language 
feature - or you need to construct the query using a syntax tree of clauses 
which is a PITA. Sometimes a hybrid approach - not a full orm - but an sql 
helper works best so 

sql.Query(table name, field list, where clauses, order by clauses) can work but 
for complex sql like joins it becomes unwieldy. 

After years of doing both, I’ve settled on that using DAOs creates the simplest 
and highest performing code. 

> On Oct 6, 2021, at 8:07 AM, Brian Candler  wrote:
> 
> FWIW, I like the idea of being able to write direct SQL and still have some 
> static type checking.  ORMs are OK for simple "get" and "put", but I have 
> been bitten so many times where I *know* the exact SQL I want for a 
> particular query, but the ORM makes it so damned hard to construct it their 
> way.
> 
>> On Wednesday, 6 October 2021 at 12:45:01 UTC+1 ren...@ix.netcom.com wrote:
>> Personally, I think this is overkill (the entire concept not the rog 
>> solution)
>> 
>> Even with static checking there is no way to ensure that tablex has the 
>> needed fields. Even if you could check this at compile time - it might be 
>> different at runtime. 
>> 
>> I don’t think the juice is worth the squeeze. 
>> 
>> Either use an ORM that generates the sql, or create DAOs and rely on test 
>> cases to ensure the code is correct.
>> 
>> Far simpler and more robust in my opinion. 
>> 
 On Oct 6, 2021, at 6:35 AM, roger peppe  wrote:
 
>>> 
> On Wed, 6 Oct 2021 at 09:21, mi...@ubo.ro  wrote:
 
> Hi Ian ,
> 
> I've modified the example towards a more specific use case. The main idea 
> in the example below  is to make code related to database operations(i.e 
> SELECT queries) safer and  easier to read. A  kind of 
> json.Unmarshal/Marshal for databases, with validation (type checking, 
> param numbers etc) to avoid a class of bugs/errors such invalid param 
> types/numbers passed, invalid queries, invalid resource type to scan into 
> etc. 
>  
> Currently the function returned by Select  throws the validation errors 
> at runtime (i.e. invalid param type passed etc). It would be great to 
> have that class of errors checked at compile type.
>  
> The only way I could achieve that kind of  type checking was  through 
> code generation. I already built a tool to generate functions with the 
> proper param types but it seems that code generation introduces a lot of 
> friction to the point that I stopped using it.
> 
> My hope is that one day a Go feature (i.e. a version of Generics) could 
> help the function returned by func Select be type checked at compile time.
> 
  
 If you change your API slightly to use a single selector argument of 
 struct type, you could do something like this:
 
 https://go2goplay.golang.org/p/QH1VCQFxrZS
 
 In summary:
 
 type Flower struct {
 Color  string
 Size   int
 Weight int
 }
 
 type ByColor struct {
 Color string
 }
 
 var FlowerByColor = Select[Flower, ByColor]("* FROM tablex WHERE Color=$ 
 LIMIT 1")
 
 type SelectFunc[Resource, Args any] func(args Args) (Resource, error)
 
 // Select returns a function that executes the given SQL query,
 // expecting results to contain fields matching Resource and
 // using fields in Args to select rows.
 //
 // Both Resource and Args must be struct types; All the fields
 // in Args must have matching fields in Resource.
 func Select[Resource, Args any](q string) SelectFunc[Resource, Args] {
 
 
 That is, we can use a combination of reflection and generics. The generics 
 keep the code type-safe. The reflection part does the more specific type 
 checking once only at init time, something I like to think of as "almost 
 statically typed". It's a powerful pattern in my view.
 
   cheers,
 rog.
 
>>> -- 
>>> You received this message because you are subscribed to the Google Groups 
>>> "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to golang-nuts...@googlegroups.com.
>> 
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/golang-nuts/CAJhgacjQGQ%2BEuGVu8%2Bp5FLDFjeE_amrU9G4zGAjderEZAUTAnA%40mail.gmail.com.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/010a11a1-262b-4699-8e70-61d9e460c270n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop 

Re: [go-nuts] Can generics help me make my library safer?

[Robert Engels]

Personally, I think this is overkill (the entire concept not the rog solution)

Even with static checking there is no way to ensure that tablex has the needed 
fields. Even if you could check this at compile time - it might be different at 
runtime. 

I don’t think the juice is worth the squeeze. 

Either use an ORM that generates the sql, or create DAOs and rely on test cases 
to ensure the code is correct.

Far simpler and more robust in my opinion. 

> On Oct 6, 2021, at 6:35 AM, roger peppe  wrote:
> 
> 
>> On Wed, 6 Oct 2021 at 09:21, mi...@ubo.ro  wrote:
> 
>> Hi Ian ,
>> 
>> I've modified the example towards a more specific use case. The main idea in 
>> the example below  is to make code related to database operations(i.e SELECT 
>> queries) safer and  easier to read. A  kind of json.Unmarshal/Marshal for 
>> databases, with validation (type checking, param numbers etc) to avoid a 
>> class of bugs/errors such invalid param types/numbers passed, invalid 
>> queries, invalid resource type to scan into etc. 
>>  
>> Currently the function returned by Select  throws the validation errors at 
>> runtime (i.e. invalid param type passed etc). It would be great to have that 
>> class of errors checked at compile type.
>>  
>> The only way I could achieve that kind of  type checking was  through code 
>> generation. I already built a tool to generate functions with the proper 
>> param types but it seems that code generation introduces a lot of friction 
>> to the point that I stopped using it.
>> 
>> My hope is that one day a Go feature (i.e. a version of Generics) could help 
>> the function returned by func Select be type checked at compile time.
>> 
>  
> If you change your API slightly to use a single selector argument of struct 
> type, you could do something like this:
> 
> https://go2goplay.golang.org/p/QH1VCQFxrZS
> 
> In summary:
> 
> type Flower struct {
> Color  string
> Size   int
> Weight int
> }
> 
> type ByColor struct {
> Color string
> }
> 
> var FlowerByColor = Select[Flower, ByColor]("* FROM tablex WHERE Color=$ 
> LIMIT 1")
> 
> type SelectFunc[Resource, Args any] func(args Args) (Resource, error)
> 
> // Select returns a function that executes the given SQL query,
> // expecting results to contain fields matching Resource and
> // using fields in Args to select rows.
> //
> // Both Resource and Args must be struct types; All the fields
> // in Args must have matching fields in Resource.
> func Select[Resource, Args any](q string) SelectFunc[Resource, Args] {
> 
> 
> That is, we can use a combination of reflection and generics. The generics 
> keep the code type-safe. The reflection part does the more specific type 
> checking once only at init time, something I like to think of as "almost 
> statically typed". It's a powerful pattern in my view.
> 
>   cheers,
> rog.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAJhgacjQGQ%2BEuGVu8%2Bp5FLDFjeE_amrU9G4zGAjderEZAUTAnA%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/FA2B1054-11DE-4C08-8988-12463FAC9981%40ix.netcom.com.

Re: [go-nuts] how atomic instruction work?

[robert engels]

This may be of interest to you: https://github.com/golang/go/issues/5045

> On Sep 21, 2021, at 4:17 PM, Ian Lance Taylor  wrote:
> 
> On Tue, Sep 21, 2021 at 12:54 PM xie cui  > wrote:
>> 
>> 
>> https://stackoverflow.com/questions/2599238/are-memory-barriers-necessary-for-atomic-reference-counting-shared-immutable-dat
>> this answer say that:
>> On x86, it will turn into a lock prefixed assembly instruction, like LOCK 
>> XADD.
>> Being a single instruction, it is non-interruptible. As an added "feature", 
>> the lock prefix results in a full memory barrier.
>> 
>> In my opinion, full memory barrier means flush store buffer and invalid 
>> queue(not very sure, is this right?)
> 
> Can you be specific about exactly what you are asking?  You originally
> said "atomic instruction," and my answer was based on the functions
> sync/atomic.LoadInt32 and sync/atomic.StoreInt32.  Here you seem to be
> talking about sync/atomic.AddInt32, which is fine, but let's agree on
> what the question is.  This is a Go language mailing list, so can you
> ask your question about a Go function, rather than about an "atomic
> instruction"?  Thanks.
> 
> Ian
> 
> 
>> On Saturday, September 18, 2021 at 3:17:26 AM UTC+8 Ian Lance Taylor wrote:
>>> 
>>> On Fri, Sep 17, 2021 at 6:25 AM xie cui  wrote:
 
 how atomic insturction work in golang at x86/amd64,
 I think the atomic insturtion will flush the store buffer and invalid 
 queue of current cpu core,
 but I am not sure, does someone know about it?
>>> 
>>> I assume that you are asking about the sync/atomic package. The
>>> functions in that package will ensure sequential consistency of atomic
>>> operationns, but they will not flush the store buffer. See
>>> https://research.swtch.com/gomm .
>>> 
>>> Ian
>> 
>> --
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/aa7487d1-13f1-451b-86a3-f7f70a7c040cn%40googlegroups.com.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAOyqgcX-12SQkcow1az%2BPhOYfubvmuD0q72yOc_xOo%3Dia2J_SQ%40mail.gmail.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/7A0E5DBB-57FB-45AD-88AE-DC06C94DA15A%40ix.netcom.com.

Re: [go-nuts] The right way for golang binary to reduce iTLB cache miss

[robert engels]

I can’t imagine the app has 34 mb of code in the hot path where this would 
matter - so I assume the application is partitioned with various Go routines 
doing a “type” of work.

Which encounters a problem with the Go design. You can’t partition Go routines, 
and you can’t set cpu/core affinity for a Go routine.

So it can be very difficult to manage the cache (code & data) in a large 
multipurpose app, or an app where there data is naturally partitioned (e.g. a 
financial exchange where different stocks and asset classes are independent).

I didn’t look but I believe there is a FR/issue to address the above 
deficiencies.

> On Sep 18, 2021, at 8:26 PM, rmfr  wrote:
> 
> I have tried to disable the inline during compiling, but it results in no big 
> difference.
> 
> On Saturday, September 18, 2021 at 9:26:45 PM UTC+8 ren...@ix.netcom.com 
> wrote:
> I will add that over aggressive inlining can cause this - it explodes the 
> binary size.
> 
> 
>> On Sep 18, 2021, at 7:37 AM, Jesper Louis Andersen > > wrote:
>> 
> 
>> On Fri, Sep 17, 2021 at 11:09 AM rmfr > > wrote:
>> One of our Golang applications has a very huge binary size and the size of 
>> the .text segment itself in the elf is approximately 34MB. The iTLB load 
>> miss reaches about 87% of all iTLB cache hits.
>> 
>> Is there any advice for big Golang applications to reduce the iTLB cache 
>> miss? Two solutions come to me, PGO 
>>  and using 
>> hugepages to load the .text segment. But they are both seem very difficult 
>> to implement in Golang.
>> 
>> 
>> 
>> Shot in the dark: is the application running next to some other application 
>> on the hardware which wipes the TLB?
>> 
> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts...@googlegroups.com 
>> .
> 
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/CAGrdgiWDY%2Bjh2%3DPzAatJU5Ckit0_maex%3DgGC3E4M-t8VB94Qig%40mail.gmail.com
>>  
>> .
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/7de12443-2916-4190-a08d-0825605f6ca5n%40googlegroups.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/03F9275B-85F4-4E25-992A-108BFE01FA83%40ix.netcom.com.

Re: [go-nuts] The right way for golang binary to reduce iTLB cache miss

[robert engels]

I will add that over aggressive inlining can cause this - it explodes the 
binary size.

> On Sep 18, 2021, at 7:37 AM, Jesper Louis Andersen 
>  wrote:
> 
> On Fri, Sep 17, 2021 at 11:09 AM rmfr  > wrote:
> One of our Golang applications has a very huge binary size and the size of 
> the .text segment itself in the elf is approximately 34MB. The iTLB load miss 
> reaches about 87% of all iTLB cache hits.
> 
> Is there any advice for big Golang applications to reduce the iTLB cache 
> miss? Two solutions come to me, PGO 
>  and using 
> hugepages to load the .text segment. But they are both seem very difficult to 
> implement in Golang.
> 
> 
> 
> Shot in the dark: is the application running next to some other application 
> on the hardware which wipes the TLB?
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAGrdgiWDY%2Bjh2%3DPzAatJU5Ckit0_maex%3DgGC3E4M-t8VB94Qig%40mail.gmail.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/571FA9E9-4C50-4643-BB9B-16C422FC6B9F%40ix.netcom.com.

Re: [go-nuts] Re: The right way for golang binary to reduce iTLB cache miss

[Robert Engels]

This is the icache or code - not the data the code is accessing. 

> On Sep 17, 2021, at 8:12 AM, peterGo  wrote:
> 
> 
> rmfr,
> 
> What is your data, what is its structure, and what are the access paths and 
> frequencies?
> 
> Your data is unlikely to be a randomly accessed blob of bits. Is there a set 
> of Pareto access paths that you can use to order your data by access 
> frequency? You want the most frequently accessed data to be adjacent. 
> 
> Peter
> 
>> On Friday, September 17, 2021 at 5:09:32 AM UTC-4 rmfr wrote:
>> One of our Golang applications has a very huge binary size and the size of 
>> the .text segment itself in the elf is approximately 34MB. The iTLB load 
>> miss reaches about 87% of all iTLB cache hits.
>> 
>> Is there any advice for big Golang applications to reduce the iTLB cache 
>> miss? Two solutions come to me, PGO and using hugepages to load the .text 
>> segment. But they are both seem very difficult to implement in Golang.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/1778bb58-e2e1-4bba-a869-1effb1e3952an%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/75B195E6-F6BE-466D-A7FB-514EBADFB659%40ix.netcom.com.

Re: [go-nuts] Creating a MQTT library

[Robert Engels]

The advantage of Go is to be able to use millions of blocking Go routines. 

Synchronous programming is much easier than async - especially for streaming 
protocols. 

> On Sep 16, 2021, at 4:17 PM, yba...@gmail.com  wrote:
> 
> Hello everyone,
> 
> I am trying to design a new library to encode/decode MQTT messages. I began 
> the work, and now I must decide how to design the TCP processing of my 
> library, and I am facing two choices:
> - Design in a non-blocking tcp way (that is let the user use 
> select()/poll()/epoll() sycall).
> - Or design it with blocking tcp io inside go routines, which would return 2 
> channels (read and write). Then let the user do a channel-select in its main 
> go routine.
> 
> I fail to see which one is superior to the other.
> Moreover: one is idiomatic go, but one the other hand I like the idea to 
> provide a thin wrapper and let the developper user of the library to choose 
> which way he handles his problematic.
> 
> What would you do?
> 
> Regards
> Yves
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/e900640f-edaa-4f6a-9d8e-7c751841e498n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/1B897582-B68B-4D78-9434-7E9E92D348FF%40ix.netcom.com.

Re: [go-nuts] Re: Debugging "fatal: morestack on gsignal"

[Robert Engels]

If the C code overruns a stack allocated variable couldn’t it easily corrupt 
the saved registers ?

> On Sep 9, 2021, at 2:14 PM, Ian Lance Taylor  wrote:
> 
> On Thu, Sep 9, 2021 at 10:29 AM Varun V  wrote:
>> 
>> @Brian, We did not try that as the issue is happening with go 1.13.7 as well
> 
> The "fatal: morestack on gsignal" error is more or less impossible.
> Some things that can cause it to happen are
> 
> 1) C code calls sigaltstack, but the alternate signal stack is too small.
> 2) C code trashes the TLS slot that stores the value of G during a cgo call.
> 3) C code fails to correctly preserve registers across a cgo call.
> 
> None of these are at all likely.
> 
> Ian
> 
> 
>>> On Thu, 9 Sep, 2021, 10:48 pm Brian Candler,  wrote:
>>> 
>>> Just a random thought, but have you tried 1.16 with 
>>> GODEBUG=asyncpreemptoff=1 ?  The preemptive scheduling stuff was introduced 
>>> in 1.14 I believe.
>>> 
>>> On Thursday, 9 September 2021 at 16:28:38 UTC+1 varun...@gmail.com wrote:
 
 @Kurtis, Thanks for the reply.
 
 Yes. We upgraded to 1.16.5 due to this issue and had to revert back to 
 1.13.7 (over earlier working version) as we are frequently seeing these 
 errors in 1.16.5.
 
 I looked into all the related issues on golang forums. We are not on 
 NetBSD (We are on CentOS7.8)
 
 We also suspect it is a bug in CGO code (specifically some memory 
 violation) but finding the bug is like finding needle in a haystack. The 
 SIGSEGV CGO signal handler did not catch anything at the time of this 
 error. The error message by itself does not mention any thing debuggable. 
 There is no stack or core dump. We tried working with GOTRACEBACK but that 
 did not generate any core dump as well.
 
 Thanks,
 Varun
 
 
 On Thursday, September 9, 2021 at 8:07:00 PM UTC+5:30 Kurtis Rader wrote:
> 
> Googling "go morestack on gsignal" turns up quite a few reports. Some of 
> which involved kernel bugs (e.g., 
> https://github.com/golang/go/issues/19652) but most seem to involve 
> SIGSEGV errors in non-kernel code; sometimes the Go runtime or stdlib 
> (e.g., https://github.com/golang/go/issues/35235) and sometimes user 
> code. Given that you're using CGO a likely explanation is a bug in your C 
> code. However, you say you're using Go 1.13.7 which is getting long in 
> the tooth. Are you sure you're seeing the same failure with Go 1.16.5? If 
> yes then I would bet the bug is in your C code. If no then it could be a 
> Go bug that has likely already been fixed.
> 
> On Thu, Sep 9, 2021 at 4:15 AM varun...@gmail.com  
> wrote:
>> 
>> Minor update:
>> 
>> a. The crash with "morestack on gsignal" happens to be independent of 
>> the kernel versions mentioned earlier
>> b. We implemented a signal handler to catch SIGSEGV from CGO. That is 
>> not helping either
>> c. From system audit logs, we can see the following message:
>> 
>> ./audit/audit.log.2:22944:type=ANOM_ABEND 
>> msg=audit(1630376735.155:5486532): auid=4294967295 uid=996 gid=994 
>> ses=4294967295 subj=system_u:system_r:unconfined_service_t:s0 pid=26422 
>> comm="indexer" reason="memory violation" sig=5
>> 
>> Sig=5 is SIGTRAP. At this point, we suspect there is a SIGSEGV in CGO 
>> layer and that is being returned as SIGTRAP by runtime.
>> 
>> I am still clueless as to why a process can crash with "morestack on 
>> gsignal".  Any pointers to debug this further would be of great help.
>> 
>> Thanks,
>> Varun
>> On Wednesday, September 1, 2021 at 2:44:52 PM UTC+5:30 
>> varun...@gmail.com wrote:
>>> 
>>> Hello,
>>> 
>>> Recently we have been seeing multiple occurrences of "fatal: morestack 
>>> on gsignal" error in our system tests. The issue happens with golang 
>>> 1.13.7 and golang 1.16.5 runtimes. The process restarts after this 
>>> error and no stack trace is dumped. Our process uses CGO calls, I/O ops 
>>> etc.
>>> 
>>> So far, the issue seem to happen only on CentOS7 with linux kernel 
>>> version: 3.10.0-1127.19.1.el7.x86_64. The issue is not seen on: 
>>> 3.10.0-693.5.2.el7.x86_64
>>> 
>>> At this point, we have no clue as to why this is happening and no stack 
>>> trace is making it difficult to debug. Any pointers on debugging the 
>>> issue would be of great help.
>>> 
>>> Thanks,
>>> Varun
>> 
>> --
>> You received this message because you are subscribed to the Google 
>> Groups "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send 
>> an email to golang-nuts...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/9f36ab30-3222-4eed-8388-93ecd259bf55n%40googlegroups.com.
> 
> 
> 
> --
> 

Re: [go-nuts] Profiling goroutines

[Robert Engels]

Check out github.com/robaho/go-analyzer. 

It has some nice enhancements for dealing with highly concurrent programs. 

> On Sep 3, 2021, at 3:20 PM, Peter Bočan  wrote:
> 
> Hey guys, 
> 
> I am trying to get a full picture using a pprof of a highly paralelised 
> binary (read many goroutines doing stuff) but I am struggling to find any 
> information on this topic - I think what I am seeing at the moment is just a 
> single goroutine/main thread in the web UI that the tool provides.
> 
> Is there more information on how to obtain a bigger picture that may contain 
> more info (although thinking about it, it may be too much info) ?
> 
> Kindly,
> Peter.
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/6beb43d9-bb32-49d2-b8b5-168c41cc1be4n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/F7B86482-DACE-4DFB-98D0-EDF0EC60E31A%40ix.netcom.com.

Re: [go-nuts] Re: Logging libraries and standard library compatibility

[Robert Engels]

I don’t agree with Rob’s assessment. Standardized interfaces is what has 
allowed the Java 3P library ecosystem to be so robust. 

It takes more design work upfront and you can get things wrong - but it’s a 
price and risk worth taking in my opinion. 

> On Aug 31, 2021, at 4:02 PM, Amit Saha  wrote:
> 
> On Tue, Aug 31, 2021 at 9:46 PM Robert Engels  wrote:
>> 
>> The io.Writer is too low level to be useful. Imagine a logger that wants to 
>> send an sms on severe errors only.
>> 
>> The best solution is to declare your own logger interface - but similar to 
>> the collections discussion it would better if the stdlib declared an ILogger 
>> interface (or similar)
> 
> I found this FAQ entry on zap's project:
> 
> """
> Why aren't Logger and SugaredLogger interfaces?
> 
> Unlike the familiar io.Writer and http.Handler, Logger and
> SugaredLogger interfaces would include many methods. As Rob Pike
> points out, "The bigger the interface, the weaker the abstraction."
> Interfaces are also rigid — any change requires releasing a new major
> version, since it breaks all third-party implementations.
> 
> Making the Logger and SugaredLogger concrete types doesn't sacrifice
> much abstraction, and it lets us add methods without introducing
> breaking changes.
> 
> Your applications should define and depend upon an interface that
> includes just the methods you use.
> """
> 
> The last sentence is interesting here which is pointing application
> authors to, I think define their own interface for logging which can
> then be implemented by a concrete type such as standard library's
> logger or zap's or zerolog's.
> 
> 
> 
>> On Aug 31, 2021, at 3:48 AM, Rupert Paddington  wrote:
>> 
>> If Logger was an interface it would be pretty large and to some extent 
>> application-specific, which is undesirable.
>> 
>> Instead inject the sink (io.Writer).
>> 
>> This is preferable in every way:
>> 
>> * the interface is small and universal
>> * it eliminates the coupling with any specific library present and future
>> * it requires to test the behaviour instead of the collaboration
>> 
>> These are the same principles why for example the http.Handler interface 
>> takes an http.ResponseWriter, and then the outcome of the response can be 
>> tested by reading from the writer that is passed at testing time.
>>> On Tuesday, 31 August 2021 at 03:19:29 UTC+1 amits...@gmail.com wrote:
>>> 
>>>> On Tue, Aug 31, 2021 at 8:58 AM Kevin Chowski  wrote:
>>> 
>>> 
>>> 
>>> 
>>>> To the original poster: can you just declare your own Logger-like 
>>>> interface and allow users of your library to pass whatever logging 
>>>> implementation they have? In what ways are you thinking that you're forced 
>>>> to be coupled with the standard library's implementation?
>>> 
>>> Yes, I can do that - that's a good idea actually - although more
>>> complicated than one I would have liked for my use-case (more on this
>>> in my original post and below), but certainly a feasible one with long
>>> term benefits.
>>> 
>>>> In what ways are you thinking that you're forced to be coupled with the 
>>>> standard library's implementation?
>>> 
>>> When I first add some logging to my application, I am doing it using
>>> the standard library's log.Printf(), log.Println(), log.Fatal() and
>>> friends. Now, I reach a point of time where I want to start structured
>>> logging but i want to do it incrementally. My existing Printf(),
>>> Println() calls should continue to build. The only change I want to do
>>> initially is simply change how I am constructing the log.Logger
>>> object.
>> 
>> --
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/c76931fb-4d7e-40f2-84d3-8923dd280748n%40googlegroups.com.
>> 
>> --
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/874D83D2-6CF1-

Re: [go-nuts] Re: Logging libraries and standard library compatibility

[Robert Engels]

The io.Writer is too low level to be useful. Imagine a logger that wants to 
send an sms on severe errors only. 

The best solution is to declare your own logger interface - but similar to the 
collections discussion it would better if the stdlib declared an ILogger 
interface (or similar)

> On Aug 31, 2021, at 3:48 AM, Rupert Paddington  wrote:
> 
> If Logger was an interface it would be pretty large and to some extent 
> application-specific, which is undesirable.
> 
> Instead inject the sink (io.Writer).
> 
> This is preferable in every way:
> 
> * the interface is small and universal
> * it eliminates the coupling with any specific library present and future
> * it requires to test the behaviour instead of the collaboration
> 
> These are the same principles why for example the http.Handler interface 
> takes an http.ResponseWriter, and then the outcome of the response can be 
> tested by reading from the writer that is passed at testing time.
>> On Tuesday, 31 August 2021 at 03:19:29 UTC+1 amits...@gmail.com wrote:
>> On Tue, Aug 31, 2021 at 8:58 AM Kevin Chowski  wrote: 
>> 
>>  
>> 
>> 
>> > To the original poster: can you just declare your own Logger-like 
>> > interface and allow users of your library to pass whatever logging 
>> > implementation they have? In what ways are you thinking that you're forced 
>> > to be coupled with the standard library's implementation? 
>> 
>> Yes, I can do that - that's a good idea actually - although more 
>> complicated than one I would have liked for my use-case (more on this 
>> in my original post and below), but certainly a feasible one with long 
>> term benefits. 
>> 
>> > In what ways are you thinking that you're forced to be coupled with the 
>> > standard library's implementation? 
>> 
>> When I first add some logging to my application, I am doing it using 
>> the standard library's log.Printf(), log.Println(), log.Fatal() and 
>> friends. Now, I reach a point of time where I want to start structured 
>> logging but i want to do it incrementally. My existing Printf(), 
>> Println() calls should continue to build. The only change I want to do 
>> initially is simply change how I am constructing the log.Logger 
>> object. 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/c76931fb-4d7e-40f2-84d3-8923dd280748n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/874D83D2-6CF1-405A-B500-1399DB77785A%40ix.netcom.com.

Re: [go-nuts] Source links on cs.opensource.google are slow (links for stdlib on pkg.go.dev)

[Robert Engels]

Every site you access probably uses Google analytics or something similar. 

It is often used to understand usage patterns, site design, etc. 


> On Aug 25, 2021, at 10:00 PM, Ben Hoyt  wrote:
> 
> 
> > I'm in Santa Clara, CA. Just a handful of miles away from Google 
> > headquarters.
> > Your two links exhibit orders of magnitude differences in load time for me.
> 
> I presume you mean orders of magnitude, with cs.opensource being slower? What 
> are the actual figures you get?
> 
> Just for the record, for my timings of ~1s vs ~4s, I was doing very rough 
> timing just counting seconds till the source code content was visible ... I 
> didn't need to do anything more accurate as they were so far apart.
> 
> > Access to XMLHttpRequest at 'https://play.google.com/log' from origin 
> > 'https://cs.opensource.google'
> 
> Ah, great, our IP addresses and traffic data's going into Google's analytics 
> engine, too. :-)
> 
> -Ben
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAL9jXCGQExy_udnCCtrgvh5ZPQUWWL8HA%3D8uG9AjCzmR1VZitA%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/A417229C-4940-4091-B531-5CA424AEC989%40ix.netcom.com.

Re: [go-nuts] unsafe string to []byte

[Robert Engels]

That’s cool. I don’t think most libraries using unsafe go to that level of 
detail or scrutiny - which can be the source of a lot of subtle failures. 

> On Aug 3, 2021, at 3:35 PM, 'Bryan C. Mills' via golang-nuts 
>  wrote:
> 
> For what it's worth, my unsafeslice.OfString makes a best effort to detect 
> mutations of the sort that would occur when a Write implementation violates 
> the io.Writer contract.
> 
> It allows for vary levels of safety. Under `-race` it successfully detects 
> pretty much every mutation and reports the exact writer goroutine. If instead 
> built with `-tags unsafe` it should produce essentially no overhead compared 
> to the hand-written `reflect.SliceHeader` transformation. The default 
> behavior is somewhere in the middle: less overhead than the race detector, 
> but still paying O(N) overhead in order to detect and report 
> otherwise-inscrutable bugs.
> 
> That won't stop an incorrect Writer from breaking your program, but it will 
> at least help you figure out where the violation occurred.
> 
>> On Tuesday, July 27, 2021 at 10:54:27 AM UTC-4 axel.wa...@googlemail.com 
>> wrote:
>>> On Tue, Jul 27, 2021 at 4:15 PM Steve Roth  wrote:
>>> The implementation of io.WriteString appears to allocate a new byte slice 
>>> and copy the string into it:
>>> w.Write([]byte(s))
>> 
>> Only if the writer does not implement `io.StringWriter`. Avoiding this 
>> allocation where possible is exactly why `io.StringWriter` exists.
>>  
>>> Many third party libraries avoid the allocation and copy with techniques 
>>> like:
>>> var b []byte
>>> sh := (*reflect.StringHeader)(unsafe.Pointer())
>>> bh := (*reflect.SliceHeader)(unsafe.Pointer())
>>> bh.Data = sh.Data
>>> bh.Len = sh.Len
>>> bh.Cap = sh.Len
>>> w.Write(b)
>>> I've seen so many different packages do this that it almost seems like a 
>>> preferred idiom. Yet, it doesn't seem to be guaranteed safe by the rules in 
>>> the "unsafe" package documentation; rule 6 comes close to allowing it but 
>>> doesn't quite get there.  And the fact that the standard library doesn't 
>>> use it, in an obviously applicable place, is telling.
>>> 
>>> So, what's the deal here?  Is it safe or not?
>> 
>> No, that code is broken. It makes assumptions about the implementation of 
>> `Write`, which is *documented*, but not enforced by the compiler - namely, 
>> that `Write` may not retain a reference to the `[]byte` and may not modify 
>> its contents. If such an incorrect `io.Writer` is used with a library like 
>> this, it might break the program in strange and unforseen ways.
>> 
>>> Can I use it in my own code?
>> 
>> There are occasions where it is safe. For example, strings.Builder does a 
>> similar thing in a safe way.
>> So, as with all unsafe: If you know it's safe, it's fine to use. Otherwise, 
>> stay away.
>> 
>>>   Must I shun libraries that use it?  (Must I read the source code of every 
>>> library I use, to see whether it uses it?)
>> 
>> Unfortunately there is no way to guarantee that a dependency contains good 
>> code.
>> This particular issue should be reasonably easy to find by grepping for 
>> `unsafe`, which is a good practice if you want to avoid potentially unsafe 
>> code anyway.
>>  
>>> 
>>> -- 
>>> You received this message because you are subscribed to the Google Groups 
>>> "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to golang-nuts...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/golang-nuts/CAAnpqKHoA74DfM5765vUfrH4V6RpBxg1DTkfn3ScN6MhjQwRTQ%40mail.gmail.com.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/8dc95c51-9faf-48db-a237-213c32cc00c0n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/B00665C4-B29D-4BFC-A3C8-03F01EFC61C2%40ix.netcom.com.

Re: [go-nuts] Re: What are the options to fake SQL database for testing?

[Robert Engels]

I don’t like the “solves the problem of your test running fine in isolation but 
fails when run in concert with other tests”. 

If you have that problem you have poorly written tests. 

> On Jul 29, 2021, at 2:05 PM, Andrew Werner  wrote:
> 
> 
> Another choice to throw into the mix is 
> https://github.com/cockroachdb/copyist. It comes with different trade offs 
> but if you buy into its framework, it should be much faster than running the 
> database. 
> 
>> On Thu, Jul 29, 2021 at 3:13 AM Brian Candler  wrote:
>> You might also want to look at podman, which runs containers directly as 
>> processes under your own userid with no docker daemon - and buildah, the 
>> corresponding tool for creating container images.  I use them quite a lot 
>> for building and running container images before deploying them to 
>> kubernetes.
>> 
>> I haven't used the go bindings, but apparently they do exist:
>> https://podman.io/blogs/2020/08/10/podman-go-bindings.html
>> 
>> According to that, they are running podman as a service with a REST API, 
>> which isn't a million miles from a docker daemon - except that the podman 
>> service runs as your own userid, not as root.
>> 
>>> On Thursday, 29 July 2021 at 06:23:05 UTC+1 amits...@gmail.com wrote:
>>> On Thu, Jul 29, 2021 at 4:39 AM Marcin Romaszewicz  
>>> wrote: 
>>> > 
>>> > I have this exact testing issue at my company, we have many Go services 
>>> > which use Postgres in production, but are unit tested against SQLite. 
>>> > 
>>> > The latest SQLite covers the vast majority of Postgres queries, so most 
>>> > tests simply use an SQLite in-memory DB. 
>>> > 
>>> > For the tests which require Postgres- specific functionality, such as 
>>> > partitioned tables, for example, we use 
>>> > https://github.com/testcontainers/testcontainers-go. This is a library 
>>> > which talks to Docker and can create your test prerequisites as docker 
>>> > containers and gives you connection information once they're up. This 
>>> > makes unit tests incredibly slower, but at least functional. 
>>> > 
>>> > The danger with mocking too much is that your unit tests end up testing 
>>> > the mocks, and not anything remotely like your runtime environment, so 
>>> > we've chosen to mock as little as possible. 
>>> 
>>> Thank you for sharing about testcontainers-go. I have come across 
>>> testcontainers, it sounds like a viable solution as well. As with a 
>>> lot of things in software, it seems like we just have to see what 
>>> works for "us". 
>>> 
>>> > 
>>> > -- Marcin 
>>> > 
>>> > On Wed, Jul 28, 2021 at 4:09 AM Henry  wrote: 
>>> >> 
>>> >> On Wednesday, July 28, 2021 at 3:05:43 PM UTC+7 amits...@gmail.com 
>>> >> wrote: 
>>> >>> 
>>> >>> That sounds interesting - is the tool generating or is able to 
>>> >>> generate SQL for different databases? That must have been a pretty big 
>>> >>> effort to create such an abstraction. 
>>> >>> 
>>> >> 
>>> >> It produces different SQL for different databases. It supports a limited 
>>> >> number of databases. Note that quite a number of Go ORM libraries 
>>> >> already support multiple databases. So it is not new. The difference is 
>>> >> that other ORM libraries usually provide a general purpose data access 
>>> >> library, whereas ours generates more specific codes. Other than that, 
>>> >> they serve a similar purpose. 
>>> >> 
>>> >> -- 
>>> >> You received this message because you are subscribed to the Google 
>>> >> Groups "golang-nuts" group. 
>>> >> To unsubscribe from this group and stop receiving emails from it, send 
>>> >> an email to golang-nuts...@googlegroups.com. 
>>> >> To view this discussion on the web visit 
>>> >> https://groups.google.com/d/msgid/golang-nuts/9c81746a-4fb4-4fb5-8e5f-605169a3f2afn%40googlegroups.com.
>>> >>  
>>> > 
>>> > -- 
>>> > You received this message because you are subscribed to the Google Groups 
>>> > "golang-nuts" group. 
>>> > To unsubscribe from this group and stop receiving emails from it, send an 
>>> > email to golang-nuts...@googlegroups.com. 
>>> > To view this discussion on the web visit 
>>> > https://groups.google.com/d/msgid/golang-nuts/CA%2Bv29Lv83-4yDijNmukf0Vx%2BoBVZXJPR11bqA_B5CY1mNhOowA%40mail.gmail.com.
>>> >  
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/4d6f1b9d-ad44-4fc5-b309-beba2a199382n%40googlegroups.com.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> 

Re: [go-nuts] unsafe string to []byte

[Robert Engels]

Agreed. If a Go library uses unsafe I avoid it. 

> On Jul 27, 2021, at 9:54 AM, 'Axel Wagner' via golang-nuts 
>  wrote:
> 
> 
>> On Tue, Jul 27, 2021 at 4:15 PM Steve Roth  wrote:
> 
>> The implementation of io.WriteString appears to allocate a new byte slice 
>> and copy the string into it:
>> w.Write([]byte(s))
> 
> Only if the writer does not implement `io.StringWriter`. Avoiding this 
> allocation where possible is exactly why `io.StringWriter` exists.
>  
>> Many third party libraries avoid the allocation and copy with techniques 
>> like:
>> var b []byte
>> sh := (*reflect.StringHeader)(unsafe.Pointer())
>> bh := (*reflect.SliceHeader)(unsafe.Pointer())
>> bh.Data = sh.Data
>> bh.Len = sh.Len
>> bh.Cap = sh.Len
>> w.Write(b)
>> I've seen so many different packages do this that it almost seems like a 
>> preferred idiom. Yet, it doesn't seem to be guaranteed safe by the rules in 
>> the "unsafe" package documentation; rule 6 comes close to allowing it but 
>> doesn't quite get there.  And the fact that the standard library doesn't use 
>> it, in an obviously applicable place, is telling.
>> 
>> So, what's the deal here?  Is it safe or not?
> 
> No, that code is broken. It makes assumptions about the implementation of 
> `Write`, which is *documented*, but not enforced by the compiler - namely, 
> that `Write` may not retain a reference to the `[]byte` and may not modify 
> its contents. If such an incorrect `io.Writer` is used with a library like 
> this, it might break the program in strange and unforseen ways.
> 
>> Can I use it in my own code?
> 
> There are occasions where it is safe. For example, strings.Builder does a 
> similar thing in a safe way.
> So, as with all unsafe: If you know it's safe, it's fine to use. Otherwise, 
> stay away.
> 
>>   Must I shun libraries that use it?  (Must I read the source code of every 
>> library I use, to see whether it uses it?)
> 
> Unfortunately there is no way to guarantee that a dependency contains good 
> code.
> This particular issue should be reasonably easy to find by grepping for 
> `unsafe`, which is a good practice if you want to avoid potentially unsafe 
> code anyway.
>  
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/CAAnpqKHoA74DfM5765vUfrH4V6RpBxg1DTkfn3ScN6MhjQwRTQ%40mail.gmail.com.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAEkBMfEf6NbkAvDRs6QFO7LDggh-n-6mT3aBnG3gNh678Z6Z1g%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/BEFE6027-3616-4654-A7D5-FBC167D0CDFF%40ix.netcom.com.

Re: [go-nuts] Re: Out of memory using golang.org/x/tools/go/packages

[Robert Engels]

Maybe the object itself has circular references - so it fails at the first 
level. 

> On Jul 21, 2021, at 11:25 AM, Brian Candler  wrote:
> 
> I had a brief look at the code and it doesn't appear to do that.  As far as 
> I can see, it indents as it goes: where you start is the "top" level, and it 
> increases the indentation for each level of children.
> 
> The individual dump functions write directly to the writer, not to a buffer 
> (except when an object has a custom dumper, which means it has a "LitterDump" 
> method, in which case the output of LitterDump *is* buffered and realigned)
> 
>> On Wednesday, 21 July 2021 at 17:17:57 UTC+1 ren...@ix.netcom.com wrote:
>> If it is pretty printing it may need to determine the lowest level for 
>> indentation - so a depth first search - causing the entire tree to be 
>> traversed and retained before it can output anything. 
>> 
 On Jul 21, 2021, at 9:40 AM, Brian Candler  wrote:
 
>>> But AFAICT, it should generate output as it runs.  The fact that it 
>>> doesn't generate any output at all is suspicious.
>> 
>>> 
>>> 
 On Wednesday, 21 July 2021 at 13:50:37 UTC+1 ren...@ix.netcom.com wrote:
 Since litter checks for circular references it needs to keep a ref to 
 every object it sees. 
 
 With a large tree you will run out of memory. 
 
>> On Jul 21, 2021, at 7:19 AM, jake...@gmail.com  wrote:
>> 
> 
 
> The first thing I would do is remove the call to litter, and see if that 
> solved the issue. That would tell you immediately if the problem was the 
> litter package or the packages package. I have so specific knowledge, but 
> it is not impossible to imagine that you are simply trying to print 
> something way to big for litter. 
> 
> After that, using pprof might be your next step.
> 
> Have you tried it on a really tiny package?
> 
>>> On Wednesday, July 21, 2021 at 6:41:39 AM UTC-4 mlevi...@gmail.com 
>>> wrote:
>>> Hi all,
>>> 
>>> I'm having a very hard time with golang.org/x/tools/go/packages. Spent 
>>> most of my evening yesterday trying to understand what's happening here.
>>> Here's my code: https://play.golang.com/p/5L1N0lSaetB
>>> 
>>> With this very simple code I would expect that the program prints 
>>> detailed information about the package path I run it with. But whatever 
>>> the package, or module, I try to use this with, the program is killed 
>>> because it takes all the memory (~32GB) of my machine in a few seconds, 
>>> nothing ever gets printed...
>>> 
>>> Here's my config: 
>>> GO111MODULE=""
>>> GOARCH="amd64"
>>> GOBIN=""
>>> GOCACHE="/home/michel/.cache/go-build"
>>> GOENV="/home/michel/.config/go/env"
>>> GOEXE=""
>>> GOFLAGS=""
>>> GOHOSTARCH="amd64"
>>> GOHOSTOS="linux"
>>> GOINSECURE=""
>>> GOMODCACHE="/home/michel/.go/pkg/mod"
>>> GONOPROXY=""
>>> GONOSUMDB=""
>>> GOOS="linux"
>>> GOPATH="/home/michel/.go"
>>> GOPRIVATE=""
>>> GOPROXY="https://proxy.golang.org,direct;
>>> GOROOT="/usr/local/go"
>>> GOSUMDB="sum.golang.org"
>>> GOTMPDIR=""
>>> GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
>>> GOVCS=""
>>> GOVERSION="go1.16"
>>> GCCGO="gccgo"
>>> AR="ar"
>>> CC="gcc"
>>> CXX="g++"
>>> CGO_ENABLED="1"
>>> GOMOD="/dev/null"
>>> CGO_CFLAGS="-g -O2"
>>> CGO_CPPFLAGS=""
>>> CGO_CXXFLAGS="-g -O2"
>>> CGO_FFLAGS="-g -O2"
>>> CGO_LDFLAGS="-g -O2"
>>> PKG_CONFIG="pkg-config"
>>> GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 
>>> -fdebug-prefix-map=/tmp/go-build3825502007=/tmp/go-build 
>>> -gno-record-gcc-switches"
>>> 
>>> 
>>> I have tried many, many things yesterday, including:
>>> - changing the `Mode` in the config
>>> - using go/types directly ("can't find import" for the package I'm look 
>>> to parse)
>>> - using different importers
>>> - trying to load and parse different packages
>>> - ...
>>> 
>>> For context, and to avoid any XY problem here, my goal is to parse a 
>>> package and find an interface based on its name. Once this interface is 
>>> found, I need to range over its method set and generate a structure 
>>> implementing this interface, with (maybe not at the beginning but) a 
>>> lot of logic, e.g. detect that an interface returns an implementation 
>>> of itself, and so on.
>>> I'm working on a tool to generate mock structures from their interface 
>>> method set, as a personal project, and this is kind of the most 
>>> important part of it (being able to automatically generate the mock).
>>> 
>>> If anyone would kindly help me find what I'm doing wrong, or at least 
>>> point me to useful resources explaining how to fix my problem, I would 
>>> be reaally delighted. This has been a problem for days now... And I 

Re: [go-nuts] Re: Out of memory using golang.org/x/tools/go/packages

[Robert Engels]

If it is pretty printing it may need to determine the lowest level for 
indentation - so a depth first search - causing the entire tree to be traversed 
and retained before it can output anything. 

> On Jul 21, 2021, at 9:40 AM, Brian Candler  wrote:
> 
> But AFAICT, it should generate output as it runs.  The fact that it doesn't 
> generate any output at all is suspicious.
> 
>> On Wednesday, 21 July 2021 at 13:50:37 UTC+1 ren...@ix.netcom.com wrote:
>> Since litter checks for circular references it needs to keep a ref to every 
>> object it sees. 
>> 
>> With a large tree you will run out of memory. 
>> 
 On Jul 21, 2021, at 7:19 AM, jake...@gmail.com  wrote:
 
>>> 
>> 
>>> The first thing I would do is remove the call to litter, and see if that 
>>> solved the issue. That would tell you immediately if the problem was the 
>>> litter package or the packages package. I have so specific knowledge, but 
>>> it is not impossible to imagine that you are simply trying to print 
>>> something way to big for litter. 
>>> 
>>> After that, using pprof might be your next step.
>>> 
>>> Have you tried it on a really tiny package?
>>> 
> On Wednesday, July 21, 2021 at 6:41:39 AM UTC-4 mlevi...@gmail.com wrote:
> Hi all,
> 
> I'm having a very hard time with golang.org/x/tools/go/packages. Spent 
> most of my evening yesterday trying to understand what's happening here.
> Here's my code: https://play.golang.com/p/5L1N0lSaetB
> 
> With this very simple code I would expect that the program prints 
> detailed information about the package path I run it with. But whatever 
> the package, or module, I try to use this with, the program is killed 
> because it takes all the memory (~32GB) of my machine in a few seconds, 
> nothing ever gets printed...
> 
> Here's my config: 
> GO111MODULE=""
> GOARCH="amd64"
> GOBIN=""
> GOCACHE="/home/michel/.cache/go-build"
> GOENV="/home/michel/.config/go/env"
> GOEXE=""
> GOFLAGS=""
> GOHOSTARCH="amd64"
> GOHOSTOS="linux"
> GOINSECURE=""
> GOMODCACHE="/home/michel/.go/pkg/mod"
> GONOPROXY=""
> GONOSUMDB=""
> GOOS="linux"
> GOPATH="/home/michel/.go"
> GOPRIVATE=""
> GOPROXY="https://proxy.golang.org,direct;
> GOROOT="/usr/local/go"
> GOSUMDB="sum.golang.org"
> GOTMPDIR=""
> GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
> GOVCS=""
> GOVERSION="go1.16"
> GCCGO="gccgo"
> AR="ar"
> CC="gcc"
> CXX="g++"
> CGO_ENABLED="1"
> GOMOD="/dev/null"
> CGO_CFLAGS="-g -O2"
> CGO_CPPFLAGS=""
> CGO_CXXFLAGS="-g -O2"
> CGO_FFLAGS="-g -O2"
> CGO_LDFLAGS="-g -O2"
> PKG_CONFIG="pkg-config"
> GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 
> -fdebug-prefix-map=/tmp/go-build3825502007=/tmp/go-build 
> -gno-record-gcc-switches"
> 
> 
> I have tried many, many things yesterday, including:
> - changing the `Mode` in the config
> - using go/types directly ("can't find import" for the package I'm look 
> to parse)
> - using different importers
> - trying to load and parse different packages
> - ...
> 
> For context, and to avoid any XY problem here, my goal is to parse a 
> package and find an interface based on its name. Once this interface is 
> found, I need to range over its method set and generate a structure 
> implementing this interface, with (maybe not at the beginning but) a lot 
> of logic, e.g. detect that an interface returns an implementation of 
> itself, and so on.
> I'm working on a tool to generate mock structures from their interface 
> method set, as a personal project, and this is kind of the most important 
> part of it (being able to automatically generate the mock).
> 
> If anyone would kindly help me find what I'm doing wrong, or at least 
> point me to useful resources explaining how to fix my problem, I would be 
> reaally delighted. This has been a problem for days now... And I 
> can't find any relevant issue or blog as this is a peculiar context.
> 
> Thanks in advance to all that will read this and have a nice day! :D
 
>>> -- 
>>> You received this message because you are subscribed to the Google Groups 
>>> "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to golang-nuts...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/golang-nuts/36cd11af-ee05-4e9e-8324-51212c80d99cn%40googlegroups.com.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> 

Re: [go-nuts] Re: Out of memory using golang.org/x/tools/go/packages

[Robert Engels]

Since litter checks for circular references it needs to keep a ref to every 
object it sees. 

With a large tree you will run out of memory. 

> On Jul 21, 2021, at 7:19 AM, jake...@gmail.com  wrote:
> 
> 
> The first thing I would do is remove the call to litter, and see if that 
> solved the issue. That would tell you immediately if the problem was the 
> litter package or the packages package. I have so specific knowledge, but it 
> is not impossible to imagine that you are simply trying to print something 
> way to big for litter. 
> 
> After that, using pprof might be your next step.
> 
> Have you tried it on a really tiny package?
> 
>> On Wednesday, July 21, 2021 at 6:41:39 AM UTC-4 mlevi...@gmail.com wrote:
>> Hi all,
>> 
>> I'm having a very hard time with golang.org/x/tools/go/packages. Spent most 
>> of my evening yesterday trying to understand what's happening here.
>> Here's my code: https://play.golang.com/p/5L1N0lSaetB
>> 
>> With this very simple code I would expect that the program prints detailed 
>> information about the package path I run it with. But whatever the package, 
>> or module, I try to use this with, the program is killed because it takes 
>> all the memory (~32GB) of my machine in a few seconds, nothing ever gets 
>> printed...
>> 
>> Here's my config: 
>> GO111MODULE=""
>> GOARCH="amd64"
>> GOBIN=""
>> GOCACHE="/home/michel/.cache/go-build"
>> GOENV="/home/michel/.config/go/env"
>> GOEXE=""
>> GOFLAGS=""
>> GOHOSTARCH="amd64"
>> GOHOSTOS="linux"
>> GOINSECURE=""
>> GOMODCACHE="/home/michel/.go/pkg/mod"
>> GONOPROXY=""
>> GONOSUMDB=""
>> GOOS="linux"
>> GOPATH="/home/michel/.go"
>> GOPRIVATE=""
>> GOPROXY="https://proxy.golang.org,direct;
>> GOROOT="/usr/local/go"
>> GOSUMDB="sum.golang.org"
>> GOTMPDIR=""
>> GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
>> GOVCS=""
>> GOVERSION="go1.16"
>> GCCGO="gccgo"
>> AR="ar"
>> CC="gcc"
>> CXX="g++"
>> CGO_ENABLED="1"
>> GOMOD="/dev/null"
>> CGO_CFLAGS="-g -O2"
>> CGO_CPPFLAGS=""
>> CGO_CXXFLAGS="-g -O2"
>> CGO_FFLAGS="-g -O2"
>> CGO_LDFLAGS="-g -O2"
>> PKG_CONFIG="pkg-config"
>> GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 
>> -fdebug-prefix-map=/tmp/go-build3825502007=/tmp/go-build 
>> -gno-record-gcc-switches"
>> 
>> 
>> I have tried many, many things yesterday, including:
>> - changing the `Mode` in the config
>> - using go/types directly ("can't find import" for the package I'm look to 
>> parse)
>> - using different importers
>> - trying to load and parse different packages
>> - ...
>> 
>> For context, and to avoid any XY problem here, my goal is to parse a package 
>> and find an interface based on its name. Once this interface is found, I 
>> need to range over its method set and generate a structure implementing this 
>> interface, with (maybe not at the beginning but) a lot of logic, e.g. detect 
>> that an interface returns an implementation of itself, and so on.
>> I'm working on a tool to generate mock structures from their interface 
>> method set, as a personal project, and this is kind of the most important 
>> part of it (being able to automatically generate the mock).
>> 
>> If anyone would kindly help me find what I'm doing wrong, or at least point 
>> me to useful resources explaining how to fix my problem, I would be 
>> reaally delighted. This has been a problem for days now... And I can't 
>> find any relevant issue or blog as this is a peculiar context.
>> 
>> Thanks in advance to all that will read this and have a nice day! :D
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/36cd11af-ee05-4e9e-8324-51212c80d99cn%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/7F218DE8-684D-4E56-8170-845A6053B0EC%40ix.netcom.com.

Re: [go-nuts] Re: heap space not released by the end of program ?

[Robert Engels]

I knew I had seen something on that. :)

I guess Go doesn’t need a compacting collector if it can allocate most objects 
on the stack to avoid fragmentation. 

> On Jul 13, 2021, at 4:34 AM, Brian Candler  wrote:
> 
> 
>> On Tuesday, 13 July 2021 at 03:52:24 UTC+1 bai...@gmail.com wrote:
>> Hi, I wrote a test program, which tests when GC starts to return heap space 
>> to OS.
>> Here's the code: https://paste.ubuntu.com/p/jxHqDnsM2T/
>> 
>> But I've waited for a long time and haven't seen the RSS memory reduced. I 
>> can't figured out why, Any help ? (go version 1.14.3)
> 
> There was a change in go 1.16 which could affect this, so I suggest you try 
> the latest go.
> 
> Previously, go just marked the pages as "can be freed when required" - but 
> the OS would not actually free them until under memory pressure. Now it tells 
> the OS to free them immediately.
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/7c99fe43-376d-4f94-9070-17dafbe6baffn%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/508DA5F7-E15C-4E95-B957-16B469AB0AD6%40ix.netcom.com.

Re: [go-nuts] heap space not released by the end of program ?

[Robert Engels]

RSS is the amount of memory mapped to physical memory. It doesn’t include 
memory swapped. Still, if GC was releasing pages back to the OS - unmapping the 
allocated and touched pages - the RSS should decrease. 

There was a previous thread on if and when GC will release pages back to the 
OS. This is hard to do if you do not have a compacting collector - Go does not 
- so it is doubtful Go ever releases the pages once allocated to the heap. 

> On Jul 12, 2021, at 11:20 PM, Kurtis Rader  wrote:
> 
> 
> I didn't look at the source for your program because you  don't seem to 
> understand what "resident set size" means in this context. The RSS of a 
> process is unrelated to its virtual size. You appear to be equating RSS and 
> virtual size. This might be an example of the XY Problem.. What Prompted you 
> to open this discussion thread?
> 
>> On Mon, Jul 12, 2021 at 7:52 PM Chan Lewis  wrote:
>> Hi, I wrote a test program, which tests when GC starts to return heap space 
>> to OS.
>> Here's the code: https://paste.ubuntu.com/p/jxHqDnsM2T/
>> 
>> But I've waited for a long time and haven't seen the RSS memory reduced. I 
>> can't figured out why, Any help ? (go version 1.14.3)
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/a511d995-7f1d-431d-9065-6b6d287a19dan%40googlegroups.com.
> 
> 
> -- 
> Kurtis Rader
> Caretaker of the exceptional canines Junior and Hank
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CABx2%3DD_BMdKn_du_REFU0NEyQaBSM979%2BHTGFf3JWmvvohNqJw%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/F1AC03F0-D92D-4087-BF13-5093A5A20DB1%40ix.netcom.com.

Re: [go-nuts] Re: Stack growing is inefficient

[Robert Engels]

If you are doing “anything” which such large objects - doesn’t that dwarf the 
time to increase the stack?

> On Jul 1, 2021, at 9:28 AM, tapi...@gmail.com  wrote:
> 
> 
> I tried to find a way to initialize the size of the stack of a new created 
> goroutine in advance to avoid several stack copies.
> Before this fix, the efficiencies of the two new goroutines are almost the 
> same.
> After the fix, the second goroutine uses much less time than the first one.
> 
> package main
> 
> import "fmt"
> import "time"
> 
> const N = 1024 * 1024
> var n byte = 123
> 
> func f(v [N]byte, n int) {
> if n > 0 {
> f(v, n-1)
> }
> }
> 
> func main() {
> var c = make(chan time.Duration, 1)
> go func() {
> start := time.Now()
> f([N]byte{}, 32)
> c <- time.Since(start)
> }()
> fmt.Println(<-c)
> go func() {
> start := time.Now()
> if n == 0 {
> var a, b, c, d, e, f, g [N*10]byte
> n = a[n] + b[n] + c[n] + d[n] + e[n] + f[n] + g[n]
> }
> f([N]byte{}, 32)
> c <- time.Since(start)
> }()
> fmt.Println(<-c)
> }
> 
> 
> 
>> On Thursday, July 1, 2021 at 2:23:45 AM UTC-4 axel.wa...@googlemail.com 
>> wrote:
>> Okay, *now* I get what you are trying to say. I agree that it seems 
>> inefficient to call it more than once, which is why the code tries to 
>> optimize for that. I don't know why that optimization doesn't trigger in 
>> your case - you might want to try and investigate that. There might be a 
>> good reason why it doesn't (for example, I'll note that your code might be 
>> using the system stack, which, AIUI, is special).
>> 
>> As always, if you are experiencing a real problem due to this, you might 
>> want to open an issue.
>> 
>>> On Thu, Jul 1, 2021 at 3:04 AM tapi...@gmail.com  wrote:
>>> 
>>> 
 On Wednesday, June 30, 2021 at 8:46:19 PM UTC-4 axel.wa...@googlemail.com 
 wrote:
> On Thu, Jul 1, 2021 at 2:34 AM tapi...@gmail.com  
> wrote:
> 
> 
>> On Wednesday, June 30, 2021 at 11:56:45 AM UTC-4 Brian Candler wrote:
>> So I think what you're asking is, "under what scenario does the code in 
>> L1066-1069 get run?" - is that right?
> 
> Almost true.
> 
> Whether or not it should run, growing the stack from 2048 to 512M in 18+ 
> steps looks not right.
 
 Why? 2048 • 2^18 = 2^11 • 2^18 = 2^29 = 536870912.
 Seems like exactly the expected result.
>>> 
>>> It looks each step calls copystack once.
>>> Isn't one step more efficient?
>>>  
  
 
>  
>> 
>>> On Wednesday, 30 June 2021 at 14:21:21 UTC+1 tapi...@gmail.com wrote:
>>> Sorry, I post the wrong anchor. It is line 1068: 
>>> https://github.com/golang/go/blob/d19a53338fa6272b4fe9c39d66812a79e1464cd2/src/runtime/stack.go#L1068
>>> 
 On Wednesday, June 30, 2021 at 5:08:30 AM UTC-4 Brian Candler wrote:
> On Wednesday, 30 June 2021 at 08:25:59 UTC+1 tapi...@gmail.com wrote:
> 
> It looks this line 
> https://github.com/golang/go/blob/master/src/runtime/stack.go#L1078 
> never gets executed.
> 
 
 Can you quote the line you're referring to?  Line numbers can shift up 
 and down as commits are made to the master branch. Right now, L1078 is 
 a blank line.
> 
 
> -- 
 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts...@googlegroups.com.
 
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/5fab5a6e-5dd5-4df2-8b31-4a51aa825f92n%40googlegroups.com.
>>> 
>>> -- 
>>> You received this message because you are subscribed to the Google Groups 
>>> "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to golang-nuts...@googlegroups.com.
>> 
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/golang-nuts/a06d022b-06c2-4379-b33b-56886ebaf1dbn%40googlegroups.com.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/42aabf7f-7aa5-43f9-915b-f49f41f8856fn%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/8AA89F7E-C48F-4CD1-8607-B1FA99C92423%40ix.netcom.com.

Re: [go-nuts] Re: How to implement this in golang?

[Robert Engels]

With Go you use a Go routine per connection and then you can read in chunks and 
delay in an imperative style. No need for generators or async. 

> On Jun 29, 2021, at 10:25 AM, LetGo  wrote:
> 
> Thanks for the answer! (:
> In python it was straightforward to implement and it works like a charm. It 
> sends small packets with delay between each other without even care if it is 
> UDP or TCP:
> 
> 
> 
> 
> 
> 
> The connection ( if  UDP/ TCP )  is handled independently, the sending part ( 
> the one I pasted ) is equal for both of them, its just an "if" in the code.
> The server ( which I control )  does riassembling by himself, so its fine.
> I know that providing a solution is not an easy task, I know it for sure.
> I'm just looking for a pro ( a saint ahaha ) with some patience that provides 
> an example of code of how it should look like to work as I wish it does.
> I do not expect do make it work in first place, its almost impossible, but we 
> all know, with some errors you get the solution before or after(:
> In this case I don't really know where to start and how to do it ( im a 
> newbie in golang ), so I came here to ask help
> 
> Il giorno martedì 29 giugno 2021 alle 14:22:32 UTC+2 Brian Candler ha scritto:
>> What sort of socket - TCP or UDP?  TCP is an infinite stream, and there is 
>> no reliable way to divide it into "chunks" (except possibly using the URG 
>> pointer, which is very rarely used).  There is no guarantee that waiting 1.6 
>> seconds between sends will actually send in separate segments being sent, 
>> nor that the receiver will be able to determine that these were separate 
>> segments - they may well be merged.
>> 
>> To do it reliably, you'd want to add your own framing, e.g. send an 4-byte 
>> length followed by that number of bytes.
>> 
>> If you show a *complete* go program, together with a description of how it 
>> isn't working to your satisfaction, then maybe someone can help you fix it.  
>> If you want to make it behave the same as some given python program, then 
>> maybe you should show the python program as well.
>> 
>> On Tuesday, 29 June 2021 at 11:51:57 UTC+1 LetGo wrote:
>>> I forgot this:
>>> // Manage connection for different behavior
>>> func handleConn(conn net.Conn, binPath string) {
>>> 
>>> msg := "Status?\n"
>>> 
>>> if binPath != "" {
>>> // Execute command and send Standard I/O net.Conn
>>> cmd := exec.Command(binPath)
>>> cmd.Stdin = conn
>>> cmd.Stdout = conn
>>> cmd.Stderr = conn
>>> cmd.Run()
>>> fmt.Println("Disconnected!?")
>>> time.Sleep(5 * time.Second)
>>> pippo, err2 := conn.Write([]byte(msg))
>>> if err2 != nil {
>>>fmt.Println(pippo)
>>> }
>>> 
>>> 
>>>   
>>> 
>>> } else {
>>> // Copy Standard I/O in a net.Conn
>>> go io.Copy(os.Stderr, conn)
>>> go io.Copy(os.Stdout, conn)
>>> io.Copy(conn, os.Stdin)
>>> 
>>> }
>>> }
>>> 
>>> Il giorno martedì 29 giugno 2021 alle 12:27:00 UTC+2 LetGo ha scritto:
 I have a proxy written in python with some logic that I would like to 
 implement in a golang tool of mine, but I don't really know how to do it.
 It splits the data sent via socket (stdout) in small chunks, with a delay 
 between each other
 
 I have a variable which gets a random number from a list:
 
  listbytes = [87, 88, 89, 90]
 
 
 I have also a function which splits in small chunk of bytes(n) the 
 data(lst):
 
 
 def chunks(lst, n):
 
 "Yield successive chunks from lst, where n is a list of possible sizes"
 
 i = 0
 
 while i < len(lst):
 
 k = min(random.choice(n), len(lst) - i)
 
 yield lst[i:i + k]
 
 i += k
 
 .
 
 Both things are executed this way:
 ...
 # get the data
 data = s.recv(BUFFER_SIZE)
  
 if s == s_src:
 d = LOCAL_DATA_HANDLER(data)
 for chunk in chunks(d, listbytes): 
 #sleep 1.6 seconds before sending the next chunk of data
 time.sleep(1.6)
 #send the chunked data
 s_dst.send(bytes(chunk) )
 ... 
 
 
 How do I implement the same exact logic illustrated here?
 
 func (nObj NetObject) RunClient(cmd string) {
 // Try connection
 
 for {
 conn, err := net.Dial(nObj.Type, nObj.Service)
 fmt.Print("connect")
 //msg := "status"
 if err != nil {
 fmt.Println("fail")
 } 
 if err == nil {
 fmt.Println("ok")
 //defer conn.Close()
 defer conn.Close()

Re: [go-nuts] Go and GPUs

[Robert Engels]

Agreed. (Didn’t see your previous message before I sent that). 

> On Jun 25, 2021, at 9:25 PM, David Riley  wrote:
> 
> On Jun 25, 2021, at 10:23 PM, Robert Engels  wrote:
>> 
>> There is also a LOT of support for Java and CUDA/OpenCL. You can essentially 
>> reimplement them Java portion in Go. There are multiple open source projects 
>> in this area. 
>> 
>> Might be a lot easier than starting from scratch. 
> 
> Yes, CGo is going to be similar to JNI in this context.
> 
> 
> - Dave

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/A8F2881B-7CC9-4E4A-9945-23F9D55BF5D0%40ix.netcom.com.

Re: [go-nuts] Go and GPUs

[Robert Engels]

There is also a LOT of support for Java and CUDA/OpenCL. You can essentially 
reimplement them Java portion in Go. There are multiple open source projects in 
this area. 

Might be a lot easier than starting from scratch. 

> On Jun 25, 2021, at 8:03 PM, Michael Poole  wrote:
> 
> On Fri, Jun 25, 2021 at 11:52 AM Nikolay Dubina wrote:
>> 
>> I tried to write down my own CUDA / NVIDIA GPU driver in native Go last 
>> weekend. To my great surprise, CUDA and pretty much all high performance 
>> software/hardware from NVIDIA is proprietary close-source C/C++ code. 
>> Meaning, you can't write native Go driver even if you wanted to. Forget Go, 
>> people are trying to reverse engineer it in C/C++ with limited success. From 
>> what I heard OpenCV is not a priority for NVIDIA either. Then I looked up 
>> what Apple is doing with their Neural Engine in latest chips. It too is 
>> closed-source Objective-C, Swift. I suspect situation with other powerful 
>> hardware is the same. Moore's law seem to be about GPU lately, and everyone 
>> is locking it in. Kind of not in the spirit open-source and Linux. That's 
>> quite bad state of affairs for Go and computing in general. Yikes!
>> 
>> Just want to see what others are thinking.
> 
> That would be a very nice thing to have.  I see four basic areas where
> this becomes tricky or requires a lot of work.
> 
> 1) Getting memory management right for sharing with the accelerator
> device.  Shared memory buffers need to stay locked for longer than
> just the call to a library or to the OS kernel, but in the worse case
> could use something like Manish Rai Jain of Dgraph described at
> https://dgraph.io/blog/post/manual-memory-management-golang-jemalloc/
> .  A lot of Nvidia's recent CUDA programmer-productivity improvements
> have focused around transparent data movement, and Go's garbage
> collector probably breaks the assumptions for those.
> 
> 2) "Friendly" source development (like CUDA C) integrates the host and
> target code into a single code base, with some kind of markup or
> compiler hint about which functions should be compiled for the
> accelerator, and which function calls need to be treated specially.
> Those "special" function calls must be translated to dispatch the call
> (usually with additional arguments like the grid/NDRange parameters)
> to the accelerator rather than as a normal function call.  This is a
> compiler front-end problem, which is almost certainly a lot easier
> with Go than with C or C++, but still requires attention and perhaps
> considerable effort because it requires multiple back-ends to run for
> some functions.  In the worst case, do like OpenCL and require the
> programmer to provide or build strings containing program text, along
> with verbose code to set up calls to the accelerator.
> 
> 3) Generating code for the accelerator.  SPIR-V is an obvious
> candidate for portable uses; it integrates reasonably with current
> OpenCL as well as Vulkan.  Nvidia makes a reasonable compromise for
> this with their PTX assembly pseudo-language: They have good
> documentation about which PTX instructions are only supported on some
> GPU generations, they document when the translation to actual machine
> code varies in major ways, and they even have a decent API for
> compiling application-provided PTX code.  This is a compiler back-end
> problem, conditional on not accepting the "worst case" in #2 above.
> 
> 4) Integrating target libraries with the rest of an application.  A
> large part of the value proposition for CUDA is that it has a lot of
> highly optimized libraries available out of the box: cuFFT,
> cuBLAS/NVBLAS, and more.  These are a hybrid between GPU elements and
> host elements, and a lot of the host elements end up being black boxes
> with respect to other languages.  The most general fix is to call out
> to C, which is not satisfying for portability.
> 
> If I were going to spend the time on this, I would probably target
> SPIR-V with Vulkan or OpenCL for portability rather than Nvidia's PTX
> or Apple's Metal.  AMD, Nvidia and Google (Android) all have good
> support for SPIR-V through their Vulkan stacks, and there are
> third-party Vulkan layers that run on MacOS.
> 
> - Michael Poole
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAOU-OAK3dUn0hDAvEt3ayhQO4Ryg2-bMpaKZTCxW086%2BYFFnpQ%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 

Re: [go-nuts] Go and GPUs

[Robert Engels]

Why not develop a Go <> CUDA binding using CGo?

> On Jun 25, 2021, at 12:11 PM, Marcin Romaszewicz  wrote:
> 
> 
> Graphics chips have a lot of proprietary IP, some of which the manufacturers 
> would like to keep secret. If you see source for one of these drivers, you 
> will have a good idea about the hardware organization, so they keep 
> everything secret. It stinks for us developers who want to write cross 
> platform open source. The best bet right now, in my opinion, is to write CGO 
> wrappers around platform native libraries, and sadly, they'll only work on 
> some OS/hardware combinations.
> 
> 
>> On Fri, Jun 25, 2021 at 8:53 AM Nikolay Dubina 
>>  wrote:
>> I tried to write down my own CUDA / NVIDIA GPU driver in native Go last 
>> weekend. To my great surprise, CUDA and pretty much all high performance 
>> software/hardware from NVIDIA is proprietary close-source C/C++ code. 
>> Meaning, you can't write native Go driver even if you wanted to. Forget Go, 
>> people are trying to reverse engineer it in C/C++ with limited success. From 
>> what I heard OpenCV is not a priority for NVIDIA either. Then I looked up 
>> what Apple is doing with their Neural Engine in latest chips. It too is 
>> closed-source Objective-C, Swift. I suspect situation with other powerful 
>> hardware is the same. Moore's law seem to be about GPU lately, and everyone 
>> is locking it in. Kind of not in the spirit open-source and Linux. That's 
>> quite bad state of affairs for Go and computing in general. Yikes!
>> 
>> Just want to see what others are thinking.
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/0913f098-700a-443f-bd02-2db7ad2408a6n%40googlegroups.com.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CA%2Bv29LvH8eZqX8ASryOgBexMVUUyDO%2BTQFtaxz8Y-EDi92hkEA%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/3D3DA44D-BEA2-4D05-9484-C2B9A573ED45%40ix.netcom.com.

Re: [go-nuts] unexpected stuck in sync.(*Pool).Get()

[Robert Engels]

With a 500k machine cluster I suggest getting professional Go support - someone 
experienced in troubleshooting that can sit with you and review the code and 
configuration to diagnose the issue. 

Personally it sounds like overallicated machines causing thrashing delays in 
the context switching.  

> On Jun 22, 2021, at 9:21 AM, Peter Z  wrote:
> 
> Sorry for a mistake: 'hyperthread closed', hyperthread is actually on.
> 
> 在2021年6月22日星期二 UTC+8 下午10:01:48 写道：
>> I just checked the monitor data and found that the machine suffered from 
>> high 'load average'(about 30+) at approximately the time the agent get stuck.
>> A 24 cores(2 CPUs * 14 cores), hyperthread closed machine with load average 
>> over 30 seems bad. But after the load average got down to below 1, the 
>> agent process still hung there.
>> 
>> 2021-6-22 Tue UTC+8 pm 8:06:55 ：
>>> Sorry, now I am completely confused. 
>>> 
> So, you have about 500,000 processes running this agent on each machine, 
> and each process has around 7,000 gorouines? Is that correct?
 
 Yes, that's  exactly what I mean. 
>>> 
>>> but then you say:  "Only one process per machine".
>>> 
>>> Is there a language barrier, or am I missing something?
>>> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/1732ca49-e526-4ec4-ac29-0f9a4b2f949cn%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/24CB21C1-0DA5-4FE7-A824-0D406AE93CDB%40ix.netcom.com.

Re: [go-nuts] unexpected stuck in sync.(*Pool).Get()

[Robert Engels]

He is stating he has a cloud cluster consisting of 500k machines - each machine 
runs an agent process - each agent has 7000 Go routines. 

> On Jun 22, 2021, at 7:07 AM, jake...@gmail.com  wrote:
> 
> 
> Sorry, now I am completely confused. 
> 
>>> So, you have about 500,000 processes running this agent on each machine, 
>>> and each process has around 7,000 gorouines? Is that correct?
>> 
>> Yes, that's  exactly what I mean. 
> 
> but then you say:  "Only one process per machine".
> 
> Is there a language barrier, or am I missing something?
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/e00c87d5-07b2-42ae-b295-880da866dc1cn%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/E933C864-6CBC-4A66-9F56-5B4E7EA80F19%40ix.netcom.com.

Re: [go-nuts] unexpected stuck in sync.(*Pool).Get()

[Robert Engels]

How many processes per machine? It seems like scheduling latency to me. 

> On Jun 21, 2021, at 6:31 AM, Peter Z  wrote:
> 
> 
>> So, you have about 500,000 processes running this agent on each machine, and 
>> each process has around 7,000 gorouines? Is that correct?
> 
> Yes, that's  exactly what I mean. 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/186e4953-93ae-456a-aa09-cfdb59a17ef5n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/C3C643CB-9FA1-4181-8E1F-BC649C656D5F%40ix.netcom.com.

Re: [go-nuts] unexpected stuck in sync.(*Pool).Get()

[Robert Engels]

You’re right. Inspecting the code it is internally partitioned by P. 

I agree that it looks like the pool is being continually created. 

> On Jun 17, 2021, at 12:18 PM, Ian Lance Taylor  wrote:
> 
> On Thu, Jun 17, 2021 at 9:19 AM Peter Z  wrote:
>> The original post is on  stackoverflow   
>> https://stackoverflow.com/questions/67999117/unexpected-stuck-in-sync-pool-get
>> Golang ENV:
>> go1.14.3 linux/amd64
>> Description:
>> We have about half a million agents running on each of our machines.The 
>> agent is written in Go. Recently we found that the agent may get stuck, no 
>> response for the sent requests. The metrics exported from the agent show 
>> that a channel in the agent(caching the request) is full. Deep into the 
>> goroutine stacks, we found that the goroutines consuming messages from the 
>> channel are all waiting for a lock.The goroutines Stack details are shown 
>> below.
> 
> That is peculiar.  What is happening under the lock is that the pool
> is allocating a slice that is GOMAXPROCS in length.  This shouldn't
> take long, obviously.  And it only needs to happen when the pool is
> first created, or when GOMAXPROCS changes.  So: how often do you
> create this pool?  Is it the case that you create the pool and then
> have a large number of goroutines try to Get a value simultaneously?
> Or, how often do you change GOMAXPROCS?  (And, if you do change
> GOMAXPROCS, why?)
> 
> 
>> The stack shows that all of the goroutines are waiting for the global lock 
>> in sync.Pool. But I can't figure out which gouroutine is holding the lock. 
>> There should be a gouroutine which has `sync.runtime_SemacquireMutex` in 
>> it's stack not at the top, but there isn't.
> 
> I don't think that is what you would see.  I think you would see a
> goroutine with pinSlow in the stack but with SemaquireMutex not in the
> stack.
> 
> 
>> Reproduce:
>> Can't find a way to reproduce this problem for now.
> 
> It's going to be pretty hard for us to solve the problem without a reproducer.
> 
> Ian
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAOyqgcW89OSL9frH%3D7f_gMga8qiKQ%2BR6cRnnWK3Kr5dKh3yLsg%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/EAB0653C-F779-46AB-B41F-CFF323455257%40ix.netcom.com.

Re: [go-nuts] unexpected stuck in sync.(*Pool).Get()

[Robert Engels]

You probably need multiple pools in and partition them. 500k accessors of a 
shared lock is going to have contention. 

github.com/robaho/go-concurrency-test might be helpful. 

> On Jun 17, 2021, at 11:19 AM, Peter Z  wrote:
> 
> 
> The original post is on  stackoverflow   
> https://stackoverflow.com/questions/67999117/unexpected-stuck-in-sync-pool-get
> 
> Golang ENV: 
> go1.14.3 linux/amd64
> 
> Description:
> We have about half a million agents running on each of our machines.The agent 
> is written in Go. Recently we found that the agent may get stuck, no response 
> for the sent requests. The metrics exported from the agent show that a 
> channel in the agent(caching the request) is full. Deep into the goroutine 
> stacks, we found that the goroutines consuming messages from the channel are 
> all waiting for a lock.The goroutines Stack details are shown below.
> 
> // Consuming Goroutine
> 166 @ 0x438cd0 0x4497e0 0x4497cb 0x449547 0x481c1c 0x482792 0x482793 
> 0x4824ee 0x4821af 0x520ebd 0x51fdff 0x51fcd0 0x737fb4 0x73a836 0x73a813 
> 0x97d660 0x97d60a 0x97d5e9 0x4689e1
> #   0x449546sync.runtime_SemacquireMutex+0x46 
>   
> /home/ferry/ONLINE_SERVICE/other/ferry/task_workspace/gopath/src/**/go-env/go1-14-linux-amd64/src/runtime/sema.go:71
> #   0x481c1bsync.(*Mutex).lockSlow+0xfb   
>   
> /home/ferry/ONLINE_SERVICE/other/ferry/task_workspace/gopath/src/**/go-env/go1-14-linux-amd64/src/sync/mutex.go:138
> #   0x482791sync.(*Mutex).Lock+0x271  
>   
> /home/ferry/ONLINE_SERVICE/other/ferry/task_workspace/gopath/src/**/go-env/go1-14-linux-amd64/src/sync/mutex.go:81
> #   0x482792sync.(*Pool).pinSlow+0x272
>   
> /home/ferry/ONLINE_SERVICE/other/ferry/task_workspace/gopath/src/**/go-env/go1-14-linux-amd64/src/sync/pool.go:213
> #   0x4824edsync.(*Pool).pin+0x5d 
>   
> /home/ferry/ONLINE_SERVICE/other/ferry/task_workspace/gopath/src/**/go-env/go1-14-linux-amd64/src/sync/pool.go:206
> #   0x4821aesync.(*Pool).Get+0x2e 
>   
> /home/ferry/ONLINE_SERVICE/other/ferry/task_workspace/gopath/src/**/go-env/go1-14-linux-amd64/src/sync/pool.go:128
> #   0x520ebcgo.uber.org/zap/zapcore.getCheckedEntry+0x2c  
>   
> /home/ferry/ONLINE_SERVICE/other/ferry/task_workspace/gopath/pkg/mod/go.uber.org/zap@v1.9.1/zapcore/entry.go:45
> #   0x51fdfe
> go.uber.org/zap/zapcore.(*CheckedEntry).AddCore+0x18e 
>   
> /home/ferry/ONLINE_SERVICE/other/ferry/task_workspace/gopath/pkg/mod/go.uber.org/zap@v1.9.1/zapcore/entry.go:240
> #   0x51fccfgo.uber.org/zap/zapcore.(*ioCore).Check+0x5f  
>   
> /home/ferry/ONLINE_SERVICE/other/ferry/task_workspace/gopath/pkg/mod/go.uber.org/zap@v1.9.1/zapcore/core.go:80
> #   0x737fb3go.uber.org/zap.(*Logger).check+0x153 
>   
> /home/ferry/ONLINE_SERVICE/other/ferry/task_workspace/gopath/pkg/mod/go.uber.org/zap@v1.9.1/logger.go:269
> #   0x73a835go.uber.org/zap.(*Logger).Check+0x85  
>   
> /home/ferry/ONLINE_SERVICE/other/ferry/task_workspace/gopath/pkg/mod/go.uber.org/zap@v1.9.1/logger.go:172
> #   0x73a812go.uber.org/zap.(*SugaredLogger).log+0x62 
>   
> /home/ferry/ONLINE_SERVICE/other/ferry/task_workspace/gopath/pkg/mod/go.uber.org/zap@v1.9.1/sugar.go:233
> #   0x97d65fgo.uber.org/zap.(*SugaredLogger).Infof+0x30f  
>   
> /home/ferry/ONLINE_SERVICE/other/ferry/task_workspace/gopath/pkg/mod/go.uber.org/zap@v1.9.1/sugar.go:138
> #   0x97d609**/log.Infof+0x2b9
>
> /home/ferry/ONLINE_SERVICE/other/ferry/task_workspace/gopath/pkg/mod/**/common@v0.0.0-20210323102343-4a6074e63e74/log/log.go:71
> #   0x97d5e8**/api.(*xxHeadServer).workerProcMsg+0x298   
> /home/ferry/ONLINE_SERVICE/other/ferry/task_workspace/gopath/src/**/xxhead_server.go:350
> 
> // Consuming Goroutine
> 119 @ 0x438cd0 0x4497e0 0x4497cb 0x449547 0x481c1c 0x482792 0x482793 
> 0x4824ee 0x4821af 0x5269e1 0x5269d2 0x526892 0x51f6cd 0x51f116 0x51ff39 
> 0x5218e7 0x73a8b0 0x97d660 0x97d60a 0x97d5e9 0x4689e1
> #   0x449546sync.runtime_SemacquireMutex+0x46 
>   
> /home/ferry/ONLINE_SERVICE/other/ferry/task_workspace/gopath/src/**/go-env/go1-14-linux-amd64/src/runtime/sema.go:71
> #   0x481c1b

Re: [go-nuts] Knowing from documentation whether an interface is holding a pointer or a struct?

[Robert Engels]

I think the playground code I supplied is essentially a test case - and it 
works in the absence of concurrency or other held references (like putting it 
in a map). 

I guess the bottom line for me is that faq says do not mix receiver types. 
There is either a valid reason for this or it should be removed. I think it is 
the former as it makes more usage patterns resilient to hard to track bugs. 

I must be very dense but I still don’t see the rationale behind not stating : 
use value receivers for immutable types and pointer receivers otherwise. There 
may be some fringe cases where this doesn’t hold - but if you can write the 
entire stdlib with this pattern it seems fairly stable. 

> On Jun 8, 2021, at 5:03 PM, 'Dan Kortschak' via golang-nuts 
>  wrote:
> 
> On Tue, 2021-06-08 at 07:57 -0500, robert engels wrote:
>> The following code is works fine from the developers perspective:
>> 
>> https://play.golang.org/p/gC1XsSLvovM
>> 
>> The developer says, oh cool, I see this great new 3P library that
>> does background logging - I want to use that instead. Hey, I already
>> implement the EventLogger interface, so no problem, I take out my
>> manual logging code, and make a call to recordEvents(EventLogger).
>> 
>> Hmm, my program isn’t logging properly any more. Oh, it says
>> background logging - that likely means concurrency - so maybe I have
>> a race condition, so I run it under —race. Hmmm, says it's fine and
>> it still isn’t working. :(
>> 
>> Eventually the developer figures out that the call to recordEvents()
>> is making a copy, and so needs pointer receiver and to create a
>> pointer based interface reference (and add the atomic calls). It’s
>> not clear to me how the library author would document things to avoid
>> this scenario.
>> 
>> If you don’t see that the above is suboptimal and an AFI I am not
>> sure what else I can say. That a person of your caliber writes code
>> that differs from the recommendation in the FAQ (the mixing of
>> receiver types) is a flag to me that one of the approaches is
>> probably not correct.
>> 
>> ‘go vet’ already has the ability to disable certain checks. Adding a
>> check to ‘go vet’ to detect mixed receiver types (which the FAQ says
>> is not recommended) seems reasonable and will make life easier for
>> many beginner Go programmers - and some seasoned ones as well :)
>> 
>> The duplicitous and transparent nature of pointer/value receivers and
>> interfaces is a source of confusion. I think being explicit would
>> have been a better choice here but that horse has left the barn.
> 
> 
> An error like this would be found in any reasonable set of tests for
> the type. Minimally, if the type is intended to be passed to functions
> in order to be able to be used, a test would involve that and that
> would immediately surface the error.
> 
> What you seem to be asking for is a way for the language to enforce
> contracts on behaviour from the source and you seem to put that at the
> feet of interface handling. However, it is clearly something that
> arises without needing interfaces in the language; a user of the
> MyEventRecorder type would already find problems if they pass the value
> to a function without needing to do that in an interface value. Without
> significant addition of annotations for implementations it is not
> possible for the language to understand the contractual obligations of
> an implementation, and these kinds of annotations are in opposition to
> the design of the language. Other languages do have these kinds of
> marks, but they are not Go (thankfully).
> 
> In terms of documentation of expectations of an interface if that were
> the situation, there are examples in the standard library where it's
> described that implementations must be able to mutate themselves — be a
> pointer receiver (or other reference-like type).
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/cce7af413d7cb56b4fd653703f1d49d6f5f7e7c6.camel%40kortschak.io.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/578FCA64-1551-4902-B09F-2269A286C32D%40ix.netcom.com.

Re: [go-nuts] Knowing from documentation whether an interface is holding a pointer or a struct?

[robert engels]

OK, I will try one more time.

The following code is works fine from the developers perspective:

https://play.golang.org/p/gC1XsSLvovM 

The developer says, oh cool, I see this great new 3P library that does 
background logging - I want to use that instead. Hey, I already implement the 
EventLogger interface, so no problem, I take out my manual logging code, and 
make a call to recordEvents(EventLogger).

Hmm, my program isn’t logging properly any more. Oh, it says background logging 
- that likely means concurrency - so maybe I have a race condition, so I run it 
under —race. Hmmm, says it's fine and it still isn’t working. :(

Eventually the developer figures out that the call to recordEvents() is making 
a copy, and so needs pointer receiver and to create a pointer based interface 
reference (and add the atomic calls). It’s not clear to me how the library 
author would document things to avoid this scenario.

If you don’t see that the above is suboptimal and an AFI I am not sure what 
else I can say. That a person of your caliber writes code that differs from the 
recommendation in the FAQ (the mixing of receiver types) is a flag to me that 
one of the approaches is probably not correct.

‘go vet’ already has the ability to disable certain checks. Adding a check to 
‘go vet’ to detect mixed receiver types (which the FAQ 
 says is not 
recommended) seems reasonable and will make life easier for many beginner Go 
programmers - and some seasoned ones as well :)

The duplicitous and transparent nature of pointer/value receivers and 
interfaces is a source of confusion. I think being explicit would have been a 
better choice here but that horse has left the barn.

> On Jun 8, 2021, at 1:54 AM, 'Axel Wagner' via golang-nuts 
>  wrote:
> 
> On Tue, Jun 8, 2021 at 6:36 AM Marvin Renich  > wrote:
> You say that test cases of Log work fine, but they are only fine in a
> non-concurrent environment.  The instant you test Log (without
> interfaces) in a concurrent program it fails in an obvious manner.
> 
> nit: I don't think concurrency has anything to do with it either. The failure 
> mode is making a copy and expecting the copy and the original to share 
> memory. If anything, concurrency (in a test) would make it more likely to get 
> hidden, by increasing the likelihood that a closure is used, which implicitly 
> shares a pointer: https://play.golang.org/p/Gwj9GScjQBJ 
> 
> 
> Of course, concurrency then also makes the failure easy to see, as long as 
> you remember to run your tests with `-race`.
>  
> FWIW I agree with Robert that it's relatively easy to write a test for this 
> that never copies the value (though even then, if you think about using 
> atomics you definitely should think about writing a concurrent test and 
> running it with `-race` enabled, which should show the problem).
> I disagree with him, however, that interfaces make it more likely to run into 
> the problem when *using* the code. Any even remotely realistic usage of that 
> code is broken. Even if you failed to write tests which surface that breakage.
> 
> 
> Interfaces, and their ability to potentially be satisfied by either a
> non-pointer type or a pointer type, depending on the type, is not the
> problem here.
> 
> ...Marvin
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/20210608043609.gguu3t3hqbsziema%40basil.wdw
>  
> .
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAEkBMfGa5B99OO8oTxJOWFSuJPinPF3b1QJWEuqMD4ZPAiCi%3DQ%40mail.gmail.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/A2CBA1A9-5342-4AAF-8393-A0DD96EC832A%40ix.netcom.com.

Re: [go-nuts] Knowing from documentation whether an interface is holding a pointer or a struct?

[Robert Engels]

Sorry - correct link. I missed the subtle change.

> On Jun 7, 2021, at 8:18 PM, Robert Engels  wrote:
> 
> 
> (I think you pasted the wrong link - that is my code). 
> 
> It is not about being unwilling to admit it. Your explanation/reasoning has 
> not convinced me. 
> 
> Imagine some library declares the EventLogger interface as shown. Acceptable. 
> Someone writes the RecordEvents() method taking an EventLogger. Acceptable. 
> 
> Now, I have a struct I want to use with as an EventLogger (badly named - 
> really EventSource). The code I wrote works fine. Test cases (of Log()) work 
> fine. It fails when used as a source to RecordEvents() (and similar held 
> reference patterns). 
> 
> How do you protect against this? What is the mechanism as a library author?
> 
> Clearly this is a trivial example but similar patterns are everywhere. 
> 
> Compare the Go interface handling with Java’s everything is a reference - 
> much simpler - and then adding value types that are explicit. Or a similar 
> implementation in Rust. In both cases knowing you wrote a correct 
> implementation is much easier. Java has since added annotations for aspects 
> like “thread safe” that cover the atomic aspects. 
> 
> I like Go. A lot. I’ve designed and built systems with millions of LOC. 
> Pointing out aspects that might benefit from changes should be encouraged - 
> if not it’s a religion not a programming language. 
> 
>>> On Jun 7, 2021, at 7:40 PM, Axel Wagner  
>>> wrote:
>>> 
>> 
>> 
>> 
>>> On Tue, Jun 8, 2021 at 2:05 AM Robert Engels  wrote:
>>> 
>>> We agree. It needs a pointer receiver to work. The atomic is also needed in 
>>> this case for background logging. 
>>> 
>>> The problem in this case is that recordEvents() has to document that the  
>>> EventLogger passed to recordEvents() must have a pointer receiver for the 
>>> Log() method. There is nothing in the language that allows me to declare it 
>>> nor the compiler to enforce it.
>> 
>> It is possible to write a working implementation of that interface without a 
>> pointer receiver - it just needs to *contain* a pointer: 
>> https://play.golang.org/p/Xm6ASGcCyhR
>> You could also have a slice type, which also can do modifications without a 
>> pointer receiver. Or a map-type. Or a channel.
>> 
>> If you would restrict an interface to require pointer-receiver, you would 
>> wrongly restrict the implementer from all these possibilities.
>> 
>> As is the common wisdom, the user of an interface should not care what the 
>> concrete type implementing an interface is (except if it needs to do a 
>> type-assertions). It's the same wisdom that applies to people wanting to 
>> check if an interface contains a nil-pointer: That check relies on the 
>> assumption that the interface contains a pointer, which shouldn't be nil and 
>> that's not something that should concern the user of an interface.
>> 
>> Again, to be abundantly clear (you still seem unwilling to acknowledge 
>> this): The problem with your code is not the definition or usage of the 
>> interface. It's the definition of the method that is wrong. The 
>> interface-definition is fine and works fine.
>> 
>>> If you don’t see this as suboptimal and an area for improvement I am not 
>>> sure what else I can say.
>> 
>> I just want to state again, clearly, that all I objected to was you calling 
>> this "the most inconsistent and obtuse aspect of the Go language", which I 
>> perceived (and still do) to be an overstatement. "It is suboptimal" or "it 
>> is an area of improvement" are both significantly weaker statements, which I 
>> find less objectionable.
>>  
>> Personally, I still don't think it is a huge problem. And the fact that you 
>> where having a lot of trouble coming up with an example showing it to be one 
>> (the one you posted doesn't - again, it doesn't, in any way, change behavior 
>> when using or not using interfaces) is, in my view, a testament to that.
>> 
>>> And by the way, linters often flag correct code - that is why they have 
>>> disable options. They try to enforce the most common cases - and by the 
>>> recommendation in the faq to only use receivers of the same type - it seems 
>>> appropriate to me to have the linter flag this. 
>> 
>> I'm opposed to a linter flag, because it would flag correct code I regularly 
>> write. In general, linters should not be ignored - they either shouldn't be 
>> run, or they should be followed. Note that golint has no option t

Re: [go-nuts] Knowing from documentation whether an interface is holding a pointer or a struct?

[Robert Engels]

(I think you pasted the wrong link - that is my code). 

It is not about being unwilling to admit it. Your explanation/reasoning has not 
convinced me. 

Imagine some library declares the EventLogger interface as shown. Acceptable. 
Someone writes the RecordEvents() method taking an EventLogger. Acceptable. 

Now, I have a struct I want to use with as an EventLogger (badly named - really 
EventSource). The code I wrote works fine. Test cases (of Log()) work fine. It 
fails when used as a source to RecordEvents() (and similar held reference 
patterns). 

How do you protect against this? What is the mechanism as a library author?

Clearly this is a trivial example but similar patterns are everywhere. 

Compare the Go interface handling with Java’s everything is a reference - much 
simpler - and then adding value types that are explicit. Or a similar 
implementation in Rust. In both cases knowing you wrote a correct 
implementation is much easier. Java has since added annotations for aspects 
like “thread safe” that cover the atomic aspects. 

I like Go. A lot. I’ve designed and built systems with millions of LOC. 
Pointing out aspects that might benefit from changes should be encouraged - if 
not it’s a religion not a programming language. 

> On Jun 7, 2021, at 7:40 PM, Axel Wagner  wrote:
> 
> 
> 
> 
>> On Tue, Jun 8, 2021 at 2:05 AM Robert Engels  wrote:
>> 
>> We agree. It needs a pointer receiver to work. The atomic is also needed in 
>> this case for background logging. 
>> 
>> The problem in this case is that recordEvents() has to document that the  
>> EventLogger passed to recordEvents() must have a pointer receiver for the 
>> Log() method. There is nothing in the language that allows me to declare it 
>> nor the compiler to enforce it.
> 
> It is possible to write a working implementation of that interface without a 
> pointer receiver - it just needs to *contain* a pointer: 
> https://play.golang.org/p/Xm6ASGcCyhR
> You could also have a slice type, which also can do modifications without a 
> pointer receiver. Or a map-type. Or a channel.
> 
> If you would restrict an interface to require pointer-receiver, you would 
> wrongly restrict the implementer from all these possibilities.
> 
> As is the common wisdom, the user of an interface should not care what the 
> concrete type implementing an interface is (except if it needs to do a 
> type-assertions). It's the same wisdom that applies to people wanting to 
> check if an interface contains a nil-pointer: That check relies on the 
> assumption that the interface contains a pointer, which shouldn't be nil and 
> that's not something that should concern the user of an interface.
> 
> Again, to be abundantly clear (you still seem unwilling to acknowledge this): 
> The problem with your code is not the definition or usage of the interface. 
> It's the definition of the method that is wrong. The interface-definition is 
> fine and works fine.
> 
>> If you don’t see this as suboptimal and an area for improvement I am not 
>> sure what else I can say.
> 
> I just want to state again, clearly, that all I objected to was you calling 
> this "the most inconsistent and obtuse aspect of the Go language", which I 
> perceived (and still do) to be an overstatement. "It is suboptimal" or "it is 
> an area of improvement" are both significantly weaker statements, which I 
> find less objectionable.
>  
> Personally, I still don't think it is a huge problem. And the fact that you 
> where having a lot of trouble coming up with an example showing it to be one 
> (the one you posted doesn't - again, it doesn't, in any way, change behavior 
> when using or not using interfaces) is, in my view, a testament to that.
> 
>> And by the way, linters often flag correct code - that is why they have 
>> disable options. They try to enforce the most common cases - and by the 
>> recommendation in the faq to only use receivers of the same type - it seems 
>> appropriate to me to have the linter flag this. 
> 
> I'm opposed to a linter flag, because it would flag correct code I regularly 
> write. In general, linters should not be ignored - they either shouldn't be 
> run, or they should be followed. Note that golint has no option to 
> selectively disable a particular instance of a warning - the only way to 
> silence a warning is to change the code. But I don't want to use a pointer 
> receiver, if a value receiver is more appropriate.
> 
> If golint or go vet would start flagging this, I would likely follow the 
> advice it's giving. Because that's how linters and static checks are supposed 
> to be used - to enforce consistency. But I'd be sad doing it. Which is why I 
> don't want them to flag it

Re: [go-nuts] Knowing from documentation whether an interface is holding a pointer or a struct?

[Robert Engels]

We agree. It needs a pointer receiver to work. The atomic is also needed in 
this case for background logging. 

The problem in this case is that recordEvents() has to document that the  
EventLogger passed to recordEvents() must have a pointer receiver for the Log() 
method. There is nothing in the language that allows me to declare it nor the 
compiler to enforce it.

If you don’t see this as suboptimal and an area for improvement I am not sure 
what else I can say.

And by the way, linters often flag correct code - that is why they have disable 
options. They try to enforce the most common cases - and by the recommendation 
in the faq to only use receivers of the same type - it seems appropriate to me 
to have the linter flag this. 

As to this being in my opinion the most inconsistent and obtuse aspect of Go - 
that is my opinion. Curious, what do you think would take the top spot?


> On Jun 7, 2021, at 6:34 PM, Axel Wagner  wrote:
> 
> 
>> On Tue, Jun 8, 2021 at 1:26 AM Robert Engels  wrote:
> 
>> The pattern of a background stats collector is a common one. The atomic is 
>> required not optional. 
> 
> It might be a common pattern, but it uses a pointer-receiver in that case. 
> The atomic operation is not required, it operates on a local variable. Again, 
> I don't understand how you can make statements that are so clearly wrong.
> 
> Feel free to try running it in the race detector without an atomic operation. 
> Feel free trying to get the race detector to trigger without the atomic 
> access, but keeping it silent when you add it. You'll find that this needs a 
> pointer receiver. Because otherwise the function is operating on a local 
> variable.
>  
>> 
>>>> On Jun 7, 2021, at 6:16 PM, 'Axel Wagner' via golang-nuts 
>>>>  wrote:
>>>> 
>>> 
>>> BTW, just to nail down the point of that code being wrong without 
>>> interfaces: Your usage of `atomic` in `Log` is superfluous. You are 
>>> operating on a local variable, so there is no possibility of concurrent 
>>> modification. Your code is equivalent to this: 
>>> https://play.golang.org/p/zYG0zTsk-2a
>>> The only reason to use `atomic` here (and why you used it) is if that 
>>> memory could be shared between goroutines. For that to happen, you need a 
>>> pointer receiver though.
>>> 
>>> I refuse to believe that interfaces have anything to do with this 
>>> obfuscation here. There is more than enough indication of it being wrong in 
>>> any case.
>>> 
>>>> On Tue, Jun 8, 2021 at 1:05 AM Axel Wagner  
>>>> wrote:
>>>>> On Mon, Jun 7, 2021 at 11:42 PM Robert Engels  
>>>>> wrote:
>>>> 
>>>>> I don’t think that represents the problem fairly. In the non interface 
>>>>> case I know I can’t accept a copy so I would declare the method as taking 
>>>>> a pointer to the struct.
>>>> 
>>>> How methods are declared should, in general, not be a matter of whether or 
>>>> not they are assigned to an interface, but to whether or not they need a 
>>>> pointer. Again: Your code is incorrect without interfaces. The problem 
>>>> doesn't happen when you put that value into an interface - it happens when 
>>>> you pass a copy of it and expect it to refer to the original. Interfaces 
>>>> are just one way to create such a copy, but they do not matter for the 
>>>> correctness of this code and for whether or not that method needs a 
>>>> pointer receiver (it does).
>>>> 
>>>> But again, to be clear: I'm not saying problems like this *never* happen 
>>>> and I'm not even saying that interfaces may obscure it in some cases. Just 
>>>> that a) the root cause here is that your method really needs to take a 
>>>> pointer-receiver, interfaces or not and b) that it seems very much an 
>>>> overstatement to me to call this "the most inconsistent and obtuse aspect 
>>>> of the Go language".
>>>> 
>>>>> With interfaces this is lost - as the interface is implicitly a pointer
>>>> 
>>>> Well, it seems a bad idea to say that interfaces are implicitly pointers 
>>>> then. That seems to indicate that Rob's original phrasing is indeed an 
>>>> important clarification - the language behaves as if the value contained 
>>>> in them is copied when the interface value is copied.
>>>> 
>>>> It seems the confusion here is, that you assume it's not. And that 
>>>> interfaces act as a pointers, when they don't.
>>> 

Re: [go-nuts] Knowing from documentation whether an interface is holding a pointer or a struct?

[Robert Engels]

The pattern of a background stats collector is a common one. The atomic is 
required not optional. 

> On Jun 7, 2021, at 6:16 PM, 'Axel Wagner' via golang-nuts 
>  wrote:
> 
> 
> BTW, just to nail down the point of that code being wrong without interfaces: 
> Your usage of `atomic` in `Log` is superfluous. You are operating on a local 
> variable, so there is no possibility of concurrent modification. Your code is 
> equivalent to this: https://play.golang.org/p/zYG0zTsk-2a
> The only reason to use `atomic` here (and why you used it) is if that memory 
> could be shared between goroutines. For that to happen, you need a pointer 
> receiver though.
> 
> I refuse to believe that interfaces have anything to do with this obfuscation 
> here. There is more than enough indication of it being wrong in any case.
> 
>> On Tue, Jun 8, 2021 at 1:05 AM Axel Wagner  
>> wrote:
>>> On Mon, Jun 7, 2021 at 11:42 PM Robert Engels  wrote:
>> 
>>> I don’t think that represents the problem fairly. In the non interface case 
>>> I know I can’t accept a copy so I would declare the method as taking a 
>>> pointer to the struct.
>> 
>> How methods are declared should, in general, not be a matter of whether or 
>> not they are assigned to an interface, but to whether or not they need a 
>> pointer. Again: Your code is incorrect without interfaces. The problem 
>> doesn't happen when you put that value into an interface - it happens when 
>> you pass a copy of it and expect it to refer to the original. Interfaces are 
>> just one way to create such a copy, but they do not matter for the 
>> correctness of this code and for whether or not that method needs a pointer 
>> receiver (it does).
>> 
>> But again, to be clear: I'm not saying problems like this *never* happen and 
>> I'm not even saying that interfaces may obscure it in some cases. Just that 
>> a) the root cause here is that your method really needs to take a 
>> pointer-receiver, interfaces or not and b) that it seems very much an 
>> overstatement to me to call this "the most inconsistent and obtuse aspect of 
>> the Go language".
>> 
>>> With interfaces this is lost - as the interface is implicitly a pointer
>> 
>> Well, it seems a bad idea to say that interfaces are implicitly pointers 
>> then. That seems to indicate that Rob's original phrasing is indeed an 
>> important clarification - the language behaves as if the value contained in 
>> them is copied when the interface value is copied.
>> 
>> It seems the confusion here is, that you assume it's not. And that 
>> interfaces act as a pointers, when they don't.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAEkBMfGVyvYYpQhCp_JkxN9EvgZ4FXJ8_WpxseJOB1OR7qt6ww%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/DC6B566F-9D93-4EE2-BFD9-4CF797298EFD%40ix.netcom.com.

Re: [go-nuts] Knowing from documentation whether an interface is holding a pointer or a struct?

[Robert Engels]

I think that is why it is inconsistent and obtuse to me.  The Log() method 
doesn’t need a pointer receiver. It works fine without it. It only needs a 
pointer receiver because when passed to a function declared as taking an 
interface a copy is made (and a reference to the copy held). This copy is 
implicit not explicit. This is a difficult model when working on large multi 
person projects - thus the “best practice” of using only pointer or value 
receivers I think. Suggesting that the linter flags code not adhering to this 
seems to have very little downside if any. 

A quick search on the web (and the faq language) makes me think I am not alone 
in this opinion.

I guess I still don’t see the downside to linter changes. As I said I reviewed 
a lot of the stdlib and I don’t see any obvious mixed receiver structs. They 
can be disabled on a per struct basis if needed. 

Anyway it was just my opinion based on my experiences. 

> On Jun 7, 2021, at 6:05 PM, Axel Wagner  wrote:
> 
> 
>> On Mon, Jun 7, 2021 at 11:42 PM Robert Engels  wrote:
> 
>> I don’t think that represents the problem fairly. In the non interface case 
>> I know I can’t accept a copy so I would declare the method as taking a 
>> pointer to the struct.
> 
> How methods are declared should, in general, not be a matter of whether or 
> not they are assigned to an interface, but to whether or not they need a 
> pointer. Again: Your code is incorrect without interfaces. The problem 
> doesn't happen when you put that value into an interface - it happens when 
> you pass a copy of it and expect it to refer to the original. Interfaces are 
> just one way to create such a copy, but they do not matter for the 
> correctness of this code and for whether or not that method needs a pointer 
> receiver (it does).
> 
> But again, to be clear: I'm not saying problems like this *never* happen and 
> I'm not even saying that interfaces may obscure it in some cases. Just that 
> a) the root cause here is that your method really needs to take a 
> pointer-receiver, interfaces or not and b) that it seems very much an 
> overstatement to me to call this "the most inconsistent and obtuse aspect of 
> the Go language".
> 
>> With interfaces this is lost - as the interface is implicitly a pointer
> 
> Well, it seems a bad idea to say that interfaces are implicitly pointers 
> then. That seems to indicate that Rob's original phrasing is indeed an 
> important clarification - the language behaves as if the value contained in 
> them is copied when the interface value is copied.
> 
> It seems the confusion here is, that you assume it's not. And that interfaces 
> act as a pointers, when they don't.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/F93E0A12-F3D2-4AAE-8D3E-091DF4A8AE60%40ix.netcom.com.

Re: [go-nuts] Knowing from documentation whether an interface is holding a pointer or a struct?

[Robert Engels]

I don’t think that represents the problem fairly. In the non interface case I 
know I can’t accept a copy so I would declare the method as taking a pointer to 
the struct. 

With interfaces this is lost - as the interface is implicitly a pointer - but 
whether it points to a copy or the original is unknown because the dev writing 
the interface declaration and function using the interface only knows they need 
something with a Log().  

It is the implicit copying for the interface which makes it obtuse. 

> On Jun 7, 2021, at 12:57 PM, Axel Wagner  
> wrote:
> 
> 
>> On Mon, Jun 7, 2021 at 7:42 PM Robert Engels  wrote:
> 
>> I think that is my point. The methods in the code I shared have a proper 
>> receiver type based on the requirements of the methods.
> 
> No, they don't. The `Log` method requires a pointer receiver, as it accesses 
> state that is supposed to be shared. Once you fix that, the compiler will 
> quite correct complain that the value does not implement the interface in 
> question.
> 
> Note that the code behaves the same even if no interfaces are involved: 
> https://play.golang.org/p/WpIzYYLOKn-
> 
>> So it seems to me that go lint should at least complain that a struct has 
>> mutating methods so all methods should take pointer receivers.
> 
> I disagree.
>  
>>  
>> 
>>>> On Jun 7, 2021, at 9:19 AM, 'Axel Wagner' via golang-nuts 
>>>>  wrote:
>>>> 
>>> 
>>> FWIW I do tend to mix value and pointer receivers occasionally.
>>> Sometimes a type needs a pointer receiver in one method, but a different 
>>> method doesn't and it's more convenient to have a copy available for 
>>> temporary operations.
>>> Usually when I implement flag.Var, I make `Set` use a pointer receiver and 
>>> `String` use a value receiver - because why not? There is no pointer needed 
>>> to stringify a value.
>>> Sometimes I have a slice type which should generally be used and passed as 
>>> a value, but for convenience, I add a `push`/`pop` method to it, with a 
>>> pointer receiver.
>>> 
>>> Of course, all of these things *could* be accomplished by just using 
>>> pointer-receivers everywhere. But by that same logic, why have value 
>>> receivers anyway? You *could* always use a pointer.
>>> 
>>> I don't think it's a huge problem to mix the two. It's true that this means 
>>> the pointer and the value type implement different interfaces - but I don't 
>>> think that's a huge problem either. I *definitely* doubt that it's "the 
>>> most inconsistent and obtuse aspect of the Go language". I'm not saying it 
>>> *never* caused any problems for me, but it definitely isn't as frequent a 
>>> source of bugs as the behavior of closures in loops, or slices 
>>> unintentionally sharing elements after append… All of which I'm fine with 
>>> as well.
>>> 
>>> But FTR, this discussion is definitely off-topic in this thread. And it's 
>>> also moot in general: It's not something we can realistically change now.
>>> 
>>>> On Mon, Jun 7, 2021 at 3:05 PM Robert Engels  wrote:
>>>> 
>>>> There is no good reason that proper behavior should be dependent on 
>>>> understanding best practices. It should help with readability not 
>>>> correctness. Seems to me the compiler or Go Vet should prohibit this - in 
>>>> my review of the stdlib and other projects I can’t see any reason why it 
>>>> doesn’t. 
>>>> 
>>>>>> On Jun 6, 2021, at 11:10 AM, jake...@gmail.com  
>>>>>> wrote:
>>>>>> 
>>>>> 
>>>>>> On Sunday, June 6, 2021 at 9:33:31 AM UTC-4 ren...@ix.netcom.com wrote:
>>>>>> For example, the fact that this code is broken is not intuitively 
>>>>>> obvious for any reader. It requires way too much scrutiny IMO. 
>>>>>> 
>>>>>> https://play.golang.org/p/-f73t_Pm7ur 
>>>>> 
>>>>> I would like to note that your example goes against the general advice 
>>>>> that all methods should be on either pointers or values. Mixing value and 
>>>>> pointer methods for the same types is a code smell. The code you posted 
>>>>> is a good example of one of the reasons why. 
>>>>> 
>>>>> The second to last paragraph in the FAQ section 
>>>>> https://golang.org/doc/faq#methods_on_values_or_pointers says:
>>>>> "If some of the methods of the type must have pointer receivers, the 

Re: [go-nuts] Knowing from documentation whether an interface is holding a pointer or a struct?

[Robert Engels]

I think that is my point. The methods in the code I shared have a proper 
receiver type based on the requirements of the methods. It only “breaks” in the 
context of the usage which isn’t at all obvious. 

So it seems to me that go lint should at least complain that a struct has 
mutating methods so all methods should take pointer receivers. 

> On Jun 7, 2021, at 9:19 AM, 'Axel Wagner' via golang-nuts 
>  wrote:
> 
> 
> FWIW I do tend to mix value and pointer receivers occasionally.
> Sometimes a type needs a pointer receiver in one method, but a different 
> method doesn't and it's more convenient to have a copy available for 
> temporary operations.
> Usually when I implement flag.Var, I make `Set` use a pointer receiver and 
> `String` use a value receiver - because why not? There is no pointer needed 
> to stringify a value.
> Sometimes I have a slice type which should generally be used and passed as a 
> value, but for convenience, I add a `push`/`pop` method to it, with a pointer 
> receiver.
> 
> Of course, all of these things *could* be accomplished by just using 
> pointer-receivers everywhere. But by that same logic, why have value 
> receivers anyway? You *could* always use a pointer.
> 
> I don't think it's a huge problem to mix the two. It's true that this means 
> the pointer and the value type implement different interfaces - but I don't 
> think that's a huge problem either. I *definitely* doubt that it's "the most 
> inconsistent and obtuse aspect of the Go language". I'm not saying it *never* 
> caused any problems for me, but it definitely isn't as frequent a source of 
> bugs as the behavior of closures in loops, or slices unintentionally sharing 
> elements after append… All of which I'm fine with as well.
> 
> But FTR, this discussion is definitely off-topic in this thread. And it's 
> also moot in general: It's not something we can realistically change now.
> 
>> On Mon, Jun 7, 2021 at 3:05 PM Robert Engels  wrote:
>> 
>> There is no good reason that proper behavior should be dependent on 
>> understanding best practices. It should help with readability not 
>> correctness. Seems to me the compiler or Go Vet should prohibit this - in my 
>> review of the stdlib and other projects I can’t see any reason why it 
>> doesn’t. 
>> 
>>>> On Jun 6, 2021, at 11:10 AM, jake...@gmail.com  wrote:
>>>> 
>>> 
>>>> On Sunday, June 6, 2021 at 9:33:31 AM UTC-4 ren...@ix.netcom.com wrote:
>>>> For example, the fact that this code is broken is not intuitively obvious 
>>>> for any reader. It requires way too much scrutiny IMO. 
>>>> 
>>>> https://play.golang.org/p/-f73t_Pm7ur 
>>> 
>>> I would like to note that your example goes against the general advice that 
>>> all methods should be on either pointers or values. Mixing value and 
>>> pointer methods for the same types is a code smell. The code you posted is 
>>> a good example of one of the reasons why. 
>>> 
>>> The second to last paragraph in the FAQ section 
>>> https://golang.org/doc/faq#methods_on_values_or_pointers says:
>>> "If some of the methods of the type must have pointer receivers, the rest 
>>> should too, so the method set is consistent regardless of how the type is 
>>> used."
>>> 
>>> In your example, if MyEventRecorder.Log() is changed to have a pointer 
>>> receiver, then the code works as expected: 
>>> https://play.golang.org/p/MG10opC6Ect
>>> 
>>> -- 
>>> You received this message because you are subscribed to the Google Groups 
>>> "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to golang-nuts+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/golang-nuts/a9b4a8b3-0b2f-4935-807e-1cbca03a3b20n%40googlegroups.com.
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/7C376006-EFA0-44BA-A036-B768078E4AA4%40ix.netcom.com.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAEkBMfGYRdadYFidvhCKnNqKDeqBSJLWBEU%3DuCJrxr1SbhPepQ%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/406A98CC-3D33-44A0-80E0-E40448DA3721%40ix.netcom.com.

Re: [go-nuts] Knowing from documentation whether an interface is holding a pointer or a struct?

[Robert Engels]

There is no good reason that proper behavior should be dependent on 
understanding best practices. It should help with readability not correctness. 
Seems to me the compiler or Go Vet should prohibit this - in my review of the 
stdlib and other projects I can’t see any reason why it doesn’t. 

> On Jun 6, 2021, at 11:10 AM, jake...@gmail.com  wrote:
> 
> On Sunday, June 6, 2021 at 9:33:31 AM UTC-4 ren...@ix.netcom.com wrote:
>> For example, the fact that this code is broken is not intuitively obvious 
>> for any reader. It requires way too much scrutiny IMO. 
>> 
>> https://play.golang.org/p/-f73t_Pm7ur
> 
> I would like to note that your example goes against the general advice that 
> all methods should be on either pointers or values. Mixing value and pointer 
> methods for the same types is a code smell. The code you posted is a good 
> example of one of the reasons why. 
> 
> The second to last paragraph in the FAQ section 
> https://golang.org/doc/faq#methods_on_values_or_pointers says:
> "If some of the methods of the type must have pointer receivers, the rest 
> should too, so the method set is consistent regardless of how the type is 
> used."
> 
> In your example, if MyEventRecorder.Log() is changed to have a pointer 
> receiver, then the code works as expected: 
> https://play.golang.org/p/MG10opC6Ect
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/a9b4a8b3-0b2f-4935-807e-1cbca03a3b20n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/7C376006-EFA0-44BA-A036-B768078E4AA4%40ix.netcom.com.

Re: [go-nuts] Knowing from documentation whether an interface is holding a pointer or a struct?

[robert engels]

For example, the fact that this code is broken is not intuitively obvious for 
any reader. It requires way too much scrutiny IMO.

https://play.golang.org/p/-f73t_Pm7ur

> On Jun 6, 2021, at 6:37 AM, Robert Engels  wrote:
> 
> For me this is the most inconsistent and obtuse aspect of the Go language. It 
> seems it would always be saner to treat interfaces as pointers. Which would 
> mean if they had non pointer receiver methods might force more objects to be 
> allocated on the heap - but it would prevent a lot of misunderstanding and 
> subtle bugs. 
> 
>> On Jun 6, 2021, at 5:43 AM, 'Dan Kortschak' via golang-nuts 
>>  wrote:
>> 
>> On Sun, 2021-06-06 at 03:17 -0700, Brian Candler wrote:
>>> When you assign a regular (non-pointer) value to an interface
>>> variable, it does take a copy of that value:
>>> https://play.golang.org/p/XyBREDL4BGw
>> 
>> It depends on whether it's safe to leave uncopied or not. You can see
>> this here https://play.golang.org/p/q1729cX09BQ
>> 
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/e77e2a94d0c16514e74340a891f02e118e185d59.camel%40kortschak.io.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/BBC4CAE4-9A6E-46EB-9E1D-681C40C5E8EF%40ix.netcom.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/673D9567-9427-4D71-B2AF-5BC5796AD494%40ix.netcom.com.

Re: [go-nuts] Knowing from documentation whether an interface is holding a pointer or a struct?

[Robert Engels]

For me this is the most inconsistent and obtuse aspect of the Go language. It 
seems it would always be saner to treat interfaces as pointers. Which would 
mean if they had non pointer receiver methods might force more objects to be 
allocated on the heap - but it would prevent a lot of misunderstanding and 
subtle bugs. 

> On Jun 6, 2021, at 5:43 AM, 'Dan Kortschak' via golang-nuts 
>  wrote:
> 
> On Sun, 2021-06-06 at 03:17 -0700, Brian Candler wrote:
>> When you assign a regular (non-pointer) value to an interface
>> variable, it does take a copy of that value:
>> https://play.golang.org/p/XyBREDL4BGw
> 
> It depends on whether it's safe to leave uncopied or not. You can see
> this here https://play.golang.org/p/q1729cX09BQ
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/e77e2a94d0c16514e74340a891f02e118e185d59.camel%40kortschak.io.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/BBC4CAE4-9A6E-46EB-9E1D-681C40C5E8EF%40ix.netcom.com.

Re: [go-nuts] Computing a hash over a uintptr ?

[Robert Engels]

 I was thinking of using C.malloc to create off heap structures. The uintptr to 
those structs should be stable and can be used as map keys or values, right?

Not suggesting anyone should do that - just feasibility. 

> On Jun 4, 2021, at 10:38 AM, Ian Lance Taylor  wrote:
> 
> On Fri, Jun 4, 2021 at 5:23 AM Robert Engels  wrote:
>> 
>> I was referring to the “special cases” in this section 
>> https://golang.org/cmd/cgo/ - wouldn’t all of these uintptr be stable?
> 
> The cgo tool gets special treatment: each cgo call is permitted to pin
> a known number of pointers.  There is some background at
> https://go.googlesource.com/proposal/+/refs/heads/master/design/12416-cgo-pointers.md.
> 
> There is a similar exception for calls to syscall.Syscall, documented
> at https://golang.org/pkg/unsafe/#Pointer.
> 
> These mechanisms are specific to those uses, and don't apply to all
> pointers in general.
> 
> Ian
> 
> 
> 
>> On Jun 4, 2021, at 12:25 AM, Ian Lance Taylor  wrote:
>> 
>> On Thu, Jun 3, 2021 at 9:13 PM Robert Engels  wrote:
>> 
>> 
>> Doesn’t that depend on what the uintptr refers to? Eg if it was allocated 
>> off heap to begin with then it should be stable  and computing a hash on it 
>> is valid.
>> 
>> 
>> That is true in current implementations, but Go, the language, does
>> not guarantee that pointers will never move.
>> 
>> Ian
>> 
>> 
>> On Jun 3, 2021, at 9:42 AM, 'Axel Wagner' via golang-nuts 
>>  wrote:
>> 
>> 
>> 
>> 
>> On Thu, Jun 3, 2021 at 4:19 PM christoph...@gmail.com 
>>  wrote:
>> 
>> 
>> The documentation doesn’t specify that we can cast an uintptr into a uint64.
>> 
>> 
>> 
>> Both are integer types, so they can be converted. You might be worried about 
>> their respective sizes and you are correct that the spec does not guarantee 
>> that a uintptr is at most 64 bit. However, at least for any implementation I 
>> know on any supported architecture, that's the case. If you *really* want to 
>> make sure, you can do something like this:
>> 
>> 
>> func toBytes(v uintptr) []byte {
>> 
>>   var out []byte
>> 
>>   mask := ^uintptr(0)
>> 
>>   for mask != 0 {
>> 
>>   out = append(out, uint8(v))
>> 
>>   v >>= 8
>> 
>>   mask >>= 8
>> 
>>   }
>> 
>>   return out
>> 
>> }
>> 
>> 
>> But really, you can just trust that a uintptr fits into a uint64. Or, if you 
>> want to future-proof, assume it's uint64, but guard by build tags to known 
>> architectures (so it at least fails to compile if that assumption ever 
>> changes).
>> 
>> 
>> To give some context, the application is for a cache with the hash key 
>> computed over a pointer to a heap allocated struct. I need to store the 
>> pointer in the cache entry to test for matching, but this would prevent the 
>> struct to be garbage collected. Storing the pointer as an uintptr would do 
>> the trick.
>> 
>> 
>> 
>> This is problematic. The address of a value can change, so the uintptr would 
>> change as well. So there is no guarantee that your hash stays the same, in 
>> subsequent calls.
>> 
>> Today, that usually doesn't happen, but there is no guarantee it stays that 
>> way. If you are willing to assume that it doesn't happen, you should 
>> definitely also be willing to assume a uintptr fits into a uint64
>> 
>> 
>> 
>> 
>> --
>> 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> 
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> 
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/7983a13f-5bf6-4299-a598-1d023ec9a9e9n%40googlegroups.com.
>> 
>> 
>> --
>> 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> 
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> 
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/CAEkBMfFsZNHLXcONF9C6Nadr6muUP_kgOJwM3QUXSZ0KxPnfYQ%40mail.gmail.com.
>> 
>> 
>> --
>> 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" gr

Re: [go-nuts] Computing a hash over a uintptr ?

[Robert Engels]

I was referring to the “special cases” in this section 
https://golang.org/cmd/cgo/ - wouldn’t all of these uintptr be stable?

> On Jun 4, 2021, at 12:25 AM, Ian Lance Taylor  wrote:
> 
> On Thu, Jun 3, 2021 at 9:13 PM Robert Engels  wrote:
>> 
>> Doesn’t that depend on what the uintptr refers to? Eg if it was allocated 
>> off heap to begin with then it should be stable  and computing a hash on it 
>> is valid.
> 
> That is true in current implementations, but Go, the language, does
> not guarantee that pointers will never move.
> 
> Ian
> 
> 
>> On Jun 3, 2021, at 9:42 AM, 'Axel Wagner' via golang-nuts 
>>  wrote:
>> 
>> 
>>> On Thu, Jun 3, 2021 at 4:19 PM christoph...@gmail.com 
>>>  wrote:
>>> 
>>> The documentation doesn’t specify that we can cast an uintptr into a uint64.
>> 
>> 
>> Both are integer types, so they can be converted. You might be worried about 
>> their respective sizes and you are correct that the spec does not guarantee 
>> that a uintptr is at most 64 bit. However, at least for any implementation I 
>> know on any supported architecture, that's the case. If you *really* want to 
>> make sure, you can do something like this:
>> 
>> func toBytes(v uintptr) []byte {
>>var out []byte
>>mask := ^uintptr(0)
>>for mask != 0 {
>>out = append(out, uint8(v))
>>v >>= 8
>>mask >>= 8
>>}
>>return out
>> }
>> 
>> But really, you can just trust that a uintptr fits into a uint64. Or, if you 
>> want to future-proof, assume it's uint64, but guard by build tags to known 
>> architectures (so it at least fails to compile if that assumption ever 
>> changes).
>> 
>>> To give some context, the application is for a cache with the hash key 
>>> computed over a pointer to a heap allocated struct. I need to store the 
>>> pointer in the cache entry to test for matching, but this would prevent the 
>>> struct to be garbage collected. Storing the pointer as an uintptr would do 
>>> the trick.
>> 
>> 
>> This is problematic. The address of a value can change, so the uintptr would 
>> change as well. So there is no guarantee that your hash stays the same, in 
>> subsequent calls.
>> Today, that usually doesn't happen, but there is no guarantee it stays that 
>> way. If you are willing to assume that it doesn't happen, you should 
>> definitely also be willing to assume a uintptr fits into a uint64
>> 
>>> 
>>> 
>>> --
>>> You received this message because you are subscribed to the Google Groups 
>>> "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to golang-nuts+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/golang-nuts/7983a13f-5bf6-4299-a598-1d023ec9a9e9n%40googlegroups.com.
>> 
>> --
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/CAEkBMfFsZNHLXcONF9C6Nadr6muUP_kgOJwM3QUXSZ0KxPnfYQ%40mail.gmail.com.
>> 
>> --
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/35E8DBC1-E067-4717-BFFF-18732D77B987%40ix.netcom.com.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAOyqgcWY%3DS%3DtriGaYtNK_4v4Nw5AAZBCFOY2hKyTHQ9guaaiTw%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/F496ED71-41E9-4311-8BF6-E6AE7BC8D8DA%40ix.netcom.com.

Re: [go-nuts] Computing a hash over a uintptr ?

[Robert Engels]

Doesn’t that depend on what the uintptr refers to? Eg if it was allocated off 
heap to begin with then it should be stable  and computing a hash on it is 
valid. 

> On Jun 3, 2021, at 9:42 AM, 'Axel Wagner' via golang-nuts 
>  wrote:
> 
> 
>> On Thu, Jun 3, 2021 at 4:19 PM christoph...@gmail.com 
>>  wrote:
> 
>> The documentation doesn’t specify that we can cast an uintptr into a uint64.
> 
> Both are integer types, so they can be converted. You might be worried about 
> their respective sizes and you are correct that the spec does not guarantee 
> that a uintptr is at most 64 bit. However, at least for any implementation I 
> know on any supported architecture, that's the case. If you *really* want to 
> make sure, you can do something like this:
> 
> func toBytes(v uintptr) []byte {
> var out []byte
> mask := ^uintptr(0)
> for mask != 0 {
> out = append(out, uint8(v))
> v >>= 8
> mask >>= 8
> }
> return out
> }
>  
> But really, you can just trust that a uintptr fits into a uint64. Or, if you 
> want to future-proof, assume it's uint64, but guard by build tags to known 
> architectures (so it at least fails to compile if that assumption ever 
> changes).
> 
>> To give some context, the application is for a cache with the hash key 
>> computed over a pointer to a heap allocated struct. I need to store the 
>> pointer in the cache entry to test for matching, but this would prevent the 
>> struct to be garbage collected. Storing the pointer as an uintptr would do 
>> the trick.
> 
> This is problematic. The address of a value can change, so the uintptr would 
> change as well. So there is no guarantee that your hash stays the same, in 
> subsequent calls.
> Today, that usually doesn't happen, but there is no guarantee it stays that 
> way. If you are willing to assume that it doesn't happen, you should 
> definitely also be willing to assume a uintptr fits into a uint64
>  
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/7983a13f-5bf6-4299-a598-1d023ec9a9e9n%40googlegroups.com.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAEkBMfFsZNHLXcONF9C6Nadr6muUP_kgOJwM3QUXSZ0KxPnfYQ%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/35E8DBC1-E067-4717-BFFF-18732D77B987%40ix.netcom.com.

Re: [go-nuts] Re: The last line makes the variable y escape. But should it?

[Robert Engels]

Whenever you take the address of something the compiler is going to have a hard 
time with escape analysis due to aliasing. Especially with a complex function 
like println with varadic args. 

> On Jun 1, 2021, at 11:55 AM, 'Axel Wagner' via golang-nuts 
>  wrote:
> 
> 
> "escape" as in "the compiler's escape analysis decides to put it on the heap, 
> instead of the stack". You can compile using `-gcflags=-m` to test that 
> yourself.
> 
>> On Tue, Jun 1, 2021 at 6:39 PM Robert Glonek  wrote:
>> What do you mean by escape? It prints the ptr to y, like the previous prints 
>> the ptr to x. Y is the same pointer throughout, as it should be.
>> 
>>> On Tuesday, 1 June 2021 at 14:51:50 UTC+1 tapi...@gmail.com wrote:
>>> 
>>> package main
>>> 
>>> func newIntPtr(n int) *int {
>>> return 
>>> }
>>> 
>>> func main() {
>>> x := newIntPtr(3)
>>> y := newIntPtr(5)
>>> c := make(chan bool)
>>> go func() {
>>> *y++
>>> close(c)
>>> }()
>>> <-c
>>> println(*x, *y)
>>> println()
>>> //println() // This line makes y escape.
>>> }
>>> 
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/5d84ba06-ec44-477d-a90e-b67dc14535fan%40googlegroups.com.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAEkBMfG5B1ptrRKywuU%2BHc2kedvL06XN871pCYusst8MP6sSjg%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/B400BC99-ADCC-4001-8B78-294C6915062E%40ix.netcom.com.

Re: [go-nuts] I think I found a bug?

[Robert Engels]

A slice is not an array. Words to live by. 

> On May 26, 2021, at 7:37 PM, Kurtis Rader  wrote:
> 
> 
> It's more likely you've misunderstood how slices work. From 
> https://golang.org/ref/spec#Slice_types:
> 
> > A slice is a descriptor for a contiguous segment of an underlying array and 
> > provides access to a numbered sequence of elements from that array.
> 
> If you still believe you've found a bug we'll need the smallest reproducer 
> you can provide.
> 
>> On Wed, May 26, 2021 at 5:27 PM John Olson  wrote:
>> I have a recursive function that seems to be reusing memory it shouldn't. 
>> I'm developing in Goland 2021.1.1 with go version 1.16.2.
>> 
>> I've written a function to generate the partitions of an integer. The 
>> partitions of n are all the sets of integers that add up to n, so the 
>> partitions of 3 are {3}, {2,1} and {1,1,1}. As n grows, the number of 
>> partitions grows exponentially (to be precise, as e^(n^½)). I wrote a 
>> recursive function, which works great up to n=7. At n=8, one of the 
>> partitions is {2,2,2,1}, which is obviously wrong; and since the function is 
>> recursive, every subsequent n is wrong, too.
>> 
>> I just spent quite a long time stepping through the code, and I found that 
>> what seems to be happening is that a slice declared in my recursive function 
>> is being reused in two different recursions. Specifically, my function 
>> declares
>> var temp [][]int
>> to build up the list of partitions. When I compute Partitions(8), I have to 
>> compute Partitions(4), 5, 6 and 7. I memoize each set of partitions so I 
>> only have to compute them once.
>> 
>> It all goes wrong when I'm working on Partitions(7). At this point, 
>> Partitions(8) has already added 6 cases, the last of which is {2,2,2,2}. 
>> This is stored in temp[5], the temp corresponding to n=8, of course. Then I 
>> compute Partitions(7), which should create a new temp [][]int; I'll call 
>> this temp//7. temp[6]//7 gets {2,2,2,1}, and at that point temp[5]//8 
>> changes from {2,2,2,2} to {2,2,2,1}. The addresses of temp[6]//7 and 
>> temp[5]//8 are different, but the addresses of the elements of these slices 
>> are the same.
>> 
>> This has to be wrong.
>> 
>> I suspect a bug in go, but I suppose it's possible there's a bug in goland. 
>> I'm running on macOS 11.3.1, just in case that's relevant.
>> 
>> I'm happy to share the source code if anyone is interested.
>> 
>> Thanks,
>> 
>> J
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/fb9031ba-bfa0-4c92-9cb7-6ad3a8781184n%40googlegroups.com.
> 
> 
> -- 
> Kurtis Rader
> Caretaker of the exceptional canines Junior and Hank
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CABx2%3DD-vkk_aOtNk%3D4woXcwDRy8_-pyeRu9NnRKF9beuOFSYTw%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/92F4E6ED-F7E2-4548-A4AA-014827E64353%40ix.netcom.com.

Re: [go-nuts] Still "missing" priority or ordered select in go?

[Robert Engels]

Robert :)

> On May 15, 2021, at 3:48 AM, Øyvind Teig  wrote:
> 
> Thanks, rog. I appreciate this! Even if I cant' stop thinking that a "pri" 
> would have been sligtly more elegant. But you shall be praised for the try! I 
> wont' take the time to fine read the code, though... 
> 
> ..Aside: Being more than busy to digest the fact that XMOS has announced a 
> paradigm shift with keeping the xC compiler, but for new projects wanting us 
> to code "in C" instead, with a library lib_xcore. There are reasons. But 
> ordered select is kept! See my reaction blog note C plus lib_xcore 
> black-boxes xC (disclaimer: no association with XMOS, no ads, no income, no 
> gifts etc with my blogs. Only fun an expensens)
> 
>> torsdag 13. mai 2021 kl. 04:48:31 UTC+2 skrev ren...@ix.netcom.com:
>> Here is a simple priority select implementation using unbuffered channels. 
>> https://play.golang.org/p/Hvl0iMr-cFW
>> 
>> Uncomment the lines as instructed and you will see that channel A is always 
>> picked.
>> 
>> What this demonstrates is that ‘priority’ and ‘event order’ is highly 
>> dependent on latency, event frequency, etc. The ’sleep’ + locks in this 
>> example function as an external clock - as long as all events are ready at 
>> the clock edge you can implement priority - but with async events and 
>> scheduling - priority is impossible.
>> 
>> (In the absence of the sleep - the Go scheduler + OS scheduler determines 
>> whether A or B will run and scheduling delays will allow B events to arrive 
>> before an A event.
>> 
>> You can also trivially change this example to implement the ‘close A, close 
>> B’ scenario using a single Go routine and will see the desired ’never ends’ 
>> state is obtained.
>> 
>> (btw - there are probably some deadlock scenarios - I didn’t spend a lot of 
>> time on this)
>> 
>> 
>> 
>> 
>> 
 On May 12, 2021, at 11:53 AM, Øyvind Teig  wrote:
 
>>> I am not certain whether you start with Java and end up with describing a 
>>> possible implementation for Go, or stay with Java. In any case Java is your 
>>> starting point. 
>>> 
>>> I guess, comparing any feature of any language, from the outside, then the 
>>> feature comparison lists turn up. But since you brought in Java..:
>>> 
>>> Then I can just as well throw in the JCSP library's Alternative (=ALT, 
>>> =select) class [1]. And here is their description of the associated 
>>> Alternative type:
>>> By invoking one of the following methods, a process may passively wait for 
>>> one or more of the guards associated with an Alternative object to become 
>>> ready. The methods differ in the way they choose which guard to select in 
>>> the case when two or more guards are ready:
>>> 
>>> select waits for one or more of the guards to become ready. If more than 
>>> one become ready, it makes an arbitrary choice between them (and 
>>> corresponds to the occam ALT).
>>> priSelect also waits for one or more of the guards to become ready. 
>>> However, if more than one becomes ready, it chooses the first one listed 
>>> (and corresponds to the occam PRI ALT). Note: the use of priSelect between 
>>> channel inputs and a skip guard (at lowest priority) gives us a polling 
>>> operation on the readiness of those channels.
>>> fairSelect also waits for one or more of the guards to become ready. If 
>>> more than one become ready, it prioritises its choice so that the guard it 
>>> chose the last time it was invoked has lowest priority this time. This 
>>> corresponds to a common occam idiom used for real-time applications. If 
>>> fairSelect is used in a loop, a ready guard has the guarantee that no other 
>>> guard will be serviced twice before it will be serviced. This enables an 
>>> upper bound on service times to be calculated and ensures that no ready 
>>> guard can be indefinitely starved.
>>> In that world Go would look like this (provided perhaps, it also supported 
>>> list type select guards):
>>> 
>>> select
>>> pri select
>>> fair select
>>> 
>>> [1] JCSP Alternative class - I am not certain how much JCSP has been used. 
>>> It's "beautiful" (my words). Observe that they didn't make any GoCSP 
>>> library..
>>> 
>>> ==
>>> Aside: After the above discussion in this thread I found what I searched 
>>> for, before I started it: a thread from 2012 [2]. Some déjà vu experience!
>>> 
>>> Is there any Go page with the rationale for having a single select type and 
>>> not the other select type(s)? Like [3] or [4]. If not, maybe a new 
>>> paragraph to [4]?
>>> 
>>> Øyvind
>>> 
>>> [2] Priority select in Go (2012)
>>> [3] Bell Labs and CSP Threads by Russ Cox
>>> [4] Why build concurrency on the ideas of CSP?
>>> ==
>>> 
 onsdag 12. mai 2021 kl. 16:52:19 UTC+2 skrev Haddock:
>> In addition to htq and ltq you have a third queue  into which you also 
>> insert a token once one has been added to htq or ltp. The 
>> thread/goroutine that serves htq, ltp, hq, lq is blocked by this 
>> additional queue.