[go-nuts] Instrumenting in Go and code generation
Hello Gophers, I just wanted to share an article and open source tool that I've finished recently. They both are about instrumenting (aka tracing) Go components. Article covers some background on it, while tool helps to generate boilerplate code to easily provide tracing hooks in your structs. Article: https://gbws.io/articles/instrumentation-in-go/ The tool: https://github.com/gobwas/gtrace Hope it would be helpful! -- Regards, Sergey. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/20200524181654.GA52504%40MacBook-Pro-Sergej.local.
Re: [go-nuts] Stack based slice usage and unsafe
Hello Ian, Thank you for your answer. On Sun, 05/26/19, May 26, 2019 at 07:59:07PM -0400, Ian Lance Taylor wrote: > This is not valid. The rule is that SliceHeader is only valid when > inspecting an actual slice header. You have to write > > h.Data = uintptr(unsafe.Pointer()) > h.Len = len(b) > h.Cap = len(b) > > I assume this is reduced greatly from the real code, as you could also > just write > > bts = b[:] Does this mean that it is not possible to get rid of heap allocation for passing slice of bytes to some method interface like io.Reader.Read()? Both code examples above are leading b to escape to the heap when passing to the interface method. -- Regards, Sergey. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/20190527061838.GA8345%40MacBook-Pro-Sergej.local. For more options, visit https://groups.google.com/d/optout.
Re: [go-nuts] Re: Stack based slice usage and unsafe
Hi Peter,
Thank you for your answer.
Actually it is not so – please see the rule (6) of the unsafe package
documentation:
https://golang.org/pkg/unsafe/
--
Regards,
Sergey.
On Sun, 05/26/19, May 26, 2019 at 09:43:13AM -0700, peterGo wrote:
> Sergey Kamardin,
>
> Your code is invalid. A uintptr variable is an integer, not a pointer.
>
> type SliceHeader: https://golang.org/pkg/reflect/#SliceHeader
>
> SliceHeader is the runtime representation of a slice. It cannot be used
> safely or portably and its representation may change in a later release.
> Moreover, the Data field is not sufficient to guarantee the data it
> references will not be garbage collected, so programs must keep a separate,
> correctly typed pointer to the underlying data.
>
> type SliceHeader struct {
> Data uintptr
> Len int
> Cap int
> }
>
> The Go Programming Language Specification: https://golang.org/ref/spec
>
> uintptr an unsigned integer large enough to store the uninterpreted bits
> of a pointer value
>
> Peter
>
> On Sunday, May 26, 2019 at 11:30:22 AM UTC-4, Sergey Kamardin wrote:
> >
> > Hello gophers,
> >
> > I have a question which relates mostly to the ideology of unsafe usage.
> >
> > In `github.com/gobwas/ws` <http://github.com/gobwas/ws> WebSocket library
> > I used an optimization for
> > reading WebSocket frame headers into stack based slices (to reduce the
> > number of heap allocations):
> >
> > func ReadHeader(r io.Reader) (Header, error) {
> > var (
> > b [16]byte
> > bts []byte
> > )
> > h := (*reflect.SliceHeader)(unsafe.Pointer())
> > *h = reflect.SliceHeader{
> > Data: uintptr(unsafe.Pointer()),
> > Len: len(b),
> > Cap: len(b),
> > }
> > _, err = io.ReadFull(r, bts)
> >
> > // process bytes and return result.
> > }
> >
> > This works fine on Linux for a long time and under the high load, but on
> > Windows it sometimes crashes or has an unexpected behaviour.
> >
> > Crashes are happened with the message like:
> > > fatal error: found bad pointer in Go heap (incorrect use of unsafe or
> > cgo?)
> >
> > I may assume that the bad pointer errors are relate to the
> > internal/poll/fd_windows.go operation buffer implementation, which
> > stores pointer to the first byte of given destination buffer – and since
> > the buffer is stack based GC throws such error.
> >
> > I may assume that the unexpected behaviour happens after goroutine stack
> > moving/copying but can not reproduce it yet during tests.
> >
> > But nevertheless, my question is: is it legal to use unsafe in that way
> > in general? Should I split the implementation of ReadHeader() to support
> > windows separately without stack based slices?
> >
> >
> > References:
> >
> > https://github.com/golang/go/blob/d97bd5d07ac4e7b342053b335428ff9c97212f9f/src/internal/poll/fd_windows.go#L526
> >
> >
> > https://github.com/gobwas/ws/blob/89d0ae05650f2cc04354d7fef282df0db5acff80/read.go#L18
> >
> > https://github.com/gobwas/ws/issues/73
> >
> >
> > --
> > Regards,
> > Sergey.
> >
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/a7a2ad2d-a14d-4538-a90e-cf5a4363d467%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/20190526172641.GA79977%40MacBook-Pro-Sergej.local.
For more options, visit https://groups.google.com/d/optout.
[go-nuts] Stack based slice usage and unsafe
Hello gophers,
I have a question which relates mostly to the ideology of unsafe usage.
In `github.com/gobwas/ws` WebSocket library I used an optimization for
reading WebSocket frame headers into stack based slices (to reduce the
number of heap allocations):
func ReadHeader(r io.Reader) (Header, error) {
var (
b [16]byte
bts []byte
)
h := (*reflect.SliceHeader)(unsafe.Pointer())
*h = reflect.SliceHeader{
Data: uintptr(unsafe.Pointer()),
Len: len(b),
Cap: len(b),
}
_, err = io.ReadFull(r, bts)
// process bytes and return result.
}
This works fine on Linux for a long time and under the high load, but on
Windows it sometimes crashes or has an unexpected behaviour.
Crashes are happened with the message like:
> fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)
I may assume that the bad pointer errors are relate to the
internal/poll/fd_windows.go operation buffer implementation, which
stores pointer to the first byte of given destination buffer – and since
the buffer is stack based GC throws such error.
I may assume that the unexpected behaviour happens after goroutine stack
moving/copying but can not reproduce it yet during tests.
But nevertheless, my question is: is it legal to use unsafe in that way
in general? Should I split the implementation of ReadHeader() to support
windows separately without stack based slices?
References:
https://github.com/golang/go/blob/d97bd5d07ac4e7b342053b335428ff9c97212f9f/src/internal/poll/fd_windows.go#L526
https://github.com/gobwas/ws/blob/89d0ae05650f2cc04354d7fef282df0db5acff80/read.go#L18
https://github.com/gobwas/ws/issues/73
--
Regards,
Sergey.
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/20190526152959.GA55149%40MacBook-Pro-Sergej.local.
For more options, visit https://groups.google.com/d/optout.
Re: [go-nuts] Calling function with variadic arguments
Hi again,
Sorry for my previous misunderstanding. Now I realized that there no
ability in general to get the knowledge of how that slice of variadic
arguments will be used inside hidden interface method implementation.
Also, is it correct to think about variadic arguments just like about
regular slice built by compiler for us?
I mean, does compiler translates this:
```
type DoerFunc func(...int)
func caller(d DoerFunc) {
d(1, 2, 3)
}
```
To something similar:
```
type DoerFunc func([]int)
func caller(d DoerFunc) {
args := []int{1, 2, 3}
d(args)
}
```
Sergey.
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: [go-nuts] Calling function with variadic arguments
Thank you for the reply. > The optimization you are talking about is escape analysis. I was trying to point not the escape analysis itself, but the ability of the compiler to construct stack-based slice for the variadic arguments instead of heap-based slices (like it does with the interfaces). I mean, seems like that there no reason to allocate slice for passing N arguments even for an interface method call? Sergey. On Mon, 11/12/18, fff Nov 12, 2018 at 10:52:07AM -0800, Ian Lance Taylor wrote: > On Mon, Nov 12, 2018 at 4:24 AM, Sergey Kamardin wrote: > > > > Also, I may assume that this because the compiler does not knows exactly > > which function it will call in a runtime. > > Most likely, yes. > > > Am I understand it right and are there any plans for preparing such > > optimizations? > > The optimization you are talking about is escape analysis. It's very > hard to do escape analysis across an interface boundary, where you > don't know what the method does. In cases involving plugins and > shared libraries, it is completely impossible to know what the method > does. I'm not aware of any plans to extend escape analysis across > calls to methods of interface values. > > It's true that in an example like yours the compiler knows the dynamic > type stored in the interface. But such examples are relatively > uncommon in Go code, so it's not clear that it's worth the extra time > required in the compiler to look for them. > > Ian -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[go-nuts] Calling function with variadic arguments
Hello gophers,
Does Go compiler has some optimizations for function and *method* calls
with `variadic` arguments? For example, this simple benchmark:
```
package main
import "testing"
func Do(xs ...int) (s int) {
for i, x := range xs {
s += i + x
}
return s
}
func BenchmarkDo(b *testing.B) {
for i := 0; i < b.N; i++ {
Do(i, i, i, i)
}
}
```
Provides these results:
```
BenchmarkDo-8 3 4.29 ns/op 0 B/op 0 allocs/op
```
That is, no allocation for slice of arguments is done. I may assume that
it is made by preparing `xs` slice on stack of `Do()`.
But, this example:
```
package main
import "testing"
type Doer interface {
Do(...int) int
}
type impl struct {}
func (impl) Do(xs ...int) (s int) {
for i, x := range xs {
s += i + x
}
return s
}
func BenchmarkDo(b *testing.B) {
var d Doer = impl{}
for i := 0; i < b.N; i++ {
d.Do(i, i, i, i)
}
}
```
Provides different results:
```
BenchmarkDo-8 3 25.8 ns/op 32 B/op 1 allocs/op
```
That is, I assume that this allocation is made for the slice of arguments.
Also, I may assume that this because the compiler does not knows exactly
which function it will call in a runtime.
Am I understand it right and are there any plans for preparing such
optimizations?
Regards,
Sergey.
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: [go-nuts] Repeated functions with the same output in loops
> Note that calculating the length of a slice is a fast operation; it's
> a single memory load.
So, actually it just loads a slice header's field?
On Thu, 09/06/18, fff Sep 06, 2018 at 12:56:36PM -0700, Ian Lance Taylor wrote:
> On Thu, Sep 6, 2018 at 12:24 PM, mustafa katipoğlu
> <98mustafakatipo...@gmail.com> wrote:
> >
> > If I use len() function inside of a foor loop , does it calculates the
> > output each time? or if the output will be the same(like arrays) will it
> > calculates the length of the function again ?
> >
> > var letterRunes =
> > []rune("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_")
> >
> >
> > b := make([]rune, usernameLength)
> >
> >
> > for i := range b {
> >b[i] = letterRunes[rand.Intn(len(letterRunes))]
> > }
>
> It will recalculate the length each time through the loop.
>
> Note that calculating the length of a slice is a fast operation; it's
> a single memory load.
>
> Ian
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[go-nuts] WebSocket implementation in Go
Hi Gophers, I have released a stable v1.0 version of https://github.com/gobwas/ws. It implements RFC6455 and supports streaming, manual memory management and other things for writing memory efficient WebSocket applications. By the way it deals with highload in production servers for almost two years (interacting with about 3 millions connections and so on). Here is an article related to it: https://medium.freecodecamp.org/million-websockets-and-go-cc58418460bb. -- Sergey. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [go-nuts] db interface design
> 1. there are a lot of similar functions, for example NewBook and NewAuthor.
> The difference is type of argument passed into the function. Is it a good
> idea to combine those 2 functions with a generic New(interface{}) (string,
> error) function and reflect the actual type inside?
I would not recommend to do this. Eventually logic of insertion may
become much different and your generic function's code will become a
very hard to read. Also, there is a little bit overhead on using
interfaces for non-pointer structs like `Book` and so on.
> 2. Sometimes we query based on one or more conditions (where clause), for
> example find books based on author/release data/ price. Should I create a
> generic function like:
> func FindBooks(map[string]interface{}) ([]Book, error)
It can be done in this way, but I could suggest to use some struct
`BookCriteria` with non-interface fields, say `Author string` and so on.
Then, your `FindBooks` implementation may accept that struct instance.
Or, for example, `FindBooks` may accept variadic functional options that
can fill appropriate fields of criteria. For more about functional
options you could find in Dave Cheney's article:
https://dave.cheney.net/2014/10/17/functional-options-for-friendly-apis
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[go-nuts] Re: Cgo: using syscall.Select
There are methods which are using raw file descriptors (such that https://godoc.org/github.com/mailru/easygo/netpoll#NewDesc). суббота, 19 мая 2018 г., 2:52:27 UTC+3 пользователь Juliusz Chroboczek написал: > > > You could also take a look > > at https://godoc.org/github.com/mailru/easygo/netpoll > > I don't have a net.Conn, just a raw file descriptor. > > -- Juliusz > > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
