Thank you @Ugorji Nwoke <ugo...@google.com> , I appreciate your going the extra mile to include gollvm in your testing.
Cheers, Than On Tue, Mar 30, 2021 at 11:17 AM Ugorji Nwoke <ugo...@gmail.com> wrote: > Ugorji here - author of the github.com/ugorji/go/codec > <https://pkg.go.dev/github.com/ugorji/go/codec> package. > > The notes below are for folks that are interested in why we use unsafe, > and how we mitigate concerns around it. > > As Peter mentioned above, you can pass the build tag "codec.safe" to > bypass using unsafe code where we try to reach deeper into the runtime > implementation to optimize. > > I have been supporting gccgo for a while now, but never tested with > gollvm, as it has not been released at this time. I built a gollvm version > yesterday (from source) and tested it, and made some changes (see here > <https://github.com/ugorji/go/commit/1768cf4d4babceec77d8ab0da478c58c9cb9aa99> > and here > <https://github.com/ugorji/go/commit/12f7e67b2326c6cc05bd3fea410164a18108d6a3>) > so that gollvm will work in the default (high-performance) mode. I plan to > cut a new release v1.2.5 sometime this week with those changes. > > For those who care about why we support unsafe and dig into the runtime > internals, please read below. > > To illustrate the benefit, look at unsafe vs codec.safe benchmark results > below: > ``` > ---- tags: "" (default high-performance mode using unsafe carefully) ---- > Benchmark__Json_______Encode-8 6921 152808 ns/op 24 B/op > 1 allocs/op > Benchmark__Json_______Encode-8 5622 197863 ns/op 10048 B/op > 381 allocs/op > > ---- tags: "codec.safe" ---- > Benchmark__Json_______Decode-8 2587 415595 ns/op 71878 B/op > 592 allocs/op > Benchmark__Json_______Decode-8 2167 478122 ns/op 96812 B/op > 1456 allocs/op > > ---- tags: x generated ---- > Benchmark__Json_______Encode-8 8694 120519 ns/op 1528 B/op > 6 allocs/op > Benchmark__Json_______Decode-8 2960 349320 ns/op 70469 B/op > 589 allocs/op > ``` > > This benchmarks show that using unsafe carefully can cut down allocations > dramatically. Encoding allocation goes from 381 to 1, while decoding goes > from 1456 to 592. Those numbers using unsafe rival the allocation numbers > that we get using code-generation (as seen above), and the run time starts > to trend within 25% of the code-generation numbers, and 25% better than the > non-unsafe run time. > > We limited the surface area that is exposed to unsafe (basically 1 file, > helper_unsafe.go, with some variant for gccgo/gollvm where some linkname > references do not exist or work differently), so we can quickly edit the > code and know where bugs are coming from. Many other packages that try to > optimize json/cbor/msgpack encoding and decoding use some variation of the > same ideas here. > > I test the code for the last 5 go releases on each github checkin, via > travis CI. I also run with the standard compiler and gccgo locally before > cutting a release. I have now added gollvm to my pre-release validation > script, so it is validated before I cut the release. Caveat: I test with > the installed versions of gccgo from ubuntu, and locally built gollvm. > Since gollvm is not released yet, the version I test with may be old > (building gollvm takes roughly 1 hour on my computer). > > If you see any further issues, please file a bug and I will jump on it: > https://github.com/ugorji/go/issues/new > > Thanks. > On Monday, March 29, 2021 at 8:32:53 AM UTC-4 peterGo wrote: > >> You haven't said whether you followed the "safe" instructions for >> github.com/ugorji/go/codec to avoid building code/helper_unsafe.go, >> which uses go:linkname. >> >> Package Documentation for github.com/ugorji/go/codec >> https://github.com/ugorji/go/blob/master/codec/README.md >> >> This package will carefully use 'package unsafe' for performance reasons >> in specific places. You can build without unsafe use by passing the safe or >> appengine tag i.e. 'go install -tags=codec.safe >> >> You can run the tag 'codec.safe' to run tests or build in safe mode. e.g. >> >> go test -tags codec.safe -run Json >> go test -tags "alltests codec.safe" -run Suite >> >> Peter >> >> On Monday, March 29, 2021 at 3:34:44 AM UTC-4 f011...@gmail.com wrote: >> >>> > go:linkname to refer directly to functions that are defined but not >>> exported by the standard library. >>> > This is not supported and is likely to break with any new release. It >>> evidently breaks with GoLLVM. >>> >>> Thanks for your attention, but I tried to write a demo with go:linkname. >>> In fact, it works well with gollvm...So maybe it is not the exact cause >>> for the problem >>> >>> Herei is my code: >>> >>> hello/hello.go >>> ``` >>> package hello >>> >>> import ( >>> "fmt" >>> _ "unsafe" >>> ) >>> //go:linkname hellox hello.hellox >>> func hellox(x string) { >>> fmt.Println(x) >>> } >>> ``` >>> >>> main.go >>> ``` >>> package main >>> >>> import ( >>> _ "mypackage/hello" >>> _ "unsafe" >>> ) >>> >>> //go:linkname hel hello.hellox >>> func hel(x string) >>> >>> func main() { >>> hel("aaa") >>> //println("aaaa") >>> } >>> ``` >>> >>> 在2021年3月29日星期一 UTC+8 上午12:37:49<Ian Lance Taylor> 写道： >>> >>>> On Sun, Mar 28, 2021 at 9:28 AM 张嘉熙 <f011...@gmail.com> wrote: >>>> > >>>> > For s2-geojson, I meet: >>>> > ``` >>>> > # github.com/pantrif/s2-geojson/cmd/s2-geojson >>>> > >>>> /home/jx/.cache/go-build/6b/6bd003c99eb0a9e7c6ea6d372307b292ec615c75c28f9b1f696896ae2fb4272b-d(_go_.o):gomodule:function >>>> github_0com_1ugorji_1go_1codec.intf2impls.intf2impl: error: undefined >>>> reference to 'reflect.unsafe_New' >>>> > >>>> /home/jx/.cache/go-build/6b/6bd003c99eb0a9e7c6ea6d372307b292ec615c75c28f9b1f696896ae2fb4272b-d(_go_.o):gomodule:function >>>> github_0com_1ugorji_1go_1codec.Decoder.interfaceExtConvertAndDecode: error: >>>> undefined reference to 'reflect.unsafe_New' >>>> > decode.go:509: error: undefined reference to 'reflect.unsafe_New' >>>> > decode.go:14935: error: undefined reference to 'reflect.unsafe_New' >>>> >>>> This is a problem with github.com/ugorji/go. The file >>>> code/helper_unsafe.go uses go:linkname to refer directly to functions >>>> that are defined but not exported by the standard library. This is >>>> not supported and is likely to break with any new release. It >>>> evidently breaks with GoLLVM. >>>> >>>> Ian >>>> >>> -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/584aa92a-8528-45a7-870c-9153298e95adn%40googlegroups.com > <https://groups.google.com/d/msgid/golang-nuts/584aa92a-8528-45a7-870c-9153298e95adn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CA%2BUr55EL3PWdo%3Du1dSYQgx2PmbVZ%3Dqqb0LPrCxO5-ZwG%2BWmFnQ%40mail.gmail.com.
