Ugorji here - author of the github.com/ugorji/go/codec <https://pkg.go.dev/github.com/ugorji/go/codec> package.
The notes below are for folks that are interested in why we use unsafe, and how we mitigate concerns around it. As Peter mentioned above, you can pass the build tag "codec.safe" to bypass using unsafe code where we try to reach deeper into the runtime implementation to optimize. I have been supporting gccgo for a while now, but never tested with gollvm, as it has not been released at this time. I built a gollvm version yesterday (from source) and tested it, and made some changes (see here <https://github.com/ugorji/go/commit/1768cf4d4babceec77d8ab0da478c58c9cb9aa99> and here <https://github.com/ugorji/go/commit/12f7e67b2326c6cc05bd3fea410164a18108d6a3>) so that gollvm will work in the default (high-performance) mode. I plan to cut a new release v1.2.5 sometime this week with those changes. For those who care about why we support unsafe and dig into the runtime internals, please read below. To illustrate the benefit, look at unsafe vs codec.safe benchmark results below: ``` ---- tags: "" (default high-performance mode using unsafe carefully) ---- Benchmark__Json_______Encode-8 6921 152808 ns/op 24 B/op 1 allocs/op Benchmark__Json_______Encode-8 5622 197863 ns/op 10048 B/op 381 allocs/op ---- tags: "codec.safe" ---- Benchmark__Json_______Decode-8 2587 415595 ns/op 71878 B/op 592 allocs/op Benchmark__Json_______Decode-8 2167 478122 ns/op 96812 B/op 1456 allocs/op ---- tags: x generated ---- Benchmark__Json_______Encode-8 8694 120519 ns/op 1528 B/op 6 allocs/op Benchmark__Json_______Decode-8 2960 349320 ns/op 70469 B/op 589 allocs/op ``` This benchmarks show that using unsafe carefully can cut down allocations dramatically. Encoding allocation goes from 381 to 1, while decoding goes from 1456 to 592. Those numbers using unsafe rival the allocation numbers that we get using code-generation (as seen above), and the run time starts to trend within 25% of the code-generation numbers, and 25% better than the non-unsafe run time. We limited the surface area that is exposed to unsafe (basically 1 file, helper_unsafe.go, with some variant for gccgo/gollvm where some linkname references do not exist or work differently), so we can quickly edit the code and know where bugs are coming from. Many other packages that try to optimize json/cbor/msgpack encoding and decoding use some variation of the same ideas here. I test the code for the last 5 go releases on each github checkin, via travis CI. I also run with the standard compiler and gccgo locally before cutting a release. I have now added gollvm to my pre-release validation script, so it is validated before I cut the release. Caveat: I test with the installed versions of gccgo from ubuntu, and locally built gollvm. Since gollvm is not released yet, the version I test with may be old (building gollvm takes roughly 1 hour on my computer). If you see any further issues, please file a bug and I will jump on it: https://github.com/ugorji/go/issues/new Thanks. On Monday, March 29, 2021 at 8:32:53 AM UTC-4 peterGo wrote: > You haven't said whether you followed the "safe" instructions for > github.com/ugorji/go/codec to avoid building code/helper_unsafe.go, which > uses go:linkname. > > Package Documentation for github.com/ugorji/go/codec > https://github.com/ugorji/go/blob/master/codec/README.md > > This package will carefully use 'package unsafe' for performance reasons > in specific places. You can build without unsafe use by passing the safe or > appengine tag i.e. 'go install -tags=codec.safe > > You can run the tag 'codec.safe' to run tests or build in safe mode. e.g. > > go test -tags codec.safe -run Json > go test -tags "alltests codec.safe" -run Suite > > Peter > > On Monday, March 29, 2021 at 3:34:44 AM UTC-4 f011...@gmail.com wrote: > >> > go:linkname to refer directly to functions that are defined but not >> exported by the standard library. >> > This is not supported and is likely to break with any new release. It >> evidently breaks with GoLLVM. >> >> Thanks for your attention, but I tried to write a demo with go:linkname. >> In fact, it works well with gollvm...So maybe it is not the exact cause >> for the problem >> >> Herei is my code: >> >> hello/hello.go >> ``` >> package hello >> >> import ( >> "fmt" >> _ "unsafe" >> ) >> //go:linkname hellox hello.hellox >> func hellox(x string) { >> fmt.Println(x) >> } >> ``` >> >> main.go >> ``` >> package main >> >> import ( >> _ "mypackage/hello" >> _ "unsafe" >> ) >> >> //go:linkname hel hello.hellox >> func hel(x string) >> >> func main() { >> hel("aaa") >> //println("aaaa") >> } >> ``` >> >> 在2021年3月29日星期一 UTC+8 上午12:37:49<Ian Lance Taylor> 写道： >> >>> On Sun, Mar 28, 2021 at 9:28 AM 张嘉熙 <f011...@gmail.com> wrote: >>> > >>> > For s2-geojson, I meet: >>> > ``` >>> > # github.com/pantrif/s2-geojson/cmd/s2-geojson >>> > >>> /home/jx/.cache/go-build/6b/6bd003c99eb0a9e7c6ea6d372307b292ec615c75c28f9b1f696896ae2fb4272b-d(_go_.o):gomodule:function >>> >>> github_0com_1ugorji_1go_1codec.intf2impls.intf2impl: error: undefined >>> reference to 'reflect.unsafe_New' >>> > >>> /home/jx/.cache/go-build/6b/6bd003c99eb0a9e7c6ea6d372307b292ec615c75c28f9b1f696896ae2fb4272b-d(_go_.o):gomodule:function >>> >>> github_0com_1ugorji_1go_1codec.Decoder.interfaceExtConvertAndDecode: error: >>> undefined reference to 'reflect.unsafe_New' >>> > decode.go:509: error: undefined reference to 'reflect.unsafe_New' >>> > decode.go:14935: error: undefined reference to 'reflect.unsafe_New' >>> >>> This is a problem with github.com/ugorji/go. The file >>> code/helper_unsafe.go uses go:linkname to refer directly to functions >>> that are defined but not exported by the standard library. This is >>> not supported and is likely to break with any new release. It >>> evidently breaks with GoLLVM. >>> >>> Ian >>> >> -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/584aa92a-8528-45a7-870c-9153298e95adn%40googlegroups.com.
