[go-nuts] Re: decode unknown CBOR type with set of known types
Hi,
I've been informed of how exactly to use the CBOR tag solution by the
author of one of the cbor.io recommended CBOR libraries:
https://github.com/fxamacker/cbor/issues/241#issuecomment-656881180
This solution is clearly a bit nicer than the wrapping struct solution but
it's specific to the given CBOR library whereas the wrapping struct
solution would likely work with any serialization library.
Not that Brian and Jonathan both suggested this solution... however it's
very helpful to see this code example of how exactly to do it since it
wasn't immediately obvious from reading the API docs of the various CBOR
libraries.
I shall quote the author's code sample herein:
"""
// Create TagSet with tag number and Go type
tags := cbor.NewTagSet()
err := tags.Add(
cbor.TagOptions{EncTag: cbor.EncTagRequired, DecTag: cbor.DecTagRequired},
reflect.TypeOf(MyType{}),// your custom type
MyTypeCBORTagNumber, // CBOR tag number for your custom type
)
// Create decoding mode with TagSet
dm, err := cbor.DecOptions{}.DecModeWithTags(tags)
// Decode unknown CBOR blob
var v interface{}
err = dm.Unmarshal(b, )
// Use type switch to handle different types
switch v := v.(type) {
case MyType: // custom type
...
default:
...
}
"""
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/8ffa761a-bfec-4658-bccd-f9a5b1297991o%40googlegroups.com.
Re: [go-nuts] building go 1.14.4 on CentOS 7 multiple test failures
On Fri, Jul 10, 2020 at 1:19 PM Alastair Neil wrote: > > I'm not sure how I would check do you have suggestions? Also, is there a way > to just run the failing test rather than redoing the whole build? > Documentation on the build process seems sparse. To run the time package tests as the build does, run "go test -test.short time". Ian > On Friday, 10 July 2020 15:54:02 UTC-4, David Riley wrote: >> >> On Jul 10, 2020, at 3:02 PM, Alastair Neil wrote: >> > >> > >> > It's an old but not an especially slow machine: dual 8 core E5-2670 Xeon >> > with 64 GB Ram. >> > I wil try >> > GO_TEST_TIMEOUT_SCALE=4 >> >> I've successfully built on an older and slower (pre-Nehalem) 8-core Xeon >> with a similar RAM load just last week, so it's probably not that. >> >> Is something amiss with your system's timer setup? The tick tests seem >> wrong, but precisely near 100ms boundaries. It almost looks like you have >> 10 Hz scheduler ticks and it's not seeing the granularity it should (and >> maybe getting scheduled later than expected). >> >> >> - Dave >> >> > -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/63a1d572-695c-462b-a83b-b37d916c512bo%40googlegroups.com. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAOyqgcVNsv%2BxoyZ3LPCgrPTEjKu_PkD4G96qY_EPZYr4Cn_aPA%40mail.gmail.com.
Re: [go-nuts] building go 1.14.4 on CentOS 7 multiple test failures
I'm not sure how I would check do you have suggestions? Also, is there a way to just run the failing test rather than redoing the whole build? Documentation on the build process seems sparse. On Friday, 10 July 2020 15:54:02 UTC-4, David Riley wrote: > > On Jul 10, 2020, at 3:02 PM, Alastair Neil > wrote: > > > > > > It's an old but not an especially slow machine: dual 8 core E5-2670 Xeon > with 64 GB Ram. > > I wil try > > GO_TEST_TIMEOUT_SCALE=4 > > I've successfully built on an older and slower (pre-Nehalem) 8-core Xeon > with a similar RAM load just last week, so it's probably not that. > > Is something amiss with your system's timer setup? The tick tests seem > wrong, but precisely near 100ms boundaries. It almost looks like you have > 10 Hz scheduler ticks and it's not seeing the granularity it should (and > maybe getting scheduled later than expected). > > > - Dave > > > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/63a1d572-695c-462b-a83b-b37d916c512bo%40googlegroups.com.
Re: [go-nuts] building go 1.14.4 on CentOS 7 multiple test failures
On Jul 10, 2020, at 3:02 PM, Alastair Neil wrote: > > > It's an old but not an especially slow machine: dual 8 core E5-2670 Xeon > with 64 GB Ram. > I wil try > GO_TEST_TIMEOUT_SCALE=4 I've successfully built on an older and slower (pre-Nehalem) 8-core Xeon with a similar RAM load just last week, so it's probably not that. Is something amiss with your system's timer setup? The tick tests seem wrong, but precisely near 100ms boundaries. It almost looks like you have 10 Hz scheduler ticks and it's not seeing the granularity it should (and maybe getting scheduled later than expected). - Dave -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/46244F14-A133-4E5C-B54E-49C703F73921%40gmail.com.
Re: [go-nuts] building go 1.14.4 on CentOS 7 multiple test failures
It's an old but not an especially slow machine: dual 8 core E5-2670 Xeon with 64 GB Ram. I wil try GO_TEST_TIMEOUT_SCALE=4 On Friday, 10 July 2020 14:24:54 UTC-4, Ian Lance Taylor wrote: > > On Fri, Jul 10, 2020 at 11:17 AM > wrote: > > > > I am attempting to build go v 1.14.4 from the git repo. I first built > version 1.4 without issue and the build itself seems fine however many of > the tests fail, see the attached log. > > > > The OS is latest CentOS 7, and go-1.4 was built using gcc 8.3.1 > > > > any pointers gratefully received > > The failures seem to be timeouts, and the reported times seem > unusually long. Are you running on an unusually slow system? > > If so, you could try setting the environment variable > GO_TEST_TIMEOUT_SCALE=4 (or some other values) to tell the tests that > your system is slow. > > Ian > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/ea1394a9-9f24-40da-992d-752552ddd59eo%40googlegroups.com.
Re: [go-nuts] building go 1.14.4 on CentOS 7 multiple test failures
On Friday, 10 July 2020 14:24:54 UTC-4, Ian Lance Taylor wrote: > > On Fri, Jul 10, 2020 at 11:17 AM > wrote: > > > > I am attempting to build go v 1.14.4 from the git repo. I first built > version 1.4 without issue and the build itself seems fine however many of > the tests fail, see the attached log. > > > > The OS is latest CentOS 7, and go-1.4 was built using gcc 8.3.1 > > > > any pointers gratefully received > > The failures seem to be timeouts, and the reported times seem > unusually long. Are you running on an unusually slow system? > > If so, you could try setting the environment variable > GO_TEST_TIMEOUT_SCALE=4 (or some other values) to tell the tests that > your system is slow. > > Ian > Thanks Ian that resulted in a considerable improvement however still failing tin the time test --- FAIL: TestTicker (3.40s) tick_test.go:79: saw 5 errors tick_test.go:37: 10 20ms ticks took 800.863193ms, expected [240ms,360ms] tick_test.go:37: 10 20ms ticks took 796.845457ms, expected [240ms,360ms] tick_test.go:37: 10 20ms ticks took 700.044069ms, expected [240ms,360ms] tick_test.go:37: 10 20ms ticks took 599.839176ms, expected [240ms,360ms] tick_test.go:37: 10 20ms ticks took 499.717239ms, expected [240ms,360ms] FAIL FAIL time 15.827s -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/c051abb7-3719-46e3-b647-9747378a88b3o%40googlegroups.com.
Re: [go-nuts] building go 1.14.4 on CentOS 7 multiple test failures
On Fri, Jul 10, 2020 at 11:17 AM wrote: > > I am attempting to build go v 1.14.4 from the git repo. I first built > version 1.4 without issue and the build itself seems fine however many of the > tests fail, see the attached log. > > The OS is latest CentOS 7, and go-1.4 was built using gcc 8.3.1 > > any pointers gratefully received The failures seem to be timeouts, and the reported times seem unusually long. Are you running on an unusually slow system? If so, you could try setting the environment variable GO_TEST_TIMEOUT_SCALE=4 (or some other values) to tell the tests that your system is slow. Ian -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAOyqgcWwdCcD2qsTwrEXn%3Dh5GXavJUN0XQ5%3D-pky-1_U6ZDQag%40mail.gmail.com.
Re: [go-nuts] [generics] Data structures and the garbage collector
> Many data structures are implemented using pointers. How does this perform
>> wrt garbage collecting? For example take any data structure that uses some
>> tree of nodes, or even simpler just a singly linked list: there is one
>> pointer from the left node to the next. Wouldn't this incur huge GC
>> (unavoidable) cycles for large linked lists that will look at all pointers?
>> I can imagine that a large AVL tree would then cause potentially large
>> hick-ups in the GC cycle?
>>
>>
> Is your worry latency in the form of pause times while doing GC, or
> bandwidth in the form of CPU cycles when doing GC? It isn't entirely clear.
>
>
My "worry" is about CPU cycles that would not be (necessarily) needed in
let's say C++ or C. The fact that Go has a concurrent GC resulting in no
"stop the world"-hick-ups is really awesome, but was not my main point.
(btw I don't really "worry" about it for any real applications; it was more
a question out of interest)
The reason I asked is because I use a custom tree data-structure that
manages parent-child, and leaf node prev-next, relationships using
pointers. I use the tree for multiple value types, so the payload type is
interface{} now. It would be a great place for me to use generics; this
would make my code more clear and safer.
In any case, you are right that these things require attention in a garbage
> collector, both of them. But Go has addressed this for many years now.
>
> As an example, imagine you have a binary AVL or Red/Black tree taking up
> 128 Gigabyte of memory space. You now delete the root node of said tree.
>
> Example 1: in C++ we have reference counted the tree. Deleting the root
> node sets its childrens refcounts to 0, so they get deleted, and so on
> recursively down the tree. It might take some time to delete 128 Gigabyte
> of data this way. So C++ isn't exempt from memory reclamation pauses either.
>
>
In C/C++ you could better use an arena-allocator and not refcounts.
Clearing the tree would then use only (nodes / (nodes per region))
deallocation calls. As the region is typically large, this would probably
be orders of magnitude quicker. I believe compilers use arena allocators
for this reason. (Deleting individual nodes would be cheaper maybe as
well.)
Refcounting I think is best reserved for situations where the lifetime of
an object is not determined by, or escapes, the package (class internals in
C++), or for reasons of simplicity of implementation of course.
I was under the (false) impression that one of the reasons of having a
built-in array, slice and map type in Go (besides it just being really
handy) is that for example a map[string]int would relieve the GC needing to
trace the keys and values (in contrast to for example map[int]*string). But
apparently (see Ian's answer) this is not the case, and the GC always needs
to trace all elements, keys and values. In that case there is (luckily) no
reason to choose build-in map over a custom map written with generics.
> Example 2: Imagine a language using Cheney's copying collection as the
> main garbage collection scheme. It will use 256 gigabyte of memory for its
> two copying spaces, but collection time will be 0, since the tree's data
> isn't live. You are paying with memory for faster collection in this case.
>
> Example 3: Concurrent collectors, like the one Go uses nowadays, will
> collect the tree while the program is running and doing productive work.
> While doing so incurs a bandwidth loss, latency pause times do not suffer.
> If you have spare CPU cores, they'll be used to handle the collection while
> the remaining cores run the program (more or less, this is a gross
> simplification).
>
> As for generics:
>
> When you use a generic, you substitute in a type to generate a
> special-case of a function for that type. If the implementation uses
> monomorphization, it will generate a new piece of code specifically for
> that type invocation. This might mean other optimizations trigger for the
> code path, notably stripping pointers away.
>
> As for general worry:
>
> In some cases it is beneficial to construct data structures such that they
> pack data well and don't use excessive amounts of pointers. One reason is
> that collecting pointers takes time in any language. Another reason is that
> pointer indirections cost memory loads, which can be expensive. A third
> reason is that packed data has cache locality, so access to data close
> together is several orders of magnitude faster than following yet another
> pointer indirection. And finally, it might avoid memory pressure in copying
> data around.
>
>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
Re: [go-nuts] [generics] Data structures and the garbage collector
On Thu, Jul 9, 2020 at 6:40 PM Markus Heukelom wrote: > Many data structures are implemented using pointers. How does this perform > wrt garbage collecting? For example take any data structure that uses some > tree of nodes, or even simpler just a singly linked list: there is one > pointer from the left node to the next. Wouldn't this incur huge GC > (unavoidable) cycles for large linked lists that will look at all pointers? > I can imagine that a large AVL tree would then cause potentially large > hick-ups in the GC cycle? > > Is your worry latency in the form of pause times while doing GC, or bandwidth in the form of CPU cycles when doing GC? It isn't entirely clear. In any case, you are right that these things require attention in a garbage collector, both of them. But Go has addressed this for many years now. As an example, imagine you have a binary AVL or Red/Black tree taking up 128 Gigabyte of memory space. You now delete the root node of said tree. Example 1: in C++ we have reference counted the tree. Deleting the root node sets its childrens refcounts to 0, so they get deleted, and so on recursively down the tree. It might take some time to delete 128 Gigabyte of data this way. So C++ isn't exempt from memory reclamation pauses either. Example 2: Imagine a language using Cheney's copying collection as the main garbage collection scheme. It will use 256 gigabyte of memory for its two copying spaces, but collection time will be 0, since the tree's data isn't live. You are paying with memory for faster collection in this case. Example 3: Concurrent collectors, like the one Go uses nowadays, will collect the tree while the program is running and doing productive work. While doing so incurs a bandwidth loss, latency pause times do not suffer. If you have spare CPU cores, they'll be used to handle the collection while the remaining cores run the program (more or less, this is a gross simplification). As for generics: When you use a generic, you substitute in a type to generate a special-case of a function for that type. If the implementation uses monomorphization, it will generate a new piece of code specifically for that type invocation. This might mean other optimizations trigger for the code path, notably stripping pointers away. As for general worry: In some cases it is beneficial to construct data structures such that they pack data well and don't use excessive amounts of pointers. One reason is that collecting pointers takes time in any language. Another reason is that pointer indirections cost memory loads, which can be expensive. A third reason is that packed data has cache locality, so access to data close together is several orders of magnitude faster than following yet another pointer indirection. And finally, it might avoid memory pressure in copying data around. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAGrdgiWh%2By%2BN92M0%2Bas650hQ4ZLJXAnDq5vz1RqJXPZooNDi%3DA%40mail.gmail.com.
[go-nuts] Re: casting arbitrary int to unsafe.Pointer: go test -race (checkptr) complains
This is a great idea for a work-around. Fixed, thanks!
On Friday, 10 July 2020 at 00:34:58 UTC+1 keith@gmail.com wrote:
> Just write a wrapper on the C side (in the import "C" code) that takes a
> uintptr and casts it to void*.
>
> func wrapper(a uintptr, b, c, d, e ...) {
> op_open_callbacks((void*)a, b, c, d, e)
> }
>
> Then you can call C.wrapper and pass it s.id directly.
>
>
> On Thursday, July 9, 2020 at 4:17:08 PM UTC-7, Hraban Luyat wrote:
>>
>> Because the C function I'm calling takes a "void *" param. Can you pass a
>> regular Go int as a "void *" argument somehow without going through
>> unsafe.Pointer?
>>
>> Note I don't directly call go_readcallback; I call op_open_callbacks (not
>> under my control) and pass it the go_readcallback as a callback (hence the
>> name). It does some setup, and only calls go_readcallback when data comes
>> in. With that void * argument that it was passed during this initial call,
>> here.
>> On Thursday, 9 July 2020 at 20:11:24 UTC+1 Tamás Gulácsi wrote:
>>
>>> As far as I see, you pass c.id to go_readcallback, which gets the
>>> stream from streamsMap.
>>>
>>> Why unsafe.Pointer and uintptr? Why not just a simple int or uint32 ?
>>> For map key, they're indifferent, and the unsafe.Pointer is totally
>>> uneeded here.
>>>
>>>
>>>
>>> 2020. július 9., csütörtök 18:58:52 UTC+2 időpontban Hraban Luyat a
>>> következőt írta:
Hi all,
In package github.com/hraban/opus (currently 57179df), if I run go
test -race I get an error:
fatal error: checkptr: unsafe pointer arithmetic
The offending line is:
https://github.com/hraban/opus/blob/v2/stream.go#L108:
oggfile := C.op_open_callbacks(
unsafe.Pointer(s.id), // <-- this guy
,
nil,
0,
)
Background: I need to pass a reference to a go struct to that C call,
so when it calls my callback function, that callback function can find the
Go struct. The C function expects a void *, but I have a
map[uintptr]myType
locally with a fresh int for every new struct. I cast that uintptr to an
unsafe.Pointer, pass that to the C call, and once the callback is called,
I
cast it back to a uintptr, and use that to fetch the relevant struct from
the map. That's what the "s.id" thing is.
If I understand correctly, since 1.14 (?) go test -race includes
-d=checkptr, and checkptr does not like arbitrary uintptrs to be case to
unsafe.Pointer. However, as far as I understand, that error is spurious in
my specific case, because nobody ever dereferences the pointer at all, or
does any arithmetic on it; it's only cast straight back to a uintptr.
Question 1: Is my map[uintptr]myType rigmarole even necessary? It came
about from my understanding of
https://github.com/golang/proposal/blob/master/design/12416-cgo-pointers.md,
but maybe I misunderstood.
Question 2: if it is necessary, is this "arbitrary uintptr in void *"
valid? As far as I understand the C standard, it is, but again maybe I
misunderstood.
Question 3: If it is valid, what can / should I do about go test -race?
Related:
https://groups.google.com/g/golang-nuts/c/FAOZoQzodAI/m/IF_xKit2DXYJ ,
seems like a similar issue, and the reply suggests the approach is valid.
Thanks all and kind regards
Hraban
Full go test -race output:
fatal error: checkptr: unsafe pointer arithmetic
goroutine 56 [running]:
runtime.throw(0x42354f9, 0x23)
/usr/local/opt/go/libexec/src/runtime/panic.go:1116 +0x72
fp=0xc48ce0 sp=0xc48cb0 pc=0x4034c22
runtime.checkptrArithmetic(0x40, 0x0, 0x0, 0x0)
/usr/local/opt/go/libexec/src/runtime/checkptr.go:26 +0xce
fp=0xc48d10 sp=0xc48ce0 pc=0x400859e
_/Users/hraban/code/personal/opus.(*Stream).Init.func1(0xc0001525c0,
0xc000128758, 0x1)
/Users/hraban/code/personal/opus/stream.go:108 +0x6a
fp=0xc48d68 sp=0xc48d10 pc=0x419ddba
_/Users/hraban/code/personal/opus.(*Stream).Init(0xc0001525c0,
0x42630c0, 0xc0001323e0, 0x0, 0x0)
/Users/hraban/code/personal/opus/stream.go:112 +0x277
fp=0xc48e38 sp=0xc48d68 pc=0x419a9c7
_/Users/hraban/code/personal/opus.NewStream(0x42630c0, 0xc0001323e0,
0x0, 0x0, 0x0)
/Users/hraban/code/personal/opus/stream.go:75 +0xa0 fp=0xc48e88
sp=0xc48e38 pc=0x419a6b0
_/Users/hraban/code/personal/opus.TestStreamIllegal(0xc0002fcb40)
/Users/hraban/code/personal/opus/stream_test.go:20 +0xc0
fp=0xc48ed0 sp=0xc48e88 pc=0x4194320
testing.tRunner(0xc0002fcb40, 0x4238c08)
/usr/local/opt/go/libexec/src/testing/testing.go:991 +0x1ec
fp=0xc48fd0 sp=0xc48ed0 pc=0x41270ec
runtime.goexit()
Re: [go-nuts] What is the the gccgo equivalent of -X importpath.name=value
Hi Ian I am thinking of ways to work around this then. I come from C world. There we use an undefined symbol, and define it using the -D flag to the compiler. Since Go has no preprocessor, this is probably not an option. Another option could be some binutil trick to remove a symbol an re-add it. That would be dirty and dangerous. Happy hacking Atilla -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/cfb1a33e-c9ca-494c-9cf0-4870ac72075bo%40googlegroups.com.
