[go-nuts] what does capture by value means here?

[xie cui]

https://github.com/golang/go/blob/c31540364c2bc3bdb7800cfc344d85c2c9df3893/src/cmd/compile/internal/gc/main.go#L654
what is captured by value not esacpe  means here, can you show a demo code?

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/8c9361cd-7da5-4f02-9bf2-2ab1202a0a6bn%40googlegroups.com.

Re: [go-nuts] Grouped functions...

[Ian Lance Taylor]

On Thu, Feb 11, 2021 at 4:09 PM Trig  wrote:
>
> So, in Go... you can either define imports, constants, types, variables one 
> at a time... or you can group them using brackets.  Why was it decided not to 
> be able to do that with functions?  Just curious.
>
> func (
>  main() {
> fmt.Println(helloWorld())
>  }
>
>  helloWorld() string {
> return "hello, world..."
>  }
> )

One reason is that it doesn't work very well for methods.  It could be
done, of course, but the resulting code doesn't look quite right.

Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAOyqgcUQsfajRx6%3D5zxgyp19Cz5AAQu2ar3zOno1xScjrdX8Zw%40mail.gmail.com.

[go-nuts] Re: Go based xml manipulation library

[Steve Roth]

github.com/beevik/etree does much of what you're looking for.
Steve

On Thursday, February 11, 2021 at 1:10:51 PM UTC-8 sunto...@gmail.com wrote:

> Is there any Go library that allow certain following-sibling 
>  manipulation?
>
> Specifically, I want to,
>
>- locate an xml node via XPath
>- delete its following-sibling
>
> More or less like this, 
> http://zvon.org/comp/r/tut-XPath_1.html#Pages~Following-sibling_axis, but 
> to delete the node, instead of just locating it.
>
> thx
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/64d39a18-fa16-44c0-9ed8-5b3e25b77daen%40googlegroups.com.

[go-nuts] Grouped functions...

[Trig]

So, in Go... you can either define imports, constants, types, variables one 
at a time... or you can group them using brackets.  Why was it decided not 
to be able to do that with functions?  Just curious.

func (
 main() {
fmt.Println(helloWorld())
 }

 helloWorld() string {
return "hello, world..."
 }
)

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/243e5beb-903e-4c95-897e-9d53fa1b521dn%40googlegroups.com.

[go-nuts] Go based xml manipulation library

[Tong Sun]

Is there any Go library that allow certain following-sibling 
 manipulation?

Specifically, I want to,

   - locate an xml node via XPath
   - delete its following-sibling

More or less like this, 
http://zvon.org/comp/r/tut-XPath_1.html#Pages~Following-sibling_axis, but 
to delete the node, instead of just locating it.

thx



-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/446d2e41-4351-4fd3-bce5-6b2406efdfcdn%40googlegroups.com.

[go-nuts] best way and race free case to transfer some value in context to caller

[Vasiliy Tolstov]

Hi. i have func like func AAA(ctx context.Context, req interface{}) error {}
inside func i need to return some value not via error, but in context
(as i only one way do do that)
I'm write something like code below. I can't transfer code inside
error, because it predefined struct (generated from protobuf) and i
cant use grpc Outcoming context and grpc as this is not grpc
server/client code.
What the best way? And the next question - does it possible to clone
context with all deadlines and internal stuff without creating some
struct that satisfies context and uses in as parentheses?

```go
type rspCodeKey struct{}
type rspCodeVal struct {
  code int
}

func SetRspCode(ctx context.Context, code int) {
  if rsp, ok := ctx.Value(rspCodeKey{}).(*rspCodeVal); ok {
rsp.code = code
  }
}

func GetRspCode(ctx context.Context) int {
  var code int
  if rsp, ok := ctx.Value(rspCodeKey{}).(*rspCodeVal); ok {
code = rsp.code
  }
  return code
}
```

-- 
Vasiliy Tolstov,
e-mail: v.tols...@selfip.ru

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CACaajQviZSyY5V0DQWRrSCh0FQm8_wJv56b6hmUL%2B1mgLSFm9Q%40mail.gmail.com.

Re: [go-nuts] Re: Unable to find small memory leak

[Robert Engels]

Are you using CGo or libraries that use CGo? If so, you probably have an 
off-heap, i.e malloc() leak  

> On Feb 11, 2021, at 11:11 AM, Tom Mitchell  wrote:
> 
> 
> 
>> On Thu, Feb 11, 2021 at 3:40 AM Uli Kunitz  wrote:
>> You are writing: The device crashes with out of memory. What does crash? The 
>> Go program, another program or the kernel? Can you share the error message? 
>> Can you share the log file for one day of GODEBUG=gctrace=1?
>> 
>> In bash do:
>> 
>> $ export GODEBUG=gctrace=1
>> $  2>error.log
>> 
>>> On Thursday, February 11, 2021 at 9:01:52 AM UTC+1 Miles Hex wrote:
>>> To be honest, i'm not totally sure, what is see is that memory keep raising 
>>> until the device crash for out of memory, but what does it mean if 
>>> reachable object remain constant, but the " Heap objects" keep raising?
> 
> Objects plural?
> What is the size of each heap object.  With garbage collection it can help to 
> have as few sizes as possible. Pad objects so when collected the free leaves 
> a useful size space in the heap.   Linux can allow the VM system to over 
> allocate resources and only discover when written and the page fault has no 
> backing store.  Add swap to file and or tune the VM system.   Limits -- set 
> limits low enough for the program so it cannot dominate system physical 
> resources. 
> Add signal handler code to dump a summary of all your live objects.  
> Recursion? Socket(state)? Events arrive faster than event handling?  
> Locks/Mutex: is housekeeping getting CPU time [phone booth rule, give 
> priority to leaving]?  
> 
> 
> -- 
>   T o mM i t c h e l l ( o n   N i f t y E g g )
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAAMy4UQWV1Gt9UZBpjpAx01mhL5HEr8NCtGX1rnr1Hevui1Eiw%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/EFB63B8F-89F7-4D62-A476-6381F446F65B%40ix.netcom.com.

Re: [go-nuts] Re: Unable to find small memory leak

[Tom Mitchell]

On Thu, Feb 11, 2021 at 3:40 AM Uli Kunitz  wrote:

> You are writing: The device crashes with out of memory. What does crash?
> The Go program, another program or the kernel? Can you share the error
> message? Can you share the log file for one day of GODEBUG=gctrace=1?
>
> In bash do:
>
> $ export GODEBUG=gctrace=1
> $  2>error.log
>
> On Thursday, February 11, 2021 at 9:01:52 AM UTC+1 Miles Hex wrote:
>
>> To be honest, i'm not totally sure, what is see is that memory keep
>> raising until the device crash for out of memory, but what does it mean if
>> reachable object remain constant, but the " Heap objects" keep raising?
>>
>>> 
>>>
>>
Objects plural?
What is the size of each heap object.  With garbage collection it can help
to have as few sizes as possible. Pad objects so when collected the free
leaves a useful size space in the heap.   Linux can allow the VM system to
over allocate resources and only discover when written and the page fault
has no backing store.  Add swap to file and or tune the VM system.   Limits
-- set limits low enough for the program so it cannot dominate system
physical resources.
Add signal handler code to dump a summary of all your live objects.
Recursion? Socket(state)? Events arrive faster than event handling?
Locks/Mutex: is housekeeping getting CPU time [phone booth rule, give
priority to leaving]?


-- 
  T o mM i t c h e l l ( o n   N i f t y E g g )

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAAMy4UQWV1Gt9UZBpjpAx01mhL5HEr8NCtGX1rnr1Hevui1Eiw%40mail.gmail.com.

Re: [go-nuts] Recommendation for Fuzzed Types without Native On-Disk Format

[Jesper Louis Andersen]

On Mon, Feb 8, 2021 at 10:07 PM Matt Proud  wrote:

> Suppose I am interested in fuzz testing an API using dvyukov/go-fuzz
> , and the primary API under test
> accepts relatively complex composite data types itself: func F(*Table)
> error, where the types below are minimally as complex as this:
>
> type Entry struct {
> Name  string
> Date  time.Date
> Child *Entry  // optional
> Data  interface{} // optional and user-defined
> }
>
> type Table struct {
> Attrs   map[string]string
> Entries []*Entry
> }
>
>
Haphazard list of ideas follows. They are not likely to hit the vampire
with the silver bullet directly, but it might ricochet off of a tree
somewhere and hit by accident.

This just screams quickcheck-style testing to me, even though that isn't
what you want. The reason it's strong is that you are guaranteed to
generate an example which is valid for your function, and you can easily
generate millions of them. Fuzz style testing will treat your data as a
black box, and thus trades CPU time for the click-of-a-button approach. You
don't have to specify a generator, and you don't have to think about a
generator. The tricks fuzzers employ are of the one-size-fits-all kind and
will work on all data. But the trade-off is definitely more CPU cycles
burned.

If you have a binary format and you flip a bit, there is a chance your
flipped []byte is valid. If this chance is low, you don't get to test a
lot. If this chance is high, you get more tests, but you have to factor
through a decoder step of some kind. The trick is to pick a format in which
*every* representation of bits is valid in some way, or at least converges
to that point. This will cut down decoding CPU time to 0, and get you to
the point where you are testing the underlying function more. This is also
the approach you take in quickcheck-style generators of this kind: you
generate a "skeleton" of the structure, then fill in missing bits drawn
from another generator. The extreme variant is akin to mmap()'ing the
memory data out to a disk file.

If `F` is simple, then you are going to spend a lot of time in the decoder.
If `F` is complex, then it doesn't matter as the time in `F` will dwarf the
decoding time.

Hybrids are also possible: take a binary decode, then run 100 steps where
you randomly alter data before invoking `F`. Here you use the fuzzer as a
way to come up with the skeleton structure, and then you test variants of
the leaf-data within the structure.

I've done some work with quickchecking in the industry (But in Haskell,
OCaml, and Erlang rather than Go). If there's any folklore trick to hand
out it's this: use descriptive statistics! Plot the data you are generating
and run it through statistical analysis. You need to verify some
assumptions about the data you are testing, and it can inform you if the
fuzzer is able to dig deep into the structure, or if it's being stopped by
most random bit flips being uninteresting. Don't rely on intuition, but
verify through data. This is often where the insight is hidden. Another
folk-lore trick is to simplify: you don't have to test the full "width" of
the underlying `F` from the get go. We can maybe just assume `Entry.Child`
is always the `nil` value, `Entry.Data` is empty and `Table.Entries` are
all non-nil. If we can establish that *this* passes the test, we can expand
later.

As a simple example: it turns out that in many cases integers of the form
pow(2, n) +/- k are more interesting to test than just a uniform
distribution of integers. Fuzzers can spend a lot of time here, whereas a
quickcheck style generator can generate those integers 5% of the time, say.

As another simple example: when testing concurrency in a database, we just
need a database table with one row. If concurrency rules fail on the single
row, it is bound to fail on multiple rows (ramification: in this scenario,
it's probably not worthwhile for a fuzzer to start exploring multi-row
input).

The strength of fuzzers is that they work by the click of a button and burn
CPU cycles. You don't have to think about the data very much. Exploit this.
If it fails, you can always try another approach down the road. Randomized
methods are strong on modern hardware which is fast. Your goal is to break
the program and most programs break easily if there's a bug in them.
Especially with a fuzzer trying to dig into call paths.

go-fuzz expects the fuzz entrypoints to be func Fuzz(data []byte) int,
> meaning *Table cannot be used directly without some intermediate
> serialization and deserialization.  Suppose that *Table has no regular
> on-disk format; what would you recommend as the approach to take toward
> generating an initial corpus so that go-fuzz can explore the search space
> and generate prospective crashers.
>
> I've toyed around a bit with taking representative values and encoding
> them to disk with encoding/gob and then having go-fuzz deserialize,
> validate, and reject bad gob value candidate

[go-nuts] Re: Unable to find small memory leak

[Uli Kunitz]

You are writing: The device crashes with out of memory. What does crash? 
The Go program, another program or the kernel? Can you share the error 
message? Can you share the log file for one day of GODEBUG=gctrace=1?

In bash do:

$ export GODEBUG=gctrace=1
$  2>error.log

On Thursday, February 11, 2021 at 9:01:52 AM UTC+1 Miles Hex wrote:

> To be honest, i'm not totally sure, what is see is that memory keep 
> raising until the device crash for out of memory, but what does it mean if 
> reachable object remain constant, but the " Heap objects" keep raising? 
> Even when i manualy trigger the GC!
> Also the reachable object is not exported from runtime.ReadMemStats(), 
> and i'm not sure that the stats from the heapview software are correct.
> Thankyou for your help
>
>
> On Wednesday, February 10, 2021 at 12:39:13 AM UTC+1 Uli Kunitz wrote:
>
>> I looked at your data again. Are you sure that you have a memory leak? 
>> According to the heap dump you have less reachable objects after 7 days 
>> than before. The reachable object size is in both cases 1.6 MByte. The live 
>> heap is a little bit larger than seven days before, but is probably caused 
>> by fragmentation and appears to me acceptable. The overall heap size 
>> increased by 4 MB, but it means that you probably need that size to handle 
>> requests. 
>>
>> You can check that by enabling GODEBUG=gctrace=1. Your program has only a 
>> memory leak if the live heap size increases continuously. BTW you can set 
>> the environment variable and start the program that calls your program. You 
>> might keep your heap size a little bit smaller by setting GOGC to a smaller 
>> value than 100.  Here is a helpful article: 
>> https://dave.cheney.net/tag/godebug
>>
>> On Tuesday, February 9, 2021 at 10:43:42 PM UTC+1 Miles Hex wrote:
>>
>>> Thank you, tomorrow i will try it!
>>> Do you know if it possible to enable it without changing the enviroment 
>>> variable? My application is started by another one and i can't (easily) 
>>> change the enviroment variables. 
>>> On Tuesday, February 9, 2021 at 8:57:36 PM UTC+1 Uli Kunitz wrote:
>>>
 It is GODEBUG=allocfreetrace=1 . Documentation for GODEBUG can be found 
 here: https://pkg.go.dev/runtime

 On Tuesday, February 9, 2021 at 8:39:27 PM UTC+1 Uli Kunitz wrote:

> GODEBUG=allocfreetrace may be what you are looking for. You might 
> want to check the output with sort | uniq -c.
>
> On Tuesday, February 9, 2021 at 8:05:23 PM UTC+1 Miles Hex wrote:
>
>>
>> Hi,
>>
>> I'm using golang (1.15.6) to write a daemon that handle system 
>> task(network, time, updates, etc..)  in embedded device (an onion omega 
>> 2+), and i'm encountering a small memory leak that i'm unable to 
>> identify 
>> and fix. 
>> The device is using linux 4.14.171, the architecture is mips.
>>
>> At first i used "go tool pprof" and the memory profile but the 
>> "inuse" index is always empty, and that kinda match my expectation since 
>> the application spend most of his time idle awaiting commands/event (i 
>> must 
>> add that he leak occur even when the application is idle). I also 
>> checked 
>> the number of goroutine that remains constant. 
>>
>> I then added some logs to try to understand when this allocation 
>> occurs (and the subsequent leak), but i'm unable to make sense of the 
>> data.
>> I also collected a couple of heapdump that i'm trying to analyze with 
>> "github.com/temorfeouz/heapdump14" (which is a fork of 
>> https://github.com/randall77/heapdump14, and the only tool i find 
>> that can open heapdump of go 1.15)
>>
>> Following is the (partial) memstat data:
>>
>> MemStast Before:
>>   "HeapAlloc": 1811448,
>>   "HeapSys": 7176192,
>>   "HeapIdle": 4399104,
>>   "HeapInuse": 2777088,
>>   "HeapReleased": 3358720,
>>   "HeapObjects": 11200,
>>   "StackInuse": 1212416,
>>   "StackSys": 1212416,
>>
>> MemStat After 7 days:
>>   "HeapAlloc": 2033048,
>>   "HeapSys": 11403264,
>>   "HeapIdle": 8257536,
>>   "HeapInuse": 3145728,
>>   "HeapReleased": 8257536,
>>   "HeapObjects": 13060,
>>   "StackInuse": 1179648,
>>   "StackSys": 1179648,
>>
>> I also have the summary from the two heapdump
>> Before:
>> Heap size: 8.0 MB bytes 
>> Heap live: 1.8 MB bytes 
>> Heap objects: 11496 
>> Reachable objects: 10492 
>> Reachable size: 1.6 MB bytes 
>>
>> After 7 days:
>> Heap size: 12.0 MB bytes 
>> Heap live: 2.0 MB bytes 
>> Heap objects: 13198 
>> Reachable objects: 10454 
>> Reachable size: 1.6 MB bytes 
>>
>> I have collected a lot more data, including all the profile made 
>> available from the go runtime and various system information.
>>
>> If i'm reading this data correctly there is a memory leak as the heap 
>> si growi

[go-nuts] Re: Unable to find small memory leak

[Miles Hex]

To be honest, i'm not totally sure, what is see is that memory keep raising 
until the device crash for out of memory, but what does it mean if 
reachable object remain constant, but the " Heap objects" keep raising? 
Even when i manualy trigger the GC!
Also the reachable object is not exported from runtime.ReadMemStats(), and 
i'm not sure that the stats from the heapview software are correct.
Thankyou for your help


On Wednesday, February 10, 2021 at 12:39:13 AM UTC+1 Uli Kunitz wrote:

> I looked at your data again. Are you sure that you have a memory leak? 
> According to the heap dump you have less reachable objects after 7 days 
> than before. The reachable object size is in both cases 1.6 MByte. The live 
> heap is a little bit larger than seven days before, but is probably caused 
> by fragmentation and appears to me acceptable. The overall heap size 
> increased by 4 MB, but it means that you probably need that size to handle 
> requests. 
>
> You can check that by enabling GODEBUG=gctrace=1. Your program has only a 
> memory leak if the live heap size increases continuously. BTW you can set 
> the environment variable and start the program that calls your program. You 
> might keep your heap size a little bit smaller by setting GOGC to a smaller 
> value than 100.  Here is a helpful article: 
> https://dave.cheney.net/tag/godebug
>
> On Tuesday, February 9, 2021 at 10:43:42 PM UTC+1 Miles Hex wrote:
>
>> Thank you, tomorrow i will try it!
>> Do you know if it possible to enable it without changing the enviroment 
>> variable? My application is started by another one and i can't (easily) 
>> change the enviroment variables. 
>> On Tuesday, February 9, 2021 at 8:57:36 PM UTC+1 Uli Kunitz wrote:
>>
>>> It is GODEBUG=allocfreetrace=1 . Documentation for GODEBUG can be found 
>>> here: https://pkg.go.dev/runtime
>>>
>>> On Tuesday, February 9, 2021 at 8:39:27 PM UTC+1 Uli Kunitz wrote:
>>>
 GODEBUG=allocfreetrace may be what you are looking for. You might want 
 to check the output with sort | uniq -c.

 On Tuesday, February 9, 2021 at 8:05:23 PM UTC+1 Miles Hex wrote:

>
> Hi,
>
> I'm using golang (1.15.6) to write a daemon that handle system 
> task(network, time, updates, etc..)  in embedded device (an onion omega 
> 2+), and i'm encountering a small memory leak that i'm unable to identify 
> and fix. 
> The device is using linux 4.14.171, the architecture is mips.
>
> At first i used "go tool pprof" and the memory profile but the "inuse" 
> index is always empty, and that kinda match my expectation since the 
> application spend most of his time idle awaiting commands/event (i must 
> add 
> that he leak occur even when the application is idle). I also checked the 
> number of goroutine that remains constant. 
>
> I then added some logs to try to understand when this allocation 
> occurs (and the subsequent leak), but i'm unable to make sense of the 
> data.
> I also collected a couple of heapdump that i'm trying to analyze with 
> "github.com/temorfeouz/heapdump14" (which is a fork of 
> https://github.com/randall77/heapdump14, and the only tool i find 
> that can open heapdump of go 1.15)
>
> Following is the (partial) memstat data:
>
> MemStast Before:
>   "HeapAlloc": 1811448,
>   "HeapSys": 7176192,
>   "HeapIdle": 4399104,
>   "HeapInuse": 2777088,
>   "HeapReleased": 3358720,
>   "HeapObjects": 11200,
>   "StackInuse": 1212416,
>   "StackSys": 1212416,
>
> MemStat After 7 days:
>   "HeapAlloc": 2033048,
>   "HeapSys": 11403264,
>   "HeapIdle": 8257536,
>   "HeapInuse": 3145728,
>   "HeapReleased": 8257536,
>   "HeapObjects": 13060,
>   "StackInuse": 1179648,
>   "StackSys": 1179648,
>
> I also have the summary from the two heapdump
> Before:
> Heap size: 8.0 MB bytes 
> Heap live: 1.8 MB bytes 
> Heap objects: 11496 
> Reachable objects: 10492 
> Reachable size: 1.6 MB bytes 
>
> After 7 days:
> Heap size: 12.0 MB bytes 
> Heap live: 2.0 MB bytes 
> Heap objects: 13198 
> Reachable objects: 10454 
> Reachable size: 1.6 MB bytes 
>
> I have collected a lot more data, including all the profile made 
> available from the go runtime and various system information.
>
> If i'm reading this data correctly there is a memory leak as the heap 
> si growing but i'm not sure how to find it! I tough i could try to diff 
> the 
> two heapdump with excel but the tool that i'm using is not very "precise" 
> and a lot of type information seem to be lost.
>
> I don't know how to make sense of this data, what would your next 
> step? I'm quite new in the golang ecosystem and i'm unsure how to move 
> from 
> here.
>
> Thanks.
>
>
>
>
>
>

-- 
You received this messag