Re: [go-nuts] Should io.WriterTo "seek to the end"?

[Ian Lance Taylor]

On Wed, Oct 26, 2022 at 8:14 PM Nigel Tao  wrote:
>
> The io.WriterTo interface has come up in a code review discussion.
>
> ```
> // WriterTo is the interface that wraps the WriteTo method.
> //
> // WriteTo writes data to w until there's no more data to write or
> // when an error occurs. The return value n is the number of bytes
> // written. Any error encountered during the write is also returned.
> //
> // The Copy function uses WriterTo if available.
> type WriterTo interface {
> WriteTo(w Writer) (n int64, err error)
> }
> ```
>
> Should WriteTo always modify receiver state, updating its internal "file 
> position" by the number of bytes written?

I think that should be the default behavior, yes.  Of course there may
be cases where that doesn't make sense.  But in general I expect
WriteTo to behave like one or more calls to Read.


> Equivalently, if I'm re-using a WriterTo, passing it to multiple Copy calls, 
> would you expect to need Seek(0, SeekStart) calls in between those Copy 
> calls? This is a question of clarifying semantics. The compiler doesn't 
> complain if WriteTo always wrote the entire contents (and didn't have a "file 
> position").

Yes, I would expect that for cases where the WriterTo is seekable.

My reason for thinking this is that WriteTo basically exists to
support io.Copy, and in the absence of WriteTo io.Copy does advance
the file position of the source.  So I think that for WriteTo to also
behave that way is the least astonishing choice.

Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAOyqgcWBMo8P7wCzJZA3yfpZX2imVutZdfQ-xkU%3DfoCJyzEYnA%40mail.gmail.com.

[go-nuts] Should io.WriterTo "seek to the end"?

[Nigel Tao]

The io.WriterTo interface has come up in a code review discussion.

```
// WriterTo is the interface that wraps the WriteTo method.
//
// WriteTo writes data to w until there's no more data to write or
// when an error occurs. The return value n is the number of bytes
// written. Any error encountered during the write is also returned.
//
// The Copy function uses WriterTo if available.
type WriterTo interface {
WriteTo(w Writer) (n int64, err error)
}
```

Should WriteTo always modify receiver state, updating its internal "file
position" by the number of bytes written?

Equivalently, if I'm re-using a WriterTo, passing it to multiple Copy
calls, would you expect to need Seek(0, SeekStart) calls in between those
Copy calls? This is a question of clarifying semantics. The compiler
doesn't complain if WriteTo always wrote the entire contents (and didn't
have a "file position").

---

In the standard library, every WriterTo implementation (with one
exception), listed below, also "advances the file position". They all also
implement io.Reader, even though, once again, the compiler doesn't enforce
that every io.WriterTo is an io.Reader.

archive/tar.regFileReader
archive/tar.sparseFileReader
bufio.Reader
bytes.Buffer
bytes.Reader
net.Buffers
strings.Reader

There's also net/http.noBody, which doesn't explicitly update a "file
position" field but it also represents a 0-sized "file" so in some sense
it's always at the "end of the file".

The one exception is recordingConn in crypto/tls/handshake_test.go but it
looks like its WriteTo method is more about "dump some debugging
information" (it's in *_test.go code) than anything related to io.Copy.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAOeFMNXaQGic%3DYxJV64z%2BoWPMk7c4m1WR2%3Di4gTEGJOxwRq3OQ%40mail.gmail.com.

[go-nuts] Re: First stable version of several package under golang.org/x/

['Heschi Kreinick' via golang-nuts]

Yeah, text and tools were already tagged back 
from https://go.dev/blog/path-security, and we've started tagging some 
other golang.org/x repositories now. They'll be tagged every month or so if 
they have changes, and their dependencies on each other will be upgraded.

Note that so far these are all v0 versions, so they're not necessarily 
stable as you said in your subject. No stability promises have changed as a 
result of these tags. That said, now that we have a process for automatic 
tagging, some repositories may be more likely to tag to a stable version. 
See https://go.dev/issue/56325.

I didn't think this change warranted an announcement, but I'll discuss it 
and maybe reconsider. Thanks.

On Wednesday, October 26, 2022 at 2:21:37 PM UTC-4 michel@doctolib.com 
wrote:

> Hi all,
>
> I was bumping the dependencies of our codebase today, and noticed that 
> many packages under golang.org/x/ had upgraded from a pseudo-version to a 
> real, semver version:
>
> -   golang.org/x/net 
>  
> v0.0.0-20220927171203-f486391704dc // indirect 
> -golang.org/x/sync 
>  
> v0.0.0-20220923202941-7f9b1623fab7 // indirect 
> -   golang.org/x/sys 
>  
> v0.0.0-20220928140112-f11e5e49a4ec // indirect
> -   golang.org/x/time 
>  
> v0.0.0-2022090347-f3bd1da661af // indirect 
> +   golang.org/x/net  v0.1.0 
> // indirect 
> +golang.org/x/sync  v0.1.0 
> // indirect 
> +   golang.org/x/sys  v0.1.0 
> // indirect
> +   golang.org/x/time  v0.1.0 
> // indirect
>
> ...and others. Is there something we should be aware of concerning this 
> upgrade? I've seen no announcement everywhere I've been looking, which 
> makes me suppose there's nothing very interesting going on there but that 
> surprised me.
>
> If anyone has more information, I'll gladly take it! Thanks!
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/0acc9b23-f7cb-4ffe-b3a7-c1c0a2d0b01dn%40googlegroups.com.

Re: [go-nuts] Embedding any (empty interface) in a struct in Go

['Axel Wagner' via golang-nuts]

On Wed, Oct 26, 2022 at 9:17 PM sick...@gmail.com 
wrote:

> Consider a scenario where you want to embed an error within any type of
> variable.
>
> This can be achieved by embedding any in a struct and it seems to be a
> valid syntax in Go 1.9.
> type WithError struct {
> any
> err error
> }


I don't think this really "embeds an error within any type of variable". It
*groups* an error with a value of any type. If anything is embedded, it's
`any` though.


> Does this violate any community best practices?
> Is there an alternative pattern that can achieve a similar result?
>
I don't think it violates any best practices. I would find it slightly
confusing and wonder why you didn't do
type WithError struct {
value any
err error
}
which seems clearer and mostly has the same effect - the embedding of `any`
doesn't really do anything, as it has no methods and is an unexported name.

I also think this seems probably more handy:
type WithError[T any] struct {
value T
err error
}

But, really, while I don't really understand what you are trying to
accomplish, there's nothing wrong with your code.



> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/0a6bbea5-c37b-4ce4-89d6-2b44d7868639n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAEkBMfH1cptSDvMDz%3DqX4xH0DQr8k8tzVPjtpMt09Hb5d-fhJg%40mail.gmail.com.

Re: [go-nuts] what is the origin of Go's reference time system?

[Rob Pike]

I believe it's unique. I thought of it one day while walking home. It is
inspired by the way Cobol picture clauses represent number formats. (That
said, I've never programmed in Cobol.)

-rob


On Thu, Oct 27, 2022 at 4:51 AM Ayan George  wrote:

>
> I'm really impressed by the simplicity of Go's time formatting and parsing
> mechanisms -- particularly when compared with strftim().
>
> Does anyone know the origin or history of it?  Is there a precedent for
> using reference layouts that include string like 2006, January, etc.?
>
> Do other languages provide something similar or is this completely unique
> to Go?
>
> Can someone point me to or describe the history of Go's time formatting
> method?
>
> -ayan
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/baa45515-cde6-4c7c-a34c-54ddf7da807en%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAOXNBZTYk_caVpCTROmCRE4g2tmO-wWPYGTgBYZ8jOf99LDHqA%40mail.gmail.com.

[go-nuts] Reading data from GeoTIFF with the gdal package

[Rick Walters]

Hi, if anyone has any experience with Luke Roth's gdal package… I'm 
relatively new to Go, and very new to GeoTIFF and GDAL, and struggling a 
bit.

I'm trying to read data from a 32-bit float band of a GeoTIFF (representing 
elevation) and have two questions:


   1. Is there any way to directly access the very useful values returned 
   by `gdal.Info(sourceDS Dataset, options []string)`? Do I have to parse the 
   resulting string myself? Does the `options []string` argument do something 
   that might be of use to me?
   2. I can get as far as having a Go variable referencing the RasterBand 
   I'm interested in, and everything I *can* examine about it checks out, 
   but I'm stumped at how to extract the data from the band. The closest  I 
   can figure is the `RasterBand.ReadBlock(…)` method, but can't figure out 
   how to use it… admittedly, it seems to require the use of an 
   `unsafe.Pointer` and I don't really understand those. Any guidance, 
   documentation, gists, etc. that might help me out would be very much 
   appreciated!


Thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/a1b3491e-25d4-431d-9349-034e80a004b0n%40googlegroups.com.

[go-nuts] Embedding any (empty interface) in a struct in Go

[sick...@gmail.com]

Consider a scenario where you want to embed an error within any type of 
variable.

This can be achieved by embedding any in a struct and it seems to be a 
valid syntax in Go 1.9.
type WithError struct { 
any 
err error
} 
func main() { 
a := WithError{ any: 5, err: nil, } 
fmt.Printf("%+v\n", a)
} 

Does this violate any community best practices? 
Is there an alternative pattern that can achieve a similar result?

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/0a6bbea5-c37b-4ce4-89d6-2b44d7868639n%40googlegroups.com.

[go-nuts] [security] Go 1.19.3 and Go 1.18.8 pre-announcement

[announce]

Hello gophers,

We plan to issue Go 1.19.3 and Go 1.18.8 on Tuesday, November 1.

These minor releases include PRIVATE security fixes to the standard library.

Following our security policy, this is the pre-announcement of those releases.

Thanks,
Tatiana and Heschi for the Go team

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/SqV2R4AQQTOjO-b9c0RHQA%40geopod-ismtpd-2-3.

[go-nuts] First stable version of several package under golang.org/x/

['Michel Levieux' via golang-nuts]

Hi all,

I was bumping the dependencies of our codebase today, and noticed that many 
packages under golang.org/x/ had upgraded from a pseudo-version to a real, 
semver version:

-   golang.org/x/net 
 v0.
0.0-20220927171203-f486391704dc // indirect 
-golang.org/x/sync 
 
v0.0.0-20220923202941-7f9b1623fab7 // indirect 
-   golang.org/x/sys 
 v0.
0.0-20220928140112-f11e5e49a4ec // indirect
-   golang.org/x/time 
 
v0.0.0-2022090347-f3bd1da661af // indirect 
+   golang.org/x/net  v0.1.0 // 
indirect 
+golang.org/x/sync  v0.1.0 
// indirect 
+   golang.org/x/sys  v0.1.0 // 
indirect
+   golang.org/x/time  v0.1.0 
// indirect

...and others. Is there something we should be aware of concerning this 
upgrade? I've seen no announcement everywhere I've been looking, which 
makes me suppose there's nothing very interesting going on there but that 
surprised me.

If anyone has more information, I'll gladly take it! Thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/9baee648-697f-4729-ace0-bb541884f599n%40googlegroups.com.

[go-nuts] what is the origin of Go's reference time system?

[Ayan George]

I'm really impressed by the simplicity of Go's time formatting and parsing 
mechanisms -- particularly when compared with strftim().

Does anyone know the origin or history of it?  Is there a precedent for 
using reference layouts that include string like 2006, January, etc.?

Do other languages provide something similar or is this completely unique 
to Go?

Can someone point me to or describe the history of Go's time formatting 
method?

-ayan

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/baa45515-cde6-4c7c-a34c-54ddf7da807en%40googlegroups.com.

Re: [go-nuts] Any information about adding a trailing comma in ast CompositeLit (for structs)

[David Finkel]

On Mon, Oct 24, 2022 at 9:27 PM Tajmeet Singh  wrote:

> Thanks for looking at it :smile:
>
> Could you elaborate more on how the `SetLines` would work on a
> `token.FileSet`? I thought that `SetLines` is only for the `os.File` types
> :sweat: Do you mean that I should create offsets (using SetLines) before I
> create the FileSet out of it?
>
Sure, so what I was thinking is that you'd use a different fileset than the
one used to parse the AST, and only give it synthetic line-boundaries (the
line-widths don't actually have to make sense), and then, when picking the
token locations for the different elements in the CompositeLiteral, you'd
override the locations of those tokens in the AST so nodes you want to
appear on the same line will appear on the same line. (up to formatting
constraints)

Another other option (which is much simpler -- and I wish I had thought of
before) is to pass the ast.FieldList 
from the StructType  instead
of the slice of fields, then you can use the token.Pos from the Closing
field . If you use Pos()
 for the field to set the `Colon
` field on the KeyValueExpr
, the formatter will see the
line boundaries in between fields as appropriate and match the formatting
of the struct definition. (AFAICT)


>
> reference code:
> https://github.com/tjgurwara99/constr/blob/0e0f8952c652b7a7a10b7238daed90132b924084/construct.go#L31
>
> On Tuesday, 25 October 2022 at 01:07:23 UTC+1 david@gmail.com wrote:
>
>> On Sat, Oct 22, 2022 at 10:00 PM Tajmeet Singh 
>> wrote:
>>
>>> Hello,
>>>
>>> I've been working on a utility to generate constructors for me when I
>>> provide it with a path to the file containing the struct and it `Ident`.
>>> The idea was that I would just create a ast.Node (FuncDecl) with all the
>>> necessary fields but I'm not able to figure out the positioning of the
>>> commas.
>>>
>>> The function that I'm working on looks something like this:
>>>
>>> ```Go
>>> func generateConstructor(typeName string, fields ...*ast.Field)
>>> *ast.FuncDecl {
>>> var elts []ast.Expr
>>>
>>> for _, field := range fields {
>>> elts = append(elts, {
>>> Key:   field.Names[0],
>>> Value: field.Names[0],
>>> })
>>> }
>>> return {
>>> Doc:  nil,
>>> Recv: nil,
>>> Name: ast.NewIdent("New" + typeName),
>>> Type: {
>>> TypeParams: nil,
>>> Params: {
>>> List: fields,
>>> },
>>> Results: nil,
>>> },
>>> Body: {
>>> Lbrace: 1,
>>> List: []ast.Stmt{
>>> {
>>> Results: []ast.Expr{
>>> {
>>> Op: token.AND,
>>> X: {
>>> Type:   ast.NewIdent(typeName),
>>> Elts:   elts,
>>> Rbrace: 1,
>>> },
>>> },
>>> },
>>> },
>>> },
>>> Rbrace: 2,
>>> },
>>> }
>>> }
>>> ```
>>>
>>> And when I provide it with a struct like this:
>>>
>>> ```Go
>>> type Data struct {
>>> something string
>>> data  string
>>> ddint
>>> }
>>> ```
>>> I get this when I do `format.Node` on the generated node `functionDecl`.
>>>
>>> ```Go
>>> func NewData(something string,
>>> data string,
>>> dd int) {
>>> return {something: something,
>>> data: data,
>>> dd: dd}
>>> }
>>> ```
>>>
>>> However, I was thinking that it would look something like this.
>>>
>>> ```Go
>>> func NewData(something string,
>>> data string,
>>> dd int,
>>> ) {
>>> return {something: something,
>>> data: data,
>>> dd: dd,
>>> }
>>> }
>>> ```
>>> (wishful thinking :joy:)
>>>
>>> Anyways, I tried to figure out how the `token.COMMA` is used in the
>>> parser and how the ast is affected but I couldn't figure it out. Maybe it
>>> is related to the `Lbrace` and `Rbrace` values which I'm deliberately
>>> missing. Any help would be much appreciated :smile: :pray:
>>>
>>
>> Looking at the code, I think you're on the right track with Rbrace's
>> location being the determining factor when to insert a comma and move put
>> the closing brace on a new line.
>> My recommendation would be to use `SetLines`
>>  on the (hopefully
>> clean) `token.FileSet` you're passing to `format.Node` to declare some
>> new-line offsets. I don't think the offsets have to make sense as they
>> would in a file that was parsed, but the formatter needs to see the last
>> expression and close brace as being 

Re: [go-nuts] Occasional hard lockup w/ 100% CPU usage in go applications

[Konstantin Khomoutov]

On Wed, Oct 26, 2022 at 08:45:00AM -0500, Robert Engels wrote:

> Trigger a core dump then use gdb on the core file. 

Also, the gdb is usually shipped with the `gcore` helper which can attach to
the specified PID and dump its core.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/20221026150257.rtxcqfb4u2vexwlv%40carbon.

Re: [go-nuts] Occasional hard lockup w/ 100% CPU usage in go applications

[Steven Sokol]

Typo. 1.19.1

On Wednesday, October 26, 2022 at 8:52:16 AM UTC-5 Jan Mercl wrote:

> On Wed, Oct 26, 2022 at 3:28 PM Steven Sokol  
> wrote:
>
> > Currently running go 1.9.1, but I've been having this same issue since I 
> started working on this project about two years ago on 1.5.x.
>
> That version is 5 years old: https://go.dev/doc/devel/release#go1.9
>
> Can you try a recent, supported version? (Go 1.18+)
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/1f88b703-1234-425a-a471-21dfa034516dn%40googlegroups.com.

Re: [go-nuts] Occasional hard lockup w/ 100% CPU usage in go applications

[Jan Mercl]

On Wed, Oct 26, 2022 at 3:28 PM Steven Sokol  wrote:

> Currently running go 1.9.1, but I've been having this same issue since I 
> started working on this project about two years ago on 1.5.x.

That version is 5 years old: https://go.dev/doc/devel/release#go1.9

Can you try a recent, supported version? (Go 1.18+)

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAA40n-XrwHNF_0hua-_6ASG5UdRytLzn431Ra-vt73qodJbgUg%40mail.gmail.com.

Re: [go-nuts] Occasional hard lockup w/ 100% CPU usage in go applications

[Robert Engels]

Trigger a core dump then use gdb on the core file. 

> On Oct 26, 2022, at 8:33 AM, Steven Sokol  wrote:
> 
> 
> I've tried using strace and it just locks up (and requires SIGKILL to break 
> out). What's the best way to get a native sampling, gdb? (Delve won't run on 
> this platform.)
> 
> 
> 
>> On Tuesday, October 25, 2022 at 8:10:12 PM UTC-5 ren...@ix.netcom.com wrote:
>> I would try an external sampling. The stack trace you are looking at is 
>> somewhat artificial. A native sampling will help pin down what exactly is 
>> consuming the cpu (switching to thread mode in htop can be a substitute). 
>> You might also run strace on the pid when it gets in this state to give 
>> additional clues.
>> 
 On Oct 25, 2022, at 7:28 PM, Steven Sokol  wrote:
 
>>> I don't think so. I don't have it capped in the systemd configuration and 
>>> it's not using very much memory even in the runaway state. Here's a 
>>> screenshot of htop - check out the fvUnisocket process at the top of the 
>>> list:
>>> 
>>> https://user-images.githubusercontent.com/350268/197785991-a8a9691b-4499-4e7a-b0c5-557ef2924514.png
>>> 
>>> 
 On Tuesday, October 25, 2022 at 6:42:14 PM UTC-5 ren...@ix.netcom.com 
 wrote:
 Are you certain you haven’t capped the memory and all of the GC threads 
 are spinning trying to claim/allocate memory?
 
>> On Oct 25, 2022, at 4:49 PM, Steven Sokol  wrote:
>> 
> Weird one. I have a go application that, on occasion, will suddenly jump 
> from some very low CPU usage (1% - 10%) to 400% (4 core system). I run 
> this app at a heightened priority and it can nearly lock the machine (Pi 
> CM4) up.
> 
> I can kill it with SIGKILL and I'm able to get a stack trace by sending 
> SIGQUIT. (Here's a gist of the stack trace.)
> 
> The trace would seem to indicate that everything is idle - virtually all 
> the goroutines are in runtime.gopark. None of them ever get out of it. 
> I've tried adding a watchdog timer and it locks up along with all the 
> other goroutines, so it never fires once the runaway event starts.
> 
> The app itself isn't anything special. It listens for data on a set of 
> about  16 Redis pubsub channels and forwards any data it receives to a 
> very limited number of clients over UDP. Throughput ranges from 34 - 38 
> KB/sec, and the message counts are in the hundreds. Most of the time it 
> eats about 5% of one core on the CM4.
> 
> I guess this could be an error on my part, but there's nothing in my code 
> that seems likely to cause this - no obvious place for a multi-threaded 
> tight loop. I thought for a while that it might be some sort of an issue 
> in Redigo (golang redis client) but from what that stack trace shows, it 
> looks like all the redis listeners are happy and healthy and stuck 
> waiting in gopark along with all the other goroutines.
> 
> Has anyone seen this kind of weird before?
> 
> Thanks,
> 
> -S
> 
> 
 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/5465198d-9a48-4db1-98d5-498922b1ac39n%40googlegroups.com.
 
>>> 
>>> 
>>> -- 
>>> You received this message because you are subscribed to the Google Groups 
>>> "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to golang-nuts...@googlegroups.com.
>> 
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/golang-nuts/382df009-cd81-4f62-ae67-03dfc1bb338en%40googlegroups.com.
>> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/3199aecd-6a25-4d77-a103-d5dd6e96c64dn%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/B6DB14C7-EBA8-469D-BA91-F8BD56D23B0B%40ix.netcom.com.

Re: [go-nuts] Occasional hard lockup w/ 100% CPU usage in go applications

[Steven Sokol]

Currently running go 1.9.1, but I've been having this same issue since I 
started working on this project about two years ago on 1.5.x.

On Tuesday, October 25, 2022 at 8:06:35 PM UTC-5 eric.h...@gmail.com wrote:

> What version of GoLang?   tried different ones?
>
> -Eric
> http://www.google.com/profiles/eric.hubbard
>
>
> On Tue, Oct 25, 2022 at 5:28 PM Steven Sokol  
> wrote:
>
>> I don't think so. I don't have it capped in the systemd configuration and 
>> it's not using very much memory even in the runaway state. Here's a 
>> screenshot of htop - check out the fvUnisocket process at the top of the 
>> list:
>>
>>
>> https://user-images.githubusercontent.com/350268/197785991-a8a9691b-4499-4e7a-b0c5-557ef2924514.png
>>
>>
>> On Tuesday, October 25, 2022 at 6:42:14 PM UTC-5 ren...@ix.netcom.com 
>> wrote:
>>
>>> Are you certain you haven’t capped the memory and all of the GC threads 
>>> are spinning trying to claim/allocate memory?
>>>
>>> On Oct 25, 2022, at 4:49 PM, Steven Sokol  wrote:
>>>
>>> Weird one. I have a go application that, on occasion, will suddenly jump 
>>> from some very low CPU usage (1% - 10%) to 400% (4 core system). I run this 
>>> app at a heightened priority and it can nearly lock the machine (Pi CM4) up.
>>>
>>> I can kill it with SIGKILL and I'm able to get a stack trace by sending 
>>> SIGQUIT. (Here's a gist 
>>>  of 
>>> the stack trace.)
>>>
>>> The trace would seem to indicate that everything is idle - virtually all 
>>> the goroutines are in runtime.gopark. None of them ever get out of it. I've 
>>> tried adding a watchdog timer and it locks up along with all the other 
>>> goroutines, so it never fires once the runaway event starts.
>>>
>>> The app itself isn't anything special. It listens for data on a set of 
>>> about  16 Redis pubsub channels and forwards any data it receives to a very 
>>> limited number of clients over UDP. Throughput ranges from 34 - 38 KB/sec, 
>>> and the message counts are in the hundreds. Most of the time it eats about 
>>> 5% of one core on the CM4.
>>>
>>> I guess this could be an error on my part, but there's nothing in my 
>>> code that seems likely to cause this - no obvious place for a 
>>> multi-threaded tight loop. I thought for a while that it might be some sort 
>>> of an issue in Redigo (golang redis client) but from what that stack trace 
>>> shows, it looks like all the redis listeners are happy and healthy and 
>>> stuck waiting in gopark along with all the other goroutines.
>>>
>>> Has anyone seen this kind of weird before?
>>>
>>> Thanks,
>>>
>>> -S
>>>
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to golang-nuts...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/golang-nuts/5465198d-9a48-4db1-98d5-498922b1ac39n%40googlegroups.com
>>>  
>>> 
>>> .
>>>
>>>
>>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts...@googlegroups.com.
>>
> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/382df009-cd81-4f62-ae67-03dfc1bb338en%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/d8e6a578-38e7-45be-ba27-5cf146b9571dn%40googlegroups.com.

Re: [go-nuts] Occasional hard lockup w/ 100% CPU usage in go applications

[Steven Sokol]

I've tried using strace and it just locks up (and requires SIGKILL to break 
out). What's the best way to get a native sampling, gdb? (Delve won't run 
on this platform.)



On Tuesday, October 25, 2022 at 8:10:12 PM UTC-5 ren...@ix.netcom.com wrote:

> I would try an external sampling. The stack trace you are looking at is 
> somewhat artificial. A native sampling will help pin down what exactly is 
> consuming the cpu (switching to thread mode in htop can be a substitute). 
> You might also run strace on the pid when it gets in this state to give 
> additional clues.
>
> On Oct 25, 2022, at 7:28 PM, Steven Sokol  wrote:
>
> I don't think so. I don't have it capped in the systemd configuration and 
> it's not using very much memory even in the runaway state. Here's a 
> screenshot of htop - check out the fvUnisocket process at the top of the 
> list:
>
>
> https://user-images.githubusercontent.com/350268/197785991-a8a9691b-4499-4e7a-b0c5-557ef2924514.png
>
>
> On Tuesday, October 25, 2022 at 6:42:14 PM UTC-5 ren...@ix.netcom.com 
> wrote:
>
>> Are you certain you haven’t capped the memory and all of the GC threads 
>> are spinning trying to claim/allocate memory?
>>
>> On Oct 25, 2022, at 4:49 PM, Steven Sokol  wrote:
>>
>> Weird one. I have a go application that, on occasion, will suddenly jump 
>> from some very low CPU usage (1% - 10%) to 400% (4 core system). I run this 
>> app at a heightened priority and it can nearly lock the machine (Pi CM4) up.
>>
>> I can kill it with SIGKILL and I'm able to get a stack trace by sending 
>> SIGQUIT. (Here's a gist 
>>  of the 
>> stack trace.)
>>
>> The trace would seem to indicate that everything is idle - virtually all 
>> the goroutines are in runtime.gopark. None of them ever get out of it. I've 
>> tried adding a watchdog timer and it locks up along with all the other 
>> goroutines, so it never fires once the runaway event starts.
>>
>> The app itself isn't anything special. It listens for data on a set of 
>> about  16 Redis pubsub channels and forwards any data it receives to a very 
>> limited number of clients over UDP. Throughput ranges from 34 - 38 KB/sec, 
>> and the message counts are in the hundreds. Most of the time it eats about 
>> 5% of one core on the CM4.
>>
>> I guess this could be an error on my part, but there's nothing in my code 
>> that seems likely to cause this - no obvious place for a multi-threaded 
>> tight loop. I thought for a while that it might be some sort of an issue in 
>> Redigo (golang redis client) but from what that stack trace shows, it looks 
>> like all the redis listeners are happy and healthy and stuck waiting in 
>> gopark along with all the other goroutines.
>>
>> Has anyone seen this kind of weird before?
>>
>> Thanks,
>>
>> -S
>>
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/5465198d-9a48-4db1-98d5-498922b1ac39n%40googlegroups.com
>>  
>> 
>> .
>>
>>
>>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts...@googlegroups.com.
>
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/382df009-cd81-4f62-ae67-03dfc1bb338en%40googlegroups.com
>  
> 
> .
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/3199aecd-6a25-4d77-a103-d5dd6e96c64dn%40googlegroups.com.

Re: [go-nuts] Analyse postgresql error

[Konstantin Khomoutov]

On Wed, Oct 26, 2022 at 01:38:51PM +0300, David Harel wrote:

[...]
> func errorCheckResult(err error) string {
> if err == nil {
> return ""
> }
> pqerr := err.(*pq.Error)
> switch pqerr.Get('C') {
> case "23505":
> return "Key violation"
> }
> return "Error unknown"
> }
> 
> Now I want to be able to test my code thereby generating this type of error
> in the testing function.
> So I need to create an error variable of the type pq.Error and use it as a
> parameter to db.Mock. Something like:
> 
> mockDB := wrapper.NewMockQuerier(ctrl)
> // The part that breaks
> err := fmt.Errorf("pq: %s", `duplicate key value violates unique constraint
> "treatment_pk"
> {"Severity":"ERROR","Code":"23505","Message":".MoreStuff."}`)
> 
> mockDB.EXPECT().AppointmentUpdate(gomock.Any(), gomock.Any()).Return(err)
> 
> Any idea?

Well, look at what you do:

  fmt.Errorf("pq: %s", `some long text`)

creates a value of some (irrelevant) type implementing the standard interface
error and containing a string which is obtained by calling fmt.Sprintf on the
aguments. The "%s" verb in the format string means "get the string
representation of the matching argument and insert in into the format string
in place of the verb". Your argument is itself a string, and it is completely
opaque to fmt.Errorf - it's taken "as is" and is not interpreted in any way.

Please stop and think of this for a moment: even if the contents of this
string would be somehow interpreted, how would the code in package fmt guess
it has to produce a value of type pg.Error from that string, and how exactly?

So, if you need to create a *pq.Error, go on and do just that in your
mock. Based on [1], you could do something like

  return {
  Code: "23505",
  Severity: "ERROR',
  Message: `duplicate key value violates unique constraint "treatment_pk"`,
  }

 1. 
https://github.com/lib/pq/blob/d65e6ae4bdd1c86b16cd6d2bcff4fe970dc697b4/error.go#L25

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/20221026124535.rlmptmaonqsvtsup%40carbon.

Re: [go-nuts] Analyse postgresql error

[David Harel]

Thanks for your reply.

After I submitted my question I made some progress and found that:
The infrastructure of my solution is using the pq package - import "
github.com/lib/pq"
The marshaling method is not relevant anymore.
I can analyse the error by casting to pq.Error.

Code example:

func errorCheckResult(err error) string {
if err == nil {
return ""
}
pqerr := err.(*pq.Error)
switch pqerr.Get('C') {
case "23505":
return "Key violation"
}
return "Error unknown"
}

Now I want to be able to test my code thereby generating this type of error
in the testing function.
So I need to create an error variable of the type pq.Error and use it as a
parameter to db.Mock. Something like:

mockDB := wrapper.NewMockQuerier(ctrl)
// The part that breaks
err := fmt.Errorf("pq: %s", `duplicate key value violates unique constraint
"treatment_pk"
{"Severity":"ERROR","Code":"23505","Message":".MoreStuff."}`)

mockDB.EXPECT().AppointmentUpdate(gomock.Any(), gomock.Any()).Return(err)

Any idea?


On Tue, Oct 25, 2022 at 7:21 PM Konstantin Khomoutov 
wrote:

> On Fri, Oct 21, 2022 at 08:17:16AM -0700, David Harel wrote:
>
> > Newbie on golang. Using sqlc: https://github.com/kyleconroy/sqlc in the
> > environment created by Karl Keefer:
> > https://github.com/karlkeefer/pngr/actions/workflows/build.yml, thanks
> Karl.
> > My queries use querier which is auto generated by sqlc.
> >
> > Sometime I get an error result in my query such as key violation.
> > I want to check the error code of such an error.
> >
> > as one of the guys suggested I can look into the object using Marshal
> > and I noticed that the error contains a very elaborated, like:
> > {"Severity":"ERROR","Code":"23505","Message":"duplicate key value
> violates
> > unique constraint \"treatment_pk\"","Detail":"Key (customer_id,
> patient_ttz,
> > therapist_id, appointment_timestamp)=(1, 060525649, 160, 2022-10-20
> > 10:30:00) already
> > exists.","Hint":"","Position":"","InternalPosition":"",
> > "InternalQuery":"","Where":"","Schema":"public",
> > "Table":"treatment","Column":"","DataTypeName":"",
> > "Constraint":"treatment_pk","File":"nbtinsert.c","Line":"434",
> > "Routine":"_bt_check_unique"}
>
> What is "Marshal" you're referring to here?
> Is this encoding/json.Marshal called on some value returned by that
> "querier generated by sqlc"?
>
> > Trying to get the same result using straight forward approach like:
> > fmt.Printf("%+v\n", err) gives me a much simpler object display.
> >
> > How can I get the "smart" content of the error object like the "Code"
> > member?
>
> If my guesseneering is correct, you can just access the Code field of that
> value you passed through "Marshal" and analyze it using any convenient
> method
> such as an if or switch statement.
>
> From [1], I gather that "23505" is indeed a standardized error code for
> violation of unique key constraints in PostgreSQL, so basically you could
> do
>
>   const uniqueKeyViolation = "23505"
>
>   switch result.Code {
>   case uniqueKeyViolation:
> // Do something sensible
>   default:
> log.Println("unexpected failure: ", result.Code)
>   }
>
> Still, please notice that all of the above is what's called "psychic
> debugging" and might miss the point partially or completely because your
> problem statement is not an MCVE [2]. I'm not sure you could sensibly
> create
> one (as it could have required posting swaths of generated code or
> something
> like this) so please don't take it as a blame.
>
>  1. https://www.postgresql.org/docs/current/errcodes-appendix.html
>  2. https://stackoverflow.com/help/minimal-reproducible-example
>
>

-- 
דוד הראל
עמוקה
ד.נ. מרום הגליל
1380200
טל: 054-2263892

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CA%2BE_%3DOZx7apxRe2fW9q5Uqxou-1PnGRSyZLJer6Ha09MEpe7%3Dg%40mail.gmail.com.

Re: [go-nuts] gollvm build fails

[Alex Markin]

Thank you for your advice!

I compared the good and the bad build log and found out the lack of rule
zgoarchat all. So I mentioned the difference in makefiles and it turned out
that I used different revisions of gollvm. It is interesting that with gcc
this problem did not appear (but in both gcc and llvm cases I got the
linkage problem described below).

After that I got the next problem: cmake failed to set the
SIZEOF_STRUCT_EPOLL_EVENT and STRUCT_EPOLL_EVENT_FD_OFFSET. I do not know
what to do with it. So I just hardcoded them in the
cmake/modules/AutoGenGo.cmake (values 12 and 4 respectively). After that,
the cmake worked fine.

Then the build failed with other problem:

/home/alex/test/gollvm/build-debug/./bin/llvm-goc -o /home/alex/test/gollvm
/build-de
bug/tools/gollvm/gotools/go go_.o -I /home/alex/test/gollvm
/build-debug/tools/gollvm/libgo -L /home/alex/test/gollvm/build-debug/tools/
gollvm/libgo /home/ale
x/test/gollvm/build-debug/tools/gollvm/libgo/libgotool.a
/usr/lib/gcc/x86_64-pc-linux-gnu/12.2.0/../../../../x86_64-pc-linux-gnu/bin/ld.gold:
error: cannot open crtn.o: Нет такого файла или каталога
/usr/lib/gcc/x86_64-pc-linux-gnu/12.2.0/../../../../x86_64-pc-linux-gnu/bin/ld.gold:
warning: skipping incompatible /usr/lib/libm.so while searching for m
/usr/lib/gcc/x86_64-pc-linux-gnu/12.2.0/../../../../x86_64-pc-linux-gnu/bin/ld.gold:
error: cannot find -lm
/usr/lib/gcc/x86_64-pc-linux-gnu/12.2.0/../../../../x86_64-pc-linux-gnu/bin/ld.gold:
warning: skipping incompatible /usr/lib/libc.so while searching for c
/usr/lib/gcc/x86_64-pc-linux-gnu/12.2.0/../../../../x86_64-pc-linux-gnu/bin/ld.gold:
error: cannot find -lc
/usr/lib64/crt1.o:function _start: error: undefined reference to
'__libc_start_main'
/home/alex/test/gollvm/llvm-project/llvm/tools/gollvm/gofrontend/libgo/runtime/go-main.c:59:
error: undefined reference to 'abort'
/usr/lib/gcc/x86_64-pc-linux-gnu/12.2.0/libgcc.a(generic-morestack.o):function
__morestack_fail: error: undefined reference to 'writev'
/usr/lib/gcc/x86_64-pc-linux-gnu/12.2.0/libgcc.a(generic-morestack.o):function
__morestack_fail: error: undefined reference to 'abort'
/usr/lib/gcc/x86_64-pc-linux-gnu/12.2.0/libgcc.a(generic-morestack.o):function
__morestack_release_segments: error: undefined reference to 'syscall'
/usr/lib/gcc/x86_64-pc-linux-gnu/12.2.0/libgcc.a(generic-morestack.o):function
__morestack_release_segments: error: undefined reference to 'free'
/usr/lib/gcc/x86_64-pc-linux-gnu/12.2.0/libgcc.a(generic-morestack.o):function
__morestack_release_segments: error: undefined reference to 'free'
...

That was even more weird. And also that was almost like the issue 26405 (
https://github.com/golang/go/issues/26405). I do not know the good way to
solve it so I just hardcoded in the file driver/GnuTools.cpp the following
things:

cmdArgs
.push_back(args.MakeArgString(toolchain().getFilePath(/usr/lib64/crt1.o)));
cmdArgs.push_back(args.MakeArgString("/usr/lib64/crti.o"));
cmdArgs.push_back(args.MakeArgString("-L/usr/lib64/"));
cmdArgs.push_back(args.MakeArgString("/usr/lib64/crtn.o"));

After that my build finished successfully.

Again, thank you for your help.

вт, 25 окт. 2022 г. в 17:04, Than McIntosh :

> Weird. I am scratching my head.
>
> At this point if I was debugging it myself on your system I would rerun
> cmake passing it the "--trace-expand" flag (which will generate giant
> volumes of output), then look in the trace to see what is happening when
> the cmake rules in question fire. You should see something like
>
> ...
> /mybuildarea/llvm-project/llvm/tools/gollvm/libgo/CMakeLists.txt(232):
>  set(zgoarchdotgo ${libgo_binroot}/zgoarch.go )
> /mybuildarea/llvm-project/llvm/tools/gollvm/libgo/CMakeLists.txt(233):
>  set(zgoarchtmp ${libgo_binroot}/zgoarch.go.tmp )
> /mybuildarea/llvm-project/llvm/tools/gollvm/libgo/CMakeLists.txt(234):
>  mkzgoarch(${goarch} ${zgoarchtmp} ${libgo_scriptroot} )
> /mybuildarea/llvm-project/llvm/tools/gollvm/cmake/modules/AutoGenGo.cmake(75):
>  file(REMOVE ${outfile} )
> /mybuildarea/llvm-project/llvm/tools/gollvm/cmake/modules/AutoGenGo.cmake(76):
>  file(WRITE ${outfile} package goarch\n\n )
> /mybuildarea/llvm-project/llvm/tools/gollvm/cmake/modules/AutoGenGo.cmake(78):
>  file(APPEND ${outfile} const GOARCH = \"${goarch}\"\n\n )
> ...
>
> If these rules are not firing, then hopefully the trace output will have
> some suggestion as to why.
>
> Thanks, Than
>
>
>
>
> On Tue, Oct 25, 2022 at 3:29 AM Alex Markin  wrote:
>
>> > Do you have tools/gollvm/libgo/zgoarch.go.tmp in your build area? What
>> sort of content is in that file?
>>
>> No
>>
>> $ ninja -v -d explain tools/gollvm/libgo/zgoarch.go
>> ninja: error: unknown target 'tools/gollvm/libgo/zgoarch.go'
>>
>> $ gcc -dumpmachine
>> x86_64-pc-linux-gnu
>>
>> $ clang -dumpmachine
>> x86_64-pc-linux-gnu
>>
>> I tried manually setting up different target triples (for example the
>> same as on a working machine) but
>> it did not help.
>>
>>
>> чт, 20 окт. 2022 г. в 15:48,