Re: [go-nuts] Re: pkg.go.dev: redirect URL for latest major version or big red warning for old version

[Tom Payne]

I see this working for modules, but not major versions.

For example the whole github.com/golang/protobuf module is deprecated:

https://github.com/golang/protobuf/blob/75de7c059e36b64f01d0dd234ff2fff404ec3374/go.mod#L1

However, there are is no deprecation message for github.com/google/go-github
:

https://github.com/google/go-github/blob/b603120bf932e99e8cc0a9fab0693fa6b5f92c4d/go.mod

Is there any way for google/go-github add a deprecation message for their
v0/v1/.../v60 major versions?

On Mon, 8 Apr 2024 at 22:10, Jason Phillips 
wrote:

> If a major version is truly obsolete the maintainers can deprecate it. In
> which case the pkg.go.dev UI stands out a bit more. Take for example:
> github.com/golang/protobuf/proto
> .
>
> On Monday, April 8, 2024 at 2:12:54 PM UTC-4 twp...@gmail.com wrote:
>
>> Is there a way to get an unchanging URL that always redirects to the
>> latest major version of a package on pkg.go.dev?
>>
>> Right now, if I go to
>>   https://pkg.go.dev/$PACKAGE_NAME
>> I get the documentation for either v0 or v1 of a package, which is out of
>> date if there's a more recent major version.
>>
>> What I would like is a URL that given $PACKAGE_NAME always redirects to
>> the latest major version of that package. For example, a URL something like
>>   https://pkg.go.dev/$PACKAGE_NAME@latest
>> which would redirect to
>>   https://pkg.go.dev/$PACKAGE_NAME/v2
>> if the latest major version of $PACKAGE_NAME is v2.
>>
>> The reasons for this request are:
>> 1. I'm usually using the latest version of the package, and so I want to
>> see the docs for the most recent version.
>> 2. If users stumble across documentation for an old major version by
>> default then there's a risk that they'll use or evaluate the old major
>> version by default. Usually there's a good reason why users should not be
>> using the old major version.
>>
>> Right now, pkg.go.dev displays a tiny black text on a grey background
>> line at the top of  the page that says something like:
>>   The highest tagged major version is v61.
>> See https://pkg.go.dev/github.com/google/go-github for an example. This
>> is extremely easy to miss. Alternatively, this banner should be in big red
>> letters - if you're using this version of the module then you're probably
>> doing something wrong.
>>
>> If there's support for this new URL and/or the banner usability
>> improvement, then I'd be happy to contribute a fix.
>>
>> Thoughts?
>>
>> Regards,
>> Tom
>>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "golang-nuts" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/golang-nuts/BQCy9ocMVUs/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/473e82f2-c201-4f87-857c-86eea02f6fb2n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAHY_QbTosXCQTeGbPKssGuGR-sfiR8Qf9b85iWsDd%3D6TVrTAzA%40mail.gmail.com.

[go-nuts] Alternative text/template function libraries to github.com/Masterminds/sprig?

[Tom Payne]

github.com/Masterminds/sprig is a popular library of template functions,
used by some popular projects, e.g. Kubernetes Helm.

Unfortunately, Masterminds/sprig also has a number of inherent flaws:
1. The order of arguments to many of its functions is incompatible with
text/template's pipeline syntax.
2. Many of its functions do not handle strings, []bytes, variable numbers
of arguments, and other argument variations in a sensible way.
3. It has, at the time of writing, 78 open issues
, most of them unaddressed,
and has seen only minor maintenance activity over the last few years.
4. Its function names do not follow Go's naming conventions.

#1 and #2 cannot be fixed in a backwards-compatible way. #3 means that
fixes aren't practically accepted anyway. #4 just means that templates
using Masterminds/sprig are ugly to Go developer eyes.

Instead, I want a library of template functions that works well with
text/template, has decent ergonomics, follows Go's naming conventions, and
is actively maintained. I don't care about backwards compatibility with
sprig as this is impossible to achieve anyway.

Before I start such a project, are there any existing good existing
alternatives to Masterminds/sprig?

Many thanks,
Tom

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAHY_QbTxDH_h4f1_o3DFg-9Jv-VfCwPGW4V%3DBFfCYNaf3A2jAA%40mail.gmail.com.

[go-nuts] Getting cross-package test coverage data?

[Tom Payne]

What's the current state of getting combined cross-package test coverage 
data? Specifically:

   - I have a lib directory containing my library code, which has some 
   tests, but not many.
   - I have a cmd directory containing my main code and many tests that 
   exercises the code in lib .
   - I want to know what code in lib is covered by my tests in cmd (and, 
   more specifically, which code in lib is *not* covered by the tests so I 
   can add more tests).

My current understanding is I need to use a third-party tool to merge 
coverage, like https://github.com/ory/go-acc.

Is this correct?

Many thanks,
Tom

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/3e24588a-005e-4072-80fe-ff7a912bd368%40googlegroups.com.

Re: [go-nuts] Re: marshal multiple json documents which may have different format versions into the latest struct version

[Tom Payne]

Possibly also useful if you're dealing with JSON documents with different 
structures:
  https://github.com/twpayne/go-jsonstruct
This will generate the most specific Go struct possible that covers all the 
example documents that you give it. In your example it will generate:

package main

type T struct {
Cities   []string `json:"Cities,omitempty"`
Version  string   `json:"Version"`
Villages []string `json:"Villages,omitempty"`
}

which is the union of all versions.

If you already know exactly what structures you will receive, then Manilo's 
approach of first decoding into a struct only the Version field is better 
though, especially when combined with Go's runtime type switches.

Regards,
Tom

On Wednesday, April 29, 2020 at 9:31:00 AM UTC+1, Chris Burkert wrote:
>
> That sounds like a good plan. I'm going to try that. Thank you Manlio!
>
> Am Di., 28. Apr. 2020 um 15:11 Uhr schrieb Manlio Perillo <
> manlio...@gmail.com >:
>
>> On Tuesday, April 28, 2020 at 10:52:56 AM UTC+2, Chris Burkert wrote:
>>>
>>> Dear all,
>>>
>>> my application users shall be able to provide multiple json documents 
>>> (files and urls) which I'd like to marshall into one structure. 
>>> Additionally these json documents may have different versions. I know how 
>>> to marshal a document into a version specific struct if I know the format 
>>> version before (for simplicity of this example I don't handle errors): 
>>> https://play.golang.org/p/ixVI5CzPqFP
>>>
>>> What I would like (in the example the village field was renamed to 
>>> cities ) is a struct of type ModelV2 with all four values merged in Cities.
>>>
>>> Is there a best practice for a backwards compatible behavior which:
>>>
>>>- identifies the json format version of each document
>>>- skips that document if it is higher than the supported format 
>>>version in my application
>>>- merges supported format versions into ONE struct
>>>
>>> Of course I have to implement the semantics on my own but how can I 
>>> approach the topic?
>>>
>>>
>> You can first unmarshal a struct containing only the Version field.  As 
>> an example:
>> https://play.golang.org/p/1oDzdWlTCfC 
>> 
>>
>>
>> Manlio 
>>
>>> thanks
>>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golan...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/274e489d-afdb-4ec9-a5b3-26440364c489%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/d035d66f-082b-41a5-9c31-f65ffe766515%40googlegroups.com.

Re: [go-nuts] testing code that uses ioutil.ReadDir?

[Tom Payne]

Testing code that uses the os and ioutil packages is tricky. I created
  https://pkg.go.dev/github.com/twpayne/go-vfs
  https://pkg.go.dev/github.com/twpayne/go-vfs/vfst
to make it easier.

Key features:
- Everything eventually calls the underlying os/ioutil function, so you get 
real behavior, not fake (and possibly incorrect) mock behavior.
- Makes it really easy to create, populate, and tear down temporary 
directories for hermetic tests.

It's best demonstrated by example:
  
https://pkg.go.dev/github.com/twpayne/go-vfs/vfst?tab=doc#example-NewTestFS

Using vfs.FS you can create instances that fail in whatever way you want, 
for example:


package main

import (
"os"
"testing"

"github.com/twpayne/go-vfs"
"github.com/twpayne/go-vfs/vfst"
)

// a readDirFailer wraps a vfs.FS but all calls to ReadDir fail with
// os.ErrPermission.
type readDirFailer struct {
vfs.FS
}

func (readDirFailer) ReadDir(dirname string) ([]os.FileInfo, error) {
return nil, os.ErrPermission
}

// testReadDirFailer tests that calling fs.ReadDir returns an error.
func testReadDirFailer(t *testing.T, fs vfs.FS) {
if _, err := fs.ReadDir("/"); err == nil {
t.Errorf("expected error from ReadDir")
}
}

func TestX(t *testing.T) {
fs, cleanup, err := vfst.NewTestFS(nil)
if err != nil {
t.Fatal(err)
}
defer cleanup()

testReadDirFailer(t, readDirFailer{FS: fs})
}


HTH,
Tom

On Saturday, April 11, 2020 at 12:16:30 AM UTC+1, Ian Lance Taylor wrote:
>
> On Fri, Apr 10, 2020 at 3:37 PM 'K Richard Pixley' via golang-nuts 
> > wrote: 
> > 
> > I have some code.  It uses ioutil.ReadDir which returns a possible 
> error. 
> > 
> > My testing is at 99% code coverage because I can't figure out a way to 
> > set up a situation in which os.Open succeeds on a directory, but 
> > ioutil.ReadDir fails. 
> > 
> > I can get to 100% code coverage if I throw away the err rather than 
> > testing for it and reporting it but that seems a bit... well... 
> > disingenuous.  I've looked down the calls to the kernel call 
> > getdirentries but the only errno I can see being relevant to a go 
> > program is EIO and I can't think of a way to force that to happen on a 
> > real file system. 
> > 
> > I can imagine a fake file system created specifically for this purpose, 
> > but that seems an awfully long way around the barn. 
> > 
> > How can I get to 100% code coverage?  Or should I just give up on 
> > finding a way to cover that last, single line of error handling code? 
>
> I'm not proud.  Here is an example program for which os.Open(dirname) 
> succeeds but ioutil.ReadDir(dirname) fails.  You may have to adjust 
> pathMax and nameMax for your system.  Whether you actually want to use 
> this technique is left to you. 
>
> Ian 
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/0a3d1ff3-2dcb-4c75-afb5-041db4d5a435%40googlegroups.com.

[go-nuts] When to use runtime.GOOS vs. build flags/file suffixes?

[Tom Payne]

Go allows OS-specific code to be selected two different ways, either using 
the runtime.GOOS constant or with file-level build tags like "//+build 
linux" or (equivalently) OS-specific source file suffixes like _linux.

What's the recommended way to decide when to use which?

As far as I can tell:
- There is no runtime advantage to either. runtime.GOOS is a constant, so 
the compiler eliminates any unreachable branches. File-level build tags 
mean that irrelevant files don't even reach the compiler. Either way, the 
irrelevant code is not included in the final binary and incurs no runtime 
cost.
- When using build tags, the fact that irrelevant code doesn't reach the 
compiler means that errors can creep in. For example, compile errors in 
foo_linux.go will be unnoticed if you only ever compile on Windows. In 
contrast, using runtime.GOOS will mean that compile errors are found 
whatever OS you compile on.
- Only build flags and OS-specific source file suffixes give you control 
over imports.

Therefore, it would seem that the recommendation of when to use 
runtime.GOOS vs. build flags/file suffixes is:
- Use runtime.GOOS unless you need OS-specific imports or your OS-specific 
code is so different that it really deserves to be in a separate source 
file.

Is this recommendation reasonable? What could be improved?

Cheers,
Tom

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/64d5d1c5-ed1f-41f9-9190-f007e327f146%40googlegroups.com.

Re: [go-nuts] Recommended way to prevent my project being built with an old Go version?

[Tom Payne]

For what it's worth, another way to do this is to use a language feature 
that is only present in a more recent version. For example

  const _ = 0o0 // Octal integer literals with the 0o prefix need Go 1.13 
or later

Obviously this only works if there's a new language feature introduced at 
exactly the minimum Go version that you want.

On Thursday, April 2, 2020 at 1:26:04 AM UTC+1, Tom Payne wrote:
>
> Thanks Ian for the fast and authoritative answer. I'll do what you suggest.
>
> Cheers,
> Tom
>
> On Tuesday, March 31, 2020 at 2:31:19 AM UTC+1, Ian Lance Taylor wrote:
>>
>> On Mon, Mar 30, 2020 at 6:22 PM Tom Payne  wrote: 
>> > 
>> > Go's backwards compatibility guarantee is fantastic, but only applies 
>> to the language, not the standard library. How to I cause a build-time 
>> failure if someone tries to build my project with a too-old Go version? 
>>
>> Pedantically, I would say that Go's backward compatibility does apply 
>> to the standard library, but that what you are talking about is 
>> forward compatibility. 
>>
>>
>> > I have a Go project that uses (or would like to use) a few features 
>> introduced in the standard library more recently, e.g. the %w verb in 
>> fmt.Errorf for wrapping errors (introduced in Go 1.13), and a fix to the 
>> text/template library (merged for Go 1.14). The nature of these features 
>> and fixes mean that my code will compile and build fine, but will fail at 
>> runtime when a codepath that relies on the feature or fix is executed, 
>> which will result in a late, weird error. I would like an early, loud 
>> failure at build time instead. 
>> > 
>> > What's the best way to achieve this? As far as I can tell, there are a 
>> few options: 
>> > 
>> > 
>> > The Go version can be determined by either the runtime.Version() 
>> function or by the presence of build flags (e.g. go1.13, go1.14, etc.). 
>> > 
>> > 
>> > Calling runtime.Version() cannot result in a build time error (it can 
>> only be called once the code is running, which can only happen after a 
>> successful build) but could be used in either a test (so the old Go version 
>> gets caught when "go test" is run) or in an init() function to (say) panic 
>> on startup when tests or the program are run. This would look something 
>> like: 
>> > 
>> > import "runtime" 
>> > 
>> > func init() { 
>> > if runtime.Version() < "1.13" { // string comparisons are not a 
>> good way to compare version strings, but you get the idea 
>> > panic("go version too old") 
>> > } 
>> > } 
>> > 
>> > 
>> > I can create a Go file with build flags that is only built on older 
>> versions of Go, something like: 
>> > 
>> > // +build !go1.13 
>> > 
>> > build with go 1.13 or later // this is deliberately not valid go 
>> syntax 
>> > 
>> > This causes an error if built with an earlier version of Go than 1.13, 
>> but the error message isn't very intuitive (something like 
>> "filename.go:3:1: expected 'package', found build") and the invalid Go 
>> syntax might confuse other Go tooling which tends to assume that every .go 
>> file contains more-or-less valid Go code. 
>> > 
>> > 
>> > Please note that I want to use these Go standard library features and 
>> fixes and provide an early, loud warning if they are not available. I know 
>> that I can use build flags to provide different code to different Go 
>> versions, but if I do that then I still have to maintain code for older Go 
>> versions. The core of this question is: how do I get a build error if my Go 
>> version is too old? 
>> > 
>> > 
>> > What's the recommended way to ensure a minimum Go version at build 
>> time? One of the above suggestions or another way? 
>>
>>
>> I think that most people use build tags, but instead of introducing a 
>> compilation error, they either provide reduced functionality when 
>> built with older Go versions, or they write something like 
>>
>> const s = "this package requires Go 1.14" + 0 
>>
>> Ian 
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/cec1a5c1-3a2d-41ec-ac82-3942f8751451%40googlegroups.com.

Re: [go-nuts] Recommended way to prevent my project being built with an old Go version?

[Tom Payne]

Thanks Ian for the fast and authoritative answer. I'll do what you suggest.

Cheers,
Tom

On Tuesday, March 31, 2020 at 2:31:19 AM UTC+1, Ian Lance Taylor wrote:
>
> On Mon, Mar 30, 2020 at 6:22 PM Tom Payne > 
> wrote: 
> > 
> > Go's backwards compatibility guarantee is fantastic, but only applies to 
> the language, not the standard library. How to I cause a build-time failure 
> if someone tries to build my project with a too-old Go version? 
>
> Pedantically, I would say that Go's backward compatibility does apply 
> to the standard library, but that what you are talking about is 
> forward compatibility. 
>
>
> > I have a Go project that uses (or would like to use) a few features 
> introduced in the standard library more recently, e.g. the %w verb in 
> fmt.Errorf for wrapping errors (introduced in Go 1.13), and a fix to the 
> text/template library (merged for Go 1.14). The nature of these features 
> and fixes mean that my code will compile and build fine, but will fail at 
> runtime when a codepath that relies on the feature or fix is executed, 
> which will result in a late, weird error. I would like an early, loud 
> failure at build time instead. 
> > 
> > What's the best way to achieve this? As far as I can tell, there are a 
> few options: 
> > 
> > 
> > The Go version can be determined by either the runtime.Version() 
> function or by the presence of build flags (e.g. go1.13, go1.14, etc.). 
> > 
> > 
> > Calling runtime.Version() cannot result in a build time error (it can 
> only be called once the code is running, which can only happen after a 
> successful build) but could be used in either a test (so the old Go version 
> gets caught when "go test" is run) or in an init() function to (say) panic 
> on startup when tests or the program are run. This would look something 
> like: 
> > 
> > import "runtime" 
> > 
> > func init() { 
> > if runtime.Version() < "1.13" { // string comparisons are not a 
> good way to compare version strings, but you get the idea 
> > panic("go version too old") 
> > } 
> > } 
> > 
> > 
> > I can create a Go file with build flags that is only built on older 
> versions of Go, something like: 
> > 
> > // +build !go1.13 
> > 
> > build with go 1.13 or later // this is deliberately not valid go 
> syntax 
> > 
> > This causes an error if built with an earlier version of Go than 1.13, 
> but the error message isn't very intuitive (something like 
> "filename.go:3:1: expected 'package', found build") and the invalid Go 
> syntax might confuse other Go tooling which tends to assume that every .go 
> file contains more-or-less valid Go code. 
> > 
> > 
> > Please note that I want to use these Go standard library features and 
> fixes and provide an early, loud warning if they are not available. I know 
> that I can use build flags to provide different code to different Go 
> versions, but if I do that then I still have to maintain code for older Go 
> versions. The core of this question is: how do I get a build error if my Go 
> version is too old? 
> > 
> > 
> > What's the recommended way to ensure a minimum Go version at build time? 
> One of the above suggestions or another way? 
>
>
> I think that most people use build tags, but instead of introducing a 
> compilation error, they either provide reduced functionality when 
> built with older Go versions, or they write something like 
>
> const s = "this package requires Go 1.14" + 0 
>
> Ian 
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/99265a86-1f78-407f-8b70-6152a595c99a%40googlegroups.com.

[go-nuts] Recommended way to prevent my project being built with an old Go version?

[Tom Payne]

Hi,

Go's backwards compatibility guarantee is fantastic, but only applies to 
the language, not the standard library. How to I cause a build-time failure 
if someone tries to build my project with a too-old Go version?

I have a Go project that uses (or would like to use) a few features 
introduced in the standard library more recently, e.g. the %w verb in 
fmt.Errorf for wrapping errors (introduced in Go 1.13 
), and a fix to the 
text/template library (merged for Go 1.14 
). The nature of these features 
and fixes mean that my code will compile and build fine, but will fail at 
runtime when a codepath that relies on the feature or fix is executed, 
which will result in a late, weird error. I would like an early, loud 
failure at build time instead.

What's the best way to achieve this? As far as I can tell, there are a few 
options:


The Go version can be determined by either the runtime.Version() function 
or by the presence of build flags (e.g. go1.13, go1.14, etc.).


Calling runtime.Version() cannot result in a build time error (it can only 
be called once the code is running, which can only happen after a 
successful build) but could be used in either a test (so the old Go version 
gets caught when "go test" is run) or in an init() function to (say) panic 
on startup when tests or the program are run. This would look something 
like:

import "runtime"

func init() {
if runtime.Version() < "1.13" { // string comparisons are not a 
good way to compare version strings, but you get the idea
panic("go version too old")
}
}


I can create a Go file with build flags that is only built on older 
versions of Go, something like:

// +build !go1.13

build with go 1.13 or later // this is deliberately not valid go syntax

This causes an error if built with an earlier version of Go than 1.13, but 
the error message isn't very intuitive (something like "filename.go:3:1: 
expected 'package', found build") and the invalid Go syntax might confuse 
other Go tooling which tends to assume that every .go file contains 
more-or-less valid Go code.


Please note that I want to use these Go standard library features and fixes 
and provide an early, loud warning if they are not available. I know that I 
can use build flags to provide different code to different Go versions, but 
if I do that then I still have to maintain code for older Go versions. The 
core of this question is: how do I get a build error if my Go version is 
too old?


What's the recommended way to ensure a minimum Go version at build time? 
One of the above suggestions or another way?

Many thanks,
Tom



-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/7c9f2948-4187-4ccf-aebe-c524965a8be4%40googlegroups.com.

[go-nuts] Re: Experience report on a large Python-to-Go translation

[Tom Payne]

Really interesting post, thank you.

On iterators without leaking goroutines, have a look at the standard library's 
bufio.Scanner and database/sql.Rows. These provide easy iteration over 
arbitrary sequences in a compact idiomatic form.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/6811412c-cf53-4626-b26b-70e868610d59%40googlegroups.com.

Re: [go-nuts] Unexpected evaluation order in a return statement with multiple operands

[Tom Payne]

Thanks again - I'm learning a lot here!

For info, the background here is that I've been writing code like:

  type T struct {
  SomeField int
  }

  func NewTFromJSON(data []byte) (*T, error) {
  var t T
  return , json.Unmarshal(data, )
  }

which wraps up error handling into a single line. This works when T is a
non-simple type as  does not change, even if the value of t does.

Writing this with T being a simple type, like int, results in the case
described in the original post:

  // NewIntFromJSON relies on undefined behavior, don't use it!
  func NewIntFromJSON(data []byte) (int, error) {
  var i int
  return i, json.Unmarshal(data, )
  }

So, in these cases I will do as Axel suggests and avoid the implicit
dependence on evaluation order.

A go vet check for this would be wonderful, even if it's not perfect. As
long as there aren't any false positives (i.e. warning when the code does
not in fact rely on unspecified behavior), it matters less if there are
false negatives (i.e. not warning when the code actually relies on
unspecified behavior). Maybe a simple test could catch the most common
cases.

On Wed, 15 Jan 2020 at 21:29, Ian Lance Taylor  wrote:

> On Wed, Jan 15, 2020 at 12:24 PM Tom Payne  wrote:
> >
> > Thanks all for the insight and explanation. Could I suggest tweaking the
> Go language specification to emphasize the separation, so it reads:
> >
> >"when evaluating the operands of an expression, assignment, or return
> statement, then all function calls, method calls, and communication
> operations are evaluated in lexical left-to-right order. Any operands that
> are variables take the value of the variable after all function calls,
> methods calls, and communication operations have been evaluated."
>
> That suggested addition is not correct.  The language deliberately
> does not specify when operands that are variables are evaluated.  They
> may be evaluated before function calls, they may be evaluated after.
> Different Go compilers make different decisions here.
>
> If we decided to specify it, I'm sure we would specify that the
> variables are evaluated in left to right order.
>
> Personally I think one good approach would be a vet check that
> complains about statements that both read and write a variable in an
> unspecified order.  But it's fairly hard to catch all possible cases
> of this.
>
> Ian
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAHY_QbTxpmvS3735HPaDeDy9K7j5VY76CCfh%3D3QsGcA%3D3njVug%40mail.gmail.com.

Re: [go-nuts] Unexpected evaluation order in a return statement with multiple operands

[Tom Payne]

Thanks all for the insight and explanation. Could I suggest tweaking the Go
language specification to emphasize the separation, so it reads:

   "when evaluating the operands of an expression, assignment, or return
statement, *then* all function calls, method calls, and communication
operations are evaluated in lexical left-to-right order. *Any operands that
are variables take the value of the variable after all function calls,
methods calls, and communication operations have been evaluated.*"

The presence or absence of function calls does make a difference here. In this
example  the use of the identity()
function changes the result of f():

  func identity(i int) int {
  return i
  }

  func modify(i *int) error {
  *i = 1
  return nil
  }

  func f() (int, error) {
  i := 0
  return identity(i), modify() // returns 0, nil
  }


On Wed, 15 Jan 2020 at 20:50, Robert Engels  wrote:

> I disagree here. Knowing the order of operations/evaluation is part of
> knowing the language. If that’s the case it should be undefined (which
> forces a rewrite) not chose a behavior that seems contrary to the
> documentation and common practice in this area.
>
> On Jan 15, 2020, at 1:39 PM, Axel Wagner 
> wrote:
>
> 
> I don't think "clearly this behavior is not what you would want" is a
> valid generalization.
>
>   type MyStruct{
>   // …
>   }
>
>   func (x *MyStruct) initialize() error {
>   // does some common verification and populates some internal data
> structures
>   return nil
>   }
>
>   func New() (*MyStruct, error) {
>   x := {
>// default fields
>   }
>   return x, x.initialize()
>   }
>
>   func NewWithFoo(f Foo) (*MyStruct, error) {
>x := {
> Foo: f,
>}
>return x, x.intialize()
>   }
>
> You can say that you shouldn't do something like this, but it's IMO
> reasonable code to write and it expects different behavior.
>
> However, as per my previous e-mail, I would still prefer to rewrite this,
> of course :)
>
>
> On Wed, Jan 15, 2020 at 8:33 PM Robert Engels 
> wrote:
>
>> I think the way to look at it is “what would be the behavior of if you
>> were inline creating and initializing a struct containing both values” -
>> clearly this behavior is not what you would want or expect.
>>
>> > On Jan 15, 2020, at 1:25 PM, Paul Jolly  wrote:
>> >
>> > 
>> >>
>> >>  "when evaluating the operands of an expression, assignment, or return
>> statement, all function calls, method calls, and communication operations
>> are evaluated in lexical left-to-right order."
>> >
>> > My understanding goes as follows: the operands of the return statement
>> > are i and modify(). The comma after "return statement" in the above
>> > sentence is then important: because the only "function calls, method
>> > calls, and communication operations" in that list of operands are (is)
>> > modify().
>> >
>> > Hence when i (as in the first operand) is evaluated is not specified.
>> > And therefore it's dangerous to rely on it being one value or another
>> > (which it could be given the example you provide).
>> >
>> > There was even some discussion at the London Gophers November 2019
>> > meetup (where the conundrum was very related
>> >
>> https://docs.google.com/presentation/d/e/2PACX-1vQ07uP_ldnSYzpaNb5AlZZ-_tL2mZuoNfQgxvsTKSM4aglYR-nuvyrZ8nK__r3YQTo1vqH-Hmax3aXs/pub?slide=id.g6239d05d0e_0_1
>> )
>> > about whether it would be possible to statically flag potential errors
>> > like this.
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> Groups "golang-nuts" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> an email to golang-nuts+unsubscr...@googlegroups.com.
>> > To view this discussion on the web visit
>> https://groups.google.com/d/msgid/golang-nuts/CACoUkn7KA0Z-TDypdvM%3Du%3DyVPHuHFmtD%3DiTV2c98Vm%3Dqn4NcPw%40mail.gmail.com
>> .
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/golang-nuts/9EC93CAE-B11C-45E3-9D01-AC4AE5769BEE%40ix.netcom.com
>> .
>>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "golang-nuts" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/golang-nuts/Q7KVGTFt3nU/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/1388A9A2-F321-459A-893E-6D90D42E7255%40ix.netcom.com
> 
> .
>

-- 

[go-nuts] Unexpected evaluation order in a return statement with multiple operands

[Tom Payne]

The Go language specification on order of evaluation 
 states:

  "when evaluating the operands of an expression, assignment, or return 
statement, all function calls, method calls, and communication operations 
are evaluated in lexical left-to-right order."

Consider the following program (also in the Go Playground 
):

  package main

  import (
  "fmt"
  )

  func modify(i *int) error {
  *i = 1
  return nil
  }

  func f() (int, error) {
  i := 0
  return i, modify()
  }

  func main() {
  i, _ := f()
  fmt.Printf("i == %d\n", i)
  }

I would expect return value from f() to be 0, nil, as, evaluated left to 
right the operands have values 0 and nil, but instead the return value is 
1, nil.

What is the explanation for what's actually happening here? Can I rely on 
this behavior or might it change in future versions of Go?

Cheers,
Tom

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/c257ce1c-49af-4a19-8bfd-601df759ffe1%40googlegroups.com.

Re: [go-nuts] Existing hermetic end-to-end testing libraries for CLI applications?

[Tom Payne]

Many thanks for this Paul - this is exactly what I was looking for.

Cheers,
Tom

On Fri, 10 Jan 2020 at 09:55, Paul Jolly  wrote:

> Hi Tom,
>
> > tl;dr Are there any existing end-to-end testing libraries for CLI
> applications? Specifically, what I'm looking for is a library that makes it
> easy to test that "running this command should produce this output" without
> fear that a buggy application could corrupt the filesystem.
>
> I'd suggest looking at:
>
> * https://pkg.go.dev/github.com/rogpeppe/go-internal/testscript?tab=doc
> for the CLI testing aspect
> * https://pkg.go.dev/mvdan.cc/dockexec?tab=overview for the isolation
> piece
>
> There are lots of examples of where testscript is being used:
> https://github.com/rogpeppe/go-internal/wiki/Users-of-testscript
>
> govim is one such an example where we also provide an example of how
> to run those tests via dockexec:
>
> https://github.com/govim/govim/wiki/govim-tests#running-tests---using-cmdgo
>
> In our case it's not so much for isolation, rather providing dependencies.
>
> The #tools channel on slack (or mailing list
> https://groups.google.com/forum/#!forum/golang-tools) is also a good
> place to chat about this more in case you had any follow-ups.
>
> Best,
>
>
> Paul
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAHY_QbRQBKe8GGo-FDbxnha40aNX6tt7Ly4aRooSS2BM-kdUzQ%40mail.gmail.com.

[go-nuts] Existing hermetic end-to-end testing libraries for CLI applications?

[Tom Payne]

Hi,

tl;dr Are there any existing end-to-end testing libraries for CLI 
applications? Specifically, what I'm looking for is a library that makes it 
easy to test that "running this command should produce this output" without 
fear that a buggy application could corrupt the filesystem.

Background:

I have a few CLI applications (written in Go) that are becoming quite 
complex. I'd like to add some end-to-end and integration tests to ensure 
that I don't accidentally break existing functionality while refactoring or 
adding new features. The CLI applications can make arbitrary changes to the 
filesystem and I don't want it to be possible for a buggy application to 
leave any trace on the filesystem after the test terminates. I want 
something like:

- Easy setup of a chroot environment or Docker container or similar with my 
Go binary copied somewhere into $PATH.
- Easy way to specify what commands should be run.
- Easy way to write tests that check exit codes and ensure that the 
stdout/stderr output matches a string or regular expression.
- Automatic tidy-up of the chroot/container after the test is complete.
- Scripted in Go and integrated with Go's testing package, of course :)

I don't need controls over CPU usage or networking as my applications are 
not CPU bound and do not need network access, but I wouldn't complain if 
they existed.

A hypothetical test would look something like: 
https://play.golang.org/p/7L6PZIjvyy6.

I know about github.com/ory/dockertest which is fantastic for writing 
integration tests with third-party services but not set up for testing your 
own applications.

Do you know of any existing libraries that implement this sort of 
functionality?

Many thanks,
Tom

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/54136892-24a7-4f14-9340-60e4dd8b7287%40googlegroups.com.

Re: [go-nuts] regexp syntax and named Unicode character classes

[Tom Payne]

Thank you :) Is this worth adding to the regexp/syntax documentation? I'd 
happily contribute a patch.

On Tuesday, January 7, 2020 at 7:36:02 PM UTC+1, Ian Lance Taylor wrote:
>
> On Tue, Jan 7, 2020 at 10:22 AM Tom Payne > 
> wrote: 
> > 
> > tl;dr How should I use named Unicode character classes in regexps? 
> > 
> > I'm trying to write a regular expression that matches Go identifiers, 
> which start with a Unicode letter or underscore followed by zero or more 
> Unicode letters, decimal digits, and/or underscores. 
> > 
> > Based on the regexp syntax, and the variables in the unicode package 
> which mention the classes "Letter" and "Number, decimal digit", I was 
> expecting to write something like: 
> > 
> >   identiferRegexp := 
> regexp.MustCompile(`\A[[\p{Letter}]_][[\p{Letter}][\p{Number, decimal 
> digit}]_]*\z`) 
> > 
> > However, this pattern does not compile, giving the error: 
> > 
> >   regexp: Compile(`\A[[\p{Letter}]_][[\p{Letter}][\p{Number, decimal 
> digit}]_]*\z`): error parsing regexp: invalid character class range: 
> `\p{Letter}` 
> > 
> > Using the short name for character classes (L for Letter, Nd for Number, 
> decimal digit) does work however: 
> > 
> >   identiferRegexp := regexp.MustCompile(`\A[\pL_][\pL\p{Nd}_]*\z`) 
> > 
> > You can play with these regexps on play.golang.org. 
> > 
> > Is this simply an oversight that Unicode character classes like "Letter" 
> and "Number, decimal digit" are not available for use in regexps, or should 
> I be using them differently? 
>
> The strings you can use with \p are the ones listed in 
> unicode.Categories and unicode.Scripts.  So use \pL as you do in the 
> second example. 
>
> Ian 
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/a22421cc-becb-496e-8d32-b41506536a54%40googlegroups.com.

[go-nuts] regexp syntax and named Unicode character classes

[Tom Payne]

Hi,

tl;dr How should I use named Unicode character classes in regexps?

I'm trying to write a regular expression that matches Go identifiers 
, which start with a Unicode 
letter or underscore followed by zero or more Unicode letters, decimal 
digits, and/or underscores.

Based on the regexp syntax , and the variables 
in the unicode package  which 
mention the classes "Letter" and "Number, decimal digit", I was expecting 
to write something like:

  identiferRegexp := 
regexp.MustCompile(`\A[[\p{Letter}]_][[\p{Letter}][\p{Number, decimal 
digit}]_]*\z`)

However, this pattern does not compile, giving the error:

  regexp: Compile(`\A[[\p{Letter}]_][[\p{Letter}][\p{Number, decimal 
digit}]_]*\z`): error parsing regexp: invalid character class range: 
`\p{Letter}`

Using the short name for character classes (L for Letter, Nd for Number, 
decimal digit) does work however:

  identiferRegexp := regexp.MustCompile(`\A[\pL_][\pL\p{Nd}_]*\z`)

You can play with these regexps on play.golang.org 
.

Is this simply an oversight that Unicode character classes like "Letter" 
and "Number, decimal digit" are not available for use in regexps, or should 
I be using them differently?

Many thanks,
Tom

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/844b99d5-0ae1-4e29-b576-cd4f1b1c24b3%40googlegroups.com.

[go-nuts] [ANN] go-jsonstruct: generate Go structs from multiple JSON objects

[Tom Payne]

Do you have a bunch of JSON objects with no schema that you want to 
idiomatically unmarshall in Go? Existing JSON-to-Go struct generators take 
one object as input. This one takes all the JSON objects you can throw at 
it, be they API responses, unstructured JSON column values in a database, 
or a document dump from CouchDB or MongoDB. You get back the most-specific 
Go type that you can unmarshall all the observed values into with 
encoding/json.

  https://github.com/twpayne/go-jsonstruct

Feedback and improvements welcome :)

Tom


-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/1b171f33-37d8-472a-b775-d2068f7c7f7d%40googlegroups.com.

Re: [go-nuts] Automatically releasing memory resources allocated in cgo?

[Tom Payne]

Thank you very much for the fast, clear, and detailed answer :)

On Fri, 1 Nov 2019 at 15:08, Ian Lance Taylor  wrote:

> On Fri, Nov 1, 2019 at 6:31 AM Tom Payne  wrote:
> >
> > cgo is often used to provide bindings to C libraries. Any memory
> allocated in the C library is not visible to Go, so Go does not have an
> accurate view of the program's memory usage and does not run the garbage
> collector or any finalizers often enough. Consequently, memory usage for a
> Go server that uses cgo heavily can grow very large, with Go itself being
> utterly unaware of it.
> >
> > If the C functions allocate memory, historically you could set a
> finalizer to free the memory sometime after there are no remaining
> references to it in Go, as, for example, described in this blog post.
> However, the current Go docs on runtime.SetFinalizer state:
> >
> > > There is no guarantee that finalizers will run before a program exits,
> so typically they are useful only for releasing non-memory resources
> associated with an object during a long-running program.
> >
> > Are there any other ways to automatically release memory resources
> associated with an object? Or telling Go's memory manager that the small
> object it sees in the Go world is but the tip of an iceberg of memory
> allocated in the C world and therefore should be finalized?
> >
> > Not-good options include:
> > - Requiring the programmer to make explicit calls to free the memory
> resources when they believe the object is no longer needed, but this takes
> us back to the painful world of C's manual memory management and is easy to
> get wrong.
> > - Padding Go objects associated with C memory resources with large,
> unused fields (e.g. an [1024]byte) in the hope that the Go garbage
> collector will be more likely to finalize and free them when they are
> unused.
> > - Avoiding cgo in any server code.
> >
> > Are there any good options?
>
>
> If you are using C code, you are using C's memory resource policy.  So
> the best approach is to use explicit calls to free.  I agree that that
> takes you back to the painful world of C's manual memory management,
> but, after all, if you are using C then you can't pretend that you are
> not using C.
>
> That said, although Go is careful not to promise that it will actually
> execute finalizers, in practice it does run them.  Padding the Go
> object won't make any difference as to when a finalizer is run.  It
> will be run in the next full GC cycle after the Go value is no longer
> referenced.  If there are times when the program knows that there is a
> lot of C memory that is no longer required, it can help by calling
> runtime.GC itself.
>
> In practice I think the best approach is a hybrid: free the C memory
> explicitly, but also add a finalizer.  In the finalizer, free the C
> memory, and log the fact that you are doing so, along with any
> information that will help identify where the C memory was allocated.
> Then periodically check your logs for cases where the finalizer ran,
> and add the necessary calls to explicitly free the C memory manually.
> Of course, when you free the C memory manually, don't forget to clear
> the finalizer.
>
> Ian
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAHY_QbR2NB1FFobtTW6dkXN9h-Zj5uw3bfodVnFzghL_1%3DdUJg%40mail.gmail.com.

[go-nuts] Automatically releasing memory resources allocated in cgo?

[Tom Payne]

cgo is often used to provide bindings to C libraries. Any memory allocated 
in the C library is not visible to Go, so Go does not have an accurate view 
of the program's memory usage and does not run the garbage collector or any 
finalizers often enough. Consequently, memory usage for a Go server that 
uses cgo heavily can grow very large, with Go itself being utterly unaware 
of it.

If the C functions allocate memory, historically you could set a finalizer 
to free the memory sometime after there are no remaining references to it 
in Go, as, for example, described in this blog post 
.
 
However, the current Go docs on runtime.SetFinalizer 
 state:

> There is no guarantee that finalizers will run before a program exits, so 
typically they are useful only for releasing non-memory resources 
associated with an object during a long-running program.

Are there any other ways to automatically release memory resources 
associated with an object? Or telling Go's memory manager that the small 
object it sees in the Go world is but the tip of an iceberg of memory 
allocated in the C world and therefore should be finalized?

Not-good options include:
- Requiring the programmer to make explicit calls to free the memory 
resources when they believe the object is no longer needed, but this takes 
us back to the painful world of C's manual memory management and is easy to 
get wrong.
- Padding Go objects associated with C memory resources with large, unused 
fields (e.g. an [1024]byte) in the hope that the Go garbage collector will 
be more likely to finalize and free them when they are unused.
- Avoiding cgo in any server code.

Are there any good options?

Cheers,
Tom

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/e15adc2e-5d41-4413-b8ed-84c63dbbb8c7%40googlegroups.com.

[go-nuts] A better io.Reader.Read signature?

[Tom Payne]

Dave Cheney has written (yet another) excellent blog post on Go, on the 
subject of API design and caller-controlled allocations:
  
https://dave.cheney.net/2019/09/05/dont-force-allocations-on-the-callers-of-your-api

In this he compares two possible signatures for the io.Reader.Read method:
  func (r *Reader) Read(buf []byte) (int, error)
  func (r *Reader) Read() ([]byte, error)
and makes a very good case for why the first, albeit trickier to use, is 
the better API. The post also sets up a false dichotomy/straw man argument, 
comparing only two signatures when there are other alternatives too.

What about the following API:
  func (r *Reader) Read(buf []byte) (result []byte, err error)
If the buf argument is non-nil then buf is used to store the result, 
len(result) == the int returned by the actual Read(buf []byte) (int, error) 
method.
If the buf argument is nil then a new []byte is created and returned, i.e. 
the allocation is done by Read and the method is as easy to use as Read() 
([]byte, error).
The semantics of the error return remain the same (i.e. Read might return 
both some bytes read and an error), but that is orthogonal to the memory 
allocation.
An example of this pattern is here 
.

Given that slices are small and cheap to copy (a pointer to the data, a 
length, and a capacity) what are the downsides to using
  func (r *Reader) Read(buf []byte) (result []byte, err error)
as a method?

I am not suggesting that we change io.Reader (that would break 
everything!), just looking for input into good Go API design.

This would be a comment on Dave's fantastic blog, but the blog does not 
have comment functionality, so I'm asking here, and I know that Dave lurks 
here too.

Cheers,
Tom

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/8ffb9828-c6e2-47de-bca1-ef6988f081e8%40googlegroups.com.

Re: [go-nuts] Re: Disable test result caching from within Go code?

[Tom Payne]

Here's an example of a test that is cached but shouldn't be:

https://github.com/twpayne/go-geom/blob/master/examples/postgis/main_test.go#L18

This particular test uses github.com/ory/dockertest to run an integration
test against a PostgreSQL database. It uses Docker which is an external
service, and the result should not be cached.

Once the test has run one, running go test again prints:
  $ go test github.com/twpayne/go-geom/examples/postgis
  ok  github.com/twpayne/go-geom/examples/postgis (cached)
which is giving me the idea that the results are cached. go test also
terminates very quickly, whereas the test normally takes ~8s to run on my
machine.


On Fri, 12 Apr 2019 at 03:40, Robert Engels  wrote:

> Where are you getting the idea that results are cached? I can't even
> figure out how that would be useful in a testing framework - do you mean it
> auto-mocks services based on past requests? I've seen nothing like this -
> are you sure you just don't have the wrong HTTP header fields, so the HTTP
> request is getting cached by a CDN, etc.
>
> -Original Message-
> From: twpa...@gmail.com
> Sent: Apr 11, 2019 4:44 PM
> To: golang-nuts
> Subject: [go-nuts] Re: Disable test result caching from within Go code?
>
> OK, I guess that Go has no solution for this.
>
> On Friday, March 29, 2019 at 2:20:53 AM UTC+1, twp...@gmail.com wrote:
>>
>> I have a bunch of integration tests which depend on external services.
>> I've added the build tag "integration" for each source file containing such
>> tests so they are only run when I do:
>>   $ go test -tags=integration ./...
>>
>> Go 1.12, by default, caches the tests results, which is exactly what I
>> don't want because these "external services" are usually different backends
>> running on my local machine. When I'm developing code, I want to test my
>> code against the current backend running on my local machine, and not have
>> "go test" return the result from running the tests against the previous
>> backend.
>>
>> I can disable caching by passing the "-count=1" argument to "go test" but
>> this is a pretty blunt instrument: I need to remember when I'm running an
>> integration test and when I'm not, and it applies to the whole set of tests
>> that I'm running.
>>
>> Ideally I'd like to specify within each test whether the result can be
>> cached or not. I looked at the documentation for testing
>> , but the word "cache" is not present,
>>
>> Is there any way for Go test code to control whether its results should
>> be cached?
>>
>> Cheers,
>> Tom
>>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Why does "go mod" ignore tags in this case?

[Tom Payne]

Thanks Roger.

To close this topic, you were right: I'd missed that there is a special
release process for v2 of a module and above, and also go's mod cache was
out-of-sync as I'd been moving tags around. "go mod verify" is your friend.

On Thu, 21 Feb 2019 at 09:05, roger peppe  wrote:

> I suspect it's because it's version 2. Major version v2 and above needs to
> have the version in the import path.
> So the correct import path for your package would be:
>
>github.com/twpayne/go-xdg/v2
>
> It might be nice if the go tool complained about this at some stage of the
> process; I suspect that's the role of the (currently non-existent) "go
> release" tool.
>
>   cheers,
> rog.
>
> On Wed, 20 Feb 2019 at 23:54,  wrote:
>
>> tl;dr "go mod" writes a timestamp/commit hash to go.mod when a perfectly
>> good tag exists. I don't understand why go.mod doesn't use the tag.
>>
>>
>> Given this library
>>
>>   https://github.com/twpayne/go-xdg
>>
>> with a "v2.0.0" tag:
>>
>>   https://github.com/twpayne/go-xdg/releases/tag/v2.0.0
>>
>> if I create a trivial program that uses it, like
>>
>>   https://gist.github.com/twpayne/09e9135d3f62471ebd7c83359b081e80
>>
>> when I run the go command (e.g. go build), I end up with a go.mod file
>> that contains
>>
>>   require github.com/twpayne/go-xdg v0.0.0-20190214203150-05c8dc503590
>>
>> whereas I would expect go.mod to contain
>>
>>   require github.com/twpayne/go-xdg v2.0.0
>>
>> Even I run the explicit command:
>>
>>   $ go get github.com/twpayne/go-xdg@v2.0.0
>>
>> go.mod still ends up containing the v0.0.0/timestamp/commit hash instead
>> of v2.0.0.
>>
>>
>> Why does go.mod contain v0.0.0/timestamp/commitHash instead of "v2.0.0"?
>>
>>
>> Possibly relevant information:
>> - I initially made a mistake tagging "v2.0.0", so the tag has moved since
>> the go command first saw it. However, I also tried creating a new "v3.0.0"
>> tag, but also saw the v0.0.0/timestamp/commit hash instead of the expected
>> "v3.0.0".
>> - I have GO111MODULE=on.
>>
>> Thanks for any insight :)
>>
>> Tom
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: Existing code for order-preserving concurrent work queue?

[Tom Payne]

Yes, I did, thank you! My reply was to the previous message (robert engels'
post about it being "straightforward" but not providing code) and I think
we just both hit send at about the same time.

On Sat, 26 Jan 2019 at 02:52, Michael Jones  wrote:

> Did you notice that I sent you the complete code above?
>
> On Fri, Jan 25, 2019 at 2:48 PM  wrote:
>
>> For what it's worth
>>   http://www.golangpatterns.info/concurrency/parallel-for-loop
>> implements an order-preserving parallel map, but does not limit the
>> number of workers.
>>
>> In my case, I want to limit the number of workers because I'm making a
>> lot of system calls and don't want to overload the kernel. runtime.NumCPU()
>> seems like a reasonable limit.
>>
>>
>>
>> On Friday, January 25, 2019 at 8:04:31 PM UTC+1, twp...@gmail.com wrote:
>>>
>>> Hi,
>>>
>>> I have a number of slow tasks that I want to run concurrently across
>>> runtime.NumCPU() workers in a single process. The tasks have a specific
>>> input order, but they are completely independent of each other and can
>>> execute in any order. I would like to print the output of each task in the
>>> same order as the input order of tasks.
>>>
>>> This can be implemented by including each task's index in the input
>>> order as it is distributed via a channel to the workers, and the final
>>> collection of results assembled using these task indexes before the results
>>> are printed.
>>>
>>> Assumptions:
>>> - Small number of tasks (~10,000 max), i.e. this easily fits in memory.
>>> - Single Go process, i.e. I don't want/need a distributed system.
>>>
>>> This feels like it should be common problem and there's probably either
>>> a library or a standard Go pattern out there which can do it. My web search
>>> skills didn't find such a library though. Do you know of one?
>>>
>>> Cheers,
>>> Tom
>>>
>>>
>>> Background info to avoid the XY problem : this
>>> is to make chezmoi  run faster. I
>>> want to run the doctor checks
>>> 
>>> (basically os.Exec'ing a whole load of binaries to get their versions)
>>> concurrently in the short term. In the long term I want to make chezmoi's
>>> apply concurrent, so it runs faster too. In the first case, the order
>>> requirement is because I want all users to see the output in the same order
>>> so that it's easy to compare. In the second case, the order requirement
>>> comes because I need to ensure that parent directories are in the correct
>>> state before checking their children.
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
> --
>
> *Michael T. jonesmichael.jo...@gmail.com *
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Can package init() functions in different packages ever run concurrently?

[Tom Payne]

On Thursday, November 15, 2018 at 1:51:08 AM UTC+1, Ian Lance Taylor wrote:
>
> On Wed, Nov 14, 2018 at 4:43 PM, Tom Payne > 
> wrote: 
> > 
> > I couldn't find the answer to this from Googling. 
> > 
> > According to the documentation that I've found: 
> > - For any given package, its imports' init() functions are run before 
> the 
> > package's init() function. 
> > - Within a package, init() functions are more-or-less run in lexical 
> > filename order. 
> > 
> > My question: 
> > Can init() functions in different packages ever run concurrently, e.g. 
> in 
> > separate go routines, or could they be in the future? 
> > 
> > This seems to be within the spec, and could lead to faster startup on 
> > multicore machines, for example when packages A and B independently 
> depend 
> > on C: both A and B's init() functions could run in separate goroutines 
> after 
> > C's init() function, without violating the above. However, this could 
> cause 
> > problems if (for example) A and B's init() functions update a map in C 
> > without synchronization. 
>
> See https://golang.org/ref/spec#Package_initialization : "Package 
> initialization—variable initialization and the invocation of init 
> functions—happens in a single goroutine, sequentially, one package at 
> a time." 
>
> Ian 
>

Perfect, many thanks for the insanely quick answer!

 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Can package init() functions in different packages ever run concurrently?

[Tom Payne]

I couldn't find the answer to this from Googling.

According to the documentation that I've found:
- For any given package, its imports' init() functions are run before the 
package's init() function.
- Within a package, init() functions are more-or-less run in lexical 
filename order.

My question:
Can init() functions in different packages ever run concurrently, e.g. in 
separate go routines, or could they be in the future?

This seems to be within the spec, and could lead to faster startup on 
multicore machines, for example when packages A and B independently depend 
on C: both A and B's init() functions could run in separate goroutines 
after C's init() function, without violating the above. However, this could 
cause problems if (for example) A and B's init() functions update a map in 
C without synchronization.

Thanks for any answer,
Tom

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] How to get the user's umask without race condition?

[Tom Payne]

Thanks Axel for the reply - that's a clever solution.

Just to close the loop on this, the solution that I finally opted for was 
to use the umask system call twice during my program's init() and save the 
value. Given the nature of the program, the race condition should not be a 
problem - no files are created in any package's init() that I'm aware of.

On Tuesday, November 6, 2018 at 4:44:07 PM UTC+1, Axel Wagner wrote:
>
> (and, of course, O_CREATE and everything. sorry for sending multiple 
> messages ^^)
>
> On Tue, Nov 6, 2018 at 4:43 PM Axel Wagner  > wrote:
>
>> Oh, come to think of it, there's a way: Use os.OpenFile with mode=0777 
>> and then stat the file? Should work, no?
>>
>> On Tue, Nov 6, 2018 at 4:41 PM Axel Wagner > > wrote:
>>
>>> That seems to be fundamentally impossible on posix using only syscalls, 
>>> but you may be able to use /proc: 
>>> https://manpages.debian.org/stretch/manpages-dev/umask.2.en.html#NOTES
>>>
>>>
>>> On Tue, Nov 6, 2018 at 4:35 PM Tom Payne > 
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> The user's umask can be set with the umask(2) system call, this sets 
>>>> the new umask and returns the old one. As far as I can tell, the canonical 
>>>> way to get the user's current umask is to call umask twice: once to get 
>>>> the 
>>>> old value and set a temporary one, then a second call to restore the old 
>>>> value:
>>>>
>>>> func getUmask() int {
>>>> umask := syscall.Umask(0)
>>>> syscall.Umask(umask)
>>>> return umask
>>>> }
>>>>
>>>> This has an obvious race condition: if some file operations (e.g. mkdir 
>>>> or open) occur between the two calls to syscall.Umask then they will use 
>>>> the unwanted temporary umask value.
>>>>
>>>> How can I avoid this race condition?
>>>>
>>>> One heavyweight possibility is to read the umask during the main 
>>>> package's init, including a call to runtime.LockOSThread(). However, I'd 
>>>> like to be able to call my getUmask() function at any time.
>>>>
>>>> Many thanks for any pointers,
>>>> Tom
>>>>
>>>> -- 
>>>> You received this message because you are subscribed to the Google 
>>>> Groups "golang-nuts" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send 
>>>> an email to golang-nuts...@googlegroups.com .
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] How to get the user's umask without race condition?

[Tom Payne]

Hi,

The user's umask can be set with the umask(2) system call, this sets the 
new umask and returns the old one. As far as I can tell, the canonical way 
to get the user's current umask is to call umask twice: once to get the old 
value and set a temporary one, then a second call to restore the old value:

func getUmask() int {
umask := syscall.Umask(0)
syscall.Umask(umask)
return umask
}

This has an obvious race condition: if some file operations (e.g. mkdir or 
open) occur between the two calls to syscall.Umask then they will use the 
unwanted temporary umask value.

How can I avoid this race condition?

One heavyweight possibility is to read the umask during the main package's 
init, including a call to runtime.LockOSThread(). However, I'd like to be 
able to call my getUmask() function at any time.

Many thanks for any pointers,
Tom

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Potential race condition in prototypical network server code

[Tom Payne]

Ah, cool. So graceful shutdown can be achieved by calling Close() on the 
net.Listener. Nice!

Thank you :)

Tom


On Thursday, August 24, 2017 at 8:20:34 PM UTC+2, Jan Mercl wrote:
>
> net.Listener has a Close method that will make Accept return an error. 
> net.Listener.Accept is not equal to accept(2).
>
> On Thu, Aug 24, 2017, 20:13 Tom Payne <twp...@gmail.com > 
> wrote:
>
>> Thanks. Looking at the man page for Linux's accept(2) 
>> http://man7.org/linux/man-pages/man2/accept.2.html it seems that 
>> l.Accept is unlikely to ever return an error.
>>
>> accept(2) returns an error if either the connection is incorrectly set up 
>> somehow, or if a system call is interrupted by a signal, or if the process 
>> or system runs out of resources (either file handles or memory).
>>
>> Setup problems should occur immediately (before any client connection is 
>> actually accepted) so there is no race there. It looks to me that EINTR is 
>> *not* handled by Go's runtime so real world (not prototypical) server code 
>> needs to handle EINTR, but there is no race condition in the code (in the 
>> case of a signal, the function will terminate without leaking resources). 
>> In the case of resource exhaustion it's hard to implement sane behaviour in 
>> any case.
>>
>> Thanks again for the discussion - I'm learning a lot here. Is there 
>> anything above that is not correct?
>>
>> Cheers,
>> Tom
>>
>>
>> On Thursday, August 24, 2017 at 7:56:22 PM UTC+2, Jan Mercl wrote:
>>
>>> The wg.Wait will be executed after l.Accept returns an error. It's 
>>> purpose is to wait for the completions of all handlers invoked in the go 
>>> statement that did not finished already.
>>>
>>> On Thu, Aug 24, 2017, 19:49 Tom Payne <twp...@gmail.com> wrote:
>>>
>>>> Awesome, thanks Jan for the fast and clear response.
>>>>
>>>> In fact, the for {} is an infinite loop so wg.Wait() will never be 
>>>> reached and serve() will never terminate. Correct?
>>>>
>>>> I guess I over-read how much this prototypical code was representative 
>>>> of a real server loop. Sorry Dave!
>>>>
>>>> Tom
>>>>
>>>> On Thursday, August 24, 2017 at 7:41:59 PM UTC+2, Jan Mercl wrote:
>>>>
>>>>> No, wg.Add cannot "switch" to wg.Wait, they're both in the samr 
>>>>> goroutine, the go statement will be always the next one to execute after 
>>>>> wg.Add within serve().
>>>>>
>>>>> On Thu, Aug 24, 2017, 19:29 Tom Payne <twp...@gmail.com> wrote:
>>>>>
>>>> I'm not singling out Dave Cheney here, I'd just like to check my 
>>>>>> understanding of Go's resource handling and concurrency.
>>>>>>
>>>>>> In this blog post a "prototypical network server" is presented:
>>>>>>
>>>>>>https://dave.cheney.net/2017/08/20/context-isnt-for-cancellation
>>>>>>
>>>>>> Code:
>>>>>>
>>>>>> func serve(l net.Listener) error {
>>>>>> var wg sync.WaitGroup
>>>>>> var conn net.Conn
>>>>>> var err error
>>>>>> for {
>>>>>> conn, err = l.Accept()
>>>>>> if err != nil {
>>>>>> break
>>>>>> }
>>>>>> wg.Add(1)
>>>>>> go func(c net.Conn) {
>>>>>> defer wg.Done()
>>>>>> handle(c)
>>>>>> }(conn)
>>>>>> }
>>>>>> wg.Wait()
>>>>>> return err
>>>>>> }
>>>>>>
>>>>>> My understanding is that this contains a race condition. 
>>>>>> Specifically, a goroutine switch can occur at the entry to wg.Add(1), 
>>>>>> switching to wg.Wait() in the main goroutine. At this point, a 
>>>>>> connection 
>>>>>> has been accepted, but the WaitGroup counter has not been incremented, 
>>>>>> so 
>>>>>> the serve function will terminate while silently dropping a network 
>>>>>> connection (the connection will be accepted, but never handled, so will 
>>>>>> probabl

Re: [go-nuts] Potential race condition in prototypical network server code

[Tom Payne]

Thanks. Looking at the man page for Linux's 
accept(2) http://man7.org/linux/man-pages/man2/accept.2.html it seems that 
l.Accept is unlikely to ever return an error.

accept(2) returns an error if either the connection is incorrectly set up 
somehow, or if a system call is interrupted by a signal, or if the process 
or system runs out of resources (either file handles or memory).

Setup problems should occur immediately (before any client connection is 
actually accepted) so there is no race there. It looks to me that EINTR is 
*not* handled by Go's runtime so real world (not prototypical) server code 
needs to handle EINTR, but there is no race condition in the code (in the 
case of a signal, the function will terminate without leaking resources). 
In the case of resource exhaustion it's hard to implement sane behaviour in 
any case.

Thanks again for the discussion - I'm learning a lot here. Is there 
anything above that is not correct?

Cheers,
Tom

On Thursday, August 24, 2017 at 7:56:22 PM UTC+2, Jan Mercl wrote:
>
> The wg.Wait will be executed after l.Accept returns an error. It's purpose 
> is to wait for the completions of all handlers invoked in the go statement 
> that did not finished already.
>
> On Thu, Aug 24, 2017, 19:49 Tom Payne <twp...@gmail.com > 
> wrote:
>
>> Awesome, thanks Jan for the fast and clear response.
>>
>> In fact, the for {} is an infinite loop so wg.Wait() will never be 
>> reached and serve() will never terminate. Correct?
>>
>> I guess I over-read how much this prototypical code was representative of 
>> a real server loop. Sorry Dave!
>>
>> Tom
>>
>> On Thursday, August 24, 2017 at 7:41:59 PM UTC+2, Jan Mercl wrote:
>>
>>> No, wg.Add cannot "switch" to wg.Wait, they're both in the samr 
>>> goroutine, the go statement will be always the next one to execute after 
>>> wg.Add within serve().
>>>
>>> On Thu, Aug 24, 2017, 19:29 Tom Payne <twp...@gmail.com> wrote:
>>>
>> I'm not singling out Dave Cheney here, I'd just like to check my 
>>>> understanding of Go's resource handling and concurrency.
>>>>
>>>> In this blog post a "prototypical network server" is presented:
>>>>
>>>>https://dave.cheney.net/2017/08/20/context-isnt-for-cancellation
>>>>
>>>> Code:
>>>>
>>>> func serve(l net.Listener) error {
>>>> var wg sync.WaitGroup
>>>> var conn net.Conn
>>>> var err error
>>>> for {
>>>> conn, err = l.Accept()
>>>> if err != nil {
>>>> break
>>>> }
>>>> wg.Add(1)
>>>> go func(c net.Conn) {
>>>> defer wg.Done()
>>>> handle(c)
>>>> }(conn)
>>>> }
>>>> wg.Wait()
>>>> return err
>>>> }
>>>>
>>>> My understanding is that this contains a race condition. Specifically, 
>>>> a goroutine switch can occur at the entry to wg.Add(1), switching to 
>>>> wg.Wait() in the main goroutine. At this point, a connection has been 
>>>> accepted, but the WaitGroup counter has not been incremented, so the serve 
>>>> function will terminate while silently dropping a network connection (the 
>>>> connection will be accepted, but never handled, so will probably 
>>>> eventually 
>>>> time out on the client side and leak a file descriptor on the server side).
>>>>
>>>> Is this understanding correct? Furthermore, I suspect that it's 
>>>> impossible to implement race-free graceful termination using only 
>>>> sync.WaitGroups. Is it actually possible to do so?
>>>>
>>>> Thanks for any correction, and apologies to Dave for picking on his 
>>>> code.
>>>>
>>>> Cheers,
>>>> Tom
>>>>
>>>> -- 
>>>> You received this message because you are subscribed to the Google 
>>>> Groups "golang-nuts" group.
>>>>
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>>> an email to golang-nuts...@googlegroups.com.
>>>
>>>
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>> -- 
>>>
>>> -j
>>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
> -- 
>
> -j
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Potential race condition in prototypical network server code

[Tom Payne]

Awesome, thanks Jan for the fast and clear response.

In fact, the for {} is an infinite loop so wg.Wait() will never be reached 
and serve() will never terminate. Correct?

I guess I over-read how much this prototypical code was representative of a 
real server loop. Sorry Dave!

Tom

On Thursday, August 24, 2017 at 7:41:59 PM UTC+2, Jan Mercl wrote:
>
> No, wg.Add cannot "switch" to wg.Wait, they're both in the samr goroutine, 
> the go statement will be always the next one to execute after wg.Add within 
> serve().
>
> On Thu, Aug 24, 2017, 19:29 Tom Payne <twp...@gmail.com > 
> wrote:
>
>> I'm not singling out Dave Cheney here, I'd just like to check my 
>> understanding of Go's resource handling and concurrency.
>>
>> In this blog post a "prototypical network server" is presented:
>>
>>https://dave.cheney.net/2017/08/20/context-isnt-for-cancellation
>>
>> Code:
>>
>> func serve(l net.Listener) error {
>> var wg sync.WaitGroup
>> var conn net.Conn
>> var err error
>> for {
>> conn, err = l.Accept()
>> if err != nil {
>> break
>> }
>> wg.Add(1)
>> go func(c net.Conn) {
>> defer wg.Done()
>> handle(c)
>> }(conn)
>> }
>> wg.Wait()
>> return err
>> }
>>
>> My understanding is that this contains a race condition. Specifically, a 
>> goroutine switch can occur at the entry to wg.Add(1), switching to 
>> wg.Wait() in the main goroutine. At this point, a connection has been 
>> accepted, but the WaitGroup counter has not been incremented, so the serve 
>> function will terminate while silently dropping a network connection (the 
>> connection will be accepted, but never handled, so will probably eventually 
>> time out on the client side and leak a file descriptor on the server side).
>>
>> Is this understanding correct? Furthermore, I suspect that it's 
>> impossible to implement race-free graceful termination using only 
>> sync.WaitGroups. Is it actually possible to do so?
>>
>> Thanks for any correction, and apologies to Dave for picking on his code.
>>
>> Cheers,
>> Tom
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
> -- 
>
> -j
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Potential race condition in prototypical network server code

[Tom Payne]

I'm not singling out Dave Cheney here, I'd just like to check my 
understanding of Go's resource handling and concurrency.

In this blog post a "prototypical network server" is presented:

   https://dave.cheney.net/2017/08/20/context-isnt-for-cancellation

Code:

func serve(l net.Listener) error {
var wg sync.WaitGroup
var conn net.Conn
var err error
for {
conn, err = l.Accept()
if err != nil {
break
}
wg.Add(1)
go func(c net.Conn) {
defer wg.Done()
handle(c)
}(conn)
}
wg.Wait()
return err
}

My understanding is that this contains a race condition. Specifically, a 
goroutine switch can occur at the entry to wg.Add(1), switching to 
wg.Wait() in the main goroutine. At this point, a connection has been 
accepted, but the WaitGroup counter has not been incremented, so the serve 
function will terminate while silently dropping a network connection (the 
connection will be accepted, but never handled, so will probably eventually 
time out on the client side and leak a file descriptor on the server side).

Is this understanding correct? Furthermore, I suspect that it's impossible 
to implement race-free graceful termination using only sync.WaitGroups. Is 
it actually possible to do so?

Thanks for any correction, and apologies to Dave for picking on his code.

Cheers,
Tom

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Reasoning behind behavior of range, when index is maintained

[tom . payne]

A side effect of this approach is that the index after the range loop will 
be zero if slice contains zero or one elements:
  https://play.golang.org/p/F7lLZ5wcuv

This means that code using the index after the range will need to re-test 
whether the slice was empty to avoid a potential panic.

On Thursday, July 27, 2017 at 2:21:34 PM UTC+2, Christoph Berger wrote:
>
> That’s actually what I meant to indicate in the last paragraph (emphasis 
> added by me):
>
> > The *code in the Reddit post* takes advantage of the fact that the last 
> increment of the C-style loop can be observed outside the loop,
>
> But thanks for providing a clarification. I see now it has not been clear 
> to everyone.
>
> On Wed, Jul 26, 2017 at 08:44:46AM -0700, Christoph Berger wrote:
>
> someone shared [this question](
>
> https://www.reddit.com/r/golang/comments/6paqc0/bug_that_caught_me_with_range/)
>  
>
> on reddit. I must say, that I'm surprised by the behavior myself. I would 
> have expected
> for i = range v
> to be semantically equivalent to
> for i = 0; i < len(v); i++
> and don't really understand the reasoning behind choosing different 
> semantics. Note, that the difference only exists, if i is declared outside 
> of the loop, that is, this is solely about the behavior after exiting the 
> loop-body.
>
> I'd greatly appreciate some explanation :)
>
> An attempt to explain this by looking at the C-style loop only:
>
> The classic C-style for loop
>
> for i:=0; i
> is equivalent to
>
> for i:=0; i// do something with i
>i++ // This is always the very last statement in the loop body
> }
>
> The loop body runs from 0 to len(v)-1 only, because the last increment of 
> i 
> to len(v) stops the loop, and no further iteration occurs. The code in the 
> loop body never sees i being set to len(v). 
>
> And that's the same behavior as with the range operator. 
>
> The code in the Reddit post takes advantage of the fact that the last 
> increment of the C-style loop can be observed outside the loop, for 
> detecting if the loop stopped early. This is a neat side effect that is 
> not 
> possible with the range operator.
>
>
> I would point out that both Axel and you are off a tiny bit from what
> actually happens ;-)
>
> In a for loop which uses a short variable declaration, that variable's
> scope is confined to the for *statement* itself, and is also visible in
> the loop's body because its scope is defined to be nested in that of the
> loop statement.  This means in a loop like
>
>  for i := 0; i < len(s); i++ {
>  }
>
> the variable "i" is not accessible after the closing brace.
>
> The actual "problem" stated in that Reddit post is different: it uses a
> variable defined outside the "for" loop:
>
>  var i int
>  for i = 0; i < len(v); i++ {
>  }
>
> As you can see, the loop merely uses that variable; it existed before
> the loop and continued to live on after it finished executing.
>
> To recap what others have already written, since the for loop's post
> statement is defined to be executed after each execution of the body
> (unless it was exited by means of executing `break` or `return`), that
>
>  i++
>
> statement gets executed, the condition evaluates to false, and the loop
> exits -- with the variable "i" having the value equal to len(v).
>
> One could do
>
>  var x int
>
>  for i := 0; i < len(v); i, x = i+1, x*2 {
>  }
>
> and get even more interesting effect on the variable "x" after the loop
> finishes executing ;-)
>
> -- 
> You received this message because you are subscribed to a topic in the 
> Google Groups "golang-nuts" group.
> To unsubscribe from this topic, visit 
> https://groups.google.com/d/topic/golang-nuts/Xi6W3H5mlto/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to 
> golang-nuts...@googlegroups.com .
> For more options, visit https://groups.google.com/d/optout.
>
>
>
-- 
This message is for the attention of the intended recipient(s) only. It may 
contain confidential, proprietary and/or legally privileged information. 
Use, disclosure and/or retransmission of information contained in this 
email may be prohibited. If you are not an intended recipient, you are 
kindly asked to notify the sender immediately (by reply e-mail) and to 
permanently delete this message. Thank you.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] "go install": no warning/error when binaries conflict

[tom . payne]

Consider a package with multiple binaries with the same name, "cmd" in this 
case:

pkg1/cmd/main.go:
package main; import "fmt"; func main() { fmt.Println("pkg1") }

pkg2/cmd/main.go:
package main; import "fmt"; func main() { fmt.Println("pkg2") }

When running
  $ go install ./...
I would expect to get an error (or at least a warning) that the binaries 
conflict: there can only be one file called "cmd" in $GOBIN. However, "go 
install ./..." silently ignores the duplicate. One of the binaries "wins" 
and the other is lost.

For reference, the "cp" command in GNU Coreutils prints a warning when this 
occurs:
  $ cp pkg1/cmd/main.go pkg2/cmd/main.go .
  cp: will not overwrite just-created './main.go' with 'pkg2/cmd/main.go'

Would the Go team be interested in a fix for this?

Cheers,
Tom

-- 
This message is for the attention of the intended recipient(s) only. It may 
contain confidential, proprietary and/or legally privileged information. 
Use, disclosure and/or retransmission of information contained in this 
email may be prohibited. If you are not an intended recipient, you are 
kindly asked to notify the sender immediately (by reply e-mail) and to 
permanently delete this message. Thank you.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: [ANN] Go geospatial libraries: geometries, GeoJSON, WKB, KML, PostGIS, GPX, polyline

[Tom Payne]

Here's an example:

8<
package main

import (
"log"
"os"

enckml "github.com/twpayne/go-geom/encoding/kml"
"github.com/twpayne/go-geom/encoding/wkbhex"
"github.com/twpayne/go-kml"
)

func main() {
g, err := wkbhex.Decode("0140004010")
if err != nil {
log.Fatal(err)
}
e, err := enckml.Encode(g)
if err != nil {
log.Fatal(err)
}
if err := kml.KML(kml.Placemark(e)).Write(os.Stdout); err != nil {
log.Fatal(err)
}
// Output:
// 
// http://www.opengis.net/kml/2.2
">2,4
}

8<


On 27 March 2017 at 19:15, Constantine Vassilev <thst...@gmail.com> wrote:

> How to use go-geom to convert WKBHEX to KML?
>
> Are there a simple example?
>
> On Sunday, November 13, 2016 at 10:33:27 AM UTC-8, Tom Payne wrote:
>>
>> A quick announce of a few libraries for geospatial applications that are
>> now mature and battle-tested:
>>
>> https://github.com/twpayne/go-geom : efficient geometry library, using a
>> high-performance cache-friendly data representation (more info
>> <https://github.com/twpayne/go-geom/blob/master/INTERNALS.md>), with
>> import and export from multiple formats
>> <https://github.com/twpayne/go-geom/blob/master/README.md> (e.g.
>> GeoJSON, WKB, KML) and many 2D geometry functions
>> <https://godoc.org/github.com/twpayne/go-geom/xy>. Really easy PostGIS
>> integration with database/sql
>> <https://github.com/twpayne/go-geom/blob/master/encoding/wkb/scan_test.go>
>> .
>>
>> <http://goog_579174976>
>> https://github.com/twpayne/go-kml : generate KML files quickly and
>> flexibly.
>>
>> <http://goog_579174978>
>> https://github.com/twpayne/go-gpx : read and write GPX files.
>>
>> <http://goog_579174980>
>> https://github.com/twpayne/go-polyline : encode and decode data to/from
>> Google Maps Polyline format.
>>
>> I think that go-geom's internal data representation
>> <https://github.com/twpayne/go-geom/blob/master/INTERNALS.md> is a nice
>> example of high performance Go code and a reasonable attempt at practical
>> code while the language does not support generics.
>>
>> All feedback welcome!
>>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "golang-nuts" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/
> topic/golang-nuts/GdssAIB6000/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> golang-nuts+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Gob encoding of maps is not consistent, should this be fixed?

[Tom Payne]

This example demonstrates that the Gob encoding of the same map value 
varies:
  https://play.golang.org/p/DZRV9i0uf_

This is because the encoder iterates over the map using a for ... range 
loop and the iteration order for maps is not defined:
  https://github.com/golang/go/blob/master/src/encoding/gob/encode.go#L379

Is this considered a bug and would the Go maintainers consider a patch to 
fix it? Such a patch would sort the keys first and emit the key-value pairs 
in order to provide a consistent encoding. The consistent encoding would 
likely be optional to avoid the performance and memory penalty of sorting 
the keys.

The reason this is a problem is that we generate a data file in Gob format 
as part of our build process. The encoded structure contains many maps and 
so the generated file varies on every run. This makes our build process 
non-repeatable: when using Gob the same inputs do not result in the same 
outputs.

Note that consistent encoding of the same value is *not* specified in the 
list of goals in https://blog.golang.org/gobs-of-data (in fact it is not 
mentioned at all). However, it could be considered a desirable feature :)

Regards,
Tom

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: [ANN] Go geospatial libraries: geometries, GeoJSON, WKB, KML, PostGIS, GPX, polyline

[Tom Payne]

Done. I've also added a link to github.com/golang/geo, a library for 
geometry on the sphere.

On Monday, November 14, 2016 at 11:25:19 PM UTC+1, Daniel Whitenack wrote:
>
> Thanks Tom! I think it would be great to add a few of these here:
>
> https://github.com/gopherds/resources/tree/master/tooling
>
> where you think it is appropriate.  Would you consider submitting a PR 
> with some additions?
>
> Daniel
>
> On Sunday, November 13, 2016 at 1:33:27 PM UTC-5, Tom Payne wrote:
>>
>> A quick announce of a few libraries for geospatial applications that are 
>> now mature and battle-tested:
>>
>> https://github.com/twpayne/go-geom : efficient geometry library, using a 
>> high-performance cache-friendly data representation (more info 
>> <https://github.com/twpayne/go-geom/blob/master/INTERNALS.md>), with 
>> import and export from multiple formats 
>> <https://github.com/twpayne/go-geom/blob/master/README.md> (e.g. 
>> GeoJSON, WKB, KML) and many 2D geometry functions 
>> <https://godoc.org/github.com/twpayne/go-geom/xy>. Really easy PostGIS 
>> integration with database/sql 
>> <https://github.com/twpayne/go-geom/blob/master/encoding/wkb/scan_test.go>
>> .
>>
>> <http://goog_579174976>
>> https://github.com/twpayne/go-kml : generate KML files quickly and 
>> flexibly.
>>
>> <http://goog_579174978>
>> https://github.com/twpayne/go-gpx : read and write GPX files.
>>
>> <http://goog_579174980>
>> https://github.com/twpayne/go-polyline : encode and decode data to/from 
>> Google Maps Polyline format.
>>
>> I think that go-geom's internal data representation 
>> <https://github.com/twpayne/go-geom/blob/master/INTERNALS.md> is a nice 
>> example of high performance Go code and a reasonable attempt at practical 
>> code while the language does not support generics.
>>
>> All feedback welcome!
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] [ANN] Go geospatial libraries: geometries, GeoJSON, WKB, KML, PostGIS, GPX, polyline

[Tom Payne]

A quick announce of a few libraries for geospatial applications that are 
now mature and battle-tested:

https://github.com/twpayne/go-geom : efficient geometry library, using a 
high-performance cache-friendly data representation (more info 
), with import 
and export from multiple formats 
 (e.g. GeoJSON, 
WKB, KML) and many 2D geometry functions 
. Really easy PostGIS 
integration with database/sql 
.


https://github.com/twpayne/go-kml : generate KML files quickly and flexibly.


https://github.com/twpayne/go-gpx : read and write GPX files.


https://github.com/twpayne/go-polyline : encode and decode data to/from 
Google Maps Polyline format.

I think that go-geom's internal data representation 
 is a nice 
example of high performance Go code and a reasonable attempt at practical 
code while the language does not support generics.

All feedback welcome!

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.