Re: [go-nuts] Re: go/types with typedef-of-typedef?

['Tim Hockin' via golang-nuts]

How do you make the jump from a go/type.TypeName to ast Decl?  Is there
some easy crosslink?

On Fri, Jul 26, 2024, 4:24 PM Jason E. Aten  wrote:

> To be more specific,
>
> ast.TypeSpec.Name.Name is, I think, what you are after.
>
> See https://pkg.go.dev/go/ast#TypeSpec
>
> And the TypeSpec are found in the ast.File.Decls, after type switching in
> the manner of getast.go#360
>
> https://pkg.go.dev/go/ast#File
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "golang-nuts" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/golang-nuts/eQxFFSRKL24/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/01c4a98c-30f4-4cd6-bc7f-a7edf781641an%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAO_RewZPe3HoHiO8WWZSRF6EEzLf4bWhS68M-ydW7tte-nMuhQ%40mail.gmail.com.

[go-nuts] go/types with typedef-of-typedef?

['Tim Hockin' via golang-nuts]

Hi all,  I have been banging my head on this all day.  I am doinking around 
in some code-generation and find myself deep into the go/types package.

Given code like:

```
type E1 string
type E2 E1
```

I loaded the package with x/tools/go/packages.  I got the root scope via 
`pkg.Types.Scope()`.  I did a `Lookup(typename)` and got a `TypeName` 
back.  `TypeName.Type()` gets me a `Named`.

Here's where I am stuck.

In the debugger I can see that E1 and E2 are different.  E1's `fromRHS` 
field tells me it is a `go/types.Basic` with `.name` =  "string".  E2's 
`fromRHS` tells me it is `go/types.Named` which is an E1.

Awesome - that is what I want.  But I can't figure out how to learn that 
through the public API.  AFAICT, `fromRHS` is only used internally and 
calling `Underlying()` on E1 and E2 yields a string in both cases.

Is there any way to learn what I am trying to learn?  The information is 
THERE but I can't seem to get at it.

Tim

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/0e4da407-7e14-42e7-90e2-05f322509232n%40googlegroups.com.

Re: [go-nuts] GOPATH, GOBIN, go install, go build, cross-compiles

['Tim Hockin' via golang-nuts]

Thanks Sean.

On Fri, Jan 20, 2023 at 10:39 AM 'Sean Liao' via golang-nuts
 wrote:
>
> in module mode,
> GOPATH is really only used for deriving default locations for GOBIN and 
> GOMODCACHE
> setting it to /dev/null might not be great for caching dependencies if 
> GOMODCACHE isn't set

I think it would bring some closure if we could definitively say
"GOPATH isn't needed".

GOCACHE seems to default to ~/.cache/go-build on my machine.
Shouldn't GOMODCACHE do the same?

> cross compile to GOBIN was recently (re)opened
> https://go.dev/issue/57485

Great - I voted it up, but I don't know if that helps at all.  Setting
GOBIN would be a better solution for k8s, if it only worked.

Another thing that we end up writing scripts for is the various sets
of build flags.  Like, if you want to build debug binaries there are
something like 4 flags to change.  It would be super interesting to
have in-repo or in-module metadata that could be used for these flags.

For example, what if I could add something that said "GOBIN=./bin" or
even more flexible.

> There are open requests to print out the directory where it would be 
> installed,
> (presumably to update PATH) but not for the actual binaries built.
> https://go.dev/issue/45546

Also voted up, but it seems stalled.

> -o dir/ ending in a slash is documented as:
> > If the named output is an existing directory or
> > ends with a slash or backslash, then any resulting executables
> > will be written to that directory.

Sure, it's documented, but one mistake and all subsequent invocations
fail.  It works, but it's just fragile.

Tim


> On Fri, Jan 20, 2023 at 5:30 PM 'Tim Hockin' via golang-nuts 
>  wrote:
>>
>> I'm trying to make kubernetes' codebase easier to navigate by people who 
>> have muscle-memory for go.  In the past we said "you have to call our 
>> Makefile" to build things.  I'd like to make \`go install\` work but I'm 
>> struggling to find a pattern that really works - am I missing something?
>>
>> First, GOPATH is dead.  I hate that Go silently writes to my home directory, 
>> which is ALMOST NEVER what I really wanted.
>>
>> Aside: is there any way to get `go install` and `go build` to tell me what 
>> they wrote?  Would that be a reasonable feature request?  E.g.
>>
>> ```
>> $ go install --print-result 
>> /home/thockin/go/bin/cmd
>> ```
>>
>> I usually set GOPATH=/dev/null.
>>
>> ```
>> $ GOPATH=/dev/null \
>>   go install 
>> go install : mkdir /dev/null: not a directory
>> ```
>>
>> Good.  I want to install to a repo-local dir so all of our CI and stuff can 
>> find it.
>>
>> ```
>> $ GOPATH=/dev/null \
>>   GOBIN=./bin \
>>   go install 
>> ```
>>
>> That works.
>>
>> ```
>> $ GOPATH=/dev/null \
>>   GOBIN=./bin \
>>   GOARCH=arm \
>>   go install 
>> go: cannot install cross-compiled binaries when GOBIN is set
>> ```
>>
>> Well, shoot.  I can't find any other way for `go install` to work.
>>
>> ```
>> $ GOPATH=/dev/null \
>>   GOBIN=./bin \
>>   GOARCH=arm \
>>   go build 
>> ```
>>
>> That works, but splats the file into the current dir.  To be fair, that is 
>> documented behavior.
>>
>> ```
>> $ GOPATH=/dev/null \
>>   GOARCH=arm \
>>   go build -o ./bin 
>> ```
>>
>> That works, except when ./bin doesn't exist, in which case it becomes the 
>> output file (if I am building 1 thing) or I get an error (if I am building 
>> multiple things):
>>
>> ```
>> $ GOPATH=/dev/null \
>>   GOARCH=arm \
>>   go build -o ./bin  
>> go: cannot write multiple packages to non-directory ./foo
>> ```
>>
>> I can specify a directory by adding a trailing `/` it seems:
>>
>> ```
>> $ GOPATH=/dev/null \
>>   GOARCH=arm \
>>   go build -o ./bin/  
>> ```
>>
>> That seems to work, but I have to get the invocation JUST RIGHT 100% of the 
>> time or risk causing weird errors and CI fails.  The obvious answer is "wrap 
>> it in a script or Makefile", which puts me right back where I started.
>>
>> Is there a better answer?
>>
>> Tim
>>
>> --
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/d04fca5a-8b85-4386-9

[go-nuts] GOPATH, GOBIN, go install, go build, cross-compiles

['Tim Hockin' via golang-nuts]

I'm trying to make kubernetes' codebase easier to navigate by people who 
have muscle-memory for go.  In the past we said "you have to call our 
Makefile" to build things.  I'd like to make \`go install\` work but I'm 
struggling to find a pattern that really works - am I missing something?

First, GOPATH is dead.  I hate that Go silently writes to my home 
directory, which is ALMOST NEVER what I really wanted. 

Aside: is there any way to get `go install` and `go build` to tell me what 
they wrote?  Would that be a reasonable feature request?  E.g. 

```
$ go install --print-result 
/home/thockin/go/bin/cmd
```

I usually set GOPATH=/dev/null.

```
$ GOPATH=/dev/null \
  go install 
go install : mkdir /dev/null: not a directory
```

Good.  I want to install to a repo-local dir so all of our CI and stuff can 
find it.

```
$ GOPATH=/dev/null \
  GOBIN=./bin \
  go install 
```

That works.

```
$ GOPATH=/dev/null \
  GOBIN=./bin \
  GOARCH=arm \
  go install 
go: cannot install cross-compiled binaries when GOBIN is set
```

Well, shoot.  I can't find any other way for `go install` to work.

```
$ GOPATH=/dev/null \
  GOBIN=./bin \
  GOARCH=arm \
  go build 
```

That works, but splats the file into the current dir.  To be fair, that is 
documented behavior.

```
$ GOPATH=/dev/null \
  GOARCH=arm \
  go build -o ./bin 
```

That works, except when ./bin doesn't exist, in which case it becomes the 
output file (if I am building 1 thing) or I get an error (if I am building 
multiple things):

```
$ GOPATH=/dev/null \
  GOARCH=arm \
  go build -o ./bin  
go: cannot write multiple packages to non-directory ./foo
```

I can specify a directory by adding a trailing `/` it seems:

```
$ GOPATH=/dev/null \
  GOARCH=arm \
  go build -o ./bin/  
```

That seems to work, but I have to get the invocation JUST RIGHT 100% of the 
time or risk causing weird errors and CI fails.  The obvious answer is 
"wrap it in a script or Makefile", which puts me right back where I started.

Is there a better answer?

Tim

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/d04fca5a-8b85-4386-9bae-50853f317fccn%40googlegroups.com.

[go-nuts] Using an "internal" repo or org vs. vendor

['Tim Hockin' via golang-nuts]

We have some packages we would like to break out into their own repos,
but which we don't really want anyone outside of our project to depend
on.

We could, of course, just document that.  But I had this thought that
maybe Go's "internal" semantics could be applied here.  So as a test,
we redirected go-get of `k8s.io/internal` to a different github org
(e.g. go get k8s.io/internal/foo -> github.com/k8s-internal/foo).

This SEEMS to work until vendoring comes in, and we get errors like:

```
package k8s.io/kubernetes/cmd/genkubedocs
imports k8s.io/kubernetes/cmd/kubelet/app
imports k8s.io/kubernetes/pkg/kubelet
imports k8s.io/kubernetes/pkg/kubelet/images
pkg/kubelet/images/image_manager.go:26:2: use of internal package
k8s.io/kubernetes/vendor/k8s.io/internal/distribution/reference not
allowed
```

The code in `k8s.io/kubernetes/pkg/kubelet/images` imports
`k8s.io/internal/distribution/reference` but vendoring renames it
internally and now it fails the "internal" rules.

Is this perceived as a bug or just a case of "too clever for our own good" ?

Tim

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAO_RewbsnD4sH5Mqc41du0DYSsSBsnVmGi7nmW0MJpvA06_M5Q%40mail.gmail.com.

Re: [go-nuts] testing if strconv.Quote() would do change a string, without calling it

['Tim Hockin' via golang-nuts]

I tried:

strconv.Quote()
strconv.AppendQuote() (weird that this is faster than Quote)
fmt.Sprintf("%q")
scanning for !IsPrint() + bytes.Buffer
scanning for !IsPrint() + strings.Builder (sad that this is not faster than
Buffer)
scanning for !IsPrint() + string addition

```
$ go test -benchtime=5s -bench='XX' ./benchmark/
goos: linux
goarch: amd64
pkg: github.com/go-logr/logr/benchmark
cpu: Intel(R) Xeon(R) W-2135 CPU @ 3.70GHz
BenchmarkXXQuote-6 23233543   258.5 ns/op
BenchmarkXXAppendQuote-6   24370812   207.7 ns/op
BenchmarkXXSprintf-6   18040070   335.2 ns/op
BenchmarkXXScanBuffer-6 47154340   117.4 ns/op
BenchmarkXXScanBuilder-6   42295635   141.9 ns/op
BenchmarkXXScanAdd-6   43635146   137.5 ns/op
PASS
ok   github.com/go-logr/logr/benchmark 35.926s
```

code:

```
//go:noinline
func foo(s string) {
_ = s
}

func prettyBuffer(s string) string {
if needsEscape(s) {
return strconv.Quote(s)
}
b := bytes.NewBuffer(make([]byte, 0, 1024))
b.WriteByte('"')
b.WriteString(s)
b.WriteByte('"')
return b.String()
}

func prettyBuilder(s string) string {
if needsEscape(s) {
return strconv.Quote(s)
}
b := strings.Builder{}
b.WriteByte('"')
b.WriteString(s)
b.WriteByte('"')
return b.String()
}
func prettyAdd(s string) string {
if needsEscape(s) {
return strconv.Quote(s)
}
return `"` + s + `"`
}

// needsEscape determines whether the input string needs to be escaped or
not,
// without doing any allocations.
func needsEscape(s string) bool {
for _, r := range s {
if !strconv.IsPrint(r) || r == '\\' || r == '"' {
return true
}
}
return false
}

func BenchmarkXXQuote(b *testing.B) {
in := "a string with no specials"
for i := 0; i < b.N; i++ {
out := strconv.Quote(in)
foo(out)
}
}
func BenchmarkXXAppendQuote(b *testing.B) {
in := "a string with no specials"
for i := 0; i < b.N; i++ {
out := strconv.AppendQuote(make([]byte, 0, 1024), in)
foo(string(out))
}
}
func BenchmarkXXSprintf(b *testing.B) {
in := "a string with no specials"
for i := 0; i < b.N; i++ {
out := fmt.Sprintf("%q", in)
foo(out)
}
}
func BenchmarkXXScanBuffer(b *testing.B) {
in := "a string with no specials"
for i := 0; i < b.N; i++ {
out := prettyBuffer(in)
foo(out)
}
}
func BenchmarkXXScanBuilder(b *testing.B) {
in := "a string with no specials"
for i := 0; i < b.N; i++ {
out := prettyBuilder(in)
foo(out)
}
}
func BenchmarkXXScanAdd(b *testing.B) {
in := "a string with no specials"
for i := 0; i < b.N; i++ {
out := prettyAdd(in)
    foo(out)
}
}

```


On Thu, Oct 14, 2021 at 1:52 AM roger peppe  wrote:

> On Thu, 14 Oct 2021 at 04:58, 'Tim Hockin' via golang-nuts <
> golang-nuts@googlegroups.com> wrote:
>
>> Thanks for confirming.  I wrote that function and erased a good bit of
>> the overhead.
>>
>> bytes.Buffer for the no-escapes path and strconv.Quote otherwise.
>>
>
> Could you not use strconv.AppendQuote and get the advantage without
> needing the extra scan?
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAO_RewbrDW55pD74hO6eCTgUEGZh7-46_z7RC3FWonrpnypwHw%40mail.gmail.com.

Re: [go-nuts] testing if strconv.Quote() would do change a string, without calling it

['Tim Hockin' via golang-nuts]

Thanks for confirming.  I wrote that function and erased a good bit of the
overhead.

bytes.Buffer for the no-escapes path and strconv.Quote otherwise.


On Wed, Oct 13, 2021, 8:49 PM Ian Lance Taylor  wrote:

> On Wed, Oct 13, 2021 at 3:46 PM 'Tim Hockin' via golang-nuts
>  wrote:
> >
> > Is there any ready-built function that can tell me whether
> `strconv.Quote()` would produce a different string than its input, without
> actually running it?  Or is there a clearly documented set of rules one
> could use to test each rune in a string?
> >
> > I am trying to avoid allocations, and MOST of the inputs will be safe
> (but not all).  Calling strconv.Quote() has a measurable impact, so I'd
> avoid it if I could...
> >
> > Does such an animal exist?
>
> To answer your exact question, strconv.Quote always allocates a new
> string, because it always adds quotation marks around the returned
> string.  That is, the output of strconv.Quote("a") is `"a"`, with
> literal quotation characters.
>
> Other than calling strconv.Quote will replace any rune for which
> strconv.IsPrint returns false, and it will also replace `\` and `"`.
> But I don't think there is a function to check that.
>
> Ian
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAO_RewYJUMFz36u%2B%2B%3DHp3-Dqi33jhjgadehE6KbC1eaakMvpzg%40mail.gmail.com.

Re: [go-nuts] testing if strconv.Quote() would do change a string, without calling it

['Tim Hockin' via golang-nuts]

If I find a string with a stray backslash in it (which passes IsPrint()) I
still need to quote.

I dug into the Quote() impl and this seems like the right path.  Thanks!

On Wed, Oct 13, 2021 at 4:31 PM Robert Engels  wrote:

> I was thinking the other way. If !IsPrint() then strconv.Quote()
>
> On Oct 13, 2021, at 6:24 PM, Tim Hockin  wrote:
>
> 
> ` IsPrint(r) || r == '\\' || r == '"' ` passes tests.  I need to build
> confidence in that, though :)  Thanks.
>
> On Wed, Oct 13, 2021 at 4:16 PM Robert Engels 
> wrote:
>
>> A simple loop calling IsPrint is your best bet. You could then have a
>> custom implementation of Quote that started at a specified index.
>>
>> On Oct 13, 2021, at 5:46 PM, 'Tim Hockin' via golang-nuts <
>> golang-nuts@googlegroups.com> wrote:
>>
>> Is there any ready-built function that can tell me whether
>> `strconv.Quote()` would produce a different string than its input, without
>> actually running it?  Or is there a clearly documented set of rules one
>> could use to test each rune in a string?
>>
>> I am trying to avoid allocations, and MOST of the inputs will be safe
>> (but not all).  Calling strconv.Quote() has a measurable impact, so I'd
>> avoid it if I could...
>>
>> Does such an animal exist?
>>
>> Tim
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/golang-nuts/1bb3f806-5930-4866-8249-0bbc0ee383b8n%40googlegroups.com
>> <https://groups.google.com/d/msgid/golang-nuts/1bb3f806-5930-4866-8249-0bbc0ee383b8n%40googlegroups.com?utm_medium=email_source=footer>
>> .
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAO_RewYvoN-LifeTTqSdWVHyqGo%3D92RC%2BoaUR_UkVxqYQVix-g%40mail.gmail.com.

Re: [go-nuts] testing if strconv.Quote() would do change a string, without calling it

['Tim Hockin' via golang-nuts]

` IsPrint(r) || r == '\\' || r == '"' ` passes tests.  I need to build
confidence in that, though :)  Thanks.

On Wed, Oct 13, 2021 at 4:16 PM Robert Engels  wrote:

> A simple loop calling IsPrint is your best bet. You could then have a
> custom implementation of Quote that started at a specified index.
>
> On Oct 13, 2021, at 5:46 PM, 'Tim Hockin' via golang-nuts <
> golang-nuts@googlegroups.com> wrote:
>
> Is there any ready-built function that can tell me whether
> `strconv.Quote()` would produce a different string than its input, without
> actually running it?  Or is there a clearly documented set of rules one
> could use to test each rune in a string?
>
> I am trying to avoid allocations, and MOST of the inputs will be safe (but
> not all).  Calling strconv.Quote() has a measurable impact, so I'd avoid it
> if I could...
>
> Does such an animal exist?
>
> Tim
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/1bb3f806-5930-4866-8249-0bbc0ee383b8n%40googlegroups.com
> <https://groups.google.com/d/msgid/golang-nuts/1bb3f806-5930-4866-8249-0bbc0ee383b8n%40googlegroups.com?utm_medium=email_source=footer>
> .
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAO_RewbBE6RQf3p1gme%2B4TKuRpkZHmfowe_HDSsqB1QsH_x%3DJw%40mail.gmail.com.

[go-nuts] testing if strconv.Quote() would do change a string, without calling it

['Tim Hockin' via golang-nuts]

Is there any ready-built function that can tell me whether 
`strconv.Quote()` would produce a different string than its input, without 
actually running it?  Or is there a clearly documented set of rules one 
could use to test each rune in a string?

I am trying to avoid allocations, and MOST of the inputs will be safe (but 
not all).  Calling strconv.Quote() has a measurable impact, so I'd avoid it 
if I could...

Does such an animal exist?

Tim

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/1bb3f806-5930-4866-8249-0bbc0ee383b8n%40googlegroups.com.

Re: [go-nuts] `go list` across multiple modules?

['Tim Hockin' via golang-nuts]

fixed.
>>>>>>>>
>>>>>>>> > In general, it should be possible to give 'go list' an absolute 
>>>>>>>> or relative path (starting with ./ or ../) to any directory containing 
>>>>>>>> a 
>>>>>>>> package which is part of any module in the build list. For example, 
>>>>>>>> some 
>>>>>>>> tools list directories in the module cache to find out what package a 
>>>>>>>> .go 
>>>>>>>> file belongs to.
>>>>>>>> >
>>>>>>>> > As a workaround, you could put a go.mod in an otherwise empty 
>>>>>>>> directory (in /tmp or something), then require the relevant modules 
>>>>>>>> from 
>>>>>>>> the repo and replace them with absolute paths. Then you can run 'go 
>>>>>>>> list' 
>>>>>>>> in that directory with absolute paths of package directories.
>>>>>>>>
>>>>>>>> Interesting - is the difference the absolute paths vs relative?
>>>>>>>>
>>>>>>>> I hoped maybe `-modfile` would do the same trick, but alas not:
>>>>>>>>
>>>>>>>> ```
>>>>>>>> $ (cd /tmp/gomodhack/; go list /tmp/go-list-modules/submod/used/)
>>>>>>>> example.com/m/submod/used
>>>>>>>>
>>>>>>>> $ go list --modfile /tmp/gomodhack/go.mod 
>>>>>>>> /tmp/go-list-modules/submod/used/
>>>>>>>> main module (tmp) does not contain package tmp/submod/used
>>>>>>>> ```
>>>>>>>>
>>>>>>>> It also fails some cases:
>>>>>>>>
>>>>>>>> ```
>>>>>>>>  (cd /tmp/gomodhack/; go list /tmp/go-list-modules/submod/used/)
>>>>>>>> example.com/m/submod/used
>>>>>>>> thockin@thockin-glaptop4 go-list-modules main /$ (cd 
>>>>>>>> /tmp/gomodhack/;
>>>>>>>> go list /tmp/go-list-modules/staging/src/example.com/other1/used/)
>>>>>>>> go: finding module for package 
>>>>>>>> example.com/m/staging/src/example.com/other1/used
>>>>>>>> cannot find module providing package
>>>>>>>> example.com/m/staging/src/example.com/other1/used: unrecognized 
>>>>>>>> import
>>>>>>>> path "example.com/m/staging/src/example.com/other1/used": reading
>>>>>>>> https://example.com/m/staging/src/example.com/other1/used?go-get=1:
>>>>>>>> 404 Not Found
>>>>>>>> ```
>>>>>>>>
>>>>>>>> It seems that is because the "main" (top-level dir) go.mod has
>>>>>>>> `replace` directives with relative paths, which kubernetes really
>>>>>>>> does.
>>>>>>>>
>>>>>>>> > Incidentally, golang.org/x/tools/go/packages will call 'go list' 
>>>>>>>> under the hood in module mode. go/build 
>>>>>>>> <https://goto.google.com/build> might do the same, depending on 
>>>>>>>> how it's invoked. 'go list' may be the best thing to use if it gives 
>>>>>>>> the 
>>>>>>>> information you need.
>>>>>>>>
>>>>>>>> Yeah, I noticed.  When GO111MODULE=off, everything I am doing is 
>>>>>>>> much
>>>>>>>> faster.  I'm wary of depending on that forever, though.
>>>>>>>>
>>>>>>>> Stepping back, I fear I am pushing the square peg into a round hole.
>>>>>>>> Let me restate what I am trying to do.
>>>>>>>>
>>>>>>>> I want to run a slow codegen process only if the packages it depends
>>>>>>>> on have ACTUALLY changed (mtime is a good enough proxy) and I don't
>>>>>>>> know a priori which packages need codegen.  I want to scan the file
>>>>>>>> tree, find the files that need codegen, check their deps, and only
>>>>>>>> then run the codegen.
>>>>>>>>
>>>>>>>> We do this today with `go list` and GO11

Re: [go-nuts] `go list` across multiple modules?

['Tim Hockin' via golang-nuts]

ed
>>>>> additional information on top of that (parsed syntax trees, type info).
>>>>>
>>>>> > I want to run a slow codegen process only if the packages it depends
>>>>> > on have ACTUALLY changed (mtime is a good enough proxy) and I don't
>>>>> > know a priori which packages need codegen.  I want to scan the file
>>>>> > tree, find the files that need codegen, check their deps, and only
>>>>> > then run the codegen.
>>>>>
>>>>> How much dependency info do you need? If the codegen is only within
>>>>> packages with files that have changed, 'go list' might be overkill (it
>>>>> always loads dependencies, even if they aren't printed). If you need
>>>>> dependencies or reverse dependencies, 'go list' or go/packages
>>>>> <https://goto.google.com/packages> are probably the right tools.
>>>>>
>>>>> On Fri, Jan 15, 2021 at 6:43 PM Tim Hockin  wrote:
>>>>>
>>>>>> On Fri, Jan 15, 2021 at 2:17 PM Jay Conrod 
>>>>>> wrote:
>>>>>> >
>>>>>> > I was initially going to suggest adding the module subdirectories
>>>>>> as requirements to the main go.mod, then replacing those with the
>>>>>> subdirectories.
>>>>>> >
>>>>>> > module example.com/m
>>>>>> >
>>>>>> > go 1.15
>>>>>> >
>>>>>> > require (
>>>>>> >   example.com/other1 v0.0.0
>>>>>> >   example.com/other2 v0.0.0
>>>>>> >   example.com/m/submod v0.0.0
>>>>>> > )
>>>>>> >
>>>>>> > replace (
>>>>>> >   example.com/other1 => ./staging/src/example.com/other1
>>>>>> >   example.com/other2 => ./staging/src/example.com/other2
>>>>>> >   example.com/m/submod v0.0.0 => ./submod
>>>>>> > )
>>>>>> >
>>>>>> >
>>>>>> > I think you might have tried this already. It gives the same "main
>>>>>> module ... does not contain package" error. I believe that's a bug. I've
>>>>>> opened #43733 to track it.
>>>>>>
>>>>>> Interesting.  If that's a bug, then maybe I'll be able to do what I
>>>>>> need once fixed.
>>>>>>
>>>>>> > In general, it should be possible to give 'go list' an absolute or
>>>>>> relative path (starting with ./ or ../) to any directory containing a
>>>>>> package which is part of any module in the build list. For example, some
>>>>>> tools list directories in the module cache to find out what package a .go
>>>>>> file belongs to.
>>>>>> >
>>>>>> > As a workaround, you could put a go.mod in an otherwise empty
>>>>>> directory (in /tmp or something), then require the relevant modules from
>>>>>> the repo and replace them with absolute paths. Then you can run 'go list'
>>>>>> in that directory with absolute paths of package directories.
>>>>>>
>>>>>> Interesting - is the difference the absolute paths vs relative?
>>>>>>
>>>>>> I hoped maybe `-modfile` would do the same trick, but alas not:
>>>>>>
>>>>>> ```
>>>>>> $ (cd /tmp/gomodhack/; go list /tmp/go-list-modules/submod/used/)
>>>>>> example.com/m/submod/used
>>>>>>
>>>>>> $ go list --modfile /tmp/gomodhack/go.mod
>>>>>> /tmp/go-list-modules/submod/used/
>>>>>> main module (tmp) does not contain package tmp/submod/used
>>>>>> ```
>>>>>>
>>>>>> It also fails some cases:
>>>>>>
>>>>>> ```
>>>>>>  (cd /tmp/gomodhack/; go list /tmp/go-list-modules/submod/used/)
>>>>>> example.com/m/submod/used
>>>>>> thockin@thockin-glaptop4 go-list-modules main /$ (cd /tmp/gomodhack/;
>>>>>> go list /tmp/go-list-modules/staging/src/example.com/other1/used/)
>>>>>> go: finding module for package
>>>>>> example.com/m/staging/src/example.com/other1/used
>>>>>> cannot find module providing package
>>>>>> example.com/m/staging/src/example.com/

Re: [go-nuts] Trying to use a tool in my build (a friction log)

['Tim Hockin' via golang-nuts]

Thanks for feedback, comments below

On Thu, Mar 18, 2021 at 3:35 PM Jay Conrod  wrote:

>
> Installing a command within the context of the current module is more
> complicated. I'd suggest this:
>
> # Add a dependency on the tool (only need to do this once)
> go get -d github.com/google/go-licenses
>
> # Import the command from tools.go
>
> # See below
>
>
> # Sync vendor directory (only if vendoring; only after changing go.mod
> with 'go get' above)
> go mod vendor
>
> # Install the command
> go install github.com/google/go-licenses
>
> # Alternatively, build and write to some other directory
> go build -o bin/go-licenses github.com/google/go-licenses
>
>
I bent this a bit, and it seems to work:

1) add tools.go and import the tool
2) go mod vendor
3) go build

IOW, I skipped `go get -d` and it all seems OK.

Now it builds.  If I run `go mod tidy` it removes a line from my go.mod (`
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b // indirect`) and
then refuses to build:

```
go: inconsistent vendoring in /src:
github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b: is marked as
explicit in vendor/modules.txt, but not explicitly required in go.mod

run 'go mod vendor' to sync, or use -mod=mod or -mod=readonly to ignore the
vendor directory
```

Now I run `go mod tidy` again and it adds a line to `vendor/modules.txt`
and happily compiles again.

Subsequent runs of either command seem to be no-op at this point.

Is it by design that these two commands are unstable like that?

Now that it builds, extra fun.  This tool in particular seems to try to
read a DB file from a subdir of the directory its source code came from (as
per `runtime.Caller(0)`) which ...doesn't work at all with vendoring, since
the dir doesn't contain Go code.

AFAIK there's not a way to ask `go mod vendor` to include the whole repo is
there?  I can't find a spec for modules.txt - it would be super cool if I
could add some tag in there...  As it stands, all this was for naught.


> *2. "writing go.mod cache" error messages*
>
> This error message should be a lot better. Sorry about that.
>

I mean, "mkdir /home/thockin/go: not a directory" is pretty concise - it's
not a directory (on purpose) because I really don't want Go randomly
deciding to write stuff to my homedir (which I suspected was happening, and
that's why I made it a non-directory :)


> *3. 'go get' updating go.mod and go.sum*
>
> This is an unfortunate consequence of 'go get' being overloaded for
> downloading, installing, and managing package versions. Ideally, we want to
> separate these roles into different commands: 'go get' should be used to
> manage dependencies (changing requirements in go.mod and go.sum), and 'go
> install' should be used to install commands.
>
> We've taken a couple steps toward this in 1.16. The 'go install cmd@version'
> form is new. Also, 'go install', 'go build', and other commands no longer
> change go.mod or go.sum automatically when something is missing.
>

>From where I site, I like the distinction, and I might even argue for
explicitness.  E.g.  if I said `go get -mod=vendor` I feel like I am
unambiguously asking the tool to fetch the code and put it in ./vendor and
update the module files.  If I said just `go get` I (not so unambiguously)
mean "global" (for me).  I find the automatic-detection of module mode very
confusing, personally.

Tim

On Thu, Mar 18, 2021 at 3:36 PM 'Tim Hockin' via golang-nuts <
> golang-nuts@googlegroups.com> wrote:
>
>> First: go version go1.16 linux/amd64
>>
>> In one of my side-projects, I build a container image.  It turns out
>> to be useful to people, but I got a request to put the license files
>> for all code used in the binary into the container.  Lawyers, what can
>> you do?
>>
>> So these people sent me a PR to gather the licenses with `go install
>> github.com/google/go-licenses` <http://github.com/google/go-licenses> in
>> the `make container` rule.  Here's
>> the friction.
>>
>> 1) Their PR includes dozens of new lines in go.sum, which makes me
>> grumpy.  I set out to see if I can do better.
>>
>> 2) When I run that command on my workstation I get:
>>
>> ```
>> $ go install github.com/google/go-licenses
>> cannot find package "." in:
>> /home/thockin/src/go/src/
>> k8s.io/git-sync/vendor/github.com/google/go-licenses
>> ```
>>
>> I don't know what to make of that message.  It's not in vendor/ yet -
>> I want to put it there (I think?).
>>
>> So I am left wondering - what is the normal flow for "I want to use
>> this Go tool in my build" ?
>>
>> I tried `-mod=mod`:
>>
>> ```
>> $ (unset GOPATH; go 

[go-nuts] Trying to use a tool in my build (a friction log)

['Tim Hockin' via golang-nuts]

First: go version go1.16 linux/amd64

In one of my side-projects, I build a container image.  It turns out
to be useful to people, but I got a request to put the license files
for all code used in the binary into the container.  Lawyers, what can
you do?

So these people sent me a PR to gather the licenses with `go install
github.com/google/go-licenses` in the `make container` rule.  Here's
the friction.

1) Their PR includes dozens of new lines in go.sum, which makes me
grumpy.  I set out to see if I can do better.

2) When I run that command on my workstation I get:

```
$ go install github.com/google/go-licenses
cannot find package "." in:
/home/thockin/src/go/src/k8s.io/git-sync/vendor/github.com/google/go-licenses
```

I don't know what to make of that message.  It's not in vendor/ yet -
I want to put it there (I think?).

So I am left wondering - what is the normal flow for "I want to use
this Go tool in my build" ?

I tried `-mod=mod`:

```
$ (unset GOPATH; go install -mod=mod github.com/google/go-licenses)
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: finding module for package github.com/google/go-licenses
go: downloading github.com/google/go-licenses v0.0.0-20201026145851-73411c8fa237
mkdir /home/thockin/go: not a directory
```

I keep hearing that GOPATH is dead (and frankly good riddance) but
this tries to write to GOPATH or ~/go (which is also annoying).  I
don't want this project's build spilling over into my homedir.

I tried `go get` but it's the same.

```
$ (unset GOPATH; go get github.com/google/go-licenses)
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: downloading github.com/google/go-licenses v0.0.0-20201026145851-73411c8fa237
go get github.com/google/go-licenses: mkdir /home/thockin/go: not a directory
```

What do I *want*?   I want to download a tool repo, build it, drop the
binary into ./bin/tools (or something) and then either vendor it or
have no artifacts.

Since GOPATH seems to be a requirement, I thought maybe I can fake it:

```
$ (export GOPATH="`pwd`/.gopath"; export GOBIN=`pwd`/bin/tools; mkdir
-p "$GOBIN"; go get github.com/google/go-licenses)
go: downloading github.com/google/go-licenses v0.0.0-20201026145851-73411c8fa237
go: downloading github.com/otiai10/copy v1.2.0
go: downloading github.com/spf13/cobra v0.0.5
go: downloading github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b
go: downloading github.com/google/licenseclassifier
v0.0.0-20190926221455-842c0d70d702
go: downloading golang.org/x/tools v0.0.0-20191118222007-07fc4c7f2b98
go: downloading gopkg.in/src-d/go-git.v4 v4.13.1
go: downloading github.com/spf13/pflag v1.0.5
go: downloading github.com/inconshreveable/mousetrap v1.0.0
go: downloading github.com/sergi/go-diff v1.0.0
go: downloading gopkg.in/src-d/go-billy.v4 v4.3.2
go: downloading golang.org/x/crypto v0.0.0-20191117063200-497ca9f6d64f
go: downloading golang.org/x/sys v0.0.0-20191119060738-e882bf8e40c2
go: downloading github.com/emirpasic/gods v1.12.0
go: downloading github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99
go: downloading github.com/src-d/gcfg v1.4.0
go: downloading github.com/kevinburke/ssh_config
v0.0.0-20190725054713-01f96b0aa0cd
go: downloading github.com/mitchellh/go-homedir v1.1.0
go: downloading github.com/xanzy/ssh-agent v0.2.1
go: downloading golang.org/x/net v0.0.0-20191119073136-fc4aabc6c914
go: downloading gopkg.in/warnings.v0 v0.1.2
go get: added github.com/google/go-licenses v0.0.0-20201026145851-73411c8fa237
```

This does actually build 

Re: [go-nuts] `go list` across multiple modules?

['Tim Hockin' via golang-nuts]

t tools.
>>>>
>>>> On Fri, Jan 15, 2021 at 6:43 PM Tim Hockin  wrote:
>>>>
>>>>> On Fri, Jan 15, 2021 at 2:17 PM Jay Conrod 
>>>>> wrote:
>>>>> >
>>>>> > I was initially going to suggest adding the module subdirectories as
>>>>> requirements to the main go.mod, then replacing those with the
>>>>> subdirectories.
>>>>> >
>>>>> > module example.com/m
>>>>> >
>>>>> > go 1.15
>>>>> >
>>>>> > require (
>>>>> >   example.com/other1 v0.0.0
>>>>> >   example.com/other2 v0.0.0
>>>>> >   example.com/m/submod v0.0.0
>>>>> > )
>>>>> >
>>>>> > replace (
>>>>> >   example.com/other1 => ./staging/src/example.com/other1
>>>>> >   example.com/other2 => ./staging/src/example.com/other2
>>>>> >   example.com/m/submod v0.0.0 => ./submod
>>>>> > )
>>>>> >
>>>>> >
>>>>> > I think you might have tried this already. It gives the same "main
>>>>> module ... does not contain package" error. I believe that's a bug. I've
>>>>> opened #43733 to track it.
>>>>>
>>>>> Interesting.  If that's a bug, then maybe I'll be able to do what I
>>>>> need once fixed.
>>>>>
>>>>> > In general, it should be possible to give 'go list' an absolute or
>>>>> relative path (starting with ./ or ../) to any directory containing a
>>>>> package which is part of any module in the build list. For example, some
>>>>> tools list directories in the module cache to find out what package a .go
>>>>> file belongs to.
>>>>> >
>>>>> > As a workaround, you could put a go.mod in an otherwise empty
>>>>> directory (in /tmp or something), then require the relevant modules from
>>>>> the repo and replace them with absolute paths. Then you can run 'go list'
>>>>> in that directory with absolute paths of package directories.
>>>>>
>>>>> Interesting - is the difference the absolute paths vs relative?
>>>>>
>>>>> I hoped maybe `-modfile` would do the same trick, but alas not:
>>>>>
>>>>> ```
>>>>> $ (cd /tmp/gomodhack/; go list /tmp/go-list-modules/submod/used/)
>>>>> example.com/m/submod/used
>>>>>
>>>>> $ go list --modfile /tmp/gomodhack/go.mod
>>>>> /tmp/go-list-modules/submod/used/
>>>>> main module (tmp) does not contain package tmp/submod/used
>>>>> ```
>>>>>
>>>>> It also fails some cases:
>>>>>
>>>>> ```
>>>>>  (cd /tmp/gomodhack/; go list /tmp/go-list-modules/submod/used/)
>>>>> example.com/m/submod/used
>>>>> thockin@thockin-glaptop4 go-list-modules main /$ (cd /tmp/gomodhack/;
>>>>> go list /tmp/go-list-modules/staging/src/example.com/other1/used/)
>>>>> go: finding module for package
>>>>> example.com/m/staging/src/example.com/other1/used
>>>>> cannot find module providing package
>>>>> example.com/m/staging/src/example.com/other1/used: unrecognized import
>>>>> path "example.com/m/staging/src/example.com/other1/used": reading
>>>>> https://example.com/m/staging/src/example.com/other1/used?go-get=1:
>>>>> 404 Not Found
>>>>> ```
>>>>>
>>>>> It seems that is because the "main" (top-level dir) go.mod has
>>>>> `replace` directives with relative paths, which kubernetes really
>>>>> does.
>>>>>
>>>>> > Incidentally, golang.org/x/tools/go/packages will call 'go list'
>>>>> under the hood in module mode. go/build
>>>>> <https://goto.google.com/build> might do the same, depending on how
>>>>> it's invoked. 'go list' may be the best thing to use if it gives the
>>>>> information you need.
>>>>>
>>>>> Yeah, I noticed.  When GO111MODULE=off, everything I am doing is much
>>>>> faster.  I'm wary of depending on that forever, though.
>>>>>
>>>>> Stepping back, I fear I am pushing the square peg into a round hole.
>>>>> Let me restate what I am trying to do.
>&

Re: [go-nuts] `go list` across multiple modules?

['Tim Hockin' via golang-nuts]

part of any module in the build list. For example, some
>>> tools list directories in the module cache to find out what package a .go
>>> file belongs to.
>>> >
>>> > As a workaround, you could put a go.mod in an otherwise empty
>>> directory (in /tmp or something), then require the relevant modules from
>>> the repo and replace them with absolute paths. Then you can run 'go list'
>>> in that directory with absolute paths of package directories.
>>>
>>> Interesting - is the difference the absolute paths vs relative?
>>>
>>> I hoped maybe `-modfile` would do the same trick, but alas not:
>>>
>>> ```
>>> $ (cd /tmp/gomodhack/; go list /tmp/go-list-modules/submod/used/)
>>> example.com/m/submod/used
>>>
>>> $ go list --modfile /tmp/gomodhack/go.mod
>>> /tmp/go-list-modules/submod/used/
>>> main module (tmp) does not contain package tmp/submod/used
>>> ```
>>>
>>> It also fails some cases:
>>>
>>> ```
>>>  (cd /tmp/gomodhack/; go list /tmp/go-list-modules/submod/used/)
>>> example.com/m/submod/used
>>> thockin@thockin-glaptop4 go-list-modules main /$ (cd /tmp/gomodhack/;
>>> go list /tmp/go-list-modules/staging/src/example.com/other1/used/)
>>> go: finding module for package
>>> example.com/m/staging/src/example.com/other1/used
>>> cannot find module providing package
>>> example.com/m/staging/src/example.com/other1/used: unrecognized import
>>> path "example.com/m/staging/src/example.com/other1/used": reading
>>> https://example.com/m/staging/src/example.com/other1/used?go-get=1:
>>> 404 Not Found
>>> ```
>>>
>>> It seems that is because the "main" (top-level dir) go.mod has
>>> `replace` directives with relative paths, which kubernetes really
>>> does.
>>>
>>> > Incidentally, golang.org/x/tools/go/packages will call 'go list'
>>> under the hood in module mode. go/build <https://goto.google.com/build>
>>> might do the same, depending on how it's invoked. 'go list' may be the best
>>> thing to use if it gives the information you need.
>>>
>>> Yeah, I noticed.  When GO111MODULE=off, everything I am doing is much
>>> faster.  I'm wary of depending on that forever, though.
>>>
>>> Stepping back, I fear I am pushing the square peg into a round hole.
>>> Let me restate what I am trying to do.
>>>
>>> I want to run a slow codegen process only if the packages it depends
>>> on have ACTUALLY changed (mtime is a good enough proxy) and I don't
>>> know a priori which packages need codegen.  I want to scan the file
>>> tree, find the files that need codegen, check their deps, and only
>>> then run the codegen.
>>>
>>> We do this today with `go list` and GO111MODULE=off, but I was advised
>>> at some point that x/tools/go/packages was the future-safe approach.
>>>
>>> If there's a better way, I am all ears.
>>>
>>> Tim
>>> GO111MODULE=off
>>> > On Fri, Jan 15, 2021 at 11:59 AM 'Tim Hockin' via golang-nuts <
>>> golang-nuts@googlegroups.com> wrote:
>>> >>
>>> >> Hi.  This isn't exactly burning urgent, but it is a long-term issue
>>> >> for Kubernetes.  If there's anything I can do to catalyze the
>>> >> discussion - tests to run, info to dig up, etc - please let me know.
>>> >>
>>> >> On Wed, Dec 23, 2020 at 10:48 AM Tim Hockin 
>>> wrote:
>>> >> >
>>> >> > Hi Paul!
>>> >> >
>>> >> > On Wed, Dec 23, 2020 at 4:23 AM Paul Jolly  wrote:
>>> >> > >
>>> >> > > > I just can't figure out how to do this.  Maybe it can't be done
>>> in `go
>>> >> > > > list` ?  Or maybe we're just missing some detail of go modules..
>>> >> > >
>>> >> > > go list operates in the context of a single module (in the mode
>>> you
>>> >> > > are interested in), so you cannot do this with a single command
>>> across
>>> >> > > multiple modules.
>>> >> >
>>> >> > This might be a real problem for us.  For this post I am reducing it
>>> >> > to `go list`, but in actuality we have a small program that we wrote
>>> >> > which does what we need in terms of `go/build`.  It works great when
>>> >>

Re: [go-nuts] `go list` across multiple modules?

['Tim Hockin' via golang-nuts]

Interesting.  If that's a bug, then maybe I'll be able to do what I
>> need once fixed.
>>
>> > In general, it should be possible to give 'go list' an absolute or
relative path (starting with ./ or ../) to any directory containing a
package which is part of any module in the build list. For example, some
tools list directories in the module cache to find out what package a .go
file belongs to.
>> >
>> > As a workaround, you could put a go.mod in an otherwise empty
directory (in /tmp or something), then require the relevant modules from
the repo and replace them with absolute paths. Then you can run 'go list'
in that directory with absolute paths of package directories.
>>
>> Interesting - is the difference the absolute paths vs relative?
>>
>> I hoped maybe `-modfile` would do the same trick, but alas not:
>>
>> ```
>> $ (cd /tmp/gomodhack/; go list /tmp/go-list-modules/submod/used/)
>> example.com/m/submod/used
>>
>> $ go list --modfile /tmp/gomodhack/go.mod
/tmp/go-list-modules/submod/used/
>> main module (tmp) does not contain package tmp/submod/used
>> ```
>>
>> It also fails some cases:
>>
>> ```
>>  (cd /tmp/gomodhack/; go list /tmp/go-list-modules/submod/used/)
>> example.com/m/submod/used
>> thockin@thockin-glaptop4 go-list-modules main /$ (cd /tmp/gomodhack/;
>> go list /tmp/go-list-modules/staging/src/example.com/other1/used/)
>> go: finding module for package
example.com/m/staging/src/example.com/other1/used
>> cannot find module providing package
>> example.com/m/staging/src/example.com/other1/used: unrecognized import
>> path "example.com/m/staging/src/example.com/other1/used": reading
>> https://example.com/m/staging/src/example.com/other1/used?go-get=1:
>> 404 Not Found
>> ```
>>
>> It seems that is because the "main" (top-level dir) go.mod has
>> `replace` directives with relative paths, which kubernetes really
>> does.
>>
>> > Incidentally, golang.org/x/tools/go/packages will call 'go list' under
the hood in module mode. go/build might do the same, depending on how it's
invoked. 'go list' may be the best thing to use if it gives the information
you need.
>>
>> Yeah, I noticed.  When GO111MODULE=off, everything I am doing is much
>> faster.  I'm wary of depending on that forever, though.
>>
>> Stepping back, I fear I am pushing the square peg into a round hole.
>> Let me restate what I am trying to do.
>>
>> I want to run a slow codegen process only if the packages it depends
>> on have ACTUALLY changed (mtime is a good enough proxy) and I don't
>> know a priori which packages need codegen.  I want to scan the file
>> tree, find the files that need codegen, check their deps, and only
>> then run the codegen.
>>
>> We do this today with `go list` and GO111MODULE=off, but I was advised
>> at some point that x/tools/go/packages was the future-safe approach.
>>
>> If there's a better way, I am all ears.
>>
>> Tim
>> GO111MODULE=off
>> > On Fri, Jan 15, 2021 at 11:59 AM 'Tim Hockin' via golang-nuts <
golang-nuts@googlegroups.com> wrote:
>> >>
>> >> Hi.  This isn't exactly burning urgent, but it is a long-term issue
>> >> for Kubernetes.  If there's anything I can do to catalyze the
>> >> discussion - tests to run, info to dig up, etc - please let me know.
>> >>
>> >> On Wed, Dec 23, 2020 at 10:48 AM Tim Hockin 
wrote:
>> >> >
>> >> > Hi Paul!
>> >> >
>> >> > On Wed, Dec 23, 2020 at 4:23 AM Paul Jolly  wrote:
>> >> > >
>> >> > > > I just can't figure out how to do this.  Maybe it can't be done
in `go
>> >> > > > list` ?  Or maybe we're just missing some detail of go modules..
>> >> > >
>> >> > > go list operates in the context of a single module (in the mode
you
>> >> > > are interested in), so you cannot do this with a single command
across
>> >> > > multiple modules.
>> >> >
>> >> > This might be a real problem for us.  For this post I am reducing it
>> >> > to `go list`, but in actuality we have a small program that we wrote
>> >> > which does what we need in terms of `go/build`.  It works great when
>> >> > `GO111MODULE=off` but is more than 100x slower normally.  I thought
it
>> >> > was finally time to rewrite it in terms of `go/packages` and get rid
>> >> > of GO111MODULE=off.  That didn't pan out, hence this post.
>> >> >
>

Re: [go-nuts] `go list` across multiple modules?

['Tim Hockin' via golang-nuts]

On Fri, Jan 15, 2021 at 2:17 PM Jay Conrod  wrote:
>
> I was initially going to suggest adding the module subdirectories as 
> requirements to the main go.mod, then replacing those with the subdirectories.
>
> module example.com/m
>
> go 1.15
>
> require (
>   example.com/other1 v0.0.0
>   example.com/other2 v0.0.0
>   example.com/m/submod v0.0.0
> )
>
> replace (
>   example.com/other1 => ./staging/src/example.com/other1
>   example.com/other2 => ./staging/src/example.com/other2
>   example.com/m/submod v0.0.0 => ./submod
> )
>
>
> I think you might have tried this already. It gives the same "main module ... 
> does not contain package" error. I believe that's a bug. I've opened #43733 
> to track it.

Interesting.  If that's a bug, then maybe I'll be able to do what I
need once fixed.

> In general, it should be possible to give 'go list' an absolute or relative 
> path (starting with ./ or ../) to any directory containing a package which is 
> part of any module in the build list. For example, some tools list 
> directories in the module cache to find out what package a .go file belongs 
> to.
>
> As a workaround, you could put a go.mod in an otherwise empty directory (in 
> /tmp or something), then require the relevant modules from the repo and 
> replace them with absolute paths. Then you can run 'go list' in that 
> directory with absolute paths of package directories.

Interesting - is the difference the absolute paths vs relative?

I hoped maybe `-modfile` would do the same trick, but alas not:

```
$ (cd /tmp/gomodhack/; go list /tmp/go-list-modules/submod/used/)
example.com/m/submod/used

$ go list --modfile /tmp/gomodhack/go.mod /tmp/go-list-modules/submod/used/
main module (tmp) does not contain package tmp/submod/used
```

It also fails some cases:

```
 (cd /tmp/gomodhack/; go list /tmp/go-list-modules/submod/used/)
example.com/m/submod/used
thockin@thockin-glaptop4 go-list-modules main /$ (cd /tmp/gomodhack/;
go list /tmp/go-list-modules/staging/src/example.com/other1/used/)
go: finding module for package example.com/m/staging/src/example.com/other1/used
cannot find module providing package
example.com/m/staging/src/example.com/other1/used: unrecognized import
path "example.com/m/staging/src/example.com/other1/used": reading
https://example.com/m/staging/src/example.com/other1/used?go-get=1:
404 Not Found
```

It seems that is because the "main" (top-level dir) go.mod has
`replace` directives with relative paths, which kubernetes really
does.

> Incidentally, golang.org/x/tools/go/packages will call 'go list' under the 
> hood in module mode. go/build might do the same, depending on how it's 
> invoked. 'go list' may be the best thing to use if it gives the information 
> you need.

Yeah, I noticed.  When GO111MODULE=off, everything I am doing is much
faster.  I'm wary of depending on that forever, though.

Stepping back, I fear I am pushing the square peg into a round hole.
Let me restate what I am trying to do.

I want to run a slow codegen process only if the packages it depends
on have ACTUALLY changed (mtime is a good enough proxy) and I don't
know a priori which packages need codegen.  I want to scan the file
tree, find the files that need codegen, check their deps, and only
then run the codegen.

We do this today with `go list` and GO111MODULE=off, but I was advised
at some point that x/tools/go/packages was the future-safe approach.

If there's a better way, I am all ears.

Tim
GO111MODULE=off
> On Fri, Jan 15, 2021 at 11:59 AM 'Tim Hockin' via golang-nuts 
>  wrote:
>>
>> Hi.  This isn't exactly burning urgent, but it is a long-term issue
>> for Kubernetes.  If there's anything I can do to catalyze the
>> discussion - tests to run, info to dig up, etc - please let me know.
>>
>> On Wed, Dec 23, 2020 at 10:48 AM Tim Hockin  wrote:
>> >
>> > Hi Paul!
>> >
>> > On Wed, Dec 23, 2020 at 4:23 AM Paul Jolly  wrote:
>> > >
>> > > > I just can't figure out how to do this.  Maybe it can't be done in `go
>> > > > list` ?  Or maybe we're just missing some detail of go modules..
>> > >
>> > > go list operates in the context of a single module (in the mode you
>> > > are interested in), so you cannot do this with a single command across
>> > > multiple modules.
>> >
>> > This might be a real problem for us.  For this post I am reducing it
>> > to `go list`, but in actuality we have a small program that we wrote
>> > which does what we need in terms of `go/build`.  It works great when
>> > `GO111MODULE=off` but is more than 100x slower normally.  I thought it
>> > was finally time to rewrite it in terms of `go/packages` a

Re: [go-nuts] `go list` across multiple modules?

['Tim Hockin' via golang-nuts]

Hi.  This isn't exactly burning urgent, but it is a long-term issue
for Kubernetes.  If there's anything I can do to catalyze the
discussion - tests to run, info to dig up, etc - please let me know.

On Wed, Dec 23, 2020 at 10:48 AM Tim Hockin  wrote:
>
> Hi Paul!
>
> On Wed, Dec 23, 2020 at 4:23 AM Paul Jolly  wrote:
> >
> > > I just can't figure out how to do this.  Maybe it can't be done in `go
> > > list` ?  Or maybe we're just missing some detail of go modules..
> >
> > go list operates in the context of a single module (in the mode you
> > are interested in), so you cannot do this with a single command across
> > multiple modules.
>
> This might be a real problem for us.  For this post I am reducing it
> to `go list`, but in actuality we have a small program that we wrote
> which does what we need in terms of `go/build`.  It works great when
> `GO111MODULE=off` but is more than 100x slower normally.  I thought it
> was finally time to rewrite it in terms of `go/packages` and get rid
> of GO111MODULE=off.  That didn't pan out, hence this post.
>
> More inline and below
>
> > > First I do a `find` for any file that has a specific comment tag,
> > > indicating that the package needs codegen.  The results span several
> > > of the in-repo submodules.
> >
> > Just to check, I'm assuming the results of this find command are being
> > translated to a list of packages? Because the transitive dependencies
> > of a list of packages within a module can be done via a single go list
> > command.
>
> The trick is "within a module".  I'll update
> https://github.com/thockin/go-list-modules to reflect the process
> more.   I've added a
> get_codegen_deps.sh that models the behavior.  Note that I really want
> files, not packages, so I can express the dep-graph.
>
> What do you mean by "translated to a list of packages" - which specific 
> syntax?
>
> What I end up with is something like `go list ./path/to/dir1
> ./path/to/dir2 ./path/to/dir3`.  Any of those dirs might be in
> different modules.  So `go list` tells me "main module (example.com/m)
> does not contain package example.com/m/path/to/dir1" and so on.
> Setting `GO111MODULE=off` does work, but I fear the future of that.
>
> > > For each target package, I want to get the list of all deps and
> > > extract the GoFiles.  Then I can use that to determine if the codegen
> > > needs to run.
> >
> > FWIW I wrote a tool to do just this:
> > https://pkg.go.dev/myitcv.io@v0.0.0-20201125173645-a7167afc9e13/cmd/gogenerate
> > which might work in your situation.
>
> I will take a look - it seems I will need to restructure a bunch of
> tooling to prove it works for us or doesn't :)
>
> > > Where it breaks down is that I can't seem to `go list` all at once:
> > >
> > > ```
> > > # This works within the "root" module
> > > $ go list -f '{{.GoFiles}}' ./subdir
> > > [file.go]
> >
> > This will work.
> >
> > > # This does not work across modules
> > > $ go list -f '{{.GoFiles}}' ./submod/used ./submod/unused
> > > main module (example.com/m) does not contain package 
> > > example.com/m/submod/used
> > > main module (example.com/m) does not contain package 
> > > example.com/m/submod/unused
> >
> > Per above, this will not work across module boundaries.
>
> It works with `GO111MODULE=off` which means that introducing modules
> is a breaking change.  Can I depend on GO111MODULE=off to work the
> same way forever?
>
> > > # Nor does this work, even with module replacements
> > > $ go list -f '{{.GoFiles}}' ./staging/src/example.com/other1/used
> > > ./staging/src/example.com/other1/unused
> > > main module (example.com/m) does not contain package
> > > example.com/m/staging/src/example.com/other1/used
> > > main module (example.com/m) does not contain package
> > > example.com/m/staging/src/example.com/other1/unused
> > > ```
> >
> > With replace directives in place this should work, but you won't be
> > able to use the relative path to the modules (which is in fact
> > interpreted as a directory): it will need to be the full
> > module/package path.
>
> Given a "./path/to/pkg" - how do I convert that to a module/package
> path?  I can run `(cd $dir && go list -m)` but that is super slow.
> Running JUST that for each directory that needs codegen in kubernetes
> takes 20+ seconds.  Is there a better way, short of writing my own
> directory-climb and parsing go.mod?
>
> > > I can run `go list` multiple times, but that's INCREDIBLY slow - most
> > > of these submodules have common deps that are large.  This re-parses
> > > everything over and over.  It takes almost 60 seconds just to do `cd
> > > $dir; go list` (on the real kubernetes repo).
> >
> > Do you have a repro of this taking 60 seconds? Because that really
> > shouldn't be the case with a populated local module cache.
>
> github.com/kubernetes/kubernetes
>
> ```
> $ time \
> find . -type f -name \*.go \
> | xargs grep -l "^// *+k8s:" \
> | xargs -n 1 dirname \
> | sort \
> | uniq \
>  

Re: [go-nuts] `go list` across multiple modules?

['Tim Hockin' via golang-nuts]

Hi Paul!

On Wed, Dec 23, 2020 at 4:23 AM Paul Jolly  wrote:
>
> > I just can't figure out how to do this.  Maybe it can't be done in `go
> > list` ?  Or maybe we're just missing some detail of go modules..
>
> go list operates in the context of a single module (in the mode you
> are interested in), so you cannot do this with a single command across
> multiple modules.

This might be a real problem for us.  For this post I am reducing it
to `go list`, but in actuality we have a small program that we wrote
which does what we need in terms of `go/build`.  It works great when
`GO111MODULE=off` but is more than 100x slower normally.  I thought it
was finally time to rewrite it in terms of `go/packages` and get rid
of GO111MODULE=off.  That didn't pan out, hence this post.

More inline and below

> > First I do a `find` for any file that has a specific comment tag,
> > indicating that the package needs codegen.  The results span several
> > of the in-repo submodules.
>
> Just to check, I'm assuming the results of this find command are being
> translated to a list of packages? Because the transitive dependencies
> of a list of packages within a module can be done via a single go list
> command.

The trick is "within a module".  I'll update
https://github.com/thockin/go-list-modules to reflect the process
more.   I've added a
get_codegen_deps.sh that models the behavior.  Note that I really want
files, not packages, so I can express the dep-graph.

What do you mean by "translated to a list of packages" - which specific syntax?

What I end up with is something like `go list ./path/to/dir1
./path/to/dir2 ./path/to/dir3`.  Any of those dirs might be in
different modules.  So `go list` tells me "main module (example.com/m)
does not contain package example.com/m/path/to/dir1" and so on.
Setting `GO111MODULE=off` does work, but I fear the future of that.

> > For each target package, I want to get the list of all deps and
> > extract the GoFiles.  Then I can use that to determine if the codegen
> > needs to run.
>
> FWIW I wrote a tool to do just this:
> https://pkg.go.dev/myitcv.io@v0.0.0-20201125173645-a7167afc9e13/cmd/gogenerate
> which might work in your situation.

I will take a look - it seems I will need to restructure a bunch of
tooling to prove it works for us or doesn't :)

> > Where it breaks down is that I can't seem to `go list` all at once:
> >
> > ```
> > # This works within the "root" module
> > $ go list -f '{{.GoFiles}}' ./subdir
> > [file.go]
>
> This will work.
>
> > # This does not work across modules
> > $ go list -f '{{.GoFiles}}' ./submod/used ./submod/unused
> > main module (example.com/m) does not contain package 
> > example.com/m/submod/used
> > main module (example.com/m) does not contain package 
> > example.com/m/submod/unused
>
> Per above, this will not work across module boundaries.

It works with `GO111MODULE=off` which means that introducing modules
is a breaking change.  Can I depend on GO111MODULE=off to work the
same way forever?

> > # Nor does this work, even with module replacements
> > $ go list -f '{{.GoFiles}}' ./staging/src/example.com/other1/used
> > ./staging/src/example.com/other1/unused
> > main module (example.com/m) does not contain package
> > example.com/m/staging/src/example.com/other1/used
> > main module (example.com/m) does not contain package
> > example.com/m/staging/src/example.com/other1/unused
> > ```
>
> With replace directives in place this should work, but you won't be
> able to use the relative path to the modules (which is in fact
> interpreted as a directory): it will need to be the full
> module/package path.

Given a "./path/to/pkg" - how do I convert that to a module/package
path?  I can run `(cd $dir && go list -m)` but that is super slow.
Running JUST that for each directory that needs codegen in kubernetes
takes 20+ seconds.  Is there a better way, short of writing my own
directory-climb and parsing go.mod?

> > I can run `go list` multiple times, but that's INCREDIBLY slow - most
> > of these submodules have common deps that are large.  This re-parses
> > everything over and over.  It takes almost 60 seconds just to do `cd
> > $dir; go list` (on the real kubernetes repo).
>
> Do you have a repro of this taking 60 seconds? Because that really
> shouldn't be the case with a populated local module cache.

github.com/kubernetes/kubernetes

```
$ time \
find . -type f -name \*.go \
| xargs grep -l "^// *+k8s:" \
| xargs -n 1 dirname \
| sort \
| uniq \
| while read X; do \
(cd $X; go list -f '{{.Deps}}'); \
done \
> /dev/null

real 0m50.488s
user 0m46.686s
sys 0m18.416s
```

Just running that inner `go list` with GO111MODULE=off cuts the run
time in half.

Compare to:

```
time \
( \
export GO111MODULE=off; \
find . -type f -name \*.go \
| xargs grep -l "^// *+k8s:" \
| xargs -n 1 dirname \
| sort \
| uniq \
   

[go-nuts] `go list` across multiple modules?

['Tim Hockin' via golang-nuts]

I just can't figure out how to do this.  Maybe it can't be done in `go
list` ?  Or maybe we're just missing some detail of go modules..

I have a repo (kubernetes, but minimized at
https://github.com/thockin/go-list-modules) which has a "main" go
module, and several other modules in the same repo.  It's probable not
worth explaining the history right now, but here we are.

As part of the build, I need to run some code generators.  They are
slow, so I only want to run them if their inputs have actually
changed.  To do this, I need to build a list of all the individual
files that are transitively depended on by the package in question.

First I do a `find` for any file that has a specific comment tag,
indicating that the package needs codegen.  The results span several
of the in-repo submodules.

For each target package, I want to get the list of all deps and
extract the GoFiles.  Then I can use that to determine if the codegen
needs to run.

Where it breaks down is that I can't seem to `go list` all at once:

```
# This works within the "root" module
$ go list -f '{{.GoFiles}}' ./subdir
[file.go]

# This does not work across modules
$ go list -f '{{.GoFiles}}' ./submod/used ./submod/unused
main module (example.com/m) does not contain package example.com/m/submod/used
main module (example.com/m) does not contain package example.com/m/submod/unused

# Nor does this work, even with module replacements
$ go list -f '{{.GoFiles}}' ./staging/src/example.com/other1/used
./staging/src/example.com/other1/unused
main module (example.com/m) does not contain package
example.com/m/staging/src/example.com/other1/used
main module (example.com/m) does not contain package
example.com/m/staging/src/example.com/other1/unused
```

I can run `go list` multiple times, but that's INCREDIBLY slow - most
of these submodules have common deps that are large.  This re-parses
everything over and over.  It takes almost 60 seconds just to do `cd
$dir; go list` (on the real kubernetes repo).

I wrote a program against go/packages which does SLIGHTLY better, in
that it finds packages in other modules but ONLY if they a) are a dep
of something in the root module; and b) exist in vendor/.  Note here
that ./subir DOES depend on ./submod, but that is not listed.

```
$go2make2 -p ./subdir
Package "example.com/other1/used" {
  Name: "used"
  PkgPath: "example.com/other1/used"
  GoFiles: [
"/tmp/m/go-list-modules/vendor/example.com/other1/used/file.go"
  ]
  Imports: [
  ]
}
Package "example.com/other2/used" {
  Name: "used"
  PkgPath: "example.com/other2/used"
  GoFiles: [
"/tmp/m/go-list-modules/vendor/example.com/other2/used/file.go"
  ]
  Imports: [
  ]
}
Package "example.com/m/subdir" {
  Name: "subdir"
  PkgPath: "example.com/m/subdir"
  Module: "example.com/m"
  ModuleDir: "/tmp/m/go-list-modules"
  GoFiles: [
"/tmp/m/go-list-modules/subdir/file.go"
  ]
  Imports: [
"example.com/m/submod/used"
"example.com/other1/used"
"example.com/other2/used"
  ]
}
```

Also note that `Module` info is missing for the ones it does find.

So I guess I am looking for some magical incantation that lets me
process and load all of the files across multiple modules exactly
once.  I wrote a similar program against go/build which runs in < 3
seconds, but doesn't work properly outside of a GOPATH (not sure why).

Any clues on how to approach this better would be well appreciated.

Tim

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAO_RewakNiSEKk6uBFjcGbQP0AgxqT0tDcyWsY5JTz_NaZDn4A%40mail.gmail.com.

Re: [go-nuts] go install touches the result file even if no changes

['Tim Hockin' via golang-nuts]

If you want to see why people give me a hard time:
https://github.com/kubernetes/kubernetes/blob/master/build/root/Makefile.generated_files

And that's AFTER a massive simplification.

On Mon, Jan 28, 2019 at 11:55 AM Ian Lance Taylor  wrote:
>
> On Mon, Jan 28, 2019 at 11:44 AM Tim Hockin  wrote:
> >
> > Of course you are right, as usual.  I just felt like that was a lot of
> > hoop-jumping.  People already chide me for my affinity for baroque
> > Makefiles...
>
> If you can assume that you are using GNU make, the Makefile language
> is Turing complete (although some limits are applied to rule expansion
> so you can't write an infinite loop).
>
> Ian
>
>
> > On Sat, Jan 26, 2019 at 11:01 AM Ian Lance Taylor  wrote:
> > >
> > > On Fri, Jan 25, 2019 at 7:55 PM Tim Hockin  wrote:
> > > >
> > > > Fair point, of course.
> > > >
> > > > I care because Kubernetes and it's family of projects have Makefiles to 
> > > > encapsulate trickier aspects of building, including code generation.  
> > > > Compiling kubernetes takes a LONG time.  It would be nice to avoid 
> > > > re-triggering secondary actions when the primary artifacts have not 
> > > > changed.
> > > >
> > > > Could I checksum?  Sure, but then I am writing a custom builder, so I 
> > > > might as well use Bazel (which has other issues).
> > > >
> > > > It's not a huge deal, today, but I really wanted to understand it.  It 
> > > > just seemed broken.
> > >
> > > While understanding that the approach is second best, Makefiles are
> > > fully able to handle files that have the same contents but updated
> > > timestamps.  You separate the file you create and the file you use.
> > > Then you write a Makefile target like
> > >
> > > progtouse: stmp-progtouse; @true  # Yes, the "; @true" is important.
> > > stmp-progtouse: dependencies to build progtobuild
> > > # commands to build progtobuild
> > > $(SHELL) mvifdiff.sh progtobuild progtouse
> > > echo >$@
> > >
> > > The shell script mvifdiff.sh is simply
> > >
> > > if cmp -s "$1" "$2" ; then
> > >   rm -f -- "$1"
> > > else
> > >   mv -f -- "$1" "$2"
> > > fi
> > >
> > > Then everything else depends on, and uses, progtouse.
> > >
> > > Remember to remove stmp-progtouse in your clean targets.
> > >
> > > Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] go install touches the result file even if no changes

['Tim Hockin' via golang-nuts]

Of course you are right, as usual.  I just felt like that was a lot of
hoop-jumping.  People already chide me for my affinity for baroque
Makefiles...

On Sat, Jan 26, 2019 at 11:01 AM Ian Lance Taylor  wrote:
>
> On Fri, Jan 25, 2019 at 7:55 PM Tim Hockin  wrote:
> >
> > Fair point, of course.
> >
> > I care because Kubernetes and it's family of projects have Makefiles to 
> > encapsulate trickier aspects of building, including code generation.  
> > Compiling kubernetes takes a LONG time.  It would be nice to avoid 
> > re-triggering secondary actions when the primary artifacts have not changed.
> >
> > Could I checksum?  Sure, but then I am writing a custom builder, so I might 
> > as well use Bazel (which has other issues).
> >
> > It's not a huge deal, today, but I really wanted to understand it.  It just 
> > seemed broken.
>
> While understanding that the approach is second best, Makefiles are
> fully able to handle files that have the same contents but updated
> timestamps.  You separate the file you create and the file you use.
> Then you write a Makefile target like
>
> progtouse: stmp-progtouse; @true  # Yes, the "; @true" is important.
> stmp-progtouse: dependencies to build progtobuild
> # commands to build progtobuild
> $(SHELL) mvifdiff.sh progtobuild progtouse
> echo >$@
>
> The shell script mvifdiff.sh is simply
>
> if cmp -s "$1" "$2" ; then
>   rm -f -- "$1"
> else
>   mv -f -- "$1" "$2"
> fi
>
> Then everything else depends on, and uses, progtouse.
>
> Remember to remove stmp-progtouse in your clean targets.
>
> Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] go install touches the result file even if no changes

['Tim Hockin' via golang-nuts]

On Fri, Jan 25, 2019 at 11:43 PM Paul Jolly  wrote:
>
> Tim - in case it's of any interest, I am in the process of (re)writing
> a dependency-aware wrapper around go generate that caches results in
> an artefact cache (i.e. only re-runs code generation as required).

Yes interested.  Happy to talk about what we do today to be as
on-demand as possible.

> On Sat, 26 Jan 2019 at 03:56, 'Tim Hockin' via golang-nuts
>  wrote:
> >
> > Fair point, of course.
> >
> > I care because Kubernetes and it's family of projects have Makefiles to 
> > encapsulate trickier aspects of building, including code generation.  
> > Compiling kubernetes takes a LONG time.  It would be nice to avoid 
> > re-triggering secondary actions when the primary artifacts have not changed.
> >
> > Could I checksum?  Sure, but then I am writing a custom builder, so I might 
> > as well use Bazel (which has other issues).
> >
> > It's not a huge deal, today, but I really wanted to understand it.  It just 
> > seemed broken.
> >
> > On Fri, Jan 25, 2019, 5:41 PM Ian Lance Taylor  wrote:
> >>
> >> On Fri, Jan 25, 2019 at 3:51 PM Tim Hockin  wrote:
> >> >
> >> > I don't grok that reasoning - can you expand on it?  Assume for a
> >> > second that it did NOT update mtime if the result did not change.  I
> >> > can be confident that same mtime == no change, right?  It doesn't
> >> > imply that different mtime == something change, but I think that's OK
> >> > (for my use, maybe I am limited in my imagination). Even
> >> >
> >> > Does it really matter if some corner cases result in spurious updates?
> >>
> >> It would be nice if it worked that way, but I'm confident that if we
> >> avoided updating mtime when we knew the file did not change, and then
> >> later started updating mtime again, people would file bugs saying that
> >> the mtime was updated incorrectly.  Right now we provide a simple API:
> >> run "go build" or "go install" and your executable will be up to date.
> >> Why do we need a more complex API?
> >>
> >> Let me turn it around: why do you care?  For cases where you do care,
> >> could you instead keep a hash of the file contents?  For what it's
> >> worth, you can fetch a hash of a Go program by running "go tool
> >> buildid PROGRAM".  See https://golang.org/cmd/buildid.
> >>
> >> Ian
> >>
> >>
> >>
> >> > On Fri, Jan 25, 2019 at 3:02 PM Ian Lance Taylor  wrote:
> >> > >
> >> > > On Fri, Jan 25, 2019 at 1:07 PM 'Tim Hockin' via golang-nuts
> >> > >  wrote:
> >> > > >
> >> > > > Example:
> >> > > >
> >> > > > ```
> >> > > > $ which git-sync
> >> > > >
> >> > > > $ go install -installsuffix "static" ./cmd/git-sync/
> >> > > >
> >> > > > $ ls -l --full-time `which git-sync`; md5sum `which git-sync`
> >> > > > -rwxr-xr-x 1 thockin primarygroup 13956902 2019-01-25
> >> > > > 13:04:40.758632955 -0800
> >> > > > /usr/local/google/home/thockin/src/go/bin/git-sync
> >> > > > 1200f479c8ba86f70f0e4a885ecdd5f2
> >> > > > /usr/local/google/home/thockin/src/go/bin/git-sync
> >> > > >
> >> > > > $ go install -installsuffix "static" ./cmd/git-sync/
> >> > > >
> >> > > > $ ls -l --full-time `which git-sync`; md5sum `which git-sync`
> >> > > > -rwxr-xr-x 1 thockin primarygroup 13956902 2019-01-25
> >> > > > 13:04:53.817700697 -0800
> >> > > > /usr/local/google/home/thockin/src/go/bin/git-sync
> >> > > > 1200f479c8ba86f70f0e4a885ecdd5f2
> >> > > > /usr/local/google/home/thockin/src/go/bin/git-sync
> >> > > > ```
> >> > > >
> >> > > > Is the desired behavior or just a side-effect?  Is there any way to
> >> > > > defeat it?  Would a patch for this be shot down?
> >> > >
> >> > > This is intended behavior.  The comment in the code
> >> > > (https://golang.org/src/cmd/go/internal/work/exec.go) is
> >> > >
> >> > > // Whether we're smart enough to avoid a complete rebuild
> >> > > // depends on exactly what the staleness and rebuild algorithms
> >> > > // are, as well as potentially the state of th

Re: [go-nuts] go install touches the result file even if no changes

['Tim Hockin' via golang-nuts]

Fair point, of course.

I care because Kubernetes and it's family of projects have Makefiles to
encapsulate trickier aspects of building, including code generation.
Compiling kubernetes takes a LONG time.  It would be nice to avoid
re-triggering secondary actions when the primary artifacts have not changed.

Could I checksum?  Sure, but then I am writing a custom builder, so I might
as well use Bazel (which has other issues).

It's not a huge deal, today, but I really wanted to understand it.  It just
seemed broken.

On Fri, Jan 25, 2019, 5:41 PM Ian Lance Taylor  wrote:

> On Fri, Jan 25, 2019 at 3:51 PM Tim Hockin  wrote:
> >
> > I don't grok that reasoning - can you expand on it?  Assume for a
> > second that it did NOT update mtime if the result did not change.  I
> > can be confident that same mtime == no change, right?  It doesn't
> > imply that different mtime == something change, but I think that's OK
> > (for my use, maybe I am limited in my imagination). Even
> >
> > Does it really matter if some corner cases result in spurious updates?
>
> It would be nice if it worked that way, but I'm confident that if we
> avoided updating mtime when we knew the file did not change, and then
> later started updating mtime again, people would file bugs saying that
> the mtime was updated incorrectly.  Right now we provide a simple API:
> run "go build" or "go install" and your executable will be up to date.
> Why do we need a more complex API?
>
> Let me turn it around: why do you care?  For cases where you do care,
> could you instead keep a hash of the file contents?  For what it's
> worth, you can fetch a hash of a Go program by running "go tool
> buildid PROGRAM".  See https://golang.org/cmd/buildid.
>
> Ian
>
>
>
> > On Fri, Jan 25, 2019 at 3:02 PM Ian Lance Taylor 
> wrote:
> > >
> > > On Fri, Jan 25, 2019 at 1:07 PM 'Tim Hockin' via golang-nuts
> > >  wrote:
> > > >
> > > > Example:
> > > >
> > > > ```
> > > > $ which git-sync
> > > >
> > > > $ go install -installsuffix "static" ./cmd/git-sync/
> > > >
> > > > $ ls -l --full-time `which git-sync`; md5sum `which git-sync`
> > > > -rwxr-xr-x 1 thockin primarygroup 13956902 2019-01-25
> > > > 13:04:40.758632955 -0800
> > > > /usr/local/google/home/thockin/src/go/bin/git-sync
> > > > 1200f479c8ba86f70f0e4a885ecdd5f2
> > > > /usr/local/google/home/thockin/src/go/bin/git-sync
> > > >
> > > > $ go install -installsuffix "static" ./cmd/git-sync/
> > > >
> > > > $ ls -l --full-time `which git-sync`; md5sum `which git-sync`
> > > > -rwxr-xr-x 1 thockin primarygroup 13956902 2019-01-25
> > > > 13:04:53.817700697 -0800
> > > > /usr/local/google/home/thockin/src/go/bin/git-sync
> > > > 1200f479c8ba86f70f0e4a885ecdd5f2
> > > > /usr/local/google/home/thockin/src/go/bin/git-sync
> > > > ```
> > > >
> > > > Is the desired behavior or just a side-effect?  Is there any way to
> > > > defeat it?  Would a patch for this be shot down?
> > >
> > > This is intended behavior.  The comment in the code
> > > (https://golang.org/src/cmd/go/internal/work/exec.go) is
> > >
> > > // Whether we're smart enough to avoid a complete rebuild
> > > // depends on exactly what the staleness and rebuild algorithms
> > > // are, as well as potentially the state of the Go build cache.
> > > // We don't really want users to be able to infer (or worse start
> depending on)
> > > // those details from whether the modification time changes during
> > > // "go install", so do a best-effort update of the file times to make
> it
> > > // look like we rewrote a.Target even if we did not. Updating the mtime
> > > // may also help other mtime-based systems that depend on our
> > > // previous mtime updates that happened more often.
> > > // This is still not perfect - we ignore the error result, and if the
> file was
> > > // unwritable for some reason then pretending to have written it is
> also
> > > // confusing - but it's probably better than not doing the mtime
> update.
> > > //
> > > // But don't do that for the special case where building an executable
> > > // with -linkshared implicitly installs all its dependent libraries.
> > > // We want to hide that awful detail as much as possible, so don't
> > > // advertise it by touching the mtimes (usually the libraries are up
> > > // to date).
> > >
> > > Ian
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] go install touches the result file even if no changes

['Tim Hockin' via golang-nuts]

I don't grok that reasoning - can you expand on it?  Assume for a
second that it did NOT update mtime if the result did not change.  I
can be confident that same mtime == no change, right?  It doesn't
imply that different mtime == something change, but I think that's OK
(for my use, maybe I am limited in my imagination). Even

Does it really matter if some corner cases result in spurious updates?

On Fri, Jan 25, 2019 at 3:02 PM Ian Lance Taylor  wrote:
>
> On Fri, Jan 25, 2019 at 1:07 PM 'Tim Hockin' via golang-nuts
>  wrote:
> >
> > Example:
> >
> > ```
> > $ which git-sync
> >
> > $ go install -installsuffix "static" ./cmd/git-sync/
> >
> > $ ls -l --full-time `which git-sync`; md5sum `which git-sync`
> > -rwxr-xr-x 1 thockin primarygroup 13956902 2019-01-25
> > 13:04:40.758632955 -0800
> > /usr/local/google/home/thockin/src/go/bin/git-sync
> > 1200f479c8ba86f70f0e4a885ecdd5f2
> > /usr/local/google/home/thockin/src/go/bin/git-sync
> >
> > $ go install -installsuffix "static" ./cmd/git-sync/
> >
> > $ ls -l --full-time `which git-sync`; md5sum `which git-sync`
> > -rwxr-xr-x 1 thockin primarygroup 13956902 2019-01-25
> > 13:04:53.817700697 -0800
> > /usr/local/google/home/thockin/src/go/bin/git-sync
> > 1200f479c8ba86f70f0e4a885ecdd5f2
> > /usr/local/google/home/thockin/src/go/bin/git-sync
> > ```
> >
> > Is the desired behavior or just a side-effect?  Is there any way to
> > defeat it?  Would a patch for this be shot down?
>
> This is intended behavior.  The comment in the code
> (https://golang.org/src/cmd/go/internal/work/exec.go) is
>
> // Whether we're smart enough to avoid a complete rebuild
> // depends on exactly what the staleness and rebuild algorithms
> // are, as well as potentially the state of the Go build cache.
> // We don't really want users to be able to infer (or worse start depending 
> on)
> // those details from whether the modification time changes during
> // "go install", so do a best-effort update of the file times to make it
> // look like we rewrote a.Target even if we did not. Updating the mtime
> // may also help other mtime-based systems that depend on our
> // previous mtime updates that happened more often.
> // This is still not perfect - we ignore the error result, and if the file was
> // unwritable for some reason then pretending to have written it is also
> // confusing - but it's probably better than not doing the mtime update.
> //
> // But don't do that for the special case where building an executable
> // with -linkshared implicitly installs all its dependent libraries.
> // We want to hide that awful detail as much as possible, so don't
> // advertise it by touching the mtimes (usually the libraries are up
> // to date).
>
> Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] go install touches the result file even if no changes

['Tim Hockin' via golang-nuts]

Example:

```
$ which git-sync

$ go install -installsuffix "static" ./cmd/git-sync/

$ ls -l --full-time `which git-sync`; md5sum `which git-sync`
-rwxr-xr-x 1 thockin primarygroup 13956902 2019-01-25
13:04:40.758632955 -0800
/usr/local/google/home/thockin/src/go/bin/git-sync
1200f479c8ba86f70f0e4a885ecdd5f2
/usr/local/google/home/thockin/src/go/bin/git-sync

$ go install -installsuffix "static" ./cmd/git-sync/

$ ls -l --full-time `which git-sync`; md5sum `which git-sync`
-rwxr-xr-x 1 thockin primarygroup 13956902 2019-01-25
13:04:53.817700697 -0800
/usr/local/google/home/thockin/src/go/bin/git-sync
1200f479c8ba86f70f0e4a885ecdd5f2
/usr/local/google/home/thockin/src/go/bin/git-sync
```

Is the desired behavior or just a side-effect?  Is there any way to
defeat it?  Would a patch for this be shot down?

Tim

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: import paths - slash vs backslash

['Tim Hockin' via golang-nuts]

Pinging this topic - does anyone know?

On Fri, Nov 10, 2017 at 10:30 AM, Tim Hockin  wrote:
> Is it valid to say `import "github.com\foo\bar" (windows path
> separators) or must it be "github.com/foo/bar"?  I couldn't find a
> canonical answer to this.
>
> Thanks
>
> Tim

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] import paths - slash vs backslash

['Tim Hockin' via golang-nuts]

Is it valid to say `import "github.com\foo\bar" (windows path
separators) or must it be "github.com/foo/bar"?  I couldn't find a
canonical answer to this.

Thanks

Tim

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] 1.8 plugins and ABI

['Tim Hockin' via golang-nuts]

Thanks, Ian.  Very informative if somewhat disappointing.  It more or less
rules out what we wanted to do.


On Feb 23, 2017 5:23 PM, "Ian Lance Taylor" <i...@golang.org> wrote:

On Thu, Feb 23, 2017 at 4:45 PM, 'Tim Hockin' via golang-nuts
<golang-nuts@googlegroups.com> wrote:
>
> I can't seem to find this documented, and I'm me involved in some
> conversations about using plugins:
>
> What guarantees or limitations surround plugins?  Can I compile a host
> program with go-1.8.0 and have a third-party plugin compiled with 1.8.1,
and
> load it?
>
> Can I call methods on plugin-specific objects?
>
> Are there any concessions toward versioning?
>
> What happens if I add a method to a type in the host, but don't rebuild he
> plugin?
>
> Are functions/method calling conventions guaranteed to be consistent, or
do
> compiler flags change that?

Plugins are definitely in an early stage, and there are many reported
bugs with using them in 1.8.

Plugins must be built with the exact same version of Go as is used to
build the main program.  If any package appears in both the main
program and a plugin, or in two loaded plugins, that package must be
built from the exact same source code in all cases.  There are no
current plans to lift these restrictions.  I believe these
restrictions are checked at run time, though there may be bugs.

You can call methods on types defined in a plugin, yes.  You'll
presumably be doing that using some sort of interface type.

There is nothing about versioning in the plugin package.  I don't
think anybody even knows what it would look like.

There are no compiler flags to change the calling convention.  If
there were any, they should cause a failure to load the plugin at run
time.

Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] 1.8 plugins and ABI

['Tim Hockin' via golang-nuts]

Hi all,

I can't seem to find this documented, and I'm me involved in some
conversations about using plugins:

What guarantees or limitations surround plugins?  Can I compile a host
program with go-1.8.0 and have a third-party plugin compiled with 1.8.1,
and load it?

Can I call methods on plugin-specific objects?

Are there any concessions toward versioning?

What happens if I add a method to a type in the host, but don't rebuild he
plugin?

Are functions/method calling conventions guaranteed to be consistent, or do
compiler flags change that?

Thanks!

Tim

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] timer Reset concurrent to receive

['Tim Hockin' via golang-nuts]

I'm not going to debate it since I am not really volunteering to fix
the docs right now, anyway :)  My sense is that when I want to know
how to use something, I run `go doc` on it.  The doc as it stands says
"don't do it" when the reality has an extra "unless ..." clause.

Thanks for the guidance!

On Wed, Jan 25, 2017 at 11:47 AM, Ian Lance Taylor <i...@golang.org> wrote:
> On Wed, Jan 25, 2017 at 11:32 AM, Tim Hockin <thoc...@google.com> wrote:
>> Better to have a documented race than a potential hang, though, yeah?
>> If it delivers and then I Stop(), drain, Reset(), I might lose the
>> race, have the channel read by the concurrent receiver, and and just
>> block here.  Or am I missing some other nuance here?
>
> The docs right now say: don't do this with a concurrent receiver.  You
> are talking about the case where there is a concurrent receiver.  I
> agree that that case is more complicated and requires further details,
> I'm just not sure the docs for this package are the place to discuss
> them.
>
> Ian
>
>> On Wed, Jan 25, 2017 at 11:21 AM, Ian Lance Taylor <i...@golang.org> wrote:
>>> On Wed, Jan 25, 2017 at 10:30 AM, Tim Hockin <thoc...@google.com> wrote:
>>>> Thanks!  That makes sense.  Does it make sense to update the docs to
>>>> show the "select-with-default" mode of draining the channel instead?
>>>
>>> I guess I don't think so, as there is still a potential race with the
>>> other receive.  I mean, we can make the docs arbitrarily complicated,
>>> but at some point it should be on the wiki or something.
>>>
>>> Ian
>>>
>>>> On Wed, Jan 25, 2017 at 10:20 AM, Ian Lance Taylor <i...@golang.org> wrote:
>>>>> On Wed, Jan 25, 2017 at 9:34 AM, 'Tim Hockin' via golang-nuts
>>>>> <golang-nuts@googlegroups.com> wrote:
>>>>>> I'm not convinced that the docs quite cover the case I am looking, so
>>>>>> I am posting here.
>>>>>>
>>>>>> https://golang.org/pkg/time/#Timer.Reset says "This should not be done
>>>>>> concurrent to other receives from the Timer's channel" but it's not
>>>>>> clear what the repercussions are.
>>>>>>
>>>>>> In our case, I have a function to be run periodically, on a timer, but
>>>>>> it can be run manually too.  When run manually, I want to push the
>>>>>> timer out (restart the period).
>>>>>>
>>>>>> I have a goroutine doing:
>>>>>>
>>>>>> ```
>>>>>> for {
>>>>>> select {
>>>>>> case <-stop:
>>>>>> pr.stop()
>>>>>> return
>>>>>> case <-timer.C:
>>>>>> run()
>>>>>> }
>>>>>> }
>>>>>> ```
>>>>>>
>>>>>> deep inside run(), we have:
>>>>>>
>>>>>> ```
>>>>>> timer.Stop()
>>>>>> timer.Reset(period)
>>>>>> ```
>>>>>>
>>>>>> I understand that I could lose the race and deliver on timer.C _just
>>>>>> before_ this runs, and that is fine.  What I am seeking to know is
>>>>>> whether this is considered "safe"?  The receive is running
>>>>>> concurrently to the Reset().  Will this cause problems inside Timer,
>>>>>> beyond the potential "extra" delivery?  Do I need to break the loop
>>>>>> and stop receiving on it while the Reset() happens?
>>>>>
>>>>> The sentence "This should not be done concurrent to other receives
>>>>> from the Timer's channel." is intended to apply to the description of
>>>>> how to use t.Stop safely.  It's there because if you use the code
>>>>> fragment described there and there is a concurrent receive, you don't
>>>>> know which channel receive will succeed.  In other words, the channel
>>>>> receive in the code fragment might hang.
>>>>>
>>>>> It is safe to use Reset as you describe, as long as you understand
>>>>> that the timer may expire, and send a value to the channel, as you
>>>>> call Stop and Reset.  If you don't care about that--if an extra value
>>>>> sent to the channel doesn't matter--then your code is fine.
>>>>>
>>>>> Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] timer Reset concurrent to receive

['Tim Hockin' via golang-nuts]

Better to have a documented race than a potential hang, though, yeah?
If it delivers and then I Stop(), drain, Reset(), I might lose the
race, have the channel read by the concurrent receiver, and and just
block here.  Or am I missing some other nuance here?

On Wed, Jan 25, 2017 at 11:21 AM, Ian Lance Taylor <i...@golang.org> wrote:
> On Wed, Jan 25, 2017 at 10:30 AM, Tim Hockin <thoc...@google.com> wrote:
>> Thanks!  That makes sense.  Does it make sense to update the docs to
>> show the "select-with-default" mode of draining the channel instead?
>
> I guess I don't think so, as there is still a potential race with the
> other receive.  I mean, we can make the docs arbitrarily complicated,
> but at some point it should be on the wiki or something.
>
> Ian
>
>> On Wed, Jan 25, 2017 at 10:20 AM, Ian Lance Taylor <i...@golang.org> wrote:
>>> On Wed, Jan 25, 2017 at 9:34 AM, 'Tim Hockin' via golang-nuts
>>> <golang-nuts@googlegroups.com> wrote:
>>>> I'm not convinced that the docs quite cover the case I am looking, so
>>>> I am posting here.
>>>>
>>>> https://golang.org/pkg/time/#Timer.Reset says "This should not be done
>>>> concurrent to other receives from the Timer's channel" but it's not
>>>> clear what the repercussions are.
>>>>
>>>> In our case, I have a function to be run periodically, on a timer, but
>>>> it can be run manually too.  When run manually, I want to push the
>>>> timer out (restart the period).
>>>>
>>>> I have a goroutine doing:
>>>>
>>>> ```
>>>> for {
>>>> select {
>>>> case <-stop:
>>>> pr.stop()
>>>> return
>>>> case <-timer.C:
>>>> run()
>>>> }
>>>> }
>>>> ```
>>>>
>>>> deep inside run(), we have:
>>>>
>>>> ```
>>>> timer.Stop()
>>>> timer.Reset(period)
>>>> ```
>>>>
>>>> I understand that I could lose the race and deliver on timer.C _just
>>>> before_ this runs, and that is fine.  What I am seeking to know is
>>>> whether this is considered "safe"?  The receive is running
>>>> concurrently to the Reset().  Will this cause problems inside Timer,
>>>> beyond the potential "extra" delivery?  Do I need to break the loop
>>>> and stop receiving on it while the Reset() happens?
>>>
>>> The sentence "This should not be done concurrent to other receives
>>> from the Timer's channel." is intended to apply to the description of
>>> how to use t.Stop safely.  It's there because if you use the code
>>> fragment described there and there is a concurrent receive, you don't
>>> know which channel receive will succeed.  In other words, the channel
>>> receive in the code fragment might hang.
>>>
>>> It is safe to use Reset as you describe, as long as you understand
>>> that the timer may expire, and send a value to the channel, as you
>>> call Stop and Reset.  If you don't care about that--if an extra value
>>> sent to the channel doesn't matter--then your code is fine.
>>>
>>> Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] timer Reset concurrent to receive

['Tim Hockin' via golang-nuts]

Thanks!  That makes sense.  Does it make sense to update the docs to
show the "select-with-default" mode of draining the channel instead?

On Wed, Jan 25, 2017 at 10:20 AM, Ian Lance Taylor <i...@golang.org> wrote:
> On Wed, Jan 25, 2017 at 9:34 AM, 'Tim Hockin' via golang-nuts
> <golang-nuts@googlegroups.com> wrote:
>> I'm not convinced that the docs quite cover the case I am looking, so
>> I am posting here.
>>
>> https://golang.org/pkg/time/#Timer.Reset says "This should not be done
>> concurrent to other receives from the Timer's channel" but it's not
>> clear what the repercussions are.
>>
>> In our case, I have a function to be run periodically, on a timer, but
>> it can be run manually too.  When run manually, I want to push the
>> timer out (restart the period).
>>
>> I have a goroutine doing:
>>
>> ```
>> for {
>> select {
>> case <-stop:
>> pr.stop()
>> return
>> case <-timer.C:
>> run()
>> }
>> }
>> ```
>>
>> deep inside run(), we have:
>>
>> ```
>> timer.Stop()
>> timer.Reset(period)
>> ```
>>
>> I understand that I could lose the race and deliver on timer.C _just
>> before_ this runs, and that is fine.  What I am seeking to know is
>> whether this is considered "safe"?  The receive is running
>> concurrently to the Reset().  Will this cause problems inside Timer,
>> beyond the potential "extra" delivery?  Do I need to break the loop
>> and stop receiving on it while the Reset() happens?
>
> The sentence "This should not be done concurrent to other receives
> from the Timer's channel." is intended to apply to the description of
> how to use t.Stop safely.  It's there because if you use the code
> fragment described there and there is a concurrent receive, you don't
> know which channel receive will succeed.  In other words, the channel
> receive in the code fragment might hang.
>
> It is safe to use Reset as you describe, as long as you understand
> that the timer may expire, and send a value to the channel, as you
> call Stop and Reset.  If you don't care about that--if an extra value
> sent to the channel doesn't matter--then your code is fine.
>
> Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] `go test -i` considered helpful?

['Tim Hockin' via golang-nuts]

What's weird to me is that go test -i doesn't actually run tests.  Maybe go
test -I which does both?

On Sep 1, 2016 11:41 AM, "Ian Lance Taylor" <i...@golang.org> wrote:

> On Thu, Sep 1, 2016 at 11:24 AM, 'Tim Hockin' via golang-nuts
> <golang-nuts@googlegroups.com> wrote:
> > I was running tests against a new project, and it was really slow.
> > When I ran with -x I saw a bunch of build steps that didn't seem to be
> > necessary.  Spelunking through docs, reveals that `go test` doesn't
> > install artifacts.  `go test -i` does install artifacts but doesn't
> > run the tests.
> >
> > So I changed my script to always `go test -i` followed by `go test`
> > and lo, my tests run in 20% of the time.
> >
> > Am I doing something wrong?  This feels really hacky - is it correct?
>
> That is indeed how it works today.  Usually I'm just working on one
> package, so I run `go test -i` once for that package and then don't
> worry about it until I'm working on a different package.
>
> I'm not sure how best to change this.  It would seem weird for `go
> test` to install packages.
>
> Ian
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] `go test -i` considered helpful?

['Tim Hockin' via golang-nuts]

I was running tests against a new project, and it was really slow.
When I ran with -x I saw a bunch of build steps that didn't seem to be
necessary.  Spelunking through docs, reveals that `go test` doesn't
install artifacts.  `go test -i` does install artifacts but doesn't
run the tests.

So I changed my script to always `go test -i` followed by `go test`
and lo, my tests run in 20% of the time.

Am I doing something wrong?  This feels really hacky - is it correct?

Tim

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] runtime.Caller ->

['Tim Hockin' via golang-nuts]

OK, is there ANY heuristic I can rely on find the "real" call frame?

On Mon, Aug 15, 2016 at 10:17 AM, Ian Lance Taylor  wrote:
> On Sun, Aug 14, 2016 at 10:07 PM, Tim Hockin  wrote:
>> Can I rely on "" not changing?
>
> I'm sorry, that's a hard question to answer, because other compilers
> do not use that string.
>
> There are no plans to change that string for the gc toolchain.
>
> Ian
>
>> On Sun, Aug 14, 2016 at 9:55 PM, Ian Lance Taylor  wrote:
>>> On Sun, Aug 14, 2016 at 9:41 PM, Tim Hockin  wrote:
 On Sun, Aug 14, 2016 at 8:31 PM, Ian Lance Taylor  wrote:
> On Sun, Aug 14, 2016 at 3:33 PM, Tim Hockin  wrote:
>> Edit:  It looks like this has more to do with being an interface
>> method than an embedded type.
>>
>> https://play.golang.org/p/I5XPdWR_O0
>
> Hmmm, you're right.  It only happens for a value method.

 Is this likely to change? I.e. can I hardcode "2" or should I actually
 write the loop to climb frames?  Is there a limit to the number of
 frames I should inspect before I give up?  Is the string
 "" stable?
>>>
>>> Well, unfortunately, it's different for different compilers.  I don't
>>> have a good answer here.  Except to say that you should never need
>>> more than 2 frames; it should never be the case that autogenerated
>>> code calls autogenerated code.
>>>
>>> Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] runtime.Caller ->

['Tim Hockin' via golang-nuts]

On Sun, Aug 14, 2016 at 8:31 PM, Ian Lance Taylor  wrote:
> On Sun, Aug 14, 2016 at 3:33 PM, Tim Hockin  wrote:
>> Edit:  It looks like this has more to do with being an interface
>> method than an embedded type.
>>
>> https://play.golang.org/p/I5XPdWR_O0
>
> Hmmm, you're right.  It only happens for a value method.

Is this likely to change? I.e. can I hardcode "2" or should I actually
write the loop to climb frames?  Is there a limit to the number of
frames I should inspect before I give up?  Is the string
"" stable?

>> Is there a generic way to get the caller safely, or should I just
>> check for  and increment?
>
> I think that is what you should do here.
>
> Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] runtime.Caller ->

['Tim Hockin' via golang-nuts]

Edit:  It looks like this has more to do with being an interface
method than an embedded type.

https://play.golang.org/p/I5XPdWR_O0

Is there a generic way to get the caller safely, or should I just
check for  and increment?

On Sun, Aug 14, 2016 at 3:02 PM, Tim Hockin <thoc...@google.com> wrote:
> It is, in fact, a method on an embedded type.  That means I can document it
> and it ceases to be a magic number!  Thanks.
>
>
> On Aug 14, 2016 12:27 PM, "Ian Lance Taylor" <i...@golang.org> wrote:
>>
>> On Sun, Aug 14, 2016 at 11:10 AM, 'Tim Hockin' via golang-nuts
>> <golang-nuts@googlegroups.com> wrote:
>> > I was playing with a method that wants to print the file:line of the
>> > caller.  `runtime.Caller(1)` seems like it should do the trick, but I
>> > get ':2'.  If I call `runtime.Caller(2)` instead it
>> > seems to work, but I don't know why, so I am reluctant to depend on
>> > that.
>> >
>> > Can anyone enlighten me?  How can I know how many frames to crawl back
>> > to my real caller?
>> >
>> > go1.6.1 on linux/amd64
>>
>> An `autogenerated` frame means that you are calling a method that is
>> defined on some type T1 that is embedded in some other type T2.  The
>> `autogenerated` code is the little wrapper that takes a value of type
>> T2 and invokes the method on the embedded field of type T1.  As far as
>> I know you will never see autogenerated code called by autogenerated
>> code.  So you only need to worry about this in a method of a type that
>> your program will embed into some other type.  If you don't know at
>> run time whether the type is embedded or not, call runtime.Caller(1),
>> and if you see autogenerated call runtime.Caller(2).
>>
>> Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] runtime.Caller ->

['Tim Hockin' via golang-nuts]

It is, in fact, a method on an embedded type.  That means I can document it
and it ceases to be a magic number!  Thanks.

On Aug 14, 2016 12:27 PM, "Ian Lance Taylor" <i...@golang.org> wrote:

> On Sun, Aug 14, 2016 at 11:10 AM, 'Tim Hockin' via golang-nuts
> <golang-nuts@googlegroups.com> wrote:
> > I was playing with a method that wants to print the file:line of the
> > caller.  `runtime.Caller(1)` seems like it should do the trick, but I
> > get ':2'.  If I call `runtime.Caller(2)` instead it
> > seems to work, but I don't know why, so I am reluctant to depend on
> > that.
> >
> > Can anyone enlighten me?  How can I know how many frames to crawl back
> > to my real caller?
> >
> > go1.6.1 on linux/amd64
>
> An `autogenerated` frame means that you are calling a method that is
> defined on some type T1 that is embedded in some other type T2.  The
> `autogenerated` code is the little wrapper that takes a value of type
> T2 and invokes the method on the embedded field of type T1.  As far as
> I know you will never see autogenerated code called by autogenerated
> code.  So you only need to worry about this in a method of a type that
> your program will embed into some other type.  If you don't know at
> run time whether the type is embedded or not, call runtime.Caller(1),
> and if you see autogenerated call runtime.Caller(2).
>
> Ian
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] runtime.Caller ->

['Tim Hockin' via golang-nuts]

I was playing with a method that wants to print the file:line of the
caller.  `runtime.Caller(1)` seems like it should do the trick, but I
get ':2'.  If I call `runtime.Caller(2)` instead it
seems to work, but I don't know why, so I am reluctant to depend on
that.

Can anyone enlighten me?  How can I know how many frames to crawl back
to my real caller?

go1.6.1 on linux/amd64

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.