Hi all. I realized that https://blog.golang.org/go116-module-changes says "We plan to drop support for GOPATH mode in Go 1.17. In other words, Go 1.17 will ignore GO111MODULE". This will break Kubernetes - our builds will be orders of magnitude slower.
AFAICT, https://github.com/golang/go/issues/43806 has had no progress and https://github.com/golang/go/issues/43733 is still open, too. I have zero confidence that these are the ONLY issues, and until I can get past them, I can't really find out. Please advise - how should I proceed? Tim On Wed, Jan 20, 2021 at 12:57 PM Tim Hockin <thoc...@google.com> wrote: > As long as these things are bugs and not "that's not how it works or will > ever work", I have some hope. This is a back-burner exploration for me, so > I am not in a panic. Every now and again I will circle back and see if > anything new happens :) > > On Wed, Jan 20, 2021 at 10:50 AM Jay Conrod <jaycon...@google.com> wrote: > >> You appear to have discovered another new bug. Sorry for that. I've >> opened #43806 <https://github.com/golang/go/issues/43806> to track it. >> >> With that bug, it doesn't look like `go list` with local paths (relative >> or absolute) in a repository with nested replacement directories will work >> (at least not yet). >> >> `go list` should still work on full packages paths, but I'm not sure >> there's an easy way to resolve directory paths to package paths, other than >> by walking up the directory tree to the nearest go.mod and reading the >> module path from there. >> >> On Tue, Jan 19, 2021 at 8:06 PM Tim Hockin <thoc...@google.com> wrote: >> >>> >>> >>> On Tue, Jan 19, 2021 at 7:54 AM Jay Conrod <jaycon...@google.com> wrote: >>> >>>> By the way, I'm not sure if you're already doing this, but if you can >>>> batch all of the `go list` runs (or go/packages >>>> <https://goto.google.com/packages>.Load) together per module (passing >>>> an argument per package), that will be much faster than loading individual >>>> packages with separate `go list` calls. It will save `go list` from having >>>> to load the module graph and common sets of dependencies multiple times. >>>> >>>> If you need to load all the packages in a module, you can also use an >>>> argument like ./... from the root directory of the module to load >>>> everything there (excluding modules in subdirectories). >>>> >>> >>> Yeah, I am trying that but tripping on the "does not contain package" >>> error. >>> >>> >>>> >>>> On Tue, Jan 19, 2021 at 10:02 AM Jay Conrod <jaycon...@google.com> >>>> wrote: >>>> >>>>> > Interesting - is the difference the absolute paths vs relative? >>>>> >>>>> It looks like the bug has to do with whether the directory is below >>>>> the main module root directory or not. If it is, the go command takes a >>>>> path that assumes it's part of the main module, which it's not. >>>>> >>>>> > I hoped maybe `-modfile` would do the same trick, but alas not: >>>>> >>>>> -modfile lets you change the effective content of go.mod but not the >>>>> module root directory. Unfortunately it doesn't look like that can be used >>>>> to work around the issue. >>>>> >>>>> > It seems that is because the "main" (top-level dir) go.mod has >>>>> > `replace` directives with relative paths, which kubernetes really >>>>> > does. >>>>> >>>>> You may need to copy those over to the tmp go.mod and adjust the >>>>> paths. Sorry this has gotten pretty involved. >>>>> >>>>> > Yeah, I noticed. When GO111MODULE=off, everything I am doing is much >>>>> > faster. I'm wary of depending on that forever, though. >>>>> >>>>> Module-aware mode is quite a bit more complicated than GOPATH mode, so >>>>> to some degree it's not surprising it's slower... it's surprising that >>>>> it's >>>>> a LOT slower though. I expect there's some optimization work for us to do >>>>> in the next development cycle. >>>>> >>>>> We would eventually like to deprecate GOPATH mode though, so it's a >>>>> good idea not to depend on it in new tooling today. 'go list' should be >>>>> fine to get package dependency info in either module mode or GOPATH mode. >>>>> go/packages <https://goto.google.com/packages> is useful if you need >>>>> additional information on top of that (parsed syntax trees, type info). >>>>> >>>>> > I want to run a slow codegen process only if the packages it depends >>>>> > on have ACTUALLY changed (mtime is a good enough proxy) and I don't >>>>> > know a priori which packages need codegen. I want to scan the file >>>>> > tree, find the files that need codegen, check their deps, and only >>>>> > then run the codegen. >>>>> >>>>> How much dependency info do you need? If the codegen is only within >>>>> packages with files that have changed, 'go list' might be overkill (it >>>>> always loads dependencies, even if they aren't printed). If you need >>>>> dependencies or reverse dependencies, 'go list' or go/packages >>>>> <https://goto.google.com/packages> are probably the right tools. >>>>> >>>>> On Fri, Jan 15, 2021 at 6:43 PM Tim Hockin <thoc...@google.com> wrote: >>>>> >>>>>> On Fri, Jan 15, 2021 at 2:17 PM Jay Conrod <jaycon...@google.com> >>>>>> wrote: >>>>>> > >>>>>> > I was initially going to suggest adding the module subdirectories >>>>>> as requirements to the main go.mod, then replacing those with the >>>>>> subdirectories. >>>>>> > >>>>>> > module example.com/m >>>>>> > >>>>>> > go 1.15 >>>>>> > >>>>>> > require ( >>>>>> > example.com/other1 v0.0.0 >>>>>> > example.com/other2 v0.0.0 >>>>>> > example.com/m/submod v0.0.0 >>>>>> > ) >>>>>> > >>>>>> > replace ( >>>>>> > example.com/other1 => ./staging/src/example.com/other1 >>>>>> > example.com/other2 => ./staging/src/example.com/other2 >>>>>> > example.com/m/submod v0.0.0 => ./submod >>>>>> > ) >>>>>> > >>>>>> > >>>>>> > I think you might have tried this already. It gives the same "main >>>>>> module ... does not contain package" error. I believe that's a bug. I've >>>>>> opened #43733 to track it. >>>>>> >>>>>> Interesting. If that's a bug, then maybe I'll be able to do what I >>>>>> need once fixed. >>>>>> >>>>>> > In general, it should be possible to give 'go list' an absolute or >>>>>> relative path (starting with ./ or ../) to any directory containing a >>>>>> package which is part of any module in the build list. For example, some >>>>>> tools list directories in the module cache to find out what package a .go >>>>>> file belongs to. >>>>>> > >>>>>> > As a workaround, you could put a go.mod in an otherwise empty >>>>>> directory (in /tmp or something), then require the relevant modules from >>>>>> the repo and replace them with absolute paths. Then you can run 'go list' >>>>>> in that directory with absolute paths of package directories. >>>>>> >>>>>> Interesting - is the difference the absolute paths vs relative? >>>>>> >>>>>> I hoped maybe `-modfile` would do the same trick, but alas not: >>>>>> >>>>>> ``` >>>>>> $ (cd /tmp/gomodhack/; go list /tmp/go-list-modules/submod/used/) >>>>>> example.com/m/submod/used >>>>>> >>>>>> $ go list --modfile /tmp/gomodhack/go.mod >>>>>> /tmp/go-list-modules/submod/used/ >>>>>> main module (tmp) does not contain package tmp/submod/used >>>>>> ``` >>>>>> >>>>>> It also fails some cases: >>>>>> >>>>>> ``` >>>>>> (cd /tmp/gomodhack/; go list /tmp/go-list-modules/submod/used/) >>>>>> example.com/m/submod/used >>>>>> thockin@thockin-glaptop4 go-list-modules main /$ (cd /tmp/gomodhack/; >>>>>> go list /tmp/go-list-modules/staging/src/example.com/other1/used/) >>>>>> go: finding module for package >>>>>> example.com/m/staging/src/example.com/other1/used >>>>>> cannot find module providing package >>>>>> example.com/m/staging/src/example.com/other1/used: unrecognized >>>>>> import >>>>>> path "example.com/m/staging/src/example.com/other1/used": reading >>>>>> https://example.com/m/staging/src/example.com/other1/used?go-get=1: >>>>>> 404 Not Found >>>>>> ``` >>>>>> >>>>>> It seems that is because the "main" (top-level dir) go.mod has >>>>>> `replace` directives with relative paths, which kubernetes really >>>>>> does. >>>>>> >>>>>> > Incidentally, golang.org/x/tools/go/packages will call 'go list' >>>>>> under the hood in module mode. go/build >>>>>> <https://goto.google.com/build> might do the same, depending on how >>>>>> it's invoked. 'go list' may be the best thing to use if it gives the >>>>>> information you need. >>>>>> >>>>>> Yeah, I noticed. When GO111MODULE=off, everything I am doing is much >>>>>> faster. I'm wary of depending on that forever, though. >>>>>> >>>>>> Stepping back, I fear I am pushing the square peg into a round hole. >>>>>> Let me restate what I am trying to do. >>>>>> >>>>>> I want to run a slow codegen process only if the packages it depends >>>>>> on have ACTUALLY changed (mtime is a good enough proxy) and I don't >>>>>> know a priori which packages need codegen. I want to scan the file >>>>>> tree, find the files that need codegen, check their deps, and only >>>>>> then run the codegen. >>>>>> >>>>>> We do this today with `go list` and GO111MODULE=off, but I was advised >>>>>> at some point that x/tools/go/packages was the future-safe approach. >>>>>> >>>>>> If there's a better way, I am all ears. >>>>>> >>>>>> Tim >>>>>> GO111MODULE=off >>>>>> > On Fri, Jan 15, 2021 at 11:59 AM 'Tim Hockin' via golang-nuts < >>>>>> golang-nuts@googlegroups.com> wrote: >>>>>> >> >>>>>> >> Hi. This isn't exactly burning urgent, but it is a long-term issue >>>>>> >> for Kubernetes. If there's anything I can do to catalyze the >>>>>> >> discussion - tests to run, info to dig up, etc - please let me >>>>>> know. >>>>>> >> >>>>>> >> On Wed, Dec 23, 2020 at 10:48 AM Tim Hockin <thoc...@google.com> >>>>>> wrote: >>>>>> >> > >>>>>> >> > Hi Paul! >>>>>> >> > >>>>>> >> > On Wed, Dec 23, 2020 at 4:23 AM Paul Jolly <p...@myitcv.io> >>>>>> wrote: >>>>>> >> > > >>>>>> >> > > > I just can't figure out how to do this. Maybe it can't be >>>>>> done in `go >>>>>> >> > > > list` ? Or maybe we're just missing some detail of go >>>>>> modules.. >>>>>> >> > > >>>>>> >> > > go list operates in the context of a single module (in the >>>>>> mode you >>>>>> >> > > are interested in), so you cannot do this with a single >>>>>> command across >>>>>> >> > > multiple modules. >>>>>> >> > >>>>>> >> > This might be a real problem for us. For this post I am >>>>>> reducing it >>>>>> >> > to `go list`, but in actuality we have a small program that we >>>>>> wrote >>>>>> >> > which does what we need in terms of `go/build`. It works great >>>>>> when >>>>>> >> > `GO111MODULE=off` but is more than 100x slower normally. I >>>>>> thought it >>>>>> >> > was finally time to rewrite it in terms of `go/packages` and get >>>>>> rid >>>>>> >> > of GO111MODULE=off. That didn't pan out, hence this post. >>>>>> >> > >>>>>> >> > More inline and below >>>>>> >> > >>>>>> >> > > > First I do a `find` for any file that has a specific comment >>>>>> tag, >>>>>> >> > > > indicating that the package needs codegen. The results span >>>>>> several >>>>>> >> > > > of the in-repo submodules. >>>>>> >> > > >>>>>> >> > > Just to check, I'm assuming the results of this find command >>>>>> are being >>>>>> >> > > translated to a list of packages? Because the transitive >>>>>> dependencies >>>>>> >> > > of a list of packages within a module can be done via a single >>>>>> go list >>>>>> >> > > command. >>>>>> >> > >>>>>> >> > The trick is "within a module". I'll update >>>>>> >> > https://github.com/thockin/go-list-modules to reflect the >>>>>> process >>>>>> >> > more. I've added a >>>>>> >> > get_codegen_deps.sh that models the behavior. Note that I >>>>>> really want >>>>>> >> > files, not packages, so I can express the dep-graph. >>>>>> >> > >>>>>> >> > What do you mean by "translated to a list of packages" - which >>>>>> specific syntax? >>>>>> >> > >>>>>> >> > What I end up with is something like `go list ./path/to/dir1 >>>>>> >> > ./path/to/dir2 ./path/to/dir3`. Any of those dirs might be in >>>>>> >> > different modules. So `go list` tells me "main module ( >>>>>> example.com/m) >>>>>> >> > does not contain package example.com/m/path/to/dir1" and so on. >>>>>> >> > Setting `GO111MODULE=off` does work, but I fear the future of >>>>>> that. >>>>>> >> > >>>>>> >> > > > For each target package, I want to get the list of all deps >>>>>> and >>>>>> >> > > > extract the GoFiles. Then I can use that to determine if >>>>>> the codegen >>>>>> >> > > > needs to run. >>>>>> >> > > >>>>>> >> > > FWIW I wrote a tool to do just this: >>>>>> >> > > >>>>>> https://pkg.go.dev/myitcv.io@v0.0.0-20201125173645-a7167afc9e13/cmd/gogenerate >>>>>> >> > > which might work in your situation. >>>>>> >> > >>>>>> >> > I will take a look - it seems I will need to restructure a bunch >>>>>> of >>>>>> >> > tooling to prove it works for us or doesn't :) >>>>>> >> > >>>>>> >> > > > Where it breaks down is that I can't seem to `go list` all >>>>>> at once: >>>>>> >> > > > >>>>>> >> > > > ``` >>>>>> >> > > > # This works within the "root" module >>>>>> >> > > > $ go list -f '{{.GoFiles}}' ./subdir >>>>>> >> > > > [file.go] >>>>>> >> > > >>>>>> >> > > This will work. >>>>>> >> > > >>>>>> >> > > > # This does not work across modules >>>>>> >> > > > $ go list -f '{{.GoFiles}}' ./submod/used ./submod/unused >>>>>> >> > > > main module (example.com/m) does not contain package >>>>>> example.com/m/submod/used >>>>>> >> > > > main module (example.com/m) does not contain package >>>>>> example.com/m/submod/unused >>>>>> >> > > >>>>>> >> > > Per above, this will not work across module boundaries. >>>>>> >> > >>>>>> >> > It works with `GO111MODULE=off` which means that introducing >>>>>> modules >>>>>> >> > is a breaking change. Can I depend on GO111MODULE=off to work >>>>>> the >>>>>> >> > same way forever? >>>>>> >> > >>>>>> >> > > > # Nor does this work, even with module replacements >>>>>> >> > > > $ go list -f '{{.GoFiles}}' ./staging/src/ >>>>>> example.com/other1/used >>>>>> >> > > > ./staging/src/example.com/other1/unused >>>>>> >> > > > main module (example.com/m) does not contain package >>>>>> >> > > > example.com/m/staging/src/example.com/other1/used >>>>>> >> > > > main module (example.com/m) does not contain package >>>>>> >> > > > example.com/m/staging/src/example.com/other1/unused >>>>>> >> > > > ``` >>>>>> >> > > >>>>>> >> > > With replace directives in place this should work, but you >>>>>> won't be >>>>>> >> > > able to use the relative path to the modules (which is in fact >>>>>> >> > > interpreted as a directory): it will need to be the full >>>>>> >> > > module/package path. >>>>>> >> > >>>>>> >> > Given a "./path/to/pkg" - how do I convert that to a >>>>>> module/package >>>>>> >> > path? I can run `(cd $dir && go list -m)` but that is super >>>>>> slow. >>>>>> >> > Running JUST that for each directory that needs codegen in >>>>>> kubernetes >>>>>> >> > takes 20+ seconds. Is there a better way, short of writing my >>>>>> own >>>>>> >> > directory-climb and parsing go.mod? >>>>>> >> > >>>>>> >> > > > I can run `go list` multiple times, but that's INCREDIBLY >>>>>> slow - most >>>>>> >> > > > of these submodules have common deps that are large. This >>>>>> re-parses >>>>>> >> > > > everything over and over. It takes almost 60 seconds just >>>>>> to do `cd >>>>>> >> > > > $dir; go list` (on the real kubernetes repo). >>>>>> >> > > >>>>>> >> > > Do you have a repro of this taking 60 seconds? Because that >>>>>> really >>>>>> >> > > shouldn't be the case with a populated local module cache. >>>>>> >> > >>>>>> >> > github.com/kubernetes/kubernetes >>>>>> >> > >>>>>> >> > ``` >>>>>> >> > $ time \ >>>>>> >> > find . -type f -name \*.go \ >>>>>> >> > | xargs grep -l "^// *+k8s:" \ >>>>>> >> > | xargs -n 1 dirname \ >>>>>> >> > | sort \ >>>>>> >> > | uniq \ >>>>>> >> > | while read X; do \ >>>>>> >> > (cd $X; go list -f '{{.Deps}}'); \ >>>>>> >> > done \ >>>>>> >> > > /dev/null >>>>>> >> > >>>>>> >> > real 0m50.488s >>>>>> >> > user 0m46.686s >>>>>> >> > sys 0m18.416s >>>>>> >> > ``` >>>>>> >> > >>>>>> >> > Just running that inner `go list` with GO111MODULE=off cuts the >>>>>> run >>>>>> >> > time in half. >>>>>> >> > >>>>>> >> > Compare to: >>>>>> >> > >>>>>> >> > ``` >>>>>> >> > time \ >>>>>> >> > ( \ >>>>>> >> > export GO111MODULE=off; \ >>>>>> >> > find . -type f -name \*.go \ >>>>>> >> > | xargs grep -l "^// *+k8s:" \ >>>>>> >> > | xargs -n 1 dirname \ >>>>>> >> > | sort \ >>>>>> >> > | uniq \ >>>>>> >> > | xargs go list -e -f '{{.Deps}}' \ >>>>>> >> > ) \ >>>>>> >> > > /dev/null >>>>>> >> > >>>>>> >> > real 0m1.323s >>>>>> >> > user 0m1.174s >>>>>> >> > sys 0m0.567s >>>>>> >> > ``` >>>>>> >> > >>>>>> >> > The model repo doesn't show so significantly because it is small. >>>>>> >> > Kubernetes is not small. >>>>>> >> > >>>>>> >> > I'm happy to hear better approaches - I really don't like >>>>>> relying on >>>>>> >> > GO111MODULE=off forever - it seems like the sort of thing that >>>>>> will >>>>>> >> > eventually get removed. >>>>>> >> >>>>>> >> -- >>>>>> >> You received this message because you are subscribed to the Google >>>>>> Groups "golang-nuts" group. >>>>>> >> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to golang-nuts+unsubscr...@googlegroups.com. >>>>>> >> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/golang-nuts/CAO_Rewa6rMW79iBHj2Jz6HfJ-tCFLFhNAhYiwDh%3DNy6M35Y91Q%40mail.gmail.com >>>>>> . >>>>>> >>>>> -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAO_RewaaEDAk4zVULp3ZHmCJRmXay4rXapA1cS04M-ff07nhhw%40mail.gmail.com.
