Despite what the blog post says, Go 1.17 won't drop support for GO111MODULE or GOPATH mode. It will be more or less the same as GOPATH mode.
We'll continue to work on these issues and others that are blocking people from migrating. I apologize for the slow progress–we have a lot going on. On Wed, Aug 11, 2021 at 12:34 PM Tim Hockin <thoc...@google.com> wrote: > Hi all. I realized that https://blog.golang.org/go116-module-changes > says "We plan to drop support for GOPATH mode in Go 1.17. In other words, > Go 1.17 will ignore GO111MODULE". This will break Kubernetes - our builds > will be orders of magnitude slower. > > AFAICT, https://github.com/golang/go/issues/43806 has had no progress and > https://github.com/golang/go/issues/43733 is still open, too. I have > zero confidence that these are the ONLY issues, and until I can get past > them, I can't really find out. > > Please advise - how should I proceed? > > Tim > > > On Wed, Jan 20, 2021 at 12:57 PM Tim Hockin <thoc...@google.com> wrote: > >> As long as these things are bugs and not "that's not how it works or will >> ever work", I have some hope. This is a back-burner exploration for me, so >> I am not in a panic. Every now and again I will circle back and see if >> anything new happens :) >> >> On Wed, Jan 20, 2021 at 10:50 AM Jay Conrod <jaycon...@google.com> wrote: >> >>> You appear to have discovered another new bug. Sorry for that. I've >>> opened #43806 <https://github.com/golang/go/issues/43806> to track it. >>> >>> With that bug, it doesn't look like `go list` with local paths (relative >>> or absolute) in a repository with nested replacement directories will work >>> (at least not yet). >>> >>> `go list` should still work on full packages paths, but I'm not sure >>> there's an easy way to resolve directory paths to package paths, other than >>> by walking up the directory tree to the nearest go.mod and reading the >>> module path from there. >>> >>> On Tue, Jan 19, 2021 at 8:06 PM Tim Hockin <thoc...@google.com> wrote: >>> >>>> >>>> >>>> On Tue, Jan 19, 2021 at 7:54 AM Jay Conrod <jaycon...@google.com> >>>> wrote: >>>> >>>>> By the way, I'm not sure if you're already doing this, but if you can >>>>> batch all of the `go list` runs (or go/packages >>>>> <https://goto.google.com/packages>.Load) together per module (passing >>>>> an argument per package), that will be much faster than loading individual >>>>> packages with separate `go list` calls. It will save `go list` from having >>>>> to load the module graph and common sets of dependencies multiple times. >>>>> >>>>> If you need to load all the packages in a module, you can also use an >>>>> argument like ./... from the root directory of the module to load >>>>> everything there (excluding modules in subdirectories). >>>>> >>>> >>>> Yeah, I am trying that but tripping on the "does not contain package" >>>> error. >>>> >>>> >>>>> >>>>> On Tue, Jan 19, 2021 at 10:02 AM Jay Conrod <jaycon...@google.com> >>>>> wrote: >>>>> >>>>>> > Interesting - is the difference the absolute paths vs relative? >>>>>> >>>>>> It looks like the bug has to do with whether the directory is below >>>>>> the main module root directory or not. If it is, the go command takes a >>>>>> path that assumes it's part of the main module, which it's not. >>>>>> >>>>>> > I hoped maybe `-modfile` would do the same trick, but alas not: >>>>>> >>>>>> -modfile lets you change the effective content of go.mod but not the >>>>>> module root directory. Unfortunately it doesn't look like that can be >>>>>> used >>>>>> to work around the issue. >>>>>> >>>>>> > It seems that is because the "main" (top-level dir) go.mod has >>>>>> > `replace` directives with relative paths, which kubernetes really >>>>>> > does. >>>>>> >>>>>> You may need to copy those over to the tmp go.mod and adjust the >>>>>> paths. Sorry this has gotten pretty involved. >>>>>> >>>>>> > Yeah, I noticed. When GO111MODULE=off, everything I am doing is >>>>>> much >>>>>> > faster. I'm wary of depending on that forever, though. >>>>>> >>>>>> Module-aware mode is quite a bit more complicated than GOPATH mode, >>>>>> so to some degree it's not surprising it's slower... it's surprising that >>>>>> it's a LOT slower though. I expect there's some optimization work for us >>>>>> to >>>>>> do in the next development cycle. >>>>>> >>>>>> We would eventually like to deprecate GOPATH mode though, so it's a >>>>>> good idea not to depend on it in new tooling today. 'go list' should be >>>>>> fine to get package dependency info in either module mode or GOPATH mode. >>>>>> go/packages <https://goto.google.com/packages> is useful if you need >>>>>> additional information on top of that (parsed syntax trees, type info). >>>>>> >>>>>> > I want to run a slow codegen process only if the packages it depends >>>>>> > on have ACTUALLY changed (mtime is a good enough proxy) and I don't >>>>>> > know a priori which packages need codegen. I want to scan the file >>>>>> > tree, find the files that need codegen, check their deps, and only >>>>>> > then run the codegen. >>>>>> >>>>>> How much dependency info do you need? If the codegen is only within >>>>>> packages with files that have changed, 'go list' might be overkill (it >>>>>> always loads dependencies, even if they aren't printed). If you need >>>>>> dependencies or reverse dependencies, 'go list' or go/packages >>>>>> <https://goto.google.com/packages> are probably the right tools. >>>>>> >>>>>> On Fri, Jan 15, 2021 at 6:43 PM Tim Hockin <thoc...@google.com> >>>>>> wrote: >>>>>> >>>>>>> On Fri, Jan 15, 2021 at 2:17 PM Jay Conrod <jaycon...@google.com> >>>>>>> wrote: >>>>>>> > >>>>>>> > I was initially going to suggest adding the module subdirectories >>>>>>> as requirements to the main go.mod, then replacing those with the >>>>>>> subdirectories. >>>>>>> > >>>>>>> > module example.com/m >>>>>>> > >>>>>>> > go 1.15 >>>>>>> > >>>>>>> > require ( >>>>>>> > example.com/other1 v0.0.0 >>>>>>> > example.com/other2 v0.0.0 >>>>>>> > example.com/m/submod v0.0.0 >>>>>>> > ) >>>>>>> > >>>>>>> > replace ( >>>>>>> > example.com/other1 => ./staging/src/example.com/other1 >>>>>>> > example.com/other2 => ./staging/src/example.com/other2 >>>>>>> > example.com/m/submod v0.0.0 => ./submod >>>>>>> > ) >>>>>>> > >>>>>>> > >>>>>>> > I think you might have tried this already. It gives the same "main >>>>>>> module ... does not contain package" error. I believe that's a bug. I've >>>>>>> opened #43733 to track it. >>>>>>> >>>>>>> Interesting. If that's a bug, then maybe I'll be able to do what I >>>>>>> need once fixed. >>>>>>> >>>>>>> > In general, it should be possible to give 'go list' an absolute or >>>>>>> relative path (starting with ./ or ../) to any directory containing a >>>>>>> package which is part of any module in the build list. For example, some >>>>>>> tools list directories in the module cache to find out what package a >>>>>>> .go >>>>>>> file belongs to. >>>>>>> > >>>>>>> > As a workaround, you could put a go.mod in an otherwise empty >>>>>>> directory (in /tmp or something), then require the relevant modules from >>>>>>> the repo and replace them with absolute paths. Then you can run 'go >>>>>>> list' >>>>>>> in that directory with absolute paths of package directories. >>>>>>> >>>>>>> Interesting - is the difference the absolute paths vs relative? >>>>>>> >>>>>>> I hoped maybe `-modfile` would do the same trick, but alas not: >>>>>>> >>>>>>> ``` >>>>>>> $ (cd /tmp/gomodhack/; go list /tmp/go-list-modules/submod/used/) >>>>>>> example.com/m/submod/used >>>>>>> >>>>>>> $ go list --modfile /tmp/gomodhack/go.mod >>>>>>> /tmp/go-list-modules/submod/used/ >>>>>>> main module (tmp) does not contain package tmp/submod/used >>>>>>> ``` >>>>>>> >>>>>>> It also fails some cases: >>>>>>> >>>>>>> ``` >>>>>>> (cd /tmp/gomodhack/; go list /tmp/go-list-modules/submod/used/) >>>>>>> example.com/m/submod/used >>>>>>> thockin@thockin-glaptop4 go-list-modules main /$ (cd >>>>>>> /tmp/gomodhack/; >>>>>>> go list /tmp/go-list-modules/staging/src/example.com/other1/used/) >>>>>>> go: finding module for package >>>>>>> example.com/m/staging/src/example.com/other1/used >>>>>>> cannot find module providing package >>>>>>> example.com/m/staging/src/example.com/other1/used: unrecognized >>>>>>> import >>>>>>> path "example.com/m/staging/src/example.com/other1/used": reading >>>>>>> https://example.com/m/staging/src/example.com/other1/used?go-get=1: >>>>>>> 404 Not Found >>>>>>> ``` >>>>>>> >>>>>>> It seems that is because the "main" (top-level dir) go.mod has >>>>>>> `replace` directives with relative paths, which kubernetes really >>>>>>> does. >>>>>>> >>>>>>> > Incidentally, golang.org/x/tools/go/packages will call 'go list' >>>>>>> under the hood in module mode. go/build >>>>>>> <https://goto.google.com/build> might do the same, depending on how >>>>>>> it's invoked. 'go list' may be the best thing to use if it gives the >>>>>>> information you need. >>>>>>> >>>>>>> Yeah, I noticed. When GO111MODULE=off, everything I am doing is much >>>>>>> faster. I'm wary of depending on that forever, though. >>>>>>> >>>>>>> Stepping back, I fear I am pushing the square peg into a round hole. >>>>>>> Let me restate what I am trying to do. >>>>>>> >>>>>>> I want to run a slow codegen process only if the packages it depends >>>>>>> on have ACTUALLY changed (mtime is a good enough proxy) and I don't >>>>>>> know a priori which packages need codegen. I want to scan the file >>>>>>> tree, find the files that need codegen, check their deps, and only >>>>>>> then run the codegen. >>>>>>> >>>>>>> We do this today with `go list` and GO111MODULE=off, but I was >>>>>>> advised >>>>>>> at some point that x/tools/go/packages was the future-safe approach. >>>>>>> >>>>>>> If there's a better way, I am all ears. >>>>>>> >>>>>>> Tim >>>>>>> GO111MODULE=off >>>>>>> > On Fri, Jan 15, 2021 at 11:59 AM 'Tim Hockin' via golang-nuts < >>>>>>> golang-nuts@googlegroups.com> wrote: >>>>>>> >> >>>>>>> >> Hi. This isn't exactly burning urgent, but it is a long-term >>>>>>> issue >>>>>>> >> for Kubernetes. If there's anything I can do to catalyze the >>>>>>> >> discussion - tests to run, info to dig up, etc - please let me >>>>>>> know. >>>>>>> >> >>>>>>> >> On Wed, Dec 23, 2020 at 10:48 AM Tim Hockin <thoc...@google.com> >>>>>>> wrote: >>>>>>> >> > >>>>>>> >> > Hi Paul! >>>>>>> >> > >>>>>>> >> > On Wed, Dec 23, 2020 at 4:23 AM Paul Jolly <p...@myitcv.io> >>>>>>> wrote: >>>>>>> >> > > >>>>>>> >> > > > I just can't figure out how to do this. Maybe it can't be >>>>>>> done in `go >>>>>>> >> > > > list` ? Or maybe we're just missing some detail of go >>>>>>> modules.. >>>>>>> >> > > >>>>>>> >> > > go list operates in the context of a single module (in the >>>>>>> mode you >>>>>>> >> > > are interested in), so you cannot do this with a single >>>>>>> command across >>>>>>> >> > > multiple modules. >>>>>>> >> > >>>>>>> >> > This might be a real problem for us. For this post I am >>>>>>> reducing it >>>>>>> >> > to `go list`, but in actuality we have a small program that we >>>>>>> wrote >>>>>>> >> > which does what we need in terms of `go/build`. It works great >>>>>>> when >>>>>>> >> > `GO111MODULE=off` but is more than 100x slower normally. I >>>>>>> thought it >>>>>>> >> > was finally time to rewrite it in terms of `go/packages` and >>>>>>> get rid >>>>>>> >> > of GO111MODULE=off. That didn't pan out, hence this post. >>>>>>> >> > >>>>>>> >> > More inline and below >>>>>>> >> > >>>>>>> >> > > > First I do a `find` for any file that has a specific >>>>>>> comment tag, >>>>>>> >> > > > indicating that the package needs codegen. The results >>>>>>> span several >>>>>>> >> > > > of the in-repo submodules. >>>>>>> >> > > >>>>>>> >> > > Just to check, I'm assuming the results of this find command >>>>>>> are being >>>>>>> >> > > translated to a list of packages? Because the transitive >>>>>>> dependencies >>>>>>> >> > > of a list of packages within a module can be done via a >>>>>>> single go list >>>>>>> >> > > command. >>>>>>> >> > >>>>>>> >> > The trick is "within a module". I'll update >>>>>>> >> > https://github.com/thockin/go-list-modules to reflect the >>>>>>> process >>>>>>> >> > more. I've added a >>>>>>> >> > get_codegen_deps.sh that models the behavior. Note that I >>>>>>> really want >>>>>>> >> > files, not packages, so I can express the dep-graph. >>>>>>> >> > >>>>>>> >> > What do you mean by "translated to a list of packages" - which >>>>>>> specific syntax? >>>>>>> >> > >>>>>>> >> > What I end up with is something like `go list ./path/to/dir1 >>>>>>> >> > ./path/to/dir2 ./path/to/dir3`. Any of those dirs might be in >>>>>>> >> > different modules. So `go list` tells me "main module ( >>>>>>> example.com/m) >>>>>>> >> > does not contain package example.com/m/path/to/dir1" and so on. >>>>>>> >> > Setting `GO111MODULE=off` does work, but I fear the future of >>>>>>> that. >>>>>>> >> > >>>>>>> >> > > > For each target package, I want to get the list of all deps >>>>>>> and >>>>>>> >> > > > extract the GoFiles. Then I can use that to determine if >>>>>>> the codegen >>>>>>> >> > > > needs to run. >>>>>>> >> > > >>>>>>> >> > > FWIW I wrote a tool to do just this: >>>>>>> >> > > >>>>>>> https://pkg.go.dev/myitcv.io@v0.0.0-20201125173645-a7167afc9e13/cmd/gogenerate >>>>>>> >> > > which might work in your situation. >>>>>>> >> > >>>>>>> >> > I will take a look - it seems I will need to restructure a >>>>>>> bunch of >>>>>>> >> > tooling to prove it works for us or doesn't :) >>>>>>> >> > >>>>>>> >> > > > Where it breaks down is that I can't seem to `go list` all >>>>>>> at once: >>>>>>> >> > > > >>>>>>> >> > > > ``` >>>>>>> >> > > > # This works within the "root" module >>>>>>> >> > > > $ go list -f '{{.GoFiles}}' ./subdir >>>>>>> >> > > > [file.go] >>>>>>> >> > > >>>>>>> >> > > This will work. >>>>>>> >> > > >>>>>>> >> > > > # This does not work across modules >>>>>>> >> > > > $ go list -f '{{.GoFiles}}' ./submod/used ./submod/unused >>>>>>> >> > > > main module (example.com/m) does not contain package >>>>>>> example.com/m/submod/used >>>>>>> >> > > > main module (example.com/m) does not contain package >>>>>>> example.com/m/submod/unused >>>>>>> >> > > >>>>>>> >> > > Per above, this will not work across module boundaries. >>>>>>> >> > >>>>>>> >> > It works with `GO111MODULE=off` which means that introducing >>>>>>> modules >>>>>>> >> > is a breaking change. Can I depend on GO111MODULE=off to work >>>>>>> the >>>>>>> >> > same way forever? >>>>>>> >> > >>>>>>> >> > > > # Nor does this work, even with module replacements >>>>>>> >> > > > $ go list -f '{{.GoFiles}}' ./staging/src/ >>>>>>> example.com/other1/used >>>>>>> >> > > > ./staging/src/example.com/other1/unused >>>>>>> >> > > > main module (example.com/m) does not contain package >>>>>>> >> > > > example.com/m/staging/src/example.com/other1/used >>>>>>> >> > > > main module (example.com/m) does not contain package >>>>>>> >> > > > example.com/m/staging/src/example.com/other1/unused >>>>>>> >> > > > ``` >>>>>>> >> > > >>>>>>> >> > > With replace directives in place this should work, but you >>>>>>> won't be >>>>>>> >> > > able to use the relative path to the modules (which is in fact >>>>>>> >> > > interpreted as a directory): it will need to be the full >>>>>>> >> > > module/package path. >>>>>>> >> > >>>>>>> >> > Given a "./path/to/pkg" - how do I convert that to a >>>>>>> module/package >>>>>>> >> > path? I can run `(cd $dir && go list -m)` but that is super >>>>>>> slow. >>>>>>> >> > Running JUST that for each directory that needs codegen in >>>>>>> kubernetes >>>>>>> >> > takes 20+ seconds. Is there a better way, short of writing my >>>>>>> own >>>>>>> >> > directory-climb and parsing go.mod? >>>>>>> >> > >>>>>>> >> > > > I can run `go list` multiple times, but that's INCREDIBLY >>>>>>> slow - most >>>>>>> >> > > > of these submodules have common deps that are large. This >>>>>>> re-parses >>>>>>> >> > > > everything over and over. It takes almost 60 seconds just >>>>>>> to do `cd >>>>>>> >> > > > $dir; go list` (on the real kubernetes repo). >>>>>>> >> > > >>>>>>> >> > > Do you have a repro of this taking 60 seconds? Because that >>>>>>> really >>>>>>> >> > > shouldn't be the case with a populated local module cache. >>>>>>> >> > >>>>>>> >> > github.com/kubernetes/kubernetes >>>>>>> >> > >>>>>>> >> > ``` >>>>>>> >> > $ time \ >>>>>>> >> > find . -type f -name \*.go \ >>>>>>> >> > | xargs grep -l "^// *+k8s:" \ >>>>>>> >> > | xargs -n 1 dirname \ >>>>>>> >> > | sort \ >>>>>>> >> > | uniq \ >>>>>>> >> > | while read X; do \ >>>>>>> >> > (cd $X; go list -f '{{.Deps}}'); \ >>>>>>> >> > done \ >>>>>>> >> > > /dev/null >>>>>>> >> > >>>>>>> >> > real 0m50.488s >>>>>>> >> > user 0m46.686s >>>>>>> >> > sys 0m18.416s >>>>>>> >> > ``` >>>>>>> >> > >>>>>>> >> > Just running that inner `go list` with GO111MODULE=off cuts the >>>>>>> run >>>>>>> >> > time in half. >>>>>>> >> > >>>>>>> >> > Compare to: >>>>>>> >> > >>>>>>> >> > ``` >>>>>>> >> > time \ >>>>>>> >> > ( \ >>>>>>> >> > export GO111MODULE=off; \ >>>>>>> >> > find . -type f -name \*.go \ >>>>>>> >> > | xargs grep -l "^// *+k8s:" \ >>>>>>> >> > | xargs -n 1 dirname \ >>>>>>> >> > | sort \ >>>>>>> >> > | uniq \ >>>>>>> >> > | xargs go list -e -f '{{.Deps}}' \ >>>>>>> >> > ) \ >>>>>>> >> > > /dev/null >>>>>>> >> > >>>>>>> >> > real 0m1.323s >>>>>>> >> > user 0m1.174s >>>>>>> >> > sys 0m0.567s >>>>>>> >> > ``` >>>>>>> >> > >>>>>>> >> > The model repo doesn't show so significantly because it is >>>>>>> small. >>>>>>> >> > Kubernetes is not small. >>>>>>> >> > >>>>>>> >> > I'm happy to hear better approaches - I really don't like >>>>>>> relying on >>>>>>> >> > GO111MODULE=off forever - it seems like the sort of thing that >>>>>>> will >>>>>>> >> > eventually get removed. >>>>>>> >> >>>>>>> >> -- >>>>>>> >> You received this message because you are subscribed to the >>>>>>> Google Groups "golang-nuts" group. >>>>>>> >> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to golang-nuts+unsubscr...@googlegroups.com. >>>>>>> >> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/golang-nuts/CAO_Rewa6rMW79iBHj2Jz6HfJ-tCFLFhNAhYiwDh%3DNy6M35Y91Q%40mail.gmail.com >>>>>>> . >>>>>>> >>>>>> -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAGCADbYt%2B%2BG67Cxe0RRTV3Eb%2BqPRuRPQrA7tm%3Dn8SSDTF6LA0g%40mail.gmail.com.
