As long as these things are bugs and not "that's not how it works or will ever work", I have some hope. This is a back-burner exploration for me, so I am not in a panic. Every now and again I will circle back and see if anything new happens :)
On Wed, Jan 20, 2021 at 10:50 AM Jay Conrod <jaycon...@google.com> wrote: > You appear to have discovered another new bug. Sorry for that. I've opened > #43806 <https://github.com/golang/go/issues/43806> to track it. > > With that bug, it doesn't look like `go list` with local paths (relative > or absolute) in a repository with nested replacement directories will work > (at least not yet). > > `go list` should still work on full packages paths, but I'm not sure > there's an easy way to resolve directory paths to package paths, other than > by walking up the directory tree to the nearest go.mod and reading the > module path from there. > > On Tue, Jan 19, 2021 at 8:06 PM Tim Hockin <thoc...@google.com> wrote: > >> >> >> On Tue, Jan 19, 2021 at 7:54 AM Jay Conrod <jaycon...@google.com> wrote: >> >>> By the way, I'm not sure if you're already doing this, but if you can >>> batch all of the `go list` runs (or go/packages >>> <https://goto.google.com/packages>.Load) together per module (passing >>> an argument per package), that will be much faster than loading individual >>> packages with separate `go list` calls. It will save `go list` from having >>> to load the module graph and common sets of dependencies multiple times. >>> >>> If you need to load all the packages in a module, you can also use an >>> argument like ./... from the root directory of the module to load >>> everything there (excluding modules in subdirectories). >>> >> >> Yeah, I am trying that but tripping on the "does not contain package" >> error. >> >> >>> >>> On Tue, Jan 19, 2021 at 10:02 AM Jay Conrod <jaycon...@google.com> >>> wrote: >>> >>>> > Interesting - is the difference the absolute paths vs relative? >>>> >>>> It looks like the bug has to do with whether the directory is below the >>>> main module root directory or not. If it is, the go command takes a path >>>> that assumes it's part of the main module, which it's not. >>>> >>>> > I hoped maybe `-modfile` would do the same trick, but alas not: >>>> >>>> -modfile lets you change the effective content of go.mod but not the >>>> module root directory. Unfortunately it doesn't look like that can be used >>>> to work around the issue. >>>> >>>> > It seems that is because the "main" (top-level dir) go.mod has >>>> > `replace` directives with relative paths, which kubernetes really >>>> > does. >>>> >>>> You may need to copy those over to the tmp go.mod and adjust the paths. >>>> Sorry this has gotten pretty involved. >>>> >>>> > Yeah, I noticed. When GO111MODULE=off, everything I am doing is much >>>> > faster. I'm wary of depending on that forever, though. >>>> >>>> Module-aware mode is quite a bit more complicated than GOPATH mode, so >>>> to some degree it's not surprising it's slower... it's surprising that it's >>>> a LOT slower though. I expect there's some optimization work for us to do >>>> in the next development cycle. >>>> >>>> We would eventually like to deprecate GOPATH mode though, so it's a >>>> good idea not to depend on it in new tooling today. 'go list' should be >>>> fine to get package dependency info in either module mode or GOPATH mode. >>>> go/packages <https://goto.google.com/packages> is useful if you need >>>> additional information on top of that (parsed syntax trees, type info). >>>> >>>> > I want to run a slow codegen process only if the packages it depends >>>> > on have ACTUALLY changed (mtime is a good enough proxy) and I don't >>>> > know a priori which packages need codegen. I want to scan the file >>>> > tree, find the files that need codegen, check their deps, and only >>>> > then run the codegen. >>>> >>>> How much dependency info do you need? If the codegen is only within >>>> packages with files that have changed, 'go list' might be overkill (it >>>> always loads dependencies, even if they aren't printed). If you need >>>> dependencies or reverse dependencies, 'go list' or go/packages >>>> <https://goto.google.com/packages> are probably the right tools. >>>> >>>> On Fri, Jan 15, 2021 at 6:43 PM Tim Hockin <thoc...@google.com> wrote: >>>> >>>>> On Fri, Jan 15, 2021 at 2:17 PM Jay Conrod <jaycon...@google.com> >>>>> wrote: >>>>> > >>>>> > I was initially going to suggest adding the module subdirectories as >>>>> requirements to the main go.mod, then replacing those with the >>>>> subdirectories. >>>>> > >>>>> > module example.com/m >>>>> > >>>>> > go 1.15 >>>>> > >>>>> > require ( >>>>> > example.com/other1 v0.0.0 >>>>> > example.com/other2 v0.0.0 >>>>> > example.com/m/submod v0.0.0 >>>>> > ) >>>>> > >>>>> > replace ( >>>>> > example.com/other1 => ./staging/src/example.com/other1 >>>>> > example.com/other2 => ./staging/src/example.com/other2 >>>>> > example.com/m/submod v0.0.0 => ./submod >>>>> > ) >>>>> > >>>>> > >>>>> > I think you might have tried this already. It gives the same "main >>>>> module ... does not contain package" error. I believe that's a bug. I've >>>>> opened #43733 to track it. >>>>> >>>>> Interesting. If that's a bug, then maybe I'll be able to do what I >>>>> need once fixed. >>>>> >>>>> > In general, it should be possible to give 'go list' an absolute or >>>>> relative path (starting with ./ or ../) to any directory containing a >>>>> package which is part of any module in the build list. For example, some >>>>> tools list directories in the module cache to find out what package a .go >>>>> file belongs to. >>>>> > >>>>> > As a workaround, you could put a go.mod in an otherwise empty >>>>> directory (in /tmp or something), then require the relevant modules from >>>>> the repo and replace them with absolute paths. Then you can run 'go list' >>>>> in that directory with absolute paths of package directories. >>>>> >>>>> Interesting - is the difference the absolute paths vs relative? >>>>> >>>>> I hoped maybe `-modfile` would do the same trick, but alas not: >>>>> >>>>> ``` >>>>> $ (cd /tmp/gomodhack/; go list /tmp/go-list-modules/submod/used/) >>>>> example.com/m/submod/used >>>>> >>>>> $ go list --modfile /tmp/gomodhack/go.mod >>>>> /tmp/go-list-modules/submod/used/ >>>>> main module (tmp) does not contain package tmp/submod/used >>>>> ``` >>>>> >>>>> It also fails some cases: >>>>> >>>>> ``` >>>>> (cd /tmp/gomodhack/; go list /tmp/go-list-modules/submod/used/) >>>>> example.com/m/submod/used >>>>> thockin@thockin-glaptop4 go-list-modules main /$ (cd /tmp/gomodhack/; >>>>> go list /tmp/go-list-modules/staging/src/example.com/other1/used/) >>>>> go: finding module for package >>>>> example.com/m/staging/src/example.com/other1/used >>>>> cannot find module providing package >>>>> example.com/m/staging/src/example.com/other1/used: unrecognized import >>>>> path "example.com/m/staging/src/example.com/other1/used": reading >>>>> https://example.com/m/staging/src/example.com/other1/used?go-get=1: >>>>> 404 Not Found >>>>> ``` >>>>> >>>>> It seems that is because the "main" (top-level dir) go.mod has >>>>> `replace` directives with relative paths, which kubernetes really >>>>> does. >>>>> >>>>> > Incidentally, golang.org/x/tools/go/packages will call 'go list' >>>>> under the hood in module mode. go/build >>>>> <https://goto.google.com/build> might do the same, depending on how >>>>> it's invoked. 'go list' may be the best thing to use if it gives the >>>>> information you need. >>>>> >>>>> Yeah, I noticed. When GO111MODULE=off, everything I am doing is much >>>>> faster. I'm wary of depending on that forever, though. >>>>> >>>>> Stepping back, I fear I am pushing the square peg into a round hole. >>>>> Let me restate what I am trying to do. >>>>> >>>>> I want to run a slow codegen process only if the packages it depends >>>>> on have ACTUALLY changed (mtime is a good enough proxy) and I don't >>>>> know a priori which packages need codegen. I want to scan the file >>>>> tree, find the files that need codegen, check their deps, and only >>>>> then run the codegen. >>>>> >>>>> We do this today with `go list` and GO111MODULE=off, but I was advised >>>>> at some point that x/tools/go/packages was the future-safe approach. >>>>> >>>>> If there's a better way, I am all ears. >>>>> >>>>> Tim >>>>> GO111MODULE=off >>>>> > On Fri, Jan 15, 2021 at 11:59 AM 'Tim Hockin' via golang-nuts < >>>>> golang-nuts@googlegroups.com> wrote: >>>>> >> >>>>> >> Hi. This isn't exactly burning urgent, but it is a long-term issue >>>>> >> for Kubernetes. If there's anything I can do to catalyze the >>>>> >> discussion - tests to run, info to dig up, etc - please let me know. >>>>> >> >>>>> >> On Wed, Dec 23, 2020 at 10:48 AM Tim Hockin <thoc...@google.com> >>>>> wrote: >>>>> >> > >>>>> >> > Hi Paul! >>>>> >> > >>>>> >> > On Wed, Dec 23, 2020 at 4:23 AM Paul Jolly <p...@myitcv.io> >>>>> wrote: >>>>> >> > > >>>>> >> > > > I just can't figure out how to do this. Maybe it can't be >>>>> done in `go >>>>> >> > > > list` ? Or maybe we're just missing some detail of go >>>>> modules.. >>>>> >> > > >>>>> >> > > go list operates in the context of a single module (in the mode >>>>> you >>>>> >> > > are interested in), so you cannot do this with a single command >>>>> across >>>>> >> > > multiple modules. >>>>> >> > >>>>> >> > This might be a real problem for us. For this post I am reducing >>>>> it >>>>> >> > to `go list`, but in actuality we have a small program that we >>>>> wrote >>>>> >> > which does what we need in terms of `go/build`. It works great >>>>> when >>>>> >> > `GO111MODULE=off` but is more than 100x slower normally. I >>>>> thought it >>>>> >> > was finally time to rewrite it in terms of `go/packages` and get >>>>> rid >>>>> >> > of GO111MODULE=off. That didn't pan out, hence this post. >>>>> >> > >>>>> >> > More inline and below >>>>> >> > >>>>> >> > > > First I do a `find` for any file that has a specific comment >>>>> tag, >>>>> >> > > > indicating that the package needs codegen. The results span >>>>> several >>>>> >> > > > of the in-repo submodules. >>>>> >> > > >>>>> >> > > Just to check, I'm assuming the results of this find command >>>>> are being >>>>> >> > > translated to a list of packages? Because the transitive >>>>> dependencies >>>>> >> > > of a list of packages within a module can be done via a single >>>>> go list >>>>> >> > > command. >>>>> >> > >>>>> >> > The trick is "within a module". I'll update >>>>> >> > https://github.com/thockin/go-list-modules to reflect the process >>>>> >> > more. I've added a >>>>> >> > get_codegen_deps.sh that models the behavior. Note that I really >>>>> want >>>>> >> > files, not packages, so I can express the dep-graph. >>>>> >> > >>>>> >> > What do you mean by "translated to a list of packages" - which >>>>> specific syntax? >>>>> >> > >>>>> >> > What I end up with is something like `go list ./path/to/dir1 >>>>> >> > ./path/to/dir2 ./path/to/dir3`. Any of those dirs might be in >>>>> >> > different modules. So `go list` tells me "main module ( >>>>> example.com/m) >>>>> >> > does not contain package example.com/m/path/to/dir1" and so on. >>>>> >> > Setting `GO111MODULE=off` does work, but I fear the future of >>>>> that. >>>>> >> > >>>>> >> > > > For each target package, I want to get the list of all deps >>>>> and >>>>> >> > > > extract the GoFiles. Then I can use that to determine if the >>>>> codegen >>>>> >> > > > needs to run. >>>>> >> > > >>>>> >> > > FWIW I wrote a tool to do just this: >>>>> >> > > >>>>> https://pkg.go.dev/myitcv.io@v0.0.0-20201125173645-a7167afc9e13/cmd/gogenerate >>>>> >> > > which might work in your situation. >>>>> >> > >>>>> >> > I will take a look - it seems I will need to restructure a bunch >>>>> of >>>>> >> > tooling to prove it works for us or doesn't :) >>>>> >> > >>>>> >> > > > Where it breaks down is that I can't seem to `go list` all at >>>>> once: >>>>> >> > > > >>>>> >> > > > ``` >>>>> >> > > > # This works within the "root" module >>>>> >> > > > $ go list -f '{{.GoFiles}}' ./subdir >>>>> >> > > > [file.go] >>>>> >> > > >>>>> >> > > This will work. >>>>> >> > > >>>>> >> > > > # This does not work across modules >>>>> >> > > > $ go list -f '{{.GoFiles}}' ./submod/used ./submod/unused >>>>> >> > > > main module (example.com/m) does not contain package >>>>> example.com/m/submod/used >>>>> >> > > > main module (example.com/m) does not contain package >>>>> example.com/m/submod/unused >>>>> >> > > >>>>> >> > > Per above, this will not work across module boundaries. >>>>> >> > >>>>> >> > It works with `GO111MODULE=off` which means that introducing >>>>> modules >>>>> >> > is a breaking change. Can I depend on GO111MODULE=off to work the >>>>> >> > same way forever? >>>>> >> > >>>>> >> > > > # Nor does this work, even with module replacements >>>>> >> > > > $ go list -f '{{.GoFiles}}' ./staging/src/ >>>>> example.com/other1/used >>>>> >> > > > ./staging/src/example.com/other1/unused >>>>> >> > > > main module (example.com/m) does not contain package >>>>> >> > > > example.com/m/staging/src/example.com/other1/used >>>>> >> > > > main module (example.com/m) does not contain package >>>>> >> > > > example.com/m/staging/src/example.com/other1/unused >>>>> >> > > > ``` >>>>> >> > > >>>>> >> > > With replace directives in place this should work, but you >>>>> won't be >>>>> >> > > able to use the relative path to the modules (which is in fact >>>>> >> > > interpreted as a directory): it will need to be the full >>>>> >> > > module/package path. >>>>> >> > >>>>> >> > Given a "./path/to/pkg" - how do I convert that to a >>>>> module/package >>>>> >> > path? I can run `(cd $dir && go list -m)` but that is super slow. >>>>> >> > Running JUST that for each directory that needs codegen in >>>>> kubernetes >>>>> >> > takes 20+ seconds. Is there a better way, short of writing my own >>>>> >> > directory-climb and parsing go.mod? >>>>> >> > >>>>> >> > > > I can run `go list` multiple times, but that's INCREDIBLY >>>>> slow - most >>>>> >> > > > of these submodules have common deps that are large. This >>>>> re-parses >>>>> >> > > > everything over and over. It takes almost 60 seconds just to >>>>> do `cd >>>>> >> > > > $dir; go list` (on the real kubernetes repo). >>>>> >> > > >>>>> >> > > Do you have a repro of this taking 60 seconds? Because that >>>>> really >>>>> >> > > shouldn't be the case with a populated local module cache. >>>>> >> > >>>>> >> > github.com/kubernetes/kubernetes >>>>> >> > >>>>> >> > ``` >>>>> >> > $ time \ >>>>> >> > find . -type f -name \*.go \ >>>>> >> > | xargs grep -l "^// *+k8s:" \ >>>>> >> > | xargs -n 1 dirname \ >>>>> >> > | sort \ >>>>> >> > | uniq \ >>>>> >> > | while read X; do \ >>>>> >> > (cd $X; go list -f '{{.Deps}}'); \ >>>>> >> > done \ >>>>> >> > > /dev/null >>>>> >> > >>>>> >> > real 0m50.488s >>>>> >> > user 0m46.686s >>>>> >> > sys 0m18.416s >>>>> >> > ``` >>>>> >> > >>>>> >> > Just running that inner `go list` with GO111MODULE=off cuts the >>>>> run >>>>> >> > time in half. >>>>> >> > >>>>> >> > Compare to: >>>>> >> > >>>>> >> > ``` >>>>> >> > time \ >>>>> >> > ( \ >>>>> >> > export GO111MODULE=off; \ >>>>> >> > find . -type f -name \*.go \ >>>>> >> > | xargs grep -l "^// *+k8s:" \ >>>>> >> > | xargs -n 1 dirname \ >>>>> >> > | sort \ >>>>> >> > | uniq \ >>>>> >> > | xargs go list -e -f '{{.Deps}}' \ >>>>> >> > ) \ >>>>> >> > > /dev/null >>>>> >> > >>>>> >> > real 0m1.323s >>>>> >> > user 0m1.174s >>>>> >> > sys 0m0.567s >>>>> >> > ``` >>>>> >> > >>>>> >> > The model repo doesn't show so significantly because it is small. >>>>> >> > Kubernetes is not small. >>>>> >> > >>>>> >> > I'm happy to hear better approaches - I really don't like relying >>>>> on >>>>> >> > GO111MODULE=off forever - it seems like the sort of thing that >>>>> will >>>>> >> > eventually get removed. >>>>> >> >>>>> >> -- >>>>> >> You received this message because you are subscribed to the Google >>>>> Groups "golang-nuts" group. >>>>> >> To unsubscribe from this group and stop receiving emails from it, >>>>> send an email to golang-nuts+unsubscr...@googlegroups.com. >>>>> >> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/golang-nuts/CAO_Rewa6rMW79iBHj2Jz6HfJ-tCFLFhNAhYiwDh%3DNy6M35Y91Q%40mail.gmail.com >>>>> . >>>>> >>>> -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAO_RewZV%3DwqfjxbfTwR6VuOaHpqHx8QsAi%2Bur%3DAvHm7vjDFpUw%40mail.gmail.com.
