I am not sure what "import external packages" means. Apart dot imports (which I have never seen used for real) why would you need to load the imported packages?
On Saturday, 16 October 2021 at 20:34:17 UTC+1 Steven Hartland wrote: > Thanks Richard, that allowed me to replace a hand rolled universe scope 👍 > > My importer varies from yours in that for correct lookups for versioned > packages or those with '-' in I had to copy ImportPathToAssumedName from > x/tools/internal/imports/fix.go. > > func simpleImporter(imports map[string]*ast.Object, path string) > (*ast.Object, error) { > pkg := imports[path] > if pkg == nil { > pkg = ast.NewObj(ast.Pkg, ImportPathToAssumedName(path)) > pkg.Data = ast.NewScope(nil) // required by ast.NewPackage > for dot-import > imports[path] = pkg > } > return pkg, nil > } > > This now works for all cases which don't import external packages. So now > I just need to do the on demand load of packages, which I suspect will lead > me right back to packages.Load. > > On Sat, 16 Oct 2021 at 15:59, 'Richard Oudkerk' via golang-nuts < > golan...@googlegroups.com> wrote: > >> You could try building the universe scope for ast.NewPackage from >> types.Universe. For example >> >> https://play.golang.org/p/1E5Iu4vW3g9 >> >> func NewPackage(fset *token.FileSet, files map[string]*ast.File) >> (*ast.Package, error) { >> univ, err := universe() >> if err != nil { >> return nil, err >> } >> return ast.NewPackage(fset, files, dummyImporter, univ) >> } >> >> func dummyImporter(imports map[string]*ast.Object, importPath string) >> (*ast.Object, error) { >> pkg := imports[importPath] >> if pkg == nil { >> pkg = ast.NewObj(ast.Pkg, path.Base(importPath)) >> pkg.Data = ast.NewScope(nil) >> imports[importPath] = pkg >> } >> return pkg, nil >> } >> >> func universe() (*ast.Scope, error) { >> u := ast.NewScope(nil) >> for _, name := range types.Universe.Names() { >> o := types.Universe.Lookup(name) >> if o == nil { >> return nil, fmt.Errorf("failed to lookup %s in universe scope", name) >> } >> var objKind ast.ObjKind >> switch o.(type) { >> case *types.Const, *types.Nil: >> objKind = ast.Con >> case *types.TypeName: >> objKind = ast.Typ >> case *types.Builtin: >> objKind = ast.Fun >> default: >> return nil, fmt.Errorf("unexpected builtin %s of type %T", o.Name(), o) >> } >> obj := ast.NewObj(objKind, name) >> if u.Insert(obj) != nil { >> return nil, fmt.Errorf("types internal error: double declaration") >> } >> obj.Decl = u >> } >> return u, nil >> } >> >> On Saturday, 16 October 2021 at 14:38:43 UTC+1 eli...@gmail.com wrote: >> >>> On Fri, Oct 15, 2021 at 2:13 PM Steven Hartland <ste...@multiplay.co.uk> >>> wrote: >>> >>>> I converted my code to x/tools/go/packages >>>> <https://pkg.go.dev/golang.org/x/tools@v0.1.7/go/packages> and while >>>> it did solve the problem it's VERY slow in comparison. >>>> >>>> I have a set of 21 tests operating on a single package which has at >>>> most two very basic types, no imports and using go/parser >>>> <https://pkg.go.dev/go/parser> they take 0.011s but with go/packages >>>> <https://pkg.go.dev/golang.org/x/tools@v0.1.7/go/packages> that >>>> increases to 3.548s a 300x slow down. >>>> >>>> I'm setting a basic mode: packages.NeedName | packages.NeedSyntax >>>> >>>> The package.Load call takes ~220ms whereas ast.NewPackage only >>>> takes 2.7µs. >>>> >>> >>> Could you post a reproducer of your target package and analysis >>> somewhere? 220ms for packages.Load sounds like a lot. It's true that >>> packages does a lot more work than just the parser (*), but it's not >>> supposed to be that slow. In my tests a simple Load with more functionality >>> takes 60-70ms >>> >>> (*) The type checking takes a bit of time over just parsing to AST, but >>> the biggest difference is loading multiple files from imports. For type >>> checking you need to know, when you see: >>> >>> import foo >>> >>> x := foo.Foo() >>> >>> What the type of `x` is, so go/packages has to analyze the `foo` package >>> as well. >>> >>> >>> >>>> >>>> As the resulting ast.File's are pretty much the same, I'm wondering if >>>> for my use case packages.Load is doing way more than I need? >>>> >>>> Another downside is for tests run in a temporary directory outside of >>>> the package space package.Load fails with: >>>> directory /tmp/tests76985775 outside available modules >>>> >>>> I fixed it by calling ioutil.TempDir with "." but that's not ideal. >>>> >>>> Thoughts? >>>> >>>> On Tue, 12 Oct 2021 at 13:42, Steven Hartland <ste...@multiplay.co.uk> >>>> wrote: >>>> >>>>> Thanks David, much appreciated, I will have a look at both. >>>>> >>>>> When migrating from go/ast to go/types did you hit anything of note I >>>>> should look out for? >>>>> >>>>> On Mon, 11 Oct 2021 at 17:06, David Finkel <david....@gmail.com> >>>>> wrote: >>>>> >>>>>> >>>>>> >>>>>> On Mon, Oct 11, 2021 at 5:48 AM Steven Hartland < >>>>>> ste...@multiplay.co.uk> wrote: >>>>>> >>>>>>> If the ast.Files passed to ast.NewPackage includes built in types >>>>>>> such as int it returns an error e.g. >>>>>>> file1.go:5:6: undeclared name: int >>>>>>> >>>>>>> Is there a way to prevent that? >>>>>>> >>>>>> >>>>>> Generally, I always add the `builtin` package to the list of packages >>>>>> I'm parsing. >>>>>> I wrote a little library for exactly this kind of package loading a >>>>>> few years ago: >>>>>> https://gitlab.com/dfinkel/goastpkg/-/blob/master/go_ast_parser.go >>>>>> (https://pkg.go.dev/golang.spin-2.net/astpkg) >>>>>> >>>>>>> >>>>>>> Playground example: https://play.golang.org/p/Yg30TTzoLHP >>>>>>> >>>>>>> My goal is to take multiple files, resolve inter file dependencies >>>>>>> e.g. a type referencing another type in a different file and process >>>>>>> the >>>>>>> resulting ast.Files. So if there is a better way to achieve this I'm >>>>>>> all >>>>>>> ears. >>>>>>> >>>>>> >>>>>> In general, I've stopped using the `go/ast` internal references as >>>>>> much and have started using resolved `go/types` references as they're >>>>>> more >>>>>> reliable and better-specified. >>>>>> (golang.org/x/tools/go/packages >>>>>> <https://pkg.go.dev/golang.org/x/tools@v0.1.7/go/packages> has a >>>>>> LoadMode flag for generating `go/types.Info` (NeedTypesInfo >>>>>> <https://pkg.go.dev/golang.org/x/tools@v0.1.7/go/packages#NeedTypesInfo> >>>>>> )) >>>>>> >>>>>>> >>>>>>> Regards >>>>>>> Steve >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "golang-nuts" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to golang-nuts...@googlegroups.com. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/golang-nuts/CAHEMsqbJoJxuo3c-mofMtzXXJhYCzV2skW2ZB3ZPY6WtA8%2BxHw%40mail.gmail.com >>>>>>> >>>>>>> <https://groups.google.com/d/msgid/golang-nuts/CAHEMsqbJoJxuo3c-mofMtzXXJhYCzV2skW2ZB3ZPY6WtA8%2BxHw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> >>>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "golang-nuts" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to golang-nuts...@googlegroups.com. >>>> >>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/golang-nuts/CAHEMsqYMSBUfuOUvptv6UrvBFTwFxjOhJZ5sMN-omOx5ESL5hw%40mail.gmail.com >>>> >>>> <https://groups.google.com/d/msgid/golang-nuts/CAHEMsqYMSBUfuOUvptv6UrvBFTwFxjOhJZ5sMN-omOx5ESL5hw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "golang-nuts" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to golang-nuts...@googlegroups.com. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/golang-nuts/d570a7ce-a780-46d8-a323-f9c26a6c2561n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/golang-nuts/d570a7ce-a780-46d8-a323-f9c26a6c2561n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/6aaa7c3a-7ef5-47ea-9f29-75443a4599b6n%40googlegroups.com.