I'm guessing here, but you probably do need the full type-checker for that,
depending on what you mean by "a native type".
In go we can alias types in two ways.
type A uint8
type B = uint8
There are subtle differences between the two which I don't claim to
understand... In any case, AFAIK both are essentially built-in types at
runtime.
Suppose these types are declared in a third-party package on GitHub and
another package uses A and B (since they're exported) to declare a struct
like this.
import "github.com/foo/bar"
type MyStruct struct {
Byte1 bar.A
Byte2 bar.B
}
AFAIK, the compiler cannot tell whether Byte1 and Byte2 are builtin types
or a struct without seeing their definition hosted on GitHub. Which means
downloading, parsing, and type-checking the code from GitHub.
But it gets a little worse. Type-checking the remote code entails
downloading any of its dependencies recursively and doing the same thing...
I'm sure you can see how this can take some time -- especially for large
projects.
Anyway, it sounds to me like what you're asking to do requires the use of
the type-checker and because of the dependency management involved I
believe the most convenient route to type-checking (at this time) lives in
the packages package.
Happy to learn if anyone else knows otherwise.
On Sun, Oct 17, 2021, 5:00 AM Steven Hartland <ste...@multiplay.co.uk>
wrote:
> I need to be able to tell the types of fields, in particular are fields of
> a struct a native type or a struct themselves.
>
> The ast parse even with a simple importer don’t provide that info.
>
> On Sat, 16 Oct 2021 at 21:06, 'Richard Oudkerk' via golang-nuts <
> golang-nuts@googlegroups.com> wrote:
>
>> I am not sure what "import external packages" means.
>>
>> Apart dot imports (which I have never seen used for real) why would you
>> need to load the imported packages?
>>
>> On Saturday, 16 October 2021 at 20:34:17 UTC+1 Steven Hartland wrote:
>>
>>> Thanks Richard, that allowed me to replace a hand rolled
>>> universe scope 👍
>>>
>>> My importer varies from yours in that for correct lookups for versioned
>>> packages or those with '-' in I had to copy ImportPathToAssumedName
>>> from x/tools/internal/imports/fix.go.
>>>
>>> func simpleImporter(imports map[string]*ast.Object, path string)
>>> (*ast.Object, error) {
>>> pkg := imports[path]
>>> if pkg == nil {
>>> pkg = ast.NewObj(ast.Pkg, ImportPathToAssumedName(path))
>>> pkg.Data = ast.NewScope(nil) // required by
>>> ast.NewPackage for dot-import
>>> imports[path] = pkg
>>> }
>>> return pkg, nil
>>> }
>>>
>>> This now works for all cases which don't import external packages. So
>>> now I just need to do the on demand load of packages, which I suspect will
>>> lead me right back to packages.Load.
>>>
>>> On Sat, 16 Oct 2021 at 15:59, 'Richard Oudkerk' via golang-nuts <
>>> golan...@googlegroups.com> wrote:
>>>
>>>> You could try building the universe scope for ast.NewPackage from
>>>> types.Universe. For example
>>>>
>>>> https://play.golang.org/p/1E5Iu4vW3g9
>>>>
>>>> func NewPackage(fset *token.FileSet, files map[string]*ast.File)
>>>> (*ast.Package, error) {
>>>> univ, err := universe()
>>>> if err != nil {
>>>> return nil, err
>>>> }
>>>> return ast.NewPackage(fset, files, dummyImporter, univ)
>>>> }
>>>>
>>>> func dummyImporter(imports map[string]*ast.Object, importPath string)
>>>> (*ast.Object, error) {
>>>> pkg := imports[importPath]
>>>> if pkg == nil {
>>>> pkg = ast.NewObj(ast.Pkg, path.Base(importPath))
>>>> pkg.Data = ast.NewScope(nil)
>>>> imports[importPath] = pkg
>>>> }
>>>> return pkg, nil
>>>> }
>>>>
>>>> func universe() (*ast.Scope, error) {
>>>> u := ast.NewScope(nil)
>>>> for _, name := range types.Universe.Names() {
>>>> o := types.Universe.Lookup(name)
>>>> if o == nil {
>>>> return nil, fmt.Errorf("failed to lookup %s in universe scope", name)
>>>> }
>>>> var objKind ast.ObjKind
>>>> switch o.(type) {
>>>> case *types.Const, *types.Nil:
>>>> objKind = ast.Con
>>>> case *types.TypeName:
>>>> objKind = ast.Typ
>>>> case *types.Builtin:
>>>> objKind = ast.Fun
>>>> default:
>>>> return nil, fmt.Errorf("unexpected builtin %s of type %T", o.Name(), o)
>>>> }
>>>> obj := ast.NewObj(objKind, name)
>>>> if u.Insert(obj) != nil {
>>>> return nil, fmt.Errorf("types internal error: double declaration")
>>>> }
>>>> obj.Decl = u
>>>> }
>>>> return u, nil
>>>> }
>>>>
>>>> On Saturday, 16 October 2021 at 14:38:43 UTC+1 eli...@gmail.com wrote:
>>>>
>>>>> On Fri, Oct 15, 2021 at 2:13 PM Steven Hartland <
>>>>> ste...@multiplay.co.uk> wrote:
>>>>>
>>>>>> I converted my code to x/tools/go/packages
>>>>>> <https://pkg.go.dev/golang.org/x/tools@v0.1.7/go/packages> and while
>>>>>> it did solve the problem it's VERY slow in comparison.
>>>>>>
>>>>>> I have a set of 21 tests operating on a single package which has at
>>>>>> most two very basic types, no imports and using go/parser
>>>>>> <https://pkg.go.dev/go/parser> they take 0.011s but with go/packages
>>>>>> <https://pkg.go.dev/golang.org/x/tools@v0.1.7/go/packages> that
>>>>>> increases to 3.548s a 300x slow down.
>>>>>>
>>>>>> I'm setting a basic mode: packages.NeedName | packages.NeedSyntax
>>>>>>
>>>>>> The package.Load call takes ~220ms whereas ast.NewPackage only
>>>>>> takes 2.7µs.
>>>>>>
>>>>>
>>>>> Could you post a reproducer of your target package and analysis
>>>>> somewhere? 220ms for packages.Load sounds like a lot. It's true that
>>>>> packages does a lot more work than just the parser (*), but it's not
>>>>> supposed to be that slow. In my tests a simple Load with more
>>>>> functionality
>>>>> takes 60-70ms
>>>>>
>>>>> (*) The type checking takes a bit of time over just parsing to AST,
>>>>> but the biggest difference is loading multiple files from imports. For
>>>>> type
>>>>> checking you need to know, when you see:
>>>>>
>>>>> import foo
>>>>>
>>>>> x := foo.Foo()
>>>>>
>>>>> What the type of `x` is, so go/packages has to analyze the `foo`
>>>>> package as well.
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> As the resulting ast.File's are pretty much the same, I'm wondering
>>>>>> if for my use case packages.Load is doing way more than I need?
>>>>>>
>>>>>> Another downside is for tests run in a temporary directory outside of
>>>>>> the package space package.Load fails with:
>>>>>> directory /tmp/tests76985775 outside available modules
>>>>>>
>>>>>> I fixed it by calling ioutil.TempDir with "." but that's not ideal.
>>>>>>
>>>>>> Thoughts?
>>>>>>
>>>>>> On Tue, 12 Oct 2021 at 13:42, Steven Hartland <ste...@multiplay.co.uk>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks David, much appreciated, I will have a look at both.
>>>>>>>
>>>>>>> When migrating from go/ast to go/types did you hit anything of note
>>>>>>> I should look out for?
>>>>>>>
>>>>>>> On Mon, 11 Oct 2021 at 17:06, David Finkel <david....@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Oct 11, 2021 at 5:48 AM Steven Hartland <
>>>>>>>> ste...@multiplay.co.uk> wrote:
>>>>>>>>
>>>>>>>>> If the ast.Files passed to ast.NewPackage includes built in types
>>>>>>>>> such as int it returns an error e.g.
>>>>>>>>> file1.go:5:6: undeclared name: int
>>>>>>>>>
>>>>>>>>> Is there a way to prevent that?
>>>>>>>>>
>>>>>>>>
>>>>>>>> Generally, I always add the `builtin` package to the list of
>>>>>>>> packages I'm parsing.
>>>>>>>> I wrote a little library for exactly this kind of package loading a
>>>>>>>> few years ago:
>>>>>>>> https://gitlab.com/dfinkel/goastpkg/-/blob/master/go_ast_parser.go
>>>>>>>> (https://pkg.go.dev/golang.spin-2.net/astpkg)
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Playground example: https://play.golang.org/p/Yg30TTzoLHP
>>>>>>>>>
>>>>>>>>> My goal is to take multiple files, resolve inter file dependencies
>>>>>>>>> e.g. a type referencing another type in a different file and process
>>>>>>>>> the
>>>>>>>>> resulting ast.Files. So if there is a better way to achieve this I'm
>>>>>>>>> all
>>>>>>>>> ears.
>>>>>>>>>
>>>>>>>>
>>>>>>>> In general, I've stopped using the `go/ast` internal references as
>>>>>>>> much and have started using resolved `go/types` references as they're
>>>>>>>> more
>>>>>>>> reliable and better-specified.
>>>>>>>> (golang.org/x/tools/go/packages
>>>>>>>> <https://pkg.go.dev/golang.org/x/tools@v0.1.7/go/packages> has a
>>>>>>>> LoadMode flag for generating `go/types.Info` (NeedTypesInfo
>>>>>>>> <https://pkg.go.dev/golang.org/x/tools@v0.1.7/go/packages#NeedTypesInfo>
>>>>>>>> ))
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regards
>>>>>>>>> Steve
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>>> Groups "golang-nuts" group.
>>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>>> send an email to golang-nuts...@googlegroups.com.
>>>>>>>>> To view this discussion on the web visit
>>>>>>>>> https://groups.google.com/d/msgid/golang-nuts/CAHEMsqbJoJxuo3c-mofMtzXXJhYCzV2skW2ZB3ZPY6WtA8%2BxHw%40mail.gmail.com
>>>>>>>>> <https://groups.google.com/d/msgid/golang-nuts/CAHEMsqbJoJxuo3c-mofMtzXXJhYCzV2skW2ZB3ZPY6WtA8%2BxHw%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>>>>>> .
>>>>>>>>>
>>>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "golang-nuts" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to golang-nuts...@googlegroups.com.
>>>>>>
>>>>> To view this discussion on the web visit
>>>>>> https://groups.google.com/d/msgid/golang-nuts/CAHEMsqYMSBUfuOUvptv6UrvBFTwFxjOhJZ5sMN-omOx5ESL5hw%40mail.gmail.com
>>>>>> <https://groups.google.com/d/msgid/golang-nuts/CAHEMsqYMSBUfuOUvptv6UrvBFTwFxjOhJZ5sMN-omOx5ESL5hw%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>>> .
>>>>>
>>>>>
>>>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "golang-nuts" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to golang-nuts...@googlegroups.com.
>>>>
>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/golang-nuts/d570a7ce-a780-46d8-a323-f9c26a6c2561n%40googlegroups.com
>>>> <https://groups.google.com/d/msgid/golang-nuts/d570a7ce-a780-46d8-a323-f9c26a6c2561n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>> .
>>>>
>>> --
>> You received this message because you are subscribed to the Google Groups
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/golang-nuts/6aaa7c3a-7ef5-47ea-9f29-75443a4599b6n%40googlegroups.com
>> <https://groups.google.com/d/msgid/golang-nuts/6aaa7c3a-7ef5-47ea-9f29-75443a4599b6n%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/CAHEMsqaQ9%3Dia2MXrVEXt--Qk3Nrx1UXK2JYU0D2BxBcZX%2B5mxw%40mail.gmail
> <https://groups.google.com/d/msgid/golang-nuts/CAHEMsqaQ9%3Dia2MXrVEXt--Qk3Nrx1UXK2JYU0D2BxBcZX%2B5mxw%40mail.gmail.com?utm_medium=email&utm_source=footer>
>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/CALJzkY_StxPKx7Riaz10RjfKwFAYtrYQ_zSmHRHvwLRi5JFsUw%40mail.gmail.com.
