On Sat, Oct 16, 2021 at 11:58 AM Steven Hartland <ste...@multiplay.co.uk>
wrote:
> This is an example of code resulting in ~200ms.
>
> This was measured on a laptop with i7-7700HQ under WSL2, so that could be
> a contributing factor.
>
> package mypkg
>
> type MyType struct {
> String string
> }
>
On my machine (oldish i7-4771 running Ubuntu 20.04) a simple load (with
pretty much the tutorial use of x/tools/go/packages) of this code takes
12-18 ms per run.
x/tools/go/packages (XTGP) does a bunch of work - mostly around calling `go
list -json` go discover the files containing code in the module, to parse
and typecheck. It also runs `go env`, and opens and reads a bunch of files.
You can do this work on your own like the other answer demonstrates, but
this will become tricky with external dependencies.
Not sure how you're getting to 200 ms on this; it's possible that in your
environment (WSL2 or what not) invoking subprocesses and stat-ing the
filesystem is particularly costly
>
>
> On Sat, 16 Oct 2021 at 14:38, Eli Bendersky <eli...@gmail.com> wrote:
>
>>
>>
>> On Fri, Oct 15, 2021 at 2:13 PM Steven Hartland <ste...@multiplay.co.uk>
>> wrote:
>>
>>> I converted my code to x/tools/go/packages
>>> <https://pkg.go.dev/golang.org/x/tools@v0.1.7/go/packages> and while it
>>> did solve the problem it's VERY slow in comparison.
>>>
>>> I have a set of 21 tests operating on a single package which has at most
>>> two very basic types, no imports and using go/parser
>>> <https://pkg.go.dev/go/parser> they take 0.011s but with go/packages
>>> <https://pkg.go.dev/golang.org/x/tools@v0.1.7/go/packages> that
>>> increases to 3.548s a 300x slow down.
>>>
>>> I'm setting a basic mode: packages.NeedName | packages.NeedSyntax
>>>
>>> The package.Load call takes ~220ms whereas ast.NewPackage only
>>> takes 2.7µs.
>>>
>>
>> Could you post a reproducer of your target package and analysis
>> somewhere? 220ms for packages.Load sounds like a lot. It's true that
>> packages does a lot more work than just the parser (*), but it's not
>> supposed to be that slow. In my tests a simple Load with more functionality
>> takes 60-70ms
>>
>> (*) The type checking takes a bit of time over just parsing to AST, but
>> the biggest difference is loading multiple files from imports. For type
>> checking you need to know, when you see:
>>
>> import foo
>>
>> x := foo.Foo()
>>
>> What the type of `x` is, so go/packages has to analyze the `foo` package
>> as well.
>>
>>
>>
>>>
>>> As the resulting ast.File's are pretty much the same, I'm wondering if
>>> for my use case packages.Load is doing way more than I need?
>>>
>>> Another downside is for tests run in a temporary directory outside of
>>> the package space package.Load fails with:
>>> directory /tmp/tests76985775 outside available modules
>>>
>>> I fixed it by calling ioutil.TempDir with "." but that's not ideal.
>>>
>>> Thoughts?
>>>
>>> On Tue, 12 Oct 2021 at 13:42, Steven Hartland <ste...@multiplay.co.uk>
>>> wrote:
>>>
>>>> Thanks David, much appreciated, I will have a look at both.
>>>>
>>>> When migrating from go/ast to go/types did you hit anything of note I
>>>> should look out for?
>>>>
>>>> On Mon, 11 Oct 2021 at 17:06, David Finkel <david.fin...@gmail.com>
>>>> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Mon, Oct 11, 2021 at 5:48 AM Steven Hartland <
>>>>> ste...@multiplay.co.uk> wrote:
>>>>>
>>>>>> If the ast.Files passed to ast.NewPackage includes built in types
>>>>>> such as int it returns an error e.g.
>>>>>> file1.go:5:6: undeclared name: int
>>>>>>
>>>>>> Is there a way to prevent that?
>>>>>>
>>>>>
>>>>> Generally, I always add the `builtin` package to the list of packages
>>>>> I'm parsing.
>>>>> I wrote a little library for exactly this kind of package loading a
>>>>> few years ago:
>>>>> https://gitlab.com/dfinkel/goastpkg/-/blob/master/go_ast_parser.go
>>>>> (https://pkg.go.dev/golang.spin-2.net/astpkg)
>>>>>
>>>>>>
>>>>>> Playground example: https://play.golang.org/p/Yg30TTzoLHP
>>>>>>
>>>>>> My goal is to take multiple files, resolve inter file dependencies
>>>>>> e.g. a type referencing another type in a different file and process the
>>>>>> resulting ast.Files. So if there is a better way to achieve this I'm all
>>>>>> ears.
>>>>>>
>>>>>
>>>>> In general, I've stopped using the `go/ast` internal references as
>>>>> much and have started using resolved `go/types` references as they're more
>>>>> reliable and better-specified.
>>>>> (golang.org/x/tools/go/packages
>>>>> <https://pkg.go.dev/golang.org/x/tools@v0.1.7/go/packages> has a
>>>>> LoadMode flag for generating `go/types.Info` (NeedTypesInfo
>>>>> <https://pkg.go.dev/golang.org/x/tools@v0.1.7/go/packages#NeedTypesInfo>
>>>>> ))
>>>>>
>>>>>>
>>>>>> Regards
>>>>>> Steve
>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "golang-nuts" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to golang-nuts+unsubscr...@googlegroups.com.
>>>>>> To view this discussion on the web visit
>>>>>> https://groups.google.com/d/msgid/golang-nuts/CAHEMsqbJoJxuo3c-mofMtzXXJhYCzV2skW2ZB3ZPY6WtA8%2BxHw%40mail.gmail.com
>>>>>> <https://groups.google.com/d/msgid/golang-nuts/CAHEMsqbJoJxuo3c-mofMtzXXJhYCzV2skW2ZB3ZPY6WtA8%2BxHw%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>>> .
>>>>>>
>>>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to golang-nuts+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/golang-nuts/CAHEMsqYMSBUfuOUvptv6UrvBFTwFxjOhJZ5sMN-omOx5ESL5hw%40mail.gmail.com
>>> <https://groups.google.com/d/msgid/golang-nuts/CAHEMsqYMSBUfuOUvptv6UrvBFTwFxjOhJZ5sMN-omOx5ESL5hw%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/CAF-Rda-xd5kHFkGwhWoZYUGE_%3Du%3DWnW4hq_WogsexbyL-Een6A%40mail.gmail.com.
