Re: [go-nuts] The GitHub Vet Project

[Tyler Compton]

>
> This is a valid concern but since the tool remains under active
> development, I would caution against making an assessment based on the
> number of issues currently reported in the tracker.
>

Fair point! I'll reserve judgement in that case then.

On Wed, Dec 30, 2020 at 9:57 PM K. Alex Mills 
wrote:

> On Wed, Dec 30, 2020, 10:34 PM Tyler Compton  wrote:
>
>> This is a very interesting project! I went ahead and labeled a few of the
>> issues. However, much as I hate to pour cold water on the idea, I have some
>> doubts about how effective this approach can be. The githubvet bot has
>> reported over ten thousand issues so far, which is quite a lot.
>>
>
> This is a valid concern but since the tool remains under active
> development, I would caution against making an assessment based on the
> number of issues currently reported in the tracker.
>
> Part of the problem is that the current tracker contains issues from
> multiple runs, since I've rescanned during testing, as I find and fixing
> issues.
>
> There are still some things that can be done to bring down the
> false-positive rate, but before addressing that I need to spend time on
> some of the false-negatives that I am currently aware of.
>
> My gut feeling is that if code does exist that correctly relies on the
>> current behavior, it would be extremely subtle. If we're going to get
>> through all of these issues, we won't be able to give each example a
>> complete deep dive which I think means we'll likely miss very subtle cases.
>> Not that I have any better ideas :)
>>
>
> Part of the hope has been that a large number of community participants
> would help parallelize the effort and (perhaps) even get multiple pairs of
> eyes on each issue. Whether that will work or not remains to be seen.
>
> I will also say that many of the issues I have looked at have seemed to be
> classifiable rather quickly. OTOH, that could also be an indication that
> the false-positive rate remains a little high.
>
> All that said, so far, the issues reported only focus on race-conditions
> or incorrectly storing a reference to a range-loop pointer. Tim King has
> just pointed out a few other concerns like pointer comparison and
> finalizers for which analyzers have yet to be written. My perception is
> that these two features aren't very widely used, so I wouldn't expect to
> find a large number of instances like these.
>
>
>> On Sun, Dec 20, 2020 at 12:38 PM K. Alex Mills 
>> wrote:
>>
>>> Hello Gophers!
>>>
>>> During a Q session at this year's GopherCon, some members of the Go
>>> language team indicated a willingness to modify the behavior of range
>>> loops  *if* we can be
>>> reasonably certain that the change will not cause incorrect behavior in
>>> current programs. To make that determination, a large collection of
>>> real-world go programs would need to be vetted. If we can find that, in
>>> every case, modifying the compiler behavior does not lead to an undesirable
>>> outcome, we will have strong evidence that the change may only have a small
>>> impact.
>>>
>>> I've been working on a project to gather and crowdsource data for that
>>> sort of analysis. It has reached a point where I think that it's time to
>>> share it with the Go community.
>>>
>>> The GitHub Vet Project  performs static
>>> analysis on public Go repositories hosted on GitHub in order to better
>>> understand the impact of this proposed language change
>>> . It consists of two bots.
>>> VetBot crawls GitHub to snippets of code it thinks could be interesting, it
>>> reports it as an issue in a separate repository
>>> , for humans
>>> to analyze via crowdsourcing. TrackBot
>>>  manages
>>> the crowdsourcing workflow.
>>>
>>> At this point, all of the features I think are necessary for the
>>> project's success have been implemented. But I am only one Gopher, and so I
>>> would like to a) make the community aware of the project and b) ask the
>>> community to help review it in three ways:
>>>
>>> 1) Test out TrackBot and the project instructions by actually
>>> crowdsourcing through the issues found so far
>>> .
>>> 2) Review the output
>>>  and
>>> raise concern if anything that I claim ought to be excluded
>>> 
>>> is somehow making it through.
>>> 3) Static analysis experts: review the analyzers
>>>  and
>>> ruthlessly question anything that could lead to a false negative. I don't
>>> think anything exists, but one pair of eyes is often wrong, and there are
>>> many folks on this list with (much) 

Re: [go-nuts] The GitHub Vet Project

[K. Alex Mills]

On Wed, Dec 30, 2020, 10:34 PM Tyler Compton  wrote:

> This is a very interesting project! I went ahead and labeled a few of the
> issues. However, much as I hate to pour cold water on the idea, I have some
> doubts about how effective this approach can be. The githubvet bot has
> reported over ten thousand issues so far, which is quite a lot.
>

This is a valid concern but since the tool remains under active
development, I would caution against making an assessment based on the
number of issues currently reported in the tracker.

Part of the problem is that the current tracker contains issues from
multiple runs, since I've rescanned during testing, as I find and fixing
issues.

There are still some things that can be done to bring down the
false-positive rate, but before addressing that I need to spend time on
some of the false-negatives that I am currently aware of.

My gut feeling is that if code does exist that correctly relies on the
> current behavior, it would be extremely subtle. If we're going to get
> through all of these issues, we won't be able to give each example a
> complete deep dive which I think means we'll likely miss very subtle cases.
> Not that I have any better ideas :)
>

Part of the hope has been that a large number of community participants
would help parallelize the effort and (perhaps) even get multiple pairs of
eyes on each issue. Whether that will work or not remains to be seen.

I will also say that many of the issues I have looked at have seemed to be
classifiable rather quickly. OTOH, that could also be an indication that
the false-positive rate remains a little high.

All that said, so far, the issues reported only focus on race-conditions or
incorrectly storing a reference to a range-loop pointer. Tim King has just
pointed out a few other concerns like pointer comparison and finalizers for
which analyzers have yet to be written. My perception is that these two
features aren't very widely used, so I wouldn't expect to find a large
number of instances like these.


> On Sun, Dec 20, 2020 at 12:38 PM K. Alex Mills 
> wrote:
>
>> Hello Gophers!
>>
>> During a Q session at this year's GopherCon, some members of the Go
>> language team indicated a willingness to modify the behavior of range
>> loops  *if* we can be
>> reasonably certain that the change will not cause incorrect behavior in
>> current programs. To make that determination, a large collection of
>> real-world go programs would need to be vetted. If we can find that, in
>> every case, modifying the compiler behavior does not lead to an undesirable
>> outcome, we will have strong evidence that the change may only have a small
>> impact.
>>
>> I've been working on a project to gather and crowdsource data for that
>> sort of analysis. It has reached a point where I think that it's time to
>> share it with the Go community.
>>
>> The GitHub Vet Project  performs static
>> analysis on public Go repositories hosted on GitHub in order to better
>> understand the impact of this proposed language change
>> . It consists of two bots.
>> VetBot crawls GitHub to snippets of code it thinks could be interesting, it
>> reports it as an issue in a separate repository
>> , for humans
>> to analyze via crowdsourcing. TrackBot
>>  manages the
>> crowdsourcing workflow.
>>
>> At this point, all of the features I think are necessary for the
>> project's success have been implemented. But I am only one Gopher, and so I
>> would like to a) make the community aware of the project and b) ask the
>> community to help review it in three ways:
>>
>> 1) Test out TrackBot and the project instructions by actually
>> crowdsourcing through the issues found so far
>> .
>> 2) Review the output
>>  and
>> raise concern if anything that I claim ought to be excluded
>> 
>> is somehow making it through.
>> 3) Static analysis experts: review the analyzers
>>  and
>> ruthlessly question anything that could lead to a false negative. I don't
>> think anything exists, but one pair of eyes is often wrong, and there are
>> many folks on this list with (much) more expertise than me.
>>
>> One final thing. The crowd-sourcing workflow I've designed relies on the
>> opinion of experts to help estimate the reliability of the community
>> assessment. In order for that to work, I need the help of some Go experts
>> who are willing to commit some time to reviewing the findings. If you'd
>> like to participate in this way, first, read the TrackBot documentation
>> 

Re: [go-nuts] The GitHub Vet Project

[Tyler Compton]

This is a very interesting project! I went ahead and labeled a few of the
issues. However, much as I hate to pour cold water on the idea, I have some
doubts about how effective this approach can be. The githubvet bot has
reported over ten thousand issues so far, which is quite a lot. My gut
feeling is that if code does exist that correctly relies on the current
behavior, it would be extremely subtle. If we're going to get through all
of these issues, we won't be able to give each example a complete deep dive
which I think means we'll likely miss very subtle cases. Not that I have
any better ideas :)

On Sun, Dec 20, 2020 at 12:38 PM K. Alex Mills 
wrote:

> Hello Gophers!
>
> During a Q session at this year's GopherCon, some members of the Go
> language team indicated a willingness to modify the behavior of range
> loops  *if* we can be
> reasonably certain that the change will not cause incorrect behavior in
> current programs. To make that determination, a large collection of
> real-world go programs would need to be vetted. If we can find that, in
> every case, modifying the compiler behavior does not lead to an undesirable
> outcome, we will have strong evidence that the change may only have a small
> impact.
>
> I've been working on a project to gather and crowdsource data for that
> sort of analysis. It has reached a point where I think that it's time to
> share it with the Go community.
>
> The GitHub Vet Project  performs static
> analysis on public Go repositories hosted on GitHub in order to better
> understand the impact of this proposed language change
> . It consists of two bots.
> VetBot crawls GitHub to snippets of code it thinks could be interesting, it
> reports it as an issue in a separate repository
> , for humans to
> analyze via crowdsourcing. TrackBot
>  manages the
> crowdsourcing workflow.
>
> At this point, all of the features I think are necessary for the project's
> success have been implemented. But I am only one Gopher, and so I would
> like to a) make the community aware of the project and b) ask the community
> to help review it in three ways:
>
> 1) Test out TrackBot and the project instructions by actually
> crowdsourcing through the issues found so far
> .
> 2) Review the output
>  and
> raise concern if anything that I claim ought to be excluded
> 
> is somehow making it through.
> 3) Static analysis experts: review the analyzers
>  and ruthlessly
> question anything that could lead to a false negative. I don't think
> anything exists, but one pair of eyes is often wrong, and there are many
> folks on this list with (much) more expertise than me.
>
> One final thing. The crowd-sourcing workflow I've designed relies on the
> opinion of experts to help estimate the reliability of the community
> assessment. In order for that to work, I need the help of some Go experts
> who are willing to commit some time to reviewing the findings. If you'd
> like to participate in this way, first, read the TrackBot documentation
>  to
> understand what being an expert entails. If you're still interested, email
> me with the subject line 'GitHub Vet Expert' and include your GitHub
> username and a brief outline of your experience with Go.
>
> It's my hope that this project can provide some data that can help to move
> Go forward. To that end, I'm also interested in any and all feedback and
> suggestions. Contributions are also welcome
> .
>
> Thanks for reading,
>
> K. Alex Mills, Ph.D
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/CALJzkY86ZgQMQBqiAyDOKHHionjLZJbB6f%2BEDcMp82ZaPCKmXw%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 

Re: [go-nuts] The GitHub Vet Project

[Jan Mercl]

On Mon, Dec 21, 2020 at 8:30 PM K. Alex Mills  wrote:

> I think that it would be best to keep our discussion of the proposed change 
> part of the relevant GitHub tracker to ensure the conversation can happen in 
> context of prior discussions.

Unfortunately, that's not going to happen, and I am truly sorry about that.

Go is a Google initiated, funded and directed project. Please explain
why a Microsoft controlled account is required to continue the
discussion of this Go issue.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAA40n-UPvN7erC6FAqBYO%2BBuHrK--uK59ysGwr0MEPh8jzDq4A%40mail.gmail.com.

Re: [go-nuts] The GitHub Vet Project

[K. Alex Mills]

On Sun, Dec 20, 2020, 2:01 PM Jan Mercl <0xj...@gmail.com> wrote:

>
> I expressed my concerns at the issue tracker in 2017 when I still had
> a Github account. That's no more the case, so I'll take the
> opportunity to make a comment here. I still believe that the intent to
> change the behavior the way issue #20733 proposes is [plain] wrong. It
> can turn loops in existing code that do not allocate to loops that
> allocate. That can be avoided by certain compiler optimizations that
> may be already present in [some or all] known Go compilers. But such
> particular optimization is not part of the language specification so
> the runtime performance of some code may differ dramatically between
> [future] compilers.
>
> Go is not, and I hope should not become, a language defined by [any
> particular] implementation.
>

For the purposes of this project, I am planning to remain agnostic as to
the specific proposal which gets implemented to resolve the range loop
issue. That sort of discussion relies heavily on the design principles used
to write Go, and I am not yet informed enough about them to have an opinion.

That said...

AFAIK, the proposal I linked to provides a good overview of the problem,
and, despite a robust discussion, remains a viable implementation
candidate.

I think that it would be best to keep our discussion of the proposed change
part of the relevant GitHub tracker to ensure the conversation can happen
in context of prior discussions. For my part, after having read through the
proposal and the subsequent discussion, I wouldn't have anything to add
other than my opinion.

It is my hope that the GitHub Vet Project has been designed in a way that
the data collected can be useful and interesting regardless of whatever
change is eventually implemented. Unfortunately, the project only has a
chance of success if enough Gophers participate in the crowdsourcing
effort. I hope that the community won't be discouraged from participation
based on the perception that we don't yet have a viable fix for this
problem since, as far as I am aware, that is not the case.

>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CALJzkY_7bdqLzjaKt2EVBiAyznkMj_twQqeb6xJDyu2wQKjUZg%40mail.gmail.com.

Re: [go-nuts] The GitHub Vet Project

[Jan Mercl]

On Sun, Dec 20, 2020 at 8:38 PM K. Alex Mills  wrote:

> During a Q session at this year's GopherCon, some members of the Go 
> language team indicated a willingness to modify the behavior of range loops 
> if we can be reasonably certain that the change will not cause incorrect 
> behavior in current programs.

I expressed my concerns at the issue tracker in 2017 when I still had
a Github account. That's no more the case, so I'll take the
opportunity to make a comment here. I still believe that the intent to
change the behavior the way issue #20733 proposes is [plain] wrong. It
can turn loops in existing code that do not allocate to loops that
allocate. That can be avoided by certain compiler optimizations that
may be already present in [some or all] known Go compilers. But such
particular optimization is not part of the language specification so
the runtime performance of some code may differ dramatically between
[future] compilers.

Go is not, and I hope should not become, a language defined by [any
particular] implementation.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAA40n-V3HRfbqWVwBZ%2BFGEZnMaMyaFLR9sX4yZmZX1h1ccqV-g%40mail.gmail.com.