Re: Updated Sourceware infrastructure plans

[Mark Wielaard]

Hi Joseph,

On Thu, 2024-04-18 at 15:56 +, Joseph Myers wrote:
> On Thu, 18 Apr 2024, Mark Wielaard wrote:
> 
> > But we like to get more feedback on what people really think a
> > "pull-request" style framework should look like. We used to have a
> > gerrit setup which wasn't really popular. And we already have a
> > sourcehut mirror that can be used to turn your "pull-requests" into a
> > git send-email style submission (without having to setup any
> > email/smtp yourself): https://sr.ht/~sourceware/
> 
> The xz backdoor showed up one issue with some implementations of 
> pull-request systems: GitHub removed access to the repository, and with it 
> access to the past pull requests, so disrupting investigation into the 
> sequence of bad-faith contributions.

Agreed. I tried to analyze the valgrind issues after the fact (we
clearly missed them before, there were warning, but they were fixed so
quickly we didn't really look into them as we should have). And it was
a bit difficult because the github repository had disappeared. But
luckily the project did have a "real" git repo:
https://git.tukaani.org/
This obviously didn't contain any "pull requests" but I am not sure
they were used on the xz github mirror. Does github require pull
requests to keep around? What if someone closes/removes their own
fork/repo/account, do the commits transfer to the project?

>   I suggest that a basic principle for 
> such a system is that it should be *easy* to obtain and maintain a local 
> copy of the history of all pull requests.  That includes all versions of a 
> pull request, if it gets rebased, and all versions of comments, if the 
> system allows editing comments.

So in a somewhat crude form we now have that with our email workflow.
In theory every patch is posted and reviewed on one of the patches
mailinglists and the public-inbox instance at
https://inbox.sourceware.org/ allows you to git clone the whole archive
for local inspection.

>   A system that uses git as the source of 
> truth for all the pull request data and has refs through which all this 
> can be located (with reasonably straightforward, documented formats for 
> the data, not too closely tied to any particular implementation of a 
> pull-request system), so that a single clone --mirror has all the data, 
> might be suitable (people have worked on ensuring git scales well with 
> very large numbers of refs, which you'd probably get in such a system 
> storing all the data in git);

Yes, git is pretty nice for storing lots of variants of somewhat
identical sources/texts. But this also seems to imply that when we
offer a system to store "contributor" git trees/forks of projects to
easily create "pull requests" then we can never remove such users/forks
and must disallow rebasing any trees that have been "submitted".

That can probably be done, but it is different from what we now require
from user or devel branches in our git repos. Where we do allow users
to delete their branches and devel branches can be rebased. Should such
branches also become "immutable"?

Cheers,

Mark

Re: Updated Sourceware infrastructure plans

[Joseph Myers via Gcc]

On Mon, 22 Apr 2024, Mark Wielaard wrote:

> >   A system that uses git as the source of 
> > truth for all the pull request data and has refs through which all this 
> > can be located (with reasonably straightforward, documented formats for 
> > the data, not too closely tied to any particular implementation of a 
> > pull-request system), so that a single clone --mirror has all the data, 
> > might be suitable (people have worked on ensuring git scales well with 
> > very large numbers of refs, which you'd probably get in such a system 
> > storing all the data in git);
> 
> Yes, git is pretty nice for storing lots of variants of somewhat
> identical sources/texts. But this also seems to imply that when we
> offer a system to store "contributor" git trees/forks of projects to
> easily create "pull requests" then we can never remove such users/forks
> and must disallow rebasing any trees that have been "submitted".

For example, GitHub has some version of the source branch for a pull 
request under refs/pull/ in the target respository - that doesn't rely on 
the source branch or repository staying around.  However, that's only one 
version - it doesn't work so well when the source branch is rebased 
(though GitHub itself is reported to keep all forks of a repository in a 
single repository internally, rarely garbage collected, so the previous 
versions probably remain there, just not accessible from any ref).  But 
you could certainly have a convention for ref naming that ensures all 
versions of a PR are available even when it's rebased.  Things like the 
"git evolve" proposal  could also be 
relevant (maybe that particular proposal wasn't intended for the goal of 
ensuring all submitted versions of a change remain permanently available, 
but at least it's dealing with a similar problem - and the more you have a 
standard way of representing this kind of information in git, rather than 
something very specific to a particular system built on top of git, the 
better).

-- 
Joseph S. Myers
josmy...@redhat.com

Re: Updated Sourceware infrastructure plans

[Tom Tromey]

> "Frank" == Frank Ch Eigler  writes:

>> [...]  I suggest that a basic principle for such a system is that it
>> should be *easy* to obtain and maintain a local copy of the history
>> of all pull requests.  That includes all versions of a pull request,
>> if it gets rebased, and all versions of comments, if the system
>> allows editing comments.  A system that uses git as the source of
>> truth for all the pull request data and has refs [...]

Frank> Do you know of a system with these characteristics?

Based on:

https://gerrit-review.googlesource.com/Documentation/dev-design.html#_notedb

... it sounds like this is what gerrit does.

Tom

Re: warnings and warnings

[Richard Biener via Gcc]

On Sun, Apr 21, 2024 at 11:06 PM James K. Lowden
 wrote:
>
> I have two simple questions, I hope!
>
> 1.  Is there a set of flags that, when compiling gcc, is meant to
> produce no warnings?  I get a surfeit of warnings with my particular
> favorite options.

-w is supposed to do that

> 2.  Are the libgcc functions warning_at() and error_at() intended for
> use by all front-ends? As of now, our COBOL front-end formats its own
> messages with fprintf(3).  I would like to emulate normal gcc behavior,
> where warnings can be turned on and off, and (especially) elevated to
> errors with -Werror.  I'm guessing we'd gain access to that
> functionality automatically if we engaged with the standard diagnositic
> framework, if there is one.
>
> I'm a little doubtful these are the keys to that kingdom, though.  The
> comments regarding location_t and the location database seem very
> C-specific.

Yes, the tools from diagnostic{-core,}.h are usable from all frontends.
The location_t is a libcpp thing but it's the location how it is tracked
through the middle-end and also for debug information generation
so you might want to at least emulate that even when you are not
using the preprocessor.  The important part is to manage a line-map
here.

Richard.

> --jkl

Re: Updated Sourceware infrastructure plans

[Frank Ch. Eigler via Gcc]

Hi -

> Would it be possible for gitsigur to support signing commits with ssh
> keys as well as gpg? Git supports this, and it's much easier for
> everybody than having to set up gpg. [...]

It would save some effort, but OTOH plenty of people have gpg keys
too, and the common desktop key agents support both.

> We already need an SSH key on sourceware.org to push to Git, so all
> those public keys could be treated as trusted (via git config
> gpg.ssh.allowedSignersFile). [...]

One difference is that gitsigur aims to prevent impersonation, by
tying the recorded committer to a designated set of keys for that
committer.  The git builtin ssh-signing gadget doesn't attempt this.
But maybe just a small matter of wrapping might do the job.

Filed https://sourceware.org/bugzilla/show_bug.cgi?id=31670 .

> I'm already signing my GCC commits that way, without needing to use
> gpg or gitsigur:

Great, keep it up!  Nothing has been stopping people from signing
their commits any way they like, including even more complex ways like
sigstore.  gitsigur verification is not enabled (even in permissive
mode) at all for gcc at this time.

> commit 7c2a9dbcc2c1cb1563774068c59d5e09edc59f06 [r14-10008-g7c2a9dbcc2c1cb]
> Good "git" signature for jwak...@redhat.com with RSA key
> SHA256:8rFaYhDWn09c3vjsYIg2JE9aSpcxzTnCqajoKevrUUo

Thanks, this will help test a prototype later on.

- FChE

Re: Updated Sourceware infrastructure plans

[Jason Merrill via Gcc]

On Mon, Apr 22, 2024 at 11:42 AM Tom Tromey  wrote:

> > "Frank" == Frank Ch Eigler  writes:
>
> >> [...]  I suggest that a basic principle for such a system is that it
> >> should be *easy* to obtain and maintain a local copy of the history
> >> of all pull requests.  That includes all versions of a pull request,
> >> if it gets rebased, and all versions of comments, if the system
> >> allows editing comments.  A system that uses git as the source of
> >> truth for all the pull request data and has refs [...]
>
> Frank> Do you know of a system with these characteristics?
>
> Based on:
>
>
> https://gerrit-review.googlesource.com/Documentation/dev-design.html#_notedb
>
> ... it sounds like this is what gerrit does.
>

Someone mentioned earlier that gerrit was previously tried unsuccessfully.
I think this is a common pattern in GCC at least: someone has an idea for a
workflow improvement, and gets it working, but it isn't widely adopted.  I
think this is a problem with the "if you build it he will come" model
rather than with any particular technology; people are more or less
comfortable with their current workflow and uninclined to experiment with
someone else's new thing, even if it could eventually be a big improvement.

I think that the office hours, for both sourceware and the toolchain, offer
a better path for building enthusiasm about a new direction.

Is gerrit still installed?

Jason

Re: Updated Sourceware infrastructure plans

[Simon Marchi via Gcc]

On 2024-04-22 22:55, Jason Merrill via Overseers wrote:
> On Mon, Apr 22, 2024 at 11:42 AM Tom Tromey  wrote:
> 
>>> "Frank" == Frank Ch Eigler  writes:
>>
 [...]  I suggest that a basic principle for such a system is that it
 should be *easy* to obtain and maintain a local copy of the history
 of all pull requests.  That includes all versions of a pull request,
 if it gets rebased, and all versions of comments, if the system
 allows editing comments.  A system that uses git as the source of
 truth for all the pull request data and has refs [...]
>>
>> Frank> Do you know of a system with these characteristics?
>>
>> Based on:
>>
>>
>> https://gerrit-review.googlesource.com/Documentation/dev-design.html#_notedb
>>
>> ... it sounds like this is what gerrit does.
>>
> 
> Someone mentioned earlier that gerrit was previously tried unsuccessfully.
> I think this is a common pattern in GCC at least: someone has an idea for a
> workflow improvement, and gets it working, but it isn't widely adopted.  I
> think this is a problem with the "if you build it he will come" model
> rather than with any particular technology; people are more or less
> comfortable with their current workflow and uninclined to experiment with
> someone else's new thing, even if it could eventually be a big improvement.

Agreed.

Gerrit has many nice features, but using it would require doing some
compromises over some things that some community members consider very
important.  Would have to give up some things we take for granted today,
such as being able to interact 100% by email.  But staying with the
current way of working because we can't find another way of working that
checks 100% of the checkboxes also has an opportunity cost.  I doubt
we'll ever find a system that checks absolutely all the checkboxes, but
that doesn't mean that the current system is the best.

> I think that the office hours, for both sourceware and the toolchain, offer
> a better path for building enthusiasm about a new direction.

As someone who pushed to try Gerrit back then, I'd be happy to chat at
the next office hours (if you remind me).

> Is gerrit still installed?

Hum yes actually, you can check `gnutoolchain-gerrit dot osci dot io`.
But it should really be taken down, it's not responsible to leave an
unattended outdated web service like that.

Simon

Re: Updated Sourceware infrastructure plans

[Tom Tromey]

Jason> Someone mentioned earlier that gerrit was previously tried
Jason> unsuccessfully.

We tried it and gdb and then abandoned it.  We tried to integrate it
into the traditional gdb development style, having it send email to
gdb-patches.  I found these somewhat hard to read and in the end we
agreed not to use it.

I've come around again to thinking we should probably abandon email
instead.  For me the main benefit is that gerrit has patch tracking,
unlike our current system, where losing patches is fairly routine.

Jason> I think this is a common pattern in GCC at least: someone has an
Jason> idea for a workflow improvement, and gets it working, but it
Jason> isn't widely adopted.

It essentially has to be mandated, IMO.

For GCC this seems somewhat harder since the community is larger, so
there's more people to convince.

Tom

Re: Updated Sourceware infrastructure plans

[Jason Merrill via Gcc]

On Mon, Apr 22, 2024 at 11:24 PM Tom Tromey  wrote:

> Jason> Someone mentioned earlier that gerrit was previously tried
> Jason> unsuccessfully.
>
> We tried it and gdb and then abandoned it.  We tried to integrate it
> into the traditional gdb development style, having it send email to
> gdb-patches.  I found these somewhat hard to read and in the end we
> agreed not to use it.
>
> I've come around again to thinking we should probably abandon email
> instead.  For me the main benefit is that gerrit has patch tracking,
> unlike our current system, where losing patches is fairly routine.
>

Indeed.  Though Patchwork is another option for patch tracking, that glibc
seem to be having success with.

Jason> I think this is a common pattern in GCC at least: someone has an
> Jason> idea for a workflow improvement, and gets it working, but it
> Jason> isn't widely adopted.
>
> It essentially has to be mandated, IMO.
>
> For GCC this seems somewhat harder since the community is larger, so
> there's more people to convince.
>

Absolutely, but now with the office hours it seems more feasible to build
momentum (or see that there isn't enough support) without having to wait
until Cauldron.

Jason

Re: Updated Sourceware infrastructure plans

[Ian Lance Taylor]

Tom Tromey via Overseers  writes:

> Jason> Someone mentioned earlier that gerrit was previously tried
> Jason> unsuccessfully.
>
> We tried it and gdb and then abandoned it.  We tried to integrate it
> into the traditional gdb development style, having it send email to
> gdb-patches.  I found these somewhat hard to read and in the end we
> agreed not to use it.

Current Gerrit e-mails are pretty nice, with a nice diff of the change.
And patches can be submitted entirely via git, which is not the same as
today but should be acceptable for almost all contributors.  What
doesn't work in Gerrit, as far as I know, is a pure e-mail based
workflow for maintainers.  To approve a patch, maintainers have to go to
a web site and click a button, or they have to run a command line tool
("ssh  gerrit review").


> I've come around again to thinking we should probably abandon email
> instead.  For me the main benefit is that gerrit has patch tracking,
> unlike our current system, where losing patches is fairly routine.

You can lose patches in Gerrit quite easily, but at least there is a
dashboard showing all the ones you lost.

I'm definitely a Gerrit fan.

Ian