Re: [sage-devel] Re: Why doesn't #23931 get merged?

[William Stein]

On Sat, Dec 9, 2017 at 5:41 AM Volker Braun  wrote:

> First of all, what sets us apart from most open source projects is that
> the time required for a CI (buildbot) run is about a day. And we produce
> more more tickets than one per day. So unless we massively throw hardware
> at this, we have to collect a couple of tickets for each buildbot run and
> then backtrack if there is a failure.
>

I can help with that...

Email me off list.  I just got a brand new grant-funded machine at UW...


> The biggest flaw of the current process is IMHO that not enough things are
> automatized; Most build failures are actually trivial (like documentation
> doesn't build) and should be caught without human interaction before doing
> a full buildbot run. The problem is that our tests are too flaky to be 100%
> reliable. For example #23391 often causes at least one buildbot to crap
> out. And patchbots often produce spurious errors, which leads to people
> ignoring their output. In part this is because we don't have a lot of
> incentive to hunt down rare failures, most people just want their ticket
> merged and don't run tests 100x. Another sign of things not being automated
> is that tarball updates require human (i.e. my) intervention to upload
> them. And as long as they aren't uploaded they can't be tested.
>
> To address these issues, I/we should:
>
> 1) address the tarball issue, either add a download url to, say,
> checksums.ini and code to auto-download from that url if it can't be found
> on the mirrors. Or set up a web service where tarball updates must be
> submitted to be accessible by build/patchbots (any volunteers for hardware
> / hosting?)
>
> 2) work towards a release script that automatically merges or sets back to
> "needs review" based on tests on a fast machine; This will also increase
> the visibility and pain of random testsuite failures...
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sage-devel+unsubscr...@googlegroups.com.
> To post to this group, send email to sage-devel@googlegroups.com.
> Visit this group at https://groups.google.com/group/sage-devel.
> For more options, visit https://groups.google.com/d/optout.
>
-- 
-- William Stein

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

Re: [sage-devel] Re: Why doesn't #23931 get merged?

[Volker Braun]

First of all, what sets us apart from most open source projects is that the 
time required for a CI (buildbot) run is about a day. And we produce more 
more tickets than one per day. So unless we massively throw hardware at 
this, we have to collect a couple of tickets for each buildbot run and then 
backtrack if there is a failure.

The biggest flaw of the current process is IMHO that not enough things are 
automatized; Most build failures are actually trivial (like documentation 
doesn't build) and should be caught without human interaction before doing 
a full buildbot run. The problem is that our tests are too flaky to be 100% 
reliable. For example #23391 often causes at least one buildbot to crap 
out. And patchbots often produce spurious errors, which leads to people 
ignoring their output. In part this is because we don't have a lot of 
incentive to hunt down rare failures, most people just want their ticket 
merged and don't run tests 100x. Another sign of things not being automated 
is that tarball updates require human (i.e. my) intervention to upload 
them. And as long as they aren't uploaded they can't be tested.

To address these issues, I/we should:

1) address the tarball issue, either add a download url to, say, 
checksums.ini and code to auto-download from that url if it can't be found 
on the mirrors. Or set up a web service where tarball updates must be 
submitted to be accessible by build/patchbots (any volunteers for hardware 
/ hosting?)

2) work towards a release script that automatically merges or sets back to 
"needs review" based on tests on a fast machine; This will also increase 
the visibility and pain of random testsuite failures...


-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

Re: [sage-devel] Re: Why doesn't #23931 get merged?

[Maarten Derickx]

On Friday, 8 December 2017 10:26:06 UTC+1, Erik Bray wrote:
>
> On Thu, Dec 7, 2017 at 6:26 PM, William Stein  > wrote: 
> > On Thu, Dec 7, 2017 at 7:02 AM, Frédéric Chapoton  > wrote: 
> >> We have currently 115 such positive-reviewed tickets waiting for 
> inclusion, 
> > 
> > Maybe the release manager could use some assistance? Or we could 
> > rotate to more release managers like we used to do for many years... 
>
> Two things that would be helpful and more like "most" open source 
> projects (quotes because I have no data to back this up, but from 
> experience): 
>
> a) Have a normal "bleeding edge" master branch into which all accepted 
> changes are immediately merged--no waiting (for developers) to get a 
> development "release" before getting new changes in. 
>
>

The branch could already be programmatically created by a bot based on the 
positive review status of trac. I think this could be of huge value in 
terms of seeing wether your code cleanly merges with already positively 
reviewed code, and for testing purposes as well. However if something like 
this is done it should be made very clear to everybody to not base any code 
on this branch (only merge your code with it on a separate branch from time 
to time to see wether there are merge conflicts and passes all test after 
the merge).

To actually make code get included faster in some branch that at some point 
will be the next release I think there indeed needs to be some extra 
"acceptance" mechanism on top of the current positive review one. Since the 
current positive review does not provide enough guarantees that a ticket is 
actually good enough that we as a community want to commit to actually get 
that ticket into the next release, and timely fix the unsuspected blocker 
tickets it might introduce. I do like the idea of having certain trusted 
core developers helping to distribute the work of the release manager, but 
they would have to be really good and trustworthy enough so that they 
actually save work and not cause extra work and frustration.

 

> b) More people who can merge into master--while it makes sense to have 
> one "release manager" responsible for keeping a schedule and ensuring 
> stability of the release (including occasionally putting on hold big 
> changes that might hold up a release), it also makes sense to have 
> multiple delegates of approving and merging changes in different areas 
> of the code, helping to set priorities, and triaging.  This can be 
> entrusted to multiple people with the help of guidelines to make sure 
> they check all the right boxes when approving a change.  For example: 
>
> http://docs.astropy.org/en/stable/development/workflow/maintainer_workflow.html
>  
> 
>  
>  .  CPython has a similar document I've seen before, but I can't find 
> it at the moment, as do many other large projects.  I believe Sage 
> might too... 
>

I also think this blogpost has a very nice description on how to deal with 
git and different releases:  
http://nvie.com/posts/a-successful-git-branching-model/ . The develop 
branch there is playing the role of your "bleeding edge master", and the 
master branch is only used for official releases. But the underlying 
principle is the same. There could be some core developers who have right 
to commit things to the develop aka "bleeding edge master" branch.

To make it match also with all the beta's we do the master branch there 
should maybe split up into a "beta-master" and a "master" branch. With the 
"beta-master" containing all releases including beta's rc's and the 
"master" branch just containing the actual releases like 8.0 and 8.1.

Depending on how much time it would cost Volker to already start the 
develop branch for the next release while waiting for the blockers to get 
fixed in the rc of the current release. Having both of these exist at the 
same time could solve some of the frustration of things not getting merged. 
Although I think in a certain sense the not having other things get merged 
could also be seen as a feature instead of an annoyance, since it provides 
a good natural incentive for fixing the blockers.

That being said, I feel that the people who have actually have been release 
manager should have a larger say in this then me, since they actually know 
how it is to do all the hard and dirty work.


 

>
> >> see 
> >> 
> >> 
> https://trac.sagemath.org/query?status=positive_review&milestone=!sage-duplicate%2Finvalid%2Fwontfix&group=milestone&col=id&col=summary&col=type&col=component&col=changetime&col=author&col=reviewer&col=dependencies&report=40&desc=1&order=changetime
>  
> >> 
> >> 
> >> Le jeudi 7 décembre 2017 15:39:07 UTC+1, Friedrich Wiemer a écrit : 
> >>> 
> >>> The ticket #23931 has a positive review for quite some time and I'm 
> not 
> >>> aware o

Re: [sage-devel] Re: Why doesn't #23931 get merged?

[Jeroen Demeyer]

On 2017-12-08 15:56, Erik Bray wrote:

I don't understand the point about the "buildbot
workflow" but it sounds like that might be a problem with that
workflow.


This was certainly the bottleneck when I was release manager. But I 
cannot speak for the current release manager.



On most projects accepted changes are merged directly
into master


Obviously. And I would argue that this is actually the case for Sage: 
it's just that accepting changes takes a long time. That's the problem 
that you should focus on.


--
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

Re: [sage-devel] Re: Why doesn't #23931 get merged?

[Jeroen Demeyer]

On 2017-12-08 15:56, Erik Bray wrote:

Right now the way it (seems to) work is new tickets get assigned to
the next release milestone.  That release is then
made...whenever...regardless of what tickets are still open or not,
and then all those tickets (typically) remain in old milestones for
completed releases.  This is nobody's fault; it just doesn't seem to
be part of the practice or culture of the project to really use
milestones in a consistent manner.


What you are saying is totally true, but I just don't see it as a 
problem. Probably we should be more explicit about it and just stop 
using the "milestone" field on Trac.


However, I don't think that there is anything fundamentally wrong with 
"That release is then made...whenever...regardless of what tickets are 
still open or not". One exception is blocker tickets and the release 
manager does consider those: a blocker ticket will indeed block a 
release. But for non-blocker tickets, it doesn't really matter much in 
which release those tickets are merged.


--
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

Re: [sage-devel] Re: Why doesn't #23931 get merged?

[Jeroen Demeyer]

Your email was quite long so I'll just reply bit by bit...

On 2017-12-08 15:56, Erik Bray wrote:

What you really
want is to have some set of "core maintainers", possibly with
different responsibilities for different areas of the code depending
on their expertise.  And a sign-off from one of them (plus positive
results of continuous integration and other checklist items) means the
change should be merged.


De facto, we already have a set of core maintainers, each looking after 
a specific part of the code. For example, I would call you one of the 
core maintainers for build issues and Python 3. We could formalize this, 
but again: I don't think that there is a problem to be solved here.


The thing you put in parentheses "(plus positive results of continuous 
integration and other checklist items)" is the difficult part. In my 
release manager days, that was certainly the bottleneck.



And
a single release manager can't be a subject-matter expert on all
topics either, nor have a 100% top-down view of what to
prioritize


Is that needed? I would argue that it's not the job of the release 
manager to understand the subject matter of the individual tickets. He 
just needs to collect tickets, make a release and test it.


--
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

Re: [sage-devel] Re: Why doesn't #23931 get merged?

[Erik Bray]

On Fri, Dec 8, 2017 at 1:59 PM, Jeroen Demeyer  wrote:
> On 2017-12-08 10:26, Erik Bray wrote:
>>
>> a) Have a normal "bleeding edge" master branch into which all accepted
>> changes are immediately merged
>
>
> Now everything boils down to defining "accepted changes".

Of course--this isn't trivial, but there's going to be some degree of
subjectivity involved no matter what.  Right now (it seems?) just
about anyone can give a ticket "positive review".  What you really
want is to have some set of "core maintainers", possibly with
different responsibilities for different areas of the code depending
on their expertise.  And a sign-off from one of them (plus positive
results of continuous integration and other checklist items) means the
change should be merged.

Right now there is really just one "core maintainer" in the sense I
described above, and that's the "release manager", who in Sage is
currently wearing a lot more hats than merely release management.  And
a single release manager can't be a subject-matter expert on all
topics either, nor have a 100% top-down view of what to
prioritize--certainly not without input.  This is really a massive
bottleneck to development iMO.

> Just having a ticket set to positive review does not mean that much. It is a
> bad idea to merge them in your bleeding edge branch because those tickets
> might not get merged in master (because it breaks something or it conflicts
> with other ticket). So you need some mechanism anyway to check tickets,
> which is the current situation.

I'm not sure what you mean here.  I'm saying "master" is the
bleeding-edge.  On most projects accepted changes are merged directly
into master--they've been checked to work against the current HEAD of
the master branch.  This *can* move quickly, which means some proposed
changes can become outdated quickly, but only in cases where there is
a conflict.  This does not happen with every change.  It also happens
less often if changes are being approved and merged more quickly,
which does not happen with Sage because again there is a single
bottleneck.

> When I was release manager, checking tickets was the most time-consuming
> step in the release process. And a rather large fraction of tickets (say,
> about 10%) didn't pass buildbot testing. Now that we have the patchbot in
> reasonable shape, that is hopefully better now.

It can be time-consuming, yes.  However, having other trusted people
who know what they're doing help with that task alleviates quite a bit
of burden; something I know from experience.  Knowing that Trusted
Person A who is a subject matter expert on the topic of the change has
signed off on a pull request, for example, increases my confidence
enough that I will scrutinize the change less myself.  For the
pedants, a number of checks can be scripted as well (even checked by
bots--see CPython's excellent fleet of bots that have been deployed on
their GitHub project).

> Once all tickets pass buildbot testing, actually making a release is easy.
>
> It would be good to have some feedback from the current release manager to
> identify what the bottleneck currently is.
>
>> More people who can merge into master
>
>
> I actually like the idea of having a single release manager whose job it is
> to do exactly this. With more people, things are no doubt going to become
> sloppier. Also, it is harder to organize/synchronize the buildbot workflow
> with many people.

See above.  I think this attitude that one and only one person can
know what they're doing with respect to accepting changes is pretty
presumptuous.  I don't understand the point about the "buildbot
workflow" but it sounds like that might be a problem with that
workflow.

>> helping to set priorities, and triaging.
>
>
> This *could* be done by other people apart from the release manager, but I
> don't think that there is a problem to be solved here.

Yes and no.  Right now it doesn't seem to get done very much.  The
"milestone" field on Trac seems all but useless.  If I put a ticket in
the "sage-8.1" milestone it means I think that ticket would be well
suited to go into that release.  That doesn't mean the release must be
held up for every ticket assigned to the milestone--oftentimes an
issue will be assigned to a milestone, but won't be possible to
resolve for that release due to time constraints of the interested
parties, unresolved questions, etc.  In this case it can be deferred,
either to a future release milestone, or a more "pie-in-the-sky"
category.  But this should be done explicitly during a pre-release
triaging period in which others can participate, to decide what we
really think will or will not go in a given release.

Right now the way it (seems to) work is new tickets get assigned to
the next release milestone.  That release is then
made...whenever...regardless of what tickets are still open or not,
and then all those tickets (typically) remain in old milestones for
completed releases.  This is nobody's fault; it ju

Re: [sage-devel] Re: Why doesn't #23931 get merged?

[Jeroen Demeyer]

On 2017-12-08 10:26, Erik Bray wrote:

a) Have a normal "bleeding edge" master branch into which all accepted
changes are immediately merged


Now everything boils down to defining "accepted changes".

Just having a ticket set to positive review does not mean that much. It 
is a bad idea to merge them in your bleeding edge branch because those 
tickets might not get merged in master (because it breaks something or 
it conflicts with other ticket). So you need some mechanism anyway to 
check tickets, which is the current situation.


When I was release manager, checking tickets was the most time-consuming 
step in the release process. And a rather large fraction of tickets 
(say, about 10%) didn't pass buildbot testing. Now that we have the 
patchbot in reasonable shape, that is hopefully better now.


Once all tickets pass buildbot testing, actually making a release is easy.

It would be good to have some feedback from the current release manager 
to identify what the bottleneck currently is.



More people who can merge into master


I actually like the idea of having a single release manager whose job it 
is to do exactly this. With more people, things are no doubt going to 
become sloppier. Also, it is harder to organize/synchronize the buildbot 
workflow with many people.



helping to set priorities, and triaging.


This *could* be done by other people apart from the release manager, but 
I don't think that there is a problem to be solved here.


--
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

Re: [sage-devel] Re: Why doesn't #23931 get merged?

[Erik Bray]

On Thu, Dec 7, 2017 at 6:26 PM, William Stein  wrote:
> On Thu, Dec 7, 2017 at 7:02 AM, Frédéric Chapoton  
> wrote:
>> We have currently 115 such positive-reviewed tickets waiting for inclusion,
>
> Maybe the release manager could use some assistance? Or we could
> rotate to more release managers like we used to do for many years...

Two things that would be helpful and more like "most" open source
projects (quotes because I have no data to back this up, but from
experience):

a) Have a normal "bleeding edge" master branch into which all accepted
changes are immediately merged--no waiting (for developers) to get a
development "release" before getting new changes in.

b) More people who can merge into master--while it makes sense to have
one "release manager" responsible for keeping a schedule and ensuring
stability of the release (including occasionally putting on hold big
changes that might hold up a release), it also makes sense to have
multiple delegates of approving and merging changes in different areas
of the code, helping to set priorities, and triaging.  This can be
entrusted to multiple people with the help of guidelines to make sure
they check all the right boxes when approving a change.  For example:
http://docs.astropy.org/en/stable/development/workflow/maintainer_workflow.html
 .  CPython has a similar document I've seen before, but I can't find
it at the moment, as do many other large projects.  I believe Sage
might too...

>> see
>>
>> https://trac.sagemath.org/query?status=positive_review&milestone=!sage-duplicate%2Finvalid%2Fwontfix&group=milestone&col=id&col=summary&col=type&col=component&col=changetime&col=author&col=reviewer&col=dependencies&report=40&desc=1&order=changetime
>>
>>
>> Le jeudi 7 décembre 2017 15:39:07 UTC+1, Friedrich Wiemer a écrit :
>>>
>>> The ticket #23931 has a positive review for quite some time and I'm not
>>> aware of any changes left for it, so my humble question is: Is there a
>>> reason this ticket isn't merged yet?

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

[sage-devel] Re: Why doesn't #23931 get merged?

[Volker Braun]

I'm currently preparing 8.1 which was held up for a while because of some 
last-minute fixes. When that is out we'll be back to the usual pace.

On Thursday, December 7, 2017 at 3:39:07 PM UTC+1, Friedrich Wiemer wrote:
>
> The ticket #23931 has a positive 
> review for quite some time and I'm not aware of any changes left for it, so 
> my humble question is: Is there a reason this ticket isn't merged yet?
> 
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

Re: [sage-devel] Re: Why doesn't #23931 get merged?

[William Stein]

On Thu, Dec 7, 2017 at 7:02 AM, Frédéric Chapoton  wrote:
> We have currently 115 such positive-reviewed tickets waiting for inclusion,

Maybe the release manager could use some assistance? Or we could
rotate to more release managers like we used to do for many years...

> see
>
> https://trac.sagemath.org/query?status=positive_review&milestone=!sage-duplicate%2Finvalid%2Fwontfix&group=milestone&col=id&col=summary&col=type&col=component&col=changetime&col=author&col=reviewer&col=dependencies&report=40&desc=1&order=changetime
>
>
> Le jeudi 7 décembre 2017 15:39:07 UTC+1, Friedrich Wiemer a écrit :
>>
>> The ticket #23931 has a positive review for quite some time and I'm not
>> aware of any changes left for it, so my humble question is: Is there a
>> reason this ticket isn't merged yet?
>
> --
> You received this message because you are subscribed to the Google Groups
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sage-devel+unsubscr...@googlegroups.com.
> To post to this group, send email to sage-devel@googlegroups.com.
> Visit this group at https://groups.google.com/group/sage-devel.
> For more options, visit https://groups.google.com/d/optout.



-- 
William (http://wstein.org)

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

[sage-devel] Re: Why doesn't #23931 get merged?

[Friedrich Wiemer]

OK, so its basically because lack of time?
Is this something a newbie contributer like me can help with?


Am Donnerstag, 7. Dezember 2017 16:02:56 UTC+1 schrieb Frédéric Chapoton:
>
> We have currently 115 such positive-reviewed tickets waiting for 
> inclusion, see
>
>
> https://trac.sagemath.org/query?status=positive_review&milestone=!sage-duplicate%2Finvalid%2Fwontfix&group=milestone&col=id&col=summary&col=type&col=component&col=changetime&col=author&col=reviewer&col=dependencies&report=40&desc=1&order=changetime
>
> Le jeudi 7 décembre 2017 15:39:07 UTC+1, Friedrich Wiemer a écrit :
>>
>> The ticket #23931 has a positive 
>> review for quite some time and I'm not aware of any changes left for it, so 
>> my humble question is: Is there a reason this ticket isn't merged yet?
>> 
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

[sage-devel] Re: Why doesn't #23931 get merged?

[Frédéric Chapoton]

We have currently 115 such positive-reviewed tickets waiting for inclusion, 
see

https://trac.sagemath.org/query?status=positive_review&milestone=!sage-duplicate%2Finvalid%2Fwontfix&group=milestone&col=id&col=summary&col=type&col=component&col=changetime&col=author&col=reviewer&col=dependencies&report=40&desc=1&order=changetime

Le jeudi 7 décembre 2017 15:39:07 UTC+1, Friedrich Wiemer a écrit :
>
> The ticket #23931 has a positive 
> review for quite some time and I'm not aware of any changes left for it, so 
> my humble question is: Is there a reason this ticket isn't merged yet?
> 
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.