Re: Proposal for an inbound2 branch
On 2013-05-05 1:20 PM, Gregory Szorc wrote: On 5/4/2013 9:59 PM, Justin Dolske wrote: On 5/1/13 8:41 PM, Ehsan Akhgari wrote: Another disadvantage of project branches in addition to the ones mentioned before is that it limits/delays the amount of testing that you can get on Nightly and from all of the developers who run things like fuzzers on ours code. Not everyone's project has enough manpower to get that level of testing on a project branch before their stuff gets merged to central/inbound. I personally cannot imagine doing my development in a project branch silo and deprive myself of the huge advantage that this kind of testing brings about. That all depends on how one uses a "project branch". If you use a project branch as a traditional staging area for work that is large/complex and merge it infrequently, then yes. Those are absolutely real problems. But if you use a project branch as just a frequently-merged integration branch, then I don't see these things as being problems. I think that's what gps is proposing -- not the old-school usage of project branches (maybe we need a new term?). Such branches would be almost exactly the same thing as mozilla-inbound, and we don't worry about those problems on inbound. We did exactly this in the past with JS and Services (and to a lesser extent with fx-team)... We'd just call them inbound-js and inbound-services now. Frequently-merged integration branches are exactly what I'm proposing. Naming is hard. What would be the benefit of that compared to Ryan's proposal? The downsides are clear: more branches to be sheriffed, more merge pain, and worse usage of our infra capacity... [AIUI, those teams stopped because inbound was working so well -- why spend time running your own branch when m-i is just as good and requires no effort? The twist now is that m-i is a victim of it's own success, and so we should think about making the costs more directly visible to project teams.] I actually started reusing services-central as an inbound-like branch because I was so frustrated dealing with inbound closures. It's merged by not-the-sheriffs, does not get the normal starring love (I don't mind starring my own pushes), and is merged less often. But, it works for our needs. I think Ryan's proposal addresses the inbound closure problem. Do you see more problems that need to be addressed by using s-c as an integration branch? I doubt that it is a scalable approach to ask every team to maintain their own branches this way. Ehsan ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposal for an inbound2 branch
On 5/4/2013 9:59 PM, Justin Dolske wrote: On 5/1/13 8:41 PM, Ehsan Akhgari wrote: Another disadvantage of project branches in addition to the ones mentioned before is that it limits/delays the amount of testing that you can get on Nightly and from all of the developers who run things like fuzzers on ours code. Not everyone's project has enough manpower to get that level of testing on a project branch before their stuff gets merged to central/inbound. I personally cannot imagine doing my development in a project branch silo and deprive myself of the huge advantage that this kind of testing brings about. That all depends on how one uses a "project branch". If you use a project branch as a traditional staging area for work that is large/complex and merge it infrequently, then yes. Those are absolutely real problems. But if you use a project branch as just a frequently-merged integration branch, then I don't see these things as being problems. I think that's what gps is proposing -- not the old-school usage of project branches (maybe we need a new term?). Such branches would be almost exactly the same thing as mozilla-inbound, and we don't worry about those problems on inbound. We did exactly this in the past with JS and Services (and to a lesser extent with fx-team)... We'd just call them inbound-js and inbound-services now. Frequently-merged integration branches are exactly what I'm proposing. Naming is hard. [AIUI, those teams stopped because inbound was working so well -- why spend time running your own branch when m-i is just as good and requires no effort? The twist now is that m-i is a victim of it's own success, and so we should think about making the costs more directly visible to project teams.] I actually started reusing services-central as an inbound-like branch because I was so frustrated dealing with inbound closures. It's merged by not-the-sheriffs, does not get the normal starring love (I don't mind starring my own pushes), and is merged less often. But, it works for our needs. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposal for an inbound2 branch
On 5/1/13 8:41 PM, Ehsan Akhgari wrote: Another disadvantage of project branches in addition to the ones mentioned before is that it limits/delays the amount of testing that you can get on Nightly and from all of the developers who run things like fuzzers on ours code. Not everyone's project has enough manpower to get that level of testing on a project branch before their stuff gets merged to central/inbound. I personally cannot imagine doing my development in a project branch silo and deprive myself of the huge advantage that this kind of testing brings about. That all depends on how one uses a "project branch". If you use a project branch as a traditional staging area for work that is large/complex and merge it infrequently, then yes. Those are absolutely real problems. But if you use a project branch as just a frequently-merged integration branch, then I don't see these things as being problems. I think that's what gps is proposing -- not the old-school usage of project branches (maybe we need a new term?). Such branches would be almost exactly the same thing as mozilla-inbound, and we don't worry about those problems on inbound. We did exactly this in the past with JS and Services (and to a lesser extent with fx-team)... We'd just call them inbound-js and inbound-services now. [AIUI, those teams stopped because inbound was working so well -- why spend time running your own branch when m-i is just as good and requires no effort? The twist now is that m-i is a victim of it's own success, and so we should think about making the costs more directly visible to project teams.] Justin ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposal for an inbound2 branch
L. David Baron wrote: > On Saturday 2013-04-27 08:26 +1000, Nicholas Nethercote wrote: > > If I have a patch ready to land when inbound closes, what would be > > the sequence of steps that I need to do to land it on inbound2? > > Would I need to have an up-to-date inbound2 clone and transplant > > the patch across? Or is it possible to push from an inbound clone? > > For what it's worth, what I'd do is qpop the patch, pull the tip of > inbound2 into my inbound clone and update to tip, qpush and qfin the > patch, and then "hg out -rtip inbound2" (and after checking it's > right), "hg push -rtip inbound2". I would: hg pull inbound2 hg update -c tip (not always correct, but usually) hg graft The horrible thing about qpop and qpush is that they deal with conflicts very, very poorly, whereas hg graft, hg rebase, and hg pull --rebase allow me to use my three-way merge tool to resolve conflicts. That ends up being a big time saver and IMO it is also much less error-prone. Also, I am not sure "-rtip" is the safest thing to use when you have a multi-headed local repo. I always use "-r ." myself. Cheers, Brian ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposal for an inbound2 branch
> Given the whole point of this thread is about how unreliable inbound is, why > are people trying to develop against it? You still need a copy of inbound to rebase your patches against when pushing. Whatever your personal opinions about git happen to be, I don't think a "git doesn't need a copy of inbound" approach is going to fly. We're simply too invested in git at this point to be OK adding more circumstances under which we have to say "well, if you want to do /that/, you need to use hg." ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposal for an inbound2 branch
Ehsan Akhgari wrote: On 2013-05-02 9:29 AM, Neil wrote: Ehsan Akhgari wrote: Multiheaded repos are evil. They're very hard to deal with, and I don't think that buildbot can deal with them properly. Also, they cannot be represented in git, which means that they will break the git mirror. Why does mozilla-inbound need a git mirror? Because a lot of our developers are using git locally, and the development of Firefox OS relies on working git mirrors of a number of branches, including inbound. Given the whole point of this thread is about how unreliable inbound is, why are people trying to develop against it? -- Warning: May contain traces of nuts. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposal for an inbound2 branch
On 5/3/2013 12:20 AM, Dirkjan Ochtman wrote: On Fri, May 3, 2013 at 8:43 AM, Ehsan Akhgari wrote: In git, the only way to have something in your history is for it to be reachable by a ref (for example, a branch name). When you convert an hg repo with a multi-headed branch to git, you can only choose to represent one of those heads by a branch name, and the other head will be left stranded, which means that it cannot be used normally (for example you cannot push it when you push your branch somewhere) and git gc will remove it from your repository since it thinks that it's an unneeded commit. It seems like a hg bookmark could be used to keep a pointer to the separate heads. This should create the refs you need when using hg-git. (FWIW, I fully agree that keeping two anonymous heads is probably a bad idea.) I was about to say the same thing. Bookmarks are how this should be done. hg-git converts bookmarks to and from Git branches. Also, if the hg-git tool isn't satisfying our needs, we can always modify hg-git. http://mercurial.selenic.com/wiki/Bookmarks/ ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposal for an inbound2 branch
On 2013-05-03 3:19 AM, Mike Hommey wrote: On Fri, May 03, 2013 at 02:59:21AM -0400, Ehsan Akhgari wrote: On 2013-05-03 2:52 AM, Mike Hommey wrote: On Fri, May 03, 2013 at 02:43:36AM -0400, Ehsan Akhgari wrote: On 2013-05-02 9:40 AM, Mike Hommey wrote: On Thu, May 02, 2013 at 09:21:57AM -0400, Ehsan Akhgari wrote: Also, they cannot be represented in git I doubt this is true. The bridging tools may well not support it, but there's nothing structural about multiple-head repository that would prevent them to be mirrored in git. In git, the only way to have something in your history is for it to be reachable by a ref (for example, a branch name). When you convert an hg repo with a multi-headed branch to git, you can only choose to represent one of those heads by a branch name, and the other head will be left stranded, which means that it cannot be used normally (for example you cannot push it when you push your branch somewhere) and git gc will remove it from your repository since it thinks that it's an unneeded commit. The hg/git bridge could keep refs with arbitrary names. Or whatever. Really, there's no reason that should be a problem. That is easier said than done. hg-git supports bidirectioal conversion, so such a solution should deal with going back from git to hg some way. And it should also delete the arbitrary name when the extra head goes away somehow, and perhaps other problems that I can't think of right now. And doing any of that is entirely beyond my abilities! :-) I don't understand why we need to switch to a solution which would make maintaining the inbound branch in the git mirror impossible (at least in the near future.) IMHO, the multiplication of branches is not helping the mercurial server performance. Each new branch takes disk space, which is not really a problem, but most branches can't stay in memory caches since other branches are going to throw them away from memory. Arguably, we're already using too many branches. That seems like a problem that can be solved by having a decent amount of memory on the hg server. Or, use the share extension. Or whatever. Remember that this setup _is_ the "standard" way to do branches in Mercurial. I agree that it's horrible, but that's something that Mercurial users have to deal with. That's also why mercurial was a terrible choice for the long term. Easy to say in retrospective, but the more we talk about mercurial problems, the clearer it gets. You're not going to hear an argument against that from me! :-) Ehsan ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposal for an inbound2 branch
On Fri, May 3, 2013 at 8:43 AM, Ehsan Akhgari wrote: > In git, the only way to have something in your history is for it to be > reachable by a ref (for example, a branch name). When you convert an hg > repo with a multi-headed branch to git, you can only choose to represent one > of those heads by a branch name, and the other head will be left stranded, > which means that it cannot be used normally (for example you cannot push it > when you push your branch somewhere) and git gc will remove it from your > repository since it thinks that it's an unneeded commit. It seems like a hg bookmark could be used to keep a pointer to the separate heads. This should create the refs you need when using hg-git. (FWIW, I fully agree that keeping two anonymous heads is probably a bad idea.) Cheers, Dirkjan ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposal for an inbound2 branch
On Fri, May 03, 2013 at 02:59:21AM -0400, Ehsan Akhgari wrote: > On 2013-05-03 2:52 AM, Mike Hommey wrote: > >On Fri, May 03, 2013 at 02:43:36AM -0400, Ehsan Akhgari wrote: > >>On 2013-05-02 9:40 AM, Mike Hommey wrote: > >>>On Thu, May 02, 2013 at 09:21:57AM -0400, Ehsan Akhgari wrote: > Also, they cannot be represented in git > >>> > >>>I doubt this is true. The bridging tools may well not support it, but > >>>there's nothing structural about multiple-head repository that would > >>>prevent them to be mirrored in git. > >> > >>In git, the only way to have something in your history is for it to > >>be reachable by a ref (for example, a branch name). When you > >>convert an hg repo with a multi-headed branch to git, you can only > >>choose to represent one of those heads by a branch name, and the > >>other head will be left stranded, which means that it cannot be used > >>normally (for example you cannot push it when you push your branch > >>somewhere) and git gc will remove it from your repository since it > >>thinks that it's an unneeded commit. > > > >The hg/git bridge could keep refs with arbitrary names. Or whatever. > >Really, there's no reason that should be a problem. > > That is easier said than done. hg-git supports bidirectioal > conversion, so such a solution should deal with going back from git > to hg some way. And it should also delete the arbitrary name when > the extra head goes away somehow, and perhaps other problems that I > can't think of right now. And doing any of that is entirely beyond > my abilities! :-) > > I don't understand why we need to switch to a solution which would > make maintaining the inbound branch in the git mirror impossible (at > least in the near future.) IMHO, the multiplication of branches is not helping the mercurial server performance. Each new branch takes disk space, which is not really a problem, but most branches can't stay in memory caches since other branches are going to throw them away from memory. Arguably, we're already using too many branches. That's also why mercurial was a terrible choice for the long term. Easy to say in retrospective, but the more we talk about mercurial problems, the clearer it gets. Mike ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposal for an inbound2 branch
On 2013-05-03 2:52 AM, Mike Hommey wrote: On Fri, May 03, 2013 at 02:43:36AM -0400, Ehsan Akhgari wrote: On 2013-05-02 9:40 AM, Mike Hommey wrote: On Thu, May 02, 2013 at 09:21:57AM -0400, Ehsan Akhgari wrote: Also, they cannot be represented in git I doubt this is true. The bridging tools may well not support it, but there's nothing structural about multiple-head repository that would prevent them to be mirrored in git. In git, the only way to have something in your history is for it to be reachable by a ref (for example, a branch name). When you convert an hg repo with a multi-headed branch to git, you can only choose to represent one of those heads by a branch name, and the other head will be left stranded, which means that it cannot be used normally (for example you cannot push it when you push your branch somewhere) and git gc will remove it from your repository since it thinks that it's an unneeded commit. The hg/git bridge could keep refs with arbitrary names. Or whatever. Really, there's no reason that should be a problem. That is easier said than done. hg-git supports bidirectioal conversion, so such a solution should deal with going back from git to hg some way. And it should also delete the arbitrary name when the extra head goes away somehow, and perhaps other problems that I can't think of right now. And doing any of that is entirely beyond my abilities! :-) I don't understand why we need to switch to a solution which would make maintaining the inbound branch in the git mirror impossible (at least in the near future.) Ehsan ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposal for an inbound2 branch
On Fri, May 03, 2013 at 02:43:36AM -0400, Ehsan Akhgari wrote: > On 2013-05-02 9:40 AM, Mike Hommey wrote: > >On Thu, May 02, 2013 at 09:21:57AM -0400, Ehsan Akhgari wrote: > >>Also, they cannot be represented in git > > > >I doubt this is true. The bridging tools may well not support it, but > >there's nothing structural about multiple-head repository that would > >prevent them to be mirrored in git. > > In git, the only way to have something in your history is for it to > be reachable by a ref (for example, a branch name). When you > convert an hg repo with a multi-headed branch to git, you can only > choose to represent one of those heads by a branch name, and the > other head will be left stranded, which means that it cannot be used > normally (for example you cannot push it when you push your branch > somewhere) and git gc will remove it from your repository since it > thinks that it's an unneeded commit. The hg/git bridge could keep refs with arbitrary names. Or whatever. Really, there's no reason that should be a problem. Mike ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposal for an inbound2 branch
On 2013-05-02 10:58 AM, Boris Zbarsky wrote: On 5/2/13 10:09 AM, Matt Brubeck wrote: The suggested workflow with inbound2 would lead to almost-identical multi-headed repos in developers' local clones anyway. Only temporary ones, until inbound and inbound2 merge. Using multiple mozilla repositories in a single hg clone means that you already have to deal with multiple heads, so this won't change anything for those people. Ehsan ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposal for an inbound2 branch
On 2013-05-02 9:29 AM, Neil wrote: Ehsan Akhgari wrote: Multiheaded repos are evil. They're very hard to deal with, and I don't think that buildbot can deal with them properly. Also, they cannot be represented in git, which means that they will break the git mirror. Why does mozilla-inbound need a git mirror? Because a lot of our developers are using git locally, and the development of Firefox OS relies on working git mirrors of a number of branches, including inbound. Ehsan ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposal for an inbound2 branch
On 2013-05-02 9:40 AM, Mike Hommey wrote: On Thu, May 02, 2013 at 09:21:57AM -0400, Ehsan Akhgari wrote: Also, they cannot be represented in git I doubt this is true. The bridging tools may well not support it, but there's nothing structural about multiple-head repository that would prevent them to be mirrored in git. In git, the only way to have something in your history is for it to be reachable by a ref (for example, a branch name). When you convert an hg repo with a multi-headed branch to git, you can only choose to represent one of those heads by a branch name, and the other head will be left stranded, which means that it cannot be used normally (for example you cannot push it when you push your branch somewhere) and git gc will remove it from your repository since it thinks that it's an unneeded commit. Ehsan ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposal for an inbound2 branch
On 5/2/13 10:09 AM, Matt Brubeck wrote: The suggested workflow with inbound2 would lead to almost-identical multi-headed repos in developers' local clones anyway. Only temporary ones, until inbound and inbound2 merge. -Boris ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposal for an inbound2 branch
On 5/2/2013 6:21 AM, Ehsan Akhgari wrote: On 2013-05-02 5:46 AM, Neil wrote: Why do we need a separate repo? Can we not simply close the broken head and start again at the last green changeset? Multiheaded repos are evil. They're very hard to deal with, and I don't think that buildbot can deal with them properly. Buildbot can deal with a multi-headed Try, so it can probably be convinced to deal with a multi-headed inbound, even if it needs to be a special case like Try. Individual developers might have more problems (they'll want to pull from inbound, unlike Try) but I think as long as the inactive heads are closed it won't freak out Mercurial too much... The suggested workflow with inbound2 would lead to almost-identical multi-headed repos in developers' local clones anyway. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposal for an inbound2 branch
On Thu, May 02, 2013 at 09:21:57AM -0400, Ehsan Akhgari wrote: > Also, they cannot be represented in git I doubt this is true. The bridging tools may well not support it, but there's nothing structural about multiple-head repository that would prevent them to be mirrored in git. Mike ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposal for an inbound2 branch
Ehsan Akhgari wrote: Multiheaded repos are evil. They're very hard to deal with, and I don't think that buildbot can deal with them properly. Also, they cannot be represented in git, which means that they will break the git mirror. Why does mozilla-inbound need a git mirror? -- Warning: May contain traces of nuts. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposal for an inbound2 branch
On 2013-05-02 5:46 AM, Neil wrote: Ryan VanderMeulen wrote: My proposal: -Create an inbound2 branch identically configured to mozilla-inbound. -Under normal circumstances (i.e. m-i open), inbound2 will be CLOSED. -In the event of a long tree closure, the last green changeset from m-i will be merged to inbound2 and inbound2 will be opened for checkins. ---It will be a judgment call for sheriffs as to how long of a closure will suffice for opening inbound2. -When the bustage on m-i is resolved and it is again passing tests, inbound2 will be closed again. -When all pending jobs on inbound2 are completed, it will be merged to m-i. Why do we need a separate repo? Can we not simply close the broken head and start again at the last green changeset? Multiheaded repos are evil. They're very hard to deal with, and I don't think that buildbot can deal with them properly. Also, they cannot be represented in git, which means that they will break the git mirror. Cheers, Ehsan ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposal for an inbound2 branch
Ryan VanderMeulen wrote: My proposal: -Create an inbound2 branch identically configured to mozilla-inbound. -Under normal circumstances (i.e. m-i open), inbound2 will be CLOSED. -In the event of a long tree closure, the last green changeset from m-i will be merged to inbound2 and inbound2 will be opened for checkins. ---It will be a judgment call for sheriffs as to how long of a closure will suffice for opening inbound2. -When the bustage on m-i is resolved and it is again passing tests, inbound2 will be closed again. -When all pending jobs on inbound2 are completed, it will be merged to m-i. Why do we need a separate repo? Can we not simply close the broken head and start again at the last green changeset? -- Warning: May contain traces of nuts. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposal for an inbound2 branch
Another disadvantage of project branches in addition to the ones
mentioned before is that it limits/delays the amount of testing that you
can get on Nightly and from all of the developers who run things like
fuzzers on ours code. Not everyone's project has enough manpower to get
that level of testing on a project branch before their stuff gets merged
to central/inbound. I personally cannot imagine doing my development in
a project branch silo and deprive myself of the huge advantage that this
kind of testing brings about.
Cheers,
Ehsan
On 2013-04-30 2:46 AM, Gregory Szorc wrote:
On 4/26/2013 12:17 PM, Ryan VanderMeulen wrote:
Specific goals:
-Offer an alternative branch for developers to push to during extended
inbound closures
-Avoid patch pile-up after inbound re-opens from a long closure
Specific non-goals:
-Reducing infrastructure load
-Changing pushing strategies from the widely-accepted status quo (i.e.
multi-headed approach)
-Creating multiple integration branches that allow for simultaneous
pushing (i.e. inbound-b2g, inbound-gfx, etc)
My proposal:
-Create an inbound2 branch identically configured to mozilla-inbound.
-Under normal circumstances (i.e. m-i open), inbound2 will be CLOSED.
-In the event of a long tree closure, the last green changeset from
m-i will be merged to inbound2 and inbound2 will be opened for checkins.
---It will be a judgment call for sheriffs as to how long of a closure
will suffice for opening inbound2.
-When the bustage on m-i is resolved and it is again passing tests,
inbound2 will be closed again.
-When all pending jobs on inbound2 are completed, it will be merged to
m-i.
-Except under extraordinary circumstances, all merges to
mozilla-central will continue to come from m-i ONLY.
-If bustage lands on inbound2, then both trees will be closed until
resolved. Tough. We apparently can't always have nice things.
If you consider that every repository is essentially a clone of
mozilla-central, what we have *now* is effectively equivalent to a
single repository with multiple heads/branches/bookmarks. However, the
different heads/branches/bookmarks differ in:
* How much attention sheriffs give them.
* The automation configuration (coalescing, priority, etc).
* Policies around landing.
* How developers use it.
These are all knobs in our control.
When we say "create an inbound2," we're essentially establishing a new
head/branch/bookmark that behaves much like "inbound1" with a slightly
different landing policy. If that's what we want to do, sure. I think
it's better than a single, frequently closed inbound.
Anyway, no matter how much I think about this proposal, I keep coming
back to the question of "why don't we use project branches more?"
Instead of everyone and her brother landing on inbound, what if more
landings were performed on {fx-team, services-central, , etc}? I /think/ the worst that can happen is merge conflicts and
bit rot. And, we can abate that through intelligent grouping of related
commits in the same repository, frequent merges, and maybe even better
communication (perhaps even automatically with tools that alert
developers to potential conflicts - wouldn't it be cool if you updated a
patch and Mercurial was like "o hai - Ehsan recently pushed a Try push
that conflicts with your change: you two should talk.").
As a counter-proposal, I propose that we start shifting landings to
project branches/twigs. We should aim for a small and well-defined set
of repositories (say 3 to 5) sharing similar automation configuration
and sheriff love. By keeping the number small, it's easy to figure out
where something should land and it's not too much of an extra burden on
sheriffs. We can still keep inbound, but it should only be reserved for
major, cross-repository landings with multi-module impact (e.g. build
system changes), merges from the main landing repositories (unless we
merge straight to central), and possibly as a backup in case one of the
landing repositories is closed.
We can test this today with very little effort: we figure out how to
bucket commits, re-purpose existing repositories/twigs, and see what
happens. If it works: great - we've just validated that distributed
version control works for Firefox development (as opposed to the
CVS/Subversion workflow we're currently using with inbound). If not, we
can try variations and/or the inbound2 idea.
Is there sanity to this proposal or am I still crazy?
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
Re: Proposal for an inbound2 branch
On 5/1/2013 2:15 PM, L. David Baron wrote:
On Saturday 2013-04-27 08:26 +1000, Nicholas Nethercote wrote:
On Sat, Apr 27, 2013 at 5:17 AM, Ryan VanderMeulen wrote:
-In the event of a long tree closure, the last green changeset from m-i will
be merged to inbound2 and inbound2 will be opened for checkins.
If I have a patch ready to land when inbound closes, what would be the
sequence of steps that I need to do to land it on inbound2? Would I
need to have an up-to-date inbound2 clone and transplant the patch
across? Or is it possible to push from an inbound clone?
For what it's worth, what I'd do is qpop the patch, pull the tip of
inbound2 into my inbound clone and update to tip, qpush and qfin the
patch, and then "hg out -rtip inbound2" (and after checking it's
right), "hg push -rtip inbound2".
I wouldn't do anything involving bookmarks; the way of checking that
I'm pushing the right thing is to run "hg out" before "hg push" (as
one should always do... and as I don't know how to do in git).
$ git log HEAD..origin/master
However, this will compare the differences based on what the local Git
repository knows about. To ensure it is up to date, you'll need to fetch
the remote first:
$ git fetch origin
$ git log HEAD..origin/master
Even then this isn't entirely the same as `hg out`: this just tells you
which commits are between the local and remote ref. It's possible some
of the {commits, trees, blobs} already exist on the remote. The actual
details of which Git objects will be pushed is worked out at push time.
There are low-level Git commands to get at this set. But, I'd argue the
commit delta is what you care about, not which raw Git objects need to
be sent over the wire.
Anyway, if you had a Mozilla Mercurial extension installed, you could
just run |hg land| and magic would ensue.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
Re: Proposal for an inbound2 branch
On Saturday 2013-04-27 08:26 +1000, Nicholas Nethercote wrote: > On Sat, Apr 27, 2013 at 5:17 AM, Ryan VanderMeulen wrote: > > > > -In the event of a long tree closure, the last green changeset from m-i will > > be merged to inbound2 and inbound2 will be opened for checkins. > > If I have a patch ready to land when inbound closes, what would be the > sequence of steps that I need to do to land it on inbound2? Would I > need to have an up-to-date inbound2 clone and transplant the patch > across? Or is it possible to push from an inbound clone? For what it's worth, what I'd do is qpop the patch, pull the tip of inbound2 into my inbound clone and update to tip, qpush and qfin the patch, and then "hg out -rtip inbound2" (and after checking it's right), "hg push -rtip inbound2". I wouldn't do anything involving bookmarks; the way of checking that I'm pushing the right thing is to run "hg out" before "hg push" (as one should always do... and as I don't know how to do in git). -David -- 𝄞 L. David Baron http://dbaron.org/ 𝄂 𝄢 Mozilla http://www.mozilla.org/ 𝄂 ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposal for an inbound2 branch
On 4/30/2013 11:19 AM, Chris AtLee wrote: It seems like a tree used by a smaller, more focused group of people could cope better with leaving some orange on the tree for short periods of time. Instead of backing out suspect revisions, and closing the tree to wait for the results of the backout to come back, could the tree remain open to landings while the test failures are being investigated. I think this is easier to coordinate with a smaller group of people, and with a slower check-in cadence. I've actually been doing that with birch. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposal for an inbound2 branch
On 02:54, Tue, 30 Apr, Justin Lebar wrote: Is there sanity to this proposal or am I still crazy? If we had a lot more project branches, wouldn't that increase the load on infra dramatically, because we'd have less coalescing? Yes, it would decrease coalescing. I wonder how many tree closures and backouts we'd have though? It seems like a tree used by a smaller, more focused group of people could cope better with leaving some orange on the tree for short periods of time. Instead of backing out suspect revisions, and closing the tree to wait for the results of the backout to come back, could the tree remain open to landings while the test failures are being investigated. I think this is easier to coordinate with a smaller group of people, and with a slower check-in cadence. Cheers, Chris signature.asc Description: Digital signature ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposal for an inbound2 branch
On 4/30/2013 2:46 AM, Gregory Szorc wrote: > As a counter-proposal, I propose that we start shifting landings to > project branches/twigs. We should aim for a small and well-defined set > of repositories (say 3 to 5) sharing similar automation configuration > and sheriff love. By keeping the number small, it's easy to figure out > where something should land and it's not too much of an extra burden on > sheriffs. We can still keep inbound, but it should only be reserved for > major, cross-repository landings with multi-module impact (e.g. build > system changes), merges from the main landing repositories (unless we > merge straight to central), and possibly as a backup in case one of the > landing repositories is closed. > > We can test this today with very little effort: we figure out how to > bucket commits, re-purpose existing repositories/twigs, and see what > happens. If it works: great - we've just validated that distributed > version control works for Firefox development (as opposed to the > CVS/Subversion workflow we're currently using with inbound). If not, we > can try variations and/or the inbound2 idea. > > Is there sanity to this proposal or am I still crazy? > I think the key point you're missing here is that the appeal of inbound is based on the fact that we have sheriffs willing to monitor it. Using project branches assumes that you can either coerce the sheriffs to monitor a bunch more trees, or project members using those branches will act as sheriffs. Neither of those seems particularly likely. RyanVM's proposal is fairly tightly scoped so that it doesn't add a lot of undue burden on sheriffs. Given the discussion, I think this is probably the best short-term project to try. -Ted ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposal for an inbound2 branch
On 4/30/13 8:46 AM, Gregory Szorc wrote:
On 4/26/2013 12:17 PM, Ryan VanderMeulen wrote:
Specific goals:
-Offer an alternative branch for developers to push to during extended
inbound closures
-Avoid patch pile-up after inbound re-opens from a long closure
Specific non-goals:
-Reducing infrastructure load
-Changing pushing strategies from the widely-accepted status quo (i.e.
multi-headed approach)
-Creating multiple integration branches that allow for simultaneous
pushing (i.e. inbound-b2g, inbound-gfx, etc)
My proposal:
-Create an inbound2 branch identically configured to mozilla-inbound.
-Under normal circumstances (i.e. m-i open), inbound2 will be CLOSED.
-In the event of a long tree closure, the last green changeset from
m-i will be merged to inbound2 and inbound2 will be opened for checkins.
---It will be a judgment call for sheriffs as to how long of a closure
will suffice for opening inbound2.
-When the bustage on m-i is resolved and it is again passing tests,
inbound2 will be closed again.
-When all pending jobs on inbound2 are completed, it will be merged to
m-i.
-Except under extraordinary circumstances, all merges to
mozilla-central will continue to come from m-i ONLY.
-If bustage lands on inbound2, then both trees will be closed until
resolved. Tough. We apparently can't always have nice things.
If you consider that every repository is essentially a clone of
mozilla-central, what we have *now* is effectively equivalent to a
single repository with multiple heads/branches/bookmarks. However, the
different heads/branches/bookmarks differ in:
* How much attention sheriffs give them.
* The automation configuration (coalescing, priority, etc).
* Policies around landing.
* How developers use it.
These are all knobs in our control.
When we say "create an inbound2," we're essentially establishing a new
head/branch/bookmark that behaves much like "inbound1" with a slightly
different landing policy. If that's what we want to do, sure. I think
it's better than a single, frequently closed inbound.
Anyway, no matter how much I think about this proposal, I keep coming
back to the question of "why don't we use project branches more?"
Instead of everyone and her brother landing on inbound, what if more
landings were performed on {fx-team, services-central, , etc}? I /think/ the worst that can happen is merge conflicts and
bit rot. And, we can abate that through intelligent grouping of related
commits in the same repository, frequent merges, and maybe even better
communication (perhaps even automatically with tools that alert
developers to potential conflicts - wouldn't it be cool if you updated a
patch and Mercurial was like "o hai - Ehsan recently pushed a Try push
that conflicts with your change: you two should talk.").
As a counter-proposal, I propose that we start shifting landings to
project branches/twigs. We should aim for a small and well-defined set
of repositories (say 3 to 5) sharing similar automation configuration
and sheriff love. By keeping the number small, it's easy to figure out
where something should land and it's not too much of an extra burden on
sheriffs. We can still keep inbound, but it should only be reserved for
major, cross-repository landings with multi-module impact (e.g. build
system changes), merges from the main landing repositories (unless we
merge straight to central), and possibly as a backup in case one of the
landing repositories is closed.
We can test this today with very little effort: we figure out how to
bucket commits, re-purpose existing repositories/twigs, and see what
happens. If it works: great - we've just validated that distributed
version control works for Firefox development (as opposed to the
CVS/Subversion workflow we're currently using with inbound). If not, we
can try variations and/or the inbound2 idea.
Is there sanity to this proposal or am I still crazy?
To me the caveat here is that merge conflicts aren't restricted to VC
merge conflicts. Changes to APIs in one branch with a new customer in
another is probably the most frequent. Naming conflicts between two jsms
in different modules, conflicting contract ids, etc.
I think that the cost of getting to a single revision that works out of
multiple independent branches in a piece of software as big and modular
as firefox has to be significant.
Axel
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
Re: Proposal for an inbound2 branch
> Is there sanity to this proposal or am I still crazy?
If we had a lot more project branches, wouldn't that increase the load
on infra dramatically, because we'd have less coalescing?
This is of course a solvable problem, but the fact that the problem
exists suggests to me that your proposal here should be considered
separately from RyanVM's tightly-scoped proposal here.
On Tue, Apr 30, 2013 at 2:46 AM, Gregory Szorc wrote:
> On 4/26/2013 12:17 PM, Ryan VanderMeulen wrote:
>> Specific goals:
>> -Offer an alternative branch for developers to push to during extended
>> inbound closures
>> -Avoid patch pile-up after inbound re-opens from a long closure
>>
>> Specific non-goals:
>> -Reducing infrastructure load
>> -Changing pushing strategies from the widely-accepted status quo (i.e.
>> multi-headed approach)
>> -Creating multiple integration branches that allow for simultaneous
>> pushing (i.e. inbound-b2g, inbound-gfx, etc)
>>
>> My proposal:
>> -Create an inbound2 branch identically configured to mozilla-inbound.
>> -Under normal circumstances (i.e. m-i open), inbound2 will be CLOSED.
>> -In the event of a long tree closure, the last green changeset from
>> m-i will be merged to inbound2 and inbound2 will be opened for checkins.
>> ---It will be a judgment call for sheriffs as to how long of a closure
>> will suffice for opening inbound2.
>> -When the bustage on m-i is resolved and it is again passing tests,
>> inbound2 will be closed again.
>> -When all pending jobs on inbound2 are completed, it will be merged to
>> m-i.
>> -Except under extraordinary circumstances, all merges to
>> mozilla-central will continue to come from m-i ONLY.
>> -If bustage lands on inbound2, then both trees will be closed until
>> resolved. Tough. We apparently can't always have nice things.
>
> If you consider that every repository is essentially a clone of
> mozilla-central, what we have *now* is effectively equivalent to a
> single repository with multiple heads/branches/bookmarks. However, the
> different heads/branches/bookmarks differ in:
>
> * How much attention sheriffs give them.
> * The automation configuration (coalescing, priority, etc).
> * Policies around landing.
> * How developers use it.
>
> These are all knobs in our control.
>
> When we say "create an inbound2," we're essentially establishing a new
> head/branch/bookmark that behaves much like "inbound1" with a slightly
> different landing policy. If that's what we want to do, sure. I think
> it's better than a single, frequently closed inbound.
>
> Anyway, no matter how much I think about this proposal, I keep coming
> back to the question of "why don't we use project branches more?"
> Instead of everyone and her brother landing on inbound, what if more
> landings were performed on {fx-team, services-central, twig>, etc}? I /think/ the worst that can happen is merge conflicts and
> bit rot. And, we can abate that through intelligent grouping of related
> commits in the same repository, frequent merges, and maybe even better
> communication (perhaps even automatically with tools that alert
> developers to potential conflicts - wouldn't it be cool if you updated a
> patch and Mercurial was like "o hai - Ehsan recently pushed a Try push
> that conflicts with your change: you two should talk.").
>
> As a counter-proposal, I propose that we start shifting landings to
> project branches/twigs. We should aim for a small and well-defined set
> of repositories (say 3 to 5) sharing similar automation configuration
> and sheriff love. By keeping the number small, it's easy to figure out
> where something should land and it's not too much of an extra burden on
> sheriffs. We can still keep inbound, but it should only be reserved for
> major, cross-repository landings with multi-module impact (e.g. build
> system changes), merges from the main landing repositories (unless we
> merge straight to central), and possibly as a backup in case one of the
> landing repositories is closed.
>
> We can test this today with very little effort: we figure out how to
> bucket commits, re-purpose existing repositories/twigs, and see what
> happens. If it works: great - we've just validated that distributed
> version control works for Firefox development (as opposed to the
> CVS/Subversion workflow we're currently using with inbound). If not, we
> can try variations and/or the inbound2 idea.
>
> Is there sanity to this proposal or am I still crazy?
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
Re: Proposal for an inbound2 branch
On 4/26/2013 12:17 PM, Ryan VanderMeulen wrote:
> Specific goals:
> -Offer an alternative branch for developers to push to during extended
> inbound closures
> -Avoid patch pile-up after inbound re-opens from a long closure
>
> Specific non-goals:
> -Reducing infrastructure load
> -Changing pushing strategies from the widely-accepted status quo (i.e.
> multi-headed approach)
> -Creating multiple integration branches that allow for simultaneous
> pushing (i.e. inbound-b2g, inbound-gfx, etc)
>
> My proposal:
> -Create an inbound2 branch identically configured to mozilla-inbound.
> -Under normal circumstances (i.e. m-i open), inbound2 will be CLOSED.
> -In the event of a long tree closure, the last green changeset from
> m-i will be merged to inbound2 and inbound2 will be opened for checkins.
> ---It will be a judgment call for sheriffs as to how long of a closure
> will suffice for opening inbound2.
> -When the bustage on m-i is resolved and it is again passing tests,
> inbound2 will be closed again.
> -When all pending jobs on inbound2 are completed, it will be merged to
> m-i.
> -Except under extraordinary circumstances, all merges to
> mozilla-central will continue to come from m-i ONLY.
> -If bustage lands on inbound2, then both trees will be closed until
> resolved. Tough. We apparently can't always have nice things.
If you consider that every repository is essentially a clone of
mozilla-central, what we have *now* is effectively equivalent to a
single repository with multiple heads/branches/bookmarks. However, the
different heads/branches/bookmarks differ in:
* How much attention sheriffs give them.
* The automation configuration (coalescing, priority, etc).
* Policies around landing.
* How developers use it.
These are all knobs in our control.
When we say "create an inbound2," we're essentially establishing a new
head/branch/bookmark that behaves much like "inbound1" with a slightly
different landing policy. If that's what we want to do, sure. I think
it's better than a single, frequently closed inbound.
Anyway, no matter how much I think about this proposal, I keep coming
back to the question of "why don't we use project branches more?"
Instead of everyone and her brother landing on inbound, what if more
landings were performed on {fx-team, services-central, , etc}? I /think/ the worst that can happen is merge conflicts and
bit rot. And, we can abate that through intelligent grouping of related
commits in the same repository, frequent merges, and maybe even better
communication (perhaps even automatically with tools that alert
developers to potential conflicts - wouldn't it be cool if you updated a
patch and Mercurial was like "o hai - Ehsan recently pushed a Try push
that conflicts with your change: you two should talk.").
As a counter-proposal, I propose that we start shifting landings to
project branches/twigs. We should aim for a small and well-defined set
of repositories (say 3 to 5) sharing similar automation configuration
and sheriff love. By keeping the number small, it's easy to figure out
where something should land and it's not too much of an extra burden on
sheriffs. We can still keep inbound, but it should only be reserved for
major, cross-repository landings with multi-module impact (e.g. build
system changes), merges from the main landing repositories (unless we
merge straight to central), and possibly as a backup in case one of the
landing repositories is closed.
We can test this today with very little effort: we figure out how to
bucket commits, re-purpose existing repositories/twigs, and see what
happens. If it works: great - we've just validated that distributed
version control works for Firefox development (as opposed to the
CVS/Subversion workflow we're currently using with inbound). If not, we
can try variations and/or the inbound2 idea.
Is there sanity to this proposal or am I still crazy?
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
Re: Proposal for an inbound2 branch
Multiple inbounds are not a great idea for smarter usage of our infra capacity or for load balancing, but this proposal doesn't attempt to address the former and avoids the latter by making the availability of the two inbounds mutually exclusive, so, it looks good. I also doubt that it's going to have a lot of bad impact on the infra load because of the mutual exclusion. Hopefully we'll monitor the infra load after switching to this model to keep an eye on how we're doing there. Cheers, Ehsan On 2013-04-26 3:17 PM, Ryan VanderMeulen wrote: As has been discussed at length in the various infrastructure meetings, one common point of frustration for developers is frequent tree closures due to bustage on inbound. While there are other issues and ideas for how to improve the inbound bustage situation, one problem I'm seeing is that multiple different issues with different solutions are being lumped together into one discussion, which makes it hard to gain traction on getting anything done. For that reason, I would like to specifically separate out the specific issue of inbound closures negatively affecting developer productivity and offer a more fleshed-out solution that can be implemented now independent of any other ideas on the table. Specific goals: -Offer an alternative branch for developers to push to during extended inbound closures -Avoid patch pile-up after inbound re-opens from a long closure Specific non-goals: -Reducing infrastructure load -Changing pushing strategies from the widely-accepted status quo (i.e. multi-headed approach) -Creating multiple integration branches that allow for simultaneous pushing (i.e. inbound-b2g, inbound-gfx, etc) My proposal: -Create an inbound2 branch identically configured to mozilla-inbound. -Under normal circumstances (i.e. m-i open), inbound2 will be CLOSED. -In the event of a long tree closure, the last green changeset from m-i will be merged to inbound2 and inbound2 will be opened for checkins. ---It will be a judgment call for sheriffs as to how long of a closure will suffice for opening inbound2. -When the bustage on m-i is resolved and it is again passing tests, inbound2 will be closed again. -When all pending jobs on inbound2 are completed, it will be merged to m-i. -Except under extraordinary circumstances, all merges to mozilla-central will continue to come from m-i ONLY. -If bustage lands on inbound2, then both trees will be closed until resolved. Tough. We apparently can't always have nice things. As stated above, I believe that this will solve one of the biggest painpoints of long tree closures without adding tons of extra complexity to what we're already doing and what developers are used to. The affect on infrastructure load should be close to neutral since at any given time, patches will only be getting checked into one branch. This proposal also has the advantage of being easy to implement since it's simply a clone of an existing repo, with a little extra sheriff overhead. It also helps to mitigate the pile-up of pushes we tend to see after a long closure which increase the likelihood of another long closure in the event of any bustage due to a high level of coalescing. To be clear, this proposal is NOT intended to solve all of the various problems that have been raised with respect to infrastructure load, good coding practices, bustage minimization, good Try usage, etc. This is only looking to reduce the impact such issues have on developer workflow and make sheriffing easier after the tree reopens. Feedback? ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposal for an inbound2 branch
On 4/29/2013 9:32 AM, Boris Zbarsky wrote:
On 4/26/13 6:26 PM, Nicholas Nethercote wrote:
If I have a patch ready to land when inbound closes, what would be the
sequence of steps that I need to do to land it on inbound2?
The following steps assume that the default and default-push for your
repo are both appropriate versions of inbound and that "i2" and
"i2-push" are the appropriate versions of inbound2 (you can create
such aliases in your .hgrc). They also assume that since inbound just
closed there is bustage on inbound that is not going to be present on
inbound2 and that you have already pulled this bustage into your tree.
1) Make sure your patch-to-land is tracked by mq
2) hg qpop -a
3) hg strip "roots(outgoing('i2'))"
4) hg pull -u i2
5) hg qpush your patches
6) hg qfin -a && hg push i2-push
The strip really isn't necessary, as long as you're ok with your tree
having multiple heads for a bit.
1) Make sure your patches are tracked by mq
2) hg qpop -a
3) hg pull inbound2
4) hg up -c tip # tip is what you just pulled from inbound2. The update
crosses branches, but -c says that's ok
5) hg qpush # your patches
6) hg qfin -a && hg push -r. inbound2 # note -r. is necessary here so
you're only pushing your inbound2 head, not your inbound head
I use a setup like this to push to the release branches, keeping all of
aurora/beta/release as multiple heads of one tree.
--BDS
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
Re: Proposal for an inbound2 branch
On 4/26/13 6:26 PM, Nicholas Nethercote wrote:
If I have a patch ready to land when inbound closes, what would be the
sequence of steps that I need to do to land it on inbound2?
The following steps assume that the default and default-push for your
repo are both appropriate versions of inbound and that "i2" and
"i2-push" are the appropriate versions of inbound2 (you can create such
aliases in your .hgrc). They also assume that since inbound just closed
there is bustage on inbound that is not going to be present on inbound2
and that you have already pulled this bustage into your tree.
1) Make sure your patch-to-land is tracked by mq
2) hg qpop -a
3) hg strip "roots(outgoing('i2'))"
4) hg pull -u i2
5) hg qpush your patches
6) hg qfin -a && hg push i2-push
Step 3 is slightly annoying because you have to quote the parens from
the shell
In any case, the upshot of the above steps is that after step 4 your
tree is effectively a clone of inbound2, and then you just push to it
normally.
If your patches are not tracked by mq and you don't want them to be thus
tracked then things are more complicated: as far as I can tell you want
to pull into the same tree from inbound2, transplant your patch on top
of the tip of inbound2, strip out the bits of inbound that shouldn't be
landing, then push.
-Boris
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
Re: Proposal for an inbound2 branch
> 3/ Being a git guy, I prefer having a try-like server where you don't get > push contention or closed tree, because we are creating a new head > every-time, and let the sheriffs cherry-pick the good changes which are not > source of conflicts. And ask developers to rebase their changes otherwise. I prefer this too, and I've proposed trying this. But kats convinced me that this isn't a decent option right now, because this would disable a lot of coalescing, and we don't have the capacity to handle this. It's possible that we'd have enough capacity, but end up shifting load to the time when the sun is over the Pacific ocean, but even that isn't great, since it significantly increases the turn-around time for landing patches. Anyway, RyanVM wanted us to limit discussion here to a yes/no vote on his proposal, and I think we should respect that. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposal for an inbound2 branch
On 04/26/2013 06:11 PM, Matt Brubeck wrote: On 4/26/2013 4:14 PM, Justin Lebar wrote: If I have a patch ready to land when inbound closes, what would be the sequence of steps that I need to do to land it on inbound2? Would I need to have an up-to-date inbound2 clone and transplant the patch across? I think mbrubeck or someone knows how to maintain multiple hg branches in one repo, but I've never figured that out... Yes, having been a git user for years before I started with Mercurial, I simply treat Mercurial as a slightly crippled version of git. :) […] I'm actually astounded that Mercurial doesn't have better support for this built in; I see Mozilla developers doing crazy time-consuming things all the time because of Mercurial's poor support for working with remote repositories. :( Such as maintaining a bi-directional hg-git mirror of multiple branches to avoid having to deal with Mercurial at all? When I discovered the poor resolution of conflicts offered my Mercurial queues, I switched back to git. I have been pushing my patches to try & inbound & aurora and beta with git since December 2011. Back on the original subject: 1/ Having to deal with yet another Inbound would be a pain for all automation testing such as Are We Fast Yet machines as we are making our own builds out of inbound. This imply that our automation will have to switch repositories to follow the latest patches such as we can keep a minimal regression range. In addition, AWFY is often ahead of tbpl in terms of build, which means that switching from inbound1 to inbound2 will mean going backward in time, as patches would be back-ported to inbound2 as soon as they are ok on inbound1. This is definitely weird. 2/ What happens if A & B changes are exclusive? A is pushed to inbound1, inbound1 is closed for another reason. B is pushed to inbound2. A finish building is cherry-picked from inbound1 and put on top of inbound2. inbound2 fails because A does not build on top of B. Conclusion: both branches are closed, and A have to be backed out even if it came first. 3/ Being a git guy, I prefer having a try-like server where you don't get push contention or closed tree, because we are creating a new head every-time, and let the sheriffs cherry-pick the good changes which are not source of conflicts. And ask developers to rebase their changes otherwise. -- Nicolas B. Pierron ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposal for an inbound2 branch
On 4/26/2013 4:14 PM, Justin Lebar wrote: If I have a patch ready to land when inbound closes, what would be the sequence of steps that I need to do to land it on inbound2? Would I need to have an up-to-date inbound2 clone and transplant the patch across? I think mbrubeck or someone knows how to maintain multiple hg branches in one repo, but I've never figured that out... Yes, having been a git user for years before I started with Mercurial, I simply treat Mercurial as a slightly crippled version of git. :) hg clone https://hg.mozilla.org/mozilla-central/ cd mozilla-central # hack on some stuff # do some builds and tests hg new my-patch # time to push to inbound: hg qpop hg pull https://hg.mozilla.org/integration/mozilla-inbound/ hg up -c hg qpush && hg qfin my-patch hg push -r tip https://hg.mozilla.org/integration/mozilla-inbound/ # Now, let's backport something to Aurora! hg pull https://hg.mozilla.org/releases/mozilla-aurora/ hg export -r a3c55bdbe37d | hg qimport -P -n patch-to-uplift hg push -r tip https://hg.mozilla.org/releases/mozilla-aurora/ # After a good night's sleep, back to work! # hg pull -u won't work across branches, so: hg pull https://hg.mozilla.org/mozilla-central/ hg up -c # do a build # start hacking again! This sort of workflow is of course much more natural in git, which makes it easy to track the state of the remote repo(s). The bookmark workflow that gps added to MDN basically emulates part of the functionality of git remote tracking branches. I'm actually astounded that Mercurial doesn't have better support for this built in; I see Mozilla developers doing crazy time-consuming things all the time because of Mercurial's poor support for working with remote repositories. :( ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposal for an inbound2 branch
On 4/26/2013 4:14 PM, Justin Lebar wrote: >> If I have a patch ready to land when inbound closes, what would be the >> sequence of steps that I need to do to land it on inbound2? Would I >> need to have an up-to-date inbound2 clone and transplant the patch >> across? > Yes, I think so. > > I think mbrubeck or someone knows how to maintain multiple hg branches > in one repo, but I've never figured that out... I just added a section to https://developer.mozilla.org/en-US/docs/Developer_Guide/Source_Code/Mercurial detailing how to use bookmarks to produce a unified Mercurial repository. I just recently switched my workflow to this model. Honestly, I'm torn. I really like having everything in one central repository. But, the workflow is really awkward. I have to remember which bookmark is active. And when pushing you have to limit the push to just the bookmark you want pushed otherwise Mercurial will attempt to push everything (and the push hooks should reject it because it would result in new heads). Contrast this with Git, where you can control exactly what refs gets pushed to different remotes. AFAICT, there is no built-in way to do this with Mercurial. I guess you can always define aliases or write your own extension. Perhaps if we defined bookmarks on our repositories that tracked the default branch head things would be easier? I encourage people to experiment with this workflow. But, be warned it may be a bit weird. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposal for an inbound2 branch
> If I have a patch ready to land when inbound closes, what would be the > sequence of steps that I need to do to land it on inbound2? Would I > need to have an up-to-date inbound2 clone and transplant the patch > across? Yes, I think so. I think mbrubeck or someone knows how to maintain multiple hg branches in one repo, but I've never figured that out... ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposal for an inbound2 branch
hi RyanVM; I really like this proposal because it gives Mozilla a way for developers to continue doing checkins even during a prolonged mozilla-inbound closure. The experiment with birch as "b2g-inbound" has been really successful so far, which is good. To easily experiment to see if the proposal helps, I have reserved "cypress" and filed bug#866314 to setup cypress running all the same builds/tests as inbound. Once cypress is ready, lets use cypress as "inbound2"... and try this proposal to see how it goes. Unless anyone has any concerns/objections, lets try have sheriffs use cypress-as-mozilla-inbound2 when we next hit a prolonged mozilla-inbound closure. (and yes, if we find this experiment helps, we can setup branches with real names instead of using birch, cypress!) tc John. On 4/26/13 12:17 PM, Ryan VanderMeulen wrote: > As has been discussed at length in the various infrastructure meetings, > one common point of frustration for developers is frequent tree closures > due to bustage on inbound. While there are other issues and ideas for > how to improve the inbound bustage situation, one problem I'm seeing is > that multiple different issues with different solutions are being lumped > together into one discussion, which makes it hard to gain traction on > getting anything done. For that reason, I would like to specifically > separate out the specific issue of inbound closures negatively affecting > developer productivity and offer a more fleshed-out solution that can be > implemented now independent of any other ideas on the table. > > Specific goals: > -Offer an alternative branch for developers to push to during extended > inbound closures > -Avoid patch pile-up after inbound re-opens from a long closure > > Specific non-goals: > -Reducing infrastructure load > -Changing pushing strategies from the widely-accepted status quo (i.e. > multi-headed approach) > -Creating multiple integration branches that allow for simultaneous > pushing (i.e. inbound-b2g, inbound-gfx, etc) > > My proposal: > -Create an inbound2 branch identically configured to mozilla-inbound. > -Under normal circumstances (i.e. m-i open), inbound2 will be CLOSED. > -In the event of a long tree closure, the last green changeset from m-i > will be merged to inbound2 and inbound2 will be opened for checkins. > ---It will be a judgment call for sheriffs as to how long of a closure > will suffice for opening inbound2. > -When the bustage on m-i is resolved and it is again passing tests, > inbound2 will be closed again. > -When all pending jobs on inbound2 are completed, it will be merged to m-i. > -Except under extraordinary circumstances, all merges to mozilla-central > will continue to come from m-i ONLY. > -If bustage lands on inbound2, then both trees will be closed until > resolved. Tough. We apparently can't always have nice things. > > As stated above, I believe that this will solve one of the biggest > painpoints of long tree closures without adding tons of extra complexity > to what we're already doing and what developers are used to. The affect > on infrastructure load should be close to neutral since at any given > time, patches will only be getting checked into one branch. This > proposal also has the advantage of being easy to implement since it's > simply a clone of an existing repo, with a little extra sheriff > overhead. It also helps to mitigate the pile-up of pushes we tend to see > after a long closure which increase the likelihood of another long > closure in the event of any bustage due to a high level of coalescing. > > To be clear, this proposal is NOT intended to solve all of the various > problems that have been raised with respect to infrastructure load, good > coding practices, bustage minimization, good Try usage, etc. This is > only looking to reduce the impact such issues have on developer workflow > and make sheriffing easier after the tree reopens. > > Feedback? > ___ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform signature.asc Description: OpenPGP digital signature ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposal for an inbound2 branch
On Sat, Apr 27, 2013 at 5:17 AM, Ryan VanderMeulen wrote: > > -In the event of a long tree closure, the last green changeset from m-i will > be merged to inbound2 and inbound2 will be opened for checkins. If I have a patch ready to land when inbound closes, what would be the sequence of steps that I need to do to land it on inbound2? Would I need to have an up-to-date inbound2 clone and transplant the patch across? Or is it possible to push from an inbound clone? Nick ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposal for an inbound2 branch
On Fri, Apr 26, 2013 at 2:16 PM, Kartikaya Gupta wrote: > Lately I've run into this a few times, where I see regressions on > areweslimyet.com and dig into it, only to find the regression happened on a > "merge from m-c" changeset. Since the mobile AWSY only runs on m-i (on the > theory that I get more granular data that way) this then means I have to > manually bisect builds from other trees. Can't you just use hg bisect --extend for most of these cases? Gavin ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposal for an inbound2 branch
My only concern with this is that it increases the difficulty of bisecting regressions. Lately I've run into this a few times, where I see regressions on areweslimyet.com and dig into it, only to find the regression happened on a "merge from m-c" changeset. Since the mobile AWSY only runs on m-i (on the theory that I get more granular data that way) this then means I have to manually bisect builds from other trees. Having things coming in from m-i2 would just aggravate the problem. It may be that this problem is unique to AWSY since it only runs on m-i, whereas talos runs on both m-c and m-i (and presumably will run on m-i2 as well). If it only affects me then whatever, go for it. If it is a pain for many other people then maybe we can try to come up with some variation. For example, instead of merging m-i2 back to m-i, we would close the busted head on m-i and just bring m-i2 back as the active head on m-i. kats On 13-04-26 15:17 , Ryan VanderMeulen wrote: As has been discussed at length in the various infrastructure meetings, one common point of frustration for developers is frequent tree closures due to bustage on inbound. While there are other issues and ideas for how to improve the inbound bustage situation, one problem I'm seeing is that multiple different issues with different solutions are being lumped together into one discussion, which makes it hard to gain traction on getting anything done. For that reason, I would like to specifically separate out the specific issue of inbound closures negatively affecting developer productivity and offer a more fleshed-out solution that can be implemented now independent of any other ideas on the table. Specific goals: -Offer an alternative branch for developers to push to during extended inbound closures -Avoid patch pile-up after inbound re-opens from a long closure Specific non-goals: -Reducing infrastructure load -Changing pushing strategies from the widely-accepted status quo (i.e. multi-headed approach) -Creating multiple integration branches that allow for simultaneous pushing (i.e. inbound-b2g, inbound-gfx, etc) My proposal: -Create an inbound2 branch identically configured to mozilla-inbound. -Under normal circumstances (i.e. m-i open), inbound2 will be CLOSED. -In the event of a long tree closure, the last green changeset from m-i will be merged to inbound2 and inbound2 will be opened for checkins. ---It will be a judgment call for sheriffs as to how long of a closure will suffice for opening inbound2. -When the bustage on m-i is resolved and it is again passing tests, inbound2 will be closed again. -When all pending jobs on inbound2 are completed, it will be merged to m-i. -Except under extraordinary circumstances, all merges to mozilla-central will continue to come from m-i ONLY. -If bustage lands on inbound2, then both trees will be closed until resolved. Tough. We apparently can't always have nice things. As stated above, I believe that this will solve one of the biggest painpoints of long tree closures without adding tons of extra complexity to what we're already doing and what developers are used to. The affect on infrastructure load should be close to neutral since at any given time, patches will only be getting checked into one branch. This proposal also has the advantage of being easy to implement since it's simply a clone of an existing repo, with a little extra sheriff overhead. It also helps to mitigate the pile-up of pushes we tend to see after a long closure which increase the likelihood of another long closure in the event of any bustage due to a high level of coalescing. To be clear, this proposal is NOT intended to solve all of the various problems that have been raised with respect to infrastructure load, good coding practices, bustage minimization, good Try usage, etc. This is only looking to reduce the impact such issues have on developer workflow and make sheriffing easier after the tree reopens. Feedback? ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposal for an inbound2 branch
I like that inbound2 would be open only when inbound is closed. That way you don't have to make a decision wrt which tree to push to. sgtm. On Fri, Apr 26, 2013 at 3:17 PM, Ryan VanderMeulen wrote: > As has been discussed at length in the various infrastructure meetings, one > common point of frustration for developers is frequent tree closures due to > bustage on inbound. While there are other issues and ideas for how to > improve the inbound bustage situation, one problem I'm seeing is that > multiple different issues with different solutions are being lumped together > into one discussion, which makes it hard to gain traction on getting > anything done. For that reason, I would like to specifically separate out > the specific issue of inbound closures negatively affecting developer > productivity and offer a more fleshed-out solution that can be implemented > now independent of any other ideas on the table. > > Specific goals: > -Offer an alternative branch for developers to push to during extended > inbound closures > -Avoid patch pile-up after inbound re-opens from a long closure > > Specific non-goals: > -Reducing infrastructure load > -Changing pushing strategies from the widely-accepted status quo (i.e. > multi-headed approach) > -Creating multiple integration branches that allow for simultaneous pushing > (i.e. inbound-b2g, inbound-gfx, etc) > > My proposal: > -Create an inbound2 branch identically configured to mozilla-inbound. > -Under normal circumstances (i.e. m-i open), inbound2 will be CLOSED. > -In the event of a long tree closure, the last green changeset from m-i will > be merged to inbound2 and inbound2 will be opened for checkins. > ---It will be a judgment call for sheriffs as to how long of a closure will > suffice for opening inbound2. > -When the bustage on m-i is resolved and it is again passing tests, inbound2 > will be closed again. > -When all pending jobs on inbound2 are completed, it will be merged to m-i. > -Except under extraordinary circumstances, all merges to mozilla-central > will continue to come from m-i ONLY. > -If bustage lands on inbound2, then both trees will be closed until > resolved. Tough. We apparently can't always have nice things. > > As stated above, I believe that this will solve one of the biggest > painpoints of long tree closures without adding tons of extra complexity to > what we're already doing and what developers are used to. The affect on > infrastructure load should be close to neutral since at any given time, > patches will only be getting checked into one branch. This proposal also has > the advantage of being easy to implement since it's simply a clone of an > existing repo, with a little extra sheriff overhead. It also helps to > mitigate the pile-up of pushes we tend to see after a long closure which > increase the likelihood of another long closure in the event of any bustage > due to a high level of coalescing. > > To be clear, this proposal is NOT intended to solve all of the various > problems that have been raised with respect to infrastructure load, good > coding practices, bustage minimization, good Try usage, etc. This is only > looking to reduce the impact such issues have on developer workflow and make > sheriffing easier after the tree reopens. > > Feedback? > ___ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
