> On Feb 9, 2017, at 5:49 PM, Bob Wilson via swift-dev <swift-dev@swift.org> > wrote: > > Since the feedback on this was all positive, we’re moving ahead with this > change. It seems like a good time for a status update. > > Some changes are already done: > > * We test up a test bot to build against the upstream-with-swift branches of > LLVM/Clang > (https://ci.swift.org/view/swift-master-next/job/oss-swift-incremental-RA-osx-master-next-upstream-with-swift/ > > <https://ci.swift.org/view/swift-master-next/job/oss-swift-incremental-RA-osx-master-next-upstream-with-swift/>). > This has been working well.
I think we can remove this test bot now, so don’t be surprised if it goes away soon. > > * Mishal set up an automerger to continuously update the Swift master-next > branch with changes from master. > > * I audited the master-next branches in the other repos (e.g., swiftpm, etc.) > and found that none of them have any changes from master. For now, we will > simply stop using those branches, and once we’re done with this transition > and things are settled, we can remove them. > > * We’ve had some discussions about how LLDB will fit into this plan. See the > recent thread on “Changes to LLDB Branch Management” for details on that. > > Here is what is left to do: > > * Update all of our master-next bots to build against the “new” branches > (e.g., upstream-with-swift instead of stable-next). New update: Mishal switched all the master-next bots to build against Clang/LLVM upstream-with-swift, so this step is done now. The bad news is that things are in kind of bad shape. Beginning on Wednesday, the build stopped working — it is getting killed by timeouts, so maybe something is hanging. > > * Update documentation. > > * Remove unused branches. > >> On Dec 8, 2016, at 10:54 AM, Michael Gottesman <mgottes...@apple.com >> <mailto:mgottes...@apple.com>> wrote: >> >>> >>> On Dec 8, 2016, at 7:43 AM, Saleem Abdulrasool via swift-dev >>> <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote: >>> >>> Having been involved in the update process for the next branches, I'm >>> really excited to see this type of change. >>> >>> I think that the "simple" approach is both better to work and collaborate >>> in as well as closer to the llvm development model which makes it easier to >>> cross pollinate. >>> >>> The one thing that I think could be more strongly called out is that normal >>> PR CI shouldn't be gated on master next passing. >>> >>> Beyond that, I think that this proposal should ease collaboration and >>> maintenance pains in the current process. >> >> I agree with everything said here. The collaboration issue is the high order >> bit that needs to be fixed. Any other issues that come up can be fixed >> incrementally if necessary based on experience (for instance using the "back >> up" alternative). This is even more true since it is clear that that >> infrastructure would take time to develop and an incremental solution now >> that improves the collaboration will not make it more difficult to develop >> such a solution. >> >> My only strong feeling here is that I think we need documentation of this >> process in ./docs in addition to this proposal. >> >> Michael >> >>> >>> On Wed, Dec 7, 2016 at 7:30 PM Bob Wilson via swift-dev >>> <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote: >>> I would like to make a change in the way we handle the master-next branch. >>> >>> Summary: I’d like to switch to a model where we continuously test against >>> the latest upstream LLVM changes. The goal is to simplify the process and >>> make it easier to collaborate on maintaining master-next. >>> >>> Background: We develop Swift against “stable” branches of LLVM (which I am >>> using here to refer to the llvm, clang, and compiler-rt repositories) that >>> are typically rebranched from trunk once for each release, with other >>> commits individually cherry-picked for specific bug fixes and other >>> changes. This insulates Swift development from the churn of changes in >>> LLVM. At the same time, we maintain the “master-next” branches of Swift >>> repos to keep up to date with trunk LLVM. For Swift, our “trunk” comes from >>> the “upstream-with-swift” branches in our GitHub LLVM repos. We have >>> existing automation to continuously merge changes from llvm.org >>> <http://llvm.org/> into those upstream-with-swift branches. >>> >>> We currently use a manual process to update master-next. Someone on the >>> Swift team is designated as the "merge czar" and is responsible for this. >>> This merge typically happens once every few weeks. Michael Gottesman >>> developed some internal tools to help automate the process, but someone >>> still needs to drive those tools manually. The process involves merging >>> “master” to “master-next” for all the Swift repos and updating the >>> “stable-next” branches of the GitHub LLVM repos for Swift. The >>> “stable-next” branches are basically snapshots of the LLVM >>> upstream-with-swift branches at the point where master-next was most >>> recently merged. >>> >>> Swift CI includes a set of Jenkins bots to test master-next building with >>> the stable-next branches of LLVM >>> (https://ci.swift.org/view/swift-master-next >>> <https://ci.swift.org/view/swift-master-next>). The merge czar can use >>> these bots to confirm that everything is working after a merge. >>> >>> Reasons to change: The current process has the advantage that the merge >>> czar can choose when to do a merge and can schedule that around other work, >>> but it has some significant problems. >>> >>> - It is difficult for multiple people to collaborate on updating >>> master-next. The changes involved are often rev-locked between Swift and >>> the LLVM repos, so there is no good way for someone to fix a problem >>> without doing the whole merge process. >>> >>> - The current system is hard to understand. I’ve been serving as the merge >>> czar for the last few months, and it took me a while to figure out how to >>> do it well. >>> >>> - It requires extra “stable-next” branches in our GitHub LLVM repos, >>> further adding to the complexity. >>> >>> - The tools we have to help automate the process are currently internal to >>> Apple and require ongoing maintenance. They could be cleaned up to release >>> publicly but that would take more work. >>> >>> Proposal: We already have Jenkins bots testing master-next. I would like to >>> add a job to continuously merge master to master-next and change the >>> existing bots to build against the “upstream-with-swift” branches in our >>> GitHub LLVM repos. The bots would then detect any new problems soon after >>> they are introduced. Anyone could fix those problems, whether they are >>> merge conflicts, build failures, or test issues. A partial fix could be >>> applied directly without needing to resolve all of the outstanding issues. >>> >>> This would avoid the need for our current internal merging tools. We >>> already have automatic merging bots, so adding another one would not be >>> difficult. >>> >>> The biggest advantage is that it provides a straightforward model that >>> anyone can understand: master-next becomes just another branch that anyone >>> can modify, build and test in the usual way. Collaboration is no more >>> difficult than for other branches. >>> >>> The cost of this simpler approach is that we would need to be willing to >>> let the master-next branch break occasionally. An LLVM change might break >>> things in a way that takes some time to fix, and the master-next bots would >>> continue to fail during that time. Someone might want to apply a partial >>> fix that does not resolve all the issues, and we would want to allow that >>> even if the bots still fail. That would mean we would have to relax (or >>> override) the requirement for PR testing for commits in that kind of >>> situation. In the worst case, if new problems are introduced more quickly >>> than we can fix them, this approach could fall apart. My experience as >>> merge czar over the last few months suggests that is unlikely. Usually >>> there are no more than a few problems per week and most of them are easy to >>> fix. >>> >>> I propose to roll this out in steps. First, we can add a new Jenkins bot >>> that tests master-next building against upstream-with-swift. If that goes >>> well, and if there are no objections to this proposal, we can add the >>> automerger to merge master into master-next. At the same time, we would >>> update the other master-next bots to use upstream-with-swift instead of >>> stable-next for the LLVM repos. >>> >>> Alternative: We can achieve some of the same goals at a considerable >>> increase in complexity by introducing an automatic gated merge solution. We >>> would have automation perform the merge and commit it as long as everything >>> works. If there were any problems, the automation would create a pull >>> request that would need to be manually updated to resolve the problems. >>> People could still collaborate by working together on the pull request >>> branch. Until the problems were resolved, no further merging would take >>> place. I would like to try the simple approach before considering this more >>> complex solution, since I don’t think it will be necessary, at least in the >>> near future. >>> >>> Any objections to this? Comments or suggestions? >>> _______________________________________________ >>> >>> swift-dev mailing list >>> >>> swift-dev@swift.org <mailto:swift-dev@swift.org> >>> >>> https://lists.swift.org/mailman/listinfo/swift-dev >>> <https://lists.swift.org/mailman/listinfo/swift-dev> >>> >>> _______________________________________________ >>> swift-dev mailing list >>> swift-dev@swift.org <mailto:swift-dev@swift.org> >>> https://lists.swift.org/mailman/listinfo/swift-dev >>> <https://lists.swift.org/mailman/listinfo/swift-dev> > _______________________________________________ > swift-dev mailing list > swift-dev@swift.org > https://lists.swift.org/mailman/listinfo/swift-dev
_______________________________________________ swift-dev mailing list swift-dev@swift.org https://lists.swift.org/mailman/listinfo/swift-dev
