Re: [Development] Allow to do parallel integration (was: Proposal: New branch model)
On Wednesday, January 30, 2019 1:55:41 PM CET Allan Jensen wrote: > On Wednesday, 30 January 2019 13:38:46 CET Jedrzej Nowacki wrote: > > On Wednesday, January 30, 2019 11:48:40 AM CET Allan Jensen wrote: > > > On Wednesday, 30 January 2019 11:19:07 CET Edward Welbourne wrote: > > > > Jedrzej Nowacki (30 January 2019 11:08) > > > > > > > > > Mårten pointed out that we can check for conflicts up front. Not > > > > > only > > > > > against HEAD of the target branch, but also against all build > > > > > branches. > > > > > That is even nicer as there is no need to start a job that would > > > > > likely > > > > > to be rejected at the end. > > > > > > > > That'll only find VC-level conflicts. > > > > It's no help in finding the kinds of breakage that only testing > > > > reveals. > > > > It'll also sometimes lead to rejecting a change needlessly, because > > > > the > > > > already-integrating branch it conflicts with is about to fail its > > > > testing. > > > > Still, I guess restaging a little later can fix that ... > > > > > > We could do speculative merging. Guess if the previous commit will merge > > > on > > not, and stage the following commit based on that guess. Or stage two > > > > versions of the next commit, one for success of the previous and one for > > > failure. That latter would use 3x the VMs for 2x the throughput. > > > > > > 'Allan > > > > We could. In such case we do not allow regressions, but we waste ~50% of > > speculative builds. I believe that would be much better accepting the > > risks, > the throughput maybe N times higher, when N is count of parallel build > > > branches. In the end reverts are just one click. Btw. after revert coin > > will likely detect that the revision was build and it would just use > > cached > > results, so it may even avoid running tests. > > At the current pass rates it would be a better speculation to assume that > the other commits do not merge. > > 'Allan Not to my knowledge (last 7 days): total percentage Passed 153 47.37% Failed 147 45.51% Cancelled 23 7.12% ___ Development mailing list Development@qt-project.org https://lists.qt-project.org/listinfo/development
Re: [Development] Allow to do parallel integration (was: Proposal: New branch model)
On Wednesday, 30 January 2019 13:38:46 CET Jedrzej Nowacki wrote: > On Wednesday, January 30, 2019 11:48:40 AM CET Allan Jensen wrote: > > > On Wednesday, 30 January 2019 11:19:07 CET Edward Welbourne wrote: > > > > > Jedrzej Nowacki (30 January 2019 11:08) > > > > > > > > > > Mårten pointed out that we can check for conflicts up front. Not only > > > > against HEAD of the target branch, but also against all build > > > > branches. > > > > That is even nicer as there is no need to start a job that would > > > > likely > > > > to be rejected at the end. > > > > > > > > > That'll only find VC-level conflicts. > > > It's no help in finding the kinds of breakage that only testing > > > reveals. > > > It'll also sometimes lead to rejecting a change needlessly, because the > > > already-integrating branch it conflicts with is about to fail its > > > testing. > > > Still, I guess restaging a little later can fix that ... > > > > > > We could do speculative merging. Guess if the previous commit will merge > > on not, and stage the following commit based on that guess. Or stage two > > versions of the next commit, one for success of the previous and one for > > failure. That latter would use 3x the VMs for 2x the throughput. > > > > 'Allan > > > We could. In such case we do not allow regressions, but we waste ~50% of > speculative builds. I believe that would be much better accepting the risks, > the throughput maybe N times higher, when N is count of parallel build > branches. In the end reverts are just one click. Btw. after revert coin > will likely detect that the revision was build and it would just use cached > results, so it may even avoid running tests. > At the current pass rates it would be a better speculation to assume that the other commits do not merge. 'Allan ___ Development mailing list Development@qt-project.org https://lists.qt-project.org/listinfo/development
Re: [Development] Allow to do parallel integration (was: Proposal: New branch model)
On Wednesday, January 30, 2019 11:48:40 AM CET Allan Jensen wrote: > On Wednesday, 30 January 2019 11:19:07 CET Edward Welbourne wrote: > > Jedrzej Nowacki (30 January 2019 11:08) > > > > > Mårten pointed out that we can check for conflicts up front. Not only > > > against HEAD of the target branch, but also against all build branches. > > > That is even nicer as there is no need to start a job that would likely > > > to be rejected at the end. > > > > That'll only find VC-level conflicts. > > It's no help in finding the kinds of breakage that only testing reveals. > > It'll also sometimes lead to rejecting a change needlessly, because the > > already-integrating branch it conflicts with is about to fail its testing. > > Still, I guess restaging a little later can fix that ... > > We could do speculative merging. Guess if the previous commit will merge on > not, and stage the following commit based on that guess. Or stage two > versions of the next commit, one for success of the previous and one for > failure. That latter would use 3x the VMs for 2x the throughput. > > 'Allan We could. In such case we do not allow regressions, but we waste ~50% of speculative builds. I believe that would be much better accepting the risks, the throughput maybe N times higher, when N is count of parallel build branches. In the end reverts are just one click. Btw. after revert coin will likely detect that the revision was build and it would just use cached results, so it may even avoid running tests. Cheers, Jędrek ___ Development mailing list Development@qt-project.org https://lists.qt-project.org/listinfo/development
Re: [Development] Allow to do parallel integration (was: Proposal: New branch model)
On Wednesday, January 30, 2019 11:19:07 AM CET Edward Welbourne wrote: > > Mårten pointed out that we can check for conflicts up front. Not only > > against HEAD of the target branch, but also against all build branches. > > That is even nicer as there is no need to start a job that would likely > > to be rejected at the end. > > That'll only find VC-level conflicts. > It's no help in finding the kinds of breakage that only testing reveals. Of course. That is exactly what I'm proposing. Cost: CI would allow breakages caused by patches that are being integrated in parallel. Gain: CI would allow running many parallel integrations. > It'll also sometimes lead to rejecting a change needlessly, because the > already-integrating branch it conflicts with is about to fail its testing. We have that already, you can not stage if your change conflicts with changes that are being integrated. Gerrit gives you a nice and complex message. Maybe it will work out of the box, with multiple build branches. I do not know. Some research is needed. > Still, I guess restaging a little later can fix that ... Exactly. Hopefully less restaging. As cherry on top it may increases chances that the content sha1(*) optimization kicks in, so re-staging may be faster. Cheers, Jędrek (*) Coin works on the content sha1 level not on the commit sha1. That way commit message change or different commit order is irrelevant from it's perspective and it can re-use caches. ___ Development mailing list Development@qt-project.org https://lists.qt-project.org/listinfo/development
Re: [Development] Allow to do parallel integration (was: Proposal: New branch model)
On Wednesday, 30 January 2019 11:19:07 CET Edward Welbourne wrote: > Jedrzej Nowacki (30 January 2019 11:08) > > > Mårten pointed out that we can check for conflicts up front. Not only > > against HEAD of the target branch, but also against all build branches. > > That is even nicer as there is no need to start a job that would likely > > to be rejected at the end. > > That'll only find VC-level conflicts. > It's no help in finding the kinds of breakage that only testing reveals. > It'll also sometimes lead to rejecting a change needlessly, because the > already-integrating branch it conflicts with is about to fail its testing. > Still, I guess restaging a little later can fix that ... > We could do speculative merging. Guess if the previous commit will merge on not, and stage the following commit based on that guess. Or stage two versions of the next commit, one for success of the previous and one for failure. That latter would use 3x the VMs for 2x the throughput. 'Allan ___ Development mailing list Development@qt-project.org https://lists.qt-project.org/listinfo/development
Re: [Development] Allow to do parallel integration (was: Proposal: New branch model)
Jedrzej Nowacki (30 January 2019 11:08) > Mårten pointed out that we can check for conflicts up front. Not only against > HEAD of the target branch, but also against all build branches. That is even > nicer as there is no need to start a job that would likely to be rejected at > the end. That'll only find VC-level conflicts. It's no help in finding the kinds of breakage that only testing reveals. It'll also sometimes lead to rejecting a change needlessly, because the already-integrating branch it conflicts with is about to fail its testing. Still, I guess restaging a little later can fix that ... Eddy. ___ Development mailing list Development@qt-project.org https://lists.qt-project.org/listinfo/development
Re: [Development] Allow to do parallel integration (was: Proposal: New branch model)
On Wednesday, January 30, 2019 10:41:02 AM CET Jedrzej Nowacki wrote: > Cheap (a.k.a reasonably fast to implement) proposal: > - Stage button would create a stage branch as currently (through cherry-pick > on top of target branch) > - 3-5 minutes old stage branch would changed to be a build branche and all > patches in it would be changed to "integrating" state. CI would start > testing the build branch. At that time a new stage branch may be created. > As opposite to now, multiple build branches may be tested in parallel. > - Once a build branch testing passed we cherry-pick / merges (depends what > gerrit does) changes on top of the target branch, in case of conflicts we > mark it as failed. Mårten pointed out that we can check for conflicts up front. Not only against HEAD of the target branch, but also against all build branches. That is even nicer as there is no need to start a job that would likely to be rejected at the end. Cheers, Jędrek ___ Development mailing list Development@qt-project.org https://lists.qt-project.org/listinfo/development
[Development] Allow to do parallel integration (was: Proposal: New branch model)
On Friday, January 25, 2019 11:08:52 AM CET Lars Knoll wrote: > To me this means we need to seriously rethink that part of our CI system, > and ideally test changes (or patch series) individually and in parallel. So > maybe we should adjust our CI system that way: > > * test changes (or patch series) individually > * If they pass CI merge or cherry-pick them into the branch > * reject only if the merge/cherry-pick gives conflicts > > This adds a very small risk that two parallel changes don’t conflict during > the merge/cherry-pick process, but cause a test regression together. To > help with that, we can simply run a regular status check on the repo. If > this happens the repo will be blocked for further testing until someone > submits a fix or reverts an offending change, which is acceptable. > > Another advantage is that this would pave the road towards a system where CI > testing happens before human review, so we can in the longer term avoid > duplicated review/approval work. Hi, I do agree with it. Personally, I'm convinced that we are doing CI gate- keeping in a wrong way. CI is about balancing costs vs. gains. We can not test everything, we should find as many as possible bugs, while running in reasonable time without using too much resources. That is the function to optimize. We really should test things in parallel. The risk of conflicts / regression caused by parallel integration in project as big as qt is minimal. One can be even proactive and not stage changes while other similar ones are integrating. While broken repository state is bad, it is also easy to detect; nothing integrates because of one test, which is not known to be flaky (testrestults.qt.io/grafana...). Cheap (a.k.a reasonably fast to implement) proposal: - Stage button would create a stage branch as currently (through cherry-pick on top of target branch) - 3-5 minutes old stage branch would changed to be a build branche and all patches in it would be changed to "integrating" state. CI would start testing the build branch. At that time a new stage branch may be created. As opposite to now, multiple build branches may be tested in parallel. - Once a build branch testing passed we cherry-pick / merges (depends what gerrit does) changes on top of the target branch, in case of conflicts we mark it as failed. Advantages: - changes are tested in smaller batches => smaller chances to fail - we can use hardware in a more efficient way => more code integrated per hour - fairly easy to implement => we would not need to wait for it to happen years - that would be a step forward in to direction of testing before reviewing Disadvantages: - risk of breakages going through CI - someone work needs to be done to enable it Cheers, Jędrek ___ Development mailing list Development@qt-project.org https://lists.qt-project.org/listinfo/development
