Re: [Mesa-dev] [Intel-gfx] gitlab.fd.o financial situation and impact on services
On Sun, Mar 1, 2020 at 2:49 PM Nicolas Dufresne wrote: > > Hi Jason, > > I personally think the suggestion are still a relatively good > brainstorm data for those implicated. Of course, those not implicated > in the CI scripting itself, I'd say just keep in mind that nothing is > black and white and every changes end-up being time consuming. Sorry. I didn't intend to stop a useful brainstorming session. I'm just trying to say that CI is useful and we shouldn't hurt our development flows just to save a little money unless we're truly desperate. From what I understand, I don't think we're that desperate yet. So I was mostly trying to re-focus the discussion towards straightforward things we can do to get rid of pointless waste (there probably is some pretty low-hanging fruit) and away from "OMG X.org is running out of money; CI as little as possible". I don't think you're saying those things; but I've sensed a good bit of fear in this thread. (I could just be totally misreading people, but I don't think so.) One of the things that someone pointed out on this thread is that we need data. Some has been provided here but it's still a bit unclear exactly what the break-down is so it's hard for people to come up with good solutions beyond "just do less CI". We do know that the biggest cost is egress web traffic and that's something we didn't know before. My understanding is that people on the X.org board and/or Daniel are working to get better data. I'm fairly hopeful that, once we understand better what the costs are (or even with just the new data we have), we can bring it down to reasonable and/or come up with money to pay for it in fairly short order. Again, sorry I was so terse. I was just trying to slow the panic. > Le dimanche 01 mars 2020 à 14:18 -0600, Jason Ekstrand a écrit : > > I've seen a number of suggestions which will do one or both of those things > > including: > > > > - Batching merge requests > > Agreed. Or at least I foresee quite complicated code to handle the case > of one batched merge failing the tests, or worst, with flicky tests. > > > - Not running CI on the master branch > > A small clarification, this depends on the chosen work-flow. In > GStreamer, we use a rebase flow, so "merge" button isn't really > merging. It means that to merge you need your branch to be rebased on > top of the latest. As it is multi-repo, there is always a tiny chance > of breakage due to mid-air collision in changes in other repos. What we > see is that the post "merge" cannot even catch them all (as we already > observed once). In fact, it usually does not catch anything. Or each > time it cached something, we only notice on the next MR.0 So we are > really considering doing this as for this specific workflow/project, we > found very little gain of having it. > > With real merge, the code being tested before/after the merge is > different, and for that I agree with you. Even with a rebase model, it's still potentially different; though marge re-runs CI before merging. I agree the risk is low, however, and if you have GitLab set up to block MRs that don't pass CI, then you may be able to drop the master branch to a daily run or something like that. Again, should be project-by-project. > > - Shutting off CI > > Of course :-), specially that we had CI before gitlab in GStreamer > (just not pre-commit), we don't want a regress that far in the past. > > > - Preventing CI on other non-MR branches > > Another small nuance, mesa does not prevent CI, it only makes it manual > on non-MR. Users can go click run to get CI results. We could also have > option to trigger the ci (the opposite of ci.skip) from git command > line. Hence my use of "prevent". :-) It's very useful but, IMO, it should be opt-in and not opt-out. I think we agree here. :-) --Jason ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] [Intel-gfx] gitlab.fd.o financial situation and impact on services
Hi Jason, I personally think the suggestion are still a relatively good brainstorm data for those implicated. Of course, those not implicated in the CI scripting itself, I'd say just keep in mind that nothing is black and white and every changes end-up being time consuming. Le dimanche 01 mars 2020 à 14:18 -0600, Jason Ekstrand a écrit : > I've seen a number of suggestions which will do one or both of those things > including: > > - Batching merge requests Agreed. Or at least I foresee quite complicated code to handle the case of one batched merge failing the tests, or worst, with flicky tests. > - Not running CI on the master branch A small clarification, this depends on the chosen work-flow. In GStreamer, we use a rebase flow, so "merge" button isn't really merging. It means that to merge you need your branch to be rebased on top of the latest. As it is multi-repo, there is always a tiny chance of breakage due to mid-air collision in changes in other repos. What we see is that the post "merge" cannot even catch them all (as we already observed once). In fact, it usually does not catch anything. Or each time it cached something, we only notice on the next MR.0 So we are really considering doing this as for this specific workflow/project, we found very little gain of having it. With real merge, the code being tested before/after the merge is different, and for that I agree with you. > - Shutting off CI Of course :-), specially that we had CI before gitlab in GStreamer (just not pre-commit), we don't want a regress that far in the past. > - Preventing CI on other non-MR branches Another small nuance, mesa does not prevent CI, it only makes it manual on non-MR. Users can go click run to get CI results. We could also have option to trigger the ci (the opposite of ci.skip) from git command line. > - Disabling CI on WIP MRs That I'm also mitigated about. > - I'm sure there are more... regards, Nicolas ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] [Intel-gfx] gitlab.fd.o financial situation and impact on services
I don't think we need to worry so much about the cost of CI that we need to micro-optimize to to get the minimal number of CI runs. We especially shouldn't if it begins to impact coffee quality, people's ability to merge patches in a timely manner, or visibility into what went wrong when CI fails. I've seen a number of suggestions which will do one or both of those things including: - Batching merge requests - Not running CI on the master branch - Shutting off CI - Preventing CI on other non-MR branches - Disabling CI on WIP MRs - I'm sure there are more... I think there are things we can do to make CI runs more efficient with some sort of end-point caching and we can probably find some truly wasteful CI to remove. Most of the things in the list above, I've seen presented by people who are only lightly involved the project to my knowledge (no offense to anyone intended). Developers depend on the CI system for their day-to-day work and hampering it will only show down development, reduce code quality, and ultimately hurt our customers and community. If we're so desperate as to be considering painful solutions which will have a negative impact on development, we're better off trying to find more money. --Jason On March 1, 2020 13:51:32 Jacob Lifshay wrote: One idea for Marge-bot (don't know if you already do this): Rust-lang has their bot (bors) automatically group together a few merge requests into a single merge commit, which it then tests, then, then the tests pass, it merges. This could help reduce CI runs to once a day (or some other rate). If the tests fail, then it could automatically deduce which one failed, by recursive subdivision or similar. There's also a mechanism to adjust priority and grouping behavior when the defaults aren't sufficient. Jacob ___ Intel-gfx mailing list intel-...@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/intel-gfx ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] [Intel-gfx] gitlab.fd.o financial situation and impact on services
One idea for Marge-bot (don't know if you already do this): Rust-lang has their bot (bors) automatically group together a few merge requests into a single merge commit, which it then tests, then, then the tests pass, it merges. This could help reduce CI runs to once a day (or some other rate). If the tests fail, then it could automatically deduce which one failed, by recursive subdivision or similar. There's also a mechanism to adjust priority and grouping behavior when the defaults aren't sufficient. Jacob ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] [Intel-gfx] gitlab.fd.o financial situation and impact on services
Le dimanche 01 mars 2020 à 15:14 +0100, Michel Dänzer a écrit : > On 2020-02-29 8:46 p.m., Nicolas Dufresne wrote: > > Le samedi 29 février 2020 à 19:14 +0100, Timur Kristóf a écrit : > > > 1. I think we should completely disable running the CI on MRs which are > > > marked WIP. Speaking from personal experience, I usually make a lot of > > > changes to my MRs before they are merged, so it is a waste of CI > > > resources. > > Interesting idea, do you want to create an MR implementing it? > > > > In the mean time, you can help by taking the habit to use: > > > > git push -o ci.skip > > That breaks Marge Bot. > > > > Notably, we would like to get rid of the post merge CI, as in a rebase > > flow like we have in GStreamer, it's a really minor risk. > > That should be pretty easy, see Mesa and > https://docs.gitlab.com/ce/ci/variables/predefined_variables.html. > Something like this should work: > > rules: > - if: '$CI_PROJECT_NAMESPACE != "gstreamer"' > when: never > > This is another interesting idea we could consider for Mesa as well. It > would however require (mostly) banning direct pushes to the main repository. We already have this policy in GStreamer group. We rely on maintainers to make the right call though, as we have few cases in multi-repo usage where pushing manually is the only way to reduce the breakage time (e.g. when we undo a new API in development branch). (We have implemented support so that CI is run across users repository with the same branch name, so that allow doing CI with all the changes, but the merge remains non-atomic.) > > > > > 2. Maybe we could take this one step further and only allow the CI to > > > be only triggered manually instead of automatically on every push. > > That would again break Marge Bot. Marge is just a software, we can update it to trigger CI on rebases, or if the CI haven't been run. There was proposal to actually do that and let marge trigger CI on merge from maintainers. Though, from my point view, having a longer delay between submission and the author being aware of CI breakage have some side effects. Authors are often less available a week later, when someone review and try to merge, which make merging patches a lot longer. > > ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] [Intel-gfx] gitlab.fd.o financial situation and impact on services
On 2020-02-29 8:46 p.m., Nicolas Dufresne wrote: > Le samedi 29 février 2020 à 19:14 +0100, Timur Kristóf a écrit : >> >> 1. I think we should completely disable running the CI on MRs which are >> marked WIP. Speaking from personal experience, I usually make a lot of >> changes to my MRs before they are merged, so it is a waste of CI >> resources. Interesting idea, do you want to create an MR implementing it? > In the mean time, you can help by taking the habit to use: > > git push -o ci.skip That breaks Marge Bot. > Notably, we would like to get rid of the post merge CI, as in a rebase > flow like we have in GStreamer, it's a really minor risk. That should be pretty easy, see Mesa and https://docs.gitlab.com/ce/ci/variables/predefined_variables.html. Something like this should work: rules: - if: '$CI_PROJECT_NAMESPACE != "gstreamer"' when: never This is another interesting idea we could consider for Mesa as well. It would however require (mostly) banning direct pushes to the main repository. >> 2. Maybe we could take this one step further and only allow the CI to >> be only triggered manually instead of automatically on every push. That would again break Marge Bot. -- Earthling Michel Dänzer | https://redhat.com Libre software enthusiast | Mesa and X developer ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
