Hi,
Steve George <[email protected]> writes: > On Thu, Feb 12, 2026 at 09:17:30PM +0100, Rutherther wrote: > (...) >> Steve George <[email protected]> writes: > (...) >> > One of our current challenges is that we don't want to commit changes to >> > the master branch that cause large rebuilds. As I understand it the main >> > issue is that the build farm takes a while to build all the dependent >> > packages and in the meantime uses that do guix pull won't have substitutes >> > so will have to build locally. >> > >> > Nix seems to solve this by having developers push to a different branch >> > from the one that users pull from [0]. This seems like a good solution, >> > has Guix tried this previously, am I missing something? >> > >> > A rough proposal for how this would work: >> > >> > 1. Developers push to a new branch (e.g. 'main') >> >> Note that there has been a GCD to move master to this branch. So I think >> using this branch is not really wise in case we want to move to it in >> the future. Of course it's just a detail. > (...) > > Ah, yes - good point, I'd forgotten about that, thank-you! Cayetano suggested > 'devel', that works for me - I don't mind what we call it. > >> > 2. CI builds each commit >> > 3a. If the build is successful the commit is updated on master >> > 3b. The substitute from the build is available to users >> >> I think this approach has some issues. While on one hand, it makes >> substitutes better available for the users, I think there are at least >> two notable disadvantages: >> >> 1) Incentives >> >> Since the branch is no longer facing users, developers get more >> incentivized to push commits that might break something. They know that >> if they mess up, the process will get back to them and they might fix it >> easily, without much effect on the users. And the more commits like that >> there are, the more time it takes for us to get a commit that's >> buildable. While we might tell ourselves that we do not want to do that, >> I think even subconsciously this is going to have some effects. >> >> Nixpkgs suffers especially in times before a new stable channel release >> when the channels might not get build for multiple days. This slows down >> the updates that might sometimes be necessary, which gets me to the >> second point. >> >> 2) Security updates >> >> The Nixpkgs model does not take into account any other way than first >> commiting to master and then letting CI push it to unstable branch. >> So even with critical security updates that should land ASAP, you can >> see delay. On the unstable branch, it might regularly be up to a week, >> sometimes even two. On the stable branches it's usually less. >> >> What's worse, if we wanted to move closer to their model, bigger >> rebuilds (500+ packages iirc) get to a staging channel and this is then >> merged to master ~twice a month. I've seen that even for a security >> update their hands were sort of tied and they had to go through this >> branch to deliver the fix to the users. And this can take up to ~3 weeks >> if merge just appeared. >> >> So I am not completely convinced this is the way to go. >> >> Wouldn't it be enough to give users the tools to find out what commit >> already contains substitutes for the packages they're interested in? >> Cuirass/data service already sort of give out this information through >> the API, so it's just a matter of proper filtering of the informations. > (...) > > In terms of the substitute availability problem, it would definitely be > better if there was an API and a client capability to query for this. But, > that sounds like "work" and I'm not sure that Cuirass/Data Service stores. We > have the `channel-with-substitutes` and various things in ci.scm. > Yes, channel-with-substitutes is something like that, but that one only checks the guix jobset. The guix jobset is not really telling you much about the packages, it just tells you won't have to build for `guix pull`. For x86_64, it's at HEAD of master most of the time. Similar function could be done with the master jobset, asking for specific packages. I would say both Cuirass and data service already expose this information, although there might be room for improvement, ie. bulk requests where you could ask for multiple packages at once. Currently you would probably have to ask on a per-package basis, which is not so good, one could even say we would be DDoSing ourselves, providing a simple function to users that does many calls. > I agree with your points about the Security updates, thanks for the > information on how it works in Nix. We could continue to test security > updates on their own branch and at a higher priority for the the build system > - they'd continue to be deployed immediately into the user facing branch > (master). I guess there's a git cherry-pick/flow issue to be solved to have > them in (devel). Git cherry pick... won't work, because then we cannot fast forward the master to current devel. We also cannot permit rebases on either of those branches, because that would mean users cannot pull from them. And also, as we've seen with next-mater, Codeberg doesn't handle rebases well at all when people open pull requests against the rebased branch, showing many more commits than are part of the PR. (as it doesn't understand them as patches, but as full branches and it diffs between the branches) So if we want to keep pushing some commits to master independently for security fixes, we would likely have to go for merges. The automated CI won't be able to do merges, because it cannot sign, so merges from devel to master don't seem feasible. So the only option I see us left with (maybe I am missing something) is merging master to devel manually when such situation arises. I think then it should be possible to fast forward "merge" master to devel, so the CI should be fine. > > If it was your solution (API) or separating the user-facing branch from the > development branch, I think it would have the same the impact for ensuring > that users received updates with Substitutes. > > The "Incentives" part is tough because I kind of agree, it's a balance > between competing needs (stability vs throughput). We know our 'users' want > us to go faster, and we know we have many out of date packages in the > archive. Change does introduce instability, but we're also facing packages > that are very hard to update because they haven't moved forward in so long, > it makes them difficult and unstable when we update them. The base Perl > package is sort of in this position at the moment. Equally, I do accept that > just pushing things with no thought to the impact causes instability - and > users don't actually want a lowering of quality either. > > In the balance, I think we should look for ideas that lead to higher > throughput. I'm not advocating for removing the 'team' branches, though I > would like them to be smaller and folded in more incrementally [0]. I do > think that developers not updating packages because they cause 'rebuilds' is > very odd for a rolling release. Maybe we can improve tooling, but we have > people burning a lot of effort on that already. I agree there is generally a problem, especially for issues that fall out of teams scope. But I am not convinced it's a good idea to go with the route of pushing these to devel, where it 'stops' the branch from reaching users until issues are resolved. Recently there has been this idea of a 'world-rebuild' branch where you can put world rebuilds outside of teams scopes iiuc. So why not extend that idea, for example, we could start grouping multiple PRs, either for multiple teams or outside of any teams bounds and push them all there. Then ensure all is fine and push to master from there. Multiple people can be merging to it PRs that are ready. Even nixpkgs doesn't really do this, they push to master what they think is fine only. world rebuilds (where it's expected it can cause trouble) go to the staging branch. If we decided to move to the devel -> master approach, I still think it would be better to push only smaller updates to devel rather than world rebuilds. Btw we could even do this all, just backwards, we could just keep master as it is for now. Then make a branch, like, 'slow' where CI would fast forward merge from master when certain substitutes are available. Or even 'slow-x86-64', 'slow-aarch64' and so on, acknowledging that our build farms are working at different rates for different architectures. This will allow for the goal of updating to commits with substitutes available. Ideally there could also be a set of system tests that are guaranteed to be passing on these branches. Personally I like slower progressive changes rather than big jumps in the workflow, where we cannot really verify how well it will work and there is no easy way back. This seems to me like an improvement, and after we see users like it or not, we could even move to devel -> master approach from there if necessary. This would allow us to mostly keep the same workflows for the time being. However we should still be cautious with these branches so that they do not go stale, ie. if a system test has gotten broken. Rutherther > > Do you have any branching, policy/social or technical ideas on what we could > to improve our throughput? Any balance of approaches from previous experience > that you like? > > Steve / Futurile > > [0] > https://trunkbaseddevelopment.com/alternative-branching-models/#modern-claimed-high-throughput-branching-models
