Bug#1005317: git-buildpackage: Make it easier to support an extra packaging branch (for derivatives)
On 2/11/22 22:12, Guido Günther wrote: I miss one point in your enumeration: syncing with changes from Debian's salsa which should be part of the picture as well - because in that situation you want the gbp.conf that Debian ships and merge the result into the derivative's branch. We don't really use Salsa. For most of our Debian's forks, we just have minor changes to maintain, so we don't need to know about Debian's git history. So we just use import-dsc to import new versions from Debian. The only exception I know of it the debian-installer. For this package we do fetch changes from Salsa, as it's sometimes convenient to package a git snapshot. In that case, we just do plain git commands, eg. "git fetch salsa && git merge salsa/master". Does that answer your question? I'm not sure I understood exactly what you asked. So given that I wonder if having the concept of a "downstream" or "derivative" within gbp wouldn't be worthwhile to look at? I know there are more people using gbp for derivative work (including me) so making this easier (without regressing in the "pure" Debian case would make sense to me. This could also include more explicit tracking or a remote for the derivative and salsa. The concept of "downstream" would make sense indeed. I can do a bit of work on my side to see how it would look like, in order to be usable by Kali. Then I'll come back here to report what I found. No ETA though, this is something that I'll do in the background when times allow ;) Cheers, -- Arnaud Rebillout / Offensive Security / Kali Linux Developer
Bug#1005317: git-buildpackage: Make it easier to support an extra packaging branch (for derivatives)
Hi Arnaud, On Fri, Feb 11, 2022 at 01:33:47PM +0700, Arnaud Rebillout wrote: > Package: git-buildpackage > Version: 0.9.25 > Severity: normal > User: de...@kali.org > Usertags: origin-kali > > Dear Maintainer, dear Guido, > > At Kali Linux we use gbp to maintain our packages. We have around 500 > packages total, among which 40 are forked from Debian. Kali Linux being > a rolling distribution, those 40 packages are kept in sync with their > counterpart in Debian testing, meaning that we update them often. > > We use the same workflow for pretty much all of our packages: 3 git > branches (packaging, upstream and pristine-tar), and no upstream git > history. We use "gbp import-orig" for all the Kali specific packages, > and "gbp import-dsc" for all the Debian forks. > > All in all, gbp works great for us, but nothing is never perfect :) > > > THE PROBLEM > > For our Debian forks, we actually have 4 branches (or 2 when it's a > native package): we have the packaging branch (named "kali/master"), the > usual upstream and pristine-tar branches, but we also have the "upstream > distro" branch (named "debian") that contains the unmodified source from > Debian, as imported by "gbp import-dsc". > > Just to clarify, let me describe how we maintain such a 4-branches git > repo: > > * Step 1. $ gbp import-dsc --debian-branch=debian > -> ie. update the 3 branches upstream, pristine-tar and debian. > > * Step 2. $ git merge debian > -> ie. merge the debian branch into the kali/master branch. This is > when we resolve our merge conflicts, since the Kali changes live > on the kali/master branch. > > (Note: this workflow was mentioned in https://bugs.debian.org/670612) > > This workflow works very well, and keeping our Debian forks in sync with > Debian is fairly straightforward. > > Our issue is that the 4th branch (the "debian" branch) lives out of the > realm of gbp. Therefore it's ignored by the gbp commands that deal with > git branches, and we always need additional steps to handle this branch. > In particular: > > * gbp clone: Won't setup this branch, one has to do it manually: > > gbp clone > cd > git branch debian origin/debian > > * gbp pull: Won't pull it, one has to do it manually: > > gbp pull > git checkout debian > git merge > git checkout - > > (or: pull all branches) > > gbp pull --all > > * gbp push: Won't push it, one has to do it manually: > > gbp push > git checkout debian > git push > git checkout - > > (or: push with git) > > git push origin : --follow-tags > > (or: if the --tips option in #908204 was implemented, we could do:) > > gbp push --tips > > These additional steps are trivial, but they are also easy to forget. > One thing that keeps biting me is that I often forget to pull the > "debian" branch. So I work on the package, update it, make it ready for > upload, and at the last moment, when I want to push all the git > branches, git refuses, because I worked with an outdated branch and my > modifications are not fast-forwardable. So I have to restart from zero, > or force push, or whatever. > > > THE TENTATIVE IMPROVEMENTS > > So I've been thinking hard about ways to prevent that to happen, and > more generally, about how I could get gbp to clone/pull/push this 4th > branch automatically, so that there's no need for additional steps. > > Here are some ideas, starting with the worst one: > > 1) the "upstream distro" concept > > An ambitious and complicated way could be to teach gbp about the concept > of an "upstream distro" (meaningful only for Debian derivatives), and > define in the gbp.conf what is the "upstream-distro-branch", the > "upstream-distro-tag", and more if needed. And then let gbp do the right > thing (TM). In other words, hammer a new concept into gbp. > > IMO: way too complicated. > > 2) the "extra branche(s)" idea > > Instead, a lighter approach could be to teach gbp about "extra > branches", without defining what those branches are for. "gbp clone" > would setup those branches, and "gbp pull" would pull it. That's super > easy to implement. The complicated part is that "gbp push", as far as I > can see, would not know what to do with this branch, due to the way it > works. And therefore it wouldn't push it. Unless the option "--tips" > mentioned in #908204 is implemented, and in such case we could document > that gbp will push the extra branches only when --tips is enabled. > > It's not super elegant (and maybe not super useful) to introduce an > option for extra branches, and then say that it's only supported by > clone and pull, but not supported by push. So I'm not super convinced > myself... > > 3) more hooks > > I've noticed that some gbp actions have a "--postxxx" argument, to allow > users to run hooks. That could help! > > There's already a "--postclone" option. I can add this snippet in > debian/gbp.conf, and it works: > >
Bug#1005317: git-buildpackage: Make it easier to support an extra packaging branch (for derivatives)
Package: git-buildpackage Version: 0.9.25 Severity: normal User: de...@kali.org Usertags: origin-kali Dear Maintainer, dear Guido, At Kali Linux we use gbp to maintain our packages. We have around 500 packages total, among which 40 are forked from Debian. Kali Linux being a rolling distribution, those 40 packages are kept in sync with their counterpart in Debian testing, meaning that we update them often. We use the same workflow for pretty much all of our packages: 3 git branches (packaging, upstream and pristine-tar), and no upstream git history. We use "gbp import-orig" for all the Kali specific packages, and "gbp import-dsc" for all the Debian forks. All in all, gbp works great for us, but nothing is never perfect :) THE PROBLEM For our Debian forks, we actually have 4 branches (or 2 when it's a native package): we have the packaging branch (named "kali/master"), the usual upstream and pristine-tar branches, but we also have the "upstream distro" branch (named "debian") that contains the unmodified source from Debian, as imported by "gbp import-dsc". Just to clarify, let me describe how we maintain such a 4-branches git repo: * Step 1. $ gbp import-dsc --debian-branch=debian -> ie. update the 3 branches upstream, pristine-tar and debian. * Step 2. $ git merge debian -> ie. merge the debian branch into the kali/master branch. This is when we resolve our merge conflicts, since the Kali changes live on the kali/master branch. (Note: this workflow was mentioned in https://bugs.debian.org/670612) This workflow works very well, and keeping our Debian forks in sync with Debian is fairly straightforward. Our issue is that the 4th branch (the "debian" branch) lives out of the realm of gbp. Therefore it's ignored by the gbp commands that deal with git branches, and we always need additional steps to handle this branch. In particular: * gbp clone: Won't setup this branch, one has to do it manually: gbp clone cd git branch debian origin/debian * gbp pull: Won't pull it, one has to do it manually: gbp pull git checkout debian git merge git checkout - (or: pull all branches) gbp pull --all * gbp push: Won't push it, one has to do it manually: gbp push git checkout debian git push git checkout - (or: push with git) git push origin : --follow-tags (or: if the --tips option in #908204 was implemented, we could do:) gbp push --tips These additional steps are trivial, but they are also easy to forget. One thing that keeps biting me is that I often forget to pull the "debian" branch. So I work on the package, update it, make it ready for upload, and at the last moment, when I want to push all the git branches, git refuses, because I worked with an outdated branch and my modifications are not fast-forwardable. So I have to restart from zero, or force push, or whatever. THE TENTATIVE IMPROVEMENTS So I've been thinking hard about ways to prevent that to happen, and more generally, about how I could get gbp to clone/pull/push this 4th branch automatically, so that there's no need for additional steps. Here are some ideas, starting with the worst one: 1) the "upstream distro" concept An ambitious and complicated way could be to teach gbp about the concept of an "upstream distro" (meaningful only for Debian derivatives), and define in the gbp.conf what is the "upstream-distro-branch", the "upstream-distro-tag", and more if needed. And then let gbp do the right thing (TM). In other words, hammer a new concept into gbp. IMO: way too complicated. 2) the "extra branche(s)" idea Instead, a lighter approach could be to teach gbp about "extra branches", without defining what those branches are for. "gbp clone" would setup those branches, and "gbp pull" would pull it. That's super easy to implement. The complicated part is that "gbp push", as far as I can see, would not know what to do with this branch, due to the way it works. And therefore it wouldn't push it. Unless the option "--tips" mentioned in #908204 is implemented, and in such case we could document that gbp will push the extra branches only when --tips is enabled. It's not super elegant (and maybe not super useful) to introduce an option for extra branches, and then say that it's only supported by clone and pull, but not supported by push. So I'm not super convinced myself... 3) more hooks I've noticed that some gbp actions have a "--postxxx" argument, to allow users to run hooks. That could help! There's already a "--postclone" option. I can add this snippet in debian/gbp.conf, and it works: [clone] postclone = git branch debian origin/debian If I had a "--postpull" option, I could make sure that the "debian" branch is automatically pulled. This is something I'd configure in the debian/gbp.conf for all of Kali's Debian forks. Same with a "--postpush" option. And while we're at it, a "--postimport" for the "gbp import-dsc" command
