On Fri, 14 Nov 2014, Ron wrote: > > So I am a Kali Linux contributor. We use git repos to maintain all our > > packages and we use git-buildpackage. > > I guess the first question there is what were the arguments put forward > for deciding to 'standardise' on gbp? If there wasn't one, maybe that's > an argument you should have (and if there was, maybe it's one to revisit :)
1/ It's the tool I was most familiar with 2/ It was rather well documented 3/ It seemed to be the most popular tool within Debian 4/ It forces the user to follow some conventions and this is good when you have newcomers 5/ It did the job You can try to sell me gitpkg but I'm not sure it will bring this discussion forward. > If you have a clearer idea now of the problems you are facing, it might > make properly evaluating things that avoid those problems easier. My problems are not so much technical, my problem is having a general recipe that work well enough for most packages so that I have to be involved for the few special cases. Changing for another tool will not improve the situation if that other tool requires an evaluation of the layout of each repository before being able to execute the right incantation. When Kali forks a package it's usually because we need a new upstream version, or because we want to apply a pre-existing patch. It's not rocket science and it ought to be doable rather easily, whatever helper tool Debian uses. git-buildpackage has not been a major obstacle for us, on the contrary. Thanks to git-import-dsc we can hack an any package by always following the same workflow. But now if I want to go one step further and build on the existing repositories while still trying to keep the same level of simplicity, we have to go one step forward. And that's the step that I'm trying to push for with this DEP. > Even if everyone did stick to the conventions already discussed > (which in reality, they won't) there's still far too many degrees of > (quite necessary) freedom to really approach a "just follow these > three easy steps" kind of uniformity. And even if you got close to > achieving that for the "debian branches", the (again necessary) > variability between upstream repos is going to be even greater. I don't want 100% uniformity, I just want as much uniformity as possible when there's no good reason to diverge from the recommendations. For the other cases, we can stick to the current model (building on top of imported sources packages). It's not perfect but it's a lot better if most of our forked packages are based on Debian's git repo. > I think at some point you're going to have to rely on real human > intelligence to be able to look at a repo and form their own > understanding of its structure. Most really aren't all that > complicated (however much they vary between each other), and in > the worst case you can always actually ask the 'upstream' > maintainer to explain anything that is unclear. Kali contributors are humans with intelligence but I'd rather not require too much of their intelligence when it comes to trivial changes like adding a patch or importing a new upstream version. Also when you're 2-3 persons maintaining 300+ packages, you really don't want to have to remember what's special about each package. This is inavoidable for some packages (say "linux") but for the vast majority one should not need to ask himself too many questions. > git is already going to namespace them so when you add the remotes > the branch refs will be (for remotes named upstream, debian, kali): I know all this, but this requires again a deeper knowledge of git's features whereas "git checkout <branch>" will create a new local branch based on the remote branch of the same name. If you have <branch> in multiple remotes, there is ambiguity. > If it really is the 'real' debian package, you have no problem > and don't need a kali tag for it anyway. If it's a kali special, > then the package already ought to have a x.y.x-1kali2 type version > anyway. So this part already naturally avoids conflicts too. Using namespaces has other possible benefits: think of building Debian packages out of a git push. The explicit tag prefix avoids building the Debian version of the package when I want to build the Kali one (I might be the maintainer on both Debian and Kali, using the same GPG signature). > > And we could also build on work in progress that has not yet been > > released as a source package. Right now, the only packages where we > > build on top of the Debian git repositories are some native packages > > (like debian-installer). > > I can't really tell you how kali ought to work, but if you can solve > it for this case, you should be able to do the same for non-native > packages too. And I would recommend that you try to, even if you > can't get the "uniformity" that you'd thing of as ideal. > > What is the exact difference that makes native packages work for this > but others be harder? In the case of gitpkg, this is really just > the difference between passing one ref and two to export it. In the case of debian-installer, I am the one taking care of it on Kali and I have the required knowledge. And also we needed some unreleased fixes for newer kernel versions. Still, native packages are easier because they look like any upstream repo with a master branch (there's no need to have multiple branches around). And they don't pose any problem with requiring an .orig.tar.gz: the current instructions I gave to other Kali contributors include enabling the --pristine-tar flag to git-bp so they never have to download the .orig.tar.gz. Now think of the case of importing a new upstream version on a git repository: you need to know whether the packaging branch is based on the upstream git (if yes you probably need to find that one and add it as a remote) or if tarballs are imported (in that case you need to know on which branch it must be imported), it's immediately more complicated. If you consider the case of adding a patch, it gets worse. While you can rely on quilt with a freshly imported "3.0 (quilt)" source package, that's no longer the case with some of the git helper tools (i.e. those where patches are applied but have no .pc). I know that only a tiny part of this will be improved by the current DEP, but it's a start and an important one IMO. > > I can't afford to document all the possible ways Debian is maintaining > > their package but if I can write a documentation that covers the common > > case and if I can tell them "when you see those branches, you can follow > > the instructions below", then we have made some real progress. > > I really do think that the names of the branches are actually going to > be the least of your worries here, unfortunately. Even with a naming > scheme that's widely adopted, things just aren't going to be that sort > of uniform outside of (a fairly large number of) fairly small subsets. > > You're going to need a better solution than this unfortunately, if you > want it to actually work. Yes, completely agreed. But git is rather robust and smart, and even if we would all prefer that derivatives continue maintaining their repository in the same spirit that we maintained ours, I believe that until we have proper tools to do just that, it's not too difficult to create some script that allows us to fixup common problems when we fetch a Debian git repository so that the standard Kali-recommended workflow works afterwards. I'm certainly eager to explore all those options (with you and anyone else interested, the scope of the suggested sprint can include such stuff once we have cracked the easy fixup to harmonize the helper tools) but I would still like to complete this DEP first because I'm convinced that it is useful nevertheless. > Really the only way this can possibly work is if all the tools that > you consider viable candidates to use *don't* require some special > structure of their own in the repo to work (or knowing the right > magic incantation specific to each repo for them to work correctly). git-buildpackage doesn't have many hard requirements. git-dpm a bit less so IIRC. > Otherwise, the tools that do are simply never going to work with a > repo that doesn't have that. There really isn't any way around that, > except to replace those tools with better ones ... Or improve them, or merge them. Which is the outcome that I'm looking forward in this DEP. Right now teams are imposing one specific tool/workflow because there are so many differences even on basic things like the branch names. When the differences start to fade away, people look more closely at ways to allow everyone to use their tool of choices. It's a process, it takes time, but we need to start it somewhere. This DEP + a sprint looks like a good way to go forward to me. > Unlike several of them, gitpkg didn't try to be a "framework" for > package management. It simply focussed on the job of exporting a > source package, in any valid source package format, from an existing > git repo. It provides hooks to let you tweak that, and to perform > automated tasks after export (like shipping the package off to be > built), but it explicitly doesn't provide high level functions for > repo management tasks *prior* to export. Why couldn't gitpkg be an option within git-buildpackage? I believe it's important to provide a global solution (i.e. from creating/cloning the repo) and importing the history, to building the package and tagging the releases, etc. > Whether the other tools can be fixed ... yeah. After all, there > was a reason gitpkg was born as a separate thing in the first place :) I'm much more positive than you. At leas Guido is very welcome to enhance git-buildpackage with support of more workflows and I'm convinced that we could include everything that makes gitpkg/git-dpm interesting in git-buildpackage. Unfortunately Guido is lacking time lately but we should be able to work together. > ... less than 8 hours later, *all* of those patches had been cherry > picked and merged into the mainline upstream repo. This is just a proof that storing the patches as real commits is useful. And that's the point of the patch-queue tag. Instead of having the patches only as real commits in the local repository, they get pushed to the public repository too under that tag (instead of appearing only within debian/patches/). I don't think your experience contradicts anything here :-) Cheers, -- Raphaël Hertzog ◈ Debian Developer Support Debian LTS: http://www.freexian.com/services/debian-lts.html Learn to master Debian: http://debian-handbook.info/get/ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20141114102536.ga28...@home.ouaza.com