Hi fellow developers, Building Debian packages is a job that is not only part of our daily development practice, but also a task regularly performed by tools. These include: * QA tools frequently rebuild packages to find bugs * Archive rebuilds (e.g. Lucas Nussbaum, Matthias Klose, Santiago Vila, et al) * reproducible.debian.net * crossqa.debian.net * ratt * Tools whose task is larger than building packages, but happens to include doing so * https://wiki.debian.org/HelmutGrohne/rebootstrap * libdvd-pkg * apt-src
I suspect that this list is incomplete. My primary motivation for looking into this came from rebootstrap though. The task of building Debian packages has been solved numerous times. Here are some known solutions: * sbuild * pbuilder * cowbuilder * qemubuilder * docker-based tools: + debocker + debdocker + docker-buildpackage + whalebuilder * systemd-nspawn-based tools + conbuilder + debspawn * manually without chroot * Create a chroot for each build using mmdebstrap Again, the list is incomplete and if you want to improve it, please edit https://wiki.debian.org/SystemBuildTools instead. When someone writes a tool, that happens to build packages as part of its job (the earlier list), typically one of the build tools (the second list) is chosen and fixed. It's not like you can mix and match them. Here are a number of lock-ins I happen to know: * reproducible.debian.net -> pbuilder * crossqa.debian.net -> sbuild * ratt -> sbuild * libdvd-pkg -> manually * rebootstrap -> manually I see this tight coupling as a problem for mainly two reasons. * When someone else works on your tool, they have to set up your preferred builder. As a developer, you cannot just choose between pbuilder and sbuild. You effectively get to use both, because some tools use one and others use the other. * As tasks grow, there is a desire to distribute builds to multiple machines. As such, the various tools typically grow their own strategy for moving build tasks between machines. Lengthy problem description, I know. If you've read some of my bug reports, you may have noticed that only few come without solutions. So I set out to solve this. More precisely, I looked for an API that could abstract both sbuild and pbuilder (because they're the most common ones) and allow executing builds on remote machines. A tool performing builds should not have to care which one is in use and whether the build is performed locally or not. Since describing the settings of a build can be complex, I chose json for representation. Some aspects that you may want to set for a build are: * build architecture / host architecture * arch-only / indep-only / full * DEB_BUILD_PROFILES * DEB_BUILD_OPTIONS * whether network access is permissible * forcing a build path I called it "Meta Dpkg-BuildPackage" or simply mdbp for now and you can find it at: https://git.subdivi.de/?p=~helmut/mdbp.git;a=blob;f=README.md git://git.subdivi.de/~helmut/mdbp.git So with this mail I seek multiple sorts of replies: * What other tools perform builds as part of their job, but don't exactly care how they're being performed? * What build tools am I missing? * Does the problem scope I've chosen make sense? * Feedback on the proposed API * What feature is missing for your tool to use mdbp? * Which build tools should mdbp support to make it useful to you? Thank you for considering. If you want to discuss on irc, I think both #debian-bootstrap and #debian-qa would be appropriate. Helmut