Bug#746967: buildd.debian.org: d-i daily builds happen with unsigned code from alioth
Hi Aurélien, Aurelien Jarno aurel...@aurel32.net (2015-03-18): With the help of Hector Oron, we have been able to setup this on the porterbox of 4 architectures: arm64, armel, armhf and mips. This has been done by allowing the d-i role account on the porterboxes. As a nice side effect, this mean that d-i people can now do/fix the setup themselves without having to go through the buildd team. The code used is available in the porterbox branch of the di-autobuild git. Note that the chroots on the porterbox are created in a very similar way than on the newer buildds (it's even done by the same script IIRC), so the problems should be similar. For example both were affected by bug#775136. Is there any doc on how to create the needed bits on a porterbox to add d-i autobuilding? I suspect anyone in the d-i group should be able to do so (plus the ssh key dance on dillon afterwards)? I've pinged Hector but apparently he didn't have the procedure in mind. :) It'd be nice to have that (1) for later uses of course, and (2) so that I can set up d-i autobuilding on abel again since it got reinstalled recently. Mraw, KiBi. signature.asc Description: Digital signature
Bug#746967: buildd.debian.org: d-i daily builds happen with unsigned code from alioth
On 2015-07-24 19:49, Cyril Brulebois wrote: Hi Aurélien, Aurelien Jarno aurel...@aurel32.net (2015-03-18): With the help of Hector Oron, we have been able to setup this on the porterbox of 4 architectures: arm64, armel, armhf and mips. This has been done by allowing the d-i role account on the porterboxes. As a nice side effect, this mean that d-i people can now do/fix the setup themselves without having to go through the buildd team. The code used is available in the porterbox branch of the di-autobuild git. Note that the chroots on the porterbox are created in a very similar way than on the newer buildds (it's even done by the same script IIRC), so the problems should be similar. For example both were affected by bug#775136. Is there any doc on how to create the needed bits on a porterbox to add d-i autobuilding? I suspect anyone in the d-i group should be able to do so (plus the ssh key dance on dillon afterwards)? I've pinged Hector but apparently he didn't have the procedure in mind. :) It'd be nice to have that (1) for later uses of course, and (2) so that I can set up d-i autobuilding on abel again since it got reinstalled recently. I don't know if you can consider that as a documentation, but there is a procedure in the README file in the git [1]. Be sure to select the proper branch though (currently 'master' for the code for the buildds and 'porterbox' for the code for the porterboxes). Aurelien [1] https://buildd.debian.org/git/di-autobuild.git -- Aurelien Jarno GPG: 4096R/1DDD8C9B aurel...@aurel32.net http://www.aurel32.net signature.asc Description: Digital signature
Bug#746967: buildd.debian.org: d-i daily builds happen with unsigned code from alioth
Aurelien Jarno aurel...@aurel32.net (2015-07-24): On 2015-07-24 19:49, Cyril Brulebois wrote: Hi Aurélien, Aurelien Jarno aurel...@aurel32.net (2015-03-18): With the help of Hector Oron, we have been able to setup this on the porterbox of 4 architectures: arm64, armel, armhf and mips. This has been done by allowing the d-i role account on the porterboxes. As a nice side effect, this mean that d-i people can now do/fix the setup themselves without having to go through the buildd team. The code used is available in the porterbox branch of the di-autobuild git. Note that the chroots on the porterbox are created in a very similar way than on the newer buildds (it's even done by the same script IIRC), so the problems should be similar. For example both were affected by bug#775136. Is there any doc on how to create the needed bits on a porterbox to add d-i autobuilding? I suspect anyone in the d-i group should be able to do so (plus the ssh key dance on dillon afterwards)? I've pinged Hector but apparently he didn't have the procedure in mind. :) It'd be nice to have that (1) for later uses of course, and (2) so that I can set up d-i autobuilding on abel again since it got reinstalled recently. I don't know if you can consider that as a documentation, but there is a procedure in the README file in the git [1]. Be sure to select the proper branch though (currently 'master' for the code for the buildds and 'porterbox' for the code for the porterboxes). Aurelien [1] https://buildd.debian.org/git/di-autobuild.git Apparently most files are actually already there on abel, and it might be that the only missing bit is the crontab entry (I'll see what happens with the currently running manual build). But the README certainly looks like what I was hoping for, thanks! Mraw, KiBi. signature.asc Description: Digital signature
Bug#746967: buildd.debian.org: d-i daily builds happen with unsigned code from alioth
(Catching up with a number of mails on debian-boot@…) Aurelien Jarno aurel...@aurel32.net (2015-03-18): With the help of Hector Oron, we have been able to setup this on the porterbox of 4 architectures: arm64, armel, armhf and mips. This has been done by allowing the d-i role account on the porterboxes. As a nice side effect, this mean that d-i people can now do/fix the setup themselves without having to go through the buildd team. The code used is available in the porterbox branch of the di-autobuild git. Thank you so much for that! There wasn't a need for fixing the setup yet, but that's a very thing to have. Note that the chroots on the porterbox are created in a very similar way than on the newer buildds (it's even done by the same script IIRC), so the problems should be similar. For example both were affected by bug#775136. Having build environments as close as possible to the ones actually used on buildds is great: this should ensure the amount of last minute surprises is kept at a minimum. Mraw, KiBi. signature.asc Description: Digital signature
Bug#746967: buildd.debian.org: d-i daily builds happen with unsigned code from alioth
On 2014-12-28 13:34, Philipp Kern wrote: On Thu, Sep 11, 2014 at 10:49:06PM +0200, Aurelien Jarno wrote: If we choose this solution, here is a quick and dirty patch against di-autobuild to do that. It's basically changing the hardcoded paths and call to schroot. There is probably more fixes/cleanup to do, but it's just a proof of concept to show this solution works without additional privilege on the porterboxes. Note that I haven't tried the upload part, but I guess it's just a matter of having the right packages installed (if not already the case). We probably want to have a dedicated d-i account for that on the porterbox. Also we have only one amd64/i386 porterbox, and the current script doesn't support that, but that should be easy to test. If this solution is chosen, I'll be happy to continue working on this script as time permits. I'm ok with this approach for the time being. Obviously building on real infrastructure the way other stuff is built would be even better. (For With the help of Hector Oron, we have been able to setup this on the porterbox of 4 architectures: arm64, armel, armhf and mips. This has been done by allowing the d-i role account on the porterboxes. As a nice side effect, this mean that d-i people can now do/fix the setup themselves without having to go through the buildd team. The code used is available in the porterbox branch of the di-autobuild git. Note that the chroots on the porterbox are created in a very similar way than on the newer buildds (it's even done by the same script IIRC), so the problems should be similar. For example both were affected by bug#775136. instance binNMUing the unstable d-i into experimental every day, or similar.) BinNMUing the unstable d-i into experimental actually doesn't really work as it doesn't take into account the changes done on the debian-installer package itself. It is not uploaded that often, so it's important to have daily-builds to test it. Cheers, Aurelien -- Aurelien Jarno GPG: 4096R/1DDD8C9B aurel...@aurel32.net http://www.aurel32.net signature.asc Description: Digital signature
Bug#746967: buildd.debian.org: d-i daily builds happen with unsigned code from alioth
On Thu, Sep 11, 2014 at 10:49:06PM +0200, Aurelien Jarno wrote: If we choose this solution, here is a quick and dirty patch against di-autobuild to do that. It's basically changing the hardcoded paths and call to schroot. There is probably more fixes/cleanup to do, but it's just a proof of concept to show this solution works without additional privilege on the porterboxes. Note that I haven't tried the upload part, but I guess it's just a matter of having the right packages installed (if not already the case). We probably want to have a dedicated d-i account for that on the porterbox. Also we have only one amd64/i386 porterbox, and the current script doesn't support that, but that should be easy to test. If this solution is chosen, I'll be happy to continue working on this script as time permits. I'm ok with this approach for the time being. Obviously building on real infrastructure the way other stuff is built would be even better. (For instance binNMUing the unstable d-i into experimental every day, or similar.) Kind regards Philipp Kern signature.asc Description: Digital signature
Bug#746967: buildd.debian.org: d-i daily builds happen with unsigned code from alioth
On Sun, May 04, 2014 at 03:04:46PM +0200, Ansgar Burchardt wrote:
Hi,
Cyril Brulebois k...@debian.org writes:
I think I already proposed pushing d-i master to some other machine
with less liberal access than alioth's. Would that help? If so, which
machine? dillon? Would pulling from there over https help? Be
sufficient? Otherwise, what else?
I think the buildds building stuff for the offical archive should only
do that. This includes not building for other archives (ports) or daily
images for d-i (no matter the source).
Of course there needs to be a solution for the daily d-i images.
Maybe they could be built on dedicated buildds that are not building
packages for the main archive? Though that would require more hardware.
Or run the daily d-i build as a job on the porter boxes?
If we choose this solution, here is a quick and dirty patch against
di-autobuild to do that. It's basically changing the hardcoded paths
and call to schroot. There is probably more fixes/cleanup to do, but
it's just a proof of concept to show this solution works without
additional privilege on the porterboxes. Note that I haven't tried the
upload part, but I guess it's just a matter of having the right packages
installed (if not already the case).
We probably want to have a dedicated d-i account for that on the
porterbox. Also we have only one amd64/i386 porterbox, and the current
script doesn't support that, but that should be easy to test.
If this solution is chosen, I'll be happy to continue working on this
script as time permits.
Aurelien
diff --git a/buildscript b/buildscript
index 26ba6b7..23f676b 100755
--- a/buildscript
+++ b/buildscript
@@ -8,10 +8,10 @@
# preconditions:
# - a buildd with a LVM setup and hence a unstable-$ARCH-sbuild
#schroot environment
-# - create /home/buildd/di
-# - clone this script into /home/buildd/di/di-autobuild
-#(it will auto-create a d-i checkout in /home/buildd/di/debian-installer)
-# - cron /home/buildd/di/di-autobuild/buildscript to something 0:00-ish
+# - create $HOME/di
+# - clone this script into $HOME/di/di-autobuild
+#(it will auto-create a d-i checkout in $HOME/di/debian-installer)
+# - cron $HOME/di/di-autobuild/buildscript to something 0:00-ish
#as user buildd
set -e
@@ -21,23 +21,23 @@ ARCH=${ARCH:-$(dpkg --print-architecture)}
RSYNC_RETRY=5
LVM_CHROOT=yes # default should be yes; on ancina.d.o we set 'no'
-exec /home/buildd/logs/di-autobuild_daily-${ARCH}-$(date +%Y%m%d-%H%M) 21
+exec $HOME/di/logs/di-autobuild_daily-${ARCH}-$(date +%Y%m%d-%H%M) 21
echo INFO: Daily build started on $(date +%Y%m%d-%H%M)
trap cleanup ERR TERM HUP INT QUIT EXIT
checkout_installer() {
- if [ ! -d /home/buildd/di/debian-installer ]
+ if [ ! -d $HOME/di/debian-installer ]
then
-echo INFO: Doing initial checkout of debian-installer into
/home/buildd/di ...
+echo INFO: Doing initial checkout of debian-installer into $HOME/di ...
(
- cd /home/buildd/di
+ cd $HOME/di
git clone https://alioth.debian.org/anonscm/git/d-i/debian-installer.git
debian-installer
)
fi
echo INFO: Updating debian-installer checkout ...
(
-cd /home/buildd/di/debian-installer
+cd $HOME/di/debian-installer
git pull
git clean -dxf
echo INFO: debian-installer revision $(git show-ref HEAD | cut -d' ' -f1)
@@ -45,29 +45,29 @@ checkout_installer() {
unset TMPDIR
case $LVM_CHROOT in
-yes) INSTALLER_DIR=$(mktemp -d -p /home/buildd/build-trees
installer-XX) ;;
-no) INSTALLER_DIR=$(mktemp -d -p /home/buildd/chroots/sid/build
installer-XX) ;;
+yes) INSTALLER_DIR=$(mktemp -d -p $HOME/di/build installer-XX) ;;
+no) INSTALLER_DIR=$(mktemp -d -p $HOME/chroots/sid/build
installer-XX) ;;
*) echo ERROR: Cannot set installer build directory. ;;
esac
echo INFO: Checking out debian-installer into $INSTALLER_DIR/checkout ...
(
cd ${INSTALLER_DIR}
-git clone /home/buildd/di/debian-installer/ checkout
+git clone $HOME/di/debian-installer/ checkout
)
}
install_build_deps() {
echo INFO: installing packages ...
- schroot -c $SID -d /build -u root -r -- apt-get update
+ dd-schroot-cmd -c $SID apt-get update
(
-LANG=C schroot -q -c $SID -d /build/$(basename
$INSTALLER_DIR)/checkout/build -p -r -- dpkg-checkbuilddeps -B
../debian/control 21 || true
+LANG=C schroot -q -c $SID -d $HOME/di/build/$(basename
$INSTALLER_DIR)/checkout/build -p -r -- dpkg-checkbuilddeps -B
../debian/control 21 || true
) | sed -e 's%dpkg-checkbuilddeps: Unmet build dependencies: %%' -e 's,
*([^)]*),,g' \
-| DEBIAN_PRIORITY=critical DEBIAN_FRONTEND=noninteractive xargs schroot -c
$SID -d /build/$(basename $INSTALLER_DIR)/checkout/build -u root -p -r --
apt-get -y --no-install-recommends install
+| DEBIAN_PRIORITY=critical DEBIAN_FRONTEND=noninteractive xargs
dd-schroot-cmd -c $SID -y apt-get install
}
build_installer() {
echo
Bug#746967: buildd.debian.org: d-i daily builds happen with unsigned code from alioth
Package: buildd.debian.org Severity: normal So apparently nobody cared to document this issue over the past few *years*, and people like to bitch about it on #debian-buildd on a regular fashion instead. Let's address at least the first point with this bug report. I think I already proposed pushing d-i master to some other machine with less liberal access than alioth's. Would that help? If so, which machine? dillon? Would pulling from there over https help? Be sufficient? Otherwise, what else? Very much not amused, KiBi. -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#746967: buildd.debian.org: d-i daily builds happen with unsigned code from alioth
Hi, Cyril Brulebois k...@debian.org writes: I think I already proposed pushing d-i master to some other machine with less liberal access than alioth's. Would that help? If so, which machine? dillon? Would pulling from there over https help? Be sufficient? Otherwise, what else? I think the buildds building stuff for the offical archive should only do that. This includes not building for other archives (ports) or daily images for d-i (no matter the source). Of course there needs to be a solution for the daily d-i images. Maybe they could be built on dedicated buildds that are not building packages for the main archive? Though that would require more hardware. Or run the daily d-i build as a job on the porter boxes? Ansgar -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#746967: buildd.debian.org: d-i daily builds happen with unsigned code from alioth
Hi, Le 2014-05-04 15:04, Ansgar Burchardt a écrit : Hi, Cyril Brulebois k...@debian.org writes: I think I already proposed pushing d-i master to some other machine with less liberal access than alioth's. Would that help? If so, which machine? dillon? Would pulling from there over https help? Be sufficient? Otherwise, what else? I think the buildds building stuff for the offical archive should only do that. This includes not building for other archives (ports) or daily images for d-i (no matter the source). IMHO, if we want to initiate a change there, we should at least say why it is so important to separate those builds. From my POV, the only difference between the content of the Debian archive and an alioth repository is a GPG signature. Having that in mind, would it be acceptable to require a GPG-signed tag to initiate a build from? RedHat has been doing that for years and didn't hear about any major issue with that. Besides, we can also require that buildds building for d-i are configured with throwaway chroots (but i think this became the default now?). The problem is not about running blindly code coming from elsewhere. We are already doing that (DDs can upload anything to build on buildds... they can also upload/test exploits). So, we have to find better arguments before changing this workflow and propose a sustainable alternative plan to build d-i. Maybe they could be built on dedicated buildds that are not building packages for the main archive? Though that would require more hardware. This doesn't seem doable for all architectures. Or run the daily d-i build as a job on the porter boxes? Can we please define what porter boxes are for and stick to that? They are not available hardware to do any stuff. Regards, -- Mehdi -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#746967: buildd.debian.org: d-i daily builds happen with unsigned code from alioth
Mehdi Dogguy me...@dogguy.org writes: Le 2014-05-04 15:04, Ansgar Burchardt a écrit : I think the buildds building stuff for the offical archive should only do that. This includes not building for other archives (ports) or daily images for d-i (no matter the source). IMHO, if we want to initiate a change there, we should at least say why it is so important to separate those builds. From my POV, the only difference between the content of the Debian archive and an alioth repository is a GPG signature. Having that in mind, would it be acceptable to require a GPG-signed tag to initiate a build from? Sure: one problem is that repositories on Alioth should not be trusted. It's quite easy to get a shell account there, so it might get compromised at some time. This should not result in buildds being potentially compromised as well (by running code obtained from Alioth during a build). Signed tags would be an improvement[1], but still have issues: it's an extra entry point into the buildd network and certain attacks are still possible: assume a buggy version of d-i does unsafe things via network during the build, but has a signed tag. One would have to make sure an attacker cannot make the buildd network repeat the build for this old version. [1] Minus minor things like being effectively limited to SHA1 in Git. Uploading signed .changes would also address these problems, but I don't know if that would fit into the workflow the d-i team uses. RedHat has been doing that for years and didn't hear about any major issue with that. Besides, we can also require that buildds building for d-i are configured with throwaway chroots (but i think this became the default now?). Throwaway chroots don't help against processes breaking out of the chroot, see #661037. The problem is not about running blindly code coming from elsewhere. We are already doing that (DDs can upload anything to build on buildds... they can also upload/test exploits). Sure, but those are at least signed by a developer's key. Those are hopefully better protected than a host more or less open to the general public. So, we have to find better arguments before changing this workflow and propose a sustainable alternative plan to build d-i. Maybe they could be built on dedicated buildds that are not building packages for the main archive? Though that would require more hardware. This doesn't seem doable for all architectures. For which architectures would it be a problem to get additional hardware/VMs? Or run the daily d-i build as a job on the porter boxes? Can we please define what porter boxes are for and stick to that? They are not available hardware to do any stuff. Well, the same question applies to buildds. As I said I would really like buildds building packages for the main archive to stick to just doing that. (And the list of things they do in addition to that seems to currently include at least two items: d-i images and building stuff for inofficial archives/ports.d.o. No idea if they do even more.) Ansgar -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
