Your message dated Mon, 10 Sep 2018 20:57:24 +0000
with message-id <[email protected]>
and subject line [Git][qa/jenkins.debian.net][master] reproducible Debian: use
amd64 kernels on all i386 nodes (Closes: #875990]
has caused the Debian Bug report #875990,
regarding reproducible: i/o issues with profitbricks-build2-i386 since stretch
upgrade
to be marked as done.
This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.
(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact [email protected]
immediately.)
--
875990: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=875990
Debian Bug Tracking System
Contact [email protected] with problems
--- Begin Message ---
Package: jenkins.debian.org
Severity: normal
It looks like after the upgrade to stretch (late june/early july), two
of the i386 builders, profitbricks-build2-i386 and
profitbricks-build12-i386 suddenly developed large i/o issues.
You can see this on the munin graphs for the year, where the blue i/o
wait spikes:
https://jenkins.debian.net/munin/debian.net/profitbricks-build2-i386.debian.net/cpu.html
https://jenkins.debian.net/munin/debian.net/profitbricks-build12-i386.debian.net/cpu.html
Comparing this to the other i386 builders, where there is no huge
spike in i/o wait:
https://jenkins.debian.net/munin/debian.net/profitbricks-build6-i386.debian.net/cpu.html
https://jenkins.debian.net/munin/debian.net/profitbricks-build16-i386.debian.net/cpu.html
I suspect this is reducing the i386 builds per day significantly,
averaging only ~1200 in the last 3 months.
My *hunch* is that build2 and build12 are running a PAE kernel with more
than 8GB of ram, and affected by this kernel bug (introduced in linux
~4.2, possibly):
https://bugzilla.kernel.org/show_bug.cgi?id=196157
https://bugs.launchpad.net/ubuntu/+source/linux-hwe/+bug/1698118
Reducing the ram of the affected builders to 8GB and having more PAE
builders with lighter workloads might be a workaround that would get
better performance... while still testing 32/64-bit kernel
variation.
Alternately, switching to only amd64 kernels might also fix the issue,
though wouldn't test 32/64-bit kernel variations.
Running a linux 4.1 kernel from snapshot.debian.org might be another way
to test the issue, even if not running long-term.
live well,
vagrant
signature.asc
Description: PGP signature
--- End Message ---
--- Begin Message ---
----- Forwarded message from Holger Levsen <[email protected]> -----
Date: Mon, 10 Sep 2018 20:48:48 +0000
From: Holger Levsen <[email protected]>
To: [email protected]
Subject: [Qa-jenkins-scm] [Git][qa/jenkins.debian.net][master] reproducible
Debian: use amd64 kernels on all i386 nodes (Closes: #875990
List-Id: "SCM mails for the development of jenkins.debian.org"
<qa-jenkins-scm.alioth-lists.debian.net>
Reply-To: [email protected]
Holger Levsen pushed to branch master at Debian QA / jenkins.debian.net
Commits:
0fef9342 by Holger Levsen at 2018-09-10T20:48:28Z
reproducible Debian: use amd64 kernels on all i386 nodes (Closes: #875990
Signed-off-by: Holger Levsen <[email protected]>
- - - - -
3 changed files:
- − hosts/profitbricks-build12-i386/etc/apt/sources.list
- − hosts/profitbricks-build2-i386/etc/apt/sources.list
- update_jdn.sh
Changes:
=====================================
hosts/profitbricks-build12-i386/etc/apt/sources.list deleted
=====================================
@@ -1,15 +0,0 @@
-deb http://deb.debian.org/debian/ stretch main contrib non-free
-#deb-src http://deb.debian.org/debian/ stretch main contrib non-free
-
-deb http://deb.debian.org/debian/ stretch-updates main contrib non-free
-#deb-src http://deb.debian.org/debian/ stretch-updates main contrib non-free
-
-deb http://security.debian.org/ stretch/updates main contrib non-free
-#deb-src http://security.debian.org/ stretch/updates main contrib non-free
-
-deb http://deb.debian.org/debian/ stretch-backports main contrib non-free
-#deb-src http://deb.debian.org/debian/ stretch-backports main contrib non-free
-
-# workaround for i386 kernel bugs #875990 + #876035
-deb http://deb.debian.org/debian-security jessie/updates main
-deb http://deb.debian.org/debian jessie main
=====================================
hosts/profitbricks-build2-i386/etc/apt/sources.list deleted
=====================================
@@ -1,15 +0,0 @@
-deb http://deb.debian.org/debian/ stretch main contrib non-free
-#deb-src http://deb.debian.org/debian/ stretch main contrib non-free
-
-deb http://deb.debian.org/debian/ stretch-updates main contrib non-free
-#deb-src http://deb.debian.org/debian/ stretch-updates main contrib non-free
-
-deb http://security.debian.org/ stretch/updates main contrib non-free
-#deb-src http://security.debian.org/ stretch/updates main contrib non-free
-
-deb http://deb.debian.org/debian/ stretch-backports main contrib non-free
-#deb-src http://deb.debian.org/debian/ stretch-backports main contrib non-free
-
-# workaround for i386 kernel bugs #875990 + #876035
-deb http://deb.debian.org/debian-security jessie/updates main
-deb http://deb.debian.org/debian jessie main
=====================================
update_jdn.sh
=====================================
@@ -469,11 +469,12 @@ if [ -f /etc/debian_version ] ; then
$UP2DATE || sudo apt-get install mock
fi
# for varying kernels:
- # - we use bpo kernels on pb-build5+15 (and the default i386
kernel on pb-build2+12-i386)
- # - we use the default amd64 kernel on pb-build1+11 (and the
default amd64 kernel on pb-build6+16-i386)
+ # - we use bpo kernels on pb-build5+15 (and the default amd64
kernel on pb-build6+16-i386)
if [ "$HOSTNAME" = "profitbricks-build5-amd64" ] || [
"$HOSTNAME" = "profitbricks-build15-amd64" ] ; then
$UP2DATE || sudo apt-get install -t stretch-backports
linux-image-amd64
- elif [ "$HOSTNAME" = "profitbricks-build6-i386" ] || [
"$HOSTNAME" = "profitbricks-build16-i386" ] ; then
+ elif [ "$HOSTNAME" = "profitbricks-build6-i386" ] || [
"$HOSTNAME" = "profitbricks-build16-i386" ] \
+ || [ "$HOSTNAME" = "profitbricks-build2-i386" ] || [
"$HOSTNAME" = "profitbricks-build12-i386" ] ; then
+ # we dont vary the kernel on i386 atm, see #875990 +
#876035
$UP2DATE || sudo apt-get install linux-image-amd64
fi
# only needed on the main nodes
View it on GitLab:
https://salsa.debian.org/qa/jenkins.debian.net/commit/0fef93423e8f485a5d2c866b86191c859c6787a4
--
View it on GitLab:
https://salsa.debian.org/qa/jenkins.debian.net/commit/0fef93423e8f485a5d2c866b86191c859c6787a4
You're receiving this email because of your account on salsa.debian.org.
_______________________________________________
Qa-jenkins-scm mailing list
[email protected]
https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/qa-jenkins-scm
----- End forwarded message -----
--
cheers,
Holger
-------------------------------------------------------------------------------
holger@(debian|reproducible-builds|layer-acht).org
PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C
signature.asc
Description: PGP signature
--- End Message ---
