Your message dated Wed, 14 Jun 2017 17:39:12 +0000
with message-id <[email protected]>
and subject line Re: [Qa-jenkins-dev] Bug#864686: reproducible: reduce number
of armhf builds per node
has caused the Debian Bug report #864686,
regarding reproducible: reduce number of armhf builds per node
to be marked as done.
This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.
(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact [email protected]
immediately.)
--
864686: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=864686
Debian Bug Tracking System
Contact [email protected] with problems
--- Begin Message ---
Package: jenkins.debian.org
Severity: normal
Please consider merging the following patch, available in the
"armhf-reduced-builds" branch:
https://anonscm.debian.org/cgit/users/vagrant/jenkins.debian.net.git/
This may help reduce the impacts of Out-of-Memory errors on nodes when
building many packages simultaneously, though will likely result in a
slower build network...
live well,
vagrant
commit 314becd4473586e1fe47cb05477276aef9630a87
Author: Vagrant Cascadian <[email protected]>
Date: Mon Jun 12 13:08:44 2017 -0700
reproducible: Reduce number of armhf build jobs to 2 for 1gb of ram, 4
for 2gb of ram, and 6 for 4gb of ram, for a total of 51 jobs. Remove
disabled jobs for odc2a.
diff --git a/README b/README
index 02854897..ded67155 100644
--- a/README
+++ b/README
@@ -135,7 +135,7 @@ Installation tests inside chroot environments.
* The (current) purpose of https://tests.reproducible-builds.org is to show
the potential of reproducible builds for
Debian - and six other projects currently. This is research, showing what could
(and should) be done... check
https://wiki.debian.org/ReproducibleBuilds for the real status of the project
for Debian!
-* For Debian, three suites, 'testing', 'unstable' and 'experimental', are
tested on four architectures: 'amd64',
'i386', 'arm64' and 'armhf'. The tests are done using 'pbuilder' through
several concurrent workers: 40 for 'amd64',
24 for 'i386', 32 for 'arm64' and 76 for 'armhf', which are each constantly
testing packages and saving the results
of these tests. There's a single
link:https://anonscm.debian.org/git/qa/jenkins.debian.net.git/tree/bin/reproducible_build_service.sh[systemd
service]
starting all of these
link:https://anonscm.debian.org/git/qa/jenkins.debian.net.git/tree/bin/reproducible_worker.sh[workers]
which in turn
launch the actual
link:https://anonscm.debian.org/git/qa/jenkins.debian.net.git/tree/bin/reproducible_build.sh[build
script]. (So the actual builds and tests are happening outside the jenkins
service.)
+* For Debian, three suites, 'testing', 'unstable' and 'experimental', are
tested on four architectures: 'amd64',
'i386', 'arm64' and 'armhf'. The tests are done using 'pbuilder' through
several concurrent workers: 40 for 'amd64',
24 for 'i386', 32 for 'arm64' and 51 for 'armhf', which are each constantly
testing packages and saving the results
of these tests. There's a single
link:https://anonscm.debian.org/git/qa/jenkins.debian.net.git/tree/bin/reproducible_build_service.sh[systemd
service]
starting all of these
link:https://anonscm.debian.org/git/qa/jenkins.debian.net.git/tree/bin/reproducible_worker.sh[workers]
which in turn
launch the actual
link:https://anonscm.debian.org/git/qa/jenkins.debian.net.git/tree/bin/reproducible_build.sh[build
script]. (So the actual builds and tests are happening outside the jenkins
service.)
* These builds on remote nodes run on very different hardware:
** for 'amd64' we are using four virtual machines,
profitbricks-build(1+5+11+15)-amd64, which have 15 or 16 cores
and 48gb ram each. These nodes are sponsored by
link:https://jenkins.debian.net/userContent/thanks.html[Profitbricks].
diff --git a/bin/reproducible_build_service.sh
b/bin/reproducible_build_service.sh
index 79bf24ee..500f15a6 100755
--- a/bin/reproducible_build_service.sh
+++ b/bin/reproducible_build_service.sh
@@ -116,93 +116,68 @@ choose_nodes() {
arm64_31) NODE1=codethink-sled14-arm64
NODE2=codethink-sled11-arm64 ;;
arm64_32) NODE1=codethink-sled13-arm64
NODE2=codethink-sled14-arm64 ;;
# to choose new armhf jobs:
- # for i in cb3a hb0 rpi2b rpi2c wbd0 bpi0 bbx15
cbxi4pro0 ff2a ff2b ff64a jtk1a odc2a odxu4
odxu4b odxu4c odu3a opi2a opi2b opi2c p64b p64c wbq0 cbxi4a cbxi4b ff4a
jtx1a ; do echo "$i: " ; grep NODE1
bin/reproducible_build_service.sh|grep armhf|grep $i-armhf ; done
- # 8 jobs for quad-cores with 4 gb ram
- # 6 jobs for octo-cores with 2 gb ram
- # 6 jobs for hexa-cores with 2 gb ram
- # 6 jobs for quad-cores with 2 gb ram
- # 3 jobs for dual-cores with 1 gb ram
- # 3 jobs for quad-cores with 1 gb ram
+ # for i in cb3a hb0 rpi2b rpi2c wbd0 bpi0 bbx15 cbxi4pro0 ff2a
ff2b ff64a jtk1a odxu4 odxu4b odxu4c
odu3a opi2a opi2b opi2c p64b p64c wbq0 cbxi4a cbxi4b ff4a jtx1a ; do echo "$i:
" ; grep NODE1
bin/reproducible_build_service.sh|grep armhf|grep $i-armhf ; done
+ # 6 jobs for quad-cores with 4 gb ram
+ # 4 jobs for octo-cores with 2 gb ram
+ # 4 jobs for hexa-cores with 2 gb ram
+ # 4 jobs for quad-cores with 2 gb ram
+ # 4 jobs for dual-cores with 2 gb ram
+ # 2 jobs for dual-cores with 1 gb ram
+ # 2 jobs for quad-cores with 1 gb ram
#
# Don't forget to update README with the number of builders…!
#
- armhf_1) NODE1=bbx15-armhf-rb
NODE2=odxu4-armhf-rb ;;
- armhf_2) NODE1=wbq0-armhf-rb
NODE2=p64c-armhf-rb ;;
- armhf_3) NODE1=hb0-armhf-rb
NODE2=p64b-armhf-rb ;;
- armhf_4) NODE1=ff4a-armhf-rb
NODE2=wbq0-armhf-rb ;;
- armhf_5) NODE1=cbxi4pro0-armhf-rb
NODE2=bpi0-armhf-rb ;;
- armhf_6) NODE1=ff4a-armhf-rb
NODE2=cbxi4pro0-armhf-rb ;;
- armhf_7) NODE1=wbq0-armhf-rb
NODE2=odxu4-armhf-rb ;;
- armhf_8) NODE1=hb0-armhf-rb
NODE2=wbq0-armhf-rb ;;
- armhf_9) NODE1=ff4a-armhf-rb
NODE2=bpi0-armhf-rb ;;
- armhf_10) NODE1=odxu4-armhf-rb
NODE2=rpi2b-armhf-rb ;;
- armhf_11) NODE1=odxu4-armhf-rb
NODE2=wbd0-armhf-rb ;;
- armhf_12) NODE1=wbd0-armhf-rb
NODE2=cbxi4pro0-armhf-rb ;;
- armhf_13) NODE1=cbxi4pro0-armhf-rb
NODE2=rpi2b-armhf-rb ;;
- armhf_14) NODE1=cbxi4a-armhf-rb
NODE2=odxu4b-armhf-rb ;;
- armhf_15) NODE1=rpi2b-armhf-rb
NODE2=odxu4c-armhf-rb ;;
- armhf_16) NODE1=odxu4b-armhf-rb
NODE2=wbd0-armhf-rb ;;
- armhf_17) NODE1=odxu4c-armhf-rb
NODE2=hb0-armhf-rb ;;
- armhf_18) NODE1=odxu4b-armhf-rb
NODE2=odu3a-armhf-rb ;;
- armhf_19) NODE1=odxu4c-armhf-rb
NODE2=opi2c-armhf-rb ;;
- armhf_20) NODE1=opi2b-armhf-rb
NODE2=odxu4b-armhf-rb ;;
- armhf_21) NODE1=ff2a-armhf-rb
NODE2=odxu4c-armhf-rb ;;
- armhf_22) NODE1=ff2a-armhf-rb
NODE2=rpi2c-armhf-rb ;;
- armhf_23) NODE1=rpi2c-armhf-rb
NODE2=odxu4b-armhf-rb ;;
- armhf_24) NODE1=rpi2c-armhf-rb
NODE2=odxu4c-armhf-rb ;;
- armhf_25) NODE1=odxu4b-armhf-rb
NODE2=ff2b-armhf-rb ;;
- armhf_26) NODE1=jtk1a-armhf-rb
NODE2=ff2a-armhf-rb ;;
- armhf_27) NODE1=odxu4c-armhf-rb
NODE2=cbxi4a-armhf-rb ;;
- armhf_28) NODE1=jtk1a-armhf-rb
NODE2=ff2b-armhf-rb ;;
- armhf_29) NODE1=ff2b-armhf-rb
NODE2=jtk1a-armhf-rb ;;
- armhf_30) NODE1=ff2b-armhf-rb
NODE2=cbxi4b-armhf-rb ;;
- armhf_31) NODE1=ff2b-armhf-rb
NODE2=opi2b-armhf-rb ;;
- armhf_32) NODE1=jtk1a-armhf-rb
NODE2=cbxi4b-armhf-rb ;;
- armhf_33) NODE1=ff2a-armhf-rb
NODE2=opi2b-armhf-rb ;;
- armhf_34) NODE1=cbxi4a-armhf-rb
NODE2=opi2b-armhf-rb ;;
- armhf_35) NODE1=opi2a-armhf-rb
NODE2=ff2b-armhf-rb ;;
- armhf_36) NODE1=opi2a-armhf-rb
NODE2=cbxi4a-armhf-rb ;;
- armhf_37) NODE1=opi2a-armhf-rb
NODE2=wbq0-armhf-rb ;;
- armhf_38) NODE1=cbxi4b-armhf-rb
NODE2=jtk1a-armhf-rb ;;
- armhf_39) NODE1=cbxi4b-armhf-rb
NODE2=cbxi4a-armhf-rb ;;
- armhf_40) NODE1=opi2b-armhf-rb
NODE2=cbxi4b-armhf-rb ;;
- armhf_41) NODE1=opi2b-armhf-rb
NODE2=cbxi4b-armhf-rb ;;
- armhf_42) NODE1=cbxi4b-armhf-rb
NODE2=cbxi4a-armhf-rb ;;
- armhf_43) NODE1=cbxi4a-armhf-rb
NODE2=opi2c-armhf-rb ;;
- armhf_44) NODE1=bbx15-armhf-rb
NODE2=ff4a-armhf-rb ;;
- armhf_45) NODE1=ff4a-armhf-rb
NODE2=p64b-armhf-rb ;;
- armhf_46) NODE1=wbq0-armhf-rb
NODE2=bbx15-armhf-rb ;;
- armhf_47) NODE1=cbxi4pro0-armhf-rb
NODE2=bbx15-armhf-rb ;;
- armhf_48) NODE1=bbx15-armhf-rb
NODE2=p64c-armhf-rb ;;
- armhf_49) NODE1=bpi0-armhf-rb
NODE2=ff4a-armhf-rb ;;
- armhf_50) NODE1=odxu4-armhf-rb
NODE2=odu3a-armhf-rb ;;
- armhf_51) NODE1=odu3a-armhf-rb
NODE2=cb3a-armhf-rb ;;
- armhf_52) NODE1=opi2c-armhf-rb
NODE2=cb3a-armhf-rb ;;
- armhf_53) NODE1=cb3a-armhf-rb
NODE2=ff4a-armhf-rb ;;
- armhf_54) NODE1=odu3a-armhf-rb
NODE2=opi2c-armhf-rb ;;
- armhf_55) NODE1=opi2c-armhf-rb
NODE2=odu3a-armhf-rb ;;
- armhf_56) NODE1=odu3a-armhf-rb
NODE2=ff2a-armhf-rb ;;
- armhf_57) NODE1=opi2c-armhf-rb
NODE2=ff64a-armhf-rb ;;
- armhf_58) NODE1=cbxi4a-armhf-rb
NODE2=ff64a-armhf-rb ;;
- armhf_59) NODE1=jtk1a-armhf-rb
NODE2=ff64a-armhf-rb ;;
- armhf_60) NODE1=cbxi4b-armhf-rb
NODE2=jtx1a-armhf-rb ;;
- armhf_61) NODE1=p64c-armhf-rb
NODE2=jtx1a-armhf-rb ;;
- armhf_62) NODE1=p64b-armhf-rb
NODE2=jtx1a-armhf-rb ;;
- armhf_63) NODE1=p64b-armhf-rb
NODE2=jtx1a-armhf-rb ;;
-# armhf_64) NODE1=p64c-armhf-rb
NODE2=odc2a-armhf-rb ;;
-# armhf_65) NODE1=p64b-armhf-rb
NODE2=odc2a-armhf-rb ;;
-# armhf_66) NODE1=p64c-armhf-rb
NODE2=odc2a-armhf-rb ;;
- armhf_67) NODE1=ff64a-armhf-rb
NODE2=ff2a-armhf-rb ;;
- armhf_68) NODE1=ff64a-armhf-rb
NODE2=p64b-armhf-rb ;;
- armhf_69) NODE1=ff64a-armhf-rb
NODE2=p64c-armhf-rb ;;
- armhf_70) NODE1=jtx1a-armhf-rb
NODE2=opi2a-armhf-rb ;;
- armhf_71) NODE1=jtx1a-armhf-rb
NODE2=opi2a-armhf-rb ;;
- armhf_72) NODE1=jtx1a-armhf-rb
NODE2=opi2a-armhf-rb ;;
- armhf_73) NODE1=jtx1a-armhf-rb
NODE2=ff4a-armhf-rb ;;
-# armhf_74) NODE1=odc2a-armhf-rb
NODE2=bbx15-armhf-rb ;;
-# armhf_75) NODE1=odc2a-armhf-rb
NODE2=cbxi4pro0-armhf-rb ;;
-# armhf_76) NODE1=odc2a-armhf-rb
NODE2=odxu4-armhf-rb ;;
-
+ armhf_1) NODE1=bbx15-armhf-rb
NODE2=jtx1a-armhf-rb ;;
+ armhf_2) NODE1=bbx15-armhf-rb
NODE2=rpi2b-armhf-rb ;;
+ armhf_3) NODE1=cb3a-armhf-rb
NODE2=bbx15-armhf-rb ;;
+ armhf_4) NODE1=cb3a-armhf-rb
NODE2=bbx15-armhf-rb ;;
+ armhf_5) NODE1=hb0-armhf-rb
NODE2=cb3a-armhf-rb ;;
+ armhf_6) NODE1=rpi2b-armhf-rb
NODE2=cb3a-armhf-rb ;;
+ armhf_7) NODE1=rpi2c-armhf-rb
NODE2=wbd0-armhf-rb ;;
+ armhf_8) NODE1=wbd0-armhf-rb
NODE2=rpi2c-armhf-rb ;;
+ armhf_9) NODE1=bpi0-armhf-rb
NODE2=cbxi4pro0-armhf-rb ;;
+ armhf_10) NODE1=cbxi4pro0-armhf-rb
NODE2=bpi0-armhf-rb ;;
+ armhf_11) NODE1=cbxi4pro0-armhf-rb
NODE2=ff2a-armhf-rb ;;
+ armhf_12) NODE1=ff2a-armhf-rb
NODE2=cbxi4pro0-armhf-rb ;;
+ armhf_13) NODE1=ff2a-armhf-rb
NODE2=ff2b-armhf-rb ;;
+ armhf_14) NODE1=ff2b-armhf-rb
NODE2=ff2a-armhf-rb ;;
+ armhf_15) NODE1=ff2b-armhf-rb
NODE2=ff64a-armhf-rb ;;
+ armhf_16) NODE1=ff64a-armhf-rb
NODE2=ff2b-armhf-rb ;;
+ armhf_17) NODE1=ff64a-armhf-rb
NODE2=jtk1a-armhf-rb ;;
+ armhf_18) NODE1=jtk1a-armhf-rb
NODE2=ff64a-armhf-rb ;;
+ armhf_19) NODE1=jtk1a-armhf-rb
NODE2=odxu4-armhf-rb ;;
+ armhf_20) NODE1=odxu4-armhf-rb
NODE2=jtk1a-armhf-rb ;;
+ armhf_21) NODE1=odxu4-armhf-rb
NODE2=odxu4b-armhf-rb ;;
+ armhf_22) NODE1=odxu4b-armhf-rb
NODE2=odxu4-armhf-rb ;;
+ armhf_23) NODE1=odxu4b-armhf-rb
NODE2=odxu4c-armhf-rb ;;
+ armhf_24) NODE1=odxu4c-armhf-rb
NODE2=odxu4b-armhf-rb ;;
+ armhf_25) NODE1=odxu4c-armhf-rb
NODE2=odu3a-armhf-rb ;;
+ armhf_26) NODE1=odu3a-armhf-rb
NODE2=odxu4c-armhf-rb ;;
+ armhf_27) NODE1=odu3a-armhf-rb
NODE2=opi2a-armhf-rb ;;
+ armhf_28) NODE1=opi2a-armhf-rb
NODE2=odu3a-armhf-rb ;;
+ armhf_29) NODE1=opi2a-armhf-rb
NODE2=opi2b-armhf-rb ;;
+ armhf_30) NODE1=opi2b-armhf-rb
NODE2=opi2a-armhf-rb ;;
+ armhf_31) NODE1=opi2b-armhf-rb
NODE2=opi2c-armhf-rb ;;
+ armhf_32) NODE1=opi2c-armhf-rb
NODE2=opi2b-armhf-rb ;;
+ armhf_33) NODE1=opi2c-armhf-rb
NODE2=p64b-armhf-rb ;;
+ armhf_34) NODE1=p64b-armhf-rb
NODE2=opi2c-armhf-rb ;;
+ armhf_35) NODE1=p64b-armhf-rb
NODE2=p64c-armhf-rb ;;
+ armhf_36) NODE1=p64c-armhf-rb
NODE2=p64b-armhf-rb ;;
+ armhf_37) NODE1=p64c-armhf-rb
NODE2=wbq0-armhf-rb ;;
+ armhf_38) NODE1=wbq0-armhf-rb
NODE2=p64c-armhf-rb ;;
+ armhf_39) NODE1=wbq0-armhf-rb
NODE2=cbxi4a-armhf-rb ;;
+ armhf_40) NODE1=cbxi4a-armhf-rb
NODE2=wbq0-armhf-rb ;;
+ armhf_41) NODE1=cbxi4a-armhf-rb
NODE2=cbxi4b-armhf-rb ;;
+ armhf_42) NODE1=cbxi4a-armhf-rb
NODE2=cbxi4b-armhf-rb ;;
+ armhf_43) NODE1=cbxi4b-armhf-rb
NODE2=cbxi4a-armhf-rb ;;
+ armhf_44) NODE1=cbxi4b-armhf-rb
NODE2=cbxi4a-armhf-rb ;;
+ armhf_45) NODE1=cbxi4b-armhf-rb
NODE2=ff4a-armhf-rb ;;
+ armhf_46) NODE1=ff4a-armhf-rb
NODE2=cbxi4b-armhf-rb ;;
+ armhf_47) NODE1=ff4a-armhf-rb
NODE2=jtx1a-armhf-rb ;;
+ armhf_48) NODE1=ff4a-armhf-rb
NODE2=jtx1a-armhf-rb ;;
+ armhf_49) NODE1=jtx1a-armhf-rb
NODE2=ff4a-armhf-rb ;;
+ armhf_50) NODE1=jtx1a-armhf-rb
NODE2=ff4a-armhf-rb ;;
+ armhf_51) NODE1=jtx1a-armhf-rb
NODE2=hb0-armhf-rb ;;
*) NODE1=undefined
;;
esac
signature.asc
Description: PGP signature
--- End Message ---
--- Begin Message ---
hi,
On Wed, Jun 14, 2017 at 09:38:19AM -0700, Vagrant Cascadian wrote:
> Patch follows with no duplicated builds, available in the
> "armhf-reduced-builds" branch:
thanks!
> For what it's worth, this is exactly the sort of thing, "besides
> complexity" that I wanted to fix by splitting the first build, second
> build and diffoscope tasks into separate jobs/queues is meant to solve;
> having to manually shuffle build job pairings in order to reduce the
> impact of downed machines, adding new machines, etc. would be a
> non-issue with such a system.
I understand and understood.
> Maybe it's too much work to implement as a
> trade-off, but it would eliminate certain classes of busywork like this.
sure. just how much time did you spent on this now?
I've certainly spend more time arguing about this, than doing changes like this.
Plus, changing this system now does not only have the costs of doing the change,
but also making sure it runs as reliable as the current system. And please
believe
me, but *that* task is HUGE.
Plus, your itch is an optimisation for an existing system. I rather would like
to work on *new* things, like integrating other distros or rebuilding whats in
the Debian archive.
We can also continue this discussion, sometimes I do like to talk.
--
cheers,
Holger
signature.asc
Description: Digital signature
--- End Message ---
