Thcipriani has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/357741 )
Change subject: Docker for operations-puppet-tests
......................................................................
Docker for operations-puppet-tests
For future iterations, we can build out a series of base images that can
handle broad categories of job. This is very similar to how we spin up
build base images using Debian Image Builder for nodepool and should
require very little maintenance.
One part that doesn't scale with the approach in this patch is the
/run.sh file is very specific to this particular job since it runs both
tox and rake in parallel for speed. Ideally, we could figure out a way
to use the information we have in jjb to automatically generate a file
similar to this one, but this is a first pass.
The part that I do like about this approach is that it moves cloning the
git repository inside the container, so after the container has run
there is nothing left to clean up. This does rely on a bare reference
repository on the host at /srv/git/operations/puppet.git that is mounted
as a read-only volume in the container. We will need to update that repo
periodically.
The total runtime for a change in this instance is ~30 seconds since we
run both tox and rake tasks in parallel.
Bug: T166888
Change-Id: I0a06afd0dae162eb61508b9dd7ddf17f11690a68
---
M .gitignore
A dockerfiles/build.sh
A dockerfiles/contint-operations-puppet/Dockerfile
A dockerfiles/contint-operations-puppet/run.sh
A dockerfiles/share/wikimedia.list
A dockerfiles/share/wikimedia.pref
M jjb/operations-puppet.yaml
7 files changed, 150 insertions(+), 0 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/integration/config
refs/changes/41/357741/1
diff --git a/.gitignore b/.gitignore
index da95b48..e14c72b 100644
--- a/.gitignore
+++ b/.gitignore
@@ -13,3 +13,5 @@
.*.swp
/Gemfile.lock
+
+dockerfiles/share/wikimedia-archive-keyring.gpg
diff --git a/dockerfiles/build.sh b/dockerfiles/build.sh
new file mode 100755
index 0000000..2518136
--- /dev/null
+++ b/dockerfiles/build.sh
@@ -0,0 +1,17 @@
+#!/usr/bin/env bash
+# Use this file to rebuild all docker images in this repository
+
+set -eu
+
+BASE_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+
+wget -O "$BASE_DIR/share/wikimedia-archive-keyring.gpg" \
+ http://apt.wikimedia.org/autoinstall/keyring/wikimedia-archive-keyring.gpg
+
+for dockerbuild in "$BASE_DIR"/contint-*/Dockerfile; do
+ CONTAINER_DIR="${dockerbuild%/*}"
+ CONTAINER_NAME="${CONTAINER_DIR##*/}"
+ docker build -t contint/"${CONTAINER_NAME:8}" -f "${dockerbuild}"
"$BASE_DIR"
+done
+
+rm -rf "$BASE_DIR/share/wikimedia-archive-keyring.gpg" \
diff --git a/dockerfiles/contint-operations-puppet/Dockerfile
b/dockerfiles/contint-operations-puppet/Dockerfile
new file mode 100644
index 0000000..d514e90
--- /dev/null
+++ b/dockerfiles/contint-operations-puppet/Dockerfile
@@ -0,0 +1,39 @@
+FROM debian:jessie
+COPY share /opt
+
+RUN apt-key add /opt/wikimedia-archive-keyring.gpg && \
+ cp /opt/wikimedia.list /etc/apt/sources.list.d/wikimedia.list && \
+ cp /opt/wikimedia.pref /etc/apt/preferences.d/wikimedia.pref
+
+RUN apt-get update && \
+ apt-get install --yes --no-install-recommends \
+ build-essential \
+ git \
+ rubygems-integration \
+ rake \
+ ruby2.1 \
+ ruby2.1-dev \
+ bundler \
+ ruby-rspec \
+ cucumber \
+ python3 \
+ python3-dev \
+ python3-tk \
+ python-dev \
+ python-pip \
+ libmysqlclient-dev \
+ libxml2-dev \
+ libxslt1-dev \
+ libffi-dev \
+ libssl-dev \
+ && \
+ pip install pip==8.1.2 && \
+ pip install tox==1.9.2 setuptools && \
+ groupadd -g 500 wikidev && \
+ useradd -u 2947 -g 500 --home-dir /var/lib/jenkins --create-home
jenkins
+
+USER jenkins
+WORKDIR /var/lib/jenkins
+ENTRYPOINT /bin/bash /run.sh
+
+COPY contint-operations-puppet/run.sh /run.sh
diff --git a/dockerfiles/contint-operations-puppet/run.sh
b/dockerfiles/contint-operations-puppet/run.sh
new file mode 100644
index 0000000..b5a4f90
--- /dev/null
+++ b/dockerfiles/contint-operations-puppet/run.sh
@@ -0,0 +1,45 @@
+#!/usr/bin/env bash
+
+set -eux
+
+TEMP_DIR=$(mktemp -d)
+LOG_DIR="$HOME/.cache/log"
+
+git clone \
+ --reference "/srv/git/${ZUUL_PROJECT}.git" \
+ "$ZUUL_URL/$ZUUL_PROJECT" \
+ "$TEMP_DIR/puppet"
+
+cd "$TEMP_DIR/puppet"
+
+git fetch origin $ZUUL_REF
+git checkout FETCH_HEAD
+
+# Run tox tests
+{
+ set -o pipefail
+ PY_COLORS=1 tox -v | tee "${LOG_DIR}/tox.log"
+ set +o pipefail
+} &
+PID_ONE=$!
+
+# Run rake tests
+{
+ bundle install --path "${HOME}/.gems" --clean
+ bundle exec rake test | tee "${LOG_DIR}/rake.log"
+} &
+PID_TWO=$!
+
+wait $PID_ONE
+EXIT_ONE=$?
+
+wait $PID_TWO
+EXIT_TWO=$?
+
+# Save logs
+mv "${TEMP_DIR}"/puppet/.tox/*/log/*.log "${LOG_DIR}/" || /bin/true
+mv "${TEMP_DIR}"/puppet/.tox/log/* "${LOG_DIR}/" || /bin/true
+
+# Exit non-zero if any subcommand exited non-zero
+(( EXIT_ONE == 0 )) || exit 1
+(( EXIT_TWO == 0 )) || exit 2
diff --git a/dockerfiles/share/wikimedia.list b/dockerfiles/share/wikimedia.list
new file mode 100644
index 0000000..d4c524b
--- /dev/null
+++ b/dockerfiles/share/wikimedia.list
@@ -0,0 +1,2 @@
+deb http://apt.wikimedia.org/wikimedia trusty-wikimedia main universe
thirdparty
+deb-src http://apt.wikimedia.org/wikimedia trusty-wikimedia main universe
thirdparty
diff --git a/dockerfiles/share/wikimedia.pref b/dockerfiles/share/wikimedia.pref
new file mode 100644
index 0000000..d338d23
--- /dev/null
+++ b/dockerfiles/share/wikimedia.pref
@@ -0,0 +1,3 @@
+Package: *
+Pin: release o=Wikimedia
+Pin-Priority: 1001
diff --git a/jjb/operations-puppet.yaml b/jjb/operations-puppet.yaml
index 15c3492..e28f323 100644
--- a/jjb/operations-puppet.yaml
+++ b/jjb/operations-puppet.yaml
@@ -72,6 +72,48 @@
- archive-tox-logs
- castor-save
+- job:
+ name: 'operations-puppet-tests-docker'
+ node: DebianJessieDocker
+ concurrent: true
+ # Reinject Zuul parameters since JJB strip for some reason
+ triggers:
+ - zuul
+ builders:
+ - shell: |
+ #!/bin/bash -eu
+
+ set -x
+
+ rm -rf log
+ rm -rf .env
+
+ cat <<ZUUL > .env
+ ZUUL_URL=$ZUUL_URL
+ ZUUL_PROJECT=$ZUUL_PROJECT
+ ZUUL_COMMIT=$ZUUL_COMMIT
+ ZUUL_REF=$ZUUL_REF
+ HOME=/var/lib/jenkins
+ ZUUL
+
+ mkdir -p "log"
+
+ docker run \
+ --rm --tty \
+ --env-file .env \
+ --volume /srv/git:/srv/git:ro \
+ --volume "$(pwd)"/log:/var/lib/jenkins/.cache/log \
+ contint/operations-puppet
+ publishers:
+ - xunit:
+ types:
+ - junit:
+ pattern: 'log/junit*.xml'
+ # rspec integration is not merged yet
+ skip-if-no-test-files: true
+ stoponerror: false
+ - archive-log-dir
+
- project:
name: 'operations-puppet'
jobs:
--
To view, visit https://gerrit.wikimedia.org/r/357741
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I0a06afd0dae162eb61508b9dd7ddf17f11690a68
Gerrit-PatchSet: 1
Gerrit-Project: integration/config
Gerrit-Branch: master
Gerrit-Owner: Thcipriani <tcipri...@wikimedia.org>
_______________________________________________
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
