Hey Ian,
On Sep 24, 2014, at 2:04 AM, Ian Duffy <i...@ianduffy.ie> wrote:
> I noticed the filename of the generated systemvms changed on
> http://jenkins.buildacloud.org/job/build-systemvm64-master/
>
> They now include a [0-9]* before the hypervisor name representing the build
> number.
That’s one of the things coming out of the 1290e10 merge.
> Why do we do this? Its annoying for external resources linking to the last
> successful build.
The general concept is one of traceable and repeatable builds.
In RPM terms, given just a random binary RPM found on a random system
* it should be possible to unambiguously determine the source RPM from which it
was built
* it should be possible to determine who or what built it, and when
* it should be possible to trace the source RPM back to the exact version
control revision
* it should be possible to determine easily whether that binary is the same as
another one on another machine / in another repository that has the same
identifiers
In RPM land, the jenkins $BUILD_NUMBER should go into the RPM Release field and
not into the RPM Version field. But in this case we don’t have RPM, or spec
files, so there is only the filename to accomplish the same (which is why the
branch name is also in there), and it has to be in this spot or cloudstack gets
a bit confused.
What we do in our integration environment is to apply just a bit of logic to
parse out the version number from the file name. I.e. to get an RPM from
jenkins see fetch-rpms.sh included below, or to fetch the most recent systemvm
from a directory listing see download-template.sh also included below. Would
something like that work for you?
If not, the logic for creating the version is in build.sh:
version_tag=
if [ ! -z "${version}" ]; then
if [ ! -z "${BUILD_NUMBER}" ]; then
version="${version}.${BUILD_NUMBER}"
fi
version_tag="-${version}"
elif [ ! -z "${BUILD_NUMBER}" ]; then
version="${BUILD_NUMBER}"
version_tag="-${BUILD_NUMBER}”
fi
So we can change that, or we can unset BUILD_NUMBER in the jenkins build prior
to invocation. What do you think?
It would be much better to exclusively use yum/maven repositories and their
logic to centralize this kind of logic server-side, or at least to be creating
and updating symlinks to ‘latest’. That’s a work in progress...
cheers,
Leo
download-templates.sh
=====================
#!/bin/bash
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
# Download systemvm templates onto secondary storage
set -e
echo $@ | grep debug >/dev/null && DEBUG="1"
echo $@ | grep trace >/dev/null && TRACE="1"
function source_common() {
local SOURCE="${BASH_SOURCE[0]}"
DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
source "${DIR}/common.sh"
}
source_common
NFS_HOST="${1:-localhost}"
NFS_SECONDARY_EXPORT="${2:-/storage/secondary}"
MOUNT_PATH="${3:-/mnt/secondary}"
BASE_URL="${4:-https://artifacts.schubergphilis.com/artifacts/cloudstack}";
TEMPLATE="${5:-systemvmtemplate}"
DB_HOST="${6:-localhost}"
DB_USER="${7:-root}"
DB_PASSWORD="${8:-}"
function find_template() {
if ! `echo "${TEMPLATE}" | egrep '^[a-zA-Z0-9]+-[0-9]+(\.[0-9]+)*$'`; then
# (apache) directory listing
# find only links
# filter by template name
# find only xen template
# strip suffix
# take the first (newest) one
newest_first="?C=M;O=D"
TEMPLATE=$(curl "${BASE_URL}/${newest_first}" | \
egrep -o 'href="[^"]+"' | \
egrep "${TEMPLATE}" | \
egrep -o "[^/\"]+-xen\.vhd\.bz2" | \
sed 's/-xen.vhd.bz2//' |\
head -n 1)
if [[ "${TEMPLATE}" == "" ]]; then
error "No possible match found for ${TEMPLATE} at ${BASE_URL}"
fi
log INFO "Using template ${TEMPLATE}"
fi
}
function install_xen_template() {
XEN_URL="${BASE_URL}/${TEMPLATE}-xen.vhd.bz2"
# KVM_URL="${BASE_URL}/${TEMPLATE}-kvm.vhd.bz2"
[ -d "${MOUNT_PATH}" ] || mkdir "${MOUNT_PATH}"
mount -t nfs "${NFS_HOST}:${NFS_SECONDARY_EXPORT}" "${MOUNT_PATH}" || true
/usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt \
-F \
-m "${MOUNT_PATH}" \
-h "xenserver" \
-u "${XEN_URL}" \
-o "${DB_HOST}" \
-r "${DB_USER}" \
-d "${DB_PASSWORD}"
}
function main() {
find_template
install_xen_template
}
# we only run main() if not source-d
return 2>/dev/null || main
fetch-rpms.sh
=============
#!/bin/bash
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
# script to install http://jenkins.buildacloud.org/ RPMs to
# the local repo
function usage() {
cat <<END
Usage:
./fetch-rpms.sh [archiveUrl]
* Pass [archiveURL] to link to the build artifacts
(default is
http://jenkins.buildacloud.org/job/
package-rhel63-master/
lastSuccessfulBuild/artifact/*zip*
/archive.zip
)
* Set \$DEBUG=1 to enable debug logging
* Set \$TRACE=1 to enable trace logging
END
exit 0
}
echo $@ | grep help >/dev/null && usage
echo $@ | grep '\-h' >/dev/null && usage
echo $@ | grep debug >/dev/null && DEBUG="1"
echo $@ | grep trace >/dev/null && TRACE="1"
function source_common() {
# from http://stackoverflow.com/a/246128
local SOURCE="${BASH_SOURCE[0]}"
local DIR=""
while [ -h "$SOURCE" ]; do
# resolve $SOURCE until the file is no longer a symlink
DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
SOURCE="$(readlink "${SOURCE}")"
# if $SOURCE was a relative symlink, we need to resolve it relative
# to the path where the symlink file was located
[[ ${SOURCE} != /* ]] && SOURCE="${DIR}/${SOURCE}"
done
DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
source "${DIR}/common.sh"
}
source_common
###
### Configuration
###
if [[ "$1" != "" ]]; then
ARCHIVE_URL="${1}"
else
JENKINS="http://jenkins.buildacloud.org";
JOB="package-rhel63-master"
ARCHIVE_URL="${JENKINS}/job/${JOB}/lastSuccessfulBuild/artifact/*zip*/archive.zip"
fi
STORAGE_DIR="${BASEDIR}/../storage"
ARCHIVE_FILE="${STORAGE_DIR}/archive.zip"
###
### Script logic
###
function prepare() {
cd "${TARGET_DIR}"
init_settings
}
function run() {
local do_download=1
if [[ -f "${ARCHIVE_FILE}" ]]; then
if [[ "${CI_RUN}" == "1" ]]; then
log WARN "${ARCHIVE_FILE} already exists, removing"
rm -f "${ARCHIVE_FILE}"
else
log INFO "${ARCHIVE_FILE} already exists, reusing"
do_download=0
fi
fi
if [[ ${do_download} -eq 1 ]]; then
log INFO "Downloading ${ARCHIVE_URL}"
curl -o "${ARCHIVE_FILE}" "${ARCHIVE_URL}"
fi
if [[ -d "${STORAGE_DIR}/archive" ]]; then
rm -rf "${STORAGE_DIR}/archive"
fi
mkdir "${STORAGE_DIR}/archive"
cd "${STORAGE_DIR}"
unzip -o "${ARCHIVE_FILE}"
mkdir -p "${STORAGE_DIR}/rpmrepo/cloudstack-${CLOUDSTACK_VERSION}"
local numfiles=$(find \
${STORAGE_DIR}/archive \
-name "cloudstack*${CLOUDSTACK_VERSION}*.rpm" | \
wc -l | awk '{print $1}')
if [[ ${numfiles} -lt 4 ]]; then
error "Expecting at least 4 rpms matching
cloudstack*${CLOUDSTACK_VERSION}*.rpm, but got only ${numfiles}"
fi
find ${STORAGE_DIR}/archive -name "cloudstack*${CLOUDSTACK_VERSION}*.rpm"
-exec \
mv \{\} "${STORAGE_DIR}/rpmrepo/cloudstack-${CLOUDSTACK_VERSION}/" \;
rm -rf "${STORAGE_DIR}/archive"
}
# we only run main() if not source-d
return 2>/dev/null || main
