David Caro has uploaded a new change for review. Change subject: Added first mock job and some improvements ......................................................................
Added first mock job and some improvements * ovirt-hosted-engine-ha create-rpms jobs for merged patches * Added README with some info on the organization * Generic mock build scripts and yamls for other jobs to reuse Change-Id: Id77cc184adadbd3b8daa54b3b9428da511496340 Signed-off-by: David Caro <[email protected]> --- M .gitignore A jobs/confs/README A jobs/confs/shell-scripts/mock_build_onlyrpm.sh.inc A jobs/confs/shell-scripts/mock_cleanup.sh A jobs/confs/shell-scripts/mock_setup.sh A jobs/confs/yaml/builders/rpm_mock.yaml M jobs/confs/yaml/defaults/defaults.yaml A jobs/confs/yaml/jobs/create-rpms_mock.yaml R jobs/confs/yaml/jobs/jenkins/jenkins_check_yaml.yaml R jobs/confs/yaml/jobs/jenkins/jenkins_deploy_yamls.yaml A jobs/confs/yaml/jobs/ovirt-hosted-engine-ha/ovirt-hosted-engine-ha_create-rpms.yaml R jobs/confs/yaml/jobs/system/system_gerrit-garbage-collect.yaml M jobs/confs/yaml/parmeters/gerrit.yaml A jobs/confs/yaml/publishers/exported-artifacts.yaml M jobs/confs/yaml/scms/jenkins.yaml A jobs/confs/yaml/scms/ovirt-hosted-engine-ha.yaml A jobs/confs/yaml/triggers/gerrit.yaml 17 files changed, 332 insertions(+), 7 deletions(-) git pull ssh://gerrit.ovirt.org:29418/jenkins refs/changes/01/29001/1 diff --git a/.gitignore b/.gitignore index 4771855..7722882 100644 --- a/.gitignore +++ b/.gitignore @@ -1,2 +1,3 @@ .*.sw? -.*~ +*~ +*\# diff --git a/jobs/confs/README b/jobs/confs/README new file mode 100644 index 0000000..66ba0a7 --- /dev/null +++ b/jobs/confs/README @@ -0,0 +1,52 @@ +This directory contains all the files needed to generate the jobs for the +jenkins-ovirt.org jenkins instance. + +It's organized in directories, being the first level to separate between complete +yaml files and other files to be included inside the jobs. + +The complete yaml files are inside the 'yaml' directory, and for each type of +files to be included (shell-scripts, groovy-scripts, yaml-scripts, ...) there +should be a directory with their name on that same level. + +Inside the yaml dir, there will be another level separating theyamls by type, +and one dir for the jobs, job-templates and projects themselves (all that is +very likely to change to adapt the current needs) + +An example of a possible layout: + +./ + |-yaml + | |-builders + | | |-rpm_mock_builder.yaml + | | \-default_rpm_builder.yaml + | |-jobs + | | \-mycustomjob.yaml + | ... + |-shell-scripts + | |-cleanup_vdsm.sh + | |-build_rpm.sh + | ... + |-groovy-scripts + | |-check_vdsm_functional_tests.groovy + | ... + ... + +NOTE: You need a patched version of the jenkins-job-builder project, with the +recursive patch[1], the parallel patch[2] and the include files patch[3]. + +A better way of getting that modified version will be provided soon. + +Make sure you are at this same level when running jenkins-job builder or that you +have this directory in the include patch for the scripts (see [1] for more info). +To test: +> jenkins-jobs --debug test --recursive -o /tmp/outdir yaml + +To run: +> jenkins-jobs --threads 0 --debug --conf path/to/my/config update --recursive yaml + +More info about jenkins job builder at [4] + +[1] https://review.openstack.org/#/c/85106/ +[2] https://review.openstack.org/#/c/75514/ +[3] https://review.openstack.org/#/c/48783/ +[4] http://ci.openstack.org/jenkins_jobs.html diff --git a/jobs/confs/shell-scripts/mock_build_onlyrpm.sh.inc b/jobs/confs/shell-scripts/mock_build_onlyrpm.sh.inc new file mode 100644 index 0000000..69940c0 --- /dev/null +++ b/jobs/confs/shell-scripts/mock_build_onlyrpm.sh.inc @@ -0,0 +1,94 @@ +#!/bin/bash -xe +# Do some black magic +# PARAMETERS +# +# project +# Name of the project it runs on, specifically the dir where the code +# has been cloned +# +# distro +# Distribution it should cre%ate the repms for (usually el6, el7, fc19 or +# fc20) +# +# arch +# Architecture to build the packages for +# +# extra-packages +# space separated list of extra packages to install, as you would pass to +# yum +# +# extra-configure-options +# extra options to pass to configure + +distro="{distro}" +arch="{arch}" +project="{project}" +extra_packages="{extra-packages}" +extra_configure_options="{extra-configure-options}" + +# Build the src_rpms +# Get the release suffix +pushd "$project" +suffix=".$(date -u +%Y%m%d%H%M%S).git$(git rev-parse --short HEAD)" + +# make sure it's properly clean +git clean -dxf +# build tarballs +./autogen.sh --system +./configure $extra_configure_options +make dist +## build src.rpm +rpmbuild \ + -D "_topdir $WORKSPACE/rpmbuild" \ + -D "_srcrpmdir $WORKSPACE/exported-artifacts" \ + -D "release_suffix ${{suffix}}"\ + -ts *.gz +## we don't need the rpmbuild dir no more +rm -Rf "$WORKSPACE"/rpmbuild + +### Generate the mock configuration +pushd "$WORKSPACE"/jenkins/mock_configs +arch="{arch}" +case $distro in + fc*) distribution="fedora-${{distro#fc}}";; + el*) distribution="epel-${{distro#el}}";; + *) echo "Unknown distro $distro"; exit 1;; +esac +mock_conf="${{distribution}}-$arch-ovirt-snapshot" +echo "#### Generating mock configuration" +./mock_genconfig \ + --name="$mock_conf" \ + --base="$distribution-$arch.cfg" \ + --option="basedir=$WORKSPACE/mock/" \ + --repo="ovirt-master-snapshot,http://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/$distro"; \ + --repo="ovirt-master-snapshot-static,http://resources.ovirt.org/pub/ovirt-master-snapshot-static/rpm/$distro"; \ +> "$mock_conf.cfg" +sudo touch /var/cache/mock/*/root_cache/cache.tar.gz || : +cat "$mock_conf.cfg" +popd + +## prepare the command line +my_mock="/usr/bin/mock" +my_mock+=" --configdir=$WORKSPACE/jenkins/mock_configs" +my_mock+=" --root=$mock_conf" + +## init the chroot +echo "##### Initializing chroot" +$my_mock --init + +### Install any extra packages if needed +if [[ -n "$extra_packages" ]]; then + echo "##### Installing extra dependencies: $extra_packages" + $my_mock \ + --no-clean \ + --install "$extra_packages" +fi + +### Build the rpms +echo "##### Building the rpms" +$my_mock \ + --define="release_suffix $suffix" \ + --rebuild \ + --no-clean \ + --resultdir=$WORKSPACE/exported-artifacts \ + "$WORKSPACE"/exported-artifacts/*.src.rpm diff --git a/jobs/confs/shell-scripts/mock_cleanup.sh b/jobs/confs/shell-scripts/mock_cleanup.sh new file mode 100644 index 0000000..73274a3 --- /dev/null +++ b/jobs/confs/shell-scripts/mock_cleanup.sh @@ -0,0 +1,8 @@ +#!/bin/bash -xe +# remove chroot to free space +sudo rm -Rf mock mock-cache +# compress logs +pushd "$WORKSPACE"/exported-artifacts +shopt -s nullglob +tar cvjf logs.tbz *log *_pkgs "$WORKSPACE"/*log +rm -f *log *_pkgs diff --git a/jobs/confs/shell-scripts/mock_setup.sh b/jobs/confs/shell-scripts/mock_setup.sh new file mode 100644 index 0000000..955c66f --- /dev/null +++ b/jobs/confs/shell-scripts/mock_setup.sh @@ -0,0 +1,14 @@ +#!/bin/bash -xe +# cleanup and setup env +if [[ "$CLEAN_CACHE" == "true" ]]; then + sudo rm -Rf /var/cache/mock +fi +sudo rm -Rf mock mock-cache exported-artifacts +mkdir -p mock exported-artifacts +chgrp mock mock "$WORKSPACE" "$WORKSPACE"/exported-artifacts +chmod g+rws mock + +# Make sure the cache has a newer timestamp than the config file or it will +# not be used +sudo touch /var/cache/mock/*/root_cache/cache.tar.gz || : +exit 0 diff --git a/jobs/confs/yaml/builders/rpm_mock.yaml b/jobs/confs/yaml/builders/rpm_mock.yaml new file mode 100644 index 0000000..221e34c --- /dev/null +++ b/jobs/confs/yaml/builders/rpm_mock.yaml @@ -0,0 +1,7 @@ +- builder: + name: mock-onlyrpm + extra-configure: '{extra-configure}' + builders: + - shell: !include-raw shell-scripts/mock_setup.sh + - shell: !include-raw shell-scripts/mock_build_onlyrpm.sh.inc + - shell: !include-raw shell-scripts/mock_cleanup.sh diff --git a/jobs/confs/yaml/defaults/defaults.yaml b/jobs/confs/yaml/defaults/defaults.yaml index a264bd2..6cbe50c 100644 --- a/jobs/confs/yaml/defaults/defaults.yaml +++ b/jobs/confs/yaml/defaults/defaults.yaml @@ -2,6 +2,11 @@ #### Defaults ############################################################################### - defaults: + description: | + This job is automatically updated by jenkins job builder, any manual + change will be lost in the next update. If you want to make permanent + changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;hb=refs/heads/master";> + jenkins</a> repo. name: global project-type: freestyle concurrent: false diff --git a/jobs/confs/yaml/jobs/create-rpms_mock.yaml b/jobs/confs/yaml/jobs/create-rpms_mock.yaml new file mode 100644 index 0000000..65c251c --- /dev/null +++ b/jobs/confs/yaml/jobs/create-rpms_mock.yaml @@ -0,0 +1,22 @@ +- job-template: + name: '{project}_{branch}_create-rpms-{distro}-{arch}_{trigger}' + parameters: + - gerrit-params: + branch: '{branch}' + scm: + - '{project}-gerrit' + - jenkins: + branch: master + triggers: + - on-patch-{trigger}: + project: '{project}' + branch: '{branch}' + builders: + - mock-{mock-build-type}: + project: '{project}' + distro: '{distro}' + arch: '{arch}' + extra-packages: '{extra-packages}' + extra-configure-options: '{extra-configure-options}' + publishers: + - exported-artifacts diff --git a/jobs/confs/yaml/jobs/jenkins_check_yaml.yaml b/jobs/confs/yaml/jobs/jenkins/jenkins_check_yaml.yaml similarity index 96% rename from jobs/confs/yaml/jobs/jenkins_check_yaml.yaml rename to jobs/confs/yaml/jobs/jenkins/jenkins_check_yaml.yaml index 8343f07..05f930b 100644 --- a/jobs/confs/yaml/jobs/jenkins_check_yaml.yaml +++ b/jobs/confs/yaml/jobs/jenkins/jenkins_check_yaml.yaml @@ -2,7 +2,7 @@ name: jenkins_master_check-yaml_gerrit node: master parameters: - - gerrit_params: + - gerrit-params: branch: master triggers: - gerrit: diff --git a/jobs/confs/yaml/jobs/jenkins_deploy_yamls.yaml b/jobs/confs/yaml/jobs/jenkins/jenkins_deploy_yamls.yaml similarity index 96% rename from jobs/confs/yaml/jobs/jenkins_deploy_yamls.yaml rename to jobs/confs/yaml/jobs/jenkins/jenkins_deploy_yamls.yaml index 7281f3c..276f2a2 100644 --- a/jobs/confs/yaml/jobs/jenkins_deploy_yamls.yaml +++ b/jobs/confs/yaml/jobs/jenkins/jenkins_deploy_yamls.yaml @@ -2,7 +2,7 @@ name: jenkins_master_deploy-configs_merged node: master parameters: - - gerrit_params: + - gerrit-params: branch: master triggers: - gerrit: diff --git a/jobs/confs/yaml/jobs/ovirt-hosted-engine-ha/ovirt-hosted-engine-ha_create-rpms.yaml b/jobs/confs/yaml/jobs/ovirt-hosted-engine-ha/ovirt-hosted-engine-ha_create-rpms.yaml new file mode 100644 index 0000000..6bb6b7c --- /dev/null +++ b/jobs/confs/yaml/jobs/ovirt-hosted-engine-ha/ovirt-hosted-engine-ha_create-rpms.yaml @@ -0,0 +1,25 @@ +- project: + name: ovirt-hosted-engine-ha_create-rpms + project: + - ovirt-hosted-engine-ha + trigger: + - merged + branch: + - master + mock-build-type: + - onlyrpm + distro: + - fc19 + - fc20 + - el6 + arch: + - x86_64 + jobs: + - '{project}_{branch}_create-rpms-{distro}-{arch}_{trigger}': + # Ugly fix until BZ#1111601 is solved + extra-packages: 'm2crypto' + extra-configure-options: | + --prefix=/usr \ + --exec_prefix=/usr \ + --sysconfdir=/etc \ + --localstatedir=/var diff --git a/jobs/confs/yaml/jobs/system_gerrit-garbage-collect.yaml b/jobs/confs/yaml/jobs/system/system_gerrit-garbage-collect.yaml similarity index 100% rename from jobs/confs/yaml/jobs/system_gerrit-garbage-collect.yaml rename to jobs/confs/yaml/jobs/system/system_gerrit-garbage-collect.yaml diff --git a/jobs/confs/yaml/parmeters/gerrit.yaml b/jobs/confs/yaml/parmeters/gerrit.yaml index 615dd08..e671e3d 100644 --- a/jobs/confs/yaml/parmeters/gerrit.yaml +++ b/jobs/confs/yaml/parmeters/gerrit.yaml @@ -1,5 +1,5 @@ - parameter: - name: gerrit_params + name: gerrit-params parameters: - string: name: GERRIT_REFSPEC diff --git a/jobs/confs/yaml/publishers/exported-artifacts.yaml b/jobs/confs/yaml/publishers/exported-artifacts.yaml new file mode 100644 index 0000000..8fb09c5 --- /dev/null +++ b/jobs/confs/yaml/publishers/exported-artifacts.yaml @@ -0,0 +1,6 @@ +- publisher: + name: exported-artifacts + publishers: + - archive: + artifacts: 'exported-artifacts/*' + allow-empty: true diff --git a/jobs/confs/yaml/scms/jenkins.yaml b/jobs/confs/yaml/scms/jenkins.yaml index 5ba49d1..3a7bb62 100644 --- a/jobs/confs/yaml/scms/jenkins.yaml +++ b/jobs/confs/yaml/scms/jenkins.yaml @@ -1,6 +1,3 @@ -############################################################################## -### SCM Definitions -############################################################################## - scm: name: jenkins-master-gerrit scm: @@ -15,3 +12,24 @@ choosing-strategy: gerrit clean: true use-author: true + wipe-workspace: false + skip-tag: true + prune: true + + +- scm: + name: jenkins + scm: + - git: + url: git://gerrit.ovirt.org/jenkins.git + branches: + - '{branch}' + basedir: jenkins + refspec: '' + scm-name: jenkins + name: '' + clean: true + use-author: true + wipe-workspace: false + skip-tag: true + prune: true diff --git a/jobs/confs/yaml/scms/ovirt-hosted-engine-ha.yaml b/jobs/confs/yaml/scms/ovirt-hosted-engine-ha.yaml new file mode 100644 index 0000000..89ebf25 --- /dev/null +++ b/jobs/confs/yaml/scms/ovirt-hosted-engine-ha.yaml @@ -0,0 +1,16 @@ +- scm: + name: ovirt-hosted-engine-ha-gerrit + scm: + - git: + url: git://gerrit.ovirt.org/ovirt-hosted-engine-ha.git + branches: + - $GERRIT_BRANCH + basedir: ovirt-hosted-engine-ha + scm-name: ovirt-hosted-engine + name: '' + refspec: $GERRIT_REFSPEC + choosing-strategy: gerrit + use-author: true + skip-tag: true + prune: true + wipe-workspace: false diff --git a/jobs/confs/yaml/triggers/gerrit.yaml b/jobs/confs/yaml/triggers/gerrit.yaml new file mode 100644 index 0000000..f161194 --- /dev/null +++ b/jobs/confs/yaml/triggers/gerrit.yaml @@ -0,0 +1,57 @@ +- trigger: + name: on-patch-created + triggers: + - gerrit: + trigger-on-patchset-uploaded-event: true + escape-quotes: true + projects: + - project-compare-type: 'PLAIN' + project-pattern: '{project}' + branches: + - branch-compare-type: 'PLAIN' + branch-pattern: '{branch}' + +- trigger: + name: on-patch-merged + triggers: + - gerrit: + trigger-on-change-merged-event: true + escape-quotes: true + projects: + - project-compare-type: 'PLAIN' + project-pattern: '{project}' + branches: + - branch-compare-type: 'PLAIN' + branch-pattern: '{branch}' + +- trigger: + name: on-patch-created-with-files + triggers: + - gerrit: + trigger-on-patchset-uploaded-event: true + escape-quotes: true + projects: + - project-compare-type: 'PLAIN' + project-pattern: '{project}' + branches: + - branch-compare-type: 'PLAIN' + branch-pattern: '{branch}' + file-paths: + - compare-type: REG_EXP + pattern: '{files}' + +- trigger: + name: on-patch-merged-with-files + triggers: + - gerrit: + trigger-on-change-merged-event: true + escape-quotes: true + projects: + - project-compare-type: 'PLAIN' + project-pattern: '{project}' + branches: + - branch-compare-type: 'PLAIN' + branch-pattern: '{branch}' + file-paths: + - compare-type: REG_EXP + pattern: '{files}' -- To view, visit http://gerrit.ovirt.org/29001 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Id77cc184adadbd3b8daa54b3b9428da511496340 Gerrit-PatchSet: 1 Gerrit-Project: jenkins Gerrit-Branch: master Gerrit-Owner: David Caro <[email protected]> _______________________________________________ Engine-patches mailing list [email protected] http://lists.ovirt.org/mailman/listinfo/engine-patches
