Re: [PATCH v2 1/1] Jobs based on custom runners: add CentOS Stream 8

2021-11-12 Thread Willian Rampazzo
On Thu, Nov 11, 2021 at 1:06 PM Cleber Rosa  wrote:
>
> This introduces three different parts of a job designed to run
> on a custom runner managed by Red Hat.  The goals include:
>
>   a) propose a model for other organizations that want to onboard
>  their own runners, with their specific platforms, build
>  configuration and tests.
>
>   b) bring awareness to the differences between upstream QEMU and the
>  version available under CentOS Stream, which is "A preview of
>  upcoming Red Hat Enterprise Linux minor and major releases".
>
>   c) because of b), it should be easier to identify and reduce the gap
>  between Red Hat's downstream and upstream QEMU.
>
> The components of this custom job are:
>
>   I) OS build environment setup code:
>
>  - additions to the existing "build-environment.yml" playbook
>that can be used to set up CentOS/EL 8 systems.
>
>  - a CentOS Stream 8 specific "build-environment.yml" playbook
>that adds to the generic one.
>
>  II) QEMU build configuration: a script that will produce binaries with
>  features as similar as possible to the ones built and packaged on
>  CentOS stream 8.
>
> III) Scripts that define the minimum amount of testing that the
>  binaries built with the given configuration (point II) under the
>  given OS build environment (point I) should be subjected to.
>
>  IV) Job definition: GitLab CI jobs that will dispatch the build/test
>  jobs (see points #II and #III) to the machine specifically
>  configured according to #I.
>
> Signed-off-by: Cleber Rosa 
> ---
>  .gitlab-ci.d/custom-runners.yml   |  29 +++
>  docs/devel/ci-jobs.rst.inc|   7 +
>  .../org.centos/stream/8/build-environment.yml |  51 +
>  .../ci/org.centos/stream/8/x86_64/configure   | 208 ++
>  .../org.centos/stream/8/x86_64/test-avocado   |  70 ++
>  scripts/ci/org.centos/stream/README   |  17 ++
>  scripts/ci/setup/build-environment.yml|  38 
>  7 files changed, 420 insertions(+)
>  create mode 100644 scripts/ci/org.centos/stream/8/build-environment.yml
>  create mode 100755 scripts/ci/org.centos/stream/8/x86_64/configure
>  create mode 100755 scripts/ci/org.centos/stream/8/x86_64/test-avocado
>  create mode 100644 scripts/ci/org.centos/stream/README
>

Maybe it is too late, but just for the records:

Reviewed-by: Willian Rampazzo 
Tested-by: Willian Rampazzo 

CI job on a custom VM runner:
https://gitlab.com/willianrampazzo/qemu/-/jobs/1778451942




[PATCH v2 1/1] Jobs based on custom runners: add CentOS Stream 8

2021-11-11 Thread Cleber Rosa
This introduces three different parts of a job designed to run
on a custom runner managed by Red Hat.  The goals include:

  a) propose a model for other organizations that want to onboard
 their own runners, with their specific platforms, build
 configuration and tests.

  b) bring awareness to the differences between upstream QEMU and the
 version available under CentOS Stream, which is "A preview of
 upcoming Red Hat Enterprise Linux minor and major releases".

  c) because of b), it should be easier to identify and reduce the gap
 between Red Hat's downstream and upstream QEMU.

The components of this custom job are:

  I) OS build environment setup code:

 - additions to the existing "build-environment.yml" playbook
   that can be used to set up CentOS/EL 8 systems.

 - a CentOS Stream 8 specific "build-environment.yml" playbook
   that adds to the generic one.

 II) QEMU build configuration: a script that will produce binaries with
 features as similar as possible to the ones built and packaged on
 CentOS stream 8.

III) Scripts that define the minimum amount of testing that the
 binaries built with the given configuration (point II) under the
 given OS build environment (point I) should be subjected to.

 IV) Job definition: GitLab CI jobs that will dispatch the build/test
 jobs (see points #II and #III) to the machine specifically
 configured according to #I.

Signed-off-by: Cleber Rosa 
---
 .gitlab-ci.d/custom-runners.yml   |  29 +++
 docs/devel/ci-jobs.rst.inc|   7 +
 .../org.centos/stream/8/build-environment.yml |  51 +
 .../ci/org.centos/stream/8/x86_64/configure   | 208 ++
 .../org.centos/stream/8/x86_64/test-avocado   |  70 ++
 scripts/ci/org.centos/stream/README   |  17 ++
 scripts/ci/setup/build-environment.yml|  38 
 7 files changed, 420 insertions(+)
 create mode 100644 scripts/ci/org.centos/stream/8/build-environment.yml
 create mode 100755 scripts/ci/org.centos/stream/8/x86_64/configure
 create mode 100755 scripts/ci/org.centos/stream/8/x86_64/test-avocado
 create mode 100644 scripts/ci/org.centos/stream/README

diff --git a/.gitlab-ci.d/custom-runners.yml b/.gitlab-ci.d/custom-runners.yml
index a89a20da48..1f56297dfa 100644
--- a/.gitlab-ci.d/custom-runners.yml
+++ b/.gitlab-ci.d/custom-runners.yml
@@ -248,3 +248,32 @@ ubuntu-20.04-aarch64-notcg:
  - ../configure --disable-libssh --disable-tcg
  - make --output-sync -j`nproc`
  - make --output-sync -j`nproc` check V=1
+
+centos-stream-8-x86_64:
+ allow_failure: true
+ needs: []
+ stage: build
+ tags:
+ - centos_stream_8
+ - x86_64
+ rules:
+ - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH =~ 
/^staging/'
+ - if: "$CENTOS_STREAM_8_x86_64_RUNNER_AVAILABLE"
+ artifacts:
+   name: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG"
+   when: on_failure
+   expire_in: 7 days
+   paths:
+ - build/tests/results/latest/results.xml
+ - build/tests/results/latest/test-results
+   reports:
+ junit: build/tests/results/latest/results.xml
+ before_script:
+ - JOBS=$(expr $(nproc) + 1)
+ script:
+ - mkdir build
+ - cd build
+ - ../scripts/ci/org.centos/stream/8/x86_64/configure
+ - make -j"$JOBS"
+ - make NINJA=":" check
+ - ../scripts/ci/org.centos/stream/8/x86_64/test-avocado
diff --git a/docs/devel/ci-jobs.rst.inc b/docs/devel/ci-jobs.rst.inc
index 277975e4ad..db3f571d5f 100644
--- a/docs/devel/ci-jobs.rst.inc
+++ b/docs/devel/ci-jobs.rst.inc
@@ -49,3 +49,10 @@ S390X_RUNNER_AVAILABLE
 If you've got access to an IBM Z host that can be used as a gitlab-CI
 runner, you can set this variable to enable the tests that require this
 kind of host. The runner should be tagged with "s390x".
+
+CENTOS_STREAM_8_x86_64_RUNNER_AVAILABLE
+~~~
+If you've got access to a CentOS Stream 8 x86_64 host that can be
+used as a gitlab-CI runner, you can set this variable to enable the
+tests that require this kind of host. The runner should be tagged with
+both "centos_stream_8" and "x86_64".
diff --git a/scripts/ci/org.centos/stream/8/build-environment.yml 
b/scripts/ci/org.centos/stream/8/build-environment.yml
new file mode 100644
index 00..42b0471634
--- /dev/null
+++ b/scripts/ci/org.centos/stream/8/build-environment.yml
@@ -0,0 +1,51 @@
+---
+- name: Installation of extra packages to build QEMU
+  hosts: all
+  tasks:
+- name: Extra check for CentOS Stream 8
+  lineinfile:
+path: /etc/redhat-release
+line: CentOS Stream release 8
+state: present
+  check_mode: yes
+  register: centos_stream_8
+
+- name: Enable PowerTools repo on CentOS Stream 8
+  ini_file:
+path: /etc/yum.repos.d/CentOS-Stream-PowerTools.repo
+section: powertools
+option: enabled
+value: "1"
+  when:
+- ansible_facts['distribution'] == 'CentOS'
+- ansible_facts['distribution_major_version'] == '8'
+