Travis is becoming overall increasingly unreliable lately. We've already
seen that the timing sensitive tests regularly fail and thus they were
disabled.
GitHub Actions VMs are working well, possibly allowing to use custom runners
for special tasks in the future.
In addition to this better performance its workflow configuration language
is more expressive compared to the Travis CI one. Specifically the build
matrix does not need to be specified in YAML. Instead it can be generated
ad-hoc using a script. This allows us to cleanly define the various build
configurations without having an unreadable 80 line mess where the flags
are inconsistently activated. As an example in the current Travis CI
configuration the prometheus exporter is tested together with LibreSSL 2.9.2
for whatever reason.
In addition to all the previous points the UI of Travis is not that nice.
On GitHub you are just seeing that "Travis failed" without any details which
exact job failed. This requires you to visit the slow Travis page and look
up the details there. GitHub Actions creates a single entry for each
configuration that is tested, allowing you to see the details without needing
to leave GitHub.
This new GitHub Actions workflow aims to reproduce the configurations tested
in Travis. It comes close, but is not completely there, yet. Consider this
patch a proof of concept that will evolve in the future, ideally with Ilya's
expertise.
The current configurations are as follows. Each one is tested with both gcc
and clang.
- All features disabled (no USE flags)
- All features enabled (all USE flags)
- Standalone test of each of the supported compression libraries:
- USE_ZLIB=1
- USE_SLZ=1
- Standalone test of various SSL libraries:
- stock (the SSL on the VM)
- OpenSSL 1.0.2u, 1.1.0l, 1.1.1f
- LibreSSL 2.9.2, 3.0.2, 3.1.1
Future additions of new tests should take care to not test unrelated stuff.
Instead a distinct configuration should be added.
Additionally there is a Mac OS test with clang and all features disabled.
Known issues:
- Apparently the git commit is not properly detected during build. The HEAD
currently shows as 2.3-dev1.
- Mac OS currently fails.
- The detailed logs are not printed for failing tests.
---
.github/workflows/vtest.yml | 175 ++++++++++++++++++++++++++++++++++++
1 file changed, 175 insertions(+)
create mode 100644 .github/workflows/vtest.yml
diff --git a/.github/workflows/vtest.yml b/.github/workflows/vtest.yml
new file mode 100644
index 000000000..d94322474
--- /dev/null
+++ b/.github/workflows/vtest.yml
@@ -0,0 +1,175 @@
+# Copyright 2019 Ilya Shipitsin <chipits...@gmail.com>
+# Copyright 2020 Tim Duesterhus <t...@bastelstu.be>
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version
+# 2 of the License, or (at your option) any later version.
+
+name: VTest
+
+on:
+ push:
+
+jobs:
+ generate-matrix:
+ name: Generate Build Matrix
+ runs-on: ubuntu-latest
+ outputs:
+ matrix: ${{ steps.set-matrix.outputs.matrix }}
+ steps:
+ - name: Generate Build Matrix
+ id: set-matrix
+ shell: python
+ run: |
+ import json
+
+ matrix = []
+ os = "ubuntu-latest"
+ TARGET = "linux-glibc"
+ for CC in [ "gcc", "clang" ]:
+ matrix.append({
+ "name": "{} with no features enabled on {}".format(CC, os),
+ "os": os,
+ "TARGET": TARGET,
+ "CC": CC,
+ "FLAGS": ""
+ })
+
+ matrix.append({
+ "name": "{} with all features enabled on {}".format(CC, os),
+ "os": os,
+ "TARGET": TARGET,
+ "CC": CC,
+ "FLAGS": " ".join([
+ "USE_ZLIB=1",
+ "USE_PCRE=1",
+ "USE_PCRE_JIT=1",
+ "USE_LUA=1",
+ "USE_OPENSSL=1",
+ "USE_SYSTEMD=1",
+ # "USE_WURFL=1",
+ # "WURFL_INC=contrib/wurfl",
+ # "WURFL_LIB=contrib/wurfl",
+ # "USE_DEVICEATLAS=1",
+ # "DEVICEATLAS_SRC=contrib/deviceatlas",
+ # "USE_51DEGREES=1",
+ # "FIFTYONEDEGREES_SRC=contrib/51d/src/pattern",
+ ])
+ })
+
+ for compression in [
+ "USE_SLZ=1",
+ "USE_ZLIB=1"
+ ]:
+ matrix.append({
+ "name": "{} with {} on {}".format(CC, compression, os),
+ "os": os,
+ "TARGET": TARGET,
+ "CC": CC,
+ "compression": compression,
+ "FLAGS": compression
+ })
+
+ for ssl in [
+ "stock",
+ "OPENSSL_VERSION=1.0.2u",
+ "OPENSSL_VERSION=1.1.0l",
+ "OPENSSL_VERSION=1.1.1f",
+ "LIBRESSL_VERSION=2.9.2",
+ "LIBRESSL_VERSION=3.0.2",
+ "LIBRESSL_VERSION=3.1.1",
+ ]:
+ flags = ["USE_OPENSSL=1"]
+ if ssl != "stock":
+ flags.append("SSL_LIB=${HOME}/opt/lib")
+ flags.append("SSL_INC=${HOME}/opt/include")
+ matrix.append({
+ "name": "{} with {} SSL on {}".format(CC, ssl, os),
+ "os": os,
+ "TARGET": TARGET,
+ "CC": CC,
+ "ssl": ssl,
+ "FLAGS": " ".join(flags)
+ })
+
+ os = "macos-latest"
+ TARGET = "osx"
+ for CC in [ "clang" ]:
+ matrix.append({
+ "name": "{} with no features enabled on {}".format(CC, os),
+ "os": os,
+ "TARGET": TARGET,
+ "CC": CC,
+ "FLAGS": ""
+ })
+
+ print("::set-output name=matrix::{}".format(json.dumps({
+ "include": matrix
+ })))
+ Test:
+ name: ${{ matrix.name }}
+ needs: generate-matrix
+ runs-on: ${{ matrix.os }}
+ strategy:
+ matrix: ${{ fromJson(needs.generate-matrix.outputs.matrix) }}
+ fail-fast: false
+ steps:
+ - uses: actions/checkout@v2
+ with:
+ fetch-depth: 100
+ - name: Install apt dependencies.
+ if: ${{ startsWith(matrix.os, 'ubuntu-') }}
+ run: |
+ sudo apt-get install -y \
+ liblua5.3-dev \
+ libpcre2-dev \
+ libsystemd-dev \
+ ninja-build \
+ socat
+ - name: Install brew dependencies.
+ if: ${{ startsWith(matrix.os, 'macos-') }}
+ run: |
+ brew install socat
+ brew install lua
+ - name: Install VTest.
+ run: |
+ curl -fsSL https://github.com/vtest/VTest/archive/master.tar.gz -o
VTest.tar.gz
+ mkdir VTest
+ tar xvf VTest.tar.gz -C VTest --strip-components=1
+ make -C VTest -j$(nproc) FLAGS="-O2 -s -Wall"
+ sudo install -m755 VTest/vtest /usr/local/bin/vtest
+ - name: Install SLZ
+ if: ${{ matrix.compression == 'USE_SLZ=1' }}
+ run: |
+ curl -fsSL https://github.com/wtarreau/libslz/archive/master.tar.gz -o
libslz.tar.gz
+ mkdir libslz
+ tar xvf libslz.tar.gz -C libslz --strip-components=1
+ make -C libslz
+ sudo make -C libslz install
+ - name: Install SSL ${{ matrix.ssl }}
+ if: ${{ matrix.ssl && matrix.ssl != 'stock' }}
+ run: env ${{ matrix.ssl }} scripts/build-ssl.sh
+ - name: Compile HAProxy with ${{ matrix.CC }}.
+ run: |
+ make -j$(nproc) all \
+ ERR=1 \
+ TARGET=${{ matrix.TARGET }} \
+ CC=${{ matrix.CC }} \
+ ${{ matrix.FLAGS }} \
+ ADDLIB="-Wl,-rpath,/usr/local/lib/ -Wl,-rpath,$HOME/opt/lib/"
+ sudo make install
+ - name: Show HAProxy version.
+ id: show-version
+ run: |
+ echo "::group::Show dynamic libraries."
+ if command -v ldd > /dev/null; then
+ ldd $(which haproxy)
+ else
+ otool -L $(which haproxy)
+ fi
+ echo "::endgroup::"
+ haproxy -vv
+ echo "::set-output name=version::$(haproxy -v |awk 'NR==1{print $3}')"
+ - name: Run VTest for HAProxy ${{ steps.show-version.outputs.version }}
+ run: make reg-tests
--
2.28.0
