The branch master has been updated via f6070cac86caad71cde3b62cd3fd0e35c724eae3 (commit) from 35717050e0abb88170873e8403c369127fb877b1 (commit)
- Log ----------------------------------------------------------------- commit f6070cac86caad71cde3b62cd3fd0e35c724eae3 Author: Tomas Mraz <to...@openssl.org> Date: Thu Sep 2 14:04:36 2021 +0200 Remove run-checker completely as it is not used anymore Reviewed-by: Richard Levitte <levi...@openssl.org> Reviewed-by: Paul Dale <pa...@openssl.org> (Merged from https://github.com/openssl/tools/pull/93) ----------------------------------------------------------------------- Summary of changes: run-checker/README | 88 ----------- run-checker/build-gost.sh | 41 ------ run-checker/run-checker-autohooks/README | 50 ------- run-checker/run-checker-autohooks/hook-end | 100 ------------- run-checker/run-checker-autohooks/hook-prepare | 49 ------- run-checker/run-checker-autohooks/hook-start | 81 ----------- run-checker/run-checker-autohooks/hook-takedown | 7 - run-checker/run-checker-cleanup.sh | 18 --- run-checker/run-checker.sh | 186 ------------------------ 9 files changed, 620 deletions(-) delete mode 100644 run-checker/README delete mode 100755 run-checker/build-gost.sh delete mode 100644 run-checker/run-checker-autohooks/README delete mode 100755 run-checker/run-checker-autohooks/hook-end delete mode 100755 run-checker/run-checker-autohooks/hook-prepare delete mode 100755 run-checker/run-checker-autohooks/hook-start delete mode 100755 run-checker/run-checker-autohooks/hook-takedown delete mode 100755 run-checker/run-checker-cleanup.sh delete mode 100755 run-checker/run-checker.sh diff --git a/run-checker/README b/run-checker/README deleted file mode 100644 index b9540a0..0000000 --- a/run-checker/README +++ /dev/null @@ -1,88 +0,0 @@ -This tool is used to build various predefined config options of -openssl and generate okay/fail reports. The array of config options -is found in run-checker.sh, assigned to 'opts'. - -Quick manual run ----------------- - -To run a check on the master branch: - - git clone git://git.openssl.org/openssl.git openssl - /path/to/run-checker.sh - -To run a check on a release branch: - - git clone -b OpenSSL_1_1_0-stable git://git.openssl.org/openssl.git openssl - /path/to/run-checker.sh - - -Hooks ------ - -run-checker supports a few hooks, in form of scripts that are -executed: - -hook-prepare This script is run once, when run-checker is - starting. If it exits with a status other - than zero, run-checker.sh will stop. It gets - no arguments. - -hook-start This script is run before each option build. - If it exits with a status other than zero, - run-checker.sh will skip the current build. - It gets the following arguments: - - $1 The build directory. - $2 The option being checked. - $3... Configure options and arguments. - -hook-end This script is run after each option build. - If gets the following arguments: - - $1 The build directory. - $2 "pass" or "fail", depending on the - build result. - -hook-takedown This script is run once, just before - run-checker terminates. It gets no arguments. - - -The hooks and documentation in run-checker-autohooks are an advanced -example, and what the OpenSSL Team runs daily (automatically). - - -Example hooks 1 ---------------- - -The run-checker script uses disk space by leaving every build tree -behind! It may be that you want to clear the build tree after each -build. This little hook can help: - -hook-end: - - #! /bin/sh - builddir="$1" - mv "$builddir"/build.log "$builddir".log && rm -rf "$builddir" - -Example hook 2 --------------- - -This is a variant of Example hook 1 that saves away the build dir into -a tarball: - -hook-end: - - #! /bin/sh - builddir="$1" - tar --remove-files -cJf "$builddir.tar.xz" "./$builddir" - -Example hook 3 --------------- - -You might want to avoid some builds, based on the options. For -example, all the fuzz builds may require installations that you're not -willing to do. hook-start is the perfect place for this: - - #! /bin/sh - if [ echo "$2" | grep -E '.*fuzz.*' ]; then exit 1; fi - exit 0 diff --git a/run-checker/build-gost.sh b/run-checker/build-gost.sh deleted file mode 100755 index a6f3468..0000000 --- a/run-checker/build-gost.sh +++ /dev/null @@ -1,41 +0,0 @@ -#! /bin/bash -# -# Run in a directory for a gost engine build. -# Two subdirectories will be created: -# -# gost-engine a checkout of https://github.com/gost-engine/engine.git -# openssl a checkout of https://github.com/openssl/openssl.git -# -# Required ubuntu packages to run this script: -# -# build-essential -# cmake -# perl -# git - -if [ -d openssl ]; then - (cd openssl; git pull --rebase) -else - git clone -b OpenSSL_1_1_0-stable --depth 1 --single-branch \ - https://github.com/openssl/openssl.git openssl -fi - -if [ -d gost-engine ]; then - (cd gost-engine; git pull --rebase) -else - git clone https://github.com/gost-engine/engine.git gost-engine -fi - -OPENSSL_PREFIX=$(pwd)/openssl/_install -( - cd openssl - ./config --prefix=$OPENSSL_PREFIX \ - && make -j8 build_libs \ - && make install_dev -) && ( - cd gost-engine - cmake -DOPENSSL_ROOT_DIR=$OPENSSL_PREFIX \ - -DCMAKE_MODULE_LINKER_FLAGS='-Wl,--enable-new-dtags' \ - . - make -) diff --git a/run-checker/run-checker-autohooks/README b/run-checker/run-checker-autohooks/README deleted file mode 100644 index 87e4be4..0000000 --- a/run-checker/run-checker-autohooks/README +++ /dev/null @@ -1,50 +0,0 @@ -run-checker autohooks -===================== - -The purpose of these run-checker hooks is to provide a foundation for -automatic rebuilds. Setup is easy, just create an empty directory, -copy or symlink run-checker.sh and these hooks into it, clone an -openssl repo into the directory "openssl", then run run-checker.sh -repeatedly, for example as a cron job. - -Example setup: - - $ git clone git://git.openssl.org/tools.git tools - $ mkdir ~/run-checker - $ cd ~/run-checker - $ ln -s ../tools/run-checker/run-checker.sh \ - ../tools/run-checker/run-checker-autohooks/hook-{prepare,start,end,takedown} \ - . - $ git clone openssl-...@git.openssl.org:openssl.git openssl - -A cronjob would look like this: - - 2 */2 * * * cd $HOME/run-checker && bash ./run-checker.sh - - -hook-config ------------ - -In addition to the hooks, one can also have a shell script called -`hook-config` with configuration variables. Currently understood are: - -- REPORT_FROM - - The email address used in the report From: header. This MUST be - assigned a value or reports will not be posted. - -- REPORT_RECIPIENT - - The email address reports will get sent to. This MUST be assigned a - value or reports will not be posted. - -- SKIP_OPTS - - MUST be a bash array, containing run-checker options that should be - skipped. Defaults to the empty array. - -- ONLY_OPTS - - MUST be a bash array. If there are any elements, the elements are - the only run-checker options that will be built. Defaults to the - empty array. diff --git a/run-checker/run-checker-autohooks/hook-end b/run-checker/run-checker-autohooks/hook-end deleted file mode 100755 index c95051b..0000000 --- a/run-checker/run-checker-autohooks/hook-end +++ /dev/null @@ -1,100 +0,0 @@ -#! /bin/bash - -here=$(cd $(dirname $0); pwd) -rcd=$here/.run-checker-data - -REPORT_RECIPIENT= -REPORT_FROM= -if [ -f $rcd/new/hook-config ]; then - . $rcd/new/hook-config -fi - -builddir=$1 -newstatus=$2 - -# Created by hook-start -optdir="$rcd/builds/$builddir" -curoptdir="$optdir/cur" -newoptdir="$optdir/new" - -if ( - set -e - - echo $newstatus > "$newoptdir/status" - cp "./$builddir/build.log" "$newoptdir/build.log" - - # Compress the hell out of the build - tar --remove-files -cJf "./$builddir.tar.xz" "./$builddir" - - curstatus=$(if [ -f "$curoptdir/status" ]; then - cat "$curoptdir/status" - else - # If it was never built previously, pretend the previous - # build passed - echo pass - fi) - gitbranch=$(cat $rcd/new/branch) - expandedopts=$(cat "$newoptdir/options") - - # If the build failed or the status changed since last time, report - if [ "$newstatus" == "fail" -o "$newstatus" != "$curstatus" ]; then - if [ -n "$REPORT_FROM" -a -n "$REPORT_RECIPIENT" ]; then - ( - statusword=FAILED - if [ "$newstatus" == "pass" ]; then - statusword=SUCCESSFUL - elif [ "$curstatus" == "fail" ]; then - statusword="Still FAILED" - fi - echo "From: $REPORT_FROM" - echo "To: $REPORT_RECIPIENT" - echo "Subject: $statusword build of OpenSSL branch $gitbranch with options $expandedopts" - cat <<EOF - -Platform and configuration command: - -EOF - echo "\$ uname -a" - uname -a - head -1 "$newoptdir/build.log" - if [ -f $rcd/new/rc-force-build ]; then - cat <<EOF - -Forced build, latest commit is: - -EOF - else - cat <<EOF - -Commit log since last time: - -EOF - fi - cat "$newoptdir/log" - if [ "$newstatus" == "fail" ]; then - cat <<EOF - -Build log ended with (last 100 lines): - -EOF - tail -100 "$newoptdir/build.log" - fi - ) | /usr/lib/sendmail -it - fi - fi - - if [ -d "$curoptdir" ]; then - mv "$curoptdir" "$curoptdir.old.$$" - fi - mv "$newoptdir" "$curoptdir" - if [ -d "$curoptdir.old.$$" ]; then - rm -rf "$curoptdir.old.$$" - fi -); then - exit 0 -fi - -# Something went wrong. Clear away $newoptdir and let the next run do -# this again -rm -rf "$newoptdir" -exit 1 diff --git a/run-checker/run-checker-autohooks/hook-prepare b/run-checker/run-checker-autohooks/hook-prepare deleted file mode 100755 index 25ce667..0000000 --- a/run-checker/run-checker-autohooks/hook-prepare +++ /dev/null @@ -1,49 +0,0 @@ -#! /bin/bash - -here=$(cd $(dirname $0); pwd) -rcd=$here/.run-checker-data - -# The top directory has this layout: -# -# $rcd/new directory with new stuff... it mostly works as a lock -# -mkdir -p $rcd - -if ! mkdir $rcd/new; then - echo >&2 "There's already a run-checker running" - exit 1 -fi - -# Copy things that are of interest to the rest of the hooks. This allows -# the original files to be added/changed without disturbing *this* run. -if [ -f $here/hook-config ]; then - cp $here/hook-config $rcd/new -fi - -if [ -f $here/rc-force-build ]; then - mv $here/rc-force-build $rcd/new -fi - -# Every build is expected to have its own cur and new layout. For build -# of option 'no-foo', the expected layout is: -# -# $rcd/builds/no-foo/new New stuff -# $rcd/builds/no-foo/cur Stuff from the previous build -# -mkdir -p $rcd/builds - -echo Starting on $(LANG=C date) -if ( - set -e - - cd openssl - git pull --rebase - git rev-parse HEAD > $rcd/new/head - git rev-parse --abbrev-ref HEAD > $rcd/new/branch -); then - exit 0 -fi - -# Something went wrong. Clear away $rcd/new so it doesn't stop a later run -rm -rf $rcd/new -exit 1 diff --git a/run-checker/run-checker-autohooks/hook-start b/run-checker/run-checker-autohooks/hook-start deleted file mode 100755 index cbfa64d..0000000 --- a/run-checker/run-checker-autohooks/hook-start +++ /dev/null @@ -1,81 +0,0 @@ -#! /bin/bash - -here=$(cd $(dirname $0); pwd) -rcd=$here/.run-checker-data - -SKIP_OPTS=() -ONLY_OPTS=() -if [ -f $rcd/new/hook-config ]; then - . $rcd/new/hook-config -fi - -rcd=$here/.run-checker-data -builddir="$1"; shift -opt="$1"; shift - -if [ ${#SKIP_OPTS[@]} -gt 0 ]; then - for x in "${SKIP_OPTS[@]}"; do - if [ "$opt" == "$x" ]; then - echo "Skipping building with $opt (because in SKIP_OPTS)" - exit 1 - fi - done -fi - -if [ ${#ONLY_OPTS[@]} -gt 0 ]; then - bail_out=/bin/true - for x in "${ONLY_OPTS[@]}"; do - if [ "$opt" == "$x" ]; then - bail_out=/bin/false - fi - done - if $bail_out; then - echo "Skipping building with $opt (because not in ONLY_OPTS)" - exit 1 - fi -fi - -optdir="$rcd/builds/$builddir" -curoptdir="$optdir/cur" -newoptdir="$optdir/new" - -mkdir -p "$optdir" -if ( - set -e - - mkdir "$newoptdir" - - echo "$@" > "$newoptdir/options" - cp $rcd/new/head "$newoptdir/head" - newhead=$(cat "$newoptdir/head") - - # The expression in each condition branch will leave behind an exit code - # that is returned back to the calling process - if [ -f $rcd/new/rc-force-build ]; then - ( - cd openssl - git log --pretty=oneline --abbrev-commit ${newhead}^..${newhead} - ) > "$newoptdir/log" - elif [ ! -d "$curoptdir" ]; then - # This is an entirely new build - ( - cd openssl - git log --pretty=oneline --abbrev-commit ${newhead}~20..${newhead} - echo ... - ) > "$newoptdir/log" - else - curhead=$(cat "$curoptdir/head") - ( - cd openssl - git log --pretty=oneline --abbrev-commit ${curhead}..${newhead} - ) > "$newoptdir/log" && [ -n "$(cat "$newoptdir/log")" ] - fi -); then - exit 0 -fi - -# Something went wrong, or there is nothing to do. Clear away $newoptdir so -# it doesn't stop a later run -rm -rf "$newoptdir" -exit 1 - diff --git a/run-checker/run-checker-autohooks/hook-takedown b/run-checker/run-checker-autohooks/hook-takedown deleted file mode 100755 index 8017810..0000000 --- a/run-checker/run-checker-autohooks/hook-takedown +++ /dev/null @@ -1,7 +0,0 @@ -#! /bin/bash - -here=$(cd $(dirname $0); pwd) -rcd=$here/.run-checker-data - -rm -rf $rcd/new -echo Finished on $(LANG=C date) diff --git a/run-checker/run-checker-cleanup.sh b/run-checker/run-checker-cleanup.sh deleted file mode 100755 index 6e8b383..0000000 --- a/run-checker/run-checker-cleanup.sh +++ /dev/null @@ -1,18 +0,0 @@ -#!/bin/bash - -#Script for emergency cleanup of the run-checker.sh work directory -#Place this script in the same directory as run-checker.sh - -#This really just runs the takedown hook, so unless there are any -#hooks present, nothing at all will happen. - -here=$(cd $(dirname $0); pwd) - -run-hook () { - local hookname=$1; shift - if [ -x $here/hook-$hookname ]; then - (cd $here; ./hook-$hookname "$@") - fi -} - -run-hook takedown diff --git a/run-checker/run-checker.sh b/run-checker/run-checker.sh deleted file mode 100755 index b3be342..0000000 --- a/run-checker/run-checker.sh +++ /dev/null @@ -1,186 +0,0 @@ -#!/bin/bash - -#Script to check all available no- options -#Place this script in an empty directory (apart from a few hook scripts, -#read on). -#In the same directory clone openssl into an openssl subdir. -#Then run the script. - -#Some hook scripts can be placed in the same directory and are execute if -#present. They are: -# -# hook-prepare - called before anything starts -# hook-start - called before each option is getting built -# Takes one argument: -# $1 the option being built, which is also the build dir -# $2.. the expanded options -# hook-end - called after each option has been built -# Takes two arguments: -# $1 the option being built, which is also the build dir -# $2 "pass" or "fail" -# hook-takedown - called at the very end - -here=$(cd $(dirname $0); pwd) -opts=( '' ) - -run-hook () { - local hookname=$1; shift - if [ -x $here/hook-$hookname ]; then - (cd $here; ./hook-$hookname "$@") - fi -} - -log-eval () { - echo \$ "$@" - eval "$@" -} - -log-exec () { - echo \$ "$@" - exec "$@" -} - -rkill () { - local signal=$1; shift - local pid=$1; shift - local notpid=$1; shift - - if children="$(pgrep -P "$pid")"; then - for child in $children; do - rkill "$signal" "$child" "$notpid" - done - fi - if [ "$pid" != "$notpid" ]; then - kill -s "$signal" "$pid" - fi -} - -if [ ! -d openssl/.git ]; then - echo >&2 "Missing openssl checkout in openssl/" - exit 1 -fi - -if run-hook prepare; then - for req_binary in clang afl-clang-fast; do - which $req_binary >/dev/null 2>&1 - if [ "$?" != "0" ]; then - echo "Warning: $req_binary does not appear to be in PATH" - fi - done - for opt in "${opts[@]}"; - do - expandedopts="$opt" - warnopts="--strict-warnings" - optcc="clang" - ldcmd="" - gost_engine="$OPENSSL_GOST_ENGINE_SO" - - if [ "$opt" == "enable-asan" ]; then - # A documented requirement for enable-asan is no-shared - expandedopts="enable-asan no-shared no-asm -DOPENSSL_SMALL_FOOTPRINT" - elif [ "$opt" == "enable-ubsan" ]; then - # We've seen it on Travis already, ubsan requires -DPEDANTIC and - # -fno-sanitize=alignment, or crypto/modes will fail to build in - # some circumstances. Running on a VM seems to be one of them. - expandedopts="enable-ubsan no-asm -DPEDANTIC -DOPENSSL_SMALL_FOOTPRINT -fno-sanitize=alignment" - elif [ "$opt" == "enable-fuzz-afl" ]; then - warnopts="" - optcc=afl-clang-fast - expandedopts="enable-fuzz-afl no-shared no-module" - elif [ "$opt" == "enable-fuzz-libfuzzer" ]; then - warnopts="" - ldcmd=clang++ - expandedopts="enable-fuzz-libfuzzer --with-fuzzer-include=../../Fuzzer --with-fuzzer-lib=../../Fuzzer/libFuzzer -DPEDANTIC enable-asan enable-ubsan no-shared" - elif [ "$opt" == "no-static-engine" ]; then - expandedopts="no-static-engine no-shared" - elif [ "$opt" == "no-deprecated" ]; then - #The gost engine uses some deprecated symbols so we don't use it - #in a no-deprecated build - gost_engine="" - elif [ "$opt" == "no-cached-fetch" ]; then - expandedopts="no-cached-fetch enable-asan enable-ubsan" - fi - - if [ -z "$opt" ]; then - builddir=default - else - builddir="$(echo $opt | sed -e 's|[ /]|_|g')" - fi - if run-hook start "$builddir" "$opt" $warnopts $expandedopts; then - if ( - set -e - - if [ ! -d "./$builddir" ]; then - mkdir "./$builddir" - fi - cd "./$builddir" - - echo "Building with '$opt'" - log-eval \ - CC=$optcc ../openssl/config $warnopts $expandedopts \ - >build.log 2>&1 || \ - exit $? - - echo " make clean" - log-eval make clean >>build.log 2>&1 || exit $? - - echo " make depend" - log-eval make depend >>build.log 2>&1 || exit $? - - echo " make -j4" - log-eval LDCMD=$ldcmd make -j4 >>build.log 2>&1 || exit $? - - # Because 'make test' may hang under certain circumstances, - # we have a timeout mechanism around it. - ( - testpid=$BASHPID - - # Number of seconds to wait for command completion. - # (3600 = one hour) - timeout=3600 - # Interval between checks if the process is still alive. - interval=5 - # Delay between posting the SIGTERM signal and destroying - # the process by SIGKILL. - delay=1 - - # kill -0 pid - # Exit code indicates if a signal may be sent to $testpid - # process. - ( - ((t = timeout)) - - while ((t > 0)); do - sleep $interval - kill -0 $testpid || exit 0 - ((t -= interval)) - done - - # Be nice, post SIGTERM first. - # The 'exit 0' below will be executed if any preceeding - # command fails. - rkill SIGTERM $testpid $BASHPID && kill -0 $testpid \ - || exit 0 - sleep $delay - rkill SIGKILL $testpid $BASHPID - ) 2> /dev/null & - - # If not set to another value, default to 4 test jobs - echo " make test" - HARNESS_JOBS=${HARNESS_JOBS:-4} OPENSSL_GOST_ENGINE_SO="$gost_engine" log-exec make test >>build.log 2>&1 - ) - ); then - echo " PASS" - run-hook end "$builddir" pass - else - echo " FAILED" - run-hook end "$builddir" fail - if [ "$opt" = "" ]; then - break - fi - fi - fi - done - - run-hook takedown -fi