The branch, master has been updated via 88d1b44 autobuild: Add test for cross-compilation infrastructure via 4a0d599 waf: add a script that compares configurations via 13ce285 waf: add a cross-execute program that runs natively via 1c58fee waf: add a sample run-on-target script via 25b1721 waf: Fix parsing of cross-answers file in case answer includes a colon via 56114bb waf: allow cross-execute program to have arguments with spaces via c244e7d waf: improve readability of cross-answers generated by cross-execute via eb696df Adds a new mode to samba cross-compiling. via 3fb6173 waf: sanitize and fix added cross answer via 843c373 waf: always close cross answers file from 0dcf535 Factor out submodule presence checking.
https://git.samba.org/?p=samba.git;a=shortlog;h=master - Log ----------------------------------------------------------------- commit 88d1b44d150c5eaeea32c08f98c2b26f44dbeede Author: Uri Simchoni <urisimch...@gmail.com> Date: Mon May 4 10:21:53 2015 +0300 autobuild: Add test for cross-compilation infrastructure With added option to generate cross-answers by cross-execute script, a test is added to verify that at least for tested build environments, the infrastructure is self-consistent. The test runs samba configuration in three modes: 1. Native 2. Cross compile with cross-execute, using the build machine as its own target 3. Cross compile with cross-answers, using the answers generated in step 2 The build environments created by the three processes are then tested to be equal. Signed-off-by: Uri Simchoni <urisimch...@gmail.com> Reviewed-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Alexander Bokovoy <a...@samba.org> Autobuild-User(master): Andrew Bartlett <abart...@samba.org> Autobuild-Date(master): Wed May 20 14:10:05 CEST 2015 on sn-devel-104 commit 4a0d599518bc6312e887d1bb4c2b9063c1c78d83 Author: Uri Simchoni <urisimch...@gmail.com> Date: Tue May 19 11:50:41 2015 +0300 waf: add a script that compares configurations This script is intended for use with the cross-build infrastructure self-test during autobuild. The script takes two or more "default.cache.py" files which are the result of the waf configure process, and performs an intelligent comparison - allowing for a limited set of variables to be different. If the configurations are identical (apart from allowed differences) then the script exit with 0 code. If there are differences, the script outputs them to standard output in unified diff format. Signed-off-by: Uri Simchoni <urisimch...@gmail.com> Reviewed-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Alexander Bokovoy <a...@samba.org> commit 13ce285b24d4f1a3e9e1e2f6eb45a53ff9f2d642 Author: Uri Simchoni <urisimch...@gmail.com> Date: Tue May 19 11:44:03 2015 +0300 waf: add a cross-execute program that runs natively This added script is for supporting self-testing of the cross-build infrastructure using autobuild. The script can be used as the --cross-execute parameter to samba's configure process, but it actually runs the program natively on the build machine. This can be done if the build is actually a native build disguised as a cross-build, i.e. native toolchain is being used. Signed-off-by: Uri Simchoni <urisimch...@gmail.com> Reviewed-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Alexander Bokovoy <a...@samba.org> commit 1c58fee706e02410966d3f9db89a7525fd2cb8a7 Author: Uri Simchoni <urisimch...@gmail.com> Date: Mon May 4 22:03:54 2015 +0300 waf: add a sample run-on-target script This is a sample script that can be used with the --cross-execute configure switch to run configure tests on a remote target for which samba is being cross-compiled. Signed-off-by: Uri Simchoni <urisimch...@gmail.com> Reviewed-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Alexander Bokovoy <a...@samba.org> commit 25b17210753117170ba1e40a3cbda0b97e208199 Author: Uri Simchoni <urisimch...@gmail.com> Date: Mon May 4 09:12:45 2015 +0300 waf: Fix parsing of cross-answers file in case answer includes a colon The answer provided in the cross-answers file may include a colon, as in: Checking uname version type: "#57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014" Signed-off-by: Uri Simchoni <urisimch...@gmail.com> Reviewed-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Alexander Bokovoy <a...@samba.org> commit 56114bb6778a3515cf8da34a1413b86fe1f904ca Author: Uri Simchoni <urisimch...@gmail.com> Date: Tue May 19 11:09:55 2015 +0300 waf: allow cross-execute program to have arguments with spaces When cross-compiling samba using the cross-execute method, allow the cross-excute command to have arguments that contain spaces by quoting them. For example: ./configure --cross-compile '--cross-execute=ce-program "par am"' In this case, for each cross test whose binary is /path/testprog, waf shall run the equivalent of running from a shell: ce-program "par am" /path/testprog This is useful for passing an arbitrary argument list to a subprocess of the cross-execute program, such as in buildtools/examples/run_on_target.py Signed-off-by: Uri Simchoni <urisimch...@gmail.com> Reviewed-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Alexander Bokovoy <a...@samba.org> commit c244e7dc09e9c98f678cbe716fe2e640de0bd4c0 Author: Uri Simchoni <urisimch...@gmail.com> Date: Sun May 3 22:56:15 2015 +0300 waf: improve readability of cross-answers generated by cross-execute When generating a result for cross-answers from the (retcode, retstring) tuple: - (0, "output") indicated as "output" - 1 is interpreted as generic fail code, instead of 255, because most if not all tests fail with 1 as exit code rather than 255 - For failing test, use NO instead of FAIL, because that's not necessarily a failure (it could mean that something is NOT broken) Signed-off-by: Uri Simchoni <urisimch...@gmail.com> Reviewed-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Alexander Bokovoy <a...@samba.org> commit eb696df4936bc48bb8fa7077bb5b45e5e2e6cb01 Author: Uri Simchoni <urisimch...@gmail.com> Date: Mon May 18 21:15:19 2015 +0300 Adds a new mode to samba cross-compiling. When both --cross-answers and --cross-execute are set, this means: - Use cross-answers - If answer is unknown, then instead of adding UNKNOWN to the cross-answers file and failing configure, the new mode runs cross-execute to determine the answer and adds that to the cross-answers file. Signed-off-by: Uri Simchoni <urisimch...@gmail.com> Reviewed-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Alexander Bokovoy <a...@samba.org> commit 3fb617310df73795fbb761253f27052ae5afa40b Author: Uri Simchoni <urisimch...@gmail.com> Date: Mon May 18 21:12:06 2015 +0300 waf: sanitize and fix added cross answer When configuring samba for cross-compilation using the cross-answers method, the function add_answer receives the standard output and exit code of a configuration test and updates the cross-answers file accordingly. This patch sanitizes the standard output to conform to the cross-answers file format - one line of output. It also adds a missing newline. (Note - at this point add_answer is only ever called with empty output but this change is significant for the reminder of this patchset) Signed-off-by: Uri Simchoni <urisimch...@gmail.com> Reviewed-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Alexander Bokovoy <a...@samba.org> commit 843c373d008cd51940f08a7a291976b6e80a5fbf Author: Uri Simchoni <urisimch...@gmail.com> Date: Mon May 18 20:40:11 2015 +0300 waf: always close cross answers file When configuring samba for cross-compilation using the cross-answers file, the file is not closed in a couple of cases - fix that. Signed-off-by: Uri Simchoni <urisimch...@gmail.com> Reviewed-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Alexander Bokovoy <a...@samba.org> ----------------------------------------------------------------------- Summary of changes: buildtools/examples/run_on_target.py | 148 +++++++++++++++++++++++++++++++++++ buildtools/wafsamba/samba_cross.py | 75 +++++++++++++----- script/autobuild.py | 18 ++++- script/compare_cc_results.py | 40 ++++++++++ script/identity_cc.sh | 10 +++ 5 files changed, 269 insertions(+), 22 deletions(-) create mode 100755 buildtools/examples/run_on_target.py create mode 100755 script/compare_cc_results.py create mode 100755 script/identity_cc.sh Changeset truncated at 500 lines: diff --git a/buildtools/examples/run_on_target.py b/buildtools/examples/run_on_target.py new file mode 100755 index 0000000..8322759 --- /dev/null +++ b/buildtools/examples/run_on_target.py @@ -0,0 +1,148 @@ +#!/usr/bin/env python + +# +# Sample run-on-target script +# This is a script that can be used as cross-execute parameter to samba +# configuration process, running the command on a remote target for which +# the cross-compiled configure test was compiled. +# +# To use: +# ./configure \ +# --cross-compile \ +# '--cross-execute=./buildtools/example/run_on_target.py --host=<host>' +# +# A more elaborate example: +# ./configure \ +# --cross-compile \ +# '--cross-execute=./buildtools/example/run_on_target.py --host=<host> --user=<user> "--ssh=ssh -i <some key file>" --destdir=/path/to/dir' +# +# Typically this is to be used also with --cross-answers, so that the +# cross answers file gets built and further builds can be made without +# the help of a remote target. +# +# The following assumptions are made: +# 1. rsync is available on build machine and target machine +# 2. A running ssh service on target machine with password-less shell login +# 3. A directory writable by the password-less login user +# 4. The tests on the target can run and provide reliable results +# from the login account's home directory. This is significant +# for example in locking tests which +# create files in the current directory. As a workaround to this +# assumption, the TESTDIR environment variable can be set on the target +# (using ssh command line or server config) and the tests shall +# chdir to that directory. +# + +import sys +import os +import subprocess +from optparse import OptionParser + +# those are defaults, but can be overidden using command line +SSH = 'ssh' +USER = None +HOST = 'localhost' + + +def xfer_files(ssh, srcdir, host, user, targ_destdir): + """Transfer executable files to target + + Use rsync to copy the directory containing program to run + INTO a destination directory on the target. An exact copy + of the source directory is created on the target machine, + possibly deleting files on the target machine which do not + exist on the source directory. + + The idea is that the test may include files in addition to + the compiled binary, and all of those files reside alongside + the binary in a source directory. + + For example, if the test to run is /foo/bar/test and the + destination directory on the target is /tbaz, then /tbaz/bar + on the target shall be an exact copy of /foo/bar on the source, + including deletion of files inside /tbaz/bar which do not exist + on the source. + """ + + userhost = host + if user: + userhost = '%s@%s' % (user, host) + + cmd = 'rsync --verbose -rl --ignore-times --delete -e "%s" %s %s:%s/' % \ + (ssh, srcdir, userhost, targ_destdir) + p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, + stderr=subprocess.PIPE) + (out, err) = p.communicate() + if p.returncode != 0: + raise Exception('failed syncing files\n stdout:\n%s\nstderr:%s\n' + % (out, err)) + + +def exec_remote(ssh, host, user, destdir, targdir, prog, args): + """Run a test on the target + + Using password-less ssh, run the compiled binary on the target. + + An assumption is that there's no need to cd into the target dir, + same as there's no need to do it on a native build. + """ + userhost = host + if user: + userhost = '%s@%s' % (user, host) + + cmd = '%s %s %s/%s/%s' % (ssh, userhost, destdir, targdir, prog) + if args: + cmd = cmd + ' ' + ' '.join(args) + p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, + stderr=subprocess.PIPE) + (out, err) = p.communicate() + return (p.returncode, out) + + +def main(argv): + usage = "usage: %prog [options] <prog> [args]" + parser = OptionParser(usage) + + parser.add_option('--ssh', help="SSH client and additional flags", + default=SSH) + parser.add_option('--host', help="target host name or IP address", + default=HOST) + parser.add_option('--user', help="login user on target", + default=USER) + parser.add_option('--destdir', help="work directory on target", + default='~') + + (options, args) = parser.parse_args(argv) + if len(args) < 1: + parser.error("please supply test program to run") + + progpath = args[0] + + # assume that a test that was not compiled fails (e.g. getconf) + if progpath[0] != '/': + return (1, "") + + progdir = os.path.dirname(progpath) + prog = os.path.basename(progpath) + targ_progdir = os.path.basename(progdir) + + xfer_files( + options.ssh, + progdir, + options.host, + options.user, + options.destdir) + + (rc, out) = exec_remote(options.ssh, + options.host, + options.user, + options.destdir, + targ_progdir, + prog, args[1:]) + return (rc, out) + + +if __name__ == '__main__': + (rc, out) = main(sys.argv[1:]) + sys.stdout.write(out) + sys.exit(rc) diff --git a/buildtools/wafsamba/samba_cross.py b/buildtools/wafsamba/samba_cross.py index 3838e34..ed3af1e 100644 --- a/buildtools/wafsamba/samba_cross.py +++ b/buildtools/wafsamba/samba_cross.py @@ -2,11 +2,12 @@ import Utils, Logs, sys, os, Options, re from Configure import conf +import shlex real_Popen = None ANSWER_UNKNOWN = (254, "") -ANSWER_FAIL = (255, "") +ANSWER_NO = (1, "") ANSWER_OK = (0, "") cross_answers_incomplete = False @@ -19,15 +20,27 @@ def add_answer(ca_file, msg, answer): except: Logs.error("Unable to open cross-answers file %s" % ca_file) sys.exit(1) + (retcode, retstring) = answer + # if retstring is more than one line then we probably + # don't care about its actual content (the tests should + # yield one-line output in order to comply with the cross-answer + # format) + retstring = retstring.strip() + if len(retstring.split('\n')) > 1: + retstring = '' + answer = (retcode, retstring) + if answer == ANSWER_OK: f.write('%s: OK\n' % msg) elif answer == ANSWER_UNKNOWN: f.write('%s: UNKNOWN\n' % msg) - elif answer == ANSWER_FAIL: - f.write('%s: FAIL\n' % msg) + elif answer == ANSWER_NO: + f.write('%s: NO\n' % msg) else: - (retcode, retstring) = answer - f.write('%s: (%d, "%s")' % (msg, retcode, retstring)) + if retcode == 0: + f.write('%s: "%s"\n' % (msg, retstring)) + else: + f.write('%s: (%d, "%s")\n' % (msg, retcode, retstring)) f.close() @@ -36,14 +49,13 @@ def cross_answer(ca_file, msg): try: f = open(ca_file, 'r') except: - add_answer(ca_file, msg, ANSWER_UNKNOWN) return ANSWER_UNKNOWN for line in f: line = line.strip() if line == '' or line[0] == '#': continue if line.find(':') != -1: - a = line.split(':') + a = line.split(':', 1) thismsg = a[0].strip() if thismsg != msg: continue @@ -56,10 +68,12 @@ def cross_answer(ca_file, msg): return ANSWER_UNKNOWN elif ans == "FAIL" or ans == "NO": f.close() - return ANSWER_FAIL + return ANSWER_NO elif ans[0] == '"': + f.close() return (0, ans.strip('"')) elif ans[0] == "'": + f.close() return (0, ans.strip("'")) else: m = re.match('\(\s*(-?\d+)\s*,\s*\"(.*)\"\s*\)', ans) @@ -69,7 +83,6 @@ def cross_answer(ca_file, msg): else: raise Utils.WafError("Bad answer format '%s' in %s" % (line, ca_file)) f.close() - add_answer(ca_file, msg, ANSWER_UNKNOWN) return ANSWER_UNKNOWN @@ -77,24 +90,47 @@ class cross_Popen(Utils.pproc.Popen): '''cross-compilation wrapper for Popen''' def __init__(*k, **kw): (obj, args) = k - - if '--cross-execute' in args: - # when --cross-execute is set, then change the arguments - # to use the cross emulator - i = args.index('--cross-execute') - newargs = args[i+1].split() - newargs.extend(args[0:i]) - args = newargs - elif '--cross-answers' in args: + use_answers = False + ans = ANSWER_UNKNOWN + + # Three possibilities: + # 1. Only cross-answers - try the cross-answers file, and if + # there's no corresponding answer, add to the file and mark + # the configure process as unfinished. + # 2. Only cross-execute - get the answer from cross-execute + # 3. Both - try the cross-answers file, and if there is no + # corresponding answer - use cross-execute to get an answer, + # and add that answer to the file. + if '--cross-answers' in args: # when --cross-answers is set, then change the arguments # to use the cross answers if available + use_answers = True i = args.index('--cross-answers') ca_file = args[i+1] msg = args[i+2] ans = cross_answer(ca_file, msg) + + if '--cross-execute' in args and ans == ANSWER_UNKNOWN: + # when --cross-execute is set, then change the arguments + # to use the cross emulator + i = args.index('--cross-execute') + newargs = shlex.split(args[i+1]) + newargs.extend(args[0:i]) + if use_answers: + p = real_Popen(newargs, + stdout=Utils.pproc.PIPE, + stderr=Utils.pproc.PIPE) + ce_out, ce_err = p.communicate() + ans = (p.returncode, ce_out) + add_answer(ca_file, msg, ans) + else: + args = newargs + + if use_answers: if ans == ANSWER_UNKNOWN: global cross_answers_incomplete cross_answers_incomplete = True + add_answer(ca_file, msg, ans) (retcode, retstring) = ans args = ['/bin/sh', '-c', "echo -n '%s'; exit %d" % (retstring, retcode)] real_Popen.__init__(*(obj, args), **kw) @@ -115,7 +151,8 @@ def SAMBA_CROSS_ARGS(conf, msg=None): if conf.env.CROSS_EXECUTE: ret.extend(['--cross-execute', conf.env.CROSS_EXECUTE]) - elif conf.env.CROSS_ANSWERS: + + if conf.env.CROSS_ANSWERS: if msg is None: raise Utils.WafError("Cannot have NULL msg in cross-answers") ret.extend(['--cross-answers', os.path.join(Options.launch_dir, conf.env.CROSS_ANSWERS), msg]) diff --git a/script/autobuild.py b/script/autobuild.py index b2303f7..c697aad 100755 --- a/script/autobuild.py +++ b/script/autobuild.py @@ -23,6 +23,7 @@ cleanup_list = [] builddirs = { "ctdb" : "ctdb", "samba" : ".", + "samba-xc" : ".", "samba-ctdb" : ".", "samba-libs" : ".", "ldb" : "lib/ldb", @@ -36,7 +37,9 @@ builddirs = { "retry" : "." } -defaulttasks = [ "ctdb", "samba", "samba-ctdb", "samba-libs", "ldb", "tdb", "talloc", "replace", "tevent", "pidl" ] +defaulttasks = [ "ctdb", "samba", "samba-xc", "samba-ctdb", "samba-libs", "ldb", "tdb", "talloc", "replace", "tevent", "pidl" ] + +samba_configure_params = " --picky-developer ${PREFIX} --with-profiling-data" tasks = { "ctdb" : [ ("random-sleep", "../script/random-sleep.sh 60 600", "text/plain"), @@ -48,13 +51,22 @@ tasks = { ("clean", "make clean", "text/plain") ], # We have 'test' before 'install' because, 'test' should work without 'install' - "samba" : [ ("configure", "./configure.developer --picky-developer ${PREFIX} --with-selftest-prefix=./bin/ab --with-profiling-data", "text/plain"), + "samba" : [ ("configure", "./configure.developer --with-selftest-prefix=./bin/ab" + samba_configure_params, "text/plain"), ("make", "make -j", "text/plain"), ("test", "make test FAIL_IMMEDIATELY=1", "text/plain"), ("install", "make install", "text/plain"), ("check-clean-tree", "script/clean-source-tree.sh", "text/plain"), ("clean", "make clean", "text/plain") ], + # Test cross-compile infrastructure + "samba-xc" : [ ("configure-native", "./configure.developer --with-selftest-prefix=./bin/ab" + samba_configure_params, "text/plain"), + ("configure-cross-execute", "./configure.developer -b ./bin-xe --cross-compile --cross-execute=script/identity_cc.sh" \ + " --cross-answers=./bin-xe/cross-answers.txt --with-selftest-prefix=./bin-xe/ab" + samba_configure_params, "text/plain"), + ("configure-cross-answers", "./configure.developer -b ./bin-xa --cross-compile" \ + " --cross-answers=./bin-xe/cross-answers.txt --with-selftest-prefix=./bin-xa/ab" + samba_configure_params, "text/plain"), + ("compare-results", "script/compare_cc_results.py ./bin/c4che/default.cache.py ./bin-xe/c4che/default.cache.py ./bin-xa/c4che/default.cache.py", "text/plain")], + + "samba-ctdb" : [ ("random-sleep", "script/random-sleep.sh 60 600", "text/plain"), # make sure we have tdb around: @@ -519,7 +531,7 @@ A summary of the autobuild process is here: %s/autobuild.log ''' % (platform.node(), failed_task, errstr, log_base) - + if failed_task != 'rebase': text += ''' You can see logs of the failed task here: diff --git a/script/compare_cc_results.py b/script/compare_cc_results.py new file mode 100755 index 0000000..b8ed399 --- /dev/null +++ b/script/compare_cc_results.py @@ -0,0 +1,40 @@ +#!/usr/bin/env python + +# +# Compare the results of native and cross-compiled configure tests +# + +import sys +import difflib + +exceptions = ['BUILD_DIRECTORY', 'CROSS_COMPILE', 'CROSS_ANSWERS', + 'CROSS_EXECUTE', 'SELFTEST_PREFIX'] + +base_lines = list() +base_fname = '' + +found_diff = False + +for fname in sys.argv[1:]: + lines = list() + f = open(fname, 'r') + for line in f: + if len(line.split('=', 1)) == 2: + key = line.split('=', 1)[0].strip() + if key in exceptions: + continue + lines.append(line) + f.close() + if base_fname: + diff = list(difflib.unified_diff(base_lines,lines,base_fname,fname)) + if diff: + print 'configuration files %s and %s do not match' % (base_fname, fname) + for l in diff: + sys.stdout.write(l) + found_diff = True + else: + base_fname = fname + base_lines = lines + +if found_diff: + sys.exit(1) diff --git a/script/identity_cc.sh b/script/identity_cc.sh new file mode 100755 index 0000000..8b5118e --- /dev/null +++ b/script/identity_cc.sh @@ -0,0 +1,10 @@ +#!/bin/sh + +#An "identity cross-execute" script +#It can be used for testing the cross-build infrastructure +#as follows: +#./configure --cross-compile --cross-execute=./script/identity_cc.sh +#If the build is actually a native build, then the configuration +#result should be just like running ./configure without --cross-compile. + +eval "$@" -- Samba Shared Repository