OK, I completed my fedfind/wikitcms conversion patch for openqa_fedora_tools. I also fixed up some of the test case submission data bits - they broke with relval report-auto (instead of jskladan's version of non-interactive result submission) because they didn't quite grok the 'test instance' concept report-auto follows.
Note this needs the very latest git wikitcms. I don't think latest relval or fedfind is strictly required, but may as well bump them anyway. I've installed the current wikitcms on fedora-qa-01. I will cut new releases tomorrow. I have tested this on fedora-qa-01, it ran and submitted a full set of results for 2015-02-07. I believe it should work for Branched nightlies (if we wind up nominating any) and TC/RC builds also, which is a significant win. I did test this a bit: https://fedora-qa-01.lab.bos.redhat.com/tests/15 is a one-job run against F21 Final RC5, you can see it successfully got the ISO and started the test but the test fails because the 'Timbuktu' screen isn't there. I revised the openqa_trigger.py CLI to use argparse; it has the same basic modes as before but the syntax is a bit different. 'openqa_trigger.py current' runs against the 'current' compose event if it has not already done so (what just plain 'openqa_trigger.py' did before). 'openqa_trigger.py event' takes the same 'version id' arguments as 'relval report-auto' and runs tasks (but does not automatically report results) against the specified compose (what you used to get by passing args, approximately, but much more robust and capable). The basic approach is that openqa_trigger gets a ValidationEvent from python-wikitcms - either the Wiki.current_event property for 'current', or the event specified, obtained via the newly-added Wiki.get_validation_event(), for 'event'. For 'event' it then just goes ahead and runs the jobs and prints the IDs. For 'current' it checks the last run compose version for each arch and runs if needed, as before. The ValidationEvent's 'sortname' property is the value written out to PERSISTENT to track the 'last run' - this property is intended to always sort compose events 'correctly', so we should always run when appropriate even when going from Rawhide to Branched, Branched to a TC, TC to RC, RC to (next milestone) TC. On both paths it gets a fedfind.Release object via the ValidationEvent - ValidationEvents have a ff_release property which is the fedfind.Release object that matches that event. It then queries fedfind for image locations using a query that tries to get just *one* generic-ish network install image for each arch. It passes the location to download_image(), which is just download_rawhide_iso() renamed and does the same job, only it can be simpler now. >From there it works pretty much as before, except we use the ValidationEvent's 'version' property as the BUILD setting for OpenQA, and report_job_results get_relval_commands() is tweaked slightly to parse this properly to produce a correct report-auto command. Probably the most likely bits to break here are the sortname thing (see wikitcms helpers.py fedora_release_sort(), it's pretty stupid, I should re-write it) and the image query, which might wind up getting more than one image depending on how exactly the F22 Alpha composes look. I'll keep a close eye on that. We can always take the list from fedfind and further filter it so we have just one image per arch. Image objects have a .arch attribute so this will be easy to do if necessary. I *could* give the fedfind query code a 'I'm feeling lucky'- ish mode to only return one image per (whatever), but not sure if that would be too specialized, I'll think about it. Please do ask any questions :) -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net
From 55d5d1b4ce4bfd7ae7c239c8aca03bceee2f1f06 Mon Sep 17 00:00:00 2001 From: Adam Williamson <[email protected]> Date: Thu, 12 Feb 2015 19:32:27 -0800 Subject: [PATCH 1/3] use python-wikitcms and fedfind to handle events and images This removes a bunch of template-y bits that fedfind does better, and should make things work for all validation events. The CLI for openqa_trigger is completely redone to use argparse and check values and stuff. The syntax for the event sub-command is purposely similar to relval report-auto. Note this requires wikitcms changes currently in git master. --- tools/openqa_trigger/openqa_trigger.py | 159 +++++++++++++++++++---------- tools/openqa_trigger/report_job_results.py | 7 +- 2 files changed, 108 insertions(+), 58 deletions(-) diff --git a/tools/openqa_trigger/openqa_trigger.py b/tools/openqa_trigger/openqa_trigger.py index 9406d0c..7424c8b 100755 --- a/tools/openqa_trigger/openqa_trigger.py +++ b/tools/openqa_trigger/openqa_trigger.py @@ -7,15 +7,15 @@ import urlgrabber import os.path import sys import subprocess +import argparse +import wikitcms.wiki +import fedfind.release from report_job_results import report_results PERSISTENT = "/var/tmp/openqa_watcher.json" -CURRENT_TEST = "https://fedoraproject.org/wiki/Test_Results:Current_Installation_Test"; -ISO_URL = "https://kojipkgs.fedoraproject.org/mash/rawhide-%s/rawhide/%s/os/images/boot.iso"; -ISO_REGEX = re.compile(r'https://kojipkgs\.fedoraproject\.org/mash/(?P<name>rawhide-(?P<build>\d+))/rawhide/(?P<arch>x86_64|i386)/os/images/boot\.iso') ISO_PATH = "/var/lib/openqa/factory/iso/" -RUN_COMMAND = "/var/lib/openqa/script/client isos post ISO=%s DISTRI=fedora VERSION=rawhide FLAVOR=server ARCH=%s BUILD=%s_%s" +RUN_COMMAND = "/var/lib/openqa/script/client isos post ISO=%s DISTRI=fedora VERSION=rawhide FLAVOR=server ARCH=%s BUILD=%s" VERSIONS = ['i386', 'x86_64'] # read last tested version from file @@ -33,27 +33,27 @@ def read_last(): result[version] = json_parsed.get(version, None) return result, json_parsed -# read current version from Current Installation Test page -def read_currents(): - page = urllib2.urlopen(CURRENT_TEST).read() - f_regex = re.compile(r'<title>.*Fedora (?P<version>\d+).*</title>') - m = f_regex.search(page) - for match in ISO_REGEX.finditer(page): - yield m.group('version'), match.group("build"), match.group(0), match.group("name"), match.group("arch") - -# download rawhide iso from koji -def download_rawhide_iso(link, name, arch): - isoname = "%s_%s.iso" % (name, arch) +def download_image(image, event): + """Download a given image with a name that should be unique for + this event and arch (until we start testing different images + for the same event and arch). Returns the filename of the image + (not the path). + """ + isoname = "{0}_{1}.iso".format(event.version.replace(' ', '_'), image.arch) filename = os.path.join(ISO_PATH, isoname) if not os.path.isfile(filename): - link = "http://"; + link[len("https://";):] - urlgrabber.urlgrab(link, filename) + # Icky hack around a urlgrabber bug: + # https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=715416 + urlgrabber.urlgrab(image.url.replace('https', 'http'), filename) return isoname -# run OpenQA 'isos' job on selected isoname, with given arch and build -# returns list of job IDs -def run_openqa_jobs(isoname, arch, fedora_version, build): - command = RUN_COMMAND % (isoname, arch, fedora_version, build) +def run_openqa_jobs(isoname, arch, event_version): + """# run OpenQA 'isos' job on selected isoname, with given arch + and event version as the 'build'. **NOTE**: the 'build' is parsed + back into the 'relval report-auto' arguments later. Using + event.version makes sure this works. Returns list of job IDs + """ + command = RUN_COMMAND % (isoname, arch, event_version.replace(' ', '_')) # starts OpenQA jobs output = subprocess.check_output(command.split()) @@ -70,28 +70,20 @@ def run_openqa_jobs(isoname, arch, fedora_version, build): else: return [] -# run OpenQA on rawhide if there is newer version since last run -def run_if_newer(): +# run OpenQA on current compose if it is newer version since last run +def run_current(args, wiki): last_versions, json_parsed = read_last() - jobs = [] - - # for every architecture - for f_version, current_version, link, name, arch in read_currents(): - # don't run when there is newer version + currev = wiki.current_event + print("Current event: {0}".format(currev.version)) + runarches = [] + for arch in VERSIONS: last_version = last_versions.get(arch, None) - print f_version, current_version, link, name, arch, - if last_version is not None and (last_version >= current_version): - print " - Skipped" - continue - - print "" - - json_parsed[arch] = current_version - - isoname = download_rawhide_iso(link, name, arch) - job_ids = run_openqa_jobs(isoname, arch, f_version, current_version) - - jobs.extend(job_ids) + if last_version and last_version >= currev.sortname: + print("Skipped: {0}".format(arch)) + else: + runarches.append(arch) + json_parsed[arch] = currev.sortname + jobs = jobs_from_event(currev, runarches) # write info about latest versions f = open(PERSISTENT, "w") @@ -101,18 +93,81 @@ def run_if_newer(): # wait for jobs to finish and display results print jobs report_results(jobs) + sys.exit() +# run OpenQA on a specified event (don't update 'current' records +# or report results) +def run_event(args, wiki): + try: + event = wiki.get_validation_event( + args.release, args.build, args.version) + except ValueError as err: + sys.exit(err[0]) + if args.arch: + print(jobs_from_event(event, [args.arch])) + else: + print(jobs_from_event(event)) + sys.exit() + +def jobs_from_event(event, arches=VERSIONS): + """Given a python-wikitcms ValidationEvent, find the ISOs we want + and run jobs on them. arches is an iterable of arches to run on, + if not specified, we'll use our constant.""" + # Find boot.iso images for our arches; third query is a bit of a + # bodge till I know what 22 TCs/RCs will actually look like, + # ideally we want a query that will reliably return one image per + # arch without us having to filter further, but we can always just + # take the first image for each arch if necessary + jobs = [] + queries = ( + fedfind.release.Query('imagetype', ('boot',)), + fedfind.release.Query('arch', arches), + fedfind.release.Query('payload', ('server', 'generic'))) + + for image in event.ff_release.find_images(queries): + print("{0} {1}".format(image.url, image.desc)) + isoname = download_image(image, event) + job_ids = run_openqa_jobs(isoname, image.arch, event.version) + jobs.extend(job_ids) + return jobs if __name__ == "__main__": - if len(sys.argv) == 1: - run_if_newer() - elif len(sys.argv) == 3: - version = sys.argv[1] - arch = sys.argv[2] - name = "rawhide-%s" % version - link = ISO_URL % (sys.argv[1], sys.argv[2]) - isoname = download_rawhide_iso(link, name, arch) - job_ids = run_openqa_jobs(isoname, arch, "", version) - print job_ids + test_help = "Operate on the staging wiki (for testing)" + parser = argparse.ArgumentParser(description=( + "Run OpenQA tests for a release validation test event.")) + subparsers = parser.add_subparsers() + + parser_current = subparsers.add_parser( + 'current', description="Run for the current event, if needed.") + parser_current.add_argument( + '-t', '--test', help=test_help, required=False, action='store_true') + parser_current.set_defaults(func=run_current) + + parser_event = subparsers.add_parser( + 'event', description="Run for a specific event.") + parser_event.add_argument( + '-r', '--release', type=int, required=True, choices=range(12, 100), + metavar="12-99", help="Release number of a specific compose to run " + "against.") + parser_event.add_argument( + '-b', '--build', help="The build to run for; either a milestone " + "(for a TC/RC), or 'Rawhide' or 'Branched' (for a nightly build)", + required=True, + choices=['Alpha', 'Beta', 'Final', 'Branched', 'Rawhide']) + parser_event.add_argument( + '-v', '--version', help="The version to run for; either the compose " + "(for a TC/RC), or the date (for a nightly build)", required=True, + metavar="{T,R}C1-19 or YYYYMMDD") + parser_event.add_argument( + '-a', '--arch', help="The arch to run for", required=False, + choices=('x86_64', 'i386')) + parser_event.add_argument( + '-t', '--test', help=test_help, required=False, action='store_true') + parser_event.set_defaults(func=run_event) + + args = parser.parse_args() + if args.test: + wiki = wikitcms.wiki.Wiki(('https', 'stg.fedoraproject.org'), '/w/') else: - print "%s [rawhide_version arch]" % sys.arv[0] + wiki = wikitcms.wiki.Wiki(('https', 'fedoraproject.org'), '/w/') + args.func(args, wiki) diff --git a/tools/openqa_trigger/report_job_results.py b/tools/openqa_trigger/report_job_results.py index 31436a6..a617562 100644 --- a/tools/openqa_trigger/report_job_results.py +++ b/tools/openqa_trigger/report_job_results.py @@ -46,12 +46,7 @@ def get_relval_commands(passed_testcases): for key in passed_testcases: cmd_ = relval_template version, _, build, arch = key - - if version == 'rawhide': - cmd_ += ' --release "%s" --build Rawhide --version "%s"' % tuple(build.split('_')) #"22_20150110" - elif version == 'branched': - #cmd_ += ' --release "%s" --milestone "%s" --compose "%s"' % tuple(build.split('_')) #"22_Alpha_TC1" - continue + cmd_ += ' --release "%s" --build %s --version "%s"' % tuple(build.split('_')) for tc_name in passed_testcases[key]: testcase = conf_test_suites.TESTCASES[tc_name] -- 2.3.0
From 2e530cb275e7021c54fdc9a32d54fa99c0a645f6 Mon Sep 17 00:00:00 2001 From: Adam Williamson <[email protected]> Date: Fri, 13 Feb 2015 03:55:13 -0800 Subject: [PATCH 2/3] add a comment --- tools/openqa_trigger/openqa_trigger.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/tools/openqa_trigger/openqa_trigger.py b/tools/openqa_trigger/openqa_trigger.py index 7424c8b..a6fdbe0 100755 --- a/tools/openqa_trigger/openqa_trigger.py +++ b/tools/openqa_trigger/openqa_trigger.py @@ -39,6 +39,10 @@ def download_image(image, event): for the same event and arch). Returns the filename of the image (not the path). """ + # We don't *really* need to pass in event and use event.version + # here, but as we're quite strongly tied to a validation event + # and we use it as an identifier elsewhere, using it here too + # keeps things consistent. isoname = "{0}_{1}.iso".format(event.version.replace(' ', '_'), image.arch) filename = os.path.join(ISO_PATH, isoname) if not os.path.isfile(filename): -- 2.3.0
From 61f5e61d097fa585c3bb815e22aa56e0348a83bb Mon Sep 17 00:00:00 2001 From: Adam Williamson <[email protected]> Date: Fri, 13 Feb 2015 03:55:20 -0800 Subject: [PATCH 3/3] tweak the test cases a bit: fixed the wikitcms bug I fixed the column name parsing issue commented here. Also, for the Default boot and install tests, QA:Testcase_Boot_ default_install is the 'test case', 'Server offline' or 'Server netinst' is the 'test name' (--testcase and --testname arguments to relval report-auto). I was nice in relval and made it try a 'test name' match for --testcase if there's no 'test case' match, so since these names are unique within the 'Default boot and install' section we can get away with this for now. We may need to be able to pass both --testcase and --testname with different values at some point, though, at which point this will need re-thinking. --- tools/openqa_trigger/conf_test_suites.py | 38 ++++++++++++++++---------------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/tools/openqa_trigger/conf_test_suites.py b/tools/openqa_trigger/conf_test_suites.py index c136410..fd19714 100644 --- a/tools/openqa_trigger/conf_test_suites.py +++ b/tools/openqa_trigger/conf_test_suites.py @@ -1,10 +1,10 @@ TESTCASES = { - "QA:Testcase_Boot_default_install Server offline": { + "Server offline": { "section": 'Default boot and install', "env": "$RUNARCH$", "type": "Installation", }, - "QA:Testcase_Boot_default_install Server netinst": { + "Server netinst": { "section": 'Default boot and install', "env": "$RUNARCH$", "type": "Installation", @@ -16,7 +16,7 @@ TESTCASES = { }, "QA:Testcase_partitioning_guided_empty": { "section": "Guided storage configuration", - "env": "x86", # Probably a bug in relval - column name is "x86 BIOS", but there is a comment there just behind 'x86' which probably makes it strip the rest + "env": "x86 BIOS", "type": "Installation", }, "QA:Testcase_Anaconda_User_Interface_Graphical": { @@ -36,7 +36,7 @@ TESTCASES = { }, "QA:Testcase_partitioning_guided_delete_all": { "section": "Guided storage configuration", - "env": "x86", # Probably a bug in relval - column name is "x86 BIOS", but there is a comment there just behind 'x86' which probably makes it strip the rest + "env": "x86 BIOS", "type": "Installation", }, "QA:Testcase_install_to_SATA": { @@ -46,7 +46,7 @@ TESTCASES = { }, "QA:Testcase_partitioning_guided_multi_select": { "section": "Guided storage configuration", - "env": "x86", # Probably a bug in relval - column name is "x86 BIOS", but there is a comment there just behind 'x86' which probably makes it strip the rest + "env": "x86 BIOS", "type": "Installation", }, "QA:Testcase_install_to_SCSI": { @@ -71,17 +71,17 @@ TESTCASES = { }, "QA:Testcase_install_repository_Mirrorlist_graphical": { "section": "Installation repositories", - "env": "result", + "env": "Result", "type": "Installation", }, "QA:Testcase_install_repository_HTTP/FTP_graphical": { "section": "Installation repositories", - "env": "result", + "env": "Result", "type": "Installation", }, "QA:Testcase_install_repository_HTTP/FTP_variation": { "section": "Installation repositories", - "env": "result", + "env": "Result", "type": "Installation", }, "QA:Testcase_Package_Sets_Minimal_Package_Install": { @@ -91,7 +91,7 @@ TESTCASES = { }, "QA:Testcase_partitioning_guided_encrypted": { "section": "Guided storage configuration", - "env": "x86", # Probably a bug in relval - column name is "x86 BIOS", but there is a comment there just behind 'x86' which probably makes it strip the rest + "env": "x86 BIOS", "type": "Installation", }, "QA:Testcase_partitioning_guided_delete_partial": { @@ -109,7 +109,7 @@ TESTCASES = { TESTSUITES = { "server_simple":[ - "QA:Testcase_Boot_default_install Server netinst", + "Server netinst", "QA:Testcase_install_to_VirtIO", "QA:Testcase_partitioning_guided_empty", "QA:Testcase_Anaconda_User_Interface_Graphical", @@ -117,7 +117,7 @@ TESTSUITES = { "QA:Testcase_Package_Sets_Minimal_Package_Install", ], "server_delete_pata":[ - "QA:Testcase_Boot_default_install Server netinst", + "Server netinst", "QA:Testcase_install_to_PATA", "QA:Testcase_partitioning_guided_delete_all", "QA:Testcase_Anaconda_User_Interface_Graphical", @@ -125,7 +125,7 @@ TESTSUITES = { "QA:Testcase_Package_Sets_Minimal_Package_Install", ], "server_sata_multi":[ - "QA:Testcase_Boot_default_install Server netinst", + "Server netinst", "QA:Testcase_install_to_SATA", "QA:Testcase_partitioning_guided_multi_select", "QA:Testcase_Anaconda_User_Interface_Graphical", @@ -133,7 +133,7 @@ TESTSUITES = { "QA:Testcase_Package_Sets_Minimal_Package_Install", ], "server_scsi_updates_img":[ - "QA:Testcase_Boot_default_install Server netinst", + "Server netinst", "QA:Testcase_install_to_SCSI", "QA:Testcase_partitioning_guided_empty", "QA:Testcase_Anaconda_updates.img_via_URL", @@ -148,7 +148,7 @@ TESTSUITES = { "QA:Testcase_Kickstart_Http_Server_Ks_Cfg", ], "server_mirrorlist_graphical":[ - "QA:Testcase_Boot_default_install Server netinst", + "Server netinst", "QA:Testcase_install_to_VirtIO", "QA:Testcase_partitioning_guided_empty", "QA:Testcase_Anaconda_User_Interface_Graphical", @@ -157,7 +157,7 @@ TESTSUITES = { "QA:Testcase_Package_Sets_Minimal_Package_Install", ], "server_repository_http_graphical":[ - "QA:Testcase_Boot_default_install Server netinst", + "Server netinst", "QA:Testcase_install_to_VirtIO", "QA:Testcase_partitioning_guided_empty", "QA:Testcase_Anaconda_User_Interface_Graphical", @@ -166,7 +166,7 @@ TESTSUITES = { "QA:Testcase_Package_Sets_Minimal_Package_Install", ], "server_repository_http_variation":[ - "QA:Testcase_Boot_default_install Server netinst", + "Server netinst", "QA:Testcase_install_to_VirtIO", "QA:Testcase_partitioning_guided_empty", "QA:Testcase_Anaconda_User_Interface_Graphical", @@ -175,7 +175,7 @@ TESTSUITES = { "QA:Testcase_Package_Sets_Minimal_Package_Install", ], "server_mirrorlist_http_variation":[ - "QA:Testcase_Boot_default_install Server netinst", + "Server netinst", "QA:Testcase_install_to_VirtIO", "QA:Testcase_partitioning_guided_empty", "QA:Testcase_Anaconda_User_Interface_Graphical", @@ -184,7 +184,7 @@ TESTSUITES = { "QA:Testcase_Package_Sets_Minimal_Package_Install", ], "server_simple_encrypted": [ - "QA:Testcase_Boot_default_install Server netinst", + "Server netinst", "QA:Testcase_install_to_VirtIO", "QA:Testcase_partitioning_guided_empty", "QA:Testcase_Anaconda_User_Interface_Graphical", @@ -193,7 +193,7 @@ TESTSUITES = { "QA:Testcase_partitioning_guided_encrypted", ], "server_delete_partial": [ - "QA:Testcase_Boot_default_install Server netinst", + "Server netinst", "QA:Testcase_install_to_VirtIO", "QA:Testcase_partitioning_guided_delete_partial" "QA:Testcase_Anaconda_User_Interface_Graphical", -- 2.3.0
_______________________________________________ qa-devel mailing list [email protected] https://admin.fedoraproject.org/mailman/listinfo/qa-devel
