[PATCH] [RFC] KVM test: Major control file cleanup
As pointed out before, the KVM reference control files
could use a little clean up. This patch implements severe
cleanup of the main control file by:
* Refactoring the code present there, moving it to the
kvm_utils.py library
* Treat the build test exactly the same way as other
tests, moving the config stuff that used to be in the
control file realm out to its own configuration file,
for the sake of consistency.
* Since a frequent complaint amongst users is that by
default the build test will build from release, and
that build might be unsuccessful depending on the system
being used (older kernel versions and other sorts of
stuff), change the default build method to 'noinstall'.
This way the control file becomes way shorter, fairly
well organized, and we have a consistent configuration
schema across the board, based on configuration files.
If people are OK with this change, final path will
change the control.parallel file as well.
Important: For the default control file, I removed
the part of the code that parses new strings for the
test configuration because I believe those
modifications belong to the test config files, they
could be just commented. For cases where keeping
all stuff inside the config file is not possible,
appropriate code will be added (in other example
control files).
Signed-off-by: Lucas Meneghel Rodrigues
---
client/tests/kvm/control | 203 +++--
client/tests/kvm/kvm_build.cfg.sample | 82 +
client/tests/kvm/kvm_utils.py | 112 ++
3 files changed, 211 insertions(+), 186 deletions(-)
create mode 100644 client/tests/kvm/kvm_build.cfg.sample
diff --git a/client/tests/kvm/control b/client/tests/kvm/control
index 431baf9..7acec14 100644
--- a/client/tests/kvm/control
+++ b/client/tests/kvm/control
@@ -20,194 +20,25 @@ KVM (both kernelspace and userspace) code.
For online docs, please refer to http://www.linux-kvm.org/page/KVM-Autotest
"""
+import sys, os, logging
+# Add the KVM tests dir to the python path
+kvm_test_dir = os.path.join(os.environ['AUTODIR'],'tests/kvm')
+sys.path.append(kvm_test_dir)
+import kvm_utils, kvm_config
-import sys, os
-#-
-# set English environment (command output might be localized, need to be safe)
-#-
-os.environ['LANG'] = 'en_US.UTF-8'
+## Prepare the environment for the KVM test.
+kvm_utils.prepare_test_run(basedir=kvm_test_dir,
+ rootdir='/tmp/kvm_autotest_root')
-#-
-# Enable modules import from current directory (tests/kvm)
-#-
-pwd = os.path.join(os.environ['AUTODIR'],'tests/kvm')
-sys.path.append(pwd)
+cfg_names = ['kvm_build.cfg', 'kvm_tests.cfg']
-#
-# create required symlinks
-#
-# When dispatching tests from autotest-server the links we need do not exist on
-# the host (the client). The following lines create those symlinks. Change
-# 'rootdir' here and/or mount appropriate directories in it.
-#
-# When dispatching tests on local host (client mode) one can either setup kvm
-# links, or same as server mode use rootdir and set all appropriate links and
-# mount-points there. For example, guest installation tests need to know where
-# to find the iso-files.
-#
-# We create the links only if not already exist, so if one already set up the
-# links for client/local run we do not touch the links.
-rootdir='/tmp/kvm_autotest_root'
-iso=os.path.join(rootdir, 'iso')
-images=os.path.join(rootdir, 'images')
-qemu=os.path.join(rootdir, 'qemu')
-qemu_img=os.path.join(rootdir, 'qemu-img')
+for cfg_name in cfg_names:
+cfg_path = os.path.join(kvm_test_dir, cfg_name)
+cfg = kvm_config.config(cfg_path)
+list = kvm_utils.get_test_list(cfg, cfg_name, kvm_test_dir)
+logging.info("Running test set defined on config file %s", cfg_name)
+kvm_utils.run_tests(list, job)
-
-def link_if_not_exist(ldir, target, link_name):
-t = target
-l = os.path.join(ldir, link_name)
-if not os.path.exists(l):
-os.system('ln -s %s %s' % (t, l))
-
-# Create links only if not already exist
-link_if_not_exist(pwd, '../../', 'autotest')
-link_if_not_exist(pwd, iso, 'isos')
-link_if_not_exist(pwd, images, 'images')
-link_if_not_exist(pwd, qemu, 'qemu')
-link_if_not_exist(pwd, qemu_img, 'qemu-img')
-
-#
-# Params that will be passed to the KVM install/build test
-#
-params = {
-"name": "build",
-"shortname": "build",
-"type": "build",
-"mode": "release",
-#"mode": "snapshot",
-#"mode": "localtar",
-#"mode": "localsrc",
-#"mode": "git",
-#"mode": "noinstall",
-#"mode": "koji
Re: [PATCH] [RFC] KVM test: Major control file cleanup
I think this patch is a good idea. Comments: - If we've decided to put all the configuration in config files (rather than the control file) let's split up the main one into kvm_test_library.cfg and kvm_tests.cfg.sample. The latter should look like this: include kvm_test_library.cfg include kvm_cdkeys.cfg variants: - fc8_quick: ... - other_test_set: ... only fc8_quick ... and the former should contain everything else. - I think we should allow the user to modify his/her configuration in the control file using parse_string(). This is actually useful, especially when running from the server. (At least I use it.) - If kvm_build.cfg or kvm_tests.cfg are missing, we should continue without them, or create them (blank), instead of failing due to an unhandled exception raised by the config parser. This should be true at least for kvm_build.cfg, which isn't always required for the tests to run. - We should consider removing hardcoded references to /tmp/kvm_autotest_root/. We use that path locally in TLV but that's not a good reason to put it upstream IMO. We should also stop creating symlinks to that path. Instead we can allow more flexibility using config files. For example, we can put this in kvm_tests.cfg.sample, between the 'include' and the 'variants': # Uncomment and modify the following lines if you wish to modify # the paths of the image files, ISO files, step files or qemu binaries. # If you keep these lines commented, all mentioned paths are expected # to be found in the KVM test dir (tests/kvm/). For example, all # image files are expected under tests/kvm/images/. # qemu_binary ?<= /tmp/kvm_autotest_root/ # qemu_img_binary ?<= /tmp/kvm_autotest_root/ # image_name.* ?<= /tmp/kvm_autotest_root/ # cdrom.* ?<= /tmp/kvm_autotest_root/ # steps ?<= /tmp/kvm_autotest_root/ The ?<= operator prepends strings to keys that match a regex, so 'image_name.* ?<= something/' will prepend 'something/' to image_name, image_name_vm2 and so on. In kvm_test_library.cfg (currently kvm_tests.cfg.sample) all 'image_name' parameters, for example, begin with 'images/', which means all image files should be under tests/kvm/images. If we prepend /tmp/kvm_autotest_root/ to that, all image files will be expected under /tmp/kvm_autotest_root/images/. Maybe we should leave these lines uncommented by default, in case anyone uses /tmp/kvm_autotest_root and relies on the control file creating the symlinks. - kvm_utils.run_tests() should return True if all tests passed and otherwise False. We need this in order to quit if the build test(s) failed. - I suggest we use kvm_utils.parse_host_specific_cfg() instead of kvm_utils.get_test_list(). I think 'parse_host_specific_cfg' better describes what the function does. I also think it should take 'kvm_address_pools.cfg' as a parameter so we don't have to hardcode it into the function. Please see additional comments below. - "Lucas Meneghel Rodrigues" wrote: > As pointed out before, the KVM reference control files > could use a little clean up. This patch implements severe > cleanup of the main control file by: > > * Refactoring the code present there, moving it to the > kvm_utils.py library > * Treat the build test exactly the same way as other > tests, moving the config stuff that used to be in the > control file realm out to its own configuration file, > for the sake of consistency. > * Since a frequent complaint amongst users is that by > default the build test will build from release, and > that build might be unsuccessful depending on the system > being used (older kernel versions and other sorts of > stuff), change the default build method to 'noinstall'. > > This way the control file becomes way shorter, fairly > well organized, and we have a consistent configuration > schema across the board, based on configuration files. > > If people are OK with this change, final path will > change the control.parallel file as well. > > Important: For the default control file, I removed > the part of the code that parses new strings for the > test configuration because I believe those > modifications belong to the test config files, they > could be just commented. For cases where keeping > all stuff inside the config file is not possible, > appropriate code will be added (in other example > control files). > > Signed-off-by: Lucas Meneghel Rodrigues > --- > client/tests/kvm/control | 203 > +++-- > client/tests/kvm/kvm_build.cfg.sample | 82 + > client/tests/kvm/kvm_utils.py | 112 ++ > 3 files changed, 211 insertions(+), 186 deletions(-) > create mode 100644 client/tests/kvm/kvm_build.cfg.sample > > diff --git a/client/tests/kvm/control b/client/tests/kvm/control > index 431baf9..7acec14 100644 > --- a/client/tests/kvm/control > +++ b/client/tests/kvm/control > @@ -20,194 +20,25 @@ KVM (both kernelspace and userspace) code. > For online docs, please refer
Re: [PATCH] [RFC] KVM test: Major control file cleanup
One thing that just occurred to me is, if we have a test config "library" as you said, it's perfectly possible to put the actual test set definitions and other config files inside the control file as strings. This way one can control configuration inside the control file, making it more convenient for usage, let's say, inside the autotest web interface. Since the control file would be reduced in size, the configuration snippets being in the control file would not be a huge problem, while keeping the original autotest philosophy of keeping stuff inside the control file... What do you think? On Wed, 2009-10-28 at 05:48 -0400, Michael Goldish wrote: > I think this patch is a good idea. Comments: > > - If we've decided to put all the configuration in config files > (rather than the control file) let's split up the main one into > kvm_test_library.cfg and kvm_tests.cfg.sample. The latter should > look like this: > > include kvm_test_library.cfg > include kvm_cdkeys.cfg > > variants: > - fc8_quick: > ... > - other_test_set: > ... > > only fc8_quick > > ... and the former should contain everything else. > > - I think we should allow the user to modify his/her configuration > in the control file using parse_string(). This is actually useful, > especially when running from the server. (At least I use it.) > > - If kvm_build.cfg or kvm_tests.cfg are missing, we should continue > without them, or create them (blank), instead of failing due to an > unhandled exception raised by the config parser. This should be > true at least for kvm_build.cfg, which isn't always required for the > tests to run. > > - We should consider removing hardcoded references to > /tmp/kvm_autotest_root/. We use that path locally in TLV but that's > not a good reason to put it upstream IMO. > We should also stop creating symlinks to that path. > Instead we can allow more flexibility using config files. > For example, we can put this in kvm_tests.cfg.sample, between the > 'include' and the 'variants': > > # Uncomment and modify the following lines if you wish to modify > # the paths of the image files, ISO files, step files or qemu binaries. > # If you keep these lines commented, all mentioned paths are expected > # to be found in the KVM test dir (tests/kvm/). For example, all > # image files are expected under tests/kvm/images/. > > # qemu_binary ?<= /tmp/kvm_autotest_root/ > # qemu_img_binary ?<= /tmp/kvm_autotest_root/ > # image_name.* ?<= /tmp/kvm_autotest_root/ > # cdrom.* ?<= /tmp/kvm_autotest_root/ > # steps ?<= /tmp/kvm_autotest_root/ > > The ?<= operator prepends strings to keys that match a regex, so > 'image_name.* ?<= something/' will prepend 'something/' to image_name, > image_name_vm2 and so on. > In kvm_test_library.cfg (currently kvm_tests.cfg.sample) all > 'image_name' parameters, for example, begin with 'images/', which means > all image files should be under tests/kvm/images. > If we prepend /tmp/kvm_autotest_root/ to that, all image files will be > expected under /tmp/kvm_autotest_root/images/. > > Maybe we should leave these lines uncommented by default, in case anyone > uses /tmp/kvm_autotest_root and relies on the control file creating the > symlinks. > > - kvm_utils.run_tests() should return True if all tests passed and > otherwise False. We need this in order to quit if the build test(s) > failed. > > - I suggest we use kvm_utils.parse_host_specific_cfg() instead of > kvm_utils.get_test_list(). I think 'parse_host_specific_cfg' better > describes what the function does. I also think it should take > 'kvm_address_pools.cfg' as a parameter so we don't have to hardcode it > into the function. > > > Please see additional comments below. > > > - "Lucas Meneghel Rodrigues" wrote: > > > As pointed out before, the KVM reference control files > > could use a little clean up. This patch implements severe > > cleanup of the main control file by: > > > > * Refactoring the code present there, moving it to the > > kvm_utils.py library > > * Treat the build test exactly the same way as other > > tests, moving the config stuff that used to be in the > > control file realm out to its own configuration file, > > for the sake of consistency. > > * Since a frequent complaint amongst users is that by > > default the build test will build from release, and > > that build might be unsuccessful depending on the system > > being used (older kernel versions and other sorts of > > stuff), change the default build method to 'noinstall'. > > > > This way the control file becomes way shorter, fairly > > well organized, and we have a consistent configuration > > schema across the board, based on configuration files. > > > > If people are OK with this change, final path will > > change the control.parallel file as well. > > > > Important: For the default control file, I removed > > the part of the code that parses new strings for the > > test configuration because I believe those > > modi
Re: [PATCH] [RFC] KVM test: Major control file cleanup
- "Lucas Meneghel Rodrigues" wrote: > One thing that just occurred to me is, if we have a test config > "library" as you said, it's perfectly possible to put the actual test > set definitions and other config files inside the control file as > strings. This way one can control configuration inside the control > file, > making it more convenient for usage, let's say, inside the autotest > web > interface. Since the control file would be reduced in size, the > configuration snippets being in the control file would not be a huge > problem, while keeping the original autotest philosophy of keeping > stuff inside the control file... What do you think? Sounds great, except it won't allow you to debug your configuration using kvm_config.py. So the question now is what's more important -- the ability to debug or ease of use when running from the server. > On Wed, 2009-10-28 at 05:48 -0400, Michael Goldish wrote: > > I think this patch is a good idea. Comments: > > > > - If we've decided to put all the configuration in config files > > (rather than the control file) let's split up the main one into > > kvm_test_library.cfg and kvm_tests.cfg.sample. The latter should > > look like this: > > > > include kvm_test_library.cfg > > include kvm_cdkeys.cfg > > > > variants: > > - fc8_quick: > > ... > > - other_test_set: > > ... > > > > only fc8_quick > > > > ... and the former should contain everything else. > > > > - I think we should allow the user to modify his/her configuration > > in the control file using parse_string(). This is actually useful, > > especially when running from the server. (At least I use it.) > > > > - If kvm_build.cfg or kvm_tests.cfg are missing, we should continue > > without them, or create them (blank), instead of failing due to an > > unhandled exception raised by the config parser. This should be > > true at least for kvm_build.cfg, which isn't always required for > the > > tests to run. > > > > - We should consider removing hardcoded references to > > /tmp/kvm_autotest_root/. We use that path locally in TLV but > that's > > not a good reason to put it upstream IMO. > > We should also stop creating symlinks to that path. > > Instead we can allow more flexibility using config files. > > For example, we can put this in kvm_tests.cfg.sample, between the > > 'include' and the 'variants': > > > > # Uncomment and modify the following lines if you wish to modify > > # the paths of the image files, ISO files, step files or qemu > binaries. > > # If you keep these lines commented, all mentioned paths are > expected > > # to be found in the KVM test dir (tests/kvm/). For example, all > > # image files are expected under tests/kvm/images/. > > > > # qemu_binary ?<= /tmp/kvm_autotest_root/ > > # qemu_img_binary ?<= /tmp/kvm_autotest_root/ > > # image_name.* ?<= /tmp/kvm_autotest_root/ > > # cdrom.* ?<= /tmp/kvm_autotest_root/ > > # steps ?<= /tmp/kvm_autotest_root/ > > > > The ?<= operator prepends strings to keys that match a regex, so > > 'image_name.* ?<= something/' will prepend 'something/' to > image_name, > > image_name_vm2 and so on. > > In kvm_test_library.cfg (currently kvm_tests.cfg.sample) all > > 'image_name' parameters, for example, begin with 'images/', which > means > > all image files should be under tests/kvm/images. > > If we prepend /tmp/kvm_autotest_root/ to that, all image files will > be > > expected under /tmp/kvm_autotest_root/images/. > > > > Maybe we should leave these lines uncommented by default, in case > anyone > > uses /tmp/kvm_autotest_root and relies on the control file creating > the > > symlinks. > > > > - kvm_utils.run_tests() should return True if all tests passed and > > otherwise False. We need this in order to quit if the build > test(s) > > failed. > > > > - I suggest we use kvm_utils.parse_host_specific_cfg() instead of > > kvm_utils.get_test_list(). I think 'parse_host_specific_cfg' > better > > describes what the function does. I also think it should take > > 'kvm_address_pools.cfg' as a parameter so we don't have to hardcode > it > > into the function. > > > > > > Please see additional comments below. > > > > > > - "Lucas Meneghel Rodrigues" wrote: > > > > > As pointed out before, the KVM reference control files > > > could use a little clean up. This patch implements severe > > > cleanup of the main control file by: > > > > > > * Refactoring the code present there, moving it to the > > > kvm_utils.py library > > > * Treat the build test exactly the same way as other > > > tests, moving the config stuff that used to be in the > > > control file realm out to its own configuration file, > > > for the sake of consistency. > > > * Since a frequent complaint amongst users is that by > > > default the build test will build from release, and > > > that build might be unsuccessful depending on the system > > > being used (older kernel versions and other sorts of > > > stuff), change the default buil
Re: [PATCH] [RFC] KVM test: Major control file cleanup
One more comment on this patch: If you plan to make 'noinstall' the default build mode, then let's not run the build test at all by default. If we run it with 'noinstall', it will just run the preprocessor and postprocessor, and those will kill any living VMs because no VMs are required for the build test. So a user won't be able to run a reboot test, keep the VM alive, modify the config file and then run a different test on the same living VM, because the preprocessor of the second 'build' test will kill the VM. To avoid this, we can just remove the 'noinstall' variant from kvm_build.cfg, and keep the 'only noinstall' statement. Because there's no 'noinstall' variant, the resulting test list will be empty, so nothing will run. We should probably add a comment reminding the user that it's possible to disable installation by stating 'only noinstall' or 'only nothing' (in case the user replaced 'noinstall' with something else). Alternatively, we can disable installation in the control file by commenting out some line. - Original Message - From: "Lucas Meneghel Rodrigues" To: "Michael Goldish" Cc: kvm@vger.kernel.org, u...@redhat.com, autot...@test.kernel.org Sent: Wednesday, October 28, 2009 1:14:43 PM (GMT+0200) Auto-Detected Subject: Re: [PATCH] [RFC] KVM test: Major control file cleanup One thing that just occurred to me is, if we have a test config "library" as you said, it's perfectly possible to put the actual test set definitions and other config files inside the control file as strings. This way one can control configuration inside the control file, making it more convenient for usage, let's say, inside the autotest web interface. Since the control file would be reduced in size, the configuration snippets being in the control file would not be a huge problem, while keeping the original autotest philosophy of keeping stuff inside the control file... What do you think? On Wed, 2009-10-28 at 05:48 -0400, Michael Goldish wrote: > I think this patch is a good idea. Comments: > > - If we've decided to put all the configuration in config files > (rather than the control file) let's split up the main one into > kvm_test_library.cfg and kvm_tests.cfg.sample. The latter should > look like this: > > include kvm_test_library.cfg > include kvm_cdkeys.cfg > > variants: > - fc8_quick: > ... > - other_test_set: > ... > > only fc8_quick > > ... and the former should contain everything else. > > - I think we should allow the user to modify his/her configuration > in the control file using parse_string(). This is actually useful, > especially when running from the server. (At least I use it.) > > - If kvm_build.cfg or kvm_tests.cfg are missing, we should continue > without them, or create them (blank), instead of failing due to an > unhandled exception raised by the config parser. This should be > true at least for kvm_build.cfg, which isn't always required for the > tests to run. > > - We should consider removing hardcoded references to > /tmp/kvm_autotest_root/. We use that path locally in TLV but that's > not a good reason to put it upstream IMO. > We should also stop creating symlinks to that path. > Instead we can allow more flexibility using config files. > For example, we can put this in kvm_tests.cfg.sample, between the > 'include' and the 'variants': > > # Uncomment and modify the following lines if you wish to modify > # the paths of the image files, ISO files, step files or qemu binaries. > # If you keep these lines commented, all mentioned paths are expected > # to be found in the KVM test dir (tests/kvm/). For example, all > # image files are expected under tests/kvm/images/. > > # qemu_binary ?<= /tmp/kvm_autotest_root/ > # qemu_img_binary ?<= /tmp/kvm_autotest_root/ > # image_name.* ?<= /tmp/kvm_autotest_root/ > # cdrom.* ?<= /tmp/kvm_autotest_root/ > # steps ?<= /tmp/kvm_autotest_root/ > > The ?<= operator prepends strings to keys that match a regex, so > 'image_name.* ?<= something/' will prepend 'something/' to image_name, > image_name_vm2 and so on. > In kvm_test_library.cfg (currently kvm_tests.cfg.sample) all > 'image_name' parameters, for example, begin with 'images/', which means > all image files should be under tests/kvm/images. > If we prepend /tmp/kvm_autotest_root/ to that, all image files will be > expected under /tmp/kvm_autotest_root/images/. > > Maybe we should leave these lines uncommented by default, in case anyone > uses /tmp/kvm_autotest_root and relies on the control file creating the > sy
Re: [PATCH] [RFC] KVM test: Major control file cleanup
* Michael Goldish [2009-10-28 10:43]: > > - "Lucas Meneghel Rodrigues" wrote: > > > One thing that just occurred to me is, if we have a test config > > "library" as you said, it's perfectly possible to put the actual test > > set definitions and other config files inside the control file as > > strings. This way one can control configuration inside the control > > file, > > making it more convenient for usage, let's say, inside the autotest > > web > > interface. Since the control file would be reduced in size, the > > configuration snippets being in the control file would not be a huge > > problem, while keeping the original autotest philosophy of keeping > > stuff inside the control file... What do you think? > > Sounds great, except it won't allow you to debug your configuration > using kvm_config.py. So the question now is what's more important -- > the ability to debug or ease of use when running from the server. +1 debug When creating new test scenarios I make *heavy* use of kvm_config.py; I'd be lost without being able to debug test configuration files. -- Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx ry...@us.ibm.com -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Autotest] [PATCH] [RFC] KVM test: Major control file cleanup
On Wed, Oct 28, 2009 at 1:43 PM, Michael Goldish wrote: > Sounds great, except it won't allow you to debug your configuration > using kvm_config.py. So the question now is what's more important -- > the ability to debug or ease of use when running from the server. Here we have 2 use cases: 1) Users of the web interface, that (hopefully) have canned test sets that work reliably. Ability to debug stuff is less important on this scenario. 2) People developing tests, and in this case ability to debug config is very important I see the following options: 1) Document as part of the "test development guide" that, in order to be able to debug stuff, that all the test sets are to be written to the config file and then, can be parsed using kvm_config. 2) If we write all dictionaries generated by that particular configuration on files inside the job results directory, we still have debug ability for all use cases (I am starting to like this idea very much, as I type). So I'd then implement option 2) and refactor the control file with the test sets defined inside strings in the control file, then you can see how it looks? How about that? -- Lucas -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Autotest] [PATCH] [RFC] KVM test: Major control file cleanup
On Wed, Oct 28, 2009 at 1:57 PM, Michael Goldish wrote: > One more comment on this patch: > > If you plan to make 'noinstall' the default build mode, then let's > not run the build test at all by default. If we run it with 'noinstall', > it will just run the preprocessor and postprocessor, and those will kill > any living VMs because no VMs are required for the build test. So a user > won't be able to run a reboot test, keep the VM alive, modify the config > file and then run a different test on the same living VM, because the > preprocessor of the second 'build' test will kill the VM. > > To avoid this, we can just remove the 'noinstall' variant from > kvm_build.cfg, and keep the 'only noinstall' statement. Because there's > no 'noinstall' variant, the resulting test list will be empty, so nothing > will run. We should probably add a comment reminding the user that it's > possible to disable installation by stating 'only noinstall' or 'only > nothing' (in case the user replaced 'noinstall' with something else). > > Alternatively, we can disable installation in the control file by commenting > out some line. Great idea, thanks :) - Lucas -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Autotest] [PATCH] [RFC] KVM test: Major control file cleanup
- "Lucas Meneghel Rodrigues" wrote:
> On Wed, Oct 28, 2009 at 1:43 PM, Michael Goldish
> wrote:
> > Sounds great, except it won't allow you to debug your configuration
> > using kvm_config.py. So the question now is what's more important
> --
> > the ability to debug or ease of use when running from the server.
>
> Here we have 2 use cases:
>
> 1) Users of the web interface, that (hopefully) have canned test sets
> that work reliably. Ability to debug stuff is less important on this
> scenario.
> 2) People developing tests, and in this case ability to debug config
> is very important
>
> I see the following options:
>
> 1) Document as part of the "test development guide" that, in order to
> be able to debug stuff, that all the test sets are to be written to
> the config file and then, can be parsed using kvm_config.
> 2) If we write all dictionaries generated by that particular
> configuration on files inside the job results directory, we still
> have
> debug ability for all use cases (I am starting to like this idea very
> much, as I type).
>
> So I'd then implement option 2) and refactor the control file with
> the
> test sets defined inside strings in the control file, then you can
> see
> how it looks? How about that?
Sounds fine.
- Where exactly will the test list appear?
- We should also allow printing of verbose debug output ("parsing variants
block, 9000 dicts in current context...") by passing something to the
constructor of the config object.
- We should make it clear to the user that he/she must rename the control
file (to control.lucas for example) or else it may be overwritten on the
next git-fetch or -pull.
I'm still not sure it's a great idea to make config debugging harder, so
if anyone other than Lucas who uses the KVM test is reading this, please
let us know if you ever use kvm_config.py and if you think the ability to
print the list of test dicts is important.
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Autotest] [PATCH] [RFC] KVM test: Major control file cleanup
Ryan, Michael: I absolutely agree that the ability to debug stuff is important, but the ability to make things straightforward to use from the web interface or cli is also important. A longer term goal is to have our test farm and make any developer able to schedule a job on the test farm easily and conveniently. Having the dictionaries generated on the job debug directory seems like a good compromise to me. Also we can come up with a smart way of parsing the config file generated by a given control file in a similar way we do today with kvm_config.py, it shouldn't be that hard to do it... (I hope I won't burn my tongue with this statement). Thanks for your input! On Wed, Oct 28, 2009 at 4:18 PM, Ryan Harper wrote: > * Michael Goldish [2009-10-28 10:43]: >> >> - "Lucas Meneghel Rodrigues" wrote: >> >> > One thing that just occurred to me is, if we have a test config >> > "library" as you said, it's perfectly possible to put the actual test >> > set definitions and other config files inside the control file as >> > strings. This way one can control configuration inside the control >> > file, >> > making it more convenient for usage, let's say, inside the autotest >> > web >> > interface. Since the control file would be reduced in size, the >> > configuration snippets being in the control file would not be a huge >> > problem, while keeping the original autotest philosophy of keeping >> > stuff inside the control file... What do you think? >> >> Sounds great, except it won't allow you to debug your configuration >> using kvm_config.py. So the question now is what's more important -- >> the ability to debug or ease of use when running from the server. > > +1 debug > > When creating new test scenarios I make *heavy* use of kvm_config.py; > I'd be lost without being able to debug test configuration files. > > -- > Ryan Harper > Software Engineer; Linux Technology Center > IBM Corp., Austin, Tx > ry...@us.ibm.com > ___ > Autotest mailing list > autot...@test.kernel.org > http://test.kernel.org/cgi-bin/mailman/listinfo/autotest > -- Lucas -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Autotest] [PATCH] [RFC] KVM test: Major control file cleanup
It's OK to log the debug output when running the framework, but Ryan says he makes heavy use of kvm_config.py, which means he may not find this comfortable enough. Ryan, what do you think? Another option is to allow editing any file from the web GUI in addition to the control file. I don't think it'll be hard to parse the control file and automatically extract config code from it in order to debug it, but it will almost certainly be very ugly. We might want to consider writing a little program that will emulate the client/bin/autotest.py program, running the control file and providing it with a fake job object whose run_test() method will just print the params dict instead of running the test. - Original Message - From: Lucas Meneghel Rodrigues To: Ryan Harper Cc: Michael Goldish , autot...@test.kernel.org, u...@redhat.com, kvm@vger.kernel.org Sent: Wed, 28 Oct 2009 15:47:54 -0400 (EDT) Subject: Re: [Autotest] [PATCH] [RFC] KVM test: Major control file cleanup Ryan, Michael: I absolutely agree that the ability to debug stuff is important, but the ability to make things straightforward to use from the web interface or cli is also important. A longer term goal is to have our test farm and make any developer able to schedule a job on the test farm easily and conveniently. Having the dictionaries generated on the job debug directory seems like a good compromise to me. Also we can come up with a smart way of parsing the config file generated by a given control file in a similar way we do today with kvm_config.py, it shouldn't be that hard to do it... (I hope I won't burn my tongue with this statement). Thanks for your input! On Wed, Oct 28, 2009 at 4:18 PM, Ryan Harper wrote: > * Michael Goldish [2009-10-28 10:43]: >> >> - "Lucas Meneghel Rodrigues" wrote: >> >> > One thing that just occurred to me is, if we have a test config >> > "library" as you said, it's perfectly possible to put the actual test >> > set definitions and other config files inside the control file as >> > strings. This way one can control configuration inside the control >> > file, >> > making it more convenient for usage, let's say, inside the autotest >> > web >> > interface. Since the control file would be reduced in size, the >> > configuration snippets being in the control file would not be a huge >> > problem, while keeping the original autotest philosophy of keeping >> > stuff inside the control file... What do you think? >> >> Sounds great, except it won't allow you to debug your configuration >> using kvm_config.py. So the question now is what's more important -- >> the ability to debug or ease of use when running from the server. > > +1 debug > > When creating new test scenarios I make *heavy* use of kvm_config.py; > I'd be lost without being able to debug test configuration files. > > -- > Ryan Harper > Software Engineer; Linux Technology Center > IBM Corp., Austin, Tx > ry...@us.ibm.com > ___ > Autotest mailing list > autot...@test.kernel.org > http://test.kernel.org/cgi-bin/mailman/listinfo/autotest > -- Lucas -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Autotest] [PATCH] [RFC] KVM test: Major control file cleanup
* Lucas Meneghel Rodrigues [2009-10-28 14:48]: > Ryan, Michael: > > I absolutely agree that the ability to debug stuff is important, but > the ability to make things straightforward to use from the web > interface or cli is also important. A longer term goal is to have our > test farm and make any developer able to schedule a job on the test > farm easily and conveniently. > > Having the dictionaries generated on the job debug directory seems > like a good compromise to me. Also we can come up with a smart way of > parsing the config file generated by a given control file in a similar > way we do today with kvm_config.py, it shouldn't be that hard to do > it... (I hope I won't burn my tongue with this statement). If I'm understanding things, we are talking about moving the large body of kvm_tests.cfg test definitions, guest definitions into a "library", and then moving the requested test config (bottom on kvm_tests.cfg) into the control file itself which means the autotest webui would be able to control which tests get run; I like this idea very well. My concern that I mentioned is that as you edit the "library" it can be difficult to ensure you described exactly which set of tests on which guests you want to run and kvm_config.py is invaluable in the process of getting it right. Why not have kvm_config.py , or some other wrapper generate a "kvm_tests.cfg" file dynamically from the "library" and the strings from the control file? That way we could still debug configuration via kvm_config.py? I much perfer this over queueing up jobs in the webiu, waiting for it to run, checking the results in the DEBUG dir, adjusting, repeat. -- Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx ry...@us.ibm.com -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Autotest] [PATCH] [RFC] KVM test: Major control file cleanup
On Fri, 2009-10-30 at 08:54 -0500, Ryan Harper wrote: > * Lucas Meneghel Rodrigues [2009-10-28 14:48]: > > Ryan, Michael: > > > > I absolutely agree that the ability to debug stuff is important, but > > the ability to make things straightforward to use from the web > > interface or cli is also important. A longer term goal is to have our > > test farm and make any developer able to schedule a job on the test > > farm easily and conveniently. > > > > Having the dictionaries generated on the job debug directory seems > > like a good compromise to me. Also we can come up with a smart way of > > parsing the config file generated by a given control file in a similar > > way we do today with kvm_config.py, it shouldn't be that hard to do > > it... (I hope I won't burn my tongue with this statement). > > If I'm understanding things, we are talking about moving the large body > of kvm_tests.cfg test definitions, guest definitions into a "library", > and then moving the requested test config (bottom on kvm_tests.cfg) into > the control file itself which means the autotest webui would be able to > control which tests get run; That is precisely my idea! > I like this idea very well. My concern > that I mentioned is that as you edit the "library" it can be difficult > to ensure you described exactly which set of tests on which guests you > want to run and kvm_config.py is invaluable in the process of getting it > right. Yes, very valid concern. Let's try to address it. > Why not have kvm_config.py , or some other wrapper generate a > "kvm_tests.cfg" file dynamically from the "library" and the strings from > the control file? That's one idea, the concern is that the code to do that might become ugly. But I will give it a try. > That way we could still debug configuration via > kvm_config.py? I much perfer this over queueing up jobs in the webiu, > waiting for it to run, checking the results in the DEBUG dir, adjusting, > repeat. Ok, I will go after this solution. Thanks for your comments Ryan, Lucas -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Autotest] [PATCH] [RFC] KVM test: Major control file cleanup
- "Ryan Harper" wrote: > * Lucas Meneghel Rodrigues [2009-10-28 14:48]: > > Ryan, Michael: > > > > I absolutely agree that the ability to debug stuff is important, > but > > the ability to make things straightforward to use from the web > > interface or cli is also important. A longer term goal is to have > our > > test farm and make any developer able to schedule a job on the test > > farm easily and conveniently. > > > > Having the dictionaries generated on the job debug directory seems > > like a good compromise to me. Also we can come up with a smart way > of > > parsing the config file generated by a given control file in a > similar > > way we do today with kvm_config.py, it shouldn't be that hard to do > > it... (I hope I won't burn my tongue with this statement). > > If I'm understanding things, we are talking about moving the large > body > of kvm_tests.cfg test definitions, guest definitions into a > "library", > and then moving the requested test config (bottom on kvm_tests.cfg) > into > the control file itself which means the autotest webui would be able > to > control which tests get run; I like this idea very well. My concern > that I mentioned is that as you edit the "library" it can be > difficult > to ensure you described exactly which set of tests on which guests > you > want to run and kvm_config.py is invaluable in the process of getting > it > right. > > Why not have kvm_config.py , or some other wrapper generate a > "kvm_tests.cfg" file dynamically from the "library" and the strings > from > the control file? That way we could still debug configuration via > kvm_config.py? I much perfer this over queueing up jobs in the > webiu, > waiting for it to run, checking the results in the DEBUG dir, > adjusting, > repeat. I'm not sure I understand your idea: you want some program to read the control file and generate a new file (kvm_tests.cfg or something) from the control file and the library file, so that this file can be debugged with kvm_config.py? IMO this solution is "dirty" because the control file is python code, not our own format, so it's not nice to automatically extract stuff from it. It would be nice to do something that eases debugging, but if you ask me, I'd rather have something as clean as possible. Here's another idea, which I suggested but haven't received any feedback on: let's write a little proggie that runs the control file just like client/bin/autotest does. The proggie will supply the control file with a fake job object that has nothing but a run_test() method, but instead of running a test, that method will simply nicely print out the test params, like kvm_config.py does. So the user will be able to do something like './dry_run.py control.mine' which will list all the tests to be executed. We might also want to implement job.parallel() in addition to job.run_test() but that should be very easy (it doesn't really have to be parallel at all). Does that make any sense? -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Autotest] [PATCH] [RFC] KVM test: Major control file cleanup
* Michael Goldish [2009-10-31 16:02]: > > > > If I'm understanding things, we are talking about moving the large > > body > > of kvm_tests.cfg test definitions, guest definitions into a > > "library", > > and then moving the requested test config (bottom on kvm_tests.cfg) > > into > > the control file itself which means the autotest webui would be able > > to > > control which tests get run; I like this idea very well. My concern > > that I mentioned is that as you edit the "library" it can be > > difficult > > to ensure you described exactly which set of tests on which guests > > you > > want to run and kvm_config.py is invaluable in the process of getting > > it > > right. > > > > Why not have kvm_config.py , or some other wrapper generate a > > "kvm_tests.cfg" file dynamically from the "library" and the strings > > from > > the control file? That way we could still debug configuration via > > kvm_config.py? I much perfer this over queueing up jobs in the > > webiu, > > waiting for it to run, checking the results in the DEBUG dir, > > adjusting, > > repeat. > > I'm not sure I understand your idea: you want some program to read the > control file and generate a new file (kvm_tests.cfg or something) from > the control file and the library file, so that this file can be debugged > with kvm_config.py? I wanted something that would take the test description string from the control file, run that through the library such that it can print out the tests that will run -- the equivalent of running kvm_config.py on a kvm_tests.cfg file... > > IMO this solution is "dirty" because the control file is python code, not > our own format, so it's not nice to automatically extract stuff from it. > It would be nice to do something that eases debugging, but if you ask me, > I'd rather have something as clean as possible. > > Here's another idea, which I suggested but haven't received any feedback > on: let's write a little proggie that runs the control file just like > client/bin/autotest does. The proggie will supply the control file with > a fake job object that has nothing but a run_test() method, but instead > of running a test, that method will simply nicely print out the test > params, like kvm_config.py does. So the user will be able to do something > like './dry_run.py control.mine' which will list all the tests to be > executed. We might also want to implement job.parallel() in addition to > job.run_test() but that should be very easy (it doesn't really have to be > parallel at all). > Does that make any sense? It does, though I don't see how it's different then what I suggested; I don't really care how we do it; I still need something to debug test config files outside of cycling through job output via the webui; -- Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx ry...@us.ibm.com -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Autotest] [PATCH] [RFC] KVM test: Major control file cleanup
On Wed, Oct 28, 2009 at 02:04:59PM -0400, Michael Goldish wrote:
>
> - "Lucas Meneghel Rodrigues" wrote:
>
> > On Wed, Oct 28, 2009 at 1:43 PM, Michael Goldish
> > wrote:
> > > Sounds great, except it won't allow you to debug your configuration
> > > using kvm_config.py. So the question now is what's more important
> > --
> > > the ability to debug or ease of use when running from the server.
> >
> > Here we have 2 use cases:
> >
> > 1) Users of the web interface, that (hopefully) have canned test sets
> > that work reliably. Ability to debug stuff is less important on this
> > scenario.
> > 2) People developing tests, and in this case ability to debug config
> > is very important
> >
> > I see the following options:
> >
> > 1) Document as part of the "test development guide" that, in order to
> > be able to debug stuff, that all the test sets are to be written to
> > the config file and then, can be parsed using kvm_config.
> > 2) If we write all dictionaries generated by that particular
> > configuration on files inside the job results directory, we still
> > have
> > debug ability for all use cases (I am starting to like this idea very
> > much, as I type).
> >
> > So I'd then implement option 2) and refactor the control file with
> > the
> > test sets defined inside strings in the control file, then you can
> > see
> > how it looks? How about that?
>
> Sounds fine.
> - Where exactly will the test list appear?
> - We should also allow printing of verbose debug output ("parsing variants
> block, 9000 dicts in current context...") by passing something to the
> constructor of the config object.
> - We should make it clear to the user that he/she must rename the control
> file (to control.lucas for example) or else it may be overwritten on the
> next git-fetch or -pull.
>
> I'm still not sure it's a great idea to make config debugging harder, so
> if anyone other than Lucas who uses the KVM test is reading this, please
> let us know if you ever use kvm_config.py and if you think the ability to
> print the list of test dicts is important.
Hi Michael,
I had used kvm_config.py for printing lists of selected test dicts
often. And I think it's necessary to keep this feature. IMHO, option 2) Lucas
proposed is a good idea. What do you think? Hope I haven't missed
something. :)
> ___
> Autotest mailing list
> autot...@test.kernel.org
> http://test.kernel.org/cgi-bin/mailman/listinfo/autotest
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Autotest] [PATCH] [RFC] KVM test: Major control file cleanup
On Wed, Nov 4, 2009 at 5:25 AM, Yolkfull Chow wrote: > Hi Michael, > > I had used kvm_config.py for printing lists of selected test dicts > often. And I think it's necessary to keep this feature. IMHO, option 2) Lucas > proposed is a good idea. What do you think? Hope I haven't missed > something. :) 2) is a compromise solution, but we find it's cumbersome, so we are going to do something that can print all test dicts from a control file, in addition to 2). Thanks for your input! -- Lucas -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
