Re: [Autotest] [PATCH] [RFC] KVM test: Major control file cleanup
On Wed, Oct 28, 2009 at 02:04:59PM -0400, Michael Goldish wrote: - Lucas Meneghel Rodrigues l...@redhat.com wrote: On Wed, Oct 28, 2009 at 1:43 PM, Michael Goldish mgold...@redhat.com wrote: Sounds great, except it won't allow you to debug your configuration using kvm_config.py. So the question now is what's more important -- the ability to debug or ease of use when running from the server. Here we have 2 use cases: 1) Users of the web interface, that (hopefully) have canned test sets that work reliably. Ability to debug stuff is less important on this scenario. 2) People developing tests, and in this case ability to debug config is very important I see the following options: 1) Document as part of the test development guide that, in order to be able to debug stuff, that all the test sets are to be written to the config file and then, can be parsed using kvm_config. 2) If we write all dictionaries generated by that particular configuration on files inside the job results directory, we still have debug ability for all use cases (I am starting to like this idea very much, as I type). So I'd then implement option 2) and refactor the control file with the test sets defined inside strings in the control file, then you can see how it looks? How about that? Sounds fine. - Where exactly will the test list appear? - We should also allow printing of verbose debug output (parsing variants block, 9000 dicts in current context...) by passing something to the constructor of the config object. - We should make it clear to the user that he/she must rename the control file (to control.lucas for example) or else it may be overwritten on the next git-fetch or -pull. I'm still not sure it's a great idea to make config debugging harder, so if anyone other than Lucas who uses the KVM test is reading this, please let us know if you ever use kvm_config.py and if you think the ability to print the list of test dicts is important. Hi Michael, I had used kvm_config.py for printing lists of selected test dicts often. And I think it's necessary to keep this feature. IMHO, option 2) Lucas proposed is a good idea. What do you think? Hope I haven't missed something. :) ___ Autotest mailing list autot...@test.kernel.org http://test.kernel.org/cgi-bin/mailman/listinfo/autotest -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Autotest] [PATCH] [RFC] KVM test: Major control file cleanup
* Michael Goldish mgold...@redhat.com [2009-10-31 16:02]: If I'm understanding things, we are talking about moving the large body of kvm_tests.cfg test definitions, guest definitions into a library, and then moving the requested test config (bottom on kvm_tests.cfg) into the control file itself which means the autotest webui would be able to control which tests get run; I like this idea very well. My concern that I mentioned is that as you edit the library it can be difficult to ensure you described exactly which set of tests on which guests you want to run and kvm_config.py is invaluable in the process of getting it right. Why not have kvm_config.py , or some other wrapper generate a kvm_tests.cfg file dynamically from the library and the strings from the control file? That way we could still debug configuration via kvm_config.py? I much perfer this over queueing up jobs in the webiu, waiting for it to run, checking the results in the DEBUG dir, adjusting, repeat. I'm not sure I understand your idea: you want some program to read the control file and generate a new file (kvm_tests.cfg or something) from the control file and the library file, so that this file can be debugged with kvm_config.py? I wanted something that would take the test description string from the control file, run that through the library such that it can print out the tests that will run -- the equivalent of running kvm_config.py on a kvm_tests.cfg file... IMO this solution is dirty because the control file is python code, not our own format, so it's not nice to automatically extract stuff from it. It would be nice to do something that eases debugging, but if you ask me, I'd rather have something as clean as possible. Here's another idea, which I suggested but haven't received any feedback on: let's write a little proggie that runs the control file just like client/bin/autotest does. The proggie will supply the control file with a fake job object that has nothing but a run_test() method, but instead of running a test, that method will simply nicely print out the test params, like kvm_config.py does. So the user will be able to do something like './dry_run.py control.mine' which will list all the tests to be executed. We might also want to implement job.parallel() in addition to job.run_test() but that should be very easy (it doesn't really have to be parallel at all). Does that make any sense? It does, though I don't see how it's different then what I suggested; I don't really care how we do it; I still need something to debug test config files outside of cycling through job output via the webui; -- Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx ry...@us.ibm.com -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Autotest] [PATCH] [RFC] KVM test: Major control file cleanup
- Ryan Harper ry...@us.ibm.com wrote: * Lucas Meneghel Rodrigues l...@redhat.com [2009-10-28 14:48]: Ryan, Michael: I absolutely agree that the ability to debug stuff is important, but the ability to make things straightforward to use from the web interface or cli is also important. A longer term goal is to have our test farm and make any developer able to schedule a job on the test farm easily and conveniently. Having the dictionaries generated on the job debug directory seems like a good compromise to me. Also we can come up with a smart way of parsing the config file generated by a given control file in a similar way we do today with kvm_config.py, it shouldn't be that hard to do it... (I hope I won't burn my tongue with this statement). If I'm understanding things, we are talking about moving the large body of kvm_tests.cfg test definitions, guest definitions into a library, and then moving the requested test config (bottom on kvm_tests.cfg) into the control file itself which means the autotest webui would be able to control which tests get run; I like this idea very well. My concern that I mentioned is that as you edit the library it can be difficult to ensure you described exactly which set of tests on which guests you want to run and kvm_config.py is invaluable in the process of getting it right. Why not have kvm_config.py , or some other wrapper generate a kvm_tests.cfg file dynamically from the library and the strings from the control file? That way we could still debug configuration via kvm_config.py? I much perfer this over queueing up jobs in the webiu, waiting for it to run, checking the results in the DEBUG dir, adjusting, repeat. I'm not sure I understand your idea: you want some program to read the control file and generate a new file (kvm_tests.cfg or something) from the control file and the library file, so that this file can be debugged with kvm_config.py? IMO this solution is dirty because the control file is python code, not our own format, so it's not nice to automatically extract stuff from it. It would be nice to do something that eases debugging, but if you ask me, I'd rather have something as clean as possible. Here's another idea, which I suggested but haven't received any feedback on: let's write a little proggie that runs the control file just like client/bin/autotest does. The proggie will supply the control file with a fake job object that has nothing but a run_test() method, but instead of running a test, that method will simply nicely print out the test params, like kvm_config.py does. So the user will be able to do something like './dry_run.py control.mine' which will list all the tests to be executed. We might also want to implement job.parallel() in addition to job.run_test() but that should be very easy (it doesn't really have to be parallel at all). Does that make any sense? -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Autotest] [PATCH] [RFC] KVM test: Major control file cleanup
* Lucas Meneghel Rodrigues l...@redhat.com [2009-10-28 14:48]: Ryan, Michael: I absolutely agree that the ability to debug stuff is important, but the ability to make things straightforward to use from the web interface or cli is also important. A longer term goal is to have our test farm and make any developer able to schedule a job on the test farm easily and conveniently. Having the dictionaries generated on the job debug directory seems like a good compromise to me. Also we can come up with a smart way of parsing the config file generated by a given control file in a similar way we do today with kvm_config.py, it shouldn't be that hard to do it... (I hope I won't burn my tongue with this statement). If I'm understanding things, we are talking about moving the large body of kvm_tests.cfg test definitions, guest definitions into a library, and then moving the requested test config (bottom on kvm_tests.cfg) into the control file itself which means the autotest webui would be able to control which tests get run; I like this idea very well. My concern that I mentioned is that as you edit the library it can be difficult to ensure you described exactly which set of tests on which guests you want to run and kvm_config.py is invaluable in the process of getting it right. Why not have kvm_config.py , or some other wrapper generate a kvm_tests.cfg file dynamically from the library and the strings from the control file? That way we could still debug configuration via kvm_config.py? I much perfer this over queueing up jobs in the webiu, waiting for it to run, checking the results in the DEBUG dir, adjusting, repeat. -- Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx ry...@us.ibm.com -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Autotest] [PATCH] [RFC] KVM test: Major control file cleanup
On Fri, 2009-10-30 at 08:54 -0500, Ryan Harper wrote: * Lucas Meneghel Rodrigues l...@redhat.com [2009-10-28 14:48]: Ryan, Michael: I absolutely agree that the ability to debug stuff is important, but the ability to make things straightforward to use from the web interface or cli is also important. A longer term goal is to have our test farm and make any developer able to schedule a job on the test farm easily and conveniently. Having the dictionaries generated on the job debug directory seems like a good compromise to me. Also we can come up with a smart way of parsing the config file generated by a given control file in a similar way we do today with kvm_config.py, it shouldn't be that hard to do it... (I hope I won't burn my tongue with this statement). If I'm understanding things, we are talking about moving the large body of kvm_tests.cfg test definitions, guest definitions into a library, and then moving the requested test config (bottom on kvm_tests.cfg) into the control file itself which means the autotest webui would be able to control which tests get run; That is precisely my idea! I like this idea very well. My concern that I mentioned is that as you edit the library it can be difficult to ensure you described exactly which set of tests on which guests you want to run and kvm_config.py is invaluable in the process of getting it right. Yes, very valid concern. Let's try to address it. Why not have kvm_config.py , or some other wrapper generate a kvm_tests.cfg file dynamically from the library and the strings from the control file? That's one idea, the concern is that the code to do that might become ugly. But I will give it a try. That way we could still debug configuration via kvm_config.py? I much perfer this over queueing up jobs in the webiu, waiting for it to run, checking the results in the DEBUG dir, adjusting, repeat. Ok, I will go after this solution. Thanks for your comments Ryan, Lucas -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Autotest] [PATCH] [RFC] KVM test: Major control file cleanup
On Wed, Oct 28, 2009 at 1:43 PM, Michael Goldish mgold...@redhat.com wrote: Sounds great, except it won't allow you to debug your configuration using kvm_config.py. So the question now is what's more important -- the ability to debug or ease of use when running from the server. Here we have 2 use cases: 1) Users of the web interface, that (hopefully) have canned test sets that work reliably. Ability to debug stuff is less important on this scenario. 2) People developing tests, and in this case ability to debug config is very important I see the following options: 1) Document as part of the test development guide that, in order to be able to debug stuff, that all the test sets are to be written to the config file and then, can be parsed using kvm_config. 2) If we write all dictionaries generated by that particular configuration on files inside the job results directory, we still have debug ability for all use cases (I am starting to like this idea very much, as I type). So I'd then implement option 2) and refactor the control file with the test sets defined inside strings in the control file, then you can see how it looks? How about that? -- Lucas -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Autotest] [PATCH] [RFC] KVM test: Major control file cleanup
On Wed, Oct 28, 2009 at 1:57 PM, Michael Goldish mgold...@redhat.com wrote: One more comment on this patch: If you plan to make 'noinstall' the default build mode, then let's not run the build test at all by default. If we run it with 'noinstall', it will just run the preprocessor and postprocessor, and those will kill any living VMs because no VMs are required for the build test. So a user won't be able to run a reboot test, keep the VM alive, modify the config file and then run a different test on the same living VM, because the preprocessor of the second 'build' test will kill the VM. To avoid this, we can just remove the 'noinstall' variant from kvm_build.cfg, and keep the 'only noinstall' statement. Because there's no 'noinstall' variant, the resulting test list will be empty, so nothing will run. We should probably add a comment reminding the user that it's possible to disable installation by stating 'only noinstall' or 'only nothing' (in case the user replaced 'noinstall' with something else). Alternatively, we can disable installation in the control file by commenting out some line. Great idea, thanks :) - Lucas -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Autotest] [PATCH] [RFC] KVM test: Major control file cleanup
- Lucas Meneghel Rodrigues l...@redhat.com wrote: On Wed, Oct 28, 2009 at 1:43 PM, Michael Goldish mgold...@redhat.com wrote: Sounds great, except it won't allow you to debug your configuration using kvm_config.py. So the question now is what's more important -- the ability to debug or ease of use when running from the server. Here we have 2 use cases: 1) Users of the web interface, that (hopefully) have canned test sets that work reliably. Ability to debug stuff is less important on this scenario. 2) People developing tests, and in this case ability to debug config is very important I see the following options: 1) Document as part of the test development guide that, in order to be able to debug stuff, that all the test sets are to be written to the config file and then, can be parsed using kvm_config. 2) If we write all dictionaries generated by that particular configuration on files inside the job results directory, we still have debug ability for all use cases (I am starting to like this idea very much, as I type). So I'd then implement option 2) and refactor the control file with the test sets defined inside strings in the control file, then you can see how it looks? How about that? Sounds fine. - Where exactly will the test list appear? - We should also allow printing of verbose debug output (parsing variants block, 9000 dicts in current context...) by passing something to the constructor of the config object. - We should make it clear to the user that he/she must rename the control file (to control.lucas for example) or else it may be overwritten on the next git-fetch or -pull. I'm still not sure it's a great idea to make config debugging harder, so if anyone other than Lucas who uses the KVM test is reading this, please let us know if you ever use kvm_config.py and if you think the ability to print the list of test dicts is important. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Autotest] [PATCH] [RFC] KVM test: Major control file cleanup
Ryan, Michael: I absolutely agree that the ability to debug stuff is important, but the ability to make things straightforward to use from the web interface or cli is also important. A longer term goal is to have our test farm and make any developer able to schedule a job on the test farm easily and conveniently. Having the dictionaries generated on the job debug directory seems like a good compromise to me. Also we can come up with a smart way of parsing the config file generated by a given control file in a similar way we do today with kvm_config.py, it shouldn't be that hard to do it... (I hope I won't burn my tongue with this statement). Thanks for your input! On Wed, Oct 28, 2009 at 4:18 PM, Ryan Harper ry...@us.ibm.com wrote: * Michael Goldish mgold...@redhat.com [2009-10-28 10:43]: - Lucas Meneghel Rodrigues l...@redhat.com wrote: One thing that just occurred to me is, if we have a test config library as you said, it's perfectly possible to put the actual test set definitions and other config files inside the control file as strings. This way one can control configuration inside the control file, making it more convenient for usage, let's say, inside the autotest web interface. Since the control file would be reduced in size, the configuration snippets being in the control file would not be a huge problem, while keeping the original autotest philosophy of keeping stuff inside the control file... What do you think? Sounds great, except it won't allow you to debug your configuration using kvm_config.py. So the question now is what's more important -- the ability to debug or ease of use when running from the server. +1 debug When creating new test scenarios I make *heavy* use of kvm_config.py; I'd be lost without being able to debug test configuration files. -- Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx ry...@us.ibm.com ___ Autotest mailing list autot...@test.kernel.org http://test.kernel.org/cgi-bin/mailman/listinfo/autotest -- Lucas -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Autotest] [PATCH] [RFC] KVM test: Major control file cleanup
It's OK to log the debug output when running the framework, but Ryan says he makes heavy use of kvm_config.py, which means he may not find this comfortable enough. Ryan, what do you think? Another option is to allow editing any file from the web GUI in addition to the control file. I don't think it'll be hard to parse the control file and automatically extract config code from it in order to debug it, but it will almost certainly be very ugly. We might want to consider writing a little program that will emulate the client/bin/autotest.py program, running the control file and providing it with a fake job object whose run_test() method will just print the params dict instead of running the test. - Original Message - From: Lucas Meneghel Rodrigues l...@redhat.com To: Ryan Harper ry...@us.ibm.com Cc: Michael Goldish mgold...@redhat.com, autot...@test.kernel.org, u...@redhat.com, kvm@vger.kernel.org Sent: Wed, 28 Oct 2009 15:47:54 -0400 (EDT) Subject: Re: [Autotest] [PATCH] [RFC] KVM test: Major control file cleanup Ryan, Michael: I absolutely agree that the ability to debug stuff is important, but the ability to make things straightforward to use from the web interface or cli is also important. A longer term goal is to have our test farm and make any developer able to schedule a job on the test farm easily and conveniently. Having the dictionaries generated on the job debug directory seems like a good compromise to me. Also we can come up with a smart way of parsing the config file generated by a given control file in a similar way we do today with kvm_config.py, it shouldn't be that hard to do it... (I hope I won't burn my tongue with this statement). Thanks for your input! On Wed, Oct 28, 2009 at 4:18 PM, Ryan Harper ry...@us.ibm.com wrote: * Michael Goldish mgold...@redhat.com [2009-10-28 10:43]: - Lucas Meneghel Rodrigues l...@redhat.com wrote: One thing that just occurred to me is, if we have a test config library as you said, it's perfectly possible to put the actual test set definitions and other config files inside the control file as strings. This way one can control configuration inside the control file, making it more convenient for usage, let's say, inside the autotest web interface. Since the control file would be reduced in size, the configuration snippets being in the control file would not be a huge problem, while keeping the original autotest philosophy of keeping stuff inside the control file... What do you think? Sounds great, except it won't allow you to debug your configuration using kvm_config.py. So the question now is what's more important -- the ability to debug or ease of use when running from the server. +1 debug When creating new test scenarios I make *heavy* use of kvm_config.py; I'd be lost without being able to debug test configuration files. -- Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx ry...@us.ibm.com ___ Autotest mailing list autot...@test.kernel.org http://test.kernel.org/cgi-bin/mailman/listinfo/autotest -- Lucas -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
