On Mon, Oct 28, 2013 at 06:24:05AM +0000, Santhosh Edukulla wrote: > Here, we mean test setup, not marvin setup ( or setup.py ) .
Sure - I understand that :). Marvin's setup is only a build+packaging file. I should've been clear - when I say deployment scripts those are the cloud-autodeploy scripts that setup the testbed. Marvin's dumb in that respect and only comes in when cloudstack is configured and tests are launched. > > Each test has their own dependency, we are having > setup,flow,teardown for test module. By below content of bug > description, we mean verifying the test dependencies under setup > class of a test feature or module not the marvin setup. The test > feature or module setup can take care to verify these dependencies > and exit with better log and reporting as against reporting the test > cases as failures. Right - so you're saying you'll change the tests. But as I mentioned with 'netaddr' it's the problem of the python environment running the test. The test is free to import 'netaddr' but not install it. > > Some dependencies at marvin setup can be verified, but the below > purpose is to ensure test dependencies are met before running tests. > The goal is to make less or zero % test script errors, against > genuine product bugs as failures. > Okay - by 'met before running' you mean during deployment or before test launch? Or as part of the setUpClass? If it is part of the setUpClass. Isn't that just a guideline to the test case writer? > Santhosh > ________________________________________ > From: Prasanna Santhanam [t...@apache.org] > Sent: Monday, October 28, 2013 2:15 AM > To: d...@cloudstack.apache.org > Cc: issues@cloudstack.apache.org > Subject: [DISCUSS] (CLOUDSTACK-4971) Verifying Test Beds : prerequisites for > launching tests > > I like the concept of a health check before launching tests. But it's > a difficult distinction to make whether to run a specific test because > its required hardware component/external dependency is not > available/not ready in the deployment OR have the test ensure that the > deployment is suitable for its run. > > So our current health check only ensures two things > - SystemVMs are up and running > - Default Templates are 'Ready' in the system > > Take the example of LDAP here that you quoted. In such a case I'd > prefer that the test itself check for the deployment guarantees > without pushing that to the framework or the deployment scripts to > decipher. > > There is a counter case - some tests use the netaddr module to do > subnet arithmetic. For these to run we need to ensure the virtualenv > under which the test module runs contains netaddr. This is something > that can be ensured by the deployment scripts. > > So this is a question of guidelines to test authors on what to expect > and what not to expect when writing a test. Also how to have the > dependencies added automatically before the test gets executed. I > don't see a one-size-fits-all solution. > > On Mon, Oct 28, 2013 at 05:49:39AM +0000, Santhosh Kumar Edukulla (JIRA) > wrote: > > Santhosh Kumar Edukulla created CLOUDSTACK-4971: > > > > 1, Currently test scripts are running despite the test setup has > > issues. EX: In one of the test scenario, where ldap server was down, > > then test suite depending upon them was still running. > > > > 2, Ideally, the setup should check for these dependencies before > > running the test script and exit wherever possible with proper log > > and reporting . > > > > 3, We may not be able to cover all setup issues, but the purpose of > > setup is also to check for its existence and dependencies. If > > scripts depending upon particular server conditions, should have a > > proper setup verifying the server existence before proceeding > > further. > > > > 4, I believe running test cases when setup is down is also not > > idealistic. > > > > 5. Purpose is to reduce script errors vs failures due to product > > bugs. > > > > 6. Logging this bug to track the setup issue cases or script > > failures. > > > -- > Prasanna., > > ------------------------ > Powered by BigRock.com -- Prasanna., ------------------------ Powered by BigRock.com
