I like the concept of a health check before launching tests. But it's a difficult distinction to make whether to run a specific test because its required hardware component/external dependency is not available/not ready in the deployment OR have the test ensure that the deployment is suitable for its run.
So our current health check only ensures two things - SystemVMs are up and running - Default Templates are 'Ready' in the system Take the example of LDAP here that you quoted. In such a case I'd prefer that the test itself check for the deployment guarantees without pushing that to the framework or the deployment scripts to decipher. There is a counter case - some tests use the netaddr module to do subnet arithmetic. For these to run we need to ensure the virtualenv under which the test module runs contains netaddr. This is something that can be ensured by the deployment scripts. So this is a question of guidelines to test authors on what to expect and what not to expect when writing a test. Also how to have the dependencies added automatically before the test gets executed. I don't see a one-size-fits-all solution. On Mon, Oct 28, 2013 at 05:49:39AM +0000, Santhosh Kumar Edukulla (JIRA) wrote: > Santhosh Kumar Edukulla created CLOUDSTACK-4971: > > 1, Currently test scripts are running despite the test setup has > issues. EX: In one of the test scenario, where ldap server was down, > then test suite depending upon them was still running. > > 2, Ideally, the setup should check for these dependencies before > running the test script and exit wherever possible with proper log > and reporting . > > 3, We may not be able to cover all setup issues, but the purpose of > setup is also to check for its existence and dependencies. If > scripts depending upon particular server conditions, should have a > proper setup verifying the server existence before proceeding > further. > > 4, I believe running test cases when setup is down is also not > idealistic. > > 5. Purpose is to reduce script errors vs failures due to product > bugs. > > 6. Logging this bug to track the setup issue cases or script > failures. > -- Prasanna., ------------------------ Powered by BigRock.com
