Sowmya Krishnan created CLOUDSTACK-4974: -------------------------------------------
Summary: [Automation] Handle exceptions for every API call in the test and verify the output Key: CLOUDSTACK-4974 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4974 Project: CloudStack Issue Type: Test Security Level: Public (Anyone can view this level - this is the default.) Components: Automation, marvin Affects Versions: 4.2.0 Environment: Test scripts written using Marvin Reporter: Sowmya Krishnan Present design of test cases assumes that the different APIs triggered as a pre-requisite of the test would all pass. For example, if we creating a VPC, we assume that the listVPCOffering API would pass (or that it is tested as part of some other test). Or after creating a Network we assume that listNetwork on the created network id would pass and return the expected network object. Also for most tests, not every single API call is enclosed within a try block. Given this, it is likely that if an API not directly related to the test fails, the test script can still fail and throws exceptions. While running huge number of tests this could lead to difficulty in debugging. One way to deal with this is to handle exceptions for every API call, but this would bloat the test scripts to unmanageable size. Other option that came up was to include a common decorator for the entire test which would take care of handling exception for the test. We probably have a better way to handle this in the re-factored Marvin, but at some point we probably need to fix existing scripts as well which have not been re-written using the "new" Marvin. -- This message was sent by Atlassian JIRA (v6.1#6144)