On Thu, Mar 07, 2013 at 01:19:19AM +0530, Frank Zhang wrote: > > - improve simulator for runtime testability > > - customize to model and inject failures > > - make a habit of writing tests around bug reports (I started tagging tests > > since api_refactoring on JIRA already. > > look for the integration-test label) > > - make integration testing easier using factories and DSLs (from > > Chiradeep) > > Sorry for long thread again. All my opinion is "don't distract from > tools, improving > Marvin should be good enough". Below are points that you can simply skip.
Frank, no need to apologize, these are useful insights. My original proposal was only to refactor the marvin library as you suggest. It still is to provide a good library that involves less typing in python. > 2. capability > DSL varying from common language usually lacks loop, condition > statement, and sometimes variable. But > these things are very important to our test. if you strip these > abilities from your DSL, I believe you will finally > add them back that ends up with writing a common language, then you > have been distracted much from writing > test case > DSL is a good to have for non-programmers - the target audience could be those who are not comfortable writing python code and qa engineers who aren't experienced writing code. Agree that it will have its limitations. The dsl is only for quickly writing a few simple tests, it's not to replace the actual integrated test writing using python. I won't be doing that as it will convolute things. > 3. it's not our business now > The urgent need for CloudStack now is a set of reliable test cases, > which I think could be done by improving current > Marvin. Building our DSL distracts us at this point. If not for writing tests I"m sure it can be used for describing deployments for marvin. I'll figure out a compromise and your inputs on this is most welcome! > 4. you don't even need to search a test framework for this time being > When I studying DSL, I ever looked into some famous test framework. > For Behavior Driven Test I looked Cucumber(Ruby), > for Model Driven Test I looked fpmt(the name maybe wrong, written by > a French). They both use DSL. My conclusion is the > benefit of DSL is for stuff that has nothing to do with test logic > itself. For example, driving test cases, producing document/report, > scheduling random test case combination ... I did read enough dsl-bashing on the interwebs when I explored about it (terms like cargo cults from the ruby era come to mind) but it might be worth a shot. We won't know until we try. > they are nice to have, but not urgent. So for our purpose, I would > suggest using python nose which is a unit test case framework but > can still use for integration test. simple, quick, and easy. The nose support for marvin has existed for sometime and you can use many of nose's plugins today including - pdb, attributes, multiprocess etc. -- Prasanna.,
