I believe both can exist independently. The framework should have more
flexibility and facilitation to run features\test cases based upon a given
release and version on a given test bed. It should be intelligent enough to run
those cases and only those cases pertaining to that release with few config
variables. Keeping aside branch\repo terminology, for me all marvin both
framework and tests should be at one place and separate from main product code.
1. Even keeping tests as part of cs code, require the user to checkout the
marvin because of its dependencies. If a test\feature is using any marvin
library call still needs to have marvin and in that way he\she checksout marvin
anyway.
2. As a user, this is what i will do. On a given testbed,
a) I will checkout the cloudstack code
b) I will checkout the marvin code
c) For running automation, I will add few configuration parameters on that
test bed.
d) Then run the automation,
EX: sh run_marvin.sh "4.2" "bvt". This will run bvt using tests pertaining
to 4.2 release on that test bed.
EX: sh run_marvin.sh "4.1" "reg" "f1,f2,f3". This will run full regression
suite only with features f1,f2,f3
EX: sh run_marvin.sh "4.x" "reg" "all". This will run full regression
suite for me. These are specific and generic examples, the control of running
test code can be made more specific through config file\lib of marvin or
through cmd line arguments etc. That we can see to change later.
3. In a way marvin as a framework\library and tests are to some extent
dependent. Marvin is meant currently to test cs API's, so libraries written are
to enhance writing tests easy and in a way are more specific to cs product as a
whole. New features keeps getting added or features getting changed , all code
keeps changing, people writing new featuresets can add their feature to the
marvin branch containing tests as well and enhance any library call when they
see requires change.
4. As an example reference to this, we used to have multiple product branches
with varied patch and maintenance releases, with automation code maintained
separately having tests as well. We used to do as what we mentioned in step2.
It worked for us.
5. Only, thing is we need to make this transition and enhancing smoother so
that it will not effect current team activities of running automation.
Again, this is just a thought, Even keeping both cs and marvin as well will
not make much difference though i believe. Both, tests and library on which
these tests are dependent should be under one single folder i believe.
Regards,
Santhosh
________________________________________
From: Alex Huang [alex.hu...@citrix.com]
Sent: Wednesday, October 02, 2013 4:42 PM
To: dev@cloudstack.apache.org
Subject: RE: [DISCUSS] Breaking out Marvin from CloudStack
Yeah...I'm more amendable to this proposal. I just don't see tests being
separated from the source release. In fact, I see a lot of problems with
matching versions and releases.
I still question the value of separating out a framework where both ends (tests
and the server it tests) stay in the same repo but I guess there should be no
harm. I do think our time can be better spent elsewhere (for example, writing
the tests) but, if others see it as necessary, I wouldn't be against it.
@Edison
I can do what you said now. To me that's more or less maven changes and build
changes. Not a repo/separate release question.
--Alex
> IMO, we should consider Marvin the "framework" to be the thing to break
> out, and the tests should be different from the framework.
>
> Now that leads to the question: to test or not to test (in the main repo)?
>
> I'd suggest that *tests* belong in the main repo, because they are tied to the
> software's capabilities and versions.
>
> The Marvin framework, on the other hand, since the re-work that Prasanna
> did, is mostly distinct (and uses API discovery).
>
> Anyone else agree?
