On 01/19/2014 02:06 AM, Yair Fried wrote: > MT:"Is your issue here that it's just called basic ops and you don't think > that's > reflective of what is being tested in that file anymore" > > No. > My issue is, that the current scenario is, in fact, at least 2 separate > scenarios: > 1. original basic_ops > 2. reassociate_floating_ip > to which I would like to add ( https://review.openstack.org/#/c/55146/ ): > 4. check external/internal connectivity > 3. update dns > > While #2 includes #1 as part of its setup, its failing shouldn't prevent #1 > from passing. the obvious solution would be to create separate modules for > each test case, but since they all share the same setup sequence, IMO, they > should at least share code. > Notice that in my patch, #2 still includes #1. > > Actually, the more network scenario we get, the more we will need to do > something in that direction, since most of the scenarios will require the > setup of a VM with a floating-ip to ssh into. > > So either we do this, or we move all of this code to scenario.manager which > is also becoming very complicated
If #2 is always supposed to work, then I don't actually understand why
#1 being part of the test or not part of the test is really relevant.
And being part of the same test saves substantial time.
If you have tests that do:
* A -> B -> C
* A -> B -> D -> F
There really isn't value in a test for A -> B *as long* as you have
sufficient sign posting to know in the fail logs that A -> B worked fine.
And there are sufficient detriments in making it a separate test,
because it's just adding time to the runs without actually testing
anything different.
-Sean
>
> Yair
>
> ----- Original Message -----
> From: "Matthew Treinish" <mtrein...@kortar.org>
> To: "OpenStack Development Mailing List (not for usage questions)"
> <openstack-dev@lists.openstack.org>
> Sent: Friday, January 17, 2014 6:17:55 AM
> Subject: Re: [openstack-dev] [qa][Neutron][Tempest][Network] Break down
> NetworkBasicOps to smaller test cases
>
> On Wed, Jan 15, 2014 at 11:20:22AM -0500, Yair Fried wrote:
>> Hi Guys
>> As Maru pointed out - NetworkBasicOps scenario has grown out of proportion
>> and is no longer "basic" ops.
>
> Is your issue here that it's just called basic ops and you don't think that's
> reflective of what is being tested in that file anymore. If that's the case
> then just change the name.
>
>>
>> So, I started breaking it down to smaller test cases that can fail
>> independently.
>
> I'm not convinced this is needed. Some scenarios are going to be very involved
> and complex. Each scenario tests is designed to simulate real use cases in the
> cloud, so obviously some of them will be fairly large. The solution for making
> debugging easier in these cases is to make sure that any failures have clear
> messages. Also make sure there is plenty of signposting debug log messages so
> when something goes wrong you know what state the test was in.
>
> If you split things up into smaller individual tests you'll most likely end up
> making tests that are really aren't scenario tests. They'd be closer to API
> tests, just using the official clients, which really shouldn't be in the
> scenario tests.
>
>>
>> Those test cases share the same setup and tear-down code:
>> 1. create network resources (and verify them)
>> 2. create VM with floating IP.
>>
>> I see 2 options to manage these resources:
>> 1. Completely isolated - resources are created and cleaned using setUp() and
>> tearDown() methods [1]. Moving cleanup to tearDown revealed this bug [2].
>> Apparently the previous way (with tearDownClass) wasn't as fast). This has
>> the side effect of having expensive resources (ie VMs and floating IPs)
>> created and discarded multiple times though they are unchanged.
>>
>> 2. Shared Resources - Using the idea of (or actually using) Fixtures - use
>> the same resources unless a test case fails, in which case resources are
>> deleted and created by the next test case [3].
>
> If you're doing this and splitting things into smaller tests then it has to be
> option 1. Scenarios have to be isolated if there are resources shared between
> scenario tests that really is only one scenario and shouldn't be split. In
> fact
> I've been working on a change that fixes the scenario test tearDowns that has
> the
> side effect of enforcing this policy.
>
> Also just for the record we've tried doing option 2 in the past, for example
> there used to be a tenant-reuse config option. The problem with doing that was
> actually tends to cause more non-deterministic failures or adding a not
> insignificant wait time to ensure the state is clean when you start the next
> test. Which is why we ended up pulling this out of tree. What ends up
> happening
> is that you get leftover state from previous tests and the second test ends up
> failing because things aren't in the clean state that the test case assumes.
> If
> you look at some of the oneserver files in the API that is the only place we
> still do this in the tempest, and we've had many issues on making that work
> reliably. Those tests are in a relatively good place now but those are much
> simpler tests. Also between each test setUp has to check and ensure that the
> shared server is in the proper state. If it's not then the shared server has
> to
> be rebuilt. This methodology would become far more involved for the scenario
> tests where you have to manage more than one shared resource.
>
>>
>> I would like to hear your opinions, and know if anyone has any thoughts or
>> ideas on which direction is best, and why.
>>
>> Once this is completed, we can move on to other scenarios as well
>>
>> Regards
>> Yair
>>
>> [1] fully isolated - https://review.openstack.org/#/c/66879/
>> [2] https://bugs.launchpad.net/nova/+bug/1269407/+choose-affected-product
>> [3] shared resources - https://review.openstack.org/#/c/64622/
>
> -Matt Treinish
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
--
Sean Dague
Samsung Research America
s...@dague.net / sean.da...@samsung.com
http://dague.net
signature.asc
Description: OpenPGP digital signature
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
