from:"Yair Fried"

Re: [openstack-dev] [QA][Tempest] Use tempest-config for tempest-cli-improvements

2015-11-29 Thread Yair Fried

Hi,
I agree with Jordan.
We don't have to use the tool as part of the gate. It's target audience is
people and not CI systems. More specifically - new users.
However, we could add a gate (or a few) for the tool that makes sure a
proper conf file is generated. It doesn't have to run the tests, just
compare the output of the script to the conf generated by devstack.

Re Rally - I believe the best place for tempest configuration script is
within tempest. That said, if the Tempest community doesn't want this tool,
we'll have to settle for the Rally solution.

Regards
Yair

On Fri, Nov 27, 2015 at 11:31 AM, Jordan Pittier  wrote:

> Hi,
> I think this script is valuable to some users: Rally and Red Hat expressed
> their needs, they seem clear.
>
> This tool is far from bullet proof and if used blindly or in case of bugs,
> Tempest could be misconfigured. So, we could have this tool inside the
> Tempest repository (in the tools/) but not use it at all for the Gate.
>
> I am not sure I fully understand the resistance for this, if we don"t use
> this config generator for the gate, what's the risk ?
>
> Jordan
>
> On Fri, Nov 27, 2015 at 8:05 AM, Ken'ichi Ohmichi 
> wrote:
>
>> 2015-11-27 15:40 GMT+09:00 Daniel Mellado :
>> > I still do think that even if there are some issues addressed to the
>> > feature, such as skipping tests in the gate, the feature itself it's
>> still
>> > good -we just won't use it for the gates-
>> > Instead it'd be used as a wrapper for a user who would be interested on
>> > trying it against a real/reals clouds.
>> >
>> > Ken, do you really think a tempest user should know all tempest options?
>> > As you pointed out there are quite a few of them and even if they
>> should at
>> > least know their environment, this script would set a minimum acceptable
>> > default. Do you think PTL and Pre-PTL concerns that we spoke of would
>> still
>> > apply to that scenario?
>>
>> If Tempest users run part of tests of Tempest, they need to know the
>> options which are used with these tests only.
>> For example, current Tempest contains ironic API tests and the
>> corresponding options.
>> If users don't want to run these tests because the cloud don't support
>> ironic API, they don't need to know/setup these options.
>> I feel users need to know necessary options which are used on tests
>> they want, because they need to investigate the reason if facing a
>> problem during Tempest tests.
>>
>> Now Tempest options contain their default values, but you need a
>> script for changing them from the default.
>> Don't these default values work for your cloud at all?
>> If so, these values should be changed to better.
>>
>> Thanks
>> Ken Ohmichi
>>
>> ---
>>
>> > Andrey, Yaroslav. Would you like to revisit the blueprint to adapt it to
>> > tempest-cli improvements? What do you think about this, Masayuki?
>> >
>> > Thanks for all your feedback! ;)
>> >
>> > El 27/11/15 a las 00:15, Andrey Kurilin escribió:
>> >
>> > Sorry for wrong numbers. The bug-fix for issue with counters is merged.
>> > Correct numbers(latest result from rally's gate[1]):
>> >  - total number of executed tests: 1689
>> >  - success: 1155
>> >  - skipped: 534 (neutron,heat,sahara,ceilometer are disabled. [2] should
>> > enable them)
>> >  - failed: 0
>> >
>> > [1] -
>> >
>> http://logs.openstack.org/27/246627/11/gate/gate-rally-dsvm-verify-full/800bad0/rally-verify/7_verify_results_--html.html.gz
>> > [2] - https://review.openstack.org/#/c/250540/
>> >
>> > On Thu, Nov 26, 2015 at 3:23 PM, Yaroslav Lobankov <
>> yloban...@mirantis.com>
>> > wrote:
>> >>
>> >> Hello everyone,
>> >>
>> >> Yes, I am working on this now. We have some success already, but there
>> is
>> >> a lot of work to do. Of course, some things don't work ideally. For
>> example,
>> >> in [2] from the previous letter we have not 24 skipped tests, actually
>> much
>> >> more. So we have a bug somewhere :)
>> >>
>> >> Regards,
>> >> Yaroslav Lobankov.
>> >>
>> >> On Thu, Nov 26, 2015 at 3:59 PM, Andrey Kurilin > >
>> >> wrote:
>> >>>
>> >>> Hi!
>> >>> Boris P. and I tried to push a spec[1] for automation tempest config
>> >>> generator, but we did not succeed to merge it. Imo, qa-team doesn't
>> want to
>> >>> have such tool:(
>> >>>
>> >>> >However, there is a big concern:
>> >>> >If the script contain a bug and creates the configuration which makes
>> >>> >most tests skipped, we cannot do enough tests on the gate.
>> >>> >Tempest contains 1432 tests and difficult to detect which tests are
>> >>> >skipped as unexpected.
>> >>>
>> >>> Yaroslav Lobankov is working on improvement for tempest config
>> generator
>> >>> in Rally. Last time when we launch full tempest run[2], we got 1154
>> success
>> >>> tests and only 24 skipped. Also, there is a patch, which adds x-fail
>> >>> mechanism(it based on subunit-filter): you can transmit a file with
>> test
>> >>> names + reasons and rally will modify results.
>> >>>
>> >>> [1] - https://review.openstack.org/#/c/94473/
>> >>>
>>

Re: [openstack-dev] [rally] Rally boot tests fails with Error Forbidden: It is not allowed to create an interface on external networks

2015-10-20 Thread Yair Fried

On Tue, Oct 20, 2015 at 8:39 PM, Behzad Dastur 
wrote:

> Hi Yair,
> The rally version I am using is 0.1.2
>
> > rally --version
> 0.1.2
>
> Also the task file is as shown below. Do you have an example of the
> "network" context to skip creation on the interface on the xternal network?
>
have you seen the plugin reference?
https://rally.readthedocs.org/en/latest/plugin/plugin_reference.html
Looks like there's also existing_network [context] but I'm unfamiliar with
it.

> vagrant@rally:~/rally$ more /vagrant/boot.json
>
> {% set flavor_name = flavor_name or "m1.tiny" %}
>
> {
>
> "NovaServers.boot_server": [
>
> {
>
> "args": {
>
> "flavor": {
>
> "name": "{{flavor_name}}"
>
> },
>
"auto_assign_nic": true,

> "image": {
>
> "name": "cirros-0.3.1-x86_64"
>
> },
>
> "use_floatingip": false
>
I think this should be true (or maybe even removed)

> },
>
> "runner": {
>
> "type": "constant",
>
> "times": 10,
>
> "concurrency": 2
>
> },
>
>     "context": {
>
"network": {"networks_per_tenant": 1},

> "users": {
>
> "tenants": 3,
>
> "users_per_tenant": 2
>
> }
>
> }
>
> }
>
> ]
>
> }
>
> regards,
> Behzad
>
> Date: Tue, 20 Oct 2015 15:04:46 +0300
> From: Yair Fried 
> To: "OpenStack Development Mailing List (not for usage questions)"
> 
> Subject: Re: [openstack-dev] [rally] Rally boot tests fails with Error
> Forbidden: It is not allowed to create an interface on external
> networks
> Message-ID:
>  rejjeirq8m630049woa...@mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
>
> On Tue, Oct 20, 2015 at 2:06 PM, Behzad Dastur 
> wrote:
>
> > I have a contrail/OpenStack cloud deployed on which I am trying to run
> > some rally benchmarks. But I am having trouble getting the rally boot
> tests
> > to run. It throws the "Error Forbidden: It is not allowed to create an
> > interface on external network"
> >
> > It seems it is trying to create an interface on the external network,
> > however in this case that operation is not required as the contrail
> plugin
> > handles that.
> >
> What version of Rally are you using?
> Could you please provide your task file? Looks like you are explicitly
> telling rally to use your external network for the VMs.
>
>
> >  Is there a way to tell the rally scenario to avoid doing that. SImply
> > put the operations that need to happen are:
> >
> > 1. nova boot (create private network/ or use private network provided)
> >
> The "network" context should allow you to dynamically create the networks.
> Also, all scenarios that boot an instance can propagate boot arguments even
> if they aren't explicitly listed (for more details try "$ rally plugin info
> "), so you should be able to pass "{networks: {uuid: }}"
> to the scenario.
>
> 2. neutron floating ip create, and assign it to the port eg (neutron
> > floatingip-create --port-id   net=""
> > id="">)
> >
> Only in VMTask AFAIK.
>
>
> > Here is the error log:
> >
> > 2015-10-20 00:24:12.759 19075 INFO 
> > rally.plugins.openstack.context.keystone.users
> [-] Task 3000fcbd-2762-400d-920f-dfbfb667e7ec | Starting:  Enter
> context: `users`2015-10-20 00:24:14.711 19075 INFO 
> rally.plugins.openstack.context.keystone.users
> [-] Task 3000fcbd-2762-400d-920f-dfbfb667e7ec | Completed: Enter context:
> `users`2015-10-20 00:24:16.222 19264 INFO rally.task.runner [-] Task
> 3000fcbd-2762-400d-920f-dfbfb667e7ec | ITER: 0 START2015-10-20
> 00:24:16.227 19264 INFO rally.task.runner [-] Task 
> 3000fcbd-2762-400d-920f-dfbfb667e7ec
> | ITER: 1 START2015-10-20 00:24:18.420 19264 INFO rally.task.runner [-]
> Task 3000fcbd-2762-400d-920f-dfbfb667e7ec | ITER: 0 END: Error Forbidden:
> It is not allowed to create an interface on external network
> 2de28d39-34f9-48c5-bbac-609e258b7aad (HTTP 403) (Request-ID:
> req-fe32bcf8-f624-4a2d-a083-7b6c5d1f24ab)
> >
> >
> > regards,
> > Behzad
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [rally] Rally boot tests fails with Error Forbidden: It is not allowed to create an interface on external networks

2015-10-20 Thread Yair Fried

On Tue, Oct 20, 2015 at 2:06 PM, Behzad Dastur 
wrote:

> I have a contrail/OpenStack cloud deployed on which I am trying to run
> some rally benchmarks. But I am having trouble getting the rally boot tests
> to run. It throws the "Error Forbidden: It is not allowed to create an
> interface on external network"
>
> It seems it is trying to create an interface on the external network,
> however in this case that operation is not required as the contrail plugin
> handles that.
>
What version of Rally are you using?
Could you please provide your task file? Looks like you are explicitly
telling rally to use your external network for the VMs.


>  Is there a way to tell the rally scenario to avoid doing that. SImply
> put the operations that need to happen are:
>
> 1. nova boot (create private network/ or use private network provided)
>
The "network" context should allow you to dynamically create the networks.
Also, all scenarios that boot an instance can propagate boot arguments even
if they aren't explicitly listed (for more details try "$ rally plugin info
"), so you should be able to pass "{networks: {uuid: }}"
to the scenario.

2. neutron floating ip create, and assign it to the port eg (neutron
> floatingip-create --port-id   id="">)
>
Only in VMTask AFAIK.


> Here is the error log:
>
> 2015-10-20 00:24:12.759 19075 INFO 
> rally.plugins.openstack.context.keystone.users [-] Task 3000fcbd-2762-400d
> -920f-dfbfb667e7ec | Starting:  Enter context: `users`2015-10-20 00:24:14.711 
> 19075 INFO rally.plugins.openstack.context.keystone.users [-] Task 
> 3000fcbd-2762-400d-920f-dfbfb667e7ec | Completed: Enter context: 
> `users`2015-10-20 00:24:16.222 19264 INFO rally.task.runner [-] Task 
> 3000fcbd-2762-400d-920f-dfbfb667e7ec | ITER: 0 START2015-10-20 00:24:16.227 
> 19264 INFO rally.task.runner [-] Task 3000fcbd-2762-400d-920f-dfbfb667e7ec | 
> ITER: 1 START2015-10-20 00:24:18.420 19264 INFO rally.task.runner [-] Task 
> 3000fcbd-2762-400d-920f-dfbfb667e7ec | ITER: 0 END: Error Forbidden: It is 
> not allowed to create an interface on external network 
> 2de28d39-34f9-48c5-bbac-609e258b7aad (HTTP 403) (Request-ID: 
> req-fe32bcf8-f624-4a2d-a083-7b6c5d1f24ab)
>
>
> regards,
> Behzad
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [rally] Weekly meeting

2015-05-05 Thread Yair Fried

Thank you for moving it to a more reasonable time for me.

- Original Message -
From: "Boris Pavlovic" 
To: "OpenStack Development Mailing List (not for usage questions)" 
, "carlos torres" 
, yfr...@redhat.com, "yingjun li" 
, "Aleksandr Maretskiy" , 
"Andrey Kurilin" , "Mikhail Dubov" , 
"Oleg Anufriev" , "Roman Vasilets" 
, "Sergey Skripnick" 
Sent: Tuesday, May 5, 2015 6:28:52 PM
Subject: Re: [openstack-dev] [rally] Weekly meeting

+Rally team

Just to make sure that everybody saw this.

Best regards,
Boris Pavlovic

On Tue, May 5, 2015 at 6:19 PM, Mikhail Dubov  wrote:

> Hi everyone,
>
> sorry for the previous disinformative message. We have decided to move our
> weekly meeting to Wednesdays at 14:00 UTC (IRC, *#openstack-meeting*). As
> said before, all the relevant information including the meeting agenda can
> be found on the wiki page 
> .
>
> Best regards,
> Mikhail Dubov
>
> Engineering OPS
> Mirantis, Inc.
> E-Mail: mdu...@mirantis.com
> Skype: msdubov
>
> On Tue, May 5, 2015 at 6:10 PM, Mikhail Dubov  wrote:
>
>> Hi everyone,
>>
>> let me remind you that today there is the weekly Rally meeting at 17:00
>> UTC (IRC, *#openstack-meeting*).
>>
>> Starting from today, we will be posting our meeting agenda at the
>> corresponding wiki page 
>> . Fell free to comment on the agenda / to propose new topics.
>>
>> Best regards,
>> Mikhail Dubov
>>
>> Engineering OPS
>> Mirantis, Inc.
>> E-Mail: mdu...@mirantis.com
>> Skype: msdubov
>>
>> On Tue, Apr 28, 2015 at 1:05 PM, Mikhail Dubov 
>> wrote:
>>
>>> Hi everyone,
>>>
>>> let me remind you that today there is the weekly Rally meeting at 17:00
>>> UTC (IRC, *#openstack-meeting*).
>>>
>>> Here is the agenda for today:
>>>
>>>1. Rally QA week: tasks, assignees, progress
>>>2. Upcoming Rally 0.0.4 release: progress on critical patches
>>>3. Spec on refactoring scenario utils: review and discussion (
>>>https://review.openstack.org/#/c/172831/)
>>>4. Spec on in-tree functional tests: review and discussion (
>>>https://review.openstack.org/#/c/166487/)
>>>5. Free discussion
>>>
>>> The meeting is going to be chaired by Alexander Maretskiy.
>>>
>>> Fell free to comment on the agenda / to propose new topics.
>>>
>>>
>>> Best regards,
>>> Mikhail Dubov
>>>
>>> Engineering OPS
>>> Mirantis, Inc.
>>> E-Mail: mdu...@mirantis.com
>>> Skype: msdubov
>>>
>>
>>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Rally] Improve review process

2015-05-05 Thread Yair Fried

+1

- Original Message -
From: "Boris Pavlovic" 
To: "OpenStack Development Mailing List (not for usage questions)" 

Sent: Tuesday, May 5, 2015 6:09:43 PM
Subject: Re: [openstack-dev] [Rally] Improve review process

Roman, 


Well done! This really simplifies life! 


Best regards, 
Boris Pavlovic 

On Tue, May 5, 2015 at 4:07 PM, Mikhail Dubov < mdu...@mirantis.com > wrote: 



Hi Roman, 

a truly great job. Very impressive and useful. Thanks a lot! 

Best regards, 
Mikhail Dubov 

Engineering OPS 
Mirantis, Inc. 
E-Mail: mdu...@mirantis.com 
Skype: msdubov 

On Tue, May 5, 2015 at 3:11 PM, Roman Vasilets < rvasil...@mirantis.com > 
wrote: 





Hi, Rally Team. 

I have created Rally Gerrit dashboard that organized patches in groups: 
Critical for next release, Waiting for final approve, Bug fixes, Proposed 
specs, Important patches, Ready for review, Has -1 but passed tests. Please use 
link http://goo.gl/iRxA5t for you comfortable. Patch is here 
https://review.openstack.org/#/c/179610/ It was made by gerrit-dash-creator. 

First group are the patches that are needed to merge to the nearest release. 
Content of the next three groups is obvious from the titles. Important patches 
- its just patches chosen(starred) by Boris Pavlovic or Mikhail Dubov. Ready 
for review - patches that are waiting for attention. And the last section - its 
patches with -1 mark but they passed CI. 


Roman Vasilets, Mirantis Inc. 

Intern Software Engineer 

__ 
OpenStack Development Mailing List (not for usage questions) 
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe 
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev 



__ 
OpenStack Development Mailing List (not for usage questions) 
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe 
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev 



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [opentack-dev][meetings] Proposing changes in Rally meetings

2015-04-18 Thread Yair Fried

- Original Message -

> From: "Aleksandr Maretskiy" 
> To: "Andrey Kurilin" 
> Cc: "Boris Pavlovic" , "OpenStack Development Mailing
> List" , yfr...@redhat.com, "yingjun li"
> , "Mikhail Dubov" , "Oleg
> Anufriev" , "Roman Vasilets"
> , "Sergey Skripnick" 
> Sent: Saturday, April 18, 2015 1:00:55 PM
> Subject: Re: [opentack-dev][meetings] Proposing changes in Rally meetings

> Agreed with everything, but I think it would be a bit better if move one
> meeting from Monday to another day (Andrey K. is right)

> On Fri, Apr 17, 2015 at 5:35 PM, Andrey Kurilin < akuri...@mirantis.com >
> wrote:

> > > - We should start making agenda for each meeting and publish it to Rally
> > > wiki
> 

> > +1
> 

+1 

> > > * Second is release management meeting, where we are discussing
> > > priorities
> > > for
> 
> > > current & next release. So core team will know what to review first.
> 

> > It would be nice to post list of high priority patches to etherpad or
> > google
> > docs after each meeting
> 

> > > - Move meetings from #openstack-meeting to #openstack-rally chat.
> 

> > doesn't matter for me:)
> 

As long as the records are kept. 

> > > - We should adjust better time for current Rally team.
> 

> > yeah. Current time is not good:( +1 for 15:00 UTC
> 

I'd like even earlier, but if it works for everyone else, I'll make the effort. 

> > > - Do meetings every Monday and Wednesday
> 

> > Monday?) Monday is very hard day...
> 

> > On Fri, Apr 17, 2015 at 4:26 PM, Boris Pavlovic < bo...@pavlovic.me >
> > wrote:
> 

> > > Rally team,
> > 
> 

> > > I would like to propose next changes in Rally meetings:
> > 
> 

> > > - We should start making agenda for each meeting and publish it to Rally
> > > wiki
> > 
> 

> > > - We should do 2 meeting per week:
> > 
> 

> > > * First is regular meeting (like we have now) where we are discussing
> > > everything
> > 
> 
> > > * Second is release management meeting, where we are discussing
> > > priorities
> > > for
> > 
> 
> > > current & next release. So core team will know what to review first.
> > 
> 

Seems like the 2nd meeting is mainly for core, so maybe we can set a 
better(earlier) time for it among a smaller group? 

> > > - Move meetings from #openstack-meeting to #openstack-rally chat.
> > 
> 

> > > - We should adjust better time for current Rally team. Like at the moment
> > > it
> > > is too late
> > 
> 
> > > for few of cores in Rally. it's 17:00 UTC and I would like to suggest to
> > > make
> > > at 15:00 UTC.
> > 
> 

> > > - Do meetings every Monday and Wednesday
> > 
> 

> > > Thoughts ?
> > 
> 

> > > Best regards,
> > 
> 
> > > Boris Pavlovic
> > 
> 

> > --
> 
> > Best regards,
> 
> > Andrey Kurilin.
> 
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa] Lack of consistency in returning response from tempest clients

2014-08-30 Thread Yair Fried

Hi,
I'd rather not subclass dict directly.
for various reasons adding extra attributes to normal python dict seems prone 
to errors since people will be expecting regular dicts, and on the other hand 
if we want to expand it in the future we might run into problems playing with 
dict methods (such as "update")

I suggets (roughly):

class ResponseBody(dict): 
def __init__(self, body=None, resp=None): 
self_data_dict = body or {} 
self.resp = resp 

def __getitem__(self, index):
return self._data_dict[index]

Thus we can keep the previous dict interface, but protect the data and make 
sure the object will behave exactly as we expect it to. if we want it to have 
more dict attributes/method we can add them explicitly

- Original Message -
From: "Boris Pavlovic" 
To: "OpenStack Development Mailing List (not for usage questions)" 

Sent: Saturday, August 30, 2014 2:53:37 PM
Subject: Re: [openstack-dev] [qa] Lack of consistency in returning response 
from tempest clients

Sean, 

class ResponseBody(dict): 
def __init__(self, body={}, resp=None): 
self.update(body) 
self.resp = resp 

Are you sure that you would like to have default value {} for method argument 
and not something like: 

class ResponseBody(dict): 
def __init__(self, body=None, resp=None): 
body = body or {} 
self.update(body) 
self.resp = resp 

In your case you have side effect. Take a look at: 
http://stackoverflow.com/questions/1132941/least-astonishment-in-python-the-mutable-default-argument

Best regards, 
Boris Pavlovic 

On Sat, Aug 30, 2014 at 10:08 AM, GHANSHYAM MANN < ghanshyamm...@gmail.com > 
wrote: 

+1. That will also help full for API coming up with microversion like Nova. 

On Fri, Aug 29, 2014 at 11:56 PM, Sean Dague < s...@dague.net > wrote: 

On 08/29/2014 10:19 AM, David Kranz wrote: 
> While reviewing patches for moving response checking to the clients, I 
> noticed that there are places where client methods do not return any value. 
> This is usually, but not always, a delete method. IMO, every rest client 
> method should return at least the response. Some services return just 
> the response for delete methods and others return (resp, body). Does any 
> one object to cleaning this up by just making all client methods return 
> resp, body? This is mostly a change to the clients. There were only a 
> few places where a non-delete method was returning just a body that was 
> used in test code. 

Yair and I were discussing this yesterday. As the response correctness 
checking is happening deeper in the code (and you are seeing more and 
more people assigning the response object to _ ) my feeling is Tempest 
clients should probably return a body obj that's basically. 

class ResponseBody(dict): 
def __init__(self, body={}, resp=None): 
self.update(body) 
self.resp = resp 

Then all the clients would have single return values, the body would be 
the default thing you were accessing (which is usually what you want), 
and the response object is accessible if needed to examine headers. 

-Sean 

-- 
Sean Dague 
http://dague.net 

___ 
OpenStack-dev mailing list 
OpenStack-dev@lists.openstack.org 
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev 

-- 
Thanks & Regards 
Ghanshyam Mann 

___ 
OpenStack-dev mailing list 
OpenStack-dev@lists.openstack.org 
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] [LBaaS] LBaaS v2 API syntax additions/changes

2014-08-28 Thread Yair Fried

I would like to add a question to John's list



- Original Message -
> From: "John Schwarz" 
> To: "OpenStack Development Mailing List (not for usage questions)" 
> 
> Sent: Tuesday, August 26, 2014 2:22:33 PM
> Subject: Re: [openstack-dev] [Neutron] [LBaaS] LBaaS v2 API syntax
> additions/changes
> 
> 
> 
> On 08/25/2014 10:06 PM, Brandon Logan wrote:
> >>
> >> 2. Therefor, there should be some configuration to specifically enable
> >> either version (not both) in case LBaaS is needed. In this case, the
> >> other version is disabled (ie. a REST query for non-active version
> >> should return a "not activated" error). Additionally, adding a
> >> 'lb-version' command to return the version currently active seems like a
> >> good user-facing idea. We should see how this doesn't negatively effect
> >> the db migration process (for example, allowing read-only commands for
> >> both versions?)
> > 
> > A /version endpoint can be added for both v1 and v2 extensions and
> > service plugins.  If it doesn't already exist, it would be nice if
> > neutron had an endpoint that would return the list of loaded extensions
> > and their versions.
> > 
> There is 'neutron ext-list', but I'm not familiar enough with it or with
> the REST API to say if we can use that.
> >>
> >> 3. Another decision that's needed to be made is the syntax for v2. As
> >> mentioned, the current new syntax is 'neutron lbaas--'
> >> (against the old 'lb--'), keeping in mind that once v1
> >> is deprecated, a syntax like 'lbv2--' would be probably
> >> unwanted. Is 'lbaas--' okay with everyone?
> > 
> > That is the reason we with with lbaas because lbv2 looks ugly and we'd
> > be stuck with it for the lifetime of v2, unless we did another migration
> > back to lb for it.  Which seemed wrong to do, since then we'd have to
> > accept both lbv2 and lb commands, and then deprecate lbv2.
> > 
> > I assume this also means you are fine with the prefix in the API
> > resource of /lbaas as well then?
> > 
> I don't mind, as long there is a similar mechanism which disables the
> non-active REST API commands. Does anyone disagree?
> >>
> >> 4. If we are going for different API between versions, appropriate
> >> patches also need to be written for lbaas-related scripts and also
> >> Tempest, and their maintainers should probably be notified.
> > 
> > Could you elaborate on this? I don't understand what you mean by
> > "different API between version."
> > 
> The intention was that the change of the user-facing API also forces
> changes on other levels - not only neutronclient needs to be modified
> accordingly, but also tempest system tests, horizon interface regarding
> LBaaS...


5. If we accept #3 and #4 to mean that the python-client API and CLI must be 
changed/updated and so does Tempest clients and tests, then what about other 
projects consuming the Neutron API? How are Heat and Ceilometer going to be 
affected by this change?

Yair


> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa] The role of an abstract client in tempest

2014-07-28 Thread Yair Fried





- Original Message -
> From: "David Kranz" 
> To: "OpenStack Development Mailing List" 
> Sent: Friday, July 25, 2014 4:54:22 PM
> Subject: [openstack-dev] [qa] The role of an abstract client in tempest
> 
> Even as a core contributor for several years, it has never been clear
> what the scope of these tests should be.
> As we move forward with the necessity of moving functional testing to
> projects, we need to answer this question for real, understanding that
> part of the mission for these tests now is validation of clouds.  Doing
> so is made difficult by the fact that the tempest api tests take a very
> opinionated view of how services are invoked. In particular, the tempest
> client is very low-level and at present the way a functional test is
> written depends on how and where it is going to run.
> 
> In an ideal world, functional tests could execute in a variety of
> environments ranging from those that completely bypass wsgi layers and
> make project api calls directly, to running in a fully integrated "real"
> environment as the tempest tests currently do. The challenge is that
> there are mismatches between how the tempest client looks to test code
> and how doing object-model api calls looks. Most of this discrepancy is
> because many pieces of invoking a service are hard-coded into the tests
> rather than being abstracted in a client. Some examples are:
> 
> 1. Response validation
> 2. json serialization/deserialization
> 3. environment description (tempest.conf)
> 4. Forced usage of addCleanUp
> 
> Maru Newby and I have proposed changing the test code to use a more
> abstract client by defining the expected signature and functionality
> of methods on the client. Roughly, the methods would take positional
> arguments for pieces that go in the url part of a REST call, and kwargs
> for the json payload. The client would take care of these enumerated
> issues (if necessary) and return an attribute dict. The test code itself
> would then just be service calls and checks of returned data. Returned
> data would be inspected as resource.id instead of resource['id']. There
> is a strawman example of this for a few neutron apis here:
> https://review.openstack.org/#/c/106916/
> Doing this would have the twin advantages of eliminating the need for
> boilerplate code in tests and making it possible to run the tests in
> different environments. It would also allow the inclusion of project
> functional tests in more general validation scenarios.
> 
> Since we are proposing to move parts of tempest into a stable library
> https://review.openstack.org/108858, we need to define the client in a
> way that meets all the needs outlined here before doing so. The actual
> work of defining the client in tempest and changing the code that uses
> it could largely be done one service at a time, in the tempest code,
> before being split out.
> 
> What do folks think about this idea?

I agree.
I also believe that streamlining the clients should take (at leaset partial)
precedence to 
https://blueprints.launchpad.net/tempest/+spec/tempest-client-scenarios.
I fear that in their current state, the tempest client are not ready to be used
in integration testing, and using them will push as backward on the scenario 
front
since we will loose much of the reusable code.

However, I believe that a week of coordinated and concentrated effort should 
get us
to a "good enough" state, to move on both tempest-client-scenarios bp, and 
further 
enhancing the tempest clients one at a time

Yair

> 
>   -David
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [QA] Questions about test policy for scenario test

2014-06-24 Thread Yair Fried

- Original Message -

> From: "Fei Long Wang" 
> To: "OpenStack Development Mailing List (not for usage questions)"
> 
> Cc: br...@catalyst.net.nz
> Sent: Tuesday, June 24, 2014 8:29:03 AM
> Subject: [openstack-dev] [QA] Questions about test policy for scenario test

> Greetings,

> We're leveraging the scenario test of Tempest to do the end-to-end
> functional test to make sure everything work great after upgrade,
> patching, etc. And We're happy to fill the gaps we found. However, I'm a
> little bit confused about the test policy from the scenario test
> perspective, especially comparing with the API test. IMHO, scenario test
> will cover some typical work flows of one specific service or mixed
> services, and it would be nice to make sure the function is really
> working instead of just checking the object status from OpenStack
> perspective. Is that correct?

> For example, live migration of Nova, it has been covered in API test of
> Tempest (see
> https://github.com/openstack/tempest/blob/master/tempest/api/compute/test_live_block_migration.py).
> But as you see, it just checks if the instance is Active or not instead
> of checking if the instance can be login/ssh successfully
Seems to me, that what you want is to add migration test to 
https://github.com/openstack/tempest/blob/master/tempest/scenario/test_network_advanced_server_ops.py
 
This scenario does exactly what you are looking for 
1. check VM connectivity 
2. mess with VM (reboot, resize, or in your case - migrate) 
3. check VM connectivity 

> . Obviously,
> from an real world view, we'd like to check if it's working indeed. So
> the question is, should this be improved? If so, the enhanced code
> should be in API test, scenario test or any other places? Thanks you.

> --
> Cheers & Best regards,
> Fei Long Wang (王飞龙)
> --
> Senior Cloud Software Engineer
> Tel: +64-48032246
> Email: flw...@catalyst.net.nz
> Catalyst IT Limited
> Level 6, Catalyst House, 150 Willis Street, Wellington
> --

> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [QA] Tempest Release Naming

2014-05-27 Thread Yair Fried

- Original Message -
> From: "Sean Dague" 
> To: "OpenStack Development Mailing List (not for usage questions)" 
> 
> Sent: Friday, May 23, 2014 3:40:16 AM
> Subject: Re: [openstack-dev] [QA] Tempest Release Naming
> 
> On 05/22/2014 08:33 PM, Jeremy Stanley wrote:
> > On 2014-05-22 18:33:34 -0400 (-0400), Matthew Treinish wrote:
> > [...]
> >> I'd like to stick with one scheme and not decide to change it
> >> later on. I figured I should bring this out to a wider audience to
> >> see if there were other suggestions or opinions before I pushed
> >> out the tag, especially because the tags are primarily for the
> >> consumers of Tempest.
> > 
> > I think as long as your release notes clearly indicate what you
> > think can safely be tested with a given release of Tempest, it
> > should be versioned just like any other piece of software (which is
> > to say, with version numbers chosen by one or more humans to
> > reflect
> > the degree of improvement/breakage reflected within the release, a
> > la "Semantic Versioning").
> 
> Given that we're not really going to have major / minor bumps at this
> point (more a continous roll), I might argue that we should just go
> Firefox on it and bump the integer on every release.
> 
> Tempest 1 is now, Tempest 2 is Julyish, Tempest 3 is at Juno, Tempest
> 4
> is next winter, etc.
> 
>   -Sean

+1
very practical, very simple. clearly independent of Openstack releases

> 
> --
> Sean Dague
> http://dague.net
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron][QA]Request for input for Juno Design Summit, Atlanta

2014-04-28 Thread Yair Fried

Hi,
For everyone's convenience, I've added to the pad short descriptions of Network 
Scenarios that are currently in tree (or under review) that I am familiar with.
Feel free to add/edit


Regards
Yair
 

- Original Message -
> From: "Miguel Lavalle" 
> To: "OpenStack Development Mailing List" 
> Sent: Tuesday, April 29, 2014 1:48:23 AM
> Subject: [openstack-dev] [Neutron][QA]Request for input for Juno Design   
> Summit, Atlanta
> 
> 
> 
> Dear fellow Neutron'ers and QA'ers,
> 
> During the Atlanta Design Summit we have been assigned 20 minutes (
> http://junodesignsummit.sched.org/event/48ccd60090740ae80b4d1811b9a61303#.U12EsqbwBPq
> ) to agree on the Tempest testing that will be developed for Neutron
> during the Juno cycle. In order to make the most out of those 20
> minutes, we want to start the conversation ahead of time, so, to the
> extent possible, we concentrate on reaching agreement during the
> Atlanta session. To get the conversation rolling, here's an initial
> list of topics where we, as a community, need to reach consensus:
> 
> 
> * Scenario testing. While during Icehouse we achieved a good
> level of community engagement and coverage in API testing,
> scenarios have received little attention, even though a few
> developers made great contributions. During Juno, we want to
> significantly expand this effort, along the following lines:
> 
> 
> * We are looking for ideas for new scenarios from anyone and
> everyone (dev, qa, automantion, manual, users, etc). There
> are no bad ideas. We need ideas, not necessarily fully
> formed blueprints, though the latter would be even better.
> Don't let constraints (whitebox, multi-host, etc) to refrain
> you from proposing an idea. We will sort through them later.
> Ideas are our initial gap right now.
> * Creation of blueprints for the agreed upon scenarios, so
> potential contributors can volunteer to implement them and
> progress tracking can be accomplished.
> * Creation of a "how to" or "primer" wiki page on how to
> implement Neutron scenario tests
> * Documentation of scenario tests. While api tests are to a
> great extent self explanatory, scenarios are more complex
> and it's not easy for people other than the writers of a
> specific test to understand. We need to improve
> documentation.
> 
> 
> * One solution might be to assign scenario tests owners
> to keep them up to date and well documented
> *
> API tests. The challenge in this area seems to be in:
> 
> 
> * Closing the gaps that might haven been left open during
> Icehouse
> * Adding new tests needed as a consequence of changes and
> evolution of the Neutron API
> * Define an on going process to prevent api tests to become
> outdated or stale
> *
> Nova Networking - Neutron parity sub-project. Are there any specific
> needs in this sub-project that can be covered with Tempest based
> testing?
> * Other Neutron sub-projects. Are there specific needs of other
> Neutron sub-projects that can be covered with Tempest based
> testing?
> 
> 
> 
> This is a list of topics meant to start the conversation on this
> subject. Please feel free to chime in, either in the mailing list or
> at this etherpad page
> https://etherpad.openstack.org/p/TempestAndNeutronJuno
> 
> 
> Thanks in advance for your input
> 
> Miguel Lavalle
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Tempest][qa] : backporting "Test Security Groups Basic Ops" in stable/havana ?

2014-04-14 Thread Yair Fried

Hi Julien
I've tried to backport it in the past, and it was rejected, as it needs quite a 
lot of work.

If you are able to convince everyone otherwise, I will gladly help you with the 
code.

However, I think the better solution would be to somehow create a gate that 
runs the scenarios in the Master branch against Havana, because, AFAIK, most 
those scenarios should work against it.

https://bugs.launchpad.net/tempest/+bug/1277040
https://review.openstack.org/#/q/status:abandoned+project:openstack/tempest+branch:stable/havana+topic:bug/1252620,n,z


- Original Message -
> From: "LELOUP Julien" 
> To: openstack-dev@lists.openstack.org
> Sent: Wednesday, April 2, 2014 12:48:31 PM
> Subject: [openstack-dev] [Tempest][qa] : backporting "Test Security Groups 
> Basic Ops" in stable/havana ?
> 
> Hi everyone,
> 
> I'm interested in having the Tempest scenario "Test Security Groups
> Basic Ops" available in the stable/Havana branch .
> This scenario is nice for acceptance tests and it's running fine with
> an Havana deployment.
> 
> Can  someone in the Stable-maint team can backport it from master to
> stable/Havana ? If it's OK for you of course :)
> 
> Best Regards,
> 
> Julien LELOUP
> julien.lel...@3ds.com
> 
> 
> This email and any attachments are intended solely for the use of the
> individual or entity to whom it is addressed and may be confidential
> and/or privileged.
> 
> If you are not one of the named recipients or have received this
> email in error,
> 
> (i) you should not read, disclose, or copy it,
> 
> (ii) please notify sender of your receipt by reply email and delete
> this email and all attachments,
> 
> (iii) Dassault Systemes does not accept or assume any liability or
> responsibility for any use of or reliance on this email.
> 
> For other languages, go to http://www.3ds.com/terms/email-disclaimer
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [QA] The future of nosetests with Tempest

2014-03-06 Thread Yair Fried


- Original Message - 

> From: "Alexei Kornienko" 
> To: openstack-dev@lists.openstack.org
> Sent: Friday, February 28, 2014 12:43:13 PM
> Subject: Re: [openstack-dev] [QA] The future of nosetests with
> Tempest

> Hi,

> Let me express my concerns on this topic:

> > With some recent changes made to Tempest compatibility with
> 
> > nosetests is going away.
> 
> I think that we should not drop nosetests support from tempest or any
> other project. The problem with testrepository is that it's not
> providing any debugger support at all (and will never provide). It
> also has some issues with proving error traces in human readable
> form and it can be quite hard to find out what is actually broken.

> Because of this I think we should try to avoid any kind of test
> libraries that break compatibility with conventional test runners.

> Our tests should be able to run correctly with nosetests, teststools
> or plain old unittest runner. If for some reason test libraries
> (like testscenarios) doesn't provide support for this we should fix
> this libraries or avoid their usage.

+1
I have the same concern about debugging. It's an essential tool (for me, at 
least) in creating scenario tests. The more complex the test, the harder it is 
to rely on simple log-prints. 

> Regards,
> Alexei Kornienko

> On 02/27/2014 06:36 PM, Frittoli, Andrea (HP Cloud) wrote:

> > This is another example of achieving the same result (exclusion
> > from
> > a
> 
> > list):
> > https://git.openstack.org/cgit/openstack/tripleo-image-elements/tree/element
> > s/tempest/tests2skip.py
> > https://git.openstack.org/cgit/openstack/tripleo-image-elements/tree/element
> > s/tempest/tests2skip.txt
> 

> > andrea
> 

> > -Original Message-
> 
> > From: Matthew Treinish [ mailto:mtrein...@kortar.org ]
> 
> > Sent: 27 February 2014 15:49
> 
> > To: OpenStack Development Mailing List (not for usage questions)
> 
> > Subject: Re: [openstack-dev] [QA] The future of nosetests with
> > Tempest
> 

> > On Tue, Feb 25, 2014 at 07:46:23PM -0600, Matt Riedemann wrote:
> 
> > > On 2/12/2014 1:57 PM, Matthew Treinish wrote:
> > 
> 
> > > > On Wed, Feb 12, 2014 at 11:32:39AM -0700, Matt Riedemann wrote:
> > > 
> > 
> 
> > > > > On 1/17/2014 8:34 AM, Matthew Treinish wrote:
> > > > 
> > > 
> > 
> 
> > > > > > On Fri, Jan 17, 2014 at 08:32:19AM -0500, David Kranz
> > > > > > wrote:
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > On 01/16/2014 10:56 PM, Matthew Treinish wrote:
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > Hi everyone,
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > > > With some recent changes made to Tempest compatibility
> > > > > > > > with
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > nosetests is going away. We've started using newer
> > > > > > > > features
> > > > > > > > that
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > nose just doesn't support. One example of this is that
> > > > > > > > we've
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > started using testscenarios and we're planning to do
> > > > > > > > this
> > > > > > > > in
> > > > > > > > more
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > places moving forward.
> 
> > > > > > > > So at Icehouse-3 I'm planning to push the patch out to
> > > > > > > > remove
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > nosetests from the requirements list and all the
> > > > > > > > workarounds
> > > > > > > > and
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > references to nose will be pulled out of the tree.
> > > > > > > > Tempest
> > > > > > > > will
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > also start raising an unsupported exception when you
> > > > > > > > try
> > > > > > > > to
> > > > > > > > run
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > it with nose so that there isn't any confusion on this
> > > > > > > > moving
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > forward. We talked about doing this at summit briefly
> > > > > > > > and
> > > > > > > > I've
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > brought it up a couple of times before, but I believe
> > > > > > > > it
> > > > > > > > is
> > > > > > > > time
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > to do this now. I feel for tempest to move forward we
> > > > > > > > need
> > > > > > > > to
> > > > > > > > do
> > > > > > > > this
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > now so that there isn't any ambiguity as we add even more features
> > and new
> 
> > types of testing.
> 
> > > > > > > I'm with you up to here.
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > Now, this wi

[openstack-dev] [Neutron] port forwarding from gateway to internal hosts

2014-02-18 Thread Yair Fried

Hi,
What's the status of this BP? - 
https://blueprints.launchpad.net/neutron/+spec/router-port-forwarding
will it be ready for I3?
The BP is approved but the patch is abandoned

Regards
Yair

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Neutron] port forwarding from gateway to internal hosts

2014-02-17 Thread Yair Fried

Hi,
What's the status of this BP? - 
https://blueprints.launchpad.net/neutron/+spec/router-port-forwarding
will it be ready for I3?
The BP is approved but the patch is abandoned

Regards
Yair

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][Neutron][Tempest][Network] Break down NetworkBasicOps to smaller test cases

2014-01-20 Thread Yair Fried

I seem to be unable to convey my point using generalization, so I will give a 
specific example:
I would like to have "update dns server" as an additional network scenario. 
Currently I could add it to the existing module:

1. tests connectivity
2. re-associate floating ip
3. update dns server

In which case, failure to re-associate ip will prevent my test from running, 
even though these are completely unrelated scenarios, and (IMO) we would like 
to get feedback on both of them.

Another way, is to copy the entire network_basic_ops module, remove 
"re-associate floating ip" and add "update dns server". For the obvious reasons 
- this also seems like the wrong way to go.

I am looking for an elegant way to share the code of these scenarios.

Yair

- Original Message -
From: "Salvatore Orlando" 
To: "OpenStack Development Mailing List (not for usage questions)" 

Sent: Monday, January 20, 2014 7:22:22 PM
Subject: Re: [openstack-dev] [qa][Neutron][Tempest][Network] Break down 
NetworkBasicOps to smaller test cases

Yair is probably referring to statistically independent tests, or whatever case 
for which the following is true (P(x) is the probably that a test succeeds): 

P(4|3|2|1) = P(4|1) * P(3|1) * P(2|1) 

This might apply to the tests we are adding to network_basic_ops scenario; 
however it is worth noting that: 

- in some cases the above relationship does not hold. For instance a public 
network connectivity test can hardly succeeds if the private connectivity test 
failed (is that correct? I'm not sure anymore of anything this days!) 
- Sean correctly pointed out that splitting test will cause repeated activities 
which will just make the test run longer without any additional benefit. 

On the other hand, I understand and share the feeling that we are adding too 
much to the same workflow. Would it make sense to identify a few conceptually 
independent workflows, identify one or more advanced network scenarios, and 
keep only internal + public connectivity checks in basic_ops? 

Salvatore 

On 20 January 2014 09:23, Jay Pipes < jaypi...@gmail.com > wrote: 

On Sun, 2014-01-19 at 07:17 -0500, Yair Fried wrote: 
> OK, 
> but considering my pending patch (#3 and #4) 
> what about: 
> 
> #1 -> #2 
> #1 -> #3 
> #1 -> #4 
> 
> instead of 
> 
> #1 -> #2 -> #3 -> #4 
> 
> a failure in #2 will prevent #3 and #4 from running even though they are 
> completely unrelated 

Seems to me, that the above is a logical fault. If a failure in #2 
prevents #3 or #4 from running, then by nature they are related to #2. 

-jay 

___ 
OpenStack-dev mailing list 
OpenStack-dev@lists.openstack.org 
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][Neutron][Tempest][Network] Break down NetworkBasicOps to smaller test cases

2014-01-19 Thread Yair Fried



- Original Message -
> From: "Sean Dague" 
> To: "OpenStack Development Mailing List (not for usage questions)" 
> 
> Sent: Sunday, January 19, 2014 1:53:21 PM
> Subject: Re: [openstack-dev] [qa][Neutron][Tempest][Network] Break down 
> NetworkBasicOps to smaller test cases
> 
> On 01/19/2014 02:06 AM, Yair Fried wrote:
> > MT:"Is your issue here that it's just called basic ops and you
> > don't think that's
> > reflective of what is being tested in that file anymore"
> > 
> > No.
> > My issue is, that the current scenario is, in fact, at least 2
> > separate scenarios:
> > 1. original basic_ops
> > 2. reassociate_floating_ip
> > to which I would like to add (
> > https://review.openstack.org/#/c/55146/ ):
> > 4. check external/internal connectivity
> > 3. update dns
> > 
> > While #2 includes #1 as part of its setup, its failing shouldn't
> > prevent #1 from passing. the obvious solution would be to create
> > separate modules for each test case, but since they all share the
> > same setup sequence, IMO, they should at least share code.
> > Notice that in my patch, #2 still includes #1.
> > 
> > Actually, the more network scenario we get, the more we will need
> > to do something in that direction, since most of the scenarios
> > will require the setup of a VM with a floating-ip to ssh into.
> > 
> > So either we do this, or we move all of this code to
> > scenario.manager which is also becoming very complicated
> 
> If #2 is always supposed to work, then I don't actually understand
> why
> #1 being part of the test or not part of the test is really relevant.
> And being part of the same test saves substantial time.
> 
> If you have tests that do:
>  * A -> B -> C
>  * A -> B -> D -> F
> 
> There really isn't value in a test for A -> B *as long* as you have
> sufficient sign posting to know in the fail logs that A -> B worked
> fine.
> 
> And there are sufficient detriments in making it a separate test,
> because it's just adding time to the runs without actually testing
> anything different.

OK,
but considering my pending patch (#3 and #4)
what about:

#1 -> #2
#1 -> #3
#1 -> #4

instead of 

#1 -> #2 -> #3 -> #4

a failure in #2 will prevent #3 and #4 from running even though they are 
completely unrelated


> 
>   -Sean
> 
> > 
> > Yair
> > 
> > - Original Message -
> > From: "Matthew Treinish" 
> > To: "OpenStack Development Mailing List (not for usage questions)"
> > 
> > Sent: Friday, January 17, 2014 6:17:55 AM
> > Subject: Re: [openstack-dev] [qa][Neutron][Tempest][Network] Break
> > down NetworkBasicOps to smaller test cases
> > 
> > On Wed, Jan 15, 2014 at 11:20:22AM -0500, Yair Fried wrote:
> >> Hi Guys
> >> As Maru pointed out - NetworkBasicOps scenario has grown out of
> >> proportion and is no longer "basic" ops.
> > 
> > Is your issue here that it's just called basic ops and you don't
> > think that's
> > reflective of what is being tested in that file anymore. If that's
> > the case
> > then just change the name.
> > 
> >>
> >> So, I started breaking it down to smaller test cases that can fail
> >> independently.
> > 
> > I'm not convinced this is needed. Some scenarios are going to be
> > very involved
> > and complex. Each scenario tests is designed to simulate real use
> > cases in the
> > cloud, so obviously some of them will be fairly large. The solution
> > for making
> > debugging easier in these cases is to make sure that any failures
> > have clear
> > messages. Also make sure there is plenty of signposting debug log
> > messages so
> > when something goes wrong you know what state the test was in.
> > 
> > If you split things up into smaller individual tests you'll most
> > likely end up
> > making tests that are really aren't scenario tests. They'd be
> > closer to API
> > tests, just using the official clients, which really shouldn't be
> > in the
> > scenario tests.
> > 
> >>
> >> Those test cases share the same setup and tear-down code:
> >> 1. create network resources (and verify them)
> >> 2. create VM with floating IP.
> >>
> >> I see 2 options to manage these resources:
> >> 1. Completely isolated - resources are created and cleaned using
> >> setUp() and tearDown() methods [

Re: [openstack-dev] [qa][Neutron][Tempest][Network] Break down NetworkBasicOps to smaller test cases

2014-01-18 Thread Yair Fried

MT:"Is your issue here that it's just called basic ops and you don't think 
that's
reflective of what is being tested in that file anymore"

No.
My issue is, that the current scenario is, in fact, at least 2 separate 
scenarios:
1. original basic_ops
2. reassociate_floating_ip
to which I would like to add ( https://review.openstack.org/#/c/55146/ ):
4. check external/internal connectivity
3. update dns

While #2 includes #1 as part of its setup, its failing shouldn't prevent #1 
from passing. the obvious solution would be to create separate modules for each 
test case, but since they all share the same setup sequence, IMO, they should 
at least share code.
Notice that in my patch, #2 still includes #1.

Actually, the more network scenario we get, the more we will need to do 
something in that direction, since most of the scenarios will require the setup 
of a VM with a floating-ip to ssh into.

So either we do this, or we move all of this code to scenario.manager which is 
also becoming very complicated

Yair

- Original Message -
From: "Matthew Treinish" 
To: "OpenStack Development Mailing List (not for usage questions)" 

Sent: Friday, January 17, 2014 6:17:55 AM
Subject: Re: [openstack-dev] [qa][Neutron][Tempest][Network] Break down 
NetworkBasicOps to smaller test cases

On Wed, Jan 15, 2014 at 11:20:22AM -0500, Yair Fried wrote:
> Hi Guys
> As Maru pointed out - NetworkBasicOps scenario has grown out of proportion 
> and is no longer "basic" ops.

Is your issue here that it's just called basic ops and you don't think that's
reflective of what is being tested in that file anymore. If that's the case
then just change the name.

> 
> So, I started breaking it down to smaller test cases that can fail 
> independently.

I'm not convinced this is needed. Some scenarios are going to be very involved
and complex. Each scenario tests is designed to simulate real use cases in the
cloud, so obviously some of them will be fairly large. The solution for making
debugging easier in these cases is to make sure that any failures have clear
messages. Also make sure there is plenty of signposting debug log messages so
when something goes wrong you know what state the test was in. 

If you split things up into smaller individual tests you'll most likely end up
making tests that are really aren't scenario tests. They'd be closer to API
tests, just using the official clients, which really shouldn't be in the
scenario tests.

> 
> Those test cases share the same setup and tear-down code:
> 1. create network resources (and verify them)
> 2. create VM with floating IP.
> 
> I see 2 options to manage these resources:
> 1. Completely isolated - resources are created and cleaned using setUp() and 
> tearDown() methods [1]. Moving cleanup to tearDown revealed this bug [2]. 
> Apparently the previous way (with tearDownClass) wasn't as fast). This has 
> the side effect of having expensive resources (ie VMs and floating IPs) 
> created and discarded  multiple times though they are unchanged.
> 
> 2. Shared Resources - Using the idea of (or actually using) Fixtures - use 
> the same resources unless a test case fails, in which case resources are 
> deleted and created by the next test case [3].

If you're doing this and splitting things into smaller tests then it has to be
option 1. Scenarios have to be isolated if there are resources shared between
scenario tests that really is only one scenario and shouldn't be split. In fact
I've been working on a change that fixes the scenario test tearDowns that has 
the
side effect of enforcing this policy.

Also just for the record we've tried doing option 2 in the past, for example
there used to be a tenant-reuse config option. The problem with doing that was
actually tends to cause more non-deterministic failures or adding a not
insignificant wait time to ensure the state is clean when you start the next
test. Which is why we ended up pulling this out of tree. What ends up happening
is that you get leftover state from previous tests and the second test ends up
failing because things aren't in the clean state that the test case assumes. If
you look at some of the oneserver files in the API that is the only place we
still do this in the tempest, and we've had many issues on making that work
reliably. Those tests are in a relatively good place now but those are much
simpler tests. Also between each test setUp has to check and ensure that the
shared server is in the proper state. If it's not then the shared server has to
be rebuilt. This methodology would become far more involved for the scenario
tests where you have to manage more than one shared resource.

> 
> I would like to hear your opinions, and know if anyone has any thoughts or 
> ideas on which direction is best,

[openstack-dev] [qa][Neutron][Tempest][Network] Break down NetworkBasicOps to smaller test cases

2014-01-15 Thread Yair Fried

Hi Guys
As Maru pointed out - NetworkBasicOps scenario has grown out of proportion and 
is no longer "basic" ops.

So, I started breaking it down to smaller test cases that can fail 
independently.

Those test cases share the same setup and tear-down code:
1. create network resources (and verify them)
2. create VM with floating IP.

I see 2 options to manage these resources:
1. Completely isolated - resources are created and cleaned using setUp() and 
tearDown() methods [1]. Moving cleanup to tearDown revealed this bug [2]. 
Apparently the previous way (with tearDownClass) wasn't as fast). This has the 
side effect of having expensive resources (ie VMs and floating IPs) created and 
discarded  multiple times though they are unchanged.

2. Shared Resources - Using the idea of (or actually using) Fixtures - use the 
same resources unless a test case fails, in which case resources are deleted 
and created by the next test case [3].

I would like to hear your opinions, and know if anyone has any thoughts or 
ideas on which direction is best, and why.

Once this is completed, we can move on to other scenarios as well

Regards
Yair

[1] fully isolated - https://review.openstack.org/#/c/66879/
[2] https://bugs.launchpad.net/nova/+bug/1269407/+choose-affected-product
[3] shared resources - https://review.openstack.org/#/c/64622/

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [qa][FWaaS][Neutron]Firewall service disabled on gate

2013-12-29 Thread Yair Fried

Hi,
I'm trying to push a firewall api test [1] and I see it cannot run on the 
current gate.
I was FWaaS is disabled since it broke the gate.
Does anyone knows if this is still an issue?
If so - how do we overcome this?
I would like to do some work on this service (scenarios) and don't want to 
waste time if this is something that cannot be done right now

Thank you
Yair

[1] https://review.openstack.org/64362

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [QA][Neutron] About paramiko's "SSHException: Error reading SSH protocol banner"

2013-12-26 Thread Yair Fried

This might be completely off, in "isolated creds", a private network is created 
for each tenant, while the test already creates its own private tenant network, 
thereby changing the behavior from how it was intended to, and how it is in 
"simple" mode. Could this be related?
I have this patch addressing this - https://review.openstack.org/#/c/63886/
You could try and see if it makes any difference

Yair 

- Original Message -
From: "Salvatore Orlando" 
To: "OpenStack Development Mailing List" 
Sent: Friday, December 27, 2013 2:53:59 AM
Subject: [openstack-dev] [QA][Neutron] About paramiko's "SSHException: Error 
reading SSH protocol banner"



I put together all the patches which we prepared for making parallel testing 
work, and ran a few times 'check experimental' on the gate to see whether it 
worked or not. 


With parallel testing, the only really troubling issue are the scenario tests 
which require to access a VM from a floating IP, and the new patches we've 
squashed together in [1] should address this issue. However, the result is that 
timeouts are still observed but with a different message [2]. 
I'm not really familiar with it, and I've never observed it in local testing. I 
wonder if it just happens to be the usual problem about the target host not 
being reachable, or if it is something specific to paramiko. 


Any hint would be appreciated, since from the logs is appears everything is 
wired properly. 


Salvatore 


[1] https://review.openstack.org/#/c/57420/ 
[2] 
http://logs.openstack.org/20/57420/40/experimental/check-tempest-dsvm-neutron-isolated-parallel/a74bdc8/console.html#_2013-12-26_22_51_31_817
 
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [OpenStack-Infra][qa][Tempest][Network] [Infra]Test for external connectivity

2013-12-25 Thread Yair Fried

Hi, 
I wanted to retrieve the gate's DNS server (or google's 8.8.8.8) in order to 
check external connectivity, but I've been told that I'm running the risk of 
getting the Jenkins gate blacklisted by either DNS server. 
It was also brought to my attention, that testing actual external connectivity 
is not really helpful because my current test actually tests the external 
server's resolution ability and not the actual Openstack instance. 

So Here's my question: 
Infra - suppose that I can get devstack to configure it's own DNS server into 
tempest.conf (from resolve.conf), could I get the gate blacklisted for abusing 
the DNS (either local or google's) 
Neutron/Tempest/Maru - What is there to gain from pinging an external 
address/url? Could pinging the public network's default GW be enough? Please 
take into account the fact that we now have CrossTenant test checking L3 
routing 

Yair 

----- Original Message -

From: "Yair Fried"  
To: openstack-in...@lists.openstack.org 
Sent: Monday, December 23, 2013 5:57:44 PM 
Subject: [OpenStack-Infra] Fwd: [openstack-dev] 
[Openstack][qa][Tempest][Network] [Infra]Test for external connectivity 

- Forwarded Message - 
From: "Yair Fried"  
To: "OpenStack Development Mailing List (not for usage questions)" 

Sent: Sunday, December 22, 2013 10:00:52 PM 
Subject: Re: [openstack-dev] [Openstack][qa][Tempest][Network] [Infra]Test for 
external connectivity 

Hi Guys, 
So, I'm done with my patch: 
https://review.openstack.org/#/c/55146 
1. ping an external IP 
2. ping a url to test the dns 
3. check VM's resolve.conf 
4. change the dns server and recheck resolve.conf 

My issue now (same as before) is that this test cannot be executed (and 
therefore pass) on the gate unless the gate is configured for external 
connectivity. 

A. How do I get the neutron gate to allow it's VM external connectivity (i.e. 
ping 8.8.8.8)? 

Considering Jeremy's comment, I agree that depending on external ip/url 
introduces an unnecessary point of failure. However IMO, using multiple 
addresses is not the best way to proceed , I'd rather test against a local 
node. Therefore: 

B. Can I get the neutron gate to enter it's own (local) DNS server? A local 
url? 
C. I this a change I can push to some project by myself, or do I need someone 
to change this for me (infra?)? 

I would really like your input in this matter, as I am in the final stretch of 
this patch and cannot move any farther by myself 

Regards 
Yair Fried 

- Original Message -

From: "Jeremy Stanley"  
To: openstack-dev@lists.openstack.org 
Sent: Thursday, November 21, 2013 12:17:52 AM 
Subject: Re: [openstack-dev] [Openstack][qa][Tempest][Network] Test for 
external connectivity 

On 2013-11-20 14:07:49 -0800 (-0800), Sean Dague wrote: 
> On 11/18/2013 02:41 AM, Yair Fried wrote: 
> [...] 
> > 2. add fields in tempest.conf for 
> > * external connectivity = False/True 
> > * external ip to test against (ie 8.8.8.8) 
> 
> +1 for #2. In the gate we'll need to think about what that address 
> can / should be. It may be different between different AZs. At this 
> point I'd leave the rest of the options off the table until #2 is 
> working reliably. 
[...] 

Having gone down this path in the past, I suggest the test check for 
no fewer than three addresses, sending several probes to each, and 
be considered successful if at least one gets a response. 
-- 
Jeremy Stanley 

___ 
OpenStack-dev mailing list 
OpenStack-dev@lists.openstack.org 
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev 

___ 
OpenStack-dev mailing list 
OpenStack-dev@lists.openstack.org 
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev 

___ 
OpenStack-Infra mailing list 
openstack-in...@lists.openstack.org 
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Tempest][qa] Adding tags to commit messages

2013-12-24 Thread Yair Fried

- Original Message -

> From: "Masayuki Igawa" 
> To: "OpenStack Development Mailing List (not for usage questions)"
> 
> Sent: Tuesday, December 24, 2013 11:39:46 AM
> Subject: Re: [openstack-dev] [Tempest][qa] Adding tags to commit
> messages

> Hi,

> On Tue, Dec 24, 2013 at 6:20 PM, Yair Fried 
> wrote:
> >
> >
> > - Original Message -

> >> From: "Masayuki Igawa" 
> >> To: "OpenStack Development Mailing List (not for usage questions)"
> >> 
> >> Sent: Tuesday, December 24, 2013 10:42:39 AM
> >> Subject: Re: [openstack-dev] [Tempest][qa] Adding tags to commit
> >> messages
> >>
> >> Hi,
> >>
> >> On Tue, Dec 24, 2013 at 3:47 PM, Yair Fried 
> >> wrote:
> >> > Hi,
> >> > Suggestion: Please consider tagging your Tempest commit messages
> >> > the same way you do your mails in the mailing list
> >> >
> >> > Explanation: Since tempest is a single project testing multiple
> >> > Openstack project we have a very diverse collection of patches
> >> > as
> >> > well as reviewers. Tagging our commit messages will allow us to
> >> > classify patches and thus:
> >> > 1. Allow reviewer to focus on patches related to their area of
> >> > expertise
> >> > 2. Track "trends" in patches - I think we all know that we lack
> >> > in
> >> > Neutron testing for example, but can we assess how many network
> >> > related patches are for awaiting review
> >> > 3. Future automation of flagging "interesting" patches
> >> >
> >> > You can usually tell all of this from reviewing the patch, but
> >> > by
> >> > then - you've spent time on a patch you might not even be
> >> > qualified to review.
> >> > I suggest we tag our patches with, to start with, the components
> >> > we
> >> > are looking to test, and the type of test (sceanrio, api, ...)
> >> > and
> >> > that reviewers should -1 untagged patches.
> >> >
> >> > I think the tagging should be the 2nd line in the message:
> >> >
> >> > ==
> >> > Example commit message
> >> >
> >> > [Neutron][Nova][Network][Scenario]
> >> >
> >> > Explanation of how this scenario tests both Neutron and Nova
> >> > Network performance
> >> >
> >> > Chang-id XXX
> >> > ===
> >> >
> >> > I would like this to start immediately but what do you guys
> >> > think?
> >>
> >> +1
> >>
> >> And, how about do we the tagging about the services in the
> >> subject(1st line)?
> >> For example:
> >> Neutron:Example commit subject
> >>
> >> Because the dashboard of the gerrit shows the subject only now.
> >> I think reviewers can find "interesting" patches easily if the
> >> dashboard shows the tags.
> >
> > I just found out you could query search for messages like this:
> > status:open message:[Neutron]

> Oh, really. Thanks!
> But the query of 'status:open message:[Neutron]' only doesn't work
> for me.
> ===
> Application Error
> Server Error
> Invalid query: status:open message:[Neutron] line 1:20 no viable
> alternative at character '['
> ===
> Is there any way working with the brackets?

Not that I know of - could be a regex sollution? 
for now search without brackets, maybe? 

> Sorry for silly question..

> >
> >> This is not so strong opinion because some scenario tests may have
> >> several services tags.
> >
> > And you have a limited subject length that's already not enough
> > (for me at least)

> I agree.

Let's change the subject length limitation to ignore tags 

> # Maybe, it's too long for the subject if already not enough :)

> >>
> >> --
> >> Masayuki Igawa
> >>
> >> ___
> >> OpenStack-dev mailing list
> >> OpenStack-dev@lists.openstack.org
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>
> >
> > ___
> > OpenStack-dev mailing list
> > OpenStack-dev@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

> --
> Masayuki Igawa

> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Tempest][qa] Adding tags to commit messages

2013-12-24 Thread Yair Fried

Hi,
Assuming this is agreed (is it?) - here's the dilemma:
- Tagging the subject is problematic due to length limit and fails the pep8 gate
- Tagging in the message looses the visibility value

Can we check for the subject length without the tags, or disable this check 
completely?
Yair


- Original Message -
From: "Yair Fried" 
To: "OpenStack Development Mailing List (not for usage questions)" 

Sent: Tuesday, December 24, 2013 11:20:40 AM
Subject: Re: [openstack-dev] [Tempest][qa] Adding tags to commit messages



- Original Message -
> From: "Masayuki Igawa" 
> To: "OpenStack Development Mailing List (not for usage questions)" 
> 
> Sent: Tuesday, December 24, 2013 10:42:39 AM
> Subject: Re: [openstack-dev] [Tempest][qa] Adding tags to commit messages
> 
> Hi,
> 
> On Tue, Dec 24, 2013 at 3:47 PM, Yair Fried 
> wrote:
> > Hi,
> > Suggestion: Please consider tagging your Tempest commit messages
> > the same way you do your mails in the mailing list
> >
> > Explanation: Since tempest is a single project testing multiple
> > Openstack project we have a very diverse collection of patches as
> > well as reviewers. Tagging our commit messages will allow us to
> > classify patches and thus:
> > 1. Allow reviewer to focus on patches related to their area of
> > expertise
> > 2. Track "trends" in patches - I think we all know that we lack in
> > Neutron testing for example, but can we assess how many network
> > related patches are for awaiting review
> > 3. Future automation of flagging "interesting" patches
> >
> > You can usually tell all of this from reviewing the patch, but by
> > then - you've spent time on a patch you might not even be
> > qualified to review.
> > I suggest we tag our patches with, to start with, the components we
> > are looking to test, and the type of test (sceanrio, api, ...) and
> > that reviewers should -1 untagged patches.
> >
> > I think the tagging should be the 2nd line in the message:
> >
> > ==
> > Example commit message
> >
> > [Neutron][Nova][Network][Scenario]
> >
> > Explanation of how this scenario tests both Neutron and Nova
> > Network performance
> >
> > Chang-id XXX
> > ===
> >
> > I would like this to start immediately but what do you guys think?
> 
> +1
> 
> And, how about do we the tagging about the services in the
> subject(1st line)?
> For example:
>   Neutron:Example commit subject
> 
> Because the dashboard of the gerrit shows the subject only now.
> I think reviewers can find "interesting" patches easily if the
> dashboard shows the tags.

I just found out you could query search for messages like this:
status:open message:[Neutron]

> This is not so strong opinion because some scenario tests may have
> several services tags.

And you have a limited subject length that's already not enough (for me at 
least)

> 
> --
> Masayuki Igawa
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Tempest][qa] Adding tags to commit messages

2013-12-24 Thread Yair Fried



- Original Message -
> From: "Masayuki Igawa" 
> To: "OpenStack Development Mailing List (not for usage questions)" 
> 
> Sent: Tuesday, December 24, 2013 10:42:39 AM
> Subject: Re: [openstack-dev] [Tempest][qa] Adding tags to commit messages
> 
> Hi,
> 
> On Tue, Dec 24, 2013 at 3:47 PM, Yair Fried 
> wrote:
> > Hi,
> > Suggestion: Please consider tagging your Tempest commit messages
> > the same way you do your mails in the mailing list
> >
> > Explanation: Since tempest is a single project testing multiple
> > Openstack project we have a very diverse collection of patches as
> > well as reviewers. Tagging our commit messages will allow us to
> > classify patches and thus:
> > 1. Allow reviewer to focus on patches related to their area of
> > expertise
> > 2. Track "trends" in patches - I think we all know that we lack in
> > Neutron testing for example, but can we assess how many network
> > related patches are for awaiting review
> > 3. Future automation of flagging "interesting" patches
> >
> > You can usually tell all of this from reviewing the patch, but by
> > then - you've spent time on a patch you might not even be
> > qualified to review.
> > I suggest we tag our patches with, to start with, the components we
> > are looking to test, and the type of test (sceanrio, api, ...) and
> > that reviewers should -1 untagged patches.
> >
> > I think the tagging should be the 2nd line in the message:
> >
> > ==
> > Example commit message
> >
> > [Neutron][Nova][Network][Scenario]
> >
> > Explanation of how this scenario tests both Neutron and Nova
> > Network performance
> >
> > Chang-id XXX
> > ===
> >
> > I would like this to start immediately but what do you guys think?
> 
> +1
> 
> And, how about do we the tagging about the services in the
> subject(1st line)?
> For example:
>   Neutron:Example commit subject
> 
> Because the dashboard of the gerrit shows the subject only now.
> I think reviewers can find "interesting" patches easily if the
> dashboard shows the tags.

I just found out you could query search for messages like this:
status:open message:[Neutron]

> This is not so strong opinion because some scenario tests may have
> several services tags.

And you have a limited subject length that's already not enough (for me at 
least)

> 
> --
> Masayuki Igawa
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Tempest][qa] Adding tags to commit messages

2013-12-23 Thread Yair Fried

Hi,
Suggestion: Please consider tagging your Tempest commit messages the same way 
you do your mails in the mailing list

Explanation: Since tempest is a single project testing multiple Openstack 
project we have a very diverse collection of patches as well as reviewers. 
Tagging our commit messages will allow us to classify patches and thus:
1. Allow reviewer to focus on patches related to their area of expertise
2. Track "trends" in patches - I think we all know that we lack in Neutron 
testing for example, but can we assess how many network related patches are for 
awaiting review
3. Future automation of flagging "interesting" patches

You can usually tell all of this from reviewing the patch, but by then - you've 
spent time on a patch you might not even be qualified to review.
I suggest we tag our patches with, to start with, the components we are looking 
to test, and the type of test (sceanrio, api, ...) and that reviewers should -1 
untagged patches.

I think the tagging should be the 2nd line in the message:

==
Example commit message

[Neutron][Nova][Network][Scenario]

Explanation of how this scenario tests both Neutron and Nova
Network performance

Chang-id XXX
===

I would like this to start immediately but what do you guys think?

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][qa] test_network_basic_ops and the "FloatingIPChecker" control point

2013-12-23 Thread Yair Fried



- Original Message -
> From: "Brent Eagles" 
> To: "OpenStack Development Mailing List (not for usage questions)" 
> 
> Sent: Monday, December 23, 2013 10:48:50 PM
> Subject: Re: [openstack-dev] [neutron][qa] test_network_basic_ops and the 
> "FloatingIPChecker" control point
> 
> Salvatore Orlando wrote:
> > Before starting this post I confess I did not read with the
> > required level
> > of attention all this thread, so I apologise for any repetition.
> >
> > I just wanted to point out that floating IPs in neutron are created
> > asynchronously when using the l3 agent, and I think this is clear
> > to
> > everybody.
> > So when the create floating IP call returns, this does not mean the
> > floating IP has actually been wired, ie: IP configured on
> > qg-interface and
> > SNAT/DNAT rules added.
> >
> > Unfortunately, neutron lacks a concept of operational status for a
> > floating
> > IP which would tell clients, including nova (it acts as a client
> > wrt nova
> > api), when a floating IP is ready to be used. I started work in
> > this
> > direction, but it has been suspended now for a week. If anybody
> > wants to
> > take over and deems this a reasonable thing to do, it will be
> > great.
> 
> Unless somebody picks it up before I get from the break, I'd like to
> discuss this further with you.
> 
> > I think neutron tests checking connectivity might return more
> > meaningful
> > failure data if they would gather the status of the various
> > components
> > which might impact connectivity.
> > These are:
> > - The floating IP
> > - The router internal interface
> > - The VIF port
> > - The DHCP agent
I wrote something addressing at least some of these points: 
https://review.openstack.org/#/c/55146/
> 
> I agree wholeheartedly. In fact, I think if we are going to rely on
> timeouts for pass/fail we need to do more for post-mortem details.
> 
> > Collecting info about the latter is very important but a bit
> > trickier. I
> > discussed with Sean and Maru that it would be great for a starter,
> > grep the
> > console log to check whether the instance obtained an IP.
> > Other things to consider would be:
> > - adding an operational status to a subnet, which would express
> > whether the
> > DHCP agent is in sync with that subnet (this information won't make
> > sense
> > for subnets with dhcp disabled)
> > - working on a 'debug' administrative API which could return, for
> > instance,
> > for each DHCP agent the list of configured networks and leases.
> 
> Interesting!
> 
> > Regarding timeouts, I think it's fair for tempest to define a
> > timeout and
> > ask that everything from VM boot to Floating IP wiring completes
> > within
> > that timeout.
> >
> > Regards,
> > Salvatore
> 
> I would agree. It would be impossible to have reasonable automated
> testing otherwise.
> 
> Cheers,
> 
> Brent
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack][qa][Tempest][Network] [Infra]Test for external connectivity

2013-12-22 Thread Yair Fried

Hi Guys,
So, I'm done with my patch:
https://review.openstack.org/#/c/55146
1. ping an external IP
2. ping a url to test the dns
3. check VM's resolve.conf
4. change the dns server and recheck resolve.conf

My issue now (same as before) is that this test cannot be executed (and 
therefore pass) on the gate unless the gate is configured for external 
connectivity.

A. How do I get the neutron gate to allow it's VM external connectivity (i.e. 
ping 8.8.8.8)?

Considering Jeremy's comment, I agree that depending on external ip/url 
introduces an unnecessary point of failure. However IMO, using multiple 
addresses is not the best way to proceed , I'd rather test against a local 
node. Therefore:

B. Can I get the neutron gate to enter it's own (local) DNS server? A local url?
C. I this a change I can push to some project by myself, or do I need someone 
to change this for me (infra?)?

I would really like your input in this matter, as I am in the final stretch of 
this patch and cannot move any farther by myself

Regards
Yair Fried

- Original Message -
From: "Jeremy Stanley" 
To: openstack-dev@lists.openstack.org
Sent: Thursday, November 21, 2013 12:17:52 AM
Subject: Re: [openstack-dev] [Openstack][qa][Tempest][Network] Test for 
external connectivity

On 2013-11-20 14:07:49 -0800 (-0800), Sean Dague wrote:
> On 11/18/2013 02:41 AM, Yair Fried wrote:
> [...]
> > 2. add fields in tempest.conf for
> >  * external connectivity = False/True
> >  * external ip to test against (ie 8.8.8.8)
> 
> +1 for #2. In the gate we'll need to think about what that address
> can / should be. It may be different between different AZs. At this
> point I'd leave the rest of the options off the table until #2 is
> working reliably.
[...]

Having gone down this path in the past, I suggest the test check for
no fewer than three addresses, sending several probes to each, and
be considered successful if at least one gets a response.
-- 
Jeremy Stanley

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][qa] test_network_basic_ops and the "FloatingIPChecker" control point

2013-12-19 Thread Yair Fried

I would also like to point out that, since Brent used compute.build_timeout as 
the timeout value
***It takes more time to update FLIP in nova DB, than for a VM to build***

Yair

- Original Message -
From: "Sean Dague" 
To: "OpenStack Development Mailing List (not for usage questions)" 

Sent: Thursday, December 19, 2013 12:42:56 PM
Subject: Re: [openstack-dev] [neutron][qa] test_network_basic_ops and the 
"FloatingIPChecker" control point

On 12/19/2013 03:31 AM, Yair Fried wrote:
> Hi Guys,
> I run into this issue trying to incorporate this test into
> cross_tenant_connectivity scenario:
> launching 2 VMs in different tenants
> What I saw, is that in the gate it fails half the time (the original
> test passes without issues) and ONLY on the 2nd VM (the first FLIP
> propagates fine).
> https://bugs.launchpad.net/nova/+bug/1262529
> 
> I don't see this in:
> 1. my local RHOS-Havana setup
> 2. the cross_tenant_connectivity scenario without the control point
> (test passes without issues)
> 3. test_network_basic_ops runs in the gate
> 
> So here's my somewhat less experienced opinion:
> 1. this happens due to stress (more than a single FLIP/VM)
> 2. (as Brent said) Timeout interval between polling are too short
> 3. FLIP is usually reachable long before it is seen in the nova DB (also
> from manual experience), so blocking the test until it reaches the nova
> DB doesn't make sense for me. if we could do this in different thread,
> then maybe, but using a Pass/Fail criteria to test for a timing issue
> seems wrong. Especially since as I understand it, the issue is on IF it
> reaches nova DB, only WHEN.
> 
> I would like to, at least, move this check from its place as a blocker
> to later in the test. Before this is done, I would like to know if
> anyone else has seen the same problems Brent describes prior to this
> patch being merged.
>
> Regarding Jay's scenario suggestion, I think this should not be a part
> of network_basic_ops, but rather a separate stress scenario creating
> multiple VMs and testing for FLIP associations and propagation time.

+1 there is no need to overload that one scenario. A dedicated one would
be fine.

-Sean

-- 
Sean Dague
Samsung Research America
s...@dague.net / sean.da...@samsung.com
http://dague.net


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][qa] test_network_basic_ops and the "FloatingIPChecker" control point

2013-12-19 Thread Yair Fried

Hi Guys, 
I run into this issue trying to incorporate this test into 
cross_tenant_connectivity scenario: 
launching 2 VMs in different tenants 
What I saw, is that in the gate it fails half the time (the original test 
passes without issues) and ONLY on the 2nd VM (the first FLIP propagates fine). 
https://bugs.launchpad.net/nova/+bug/1262529 

I don't see this in: 
1. my local RHOS-Havana setup 
2. the cross_tenant_connectivity scenario without the control point (test 
passes without issues) 
3. test_network_basic_ops runs in the gate 

So here's my somewhat less experienced opinion: 
1. this happens due to stress (more than a single FLIP/VM) 
2. (as Brent said) Timeout interval between polling are too short 
3. FLIP is usually reachable long before it is seen in the nova DB (also from 
manual experience), so blocking the test until it reaches the nova DB doesn't 
make sense for me. if we could do this in different thread, then maybe, but 
using a Pass/Fail criteria to test for a timing issue seems wrong. Especially 
since as I understand it, the issue is on IF it reaches nova DB, only WHEN. 

I would like to, at least, move this check from its place as a blocker to later 
in the test. Before this is done, I would like to know if anyone else has seen 
the same problems Brent describes prior to this patch being merged. 

Regarding Jay's scenario suggestion, I think this should not be a part of 
network_basic_ops, but rather a separate stress scenario creating multiple VMs 
and testing for FLIP associations and propagation time. 

Regards Yair 
(Also added my comments inline) 

- Original Message -

From: "Jay Pipes"  
To: openstack-dev@lists.openstack.org 
Sent: Thursday, December 19, 2013 5:54:29 AM 
Subject: Re: [openstack-dev] [neutron][qa] test_network_basic_ops and the 
"FloatingIPChecker" control point 

On 12/18/2013 10:21 PM, Brent Eagles wrote: 
> Hi, 
> 
> Yair and I were discussing a change that I initiated and was 
> incorporated into the test_network_basic_ops test. It was intended as a 
> configuration control point for floating IP address assignments before 
> actually testing connectivity. The question we were discussing was 
> whether this check was a valid pass/fail criteria for tests like 
> test_network_basic_ops. 
> 
> The initial motivation for the change was that test_network_basic_ops 
> had a less than 50/50 chance of passing in my local environment for 
> whatever reason. After looking at the test, it seemed ridiculous that it 
> should be failing. The problem is that more often than not the data that 
> was available in the logs all pointed to it being set up correctly but 
> the ping test for connectivity was timing out. From the logs it wasn't 
> clear that the test was failing because neutron did not do the right 
> thing, did not do it fast enough, or is something else happening? Of 
> course if I paused the test for a short bit between setup and the checks 
> to manually verify everything the checks always passed. So it's a timing 
> issue right? 
> 

DID anyone else see experience this issue? locally or on the gate? 

> Two things: adding more timeout to a check is as appealing to me as 
> gargling glass AND I was less "annoyed" that the test was failing as I 
> was that it wasn't clear from reading logs what had gone wrong. I tried 
> to find an additional intermediate control point that would "split" 
> failure modes into two categories: neutron is too slow in setting things 
> up and neutron failed to set things up correctly. Granted it still is 
> adding timeout to the test, but if I could find a control point based on 
> "settling" so that if it passed, then there is a good chance that if the 
> next check failed it was because neutron actually screwed up what it was 
> trying to do. 
> 
> Waiting until the query on the nova for the floating IP information 
> seemed a relatively reasonable, if imperfect, "settling" criteria before 
> attempting to connect to the VM. Testing to see if the floating IP 
> assignment gets to the nova instance details is a valid test and, 
> AFAICT, missing from the current tests. However, Yair has the reasonable 
> point that connectivity is often available long before the floating IP 
> appears in the nova results and that it could be considered invalid to 
> use non-network specific criteria as pass/fail for this test. 

But, Tempest is all about functional integration testing. Using a call 
to Nova's server details to determine whether a dependent call to 
Neutron succeeded (setting up the floating IP) is exactly what I think 
Tempest is all about. It's validating that the integration between Nova 
and Neutron is working as expected. 

So, I actually think the assertion on the floating IP address appearing 
(after some timeout/timeout-backoff) is entirely appropriate. 

Blocking the connectivity check until DB is updated doesn't make sense to me, 
since we know FLIP is reachable well before nova DB is updated (this is seen 
als

[openstack-dev] [Openstack][qa][Tempest][Network][Neutron] tenant_networks_reachable

2013-12-04 Thread Yair Fried

Hi
In tempest.conf, Is the parameter "tenant_networks_reachable" affected by 
anything other than neutron using ip name-spaces (which is the default setting) 
or not, and if not using it - if the tempest host is the same as the neutron 
server host?

Regards
Yair Fried

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack][qa][Tempest][Network] Test for external connectivity

2013-11-28 Thread Yair Fried

Thanks for the input. I apologize for the delay.

1. A working patch is here - https://review.openstack.org/#/c/55146. Reviews 
will be much appreciated.
2. Default setting has external_connectivity=False so tempest gate doesn't 
check this. I wonder if we could somehow set the neutron gate to configure 
external access and enable this feature for all tests?
3. Tomoe "How can we test the internal network connectivity?" -- I'm pinging 
router and dhcp ports from the VM via ssh and floating IP, since l2 and l3 
agents might reside on different hosts.
4. Jeremy Stanley - "test check for no fewer than three addresses" -- Why?

- Original Message -
From: "Jeremy Stanley" 
To: openstack-dev@lists.openstack.org
Sent: Thursday, November 21, 2013 12:17:52 AM
Subject: Re: [openstack-dev] [Openstack][qa][Tempest][Network] Test for 
external connectivity

On 2013-11-20 14:07:49 -0800 (-0800), Sean Dague wrote:
> On 11/18/2013 02:41 AM, Yair Fried wrote:
> [...]
> > 2. add fields in tempest.conf for
> >  * external connectivity = False/True
> >  * external ip to test against (ie 8.8.8.8)
> 
> +1 for #2. In the gate we'll need to think about what that address
> can / should be. It may be different between different AZs. At this
> point I'd leave the rest of the options off the table until #2 is
> working reliably.
[...]

Having gone down this path in the past, I suggest the test check for
no fewer than three addresses, sending several probes to each, and
be considered successful if at least one gets a response.
-- 
Jeremy Stanley

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Openstack][qa][Tempest][Network] Test for external connectivity

2013-11-18 Thread Yair Fried

I'm editing tempest/scenario/test_network_basic_ops.py for external 
connectivity as the TODO listed in its docstring.


the test cases are for pinging against external ip and url to test 
connectivity and dns respectivly.
since default deployement (devstack gate) doesn't have external 
connectivity I was thinking on one or all of the following


1. test against public network gateway
2. add fields in tempest.conf for
 * external connectivity = False/True
 * external ip to test against (ie 8.8.8.8)
3. Regarding DNS:
1. assume that external connectivity means dns is also configured
2. activly configure dns on subnet creation (based on tempest.conf

Comments/suggestions will be much appriciated

Regards
Yair Fried

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [QA][Tempest] Use tempest-config for tempest-cli-improvements

Re: [openstack-dev] [rally] Rally boot tests fails with Error Forbidden: It is not allowed to create an interface on external networks

Re: [openstack-dev] [rally] Rally boot tests fails with Error Forbidden: It is not allowed to create an interface on external networks

Re: [openstack-dev] [rally] Weekly meeting

Re: [openstack-dev] [Rally] Improve review process

Re: [openstack-dev] [opentack-dev][meetings] Proposing changes in Rally meetings

Re: [openstack-dev] [qa] Lack of consistency in returning response from tempest clients

Re: [openstack-dev] [Neutron] [LBaaS] LBaaS v2 API syntax additions/changes

Re: [openstack-dev] [qa] The role of an abstract client in tempest

Re: [openstack-dev] [QA] Questions about test policy for scenario test

Re: [openstack-dev] [QA] Tempest Release Naming

Re: [openstack-dev] [Neutron][QA]Request for input for Juno Design Summit, Atlanta

Re: [openstack-dev] [Tempest][qa] : backporting "Test Security Groups Basic Ops" in stable/havana ?

Re: [openstack-dev] [QA] The future of nosetests with Tempest

[openstack-dev] [Neutron] port forwarding from gateway to internal hosts

[openstack-dev] [Neutron] port forwarding from gateway to internal hosts

Re: [openstack-dev] [qa][Neutron][Tempest][Network] Break down NetworkBasicOps to smaller test cases

Re: [openstack-dev] [qa][Neutron][Tempest][Network] Break down NetworkBasicOps to smaller test cases

Re: [openstack-dev] [qa][Neutron][Tempest][Network] Break down NetworkBasicOps to smaller test cases

[openstack-dev] [qa][Neutron][Tempest][Network] Break down NetworkBasicOps to smaller test cases

[openstack-dev] [qa][FWaaS][Neutron]Firewall service disabled on gate

Re: [openstack-dev] [QA][Neutron] About paramiko's "SSHException: Error reading SSH protocol banner"

Re: [openstack-dev] [OpenStack-Infra][qa][Tempest][Network] [Infra]Test for external connectivity

Re: [openstack-dev] [Tempest][qa] Adding tags to commit messages

Re: [openstack-dev] [Tempest][qa] Adding tags to commit messages

Re: [openstack-dev] [Tempest][qa] Adding tags to commit messages

[openstack-dev] [Tempest][qa] Adding tags to commit messages

Re: [openstack-dev] [neutron][qa] test_network_basic_ops and the "FloatingIPChecker" control point

Re: [openstack-dev] [Openstack][qa][Tempest][Network] [Infra]Test for external connectivity

Re: [openstack-dev] [neutron][qa] test_network_basic_ops and the "FloatingIPChecker" control point

Re: [openstack-dev] [neutron][qa] test_network_basic_ops and the "FloatingIPChecker" control point

[openstack-dev] [Openstack][qa][Tempest][Network][Neutron] tenant_networks_reachable

Re: [openstack-dev] [Openstack][qa][Tempest][Network] Test for external connectivity

[openstack-dev] [Openstack][qa][Tempest][Network] Test for external connectivity

34 matches

Site Navigation

Mail list logo

Footer information