Re: [openstack-dev] [qa] [patrole] Question regarding Patrole API coverage

[MONTEIRO, FELIPE C]

> -Original Message-
> From: Graham Hayes [mailto:g...@ham.ie]
> Sent: Wednesday, November 29, 2017 10:20 AM
> To: openstack-dev@lists.openstack.org
> Subject: Re: [openstack-dev] [qa] [patrole] Question regarding Patrole API
> coverage
> 
> 
> 
> On 29/11/17 15:11, Andrea Frittoli wrote:
> >
> >
> > On Wed, Nov 29, 2017 at 2:28 PM Peng Liu <p...@redhat.com
> > <mailto:p...@redhat.com>> wrote:
> >
> > Hi Team,
> >
> > Hi Peng
> >
> >
> > I have a question regarding to the API coverage of Patrole.
> > Currently, Patrole as a Tempest plugin heavily relys on the Tempest
> > code. However, Tempest only contains the API tests for the most
> > common APIs of the most common projects(Nova, Neutron, Cinder,
> > Glance, Swift, Keystone).
> >
> > So I want to know if it is possible to extend Patrole to:
> > 1) test the APIs of the common projects which was not yet covered in
> > Tempest.
> >
> > 2) test projects which was not covered in Tempest?
> >
> >
> > You can use the Patrole framework to test services not covered by
> > Tempest by taking advantage of Tempest plugin mechanism.
> > Patrole itself is a Tempest plugin. If you install the plugin of a
> > service that includes a service client, you should be able to use it to
> > write Patrole tests for that service.
> > I believe this has not been done yet by any project though, so there may
> > be a few technical bits to be sorted out.
> 
> There has been a start with Designate + Neutron [1], it is just underway
> now though.
> 
> It could cause an issue as all of a sudden the internals of a project's
> plugin may need to be a stable interface, which may not be something
> they expect.

Patrole doesn't change its internals overnight, so to speak. We also use 
deprecation notes and typically wait one release cycle before removing the 
feature, communicated again with a release note. That being said, Patrole's 
framework does change more often than Tempest's and will continue to do so for 
some time yet. So any reliance on Patrole will require maintenance. This is why 
Patrole uses 0.x.0 release versioning right now. 

> 1 - https://review.openstack.org/#/c/520237/
> 
> > I don't think Patrole itself will have to be extended, however Patrole
> > does not yet include stable APIs.
> > If you're going to use Patrole APIs in your project you need to be aware
> > that there may be backward incompatible changes happening without a
> > deprecation period.

The other thing to keep in mind is to ensure that the clients that are being 
used are Tempest-based. I've seen a few Tempest plugins using 
python--client integrations which is problematic, because Patrole 
expects Tempest-based exceptions to be raised.

> >
> > There are several options on where to host such tests: in a dedicated
> > plugin, in the Tempest plugin for the service or in Patrole itself.
> > The latter would probably suffer from the same scalability issues that
> > lead us to create the plugin mechanism to begin with.

There was discussion long ago about a `patrole.lib` that could be its own 
package, but anything like that will not materialize until Patrole is stable, 
pending discussion of course.

> >
> > Andrea Frittoli (andreaf)
> >
> >
> >
> > Thanks,
> > --
> > Peng Liu
> >

Thanks,

Felipe
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa] [patrole] Question regarding Patrole API coverage

[Ghanshyam Mann]

On Thu, Nov 30, 2017 at 12:11 AM, Andrea Frittoli
 wrote:
>
>
> On Wed, Nov 29, 2017 at 2:28 PM Peng Liu  wrote:
>>
>> Hi Team,
>>
> Hi Peng
>
>>
>> I have a question regarding to the API coverage of Patrole. Currently,
>> Patrole as a Tempest plugin heavily relys on the Tempest code. However,
>> Tempest only contains the API tests for the most common APIs of the most
>> common projects(Nova, Neutron, Cinder, Glance, Swift, Keystone).
>>
>> So I want to know if it is possible to extend Patrole to:
>> 1) test the APIs of the common projects which was not yet covered in
>> Tempest.
>>
>> 2) test projects which was not covered in Tempest?

In case there is any gap, Patrole will not tests the API like Tempest
does. It will test only policy enforcement on API. Tempest way to test
API is different than what Patrole does. I hope you mean the same here
which means to extend the coverage of other projects' API policy tests
in Patrole.

>
>
> You can use the Patrole framework to test services not covered by Tempest by
> taking advantage of Tempest plugin mechanism.
> Patrole itself is a Tempest plugin. If you install the plugin of a service
> that includes a service client, you should be able to use it to write
> Patrole tests for that service.
> I believe this has not been done yet by any project though, so there may be
> a few technical bits to be sorted out.
>
> I don't think Patrole itself will have to be extended, however Patrole does
> not yet include stable APIs.
> If you're going to use Patrole APIs in your project you need to be aware
> that there may be backward incompatible changes happening without a
> deprecation period.
>
> There are several options on where to host such tests: in a dedicated
> plugin, in the Tempest plugin for the service or in Patrole itself.
> The latter would probably suffer from the same scalability issues that lead
> us to create the plugin mechanism to begin with.

Yea, I remember we talked about it in some of previous PTG or Summit
also. I feel having Patrole tests for other projects in separate
plugin is not the best way. It will lead to have 2 tempest plugin on
each projects side (one for Tempest and second for Patrole).  I prefer
the solution to have all tests in Patrole repo itself if possible.
Scalability might be issue with this but Patrole test are just policy
test so should not be hard as Tempest tests.
But that is something we need to discussed in detail considering all
cases of maintenance and current resources bandwidth in Patrole.

We have 2 items here to make it possible:
1. Decide the best way to tests Policy for other projects which are
not covered in Patorle. Options are 1. "Separate Patrole Plugin per
projects" 2. "In Projects Tempest Plugin" 3. "In Patrole itself".

2. If we go for Plugin approach then. make Patrole framework stable.

I do not find these 2 items very complex and can be done soon.

-gmann

>
> Andrea Frittoli (andreaf)
>
>>
>>
>> Thanks,
>> --
>> Peng Liu
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa] [patrole] Question regarding Patrole API coverage

[Graham Hayes]

On 29/11/17 15:11, Andrea Frittoli wrote:
> 
> 
> On Wed, Nov 29, 2017 at 2:28 PM Peng Liu  > wrote:
> 
> Hi Team,
> 
> Hi Peng
>  
> 
> I have a question regarding to the API coverage of Patrole.
> Currently, Patrole as a Tempest plugin heavily relys on the Tempest
> code. However, Tempest only contains the API tests for the most
> common APIs of the most common projects(Nova, Neutron, Cinder,
> Glance, Swift, Keystone). 
> 
> So I want to know if it is possible to extend Patrole to:
> 1) test the APIs of the common projects which was not yet covered in
> Tempest.  
> 
> 2) test projects which was not covered in Tempest?
> 
> 
> You can use the Patrole framework to test services not covered by
> Tempest by taking advantage of Tempest plugin mechanism.
> Patrole itself is a Tempest plugin. If you install the plugin of a
> service that includes a service client, you should be able to use it to
> write Patrole tests for that service.
> I believe this has not been done yet by any project though, so there may
> be a few technical bits to be sorted out.

There has been a start with Designate + Neutron [1], it is just underway
now though.

It could cause an issue as all of a sudden the internals of a project's
plugin may need to be a stable interface, which may not be something
they expect.

1 - https://review.openstack.org/#/c/520237/
  
> I don't think Patrole itself will have to be extended, however Patrole
> does not yet include stable APIs.
> If you're going to use Patrole APIs in your project you need to be aware
> that there may be backward incompatible changes happening without a
> deprecation period.
> 
> There are several options on where to host such tests: in a dedicated
> plugin, in the Tempest plugin for the service or in Patrole itself.
> The latter would probably suffer from the same scalability issues that
> lead us to create the plugin mechanism to begin with.
> 
> Andrea Frittoli (andreaf)
>  
> 
> 
> Thanks,
> -- 
> Peng Liu 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> 
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 



signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa] [patrole] Question regarding Patrole API coverage

[Andrea Frittoli]

On Wed, Nov 29, 2017 at 2:28 PM Peng Liu  wrote:

> Hi Team,
>
> Hi Peng


> I have a question regarding to the API coverage of Patrole. Currently,
> Patrole as a Tempest plugin heavily relys on the Tempest code. However,
> Tempest only contains the API tests for the most common APIs of the most
> common projects(Nova, Neutron, Cinder, Glance, Swift, Keystone).
>
> So I want to know if it is possible to extend Patrole to:
> 1) test the APIs of the common projects which was not yet covered in
> Tempest.
>
2) test projects which was not covered in Tempest?
>

You can use the Patrole framework to test services not covered by Tempest
by taking advantage of Tempest plugin mechanism.
Patrole itself is a Tempest plugin. If you install the plugin of a service
that includes a service client, you should be able to use it to write
Patrole tests for that service.
I believe this has not been done yet by any project though, so there may be
a few technical bits to be sorted out.

I don't think Patrole itself will have to be extended, however Patrole does
not yet include stable APIs.
If you're going to use Patrole APIs in your project you need to be aware
that there may be backward incompatible changes happening without a
deprecation period.

There are several options on where to host such tests: in a dedicated
plugin, in the Tempest plugin for the service or in Patrole itself.
The latter would probably suffer from the same scalability issues that lead
us to create the plugin mechanism to begin with.

Andrea Frittoli (andreaf)


>
> Thanks,
> --
> Peng Liu
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [qa] [patrole] Question regarding Patrole API coverage

[Peng Liu]

Hi Team,

I have a question regarding to the API coverage of Patrole. Currently,
Patrole as a Tempest plugin heavily relys on the Tempest code. However,
Tempest only contains the API tests for the most common APIs of the most
common projects(Nova, Neutron, Cinder, Glance, Swift, Keystone).

So I want to know if it is possible to extend Patrole to:
1) test the APIs of the common projects which was not yet covered in
Tempest.
2) test projects which was not covered in Tempest?

Thanks,
-- 
Peng Liu
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev