Re: [openstack-dev] [puppet][Fuel] Using Native Ruby Client for Openstack Providers

[Rich Megginson]

On 10/22/2015 11:09 PM, Gilles Dubreuil wrote:


On 21/10/15 00:56, Sofer Athlan-Guyot wrote:

Gilles Dubreuil  writes:


On 14/10/15 17:15, Gilles Dubreuil wrote:


On 14/10/15 10:36, Colleen Murphy wrote:


On Tue, Oct 13, 2015 at 6:13 AM, Vladimir Kuklin > wrote:

 Puppetmaster and Fuelers,

 Last week I mentioned that I would like to bring the theme of using
 native ruby OpenStack client and use it within the providers.

 Emilien told me that I had already been late and the decision was
 made that puppet-openstack decided to not work with Aviator based on
 [0]. I went through this thread and did not find any unresolvable
 issues with using Aviator in comparison with potential benefits it
 could have brought up.

 What I saw actually was like that:

 * Pros

 1) It is a native ruby client
 2) We can import it in puppet and use all the power of Ruby
 3) We will not need to have a lot of forks/execs for puppet
 4) You are relying on negotiated and structured output provided by
 API (JSON) instead of introducing workarounds for client output like [1]

 * Cons

 1) Aviator is not actively supported
 2) Aviator does not track all the upstream OpenStack features while
 native OpenStack client does support them
 3) Ruby community is not really interested in OpenStack (this one is
 arguable, I think)

 * Proposed solution

 While I completely understand all the cons against using Aviator
 right now, I see that Pros above are essential enough to change our
 mind and invest our own resources into creating really good
 OpenStack binding in Ruby.
 Some are saying that there is not so big involvement of Ruby into
 OpenStack. But we are actually working with Puppet/Ruby and are
 invloved into community. So why should not we own this by ourselves
 and lead by example here?

 I understand that many of you do already have a lot of things on
 their plate and cannot or would not want to support things like
 additional library when native OpenStack client is working
 reasonably well for you. But if I propose the following scheme to
 get support of native Ruby client for OpenStack:

 1) we (community) have these resources (speaking of the company I am
 working for, we at Mirantis have a set of guys who could be very
 interested in working on Ruby client for OpenStack)
 2) we gradually improve Aviator code base up to the level that it
 eliminates issues that are mentioned in  'Cons' section
 3) we introduce additional set of providers and allow users and
 operators to pick whichever they want
 4) we leave OpenStackClient default one

 Would you support it and allow such code to be merged into upstream
 puppet-openstack modules?


 [0] 
https://groups.google.com/a/puppetlabs.com/forum/#!searchin/puppet-openstack/aviator$20openstackclient/puppet-openstack/GJwDHNAFVYw/ayN4cdg3EW0J
 [1] 
https://github.com/openstack/puppet-swift/blob/master/lib/puppet/provider/swift_ring_builder.rb#L21-L86
 --
 Yours Faithfully,
 Vladimir Kuklin,
 Fuel Library Tech Lead,
 Mirantis, Inc.
 +7 (495) 640-49-04 
 +7 (926) 702-39-68 
 Skype kuklinvv
 35bk3, Vorontsovskaya Str.
 Moscow, Russia,
 www.mirantis.com 
 www.mirantis.ru 
 vkuk...@mirantis.com 


The scale-tipping reason we went with python-openstackclient over the
Aviator library was that at the same time we were trying to switch, we
were also developing keystone v3 features and we could only get those
features from python-openstackclient.

For the first two pros you listed, I'm not convinced they're really
pros. Puppet types and providers are actually extremely well-suited to
shelling out to command-line clients. There are existing, documented
puppet APIs to do it and we get automatic debug output with it. Nearly
every existing type and provider does this. It is not well-suited to
call out to other non-standard ruby libraries because they need to be
added as a dependency somehow, and doing this is not well-established in
puppet. There are basically two options to do this:

  1) Add a gem as a package resource and make sure the package resource
is called before any of the openstack resources. I could see this
working as an opt-in thing, but not as a default, for the same reason we
don't require our users to install pip libraries - there is less
security guarantees from pypi and rubygems than from distro packages,
plus corporate infrastructure may not allow pulling packages from these
types of sources. (I don't see this policy documented anywhere, this was
just something that's been instilled in me since I started working on
this team. If we 

Re: [openstack-dev] [puppet][Fuel] Using Native Ruby Client for Openstack Providers

[Gilles Dubreuil]

On 19/10/15 22:04, Tom Fifield wrote:
> On 13/10/15 21:13, Vladimir Kuklin wrote:
>> Puppetmaster and Fuelers,
>>
>> Last week I mentioned that I would like to bring the theme of using
>> native ruby OpenStack client and use it within the providers.
>>
>> Emilien told me that I had already been late and the decision was made
>> that puppet-openstack decided to not work with Aviator based on [0]. I
>> went through this thread and did not find any unresolvable issues with
>> using Aviator in comparison with potential benefits it could have
>> brought up.
>>
>> What I saw actually was like that:
>>
>> * Pros
>>
>> 1) It is a native ruby client
>> 2) We can import it in puppet and use all the power of Ruby
>> 3) We will not need to have a lot of forks/execs for puppet
>> 4) You are relying on negotiated and structured output provided by API
>> (JSON) instead of introducing workarounds for client output like [1]
>>
>> * Cons
>>
>> 1) Aviator is not actively supported
>> 2) Aviator does not track all the upstream OpenStack features while
>> native OpenStack client does support them
>> 3) Ruby community is not really interested in OpenStack (this one is
>> arguable, I think)
>>
>> * Proposed solution
>>
>> While I completely understand all the cons against using Aviator right
>> now, I see that Pros above are essential enough to change our mind and
>> invest our own resources into creating really good OpenStack binding in
>> Ruby.
>> Some are saying that there is not so big involvement of Ruby into
>> OpenStack. But we are actually working with Puppet/Ruby and are invloved
>> into community. So why should not we own this by ourselves and lead by
>> example here?
>>
>> I understand that many of you do already have a lot of things on their
>> plate and cannot or would not want to support things like additional
>> library when native OpenStack client is working reasonably well for you.
>> But if I propose the following scheme to get support of native Ruby
>> client for OpenStack:
>>
>> 1) we (community) have these resources (speaking of the company I am
>> working for, we at Mirantis have a set of guys who could be very
>> interested in working on Ruby client for OpenStack)
>> 2) we gradually improve Aviator code base up to the level that it
>> eliminates issues that are mentioned in  'Cons' section
>> 3) we introduce additional set of providers and allow users and
>> operators to pick whichever they want
>> 4) we leave OpenStackClient default one
>>
>> Would you support it and allow such code to be merged into upstream
>> puppet-openstack modules?
> 
> Hi,
> 
> I'm just throwing this out there since I didn't see it mentioned in
> either this thread or the linked one on the puppet-openstack ML, but ...
> 
> ... has anyone looked into fog (http://fog.io/ ) ?
> 
> 
> In my experience it generally works, and is updated fairly frequently.
> 
> 

Fog is an interesting and I think very exciting and ambitious project.

Meanwhile I'm not sure it could be a candidate to go along Net/HTTP or
Faraday because it's cloud generic, so quite high level, bringing many
things we don't need at all, unless we could use only the fog/openstack
part.


> 
> Regards,
> 
> 
> Tom
> 
> 
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet][Fuel] Using Native Ruby Client for Openstack Providers

[Gilles Dubreuil]

On 21/10/15 00:56, Sofer Athlan-Guyot wrote:
> Gilles Dubreuil  writes:
> 
>> On 14/10/15 17:15, Gilles Dubreuil wrote:
>>>
>>>
>>> On 14/10/15 10:36, Colleen Murphy wrote:


 On Tue, Oct 13, 2015 at 6:13 AM, Vladimir Kuklin > wrote:

 Puppetmaster and Fuelers,

 Last week I mentioned that I would like to bring the theme of using
 native ruby OpenStack client and use it within the providers.

 Emilien told me that I had already been late and the decision was
 made that puppet-openstack decided to not work with Aviator based on
 [0]. I went through this thread and did not find any unresolvable
 issues with using Aviator in comparison with potential benefits it
 could have brought up.

 What I saw actually was like that:

 * Pros

 1) It is a native ruby client
 2) We can import it in puppet and use all the power of Ruby
 3) We will not need to have a lot of forks/execs for puppet 
 4) You are relying on negotiated and structured output provided by
 API (JSON) instead of introducing workarounds for client output like 
 [1]

 * Cons

 1) Aviator is not actively supported 
 2) Aviator does not track all the upstream OpenStack features while
 native OpenStack client does support them
 3) Ruby community is not really interested in OpenStack (this one is
 arguable, I think)

 * Proposed solution

 While I completely understand all the cons against using Aviator
 right now, I see that Pros above are essential enough to change our
 mind and invest our own resources into creating really good
 OpenStack binding in Ruby.
 Some are saying that there is not so big involvement of Ruby into
 OpenStack. But we are actually working with Puppet/Ruby and are
 invloved into community. So why should not we own this by ourselves
 and lead by example here?

 I understand that many of you do already have a lot of things on
 their plate and cannot or would not want to support things like
 additional library when native OpenStack client is working
 reasonably well for you. But if I propose the following scheme to
 get support of native Ruby client for OpenStack:

 1) we (community) have these resources (speaking of the company I am
 working for, we at Mirantis have a set of guys who could be very
 interested in working on Ruby client for OpenStack)
 2) we gradually improve Aviator code base up to the level that it
 eliminates issues that are mentioned in  'Cons' section
 3) we introduce additional set of providers and allow users and
 operators to pick whichever they want
 4) we leave OpenStackClient default one

 Would you support it and allow such code to be merged into upstream
 puppet-openstack modules?


 [0] 
 https://groups.google.com/a/puppetlabs.com/forum/#!searchin/puppet-openstack/aviator$20openstackclient/puppet-openstack/GJwDHNAFVYw/ayN4cdg3EW0J
 [1] 
 https://github.com/openstack/puppet-swift/blob/master/lib/puppet/provider/swift_ring_builder.rb#L21-L86
 -- 
 Yours Faithfully,
 Vladimir Kuklin,
 Fuel Library Tech Lead,
 Mirantis, Inc.
 +7 (495) 640-49-04 
 +7 (926) 702-39-68 
 Skype kuklinvv
 35bk3, Vorontsovskaya Str.
 Moscow, Russia,
 www.mirantis.com 
 www.mirantis.ru 
 vkuk...@mirantis.com 


 The scale-tipping reason we went with python-openstackclient over the
 Aviator library was that at the same time we were trying to switch, we
 were also developing keystone v3 features and we could only get those
 features from python-openstackclient.

 For the first two pros you listed, I'm not convinced they're really
 pros. Puppet types and providers are actually extremely well-suited to
 shelling out to command-line clients. There are existing, documented
 puppet APIs to do it and we get automatic debug output with it. Nearly
 every existing type and provider does this. It is not well-suited to
 call out to other non-standard ruby libraries because they need to be
 added as a dependency somehow, and doing this is not well-established in
 puppet. There are basically two options to do this:

  1) Add a gem as a package resource and make sure the package resource
 is called before any of the openstack resources. I could see this
 working as an opt-in thing, but not as a 

Re: [openstack-dev] [puppet][Fuel] Using Native Ruby Client for Openstack Providers

[Sofer Athlan-Guyot]

Gilles Dubreuil  writes:

> On 14/10/15 17:15, Gilles Dubreuil wrote:
>> 
>> 
>> On 14/10/15 10:36, Colleen Murphy wrote:
>>>
>>>
>>> On Tue, Oct 13, 2015 at 6:13 AM, Vladimir Kuklin >> > wrote:
>>>
>>> Puppetmaster and Fuelers,
>>>
>>> Last week I mentioned that I would like to bring the theme of using
>>> native ruby OpenStack client and use it within the providers.
>>>
>>> Emilien told me that I had already been late and the decision was
>>> made that puppet-openstack decided to not work with Aviator based on
>>> [0]. I went through this thread and did not find any unresolvable
>>> issues with using Aviator in comparison with potential benefits it
>>> could have brought up.
>>>
>>> What I saw actually was like that:
>>>
>>> * Pros
>>>
>>> 1) It is a native ruby client
>>> 2) We can import it in puppet and use all the power of Ruby
>>> 3) We will not need to have a lot of forks/execs for puppet 
>>> 4) You are relying on negotiated and structured output provided by
>>> API (JSON) instead of introducing workarounds for client output like [1]
>>>
>>> * Cons
>>>
>>> 1) Aviator is not actively supported 
>>> 2) Aviator does not track all the upstream OpenStack features while
>>> native OpenStack client does support them
>>> 3) Ruby community is not really interested in OpenStack (this one is
>>> arguable, I think)
>>>
>>> * Proposed solution
>>>
>>> While I completely understand all the cons against using Aviator
>>> right now, I see that Pros above are essential enough to change our
>>> mind and invest our own resources into creating really good
>>> OpenStack binding in Ruby.
>>> Some are saying that there is not so big involvement of Ruby into
>>> OpenStack. But we are actually working with Puppet/Ruby and are
>>> invloved into community. So why should not we own this by ourselves
>>> and lead by example here?
>>>
>>> I understand that many of you do already have a lot of things on
>>> their plate and cannot or would not want to support things like
>>> additional library when native OpenStack client is working
>>> reasonably well for you. But if I propose the following scheme to
>>> get support of native Ruby client for OpenStack:
>>>
>>> 1) we (community) have these resources (speaking of the company I am
>>> working for, we at Mirantis have a set of guys who could be very
>>> interested in working on Ruby client for OpenStack)
>>> 2) we gradually improve Aviator code base up to the level that it
>>> eliminates issues that are mentioned in  'Cons' section
>>> 3) we introduce additional set of providers and allow users and
>>> operators to pick whichever they want
>>> 4) we leave OpenStackClient default one
>>>
>>> Would you support it and allow such code to be merged into upstream
>>> puppet-openstack modules?
>>>
>>>
>>> [0] 
>>> https://groups.google.com/a/puppetlabs.com/forum/#!searchin/puppet-openstack/aviator$20openstackclient/puppet-openstack/GJwDHNAFVYw/ayN4cdg3EW0J
>>> [1] 
>>> https://github.com/openstack/puppet-swift/blob/master/lib/puppet/provider/swift_ring_builder.rb#L21-L86
>>> -- 
>>> Yours Faithfully,
>>> Vladimir Kuklin,
>>> Fuel Library Tech Lead,
>>> Mirantis, Inc.
>>> +7 (495) 640-49-04 
>>> +7 (926) 702-39-68 
>>> Skype kuklinvv
>>> 35bk3, Vorontsovskaya Str.
>>> Moscow, Russia,
>>> www.mirantis.com 
>>> www.mirantis.ru 
>>> vkuk...@mirantis.com 
>>>
>>>
>>> The scale-tipping reason we went with python-openstackclient over the
>>> Aviator library was that at the same time we were trying to switch, we
>>> were also developing keystone v3 features and we could only get those
>>> features from python-openstackclient.
>>>
>>> For the first two pros you listed, I'm not convinced they're really
>>> pros. Puppet types and providers are actually extremely well-suited to
>>> shelling out to command-line clients. There are existing, documented
>>> puppet APIs to do it and we get automatic debug output with it. Nearly
>>> every existing type and provider does this. It is not well-suited to
>>> call out to other non-standard ruby libraries because they need to be
>>> added as a dependency somehow, and doing this is not well-established in
>>> puppet. There are basically two options to do this:
>>>
>>>  1) Add a gem as a package resource and make sure the package resource
>>> is called before any of the openstack resources. I could see this
>>> working as an opt-in thing, but not as a default, for the same reason we
>>> don't require our users to install pip libraries - there is less
>>> security guarantees from pypi and rubygems than from distro 

Re: [openstack-dev] [puppet][Fuel] Using Native Ruby Client for Openstack Providers

[Tom Fifield]

On 13/10/15 21:13, Vladimir Kuklin wrote:

Puppetmaster and Fuelers,

Last week I mentioned that I would like to bring the theme of using
native ruby OpenStack client and use it within the providers.

Emilien told me that I had already been late and the decision was made
that puppet-openstack decided to not work with Aviator based on [0]. I
went through this thread and did not find any unresolvable issues with
using Aviator in comparison with potential benefits it could have
brought up.

What I saw actually was like that:

* Pros

1) It is a native ruby client
2) We can import it in puppet and use all the power of Ruby
3) We will not need to have a lot of forks/execs for puppet
4) You are relying on negotiated and structured output provided by API
(JSON) instead of introducing workarounds for client output like [1]

* Cons

1) Aviator is not actively supported
2) Aviator does not track all the upstream OpenStack features while
native OpenStack client does support them
3) Ruby community is not really interested in OpenStack (this one is
arguable, I think)

* Proposed solution

While I completely understand all the cons against using Aviator right
now, I see that Pros above are essential enough to change our mind and
invest our own resources into creating really good OpenStack binding in
Ruby.
Some are saying that there is not so big involvement of Ruby into
OpenStack. But we are actually working with Puppet/Ruby and are invloved
into community. So why should not we own this by ourselves and lead by
example here?

I understand that many of you do already have a lot of things on their
plate and cannot or would not want to support things like additional
library when native OpenStack client is working reasonably well for you.
But if I propose the following scheme to get support of native Ruby
client for OpenStack:

1) we (community) have these resources (speaking of the company I am
working for, we at Mirantis have a set of guys who could be very
interested in working on Ruby client for OpenStack)
2) we gradually improve Aviator code base up to the level that it
eliminates issues that are mentioned in  'Cons' section
3) we introduce additional set of providers and allow users and
operators to pick whichever they want
4) we leave OpenStackClient default one

Would you support it and allow such code to be merged into upstream
puppet-openstack modules?


Hi,

I'm just throwing this out there since I didn't see it mentioned in 
either this thread or the linked one on the puppet-openstack ML, but ...


... has anyone looked into fog (http://fog.io/ ) ?


In my experience it generally works, and is updated fairly frequently.



Regards,


Tom




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet][Fuel] Using Native Ruby Client for Openstack Providers

[Gilles Dubreuil]

On 14/10/15 17:15, Gilles Dubreuil wrote:
> 
> 
> On 14/10/15 10:36, Colleen Murphy wrote:
>>
>>
>> On Tue, Oct 13, 2015 at 6:13 AM, Vladimir Kuklin > > wrote:
>>
>> Puppetmaster and Fuelers,
>>
>> Last week I mentioned that I would like to bring the theme of using
>> native ruby OpenStack client and use it within the providers.
>>
>> Emilien told me that I had already been late and the decision was
>> made that puppet-openstack decided to not work with Aviator based on
>> [0]. I went through this thread and did not find any unresolvable
>> issues with using Aviator in comparison with potential benefits it
>> could have brought up.
>>
>> What I saw actually was like that:
>>
>> * Pros
>>
>> 1) It is a native ruby client
>> 2) We can import it in puppet and use all the power of Ruby
>> 3) We will not need to have a lot of forks/execs for puppet 
>> 4) You are relying on negotiated and structured output provided by
>> API (JSON) instead of introducing workarounds for client output like [1]
>>
>> * Cons
>>
>> 1) Aviator is not actively supported 
>> 2) Aviator does not track all the upstream OpenStack features while
>> native OpenStack client does support them
>> 3) Ruby community is not really interested in OpenStack (this one is
>> arguable, I think)
>>
>> * Proposed solution
>>
>> While I completely understand all the cons against using Aviator
>> right now, I see that Pros above are essential enough to change our
>> mind and invest our own resources into creating really good
>> OpenStack binding in Ruby.
>> Some are saying that there is not so big involvement of Ruby into
>> OpenStack. But we are actually working with Puppet/Ruby and are
>> invloved into community. So why should not we own this by ourselves
>> and lead by example here?
>>
>> I understand that many of you do already have a lot of things on
>> their plate and cannot or would not want to support things like
>> additional library when native OpenStack client is working
>> reasonably well for you. But if I propose the following scheme to
>> get support of native Ruby client for OpenStack:
>>
>> 1) we (community) have these resources (speaking of the company I am
>> working for, we at Mirantis have a set of guys who could be very
>> interested in working on Ruby client for OpenStack)
>> 2) we gradually improve Aviator code base up to the level that it
>> eliminates issues that are mentioned in  'Cons' section
>> 3) we introduce additional set of providers and allow users and
>> operators to pick whichever they want
>> 4) we leave OpenStackClient default one
>>
>> Would you support it and allow such code to be merged into upstream
>> puppet-openstack modules?
>>
>>
>> [0] 
>> https://groups.google.com/a/puppetlabs.com/forum/#!searchin/puppet-openstack/aviator$20openstackclient/puppet-openstack/GJwDHNAFVYw/ayN4cdg3EW0J
>> [1] 
>> https://github.com/openstack/puppet-swift/blob/master/lib/puppet/provider/swift_ring_builder.rb#L21-L86
>> -- 
>> Yours Faithfully,
>> Vladimir Kuklin,
>> Fuel Library Tech Lead,
>> Mirantis, Inc.
>> +7 (495) 640-49-04 
>> +7 (926) 702-39-68 
>> Skype kuklinvv
>> 35bk3, Vorontsovskaya Str.
>> Moscow, Russia,
>> www.mirantis.com 
>> www.mirantis.ru 
>> vkuk...@mirantis.com 
>>
>>
>> The scale-tipping reason we went with python-openstackclient over the
>> Aviator library was that at the same time we were trying to switch, we
>> were also developing keystone v3 features and we could only get those
>> features from python-openstackclient.
>>
>> For the first two pros you listed, I'm not convinced they're really
>> pros. Puppet types and providers are actually extremely well-suited to
>> shelling out to command-line clients. There are existing, documented
>> puppet APIs to do it and we get automatic debug output with it. Nearly
>> every existing type and provider does this. It is not well-suited to
>> call out to other non-standard ruby libraries because they need to be
>> added as a dependency somehow, and doing this is not well-established in
>> puppet. There are basically two options to do this:
>>
>>  1) Add a gem as a package resource and make sure the package resource
>> is called before any of the openstack resources. I could see this
>> working as an opt-in thing, but not as a default, for the same reason we
>> don't require our users to install pip libraries - there is less
>> security guarantees from pypi and rubygems than from distro packages,
>> plus corporate infrastructure may not allow pulling packages from these
>> types of sources. (I don't see this policy documented 

Re: [openstack-dev] [puppet][Fuel] Using Native Ruby Client for Openstack Providers

[Gilles Dubreuil]

On 14/10/15 10:36, Colleen Murphy wrote:
> 
> 
> On Tue, Oct 13, 2015 at 6:13 AM, Vladimir Kuklin  > wrote:
> 
> Puppetmaster and Fuelers,
> 
> Last week I mentioned that I would like to bring the theme of using
> native ruby OpenStack client and use it within the providers.
> 
> Emilien told me that I had already been late and the decision was
> made that puppet-openstack decided to not work with Aviator based on
> [0]. I went through this thread and did not find any unresolvable
> issues with using Aviator in comparison with potential benefits it
> could have brought up.
> 
> What I saw actually was like that:
> 
> * Pros
> 
> 1) It is a native ruby client
> 2) We can import it in puppet and use all the power of Ruby
> 3) We will not need to have a lot of forks/execs for puppet 
> 4) You are relying on negotiated and structured output provided by
> API (JSON) instead of introducing workarounds for client output like [1]
> 
> * Cons
> 
> 1) Aviator is not actively supported 
> 2) Aviator does not track all the upstream OpenStack features while
> native OpenStack client does support them
> 3) Ruby community is not really interested in OpenStack (this one is
> arguable, I think)
> 
> * Proposed solution
> 
> While I completely understand all the cons against using Aviator
> right now, I see that Pros above are essential enough to change our
> mind and invest our own resources into creating really good
> OpenStack binding in Ruby.
> Some are saying that there is not so big involvement of Ruby into
> OpenStack. But we are actually working with Puppet/Ruby and are
> invloved into community. So why should not we own this by ourselves
> and lead by example here?
> 
> I understand that many of you do already have a lot of things on
> their plate and cannot or would not want to support things like
> additional library when native OpenStack client is working
> reasonably well for you. But if I propose the following scheme to
> get support of native Ruby client for OpenStack:
> 
> 1) we (community) have these resources (speaking of the company I am
> working for, we at Mirantis have a set of guys who could be very
> interested in working on Ruby client for OpenStack)
> 2) we gradually improve Aviator code base up to the level that it
> eliminates issues that are mentioned in  'Cons' section
> 3) we introduce additional set of providers and allow users and
> operators to pick whichever they want
> 4) we leave OpenStackClient default one
> 
> Would you support it and allow such code to be merged into upstream
> puppet-openstack modules?
> 
> 
> [0] 
> https://groups.google.com/a/puppetlabs.com/forum/#!searchin/puppet-openstack/aviator$20openstackclient/puppet-openstack/GJwDHNAFVYw/ayN4cdg3EW0J
> [1] 
> https://github.com/openstack/puppet-swift/blob/master/lib/puppet/provider/swift_ring_builder.rb#L21-L86
> -- 
> Yours Faithfully,
> Vladimir Kuklin,
> Fuel Library Tech Lead,
> Mirantis, Inc.
> +7 (495) 640-49-04 
> +7 (926) 702-39-68 
> Skype kuklinvv
> 35bk3, Vorontsovskaya Str.
> Moscow, Russia,
> www.mirantis.com 
> www.mirantis.ru 
> vkuk...@mirantis.com 
> 
> 
> The scale-tipping reason we went with python-openstackclient over the
> Aviator library was that at the same time we were trying to switch, we
> were also developing keystone v3 features and we could only get those
> features from python-openstackclient.
> 
> For the first two pros you listed, I'm not convinced they're really
> pros. Puppet types and providers are actually extremely well-suited to
> shelling out to command-line clients. There are existing, documented
> puppet APIs to do it and we get automatic debug output with it. Nearly
> every existing type and provider does this. It is not well-suited to
> call out to other non-standard ruby libraries because they need to be
> added as a dependency somehow, and doing this is not well-established in
> puppet. There are basically two options to do this:
> 
>  1) Add a gem as a package resource and make sure the package resource
> is called before any of the openstack resources. I could see this
> working as an opt-in thing, but not as a default, for the same reason we
> don't require our users to install pip libraries - there is less
> security guarantees from pypi and rubygems than from distro packages,
> plus corporate infrastructure may not allow pulling packages from these
> types of sources. (I don't see this policy documented anywhere, this was
> just something that's been instilled in me since I started working on
> this team. If we want to revisit it, formalize 

[openstack-dev] [puppet][Fuel] Using Native Ruby Client for Openstack Providers

[Vladimir Kuklin]

Puppetmaster and Fuelers,

Last week I mentioned that I would like to bring the theme of using native
ruby OpenStack client and use it within the providers.

Emilien told me that I had already been late and the decision was made that
puppet-openstack decided to not work with Aviator based on [0]. I went
through this thread and did not find any unresolvable issues with using
Aviator in comparison with potential benefits it could have brought up.

What I saw actually was like that:

* Pros

1) It is a native ruby client
2) We can import it in puppet and use all the power of Ruby
3) We will not need to have a lot of forks/execs for puppet
4) You are relying on negotiated and structured output provided by API
(JSON) instead of introducing workarounds for client output like [1]

* Cons

1) Aviator is not actively supported
2) Aviator does not track all the upstream OpenStack features while native
OpenStack client does support them
3) Ruby community is not really interested in OpenStack (this one is
arguable, I think)

* Proposed solution

While I completely understand all the cons against using Aviator right now,
I see that Pros above are essential enough to change our mind and invest
our own resources into creating really good OpenStack binding in Ruby.
Some are saying that there is not so big involvement of Ruby into
OpenStack. But we are actually working with Puppet/Ruby and are invloved
into community. So why should not we own this by ourselves and lead by
example here?

I understand that many of you do already have a lot of things on their
plate and cannot or would not want to support things like additional
library when native OpenStack client is working reasonably well for you.
But if I propose the following scheme to get support of native Ruby client
for OpenStack:

1) we (community) have these resources (speaking of the company I am
working for, we at Mirantis have a set of guys who could be very interested
in working on Ruby client for OpenStack)
2) we gradually improve Aviator code base up to the level that it
eliminates issues that are mentioned in  'Cons' section
3) we introduce additional set of providers and allow users and operators
to pick whichever they want
4) we leave OpenStackClient default one

Would you support it and allow such code to be merged into upstream
puppet-openstack modules?


[0]
https://groups.google.com/a/puppetlabs.com/forum/#!searchin/puppet-openstack/aviator$20openstackclient/puppet-openstack/GJwDHNAFVYw/ayN4cdg3EW0J
[1]
https://github.com/openstack/puppet-swift/blob/master/lib/puppet/provider/swift_ring_builder.rb#L21-L86
-- 
Yours Faithfully,
Vladimir Kuklin,
Fuel Library Tech Lead,
Mirantis, Inc.
+7 (495) 640-49-04
+7 (926) 702-39-68
Skype kuklinvv
35bk3, Vorontsovskaya Str.
Moscow, Russia,
www.mirantis.com 
www.mirantis.ru
vkuk...@mirantis.com
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet][Fuel] Using Native Ruby Client for Openstack Providers

[Matt Fischer]

>From a technical point of view, not forking and using a native library
makes total sense. I think it would likely be faster and certainly cleaner
than parsing output. Unfortunately I don't think that we have the resources
to actively maintain the library. I think that's the main blocker for me.

On Tue, Oct 13, 2015 at 7:13 AM, Vladimir Kuklin 
wrote:

> Puppetmaster and Fuelers,
>
> Last week I mentioned that I would like to bring the theme of using native
> ruby OpenStack client and use it within the providers.
>
> Emilien told me that I had already been late and the decision was made
> that puppet-openstack decided to not work with Aviator based on [0]. I went
> through this thread and did not find any unresolvable issues with using
> Aviator in comparison with potential benefits it could have brought up.
>
> What I saw actually was like that:
>
> * Pros
>
> 1) It is a native ruby client
> 2) We can import it in puppet and use all the power of Ruby
> 3) We will not need to have a lot of forks/execs for puppet
> 4) You are relying on negotiated and structured output provided by API
> (JSON) instead of introducing workarounds for client output like [1]
>
> * Cons
>
> 1) Aviator is not actively supported
> 2) Aviator does not track all the upstream OpenStack features while native
> OpenStack client does support them
> 3) Ruby community is not really interested in OpenStack (this one is
> arguable, I think)
>
> * Proposed solution
>
> While I completely understand all the cons against using Aviator right
> now, I see that Pros above are essential enough to change our mind and
> invest our own resources into creating really good OpenStack binding in
> Ruby.
> Some are saying that there is not so big involvement of Ruby into
> OpenStack. But we are actually working with Puppet/Ruby and are invloved
> into community. So why should not we own this by ourselves and lead by
> example here?
>
> I understand that many of you do already have a lot of things on their
> plate and cannot or would not want to support things like additional
> library when native OpenStack client is working reasonably well for you.
> But if I propose the following scheme to get support of native Ruby client
> for OpenStack:
>
> 1) we (community) have these resources (speaking of the company I am
> working for, we at Mirantis have a set of guys who could be very interested
> in working on Ruby client for OpenStack)
> 2) we gradually improve Aviator code base up to the level that it
> eliminates issues that are mentioned in  'Cons' section
> 3) we introduce additional set of providers and allow users and operators
> to pick whichever they want
> 4) we leave OpenStackClient default one
>
> Would you support it and allow such code to be merged into upstream
> puppet-openstack modules?
>
>
> [0]
> https://groups.google.com/a/puppetlabs.com/forum/#!searchin/puppet-openstack/aviator$20openstackclient/puppet-openstack/GJwDHNAFVYw/ayN4cdg3EW0J
> [1]
> https://github.com/openstack/puppet-swift/blob/master/lib/puppet/provider/swift_ring_builder.rb#L21-L86
> --
> Yours Faithfully,
> Vladimir Kuklin,
> Fuel Library Tech Lead,
> Mirantis, Inc.
> +7 (495) 640-49-04
> +7 (926) 702-39-68
> Skype kuklinvv
> 35bk3, Vorontsovskaya Str.
> Moscow, Russia,
> www.mirantis.com 
> www.mirantis.ru
> vkuk...@mirantis.com
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet][Fuel] Using Native Ruby Client for Openstack Providers

[Vladimir Kuklin]

Matt

Thanks for your input. So, I mentioned the following - Fuel guys can
contribute into Ruby client for OpenStack as we are also interested in
making it faster. That's why I asked for support in case we invest
substantial effort (as we do not want to waste our time on things that will
not land into upstream) and asked if the approach that I proposed is OK
with you.

On Tue, Oct 13, 2015 at 6:07 PM, Matt Fischer  wrote:

> From a technical point of view, not forking and using a native library
> makes total sense. I think it would likely be faster and certainly cleaner
> than parsing output. Unfortunately I don't think that we have the resources
> to actively maintain the library. I think that's the main blocker for me.
>
> On Tue, Oct 13, 2015 at 7:13 AM, Vladimir Kuklin 
> wrote:
>
>> Puppetmaster and Fuelers,
>>
>> Last week I mentioned that I would like to bring the theme of using
>> native ruby OpenStack client and use it within the providers.
>>
>> Emilien told me that I had already been late and the decision was made
>> that puppet-openstack decided to not work with Aviator based on [0]. I went
>> through this thread and did not find any unresolvable issues with using
>> Aviator in comparison with potential benefits it could have brought up.
>>
>> What I saw actually was like that:
>>
>> * Pros
>>
>> 1) It is a native ruby client
>> 2) We can import it in puppet and use all the power of Ruby
>> 3) We will not need to have a lot of forks/execs for puppet
>> 4) You are relying on negotiated and structured output provided by API
>> (JSON) instead of introducing workarounds for client output like [1]
>>
>> * Cons
>>
>> 1) Aviator is not actively supported
>> 2) Aviator does not track all the upstream OpenStack features while
>> native OpenStack client does support them
>> 3) Ruby community is not really interested in OpenStack (this one is
>> arguable, I think)
>>
>> * Proposed solution
>>
>> While I completely understand all the cons against using Aviator right
>> now, I see that Pros above are essential enough to change our mind and
>> invest our own resources into creating really good OpenStack binding in
>> Ruby.
>> Some are saying that there is not so big involvement of Ruby into
>> OpenStack. But we are actually working with Puppet/Ruby and are invloved
>> into community. So why should not we own this by ourselves and lead by
>> example here?
>>
>> I understand that many of you do already have a lot of things on their
>> plate and cannot or would not want to support things like additional
>> library when native OpenStack client is working reasonably well for you.
>> But if I propose the following scheme to get support of native Ruby client
>> for OpenStack:
>>
>> 1) we (community) have these resources (speaking of the company I am
>> working for, we at Mirantis have a set of guys who could be very interested
>> in working on Ruby client for OpenStack)
>> 2) we gradually improve Aviator code base up to the level that it
>> eliminates issues that are mentioned in  'Cons' section
>> 3) we introduce additional set of providers and allow users and operators
>> to pick whichever they want
>> 4) we leave OpenStackClient default one
>>
>> Would you support it and allow such code to be merged into upstream
>> puppet-openstack modules?
>>
>>
>> [0]
>> https://groups.google.com/a/puppetlabs.com/forum/#!searchin/puppet-openstack/aviator$20openstackclient/puppet-openstack/GJwDHNAFVYw/ayN4cdg3EW0J
>> [1]
>> https://github.com/openstack/puppet-swift/blob/master/lib/puppet/provider/swift_ring_builder.rb#L21-L86
>> --
>> Yours Faithfully,
>> Vladimir Kuklin,
>> Fuel Library Tech Lead,
>> Mirantis, Inc.
>> +7 (495) 640-49-04
>> +7 (926) 702-39-68
>> Skype kuklinvv
>> 35bk3, Vorontsovskaya Str.
>> Moscow, Russia,
>> www.mirantis.com 
>> www.mirantis.ru
>> vkuk...@mirantis.com
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
Yours Faithfully,
Vladimir Kuklin,
Fuel Library Tech Lead,
Mirantis, Inc.
+7 (495) 640-49-04
+7 (926) 702-39-68
Skype kuklinvv
35bk3, Vorontsovskaya Str.
Moscow, Russia,
www.mirantis.com 
www.mirantis.ru
vkuk...@mirantis.com
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

Re: [openstack-dev] [puppet][Fuel] Using Native Ruby Client for Openstack Providers

[Vladimir Kuklin]

Rich

Thanks for your feedback - let me comment on a couple of things.

First of all, I do not think we have complete support for any action in
OpenStack client right now - we still need to rely on neutronclient,
glanceclient, etc.

Secondly, regarding Ruby power - this is about any good programming
language, not about Ruby - I can simply mention better exception handling
as you would not need to parse the output and generate your own exceptions
- this makes it easier to support the whole set of providers. As I
mentioned earlier, we do not have perfect exception handling for
intermittent operational issues.

Finally, I understand that you do not see metrics. Although, it seems to me
absolutely obvious that fork/exec is going to be the problem here, that's
OK, I will work on that and come up with quantitative analysis.


On Tue, Oct 13, 2015 at 5:18 PM, Rich Megginson  wrote:

> On 10/13/2015 07:13 AM, Vladimir Kuklin wrote:
>
> Puppetmaster and Fuelers,
>
> Last week I mentioned that I would like to bring the theme of using native
> ruby OpenStack client and use it within the providers.
>
> Emilien told me that I had already been late and the decision was made
> that puppet-openstack decided to not work with Aviator based on [0]. I went
> through this thread and did not find any unresolvable issues with using
> Aviator in comparison with potential benefits it could have brought up.
>
> What I saw actually was like that:
>
> * Pros
>
> 1) It is a native ruby client
> 2) We can import it in puppet and use all the power of Ruby
> 3) We will not need to have a lot of forks/execs for puppet
>
>
> I think 1), 2), and 3) go together - that is, the reason why 1) and 2) are
> good is because of 3) - since aviator is native ruby, there is no need to
> fork/exec.  What other "power of Ruby" is there to be taken advantage of?
>
> As for fork/exec, it remains to be seen that fork/exec is causing a
> performance problem.  Note that you can also run the openstackclient in
> "persistent" mode - that is, use it as a persistent pipe, which will read
> commands from stdin and output to stdout, which should alleviate much if
> not all of any performance problem caused by multiple fork/exec.  We just
> haven't investigated this route yet because it needs to be proven that
> fork/exec causes performance problems.
>
> 4) You are relying on negotiated and structured output provided by API
> (JSON) instead of introducing workarounds for client output like [1]
>
>
> openstackclient can output JSON.
>
>
> * Cons
>
> 1) Aviator is not actively supported
>
>
> This is huge.
>
> 2) Aviator does not track all the upstream OpenStack features while native
> OpenStack client does support them
>
>
> This is also huge.
>
> 3) Ruby community is not really interested in OpenStack (this one is
> arguable, I think)
>
> * Proposed solution
>
> While I completely understand all the cons against using Aviator right
> now, I see that Pros above are essential enough to change our mind and
> invest our own resources into creating really good OpenStack binding in
> Ruby.
>
>
> I'm still not convinced.
>
> Some are saying that there is not so big involvement of Ruby into
> OpenStack. But we are actually working with Puppet/Ruby and are invloved
> into community. So why should not we own this by ourselves and lead by
> example here?
>
>
>
>
>
> I understand that many of you do already have a lot of things on their
> plate and cannot or would not want to support things like additional
> library when native OpenStack client is working reasonably well for you.
> But if I propose the following scheme to get support of native Ruby client
> for OpenStack:
>
> 1) we (community) have these resources (speaking of the company I am
> working for, we at Mirantis have a set of guys who could be very interested
> in working on Ruby client for OpenStack)
> 2) we gradually improve Aviator code base up to the level that it
> eliminates issues that are mentioned in  'Cons' section
> 3) we introduce additional set of providers and allow users and operators
> to pick whichever they want
> 4) we leave OpenStackClient default one
>
> Would you support it and allow such code to be merged into upstream
> puppet-openstack modules?
>
>
> I would be in favor of such a plan if
> https://etherpad.openstack.org/p/puppet-openstack-weekly-meeting-20151013
> questions 0.4.1-0.4.7 could be answered in the affirmative.
>
>
>
> [0]
> https://groups.google.com/a/puppetlabs.com/forum/#!searchin/puppet-openstack/aviator$20openstackclient/puppet-openstack/GJwDHNAFVYw/ayN4cdg3EW0J
> [1]
> https://github.com/openstack/puppet-swift/blob/master/lib/puppet/provider/swift_ring_builder.rb#L21-L86
> --
> Yours Faithfully,
> Vladimir Kuklin,
> Fuel Library Tech Lead,
> Mirantis, Inc.
> +7 (495) 640-49-04
> +7 (926) 702-39-68
> Skype kuklinvv
> 35bk3, Vorontsovskaya Str.
> Moscow, Russia,
> www.mirantis.com 
> www.mirantis.ru
> vkuk...@mirantis.com
>
>
> 

Re: [openstack-dev] [puppet][Fuel] Using Native Ruby Client for Openstack Providers

[Clayton O'Neill]

I agree that ideally, using a native ruby library would be better, but I
also share Matt's concern.  We'd need a commitment from more than one
person to maintain the library if we went that route.

I think the big advantages I see with the ruby client would be:

   - Potentially better performance
   - Faster turn around time for enhancements/bug fixes.  My concern here
   is that we're currently limited by how quickly distros pick up new versions
   of OpenStack Client.


I think if we did end up using a ruby library, we'd also want to make sure
it was not only vendored, but also usable independently, to increase the
audience.

On Tue, Oct 13, 2015 at 8:16 AM, Vladimir Kuklin 
wrote:

> Matt
>
> Thanks for your input. So, I mentioned the following - Fuel guys can
> contribute into Ruby client for OpenStack as we are also interested in
> making it faster. That's why I asked for support in case we invest
> substantial effort (as we do not want to waste our time on things that will
> not land into upstream) and asked if the approach that I proposed is OK
> with you.
>
> On Tue, Oct 13, 2015 at 6:07 PM, Matt Fischer 
> wrote:
>
>> From a technical point of view, not forking and using a native library
>> makes total sense. I think it would likely be faster and certainly cleaner
>> than parsing output. Unfortunately I don't think that we have the resources
>> to actively maintain the library. I think that's the main blocker for me.
>>
>> On Tue, Oct 13, 2015 at 7:13 AM, Vladimir Kuklin 
>> wrote:
>>
>>> Puppetmaster and Fuelers,
>>>
>>> Last week I mentioned that I would like to bring the theme of using
>>> native ruby OpenStack client and use it within the providers.
>>>
>>> Emilien told me that I had already been late and the decision was made
>>> that puppet-openstack decided to not work with Aviator based on [0]. I went
>>> through this thread and did not find any unresolvable issues with using
>>> Aviator in comparison with potential benefits it could have brought up.
>>>
>>> What I saw actually was like that:
>>>
>>> * Pros
>>>
>>> 1) It is a native ruby client
>>> 2) We can import it in puppet and use all the power of Ruby
>>> 3) We will not need to have a lot of forks/execs for puppet
>>> 4) You are relying on negotiated and structured output provided by API
>>> (JSON) instead of introducing workarounds for client output like [1]
>>>
>>> * Cons
>>>
>>> 1) Aviator is not actively supported
>>> 2) Aviator does not track all the upstream OpenStack features while
>>> native OpenStack client does support them
>>> 3) Ruby community is not really interested in OpenStack (this one is
>>> arguable, I think)
>>>
>>> * Proposed solution
>>>
>>> While I completely understand all the cons against using Aviator right
>>> now, I see that Pros above are essential enough to change our mind and
>>> invest our own resources into creating really good OpenStack binding in
>>> Ruby.
>>> Some are saying that there is not so big involvement of Ruby into
>>> OpenStack. But we are actually working with Puppet/Ruby and are invloved
>>> into community. So why should not we own this by ourselves and lead by
>>> example here?
>>>
>>> I understand that many of you do already have a lot of things on their
>>> plate and cannot or would not want to support things like additional
>>> library when native OpenStack client is working reasonably well for you.
>>> But if I propose the following scheme to get support of native Ruby client
>>> for OpenStack:
>>>
>>> 1) we (community) have these resources (speaking of the company I am
>>> working for, we at Mirantis have a set of guys who could be very interested
>>> in working on Ruby client for OpenStack)
>>> 2) we gradually improve Aviator code base up to the level that it
>>> eliminates issues that are mentioned in  'Cons' section
>>> 3) we introduce additional set of providers and allow users and
>>> operators to pick whichever they want
>>> 4) we leave OpenStackClient default one
>>>
>>> Would you support it and allow such code to be merged into upstream
>>> puppet-openstack modules?
>>>
>>>
>>> [0]
>>> https://groups.google.com/a/puppetlabs.com/forum/#!searchin/puppet-openstack/aviator$20openstackclient/puppet-openstack/GJwDHNAFVYw/ayN4cdg3EW0J
>>> [1]
>>> https://github.com/openstack/puppet-swift/blob/master/lib/puppet/provider/swift_ring_builder.rb#L21-L86
>>> --
>>> Yours Faithfully,
>>> Vladimir Kuklin,
>>> Fuel Library Tech Lead,
>>> Mirantis, Inc.
>>> +7 (495) 640-49-04
>>> +7 (926) 702-39-68
>>> Skype kuklinvv
>>> 35bk3, Vorontsovskaya Str.
>>> Moscow, Russia,
>>> www.mirantis.com 
>>> www.mirantis.ru
>>> vkuk...@mirantis.com
>>>
>>>
>>> __
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> 

Re: [openstack-dev] [puppet][Fuel] Using Native Ruby Client for Openstack Providers

[Rich Megginson]

On 10/13/2015 07:13 AM, Vladimir Kuklin wrote:

Puppetmaster and Fuelers,

Last week I mentioned that I would like to bring the theme of using 
native ruby OpenStack client and use it within the providers.


Emilien told me that I had already been late and the decision was made 
that puppet-openstack decided to not work with Aviator based on [0]. I 
went through this thread and did not find any unresolvable issues with 
using Aviator in comparison with potential benefits it could have 
brought up.


What I saw actually was like that:

* Pros

1) It is a native ruby client
2) We can import it in puppet and use all the power of Ruby
3) We will not need to have a lot of forks/execs for puppet


I think 1), 2), and 3) go together - that is, the reason why 1) and 2) 
are good is because of 3) - since aviator is native ruby, there is no 
need to fork/exec.  What other "power of Ruby" is there to be taken 
advantage of?


As for fork/exec, it remains to be seen that fork/exec is causing a 
performance problem.  Note that you can also run the openstackclient in 
"persistent" mode - that is, use it as a persistent pipe, which will 
read commands from stdin and output to stdout, which should alleviate 
much if not all of any performance problem caused by multiple 
fork/exec.  We just haven't investigated this route yet because it needs 
to be proven that fork/exec causes performance problems.


4) You are relying on negotiated and structured output provided by API 
(JSON) instead of introducing workarounds for client output like [1]


openstackclient can output JSON.



* Cons

1) Aviator is not actively supported


This is huge.

2) Aviator does not track all the upstream OpenStack features while 
native OpenStack client does support them


This is also huge.

3) Ruby community is not really interested in OpenStack (this one is 
arguable, I think)


* Proposed solution

While I completely understand all the cons against using Aviator right 
now, I see that Pros above are essential enough to change our mind and 
invest our own resources into creating really good OpenStack binding 
in Ruby.


I'm still not convinced.

Some are saying that there is not so big involvement of Ruby into 
OpenStack. But we are actually working with Puppet/Ruby and are 
invloved into community. So why should not we own this by ourselves 
and lead by example here?






I understand that many of you do already have a lot of things on their 
plate and cannot or would not want to support things like additional 
library when native OpenStack client is working reasonably well for 
you. But if I propose the following scheme to get support of native 
Ruby client for OpenStack:


1) we (community) have these resources (speaking of the company I am 
working for, we at Mirantis have a set of guys who could be very 
interested in working on Ruby client for OpenStack)
2) we gradually improve Aviator code base up to the level that it 
eliminates issues that are mentioned in  'Cons' section
3) we introduce additional set of providers and allow users and 
operators to pick whichever they want

4) we leave OpenStackClient default one

Would you support it and allow such code to be merged into upstream 
puppet-openstack modules?


I would be in favor of such a plan if 
https://etherpad.openstack.org/p/puppet-openstack-weekly-meeting-20151013 questions 
0.4.1-0.4.7 could be answered in the affirmative.





[0] 
https://groups.google.com/a/puppetlabs.com/forum/#!searchin/puppet-openstack/aviator$20openstackclient/puppet-openstack/GJwDHNAFVYw/ayN4cdg3EW0J 

[1] 
https://github.com/openstack/puppet-swift/blob/master/lib/puppet/provider/swift_ring_builder.rb#L21-L86

--
Yours Faithfully,
Vladimir Kuklin,
Fuel Library Tech Lead,
Mirantis, Inc.
+7 (495) 640-49-04
+7 (926) 702-39-68
Skype kuklinvv
35bk3, Vorontsovskaya Str.
Moscow, Russia,
www.mirantis.com 
www.mirantis.ru 
vkuk...@mirantis.com 


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet][Fuel] Using Native Ruby Client for Openstack Providers

[Rich Megginson]

On 10/13/2015 09:17 AM, Vladimir Kuklin wrote:

Rich

Thanks for your feedback - let me comment on a couple of things.

First of all, I do not think we have complete support for any action 
in OpenStack client right now - we still need to rely on 
neutronclient, glanceclient, etc.


Right.  But those are all being actively maintained, and will have to 
add support for Keystone v3 in order to take advantage of Keystone v3 
features if desired by the clients of those services.




Secondly, regarding Ruby power - this is about any good programming 
language, not about Ruby - I can simply mention better exception 
handling as you would not need to parse the output and generate your 
own exceptions - this makes it easier to support the whole set of 
providers. As I mentioned earlier, we do not have perfect exception 
handling for intermittent operational issues.


"As I mentioned earlier" - not sure to what you are referring here. Can 
you please explain how you could do exception handling better with 
native ruby than with openstackclient output?  I mean, you still have to 
"parse" the return value of the http request, to get the code, the error 
message, and any returned values.




Finally, I understand that you do not see metrics. Although, it seems 
to me absolutely obvious that fork/exec is going to be the problem 
here, that's OK, I will work on that and come up with quantitative 
analysis.


It does appear obvious that getting rid of fork/exec will speed up 
puppet runs.  But it is not obvious how much that speed up will be, and 
it is not obvious about the cost of that vs. the current code, and 
cost/performance vs. using openstackclient in "persistent" mode.





On Tue, Oct 13, 2015 at 5:18 PM, Rich Megginson > wrote:


On 10/13/2015 07:13 AM, Vladimir Kuklin wrote:

Puppetmaster and Fuelers,

Last week I mentioned that I would like to bring the theme of
using native ruby OpenStack client and use it within the providers.

Emilien told me that I had already been late and the decision was
made that puppet-openstack decided to not work with Aviator based
on [0]. I went through this thread and did not find any
unresolvable issues with using Aviator in comparison with
potential benefits it could have brought up.

What I saw actually was like that:

* Pros

1) It is a native ruby client
2) We can import it in puppet and use all the power of Ruby
3) We will not need to have a lot of forks/execs for puppet


I think 1), 2), and 3) go together - that is, the reason why 1)
and 2) are good is because of 3) - since aviator is native ruby,
there is no need to fork/exec. What other "power of Ruby" is there
to be taken advantage of?

As for fork/exec, it remains to be seen that fork/exec is causing
a performance problem.  Note that you can also run the
openstackclient in "persistent" mode - that is, use it as a
persistent pipe, which will read commands from stdin and output to
stdout, which should alleviate much if not all of any performance
problem caused by multiple fork/exec.  We just haven't
investigated this route yet because it needs to be proven that
fork/exec causes performance problems.


4) You are relying on negotiated and structured output provided
by API (JSON) instead of introducing workarounds for client
output like [1]


openstackclient can output JSON.



* Cons

1) Aviator is not actively supported


This is huge.


2) Aviator does not track all the upstream OpenStack features
while native OpenStack client does support them


This is also huge.


3) Ruby community is not really interested in OpenStack (this one
is arguable, I think)

* Proposed solution

While I completely understand all the cons against using Aviator
right now, I see that Pros above are essential enough to change
our mind and invest our own resources into creating really good
OpenStack binding in Ruby.


I'm still not convinced.


Some are saying that there is not so big involvement of Ruby into
OpenStack. But we are actually working with Puppet/Ruby and are
invloved into community. So why should not we own this by
ourselves and lead by example here?






I understand that many of you do already have a lot of things on
their plate and cannot or would not want to support things like
additional library when native OpenStack client is working
reasonably well for you. But if I propose the following scheme to
get support of native Ruby client for OpenStack:

1) we (community) have these resources (speaking of the company I
am working for, we at Mirantis have a set of guys who could be
very interested in working on Ruby client for OpenStack)
2) we gradually improve Aviator code base up to the level that it
eliminates issues that are mentioned in  'Cons' 

Re: [openstack-dev] [puppet][Fuel] Using Native Ruby Client for Openstack Providers

[Rich Megginson]

On 10/13/2015 09:22 AM, Clayton O'Neill wrote:
I agree that ideally, using a native ruby library would be better, but 
I also share Matt's concern.  We'd need a commitment from more than 
one person to maintain the library if we went that route.


I think the big advantages I see with the ruby client would be:

  * Potentially better performance



But how much, and is it worth the cost, and also worth the cost vs. 
using openstackclient in "persistent" mode.



  * Faster turn around time for enhancements/bug fixes.  My concern
here is that we're currently limited by how quickly distros pick
up new versions of OpenStack Client.



IMO this is the biggest problem we have had with using openstackclient - 
being gated by distros, and having to wait months, in some cases, to use 
features supported by the services, which we could have used immediately 
using the API directly.




I think if we did end up using a ruby library, we'd also want to make 
sure it was not only vendored, but also usable independently, to 
increase the audience.


. . . and then are we also going to be gated by the distros in the same 
way, waiting for months to get an update to aviator?




On Tue, Oct 13, 2015 at 8:16 AM, Vladimir Kuklin > wrote:


Matt

Thanks for your input. So, I mentioned the following - Fuel guys
can contribute into Ruby client for OpenStack as we are also
interested in making it faster. That's why I asked for support in
case we invest substantial effort (as we do not want to waste our
time on things that will not land into upstream) and asked if the
approach that I proposed is OK with you.

On Tue, Oct 13, 2015 at 6:07 PM, Matt Fischer
> wrote:

From a technical point of view, not forking and using a native
library makes total sense. I think it would likely be faster
and certainly cleaner than parsing output. Unfortunately I
don't think that we have the resources to actively maintain
the library. I think that's the main blocker for me.

On Tue, Oct 13, 2015 at 7:13 AM, Vladimir Kuklin
> wrote:

Puppetmaster and Fuelers,

Last week I mentioned that I would like to bring the theme
of using native ruby OpenStack client and use it within
the providers.

Emilien told me that I had already been late and the
decision was made that puppet-openstack decided to not
work with Aviator based on [0]. I went through this thread
and did not find any unresolvable issues with using
Aviator in comparison with potential benefits it could
have brought up.

What I saw actually was like that:

* Pros

1) It is a native ruby client
2) We can import it in puppet and use all the power of Ruby
3) We will not need to have a lot of forks/execs for puppet
4) You are relying on negotiated and structured output
provided by API (JSON) instead of introducing workarounds
for client output like [1]

* Cons

1) Aviator is not actively supported
2) Aviator does not track all the upstream OpenStack
features while native OpenStack client does support them
3) Ruby community is not really interested in OpenStack
(this one is arguable, I think)

* Proposed solution

While I completely understand all the cons against using
Aviator right now, I see that Pros above are essential
enough to change our mind and invest our own resources
into creating really good OpenStack binding in Ruby.
Some are saying that there is not so big involvement of
Ruby into OpenStack. But we are actually working with
Puppet/Ruby and are invloved into community. So why should
not we own this by ourselves and lead by example here?

I understand that many of you do already have a lot of
things on their plate and cannot or would not want to
support things like additional library when native
OpenStack client is working reasonably well for you. But
if I propose the following scheme to get support of native
Ruby client for OpenStack:

1) we (community) have these resources (speaking of the
company I am working for, we at Mirantis have a set of
guys who could be very interested in working on Ruby
client for OpenStack)
2) we gradually improve Aviator code base up to the level
that it eliminates issues that are mentioned in  'Cons'
section
3) we introduce additional set of providers 

Re: [openstack-dev] [puppet][Fuel] Using Native Ruby Client for Openstack Providers

[Dean Troyer]

On Tue, Oct 13, 2015 at 10:33 AM, Rich Megginson 
wrote:

> I think if we did end up using a ruby library, we'd also want to make sure
> it was not only vendored, but also usable independently, to increase the
> audience.
>
>
> . . . and then are we also going to be gated by the distros in the same
> way, waiting for months to get an update to aviator?
>

Yes.  Check out the recent thread (again!) about urllib3 vendored inside
Python requests.  Distros will unvendor that for you and you will have
again the same problem in a different place.

Rich, the daemon/persistent mode has been low on my radar for a bit and I
think needs a bit of work to be fully usable the way I think you envision.
Let us know if that becomes a higher priority as we're currently focusing
on fleshing out the in-repo API support and modernizing our auth (switching
to keystoneauth1).

dt

-- 

Dean Troyer
dtro...@gmail.com
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet][Fuel] Using Native Ruby Client for Openstack Providers

[Rich Megginson]

On 10/13/2015 01:49 PM, Dean Troyer wrote:
On Tue, Oct 13, 2015 at 10:33 AM, Rich Megginson > wrote:



I think if we did end up using a ruby library, we'd also want to
make sure it was not only vendored, but also usable
independently, to increase the audience.


. . . and then are we also going to be gated by the distros in the
same way, waiting for months to get an update to aviator?


Yes.  Check out the recent thread (again!) about urllib3 vendored 
inside Python requests.  Distros will unvendor that for you and you 
will have again the same problem in a different place.


Rich, the daemon/persistent mode has been low on my radar for a bit 
and I think needs a bit of work to be fully usable the way I think you 
envision.  Let us know if that becomes a higher priority as we're 
currently focusing on fleshing out the in-repo API support and 
modernizing our auth (switching to keystoneauth1).


Ok.  It's unclear right now when/if this will be needed by puppet.



dt

--

Dean Troyer
dtro...@gmail.com 


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet][Fuel] Using Native Ruby Client for Openstack Providers

[Colleen Murphy]

On Tue, Oct 13, 2015 at 6:13 AM, Vladimir Kuklin 
wrote:

> Puppetmaster and Fuelers,
>
> Last week I mentioned that I would like to bring the theme of using native
> ruby OpenStack client and use it within the providers.
>
> Emilien told me that I had already been late and the decision was made
> that puppet-openstack decided to not work with Aviator based on [0]. I went
> through this thread and did not find any unresolvable issues with using
> Aviator in comparison with potential benefits it could have brought up.
>
> What I saw actually was like that:
>
> * Pros
>
> 1) It is a native ruby client
> 2) We can import it in puppet and use all the power of Ruby
> 3) We will not need to have a lot of forks/execs for puppet
> 4) You are relying on negotiated and structured output provided by API
> (JSON) instead of introducing workarounds for client output like [1]
>
> * Cons
>
> 1) Aviator is not actively supported
> 2) Aviator does not track all the upstream OpenStack features while native
> OpenStack client does support them
> 3) Ruby community is not really interested in OpenStack (this one is
> arguable, I think)
>
> * Proposed solution
>
> While I completely understand all the cons against using Aviator right
> now, I see that Pros above are essential enough to change our mind and
> invest our own resources into creating really good OpenStack binding in
> Ruby.
> Some are saying that there is not so big involvement of Ruby into
> OpenStack. But we are actually working with Puppet/Ruby and are invloved
> into community. So why should not we own this by ourselves and lead by
> example here?
>
> I understand that many of you do already have a lot of things on their
> plate and cannot or would not want to support things like additional
> library when native OpenStack client is working reasonably well for you.
> But if I propose the following scheme to get support of native Ruby client
> for OpenStack:
>
> 1) we (community) have these resources (speaking of the company I am
> working for, we at Mirantis have a set of guys who could be very interested
> in working on Ruby client for OpenStack)
> 2) we gradually improve Aviator code base up to the level that it
> eliminates issues that are mentioned in  'Cons' section
> 3) we introduce additional set of providers and allow users and operators
> to pick whichever they want
> 4) we leave OpenStackClient default one
>
> Would you support it and allow such code to be merged into upstream
> puppet-openstack modules?
>
>
> [0]
> https://groups.google.com/a/puppetlabs.com/forum/#!searchin/puppet-openstack/aviator$20openstackclient/puppet-openstack/GJwDHNAFVYw/ayN4cdg3EW0J
> [1]
> https://github.com/openstack/puppet-swift/blob/master/lib/puppet/provider/swift_ring_builder.rb#L21-L86
> --
> Yours Faithfully,
> Vladimir Kuklin,
> Fuel Library Tech Lead,
> Mirantis, Inc.
> +7 (495) 640-49-04
> +7 (926) 702-39-68
> Skype kuklinvv
> 35bk3, Vorontsovskaya Str.
> Moscow, Russia,
> www.mirantis.com 
> www.mirantis.ru
> vkuk...@mirantis.com
>

The scale-tipping reason we went with python-openstackclient over the
Aviator library was that at the same time we were trying to switch, we were
also developing keystone v3 features and we could only get those features
from python-openstackclient.

For the first two pros you listed, I'm not convinced they're really pros.
Puppet types and providers are actually extremely well-suited to shelling
out to command-line clients. There are existing, documented puppet APIs to
do it and we get automatic debug output with it. Nearly every existing type
and provider does this. It is not well-suited to call out to other
non-standard ruby libraries because they need to be added as a dependency
somehow, and doing this is not well-established in puppet. There are
basically two options to do this:

 1) Add a gem as a package resource and make sure the package resource is
called before any of the openstack resources. I could see this working as
an opt-in thing, but not as a default, for the same reason we don't require
our users to install pip libraries - there is less security guarantees from
pypi and rubygems than from distro packages, plus corporate infrastructure
may not allow pulling packages from these types of sources. (I don't see
this policy documented anywhere, this was just something that's been
instilled in me since I started working on this team. If we want to revisit
it, formalize it, or drop it, we can start a separate thread for that.)

2) Vendor the gem as a module. This was how we tried it before, and you can
see aimonb/aviator  for this. The
problems I see with this are that we have to keep the module in sync with
the gem, and there is no way to keep proper attribution in the module for
work that was really done in the gem.

I am not convinced that the fork/execs are really causing that much
performance issues, though I've heard from some people