[openstack-dev] [python-openstacksdk] Meeting minutes, and Next Steps/new meeting time.

[Jesse Noller]

Hi Everyone;

Our first python-openstack meeting was awesome: and I really want to thank 
everyone who came, and for Doug teaching me the meeting bot :)

Minutes:
http://eavesdrop.openstack.org/meetings/python_openstacksdk/2014/python_openstacksdk.2014-02-19-19.01.html
Minutes 
(text):http://eavesdrop.openstack.org/meetings/python_openstacksdk/2014/python_openstacksdk.2014-02-19-19.01.txt
Log:
http://eavesdrop.openstack.org/meetings/python_openstacksdk/2014/python_openstacksdk.2014-02-19-19.01.log.html

Note that coming out of this we will be moving the meetings to Tuesdays, 19:00 
UTC / 1pm CST starting on Tuesday March 4th. Next week there will not be a 
meeting while we discuss and flesh out next steps and requested items (API, 
names, extensions and internal HTTP API).

If you want to participate: please join us on free node: #openstack-sdks 

https://wiki.openstack.org/wiki/PythonOpenStackSDK

Jesse
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [python-openstacksdk] REMINDER: First meeting scheduled

[Jesse Noller]

On Feb 11, 2014, at 2:40 PM, Jesse Noller  wrote:

> As I said last week; we’re ready to kickoff and have regular meetings for the 
> “unified python SDK” project. The initial meeting is scheduled on the wiki:
> 
> https://wiki.openstack.org/wiki/Meetings#python-openstacksdk_Meeting
> 
> Date/Time: Feb. 19th - 19:00 UTC / 1pm CST 
> 
> IRC channel: #openstack-meeting-3
> 
> Meeting Agenda: 
>   https://wiki.openstack.org/wiki/Meetings/PythonOpenStackSDK
> 
> About the project: 
>   https://wiki.openstack.org/wiki/PythonOpenStackSDK
> 
> If you have questions, all of us lurk in #openstack-sdks on freenode!
> 
> See you then.
> 
> Jesse

Reminder: today’s meeting is on!


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Interested in attracting new contributors?

[Jesse Noller]

On Feb 12, 2014, at 8:30 AM, Julie Pichon  wrote:

> Hi folks,
> 
> Stefano's post on how to make contributions to OpenStack easier [1]
> finally stirred me into writing about something that vkmc and myself
> have been doing on the side for a few months to help new contributors
> to get involved.
> 
> Some of you may be aware of OpenHatch [2], a non-profit dedicated to
> helping newcomers get started in open-source. About 6 months ago we
> created a project page for Horizon [3], filled in a few high level
> details, set ourselves up as mentors. Since then people have been
> expressing interest in the project and a number of them got a patch
> submitted and approved, a couple are sticking around (often helping out
> with bug triaging, as confirming new bugs is one of the few tasks one
> can help out with when only having limited time).
> 
> I can definitely sympathise with the comment in Stefano's article that
> there are not enough easy tasks / simple issues for newcomers. There's
> a lot to learn already when you're starting out (git, gerrit, python,
> devstack, ...) and simple bugs are so hard to find - something that
> will take a few minutes to an existing contributor will take much
> longer for someone who's still figuring out where to get the code
> from. Unfortunately it's not uncommon for existing contributors to take
> on tasks marked as "low-hanging-fruit" because it's only 5 minutes (I
> can understand this coming up to an RC but otherwise low-hanging-fruits
> are often low priority nits that could wait a little bit longer). In
> Horizon the low-hanging-fruits definitely get snatched up quickly and I
> try to keep a list of typos or other low impact, trivial bugs that
> would make good first tasks for people reaching out via OpenHatch.
> 
> OpenHatch doesn't spam, you get one email a week if one or more people
> indicated they want to help. The initial effort is not time-consuming,
> following OpenHatch's advice [4] you can refine a nice "initial
> contact" email that helps you get people started and understand what
> they are interested in quickly. I don't find the time commitment to be
> too much so far, and it's incredibly gratifying to see someone
> submitting their first patch after you answered a couple of questions
> or helped resolve a hairy git issue. I'm happy to chat about it more,
> if you're curious or have any questions.
> 
> In any case if you'd like to attract more contributors to your project,
> and/or help newcomers get started in open-source, consider adding your
> project to OpenHatch too!
> 
> Cheers,
> 
> Julie
> 

+10

There’s been quite a bit of talk about this - but not necessarily on the dev 
list. I think openhatch is great - mentorship programs in general go a *long* 
way to help raise up and gain new people. Core Python has had this issue for 
awhile, and many other large OSS projects continue to suffer from it (“barrier 
to entry too high”).

Some random thoughts:

I’d like to see something like Solum’s Contributing page:

https://wiki.openstack.org/wiki/Solum/Contributing

Expanded a little and potentially be the recommended “intro to contribution” 
guide - https://wiki.openstack.org/wiki/How_To_Contribute is good, but a more 
accessible version goes a long way. You want to show them how easy / fast it 
is, not all of the options at once. I think this is somewhere where python-core 
has gotten better at with:

http://docs.python.org/devguide/

It’s not perfect, but it errors on “time to submission/fix"

> [1] http://opensource.com/business/14/2/analyzing-contributions-to-openstack
> [2] http://openhatch.org/
> [3] http://openhatch.org/+projects/OpenStack%20dashboard%20%28Horizon%29
> [4] https://openhatch.org/wiki/Contacting_new_contributors
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [python-openstacksdk] First meeting scheduled

[Jesse Noller]

On Feb 12, 2014, at 9:07 AM, Ed Leafe  wrote:

> On Feb 11, 2014, at 3:30 PM, Jesse Noller  wrote:
> 
>> I did propose in the original thread that we join efforts: in fact, we 
>> already have a fully functioning, unified SDK that *could* be checked in 
>> today - but without discussing the APIs, design and other items with the 
>> community at large, I don’t think that that would be successful.
>> 
>>> [1] 
>>> https://github.com/openstack/oslo-incubator/tree/master/openstack/common/apiclient
> 
> While there are some differences, this library shares the same 
> client/manager/resource class design and flow as the work we've done. IOW, I 
> think the approaches are more alike than different, which is encouraging. 
> 
> 
> -- Ed Leafe

The projects are not exclusive to one another - they are completely 
complimentary. 

And as of yet, we have not solidified the client/manager/resource structure - 
we should talk about that at the meeting next week.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [python-openstacksdk] First meeting scheduled

[Jesse Noller]

On Feb 11, 2014, at 3:02 PM, Boris Pavlovic  wrote:

> Jesse,
> 
> Why not next approach:
> 1) Switch all clients to apiclient[1] (internal unification)

This was discussed several weeks ago here:

http://lists.openstack.org/pipermail/openstack-dev/2014-January/024441.html

This discussion covered everything you mention and more. At that time, it was 
pointed out the blueprint that the internal unification work was being checked 
in against lacked sufficient design, thought towards the end-user (e.g. 
developers). I have spoken to *a lot* of people - across multiple projects and 
teams (and companies) who concur that wrapping the existing clients is 
insufficient and actively *harmful* to the experience of end-users and 
consumers of openstack clouds. More below:

> 2) Unify API of all clients (external unification)
> 3) Unify work with keystone
> 4) Graduate apiclient
> 5) Switch all clients to apiclient lib
> 6) Make one simple plugable mechanism. e.g. subclass based factory + steavdore
> 7) Add one by one subclasse that present client in this factory 
> 8) In one pretty day stop gates & switch to unified client.
> 
> This is actually step by step solution that works for community and could be 
> done independently by tons of developers.
> 

When this last came up, I was Linus’ed:

http://lists.openstack.org/pipermail/openstack-dev/2014-January/024583.html

And not everyone in openstack core is aligned with the common-client-library-2 
work:
https://review.openstack.org/#/q/topic:bp/common-client-library-2,n,z

And it is generally agreed that although “big up-front design” can be a drag, 
the current blueprints and work is confusing and a project like this actually 
requires a deal of design of front to ensure that the API contracts and 
interfaces we expose to users is consistent, usable and not entirely optimized 
for “developers who work on openstack”.

common-client-library-2: 
https://blueprints.launchpad.net/oslo/+spec/common-client-library-2

The blueprint is not a design, or a justification - it’s a todo list and while 
I think code cleanup and reuse is generally a Good Thing, I don’t think 
imposing a series of changes without some coordination and planning across the 
-dev group is going to end up with a good, consistent *end-user* experience. In 
fact, I think the goals are orthogonal - the goals of this BP work seems to be 
to reduce duplicated code; this is an optimization for the OpenStack project.

The python-openstacksdk and python-openstackclient are aimed at a fundamentally 
different (but overlapping set sometimes) audience: developers building 
applications that target openstack deployments, and other end-users. See the 
audience section here:

• Application Developers: Application developers are not OpenStack 
Operators or Developers. These are developers looking to consume a feature-rich 
OpenStack Cloud with its many services. These Developers require a consistent, 
single namespace API ("Application Programming Interface") that allows them to 
build and deploy their application with minimal dependencies.

In a perfect scenario: python-openstackclient’s 1 dependency becomes 
python-openstacksdk which in turn has a dependency list of 1 or two fundamental 
libraries (e.g. requests). 

I did propose in the original thread that we join efforts: in fact, we already 
have a fully functioning, unified SDK that *could* be checked in today - but 
without discussing the APIs, design and other items with the community at 
large, I don’t think that that would be successful.


> 
> [1] 
> https://github.com/openstack/oslo-incubator/tree/master/openstack/common/apiclient
> 
> 
> Best regards, 
> Boris Pavlovic 
> 
> 
> On Wed, Feb 12, 2014 at 12:40 AM, Jesse Noller  
> wrote:
> As I said last week; we’re ready to kickoff and have regular meetings for the 
> “unified python SDK” project. The initial meeting is scheduled on the wiki:
> 
> https://wiki.openstack.org/wiki/Meetings#python-openstacksdk_Meeting
> 
> Date/Time: Feb. 19th - 19:00 UTC / 1pm CST
> 
> IRC channel: #openstack-meeting-3
> 
> Meeting Agenda:
> https://wiki.openstack.org/wiki/Meetings/PythonOpenStackSDK
> 
> About the project:
> https://wiki.openstack.org/wiki/PythonOpenStackSDK
> 
> If you have questions, all of us lurk in #openstack-sdks on freenode!
> 
> See you then.
> 
> Jesse
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [python-openstacksdk] First meeting scheduled

[Jesse Noller]

As I said last week; we’re ready to kickoff and have regular meetings for the 
“unified python SDK” project. The initial meeting is scheduled on the wiki:

https://wiki.openstack.org/wiki/Meetings#python-openstacksdk_Meeting

Date/Time: Feb. 19th - 19:00 UTC / 1pm CST 

IRC channel: #openstack-meeting-3

Meeting Agenda: 
https://wiki.openstack.org/wiki/Meetings/PythonOpenStackSDK

About the project: 
https://wiki.openstack.org/wiki/PythonOpenStackSDK

If you have questions, all of us lurk in #openstack-sdks on freenode!

See you then.

Jesse
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [python-openstacksdk] Initial meeting scheduling

[Jesse Noller]

On Feb 7, 2014, at 5:57 PM, Doug Hellmann  wrote:

> 
> 
> 
> On Fri, Feb 7, 2014 at 6:15 PM, Jesse Noller  
> wrote:
> Hi Everyone:
> Circling back to the client tools / SDK discussion - we’ve gotten a 
> critical mass of people involved/interesting in the unified/common SDK back 
> end for application developers and other end-users. From the wiki page:
> 
> "This is a proposed OpenStack project that is designed to improve the 
> experience of OpenStack end-users by consolidating the various OpenStack 
> python-* client back-ends and common code into a unified, well designed and 
> user focused SDK ("Software Development Toolkit”).”
> 
> Given the interest many have expressed, we feel it’s time to kick off 
> an initial meeting to discuss design, goals and other things that are 
> pertinent prior to all of us running off and coding.
> 
> We have not determined the date/time for the meeting, and have instead set up 
> a doodle to find a time slice that can maximize participation:
> 
> http://doodle.com/qe6v2ssr28x8urqe
> 
> Agenda:
> https://wiki.openstack.org/wiki/Meetings/PythonOpenStackSDK
> 
> The Wiki pages are still in development, but give a high level view of what 
> problem we are trying to solve, and the target audience. Feedback, thoughts, 
> etc are *greatly* welcome. We’re still sorting through mounds of mail threads 
> discussing things and taking things into account.
> 
> Wiki Pages:
> https://wiki.openstack.org/wiki/PythonOpenStackSDK
> https://wiki.openstack.org/wiki/SDK-Development
> 
> Code:
> https://github.com/stackforge/python-openstacksdk
> 
> 
> 
> Can we hold the meetings in one of the regular meeting rooms? That will give 
> us access to the bot to log the meeting notes.
> 
> Doug

Yes - I was trying to find one without conflict - can you help me oh might 
Doug? :|

> 
>  
> Jesse
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [python-openstacksdk] Initial meeting scheduling

[Jesse Noller]

Hi Everyone:
Circling back to the client tools / SDK discussion - we’ve gotten a 
critical mass of people involved/interesting in the unified/common SDK back end 
for application developers and other end-users. From the wiki page:

"This is a proposed OpenStack project that is designed to improve the 
experience of OpenStack end-users by consolidating the various OpenStack 
python-* client back-ends and common code into a unified, well designed and 
user focused SDK ("Software Development Toolkit”).”

Given the interest many have expressed, we feel it’s time to kick off 
an initial meeting to discuss design, goals and other things that are pertinent 
prior to all of us running off and coding. 

We have not determined the date/time for the meeting, and have instead set up a 
doodle to find a time slice that can maximize participation:

http://doodle.com/qe6v2ssr28x8urqe

Agenda:
https://wiki.openstack.org/wiki/Meetings/PythonOpenStackSDK

The Wiki pages are still in development, but give a high level view of what 
problem we are trying to solve, and the target audience. Feedback, thoughts, 
etc are *greatly* welcome. We’re still sorting through mounds of mail threads 
discussing things and taking things into account.

Wiki Pages:
https://wiki.openstack.org/wiki/PythonOpenStackSDK
https://wiki.openstack.org/wiki/SDK-Development

Code:
https://github.com/stackforge/python-openstacksdk


Jesse
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Asynchrounous programming: replace eventlet with asyncio

[Jesse Noller]

On Feb 7, 2014, at 11:12 AM, Chris Behrens  wrote:

> 
> On Feb 7, 2014, at 8:21 AM, Jesse Noller  wrote:
> 
>> It seems that baking concurrency models into the individual clients / 
>> services adds some opinionated choices that may not scale, or fit the needs 
>> of a large-scale deployment. This is one of the things looking at the client 
>> tools I’ve noticed - don’t dictate a concurrency backend, treat it as 
>> producer/consumer/message passing and you end up with something that can 
>> potentially scale out a lot more. 
> 
> I agree, and I think we should do this with our own clients. However, on the 
> service side, there are a lot of 3rd party modules that would need the 
> support as well. libvirt, xenapi, pyamqp, qpid, kombu (sits on pyamqp), etc, 
> come to mind as the top possibilities.
> 
> I was also going to change direction in this reply and say that we should 
> back up and come up with a basic set of requirements. In this thread, I think 
> I’ve only seen arguments against various technology choices without a clear 
> list of our requirements. Since Chuck has posted in the meantime, I’m going 
> to start (what I view) should be some of our requirements in reply to him.
> 
> - Chris
> 

Definitely +1 - we’re talking solutions and technology when the requirements 
aren’t clear - I mean, heck - some might want to use Celery/etc, some might 
want to use cloud queues and other things on the server side. I’d like to help 
get us to a point where we could enable operators to swap in the client side 
and server side things that work best for their topology 
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Asynchrounous programming: replace eventlet with asyncio

[Jesse Noller]

On Feb 7, 2014, at 1:51 AM, Chris Behrens  wrote:

> 
> On Feb 6, 2014, at 11:07 PM, Joshua Harlow  wrote:
> 
>> +1
>> 
>> To give an example as to why eventlet implicit monkey patch the world isn't 
>> especially great (although it's what we are currently using throughout 
>> openstack).
>> 
>> The way I think about how it works is to think about what libraries that a 
>> single piece of code calls and how it is very hard to predict whether that 
>> code will trigger a implicit switch (conceptually similar to a context 
>> switch).
> 
> Conversely, switching to asyncio means that every single module call that 
> would have blocked before monkey patching… will now block. What is worse? :)
> 

Are we perhaps thinking about this in the wrong way?

As I’m looking at the services that make heavy use of eventlet/etc - many of 
them (to me) would benefit more from the typical task queue pattern most SOA 
systems use. At that point your producers + consumers would use a common 
abstracted back end - a simple:

class Reader(object):
 def put():
 def get():

abstraction means that the Reader class could extend from - or be extended - to 
encompass the various models out there - local threads + queue.Queue, asyncIO, 
eventlet / etc. This means that you force everyone into a message 
passing/"shared nothing” architecture where even on a deployment level, a given 
individual could swap in say, twisted, or tornado, or…

It seems that baking concurrency models into the individual clients / services 
adds some opinionated choices that may not scale, or fit the needs of a 
large-scale deployment. This is one of the things looking at the client tools 
I’ve noticed - don’t dictate a concurrency backend, treat it as 
producer/consumer/message passing and you end up with something that can 
potentially scale out a lot more. 

jesse


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Asynchrounous programming: replace eventlet with asyncio

[Jesse Noller]

On Feb 6, 2014, at 2:32 PM, Kevin Conway  wrote:

> There's an incredibly valid reason why we use green thread abstractions like 
> eventlet and gevent in Python. The CPython implementation is inherently 
> single threaded so we need some other form of concurrency to get the most 
> effective use out of our code. You can "import threading" all you want but it 
> won't work the way you expect it to. If you are considering doing anything 
> threading related in Python then http://www.youtube.com/watch?v=Obt-vMVdM8s 
> is absolutely required watching.

As a python core-dev, this isn’t *entirely* correct - or incorrect. It’s 
actually misleading though. But I don’t think -dev is where we need to boil the 
ocean about concurrency in python…

> 
> Green threads give us a powerful way to manage concurrency where it counts: 
> I/O. Everything in openstack is waiting on something else in openstack. That 
> is our natural state of being. If your plan for increasing the number of 
> concurrent requests is "fork more processes" then you're in for a rude 
> awakening when your hosts start kernel panicking from a lack of memory. With 
> green threads, on the other hand, we maintain the use of one process, one 
> thread but are able to manage multiple, concurrent network operations.
> 
> In the case of API nodes: yes, they should (at most) do some db work and drop 
> a message on the queue. That means they almost exclusively deal with I/O. 
> Expecting your wsgi server to scale that up for you is wrong and, in fact, 
> the reason we have eventlet in the first place.
> 
> What's more is this conversation has turned from "lets use asyncio" to "lets 
> make evenltet work with asyncio". If the aim is to convert eventlet to use 
> the asyncio interface then this seems like a great idea so long as it takes 
> place within the eventlet project and not openstack. I don't see the benefit 
> of shimming in asyncio and a fork/backport of asyncio into any of our code 
> bases if the point is to integrate it into a third party module.
> 
> On Thu, Feb 6, 2014 at 12:34 PM, Joshua Harlow  wrote:
> Its a good question, I see openstack as mostly like the following 2 groups of 
> applications.
> 
> Group 1: 
> 
> API entrypoints using [apache/nginx]+wsgi (nova-api, glance-api…)
> 
> In this group we can just let the underlying framework/app deal with the 
> scaling and just use native wsgi as it was intended. Scale more 
> [apache/nginx] if u need more requests per second. For any kind of long term 
> work these apps should be dropping all work to be done on a MQ and letting 
> someone pick that work up to be finished in some future time.
> 
> Group 2:
> 
> Workers that pick things up off MQ. In this area we are allowed to be a 
> little more different and change as we want, but it seems like the simple 
> approach we have been doing is the daemon model (forking N child worker 
> processes). We've also added eventlet in these children (so it becomes more 
> like NxM where M is the number of greenthreads). For the usages where workers 
> are used has it been beneficial to add those M greenthreads? If we just 
> scaled out more N (processes) how bad would it be? (I don't have the answers 
> here actually, but it does make you wonder why we couldn't just eliminate 
> eventlet/asyncio altogether and just use more N processes).
> 
> -Josh
> 
> From: Yuriy Taraday 
> 
> Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
> 
> Date: Thursday, February 6, 2014 at 10:06 AM
> 
> To: "OpenStack Development Mailing List (not for usage questions)" 
> 
> Subject: Re: [openstack-dev] Asynchrounous programming: replace eventlet with 
> asyncio
> 
> Hello.
> 
> 
> On Tue, Feb 4, 2014 at 5:38 PM, victor stinner  
> wrote:
> I would like to replace eventlet with asyncio in OpenStack for the 
> asynchronous programming. The new asyncio module has a better design and is 
> less "magical". It is now part of python 3.4 arguably becoming the de-facto 
> standard for asynchronous programming in Python world.
> 
> I think that before doing this big move to yet another asynchronous framework 
> we should ask the main question: Do we need it? Why do we actually need async 
> framework inside our code?
> There most likely is some historical reason why (almost) every OpenStack 
> project runs every its process with eventlet hub, but I think we should 
> reconsider this now when it's clear that we can't go forward with eventlet 
> (because of py3k mostly) and we're going to put considerable amount of 
> resources into switching to another async framework.
> 
> Let's take Nova for example.
> 
> There are two kinds of processes there: nova-api and others.
> 
> - nova-api process forks to a number of workers listening on one socket and 
> running a single greenthread for each incoming request;
> - other services (workers) constantly poll some queue and spawn a greenthread 
> for each incoming request.
> 
> Both kinds to basically the same job: recei

Re: [openstack-dev] Ugly Hack to deal with multiple versions

[Jesse Noller]

On Feb 4, 2014, at 1:28 PM, Sean Dague  wrote:

> On 02/05/2014 01:50 AM, Jesse Noller wrote:
>> 
>> On Feb 4, 2014, at 10:31 AM, Sean Dague  wrote:
>> 
>>> On 02/05/2014 01:09 AM, Dean Troyer wrote:
>>>> On Tue, Feb 4, 2014 at 9:00 AM, Sean Dague >>> <mailto:s...@dague.net>> wrote:
>>>> 
>>>>   Can you be more specific about what goes wrong here? I'm not entirely
>>>>   sure I understand why an old client of arbitrary age needs to be
>>>>   supported with new OpenStack. The contract is the API, not the client,
>>>>   and an old client that doesn't do version discovery is just a buggy
>>>>   client from what I'm concerned. Time to release a new version.
>>>> 
>>>> 
>>>> Problem 1: API version discovery is not universally considered to be
>>>> part of the API and therefore is not defined by most services beyond
>>>> them responding to a '/' request with a 300 response and a list of
>>>> versions. No two of these responses look alike except where the source
>>>> was copied from an existing service.
>>>> 
>>>> Problem 2: Identity is unique in that it is handed a deployment-defined
>>>> URL to authenticate and get endpoints for all other services.  Most of
>>>> these auth URLs have a version hard-coded in them because the client
>>>> didn't do version discovery or negotiation until recently.  This is what
>>>> we're talking about here, how to remove the version from this URL and
>>>> not break old clients.  We can't.  Not without doing nasty things like
>>>> detecting an old client and compensating for it server-side.  So we have
>>>> to work out a way for new clients to do discovery even when handed a URL
>>>> that has a version in it.
>>>> 
>>>> I've tested a couple of more generalized approaches, and the best
>>>> solution I have found so far is to simply special-case the known legacy
>>>> behaviour then drop in to the general discovery process. 
>>>> 
>>>>   I also wonder if this is an issue with version discovery implementation.
>>>>   It seems like if we think this is going to be affecting multiple
>>>>   services before doing an odd hack for keystone, we should actually
>>>>   figure out a pattern that works for all services, and figure out why
>>>>   this has only just become an issue. Most of the other services have done
>>>> 
>>>> 
>>>> The services that traditionally embed a version inside the URL followed
>>>> by a tenant ID or something get even deeper into parsing the URL to hack
>>>> the version.
>>>> 
>>>>   dual APIs at some point over the last 2 years, and this didn't seem to
>>>>   trip them up too badly. What happened differently in keystone that made
>>>>   this an issue? And what can be learned about how we structure APIs going
>>>>   forward.
>>>> 
>>>> 
>>>> I think the difference is this is the first API we have actually tried
>>>> to deprecate and we don't have the option to hide it in an updated SC
>>>> endpoint.  The service catalog has hidden a lot of this pain for other
>>>> services because the clients generally can use whatever endpoint the SC
>>>> gives it.
>>>> 
>>>> 
>>>> a) Version discovery needs to be rationalized across the services.
>>>> We've talked about this at summits before, and proposals have been
>>>> written.  And here we are.  We'll do it again in Atlanta, hopefully for
>>>> the last time.
>>>> 
>>>> b) Define a common structured endpoint and let the client assemble the
>>>> components into the final URL.  If the service catalog had a base URL
>>>> for compute, and a list of versions, and the additional bits to be
>>>> appended the client could make an intelligent choice and assemble the
>>>> endpoint.  It isn't like the client doesn't already have to know how the
>>>> REST URLs are constructed.
>>>> 
>>>> b-alt) Stop putting things like tenant IDs in the SC.  This has the same
>>>> issue as the auth URL in how to do this without instantly breaking the
>>>> existing clients.
>>> 
>>> Ok, much clearer now to me (though I'll still claim jetlag for some bits
>>> not sinking in).
>>> 
>>>

Re: [openstack-dev] Ugly Hack to deal with multiple versions

[Jesse Noller]

On Feb 4, 2014, at 10:31 AM, Sean Dague  wrote:

> On 02/05/2014 01:09 AM, Dean Troyer wrote:
>> On Tue, Feb 4, 2014 at 9:00 AM, Sean Dague > > wrote:
>> 
>>Can you be more specific about what goes wrong here? I'm not entirely
>>sure I understand why an old client of arbitrary age needs to be
>>supported with new OpenStack. The contract is the API, not the client,
>>and an old client that doesn't do version discovery is just a buggy
>>client from what I'm concerned. Time to release a new version.
>> 
>> 
>> Problem 1: API version discovery is not universally considered to be
>> part of the API and therefore is not defined by most services beyond
>> them responding to a '/' request with a 300 response and a list of
>> versions. No two of these responses look alike except where the source
>> was copied from an existing service.
>> 
>> Problem 2: Identity is unique in that it is handed a deployment-defined
>> URL to authenticate and get endpoints for all other services.  Most of
>> these auth URLs have a version hard-coded in them because the client
>> didn't do version discovery or negotiation until recently.  This is what
>> we're talking about here, how to remove the version from this URL and
>> not break old clients.  We can't.  Not without doing nasty things like
>> detecting an old client and compensating for it server-side.  So we have
>> to work out a way for new clients to do discovery even when handed a URL
>> that has a version in it.
>> 
>> I've tested a couple of more generalized approaches, and the best
>> solution I have found so far is to simply special-case the known legacy
>> behaviour then drop in to the general discovery process. 
>> 
>>I also wonder if this is an issue with version discovery implementation.
>>It seems like if we think this is going to be affecting multiple
>>services before doing an odd hack for keystone, we should actually
>>figure out a pattern that works for all services, and figure out why
>>this has only just become an issue. Most of the other services have done
>> 
>> 
>> The services that traditionally embed a version inside the URL followed
>> by a tenant ID or something get even deeper into parsing the URL to hack
>> the version.
>> 
>>dual APIs at some point over the last 2 years, and this didn't seem to
>>trip them up too badly. What happened differently in keystone that made
>>this an issue? And what can be learned about how we structure APIs going
>>forward.
>> 
>> 
>> I think the difference is this is the first API we have actually tried
>> to deprecate and we don't have the option to hide it in an updated SC
>> endpoint.  The service catalog has hidden a lot of this pain for other
>> services because the clients generally can use whatever endpoint the SC
>> gives it.
>> 
>> 
>> a) Version discovery needs to be rationalized across the services.
>> We've talked about this at summits before, and proposals have been
>> written.  And here we are.  We'll do it again in Atlanta, hopefully for
>> the last time.
>> 
>> b) Define a common structured endpoint and let the client assemble the
>> components into the final URL.  If the service catalog had a base URL
>> for compute, and a list of versions, and the additional bits to be
>> appended the client could make an intelligent choice and assemble the
>> endpoint.  It isn't like the client doesn't already have to know how the
>> REST URLs are constructed.
>> 
>> b-alt) Stop putting things like tenant IDs in the SC.  This has the same
>> issue as the auth URL in how to do this without instantly breaking the
>> existing clients.
> 
> Ok, much clearer now to me (though I'll still claim jetlag for some bits
> not sinking in).
> 
> I think a really important thing to keep in mind is any solution that's
> implemented client side, is something that all the other OpenStack SDKs
> are going to have to implement as well. So an ugly hack isn't just
> python-keystone... and be done. It's also just hoisted doing that ugly
> hack on the php / go sdk teams, jclouds, deltacloud, etc. Something they
> may not be aware is going to break them, or their users.

Do we have official openstack PHP / go SDKs?

> 
> So we really need version discovery rationalized once and for all,
> otherwise this nightmare goes far beyond something that's fixable within
> software that we control.
> 
>   -Sean
> 
> -- 
> Sean Dague
> Samsung Research America
> s...@dague.net / sean.da...@samsung.com
> http://dague.net
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Horizon] Upstream help needed (django-compressor)

[Jesse Noller]

Hi All;

Jannis Leidel, author of Django-Compressor which Horizon relies on recently 
sent out a message saying that he needs help maintaining/releasing 
django_compressor:

https://twitter.com/jezdez/status/423559915660382209

If we have people willing to help upstream dependencies, this would be a great 
place to help out. If you need help getting in contact with him let me know.

Jesse
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Jesse Noller]

On Jan 21, 2014, at 12:59 PM, Alexei Kornienko 
mailto:alexei.kornie...@gmail.com>> wrote:

>It is when most openstack clouds don’t just run keystone. Or nova, or swift. 
>Or when each client acts, smells and behaves differently. It matters a LOT 
>when you’re trying to write applications on top of a mature openstack 
>deployment.

I still don't understand the problem here. Installed packages usually just lie 
quietly on your disk (which size is now measured in terabytes) and they don't 
act or stink unless you ask them to. I'm pretty sure that most of the people 
are not aware of ALL packages installed in their distribution and few packages 
that are not used won't make it any worse


>I have actual experience here as a person running a cloud and supporting 
>end-users & developers: the overwhelming customer feedback (paid and unpaid 
>(exploratory users)) is that the 22+ clients are confusing, difficult to use, 
>prone to error. There are two ways of resolving this if you’re in my shoes - 
>or in a role where the primary focus is not openstack developers or builders; 
>the first is you coordinate work focusing on developer & end user experience 
>upstream, working with openstack and the teams to come up with something that 
>benefits everyone, or, you fork, and build the openstack equivalent to boto / 
>awscli so you can provide a unified “one obvious way to consume openstack 
>clouds”.

I agree that many clients can be confusing however I think that actuall problem 
here is not the clients number but discrepancies in client parameters naming 
and missing/unclear help.
In our work of clients refactoring I will also update bash completition that we 
have for our clients. So when you use nova start-server you will see options 
applicable to this command etc.

For the unified client we can make a nice commands hierarhcy and I don't think 
it will be confusing to users for example:

$ openstack help
nova cloud computing fabric controller
glance  discovering, registering, retrieving and storing virtual machine 
images
cinder
...
22 records

$ openstack help nova
OpenStack Nova provides a cloud computing fabric controller, supporting a wide 
variety of virtualization technologies, including KVM, Xen, LXC, VMware, and 
more. In addition to its native API, it includes compatibility with the 
commonly encountered Amazon EC2 and S3 APIs.

start-server
...

With proper bash completition it should be quite easy to use.


Since people are complaining about thread depth: I’ll focus on the blueprint 
and wiki page for now - we’re disagreeing on basic assumptions and facts here. 
If you are a developer or a user, trying to build an application that even uses 
50% of the currently shipping openstack software; your life is extremely sub 
optimal.

The ideas expressed is to replace the focus of the clients and CLIs from “those 
who build openstack” to “those that consume openstack” which is a very 
different profile, knowledge level and user story.



2014/1/21 Jesse Noller 
mailto:jesse.nol...@rackspace.com>>

On Jan 21, 2014, at 12:19 PM, Alexei Kornienko 
mailto:alexei.kornie...@gmail.com>> wrote:

Hello,

I would like to end this requirements talk cause it doesn't make any sense in 
term of python clients.
Initially the discussion was about "many clients projects with separate 
requirements" VS "single client project with single requirements list".


Then I suspect we’re at an impasse; I do plan on proceeding with a new wiki 
page and blueprint for a unified client, SDK (backend) for end users. I for one 
do not think things are as clear cut as you make them - and the +1s to the 
unified client thoughts clearly express the discontent with the current 
structure & packaging.

At that moment we should have stop and actually open requirements lists for 
python clients.
Basically all clients have the same requirement (cause they all do the same 
stuff - sending HTTP requests K.O.)
There is absolutely no difference in the situation of many clients vs single 
client.

Answering another question about "user only needs X (keystone) and we install 
package with clients for all openstack services":
Size of keystone client (and any other client I suppose) is ~300Kb I don't 
think that it's a big difference for the user to install package that is ~300Kb 
or ~10Mb (unless we are using openstack from Android).

It is when most openstack clouds don’t just run keystone. Or nova, or swift. Or 
when each client acts, smells and behaves differently. It matters a LOT when 
you’re trying to write applications on top of a mature openstack deployment.


>From the user perspective I think it's much easier to use client with 
>"everything included" rather than try to google for client package for some 
>rarely used service.


I have actual experience here as 

Re: [openstack-dev] a "common" client library

[Jesse Noller]

On Jan 21, 2014, at 12:19 PM, Alexei Kornienko 
mailto:alexei.kornie...@gmail.com>> wrote:

Hello,

I would like to end this requirements talk cause it doesn't make any sense in 
term of python clients.
Initially the discussion was about "many clients projects with separate 
requirements" VS "single client project with single requirements list".


Then I suspect we’re at an impasse; I do plan on proceeding with a new wiki 
page and blueprint for a unified client, SDK (backend) for end users. I for one 
do not think things are as clear cut as you make them - and the +1s to the 
unified client thoughts clearly express the discontent with the current 
structure & packaging.

At that moment we should have stop and actually open requirements lists for 
python clients.
Basically all clients have the same requirement (cause they all do the same 
stuff - sending HTTP requests K.O.)
There is absolutely no difference in the situation of many clients vs single 
client.

Answering another question about "user only needs X (keystone) and we install 
package with clients for all openstack services":
Size of keystone client (and any other client I suppose) is ~300Kb I don't 
think that it's a big difference for the user to install package that is ~300Kb 
or ~10Mb (unless we are using openstack from Android).

It is when most openstack clouds don’t just run keystone. Or nova, or swift. Or 
when each client acts, smells and behaves differently. It matters a LOT when 
you’re trying to write applications on top of a mature openstack deployment.


>From the user perspective I think it's much easier to use client with 
>"everything included" rather than try to google for client package for some 
>rarely used service.


I have actual experience here as a person running a cloud and supporting 
end-users & developers: the overwhelming customer feedback (paid and unpaid 
(exploratory users)) is that the 22+ clients are confusing, difficult to use, 
prone to error. There are two ways of resolving this if you’re in my shoes - or 
in a role where the primary focus is not openstack developers or builders; the 
first is you coordinate work focusing on developer & end user experience 
upstream, working with openstack and the teams to come up with something that 
benefits everyone, or, you fork, and build the openstack equivalent to boto / 
awscli so you can provide a unified “one obvious way to consume openstack 
clouds”.

jesse

Regards,
Alexei




2014/1/21 Sean Dague mailto:s...@dague.net>>
On 01/21/2014 11:54 AM, Renat Akhmerov wrote:
>
> On 17 Jan 2014, at 22:00, Jamie Lennox 
> mailto:jamielen...@redhat.com>
> >> wrote:
>
>> (I don't buy the problem with large amounts of dependencies, if you
>> have a meta-package you just have one line in requirements and pip
>> will figure the rest out.)
>
> +1
>
> Renat Akhmerov
> @ Mirantis Inc.

Man, where were you then when we had to spend 3 weeks unwinding global
requirements in the gate because pip was figuring it out all kinds of
wrong, and we'd do things like uninstall and reinstall
python-keystoneclient 6 times during an install. Because after that
experience, I'm very anti "pip will figure the rest out".

Because it won't, not in python, where we're talking about libraries
that are in the global namespace, where python can only have 1 version
of a dependency installed.

If the the solution is every openstack project should install a venv for
all it's dependencies to get around this issue, then we're talking a
different problem (and a different architecture from what we've been
trying to do). But I find the idea of having 12 copies of
python-keystone client installed on my openstack environment to be
distasteful.

So come spend a month working on requirements updates in OpenStack
gate... and if you still believe "pip will figure it out", well you are
a braver man than I. :)

-Sean

--
Sean Dague
Samsung Research America
s...@dague.net / 
sean.da...@samsung.com
http://dague.net


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Jesse Noller]

On Jan 21, 2014, at 10:54 AM, Renat Akhmerov 
mailto:rakhme...@mirantis.com>> wrote:


On 17 Jan 2014, at 22:00, Jamie Lennox 
mailto:jamielen...@redhat.com>> wrote:

(I don't buy the problem with large amounts of dependencies, if you have a 
meta-package you just have one line in requirements and pip will figure the 
rest out.)

+1

Renat Akhmerov
@ Mirantis Inc.

Do you use any other platform than Linux? Even donald - one of the python 
packaging leads and PyPI leads said this is a bad end-user experience for 
consumers of openstack clouds.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Jesse Noller]

On Jan 19, 2014, at 5:37 PM, Jamie Lennox 
mailto:jamielen...@redhat.com>> wrote:

On Sat, 2014-01-18 at 09:13 -0500, Doug Hellmann wrote:
I like the idea of a fresh start, but I don't think that's
incompatible with the other work to clean up the existing clients.
That cleanup work could help with creating the backwards compatibility
layer, if a new library needs to include one, for example.


As far as namespace packages and separate client libraries, I'm torn.
It makes sense, and I originally assumed we would want to take that
approach. The more I think about it, though, the more I like the
approach Dean took with the CLI, creating a single repository with a
team responsible for managing consistency in the UI.


Doug

This *is* the approach Dean took with the CLI. Have a package that
provides the CLI but then have the actual work handed off to the
individual clients (with quite a lot of glue).

And I think many of us are making the argument (or trying to) that the “a lot 
of glue” approach is wrong and unsustainable for both a unified CLI long term 
*and especially* for application developers.



On Sat, Jan 18, 2014 at 1:00 AM, Jamie Lennox 
mailto:jamielen...@redhat.com>>
wrote:
   I can't see any reason that all of these situations can't be
   met.

   We can finally take the openstack pypi namespace, move
   keystoneclient -> openstack.keystone and similar for the other
   projects. Have them all based upon openstack.base and probably
   an openstack.transport for transport.

   For the all-in-one users we can then just have
   openstack.client which depends on all of the openstack.x
   projects. This would satisfy the requirement of keeping
   projects seperate, but having the one entry point for newer
   users. Similar to the OSC project (which could acutally rely
   on the new all-in-one).

   This would also satisfy a lot of the clients who have i know
   are looking to move to a version 2 and break compatability
   with some of the crap from the early days.

   I think what is most important here is deciding what we want
   from our clients and discussing a common base that we are
   happy to support - not just renaming the existing ones.

   (I don't buy the problem with large amounts of dependencies,
   if you have a meta-package you just have one line in
   requirements and pip will figure the rest out.)

   Jamie

   - Original Message -
From: "Jonathan LaCour" 
mailto:jonathan-li...@cleverdevil.org>>
To: "OpenStack Development Mailing List (not for usage
   questions)" 
mailto:openstack-dev@lists.openstack.org>>
Sent: Saturday, 18 January, 2014 4:00:58 AM
Subject: Re: [openstack-dev] a "common" client library


On Thu, Jan 16, 2014 at 1:23 PM, Donald Stufft <
   don...@stufft.io<mailto:don...@stufft.io> > wrote:




On Jan 16, 2014, at 4:06 PM, Jesse Noller <
   jesse.nol...@rackspace.com<mailto:jesse.nol...@rackspace.com> >
wrote:





On Jan 16, 2014, at 2:22 PM, Renat Akhmerov <
   rakhme...@mirantis.com<mailto:rakhme...@mirantis.com> > wrote:




Since it’s pretty easy to get lost among all the opinions
   I’d like to
clarify/ask a couple of things:




   * Keeping all the clients physically separate/combining
   them in to a
   single library. Two things here:

   * In case of combining them, what exact project are
   we considering?
   If this list is limited to core projects like nova
   and keystone what
   policy could we have for other projects to join this
   list?
   (Incubation, graduation, something else?)

   * In terms of granularity and easiness of
   development I’m for keeping
   them separate but have them use the same boilerplate
   code, basically
   we need a OpenStack Rest Client Framework which is
   flexible enough
   to address all the needs in an abstract domain
   agnostic manner. I
   would assume that combining them would be an
   additional
   organizational burden that every stakeholder would
   have to deal
   with.

Keeping them separate is awesome for *us* but really,
   really, really sucks
for users trying to use the system.

I agree. Keeping them separate trades user usability for
   developer usability,
I think user usability is a better thing to strive for.
100% agree with this. In order for OpenStack to be its most
   successful, I
believe firmly that a focus on end-users and deployers needs
   to take the
forefront. That means making OpenStack clouds as easy to
   consume/leverage as
possible for users and tool builders, and
   simplifying/streamlining as much
as possible.

I think that a single, common client project, based upon
   package namespaces,
with a unified, cohesive feel is a big step 

Re: [openstack-dev] a "common" client library

[Jesse Noller]

On Jan 18, 2014, at 12:00 AM, Jamie Lennox  wrote:

> I can't see any reason that all of these situations can't be met. 
> 
> We can finally take the openstack pypi namespace, move keystoneclient -> 
> openstack.keystone and similar for the other projects. Have them all based 
> upon openstack.base and probably an openstack.transport for transport.
> 
> For the all-in-one users we can then just have openstack.client which depends 
> on all of the openstack.x projects. This would satisfy the requirement of 
> keeping projects seperate, but having the one entry point for newer users. 
> Similar to the OSC project (which could acutally rely on the new all-in-one).
> 
> This would also satisfy a lot of the clients who have i know are looking to 
> move to a version 2 and break compatability with some of the crap from the 
> early days.
> 
> I think what is most important here is deciding what we want from our clients 
> and discussing a common base that we are happy to support - not just renaming 
> the existing ones.
> 
> (I don't buy the problem with large amounts of dependencies, if you have a 
> meta-package you just have one line in requirements and pip will figure the 
> rest out.)

You’re assuming:

1: Pip works when installing the entire dependency graph (it often doesn’t)
2: For some of these requirements, the user has a compiler installed (they 
don’t)
3: Installing 1 “meta package” that install N+K dependencies makes end user 
consumers happy (it doesn’t)
4: All of these dependencies make shipping a single binary deployment easier 
(it doesn’t)
5: Installing and using all of these things makes using openstack within my 
code conceptually simpler (it doesn’t)

We can start with *not* renaming the sub clients (meaning) collapsing them into 
the singular namespace; but the problem is that every one of those sub 
dependencies is potential liability to someone using this single client. 

If yes, we could only target fedora, and rely on yum & rpm, I’d agree with you 
- but for python application dependencies across multiple OSes and developers 
doing ci/cd using these systems I can’t. I also don’t want user to stumble into 
the nuanced vagaries of the sub-clients when writing application code; writing 
glue code to bind them all together does work very well (we know this from 
experience).


> 
> Jamie
> 
> - Original Message -
>> From: "Jonathan LaCour" 
>> To: "OpenStack Development Mailing List (not for usage questions)" 
>> 
>> Sent: Saturday, 18 January, 2014 4:00:58 AM
>> Subject: Re: [openstack-dev] a "common" client library
>> 
>> On Thu, Jan 16, 2014 at 1:23 PM, Donald Stufft < don...@stufft.io > wrote:
>> 
>> 
>> 
>> 
>> On Jan 16, 2014, at 4:06 PM, Jesse Noller < jesse.nol...@rackspace.com >
>> wrote:
>> 
>> 
>> 
>> 
>> 
>> On Jan 16, 2014, at 2:22 PM, Renat Akhmerov < rakhme...@mirantis.com > wrote:
>> 
>> 
>> 
>> 
>> Since it’s pretty easy to get lost among all the opinions I’d like to
>> clarify/ask a couple of things:
>> 
>> 
>> 
>>* Keeping all the clients physically separate/combining them in to a
>>single library. Two things here:
>>* In case of combining them, what exact project are we considering?
>>If this list is limited to core projects like nova and keystone what
>>policy could we have for other projects to join this list?
>>(Incubation, graduation, something else?)
>>* In terms of granularity and easiness of development I’m for keeping
>>them separate but have them use the same boilerplate code, basically
>>we need a OpenStack Rest Client Framework which is flexible enough
>>to address all the needs in an abstract domain agnostic manner. I
>>would assume that combining them would be an additional
>>organizational burden that every stakeholder would have to deal
>>with.
>> 
>> Keeping them separate is awesome for *us* but really, really, really sucks
>> for users trying to use the system.
>> 
>> I agree. Keeping them separate trades user usability for developer usability,
>> I think user usability is a better thing to strive for.
>> 100% agree with this. In order for OpenStack to be its most successful, I
>> believe firmly that a focus on end-users and deployers needs to take the
>> forefront. That means making OpenStack clouds as easy to consume/leverage as
>> possible for users and tool builders, and simplifying/streamlining as much
>> as possible.
>> 
>> I think that a single, common client project, based u

Re: [openstack-dev] a "common" client library

[Jesse Noller]

On Jan 18, 2014, at 8:13 AM, Doug Hellmann 
mailto:doug.hellm...@dreamhost.com>> wrote:

I like the idea of a fresh start, but I don't think that's incompatible with 
the other work to clean up the existing clients. That cleanup work could help 
with creating the backwards compatibility layer, if a new library needs to 
include one, for example.

As far as namespace packages and separate client libraries, I'm torn. It makes 
sense, and I originally assumed we would want to take that approach. The more I 
think about it, though, the more I like the approach Dean took with the CLI, 
creating a single repository with a team responsible for managing consistency 
in the UI.

Doug



I’m going to pursue the latter - but at the same time the current effort to 
clean things up seems to be running afoul of the keystone client changes in 
flight?


On Sat, Jan 18, 2014 at 1:00 AM, Jamie Lennox 
mailto:jamielen...@redhat.com>> wrote:
I can't see any reason that all of these situations can't be met.

We can finally take the openstack pypi namespace, move keystoneclient -> 
openstack.keystone and similar for the other projects. Have them all based upon 
openstack.base and probably an openstack.transport for transport.

For the all-in-one users we can then just have openstack.client which depends 
on all of the openstack.x projects. This would satisfy the requirement of 
keeping projects seperate, but having the one entry point for newer users. 
Similar to the OSC project (which could acutally rely on the new all-in-one).

This would also satisfy a lot of the clients who have i know are looking to 
move to a version 2 and break compatability with some of the crap from the 
early days.

I think what is most important here is deciding what we want from our clients 
and discussing a common base that we are happy to support - not just renaming 
the existing ones.

(I don't buy the problem with large amounts of dependencies, if you have a 
meta-package you just have one line in requirements and pip will figure the 
rest out.)

Jamie

- Original Message -
> From: "Jonathan LaCour" 
> mailto:jonathan-li...@cleverdevil.org>>
> To: "OpenStack Development Mailing List (not for usage questions)" 
> mailto:openstack-dev@lists.openstack.org>>
> Sent: Saturday, 18 January, 2014 4:00:58 AM
> Subject: Re: [openstack-dev] a "common" client library
>
> On Thu, Jan 16, 2014 at 1:23 PM, Donald Stufft < 
> don...@stufft.io<mailto:don...@stufft.io> > wrote:
>
>
>
>
> On Jan 16, 2014, at 4:06 PM, Jesse Noller < 
> jesse.nol...@rackspace.com<mailto:jesse.nol...@rackspace.com> >
> wrote:
>
>
>
>
>
> On Jan 16, 2014, at 2:22 PM, Renat Akhmerov < 
> rakhme...@mirantis.com<mailto:rakhme...@mirantis.com> > wrote:
>
>
>
>
> Since it’s pretty easy to get lost among all the opinions I’d like to
> clarify/ask a couple of things:
>
>
>
> * Keeping all the clients physically separate/combining them in to a
> single library. Two things here:
> * In case of combining them, what exact project are we considering?
> If this list is limited to core projects like nova and keystone what
> policy could we have for other projects to join this list?
> (Incubation, graduation, something else?)
> * In terms of granularity and easiness of development I’m for keeping
> them separate but have them use the same boilerplate code, basically
> we need a OpenStack Rest Client Framework which is flexible enough
> to address all the needs in an abstract domain agnostic manner. I
> would assume that combining them would be an additional
> organizational burden that every stakeholder would have to deal
> with.
>
> Keeping them separate is awesome for *us* but really, really, really sucks
> for users trying to use the system.
>
> I agree. Keeping them separate trades user usability for developer usability,
> I think user usability is a better thing to strive for.
> 100% agree with this. In order for OpenStack to be its most successful, I
> believe firmly that a focus on end-users and deployers needs to take the
> forefront. That means making OpenStack clouds as easy to consume/leverage as
> possible for users and tool builders, and simplifying/streamlining as much
> as possible.
>
> I think that a single, common client project, based upon package namespaces,
> with a unified, cohesive feel is a big step in this direction.
>
> --
> Jonathan LaCour
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org>
> http://lists.openstack.org/cgi-bin/mailman

Re: [openstack-dev] a "common" client library

[Jesse Noller]

On Jan 17, 2014, at 10:03 PM, John Utz  wrote:

> Outlook Web MUA, forgive the top post. :-(
> 
> While a single import line that brings all the good stuff in at one shot is 
> very convenient for the creation of an application, it would muddy the 
> security picture *substantially* for the exact type of developer\customer 
> that you would be targeting with this sort of syntactic sugar.
> 
> As Jesse alludes to below, the expanding tree of dependencies would be masked 
> by the aggregation.
> 
> So, most likely, they would be pulling in vast numbers of things that they 
> don't require to get their simple app done (there's an idea! an eclipse 
> plugin that helpfully points out all the things that you are *not* using and 
> offers to redo your imports for you :-) ).
> 
> As a result, when a security defect is published concerning one of those 
> hidden dependencies, they will not have any reason to think that it effects 
> them.
> 
> just my us$0.02;
> 


Thats why when a security defect in the *client side* tools is announced it 
affects the entire client library which is then versioned. Not just a sub part: 
you re-ship the entire thing. 

This is how every other SDK *other* than openstack’s cli tools handles this.



> johnu 
> 
> From: Jesse Noller [jesse.nol...@rackspace.com]
> Sent: Thursday, January 16, 2014 5:42 PM
> To: OpenStack Development Mailing List (not for usage questions)
> Cc: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] a "common" client library
> 
> On Jan 16, 2014, at 4:59 PM, "Renat Akhmerov" 
> mailto:rakhme...@mirantis.com>> wrote:
> 
> On 16 Jan 2014, at 13:06, Jesse Noller 
> mailto:jesse.nol...@rackspace.com>> wrote:
> 
> Since it’s pretty easy to get lost among all the opinions I’d like to 
> clarify/ask a couple of things:
> 
> 
>  *   Keeping all the clients physically separate/combining them in to a 
> single library. Two things here:
> *   In case of combining them, what exact project are we considering? If 
> this list is limited to core projects like nova and keystone what policy 
> could we have for other projects to join this list? (Incubation, graduation, 
> something else?)
> *   In terms of granularity and easiness of development I’m for keeping 
> them separate but have them use the same boilerplate code, basically we need 
> a OpenStack Rest Client Framework which is flexible enough to address all the 
> needs in an abstract domain agnostic manner. I would assume that combining 
> them would be an additional organizational burden that every stakeholder 
> would have to deal with.
> 
> Keeping them separate is awesome for *us* but really, really, really sucks 
> for users trying to use the system.
> 
> You may be right but not sure that adding another line into requirements.txt 
> is a huge loss of usability.
> 
> 
> It is when that 1 dependency pulls in 6 others that pull in 10 more - every 
> little barrier or potential failure from the inability to make a static 
> binary to how each tool acts different is a paper cut of frustration to an 
> end user.
> 
> Most of the time the clients don't even properly install because of 
> dependencies on setuptools plugins and other things. For developers (as I've 
> said) the story is worse: you have potentially 22+ individual packages and 
> their dependencies to deal with if they want to use a complete openstack 
> install from their code.
> 
> So it doesn't boil down to just 1 dependency: it's a long laundry list of 
> things that make consumers' lives more difficult and painful.
> 
> This doesn't even touch on the fact there aren't blessed SDKs or tools 
> pointing users to consume openstack in their preferred programming language.
> 
> Shipping an API isn't enough - but it can be fixed easily enough.
> 
> Renat Akhmerov
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Jesse Noller]

On Jan 16, 2014, at 8:36 PM, "Renat Akhmerov" 
mailto:rakhme...@mirantis.com>> wrote:

Ok, I think most of the reasoning you’ve said makes sense. So for a 
non-incubated project we’re going to have separate clients and then get them 
integrated into this single client once the project itself gets incubated? Or 
it’s going to happen when the project gets integrated into core os projects? So 
if yes, it’s just going to be one more incubation/integration requirement, 
right?


My gut says moving more code into Oslo and incurring that change for all 22(23 
with solum) client code based short term solves one aspect of cleaning up the 
clients - if - the sub projects agree that the code they are now depending on 
presents a useful API to them. It's not clear to me that case has been 
explained, or the design of those new libraries sussed out.

My personal opinion is that if people wanted the current consolidation to 
continue we let that go, but begin fleshing out what a stand alone 
project/library akin to horizon would look like (let's call it zenith).

If we go this route we need to buildout some blueprints with design / layout / 
APIs for people to review and approve.

When talking with dean today we actually see it as fewer layers but the most 
work : benefit comes from unifying and coordinating the client code from each 
of the projects. This latter part requires consensus from the teams on the 
layout and design, contributing to it, etc

Dean and I started an etherpad today, I need to go back through this thread and 
capture more design considerations and properly capture it in the wiki.

With at least a fleshed out strawman proposal for layout and design we can 
discuss concrete things. I think some of the contention is focused on the 
currently progressing work on a blueprint that doesn't fully capture the design 
and I'd like to avoid that.

Renat

On 16 Jan 2014, at 18:09, Donald Stufft 
mailto:don...@stufft.io>> wrote:


On Jan 16, 2014, at 8:42 PM, Jesse Noller 
mailto:jesse.nol...@rackspace.com>> wrote:



On Jan 16, 2014, at 4:59 PM, "Renat Akhmerov" 
mailto:rakhme...@mirantis.com>> wrote:

On 16 Jan 2014, at 13:06, Jesse Noller 
mailto:jesse.nol...@rackspace.com>> wrote:

Since it’s pretty easy to get lost among all the opinions I’d like to 
clarify/ask a couple of things:


  *   Keeping all the clients physically separate/combining them in to a single 
library. Two things here:
 *   In case of combining them, what exact project are we considering? If 
this list is limited to core projects like nova and keystone what policy could 
we have for other projects to join this list? (Incubation, graduation, 
something else?)
 *   In terms of granularity and easiness of development I’m for keeping 
them separate but have them use the same boilerplate code, basically we need a 
OpenStack Rest Client Framework which is flexible enough to address all the 
needs in an abstract domain agnostic manner. I would assume that combining them 
would be an additional organizational burden that every stakeholder would have 
to deal with.

Keeping them separate is awesome for *us* but really, really, really sucks for 
users trying to use the system.

You may be right but not sure that adding another line into requirements.txt is 
a huge loss of usability.


It is when that 1 dependency pulls in 6 others that pull in 10 more - every 
little barrier or potential failure from the inability to make a static binary 
to how each tool acts different is a paper cut of frustration to an end user.

Most of the time the clients don't even properly install because of 
dependencies on setuptools plugins and other things. For developers (as I've 
said) the story is worse: you have potentially 22+ individual packages and 
their dependencies to deal with if they want to use a complete openstack 
install from their code.

So it doesn't boil down to just 1 dependency: it's a long laundry list of 
things that make consumers' lives more difficult and painful.

This doesn't even touch on the fact there aren't blessed SDKs or tools pointing 
users to consume openstack in their preferred programming language.

Shipping an API isn't enough - but it can be fixed easily enough.

There’s also the discovery problem, it’s incredibly frustrating if, as I’m 
starting out to use an Openstack based cloud, everytime I want to touch some 
new segment of the service I need to go find out what the client lib is for 
that, possibly download the dependencies for it, possibly get it approved, etc.

Splitting up services makes a lot of sense on the server side, but to the 
consumer a cloud often times isn’t a disjoint set of services that happen to be 
working in parallel, it is a single unified product where they may not know the 
boundary lines, or at the very least the boundaries can be fuzzy for them.


Renat Akhmerov
___

Re: [openstack-dev] a "common" client library

[Jesse Noller]

On Jan 16, 2014, at 4:59 PM, "Renat Akhmerov" 
mailto:rakhme...@mirantis.com>> wrote:

On 16 Jan 2014, at 13:06, Jesse Noller 
mailto:jesse.nol...@rackspace.com>> wrote:

Since it’s pretty easy to get lost among all the opinions I’d like to 
clarify/ask a couple of things:


  *   Keeping all the clients physically separate/combining them in to a single 
library. Two things here:
 *   In case of combining them, what exact project are we considering? If 
this list is limited to core projects like nova and keystone what policy could 
we have for other projects to join this list? (Incubation, graduation, 
something else?)
 *   In terms of granularity and easiness of development I’m for keeping 
them separate but have them use the same boilerplate code, basically we need a 
OpenStack Rest Client Framework which is flexible enough to address all the 
needs in an abstract domain agnostic manner. I would assume that combining them 
would be an additional organizational burden that every stakeholder would have 
to deal with.

Keeping them separate is awesome for *us* but really, really, really sucks for 
users trying to use the system.

You may be right but not sure that adding another line into requirements.txt is 
a huge loss of usability.


It is when that 1 dependency pulls in 6 others that pull in 10 more - every 
little barrier or potential failure from the inability to make a static binary 
to how each tool acts different is a paper cut of frustration to an end user.

Most of the time the clients don't even properly install because of 
dependencies on setuptools plugins and other things. For developers (as I've 
said) the story is worse: you have potentially 22+ individual packages and 
their dependencies to deal with if they want to use a complete openstack 
install from their code.

So it doesn't boil down to just 1 dependency: it's a long laundry list of 
things that make consumers' lives more difficult and painful.

This doesn't even touch on the fact there aren't blessed SDKs or tools pointing 
users to consume openstack in their preferred programming language.

Shipping an API isn't enough - but it can be fixed easily enough.

Renat Akhmerov
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Jesse Noller]

On Jan 16, 2014, at 2:22 PM, Renat Akhmerov 
mailto:rakhme...@mirantis.com>> wrote:

Since it’s pretty easy to get lost among all the opinions I’d like to 
clarify/ask a couple of things:


  *   Keeping all the clients physically separate/combining them in to a single 
library. Two things here:
 *   In case of combining them, what exact project are we considering? If 
this list is limited to core projects like nova and keystone what policy could 
we have for other projects to join this list? (Incubation, graduation, 
something else?)
 *   In terms of granularity and easiness of development I’m for keeping 
them separate but have them use the same boilerplate code, basically we need a 
OpenStack Rest Client Framework which is flexible enough to address all the 
needs in an abstract domain agnostic manner. I would assume that combining them 
would be an additional organizational burden that every stakeholder would have 
to deal with.

Keeping them separate is awesome for *us* but really, really, really sucks for 
users trying to use the system.


  *   Has anyone ever considered an idea of generating a fully functional REST 
client automatically based on an API specification (WADL could be used for 
that)? Not sure how convenient it would be, it really depends on a particular 
implementation, but as an idea it could be at least thought of. Sounds a little 
bit crazy though, I recognize it :).

Renat Akhmerov

On 16 Jan 2014, at 11:52, Chmouel Boudjnah 
mailto:chmo...@enovance.com>> wrote:

On Thu, Jan 16, 2014 at 8:40 PM, Donald Stufft 
mailto:don...@stufft.io>> wrote:

On Jan 16, 2014, at 2:36 PM, Joe Gordon 
mailto:joe.gord...@gmail.com>> wrote:

2) major overhaul of client libraries so they are all based off a common base 
library. This would cover namespace changes, and possible a push to move CLI 
into python-openstackclient
This seems like the biggest win to me.


+1

Chmouel.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Jesse Noller]

On Jan 16, 2014, at 11:39 AM, Mark Washenberger 
mailto:mark.washenber...@markwash.net>> wrote:




On Thu, Jan 16, 2014 at 8:06 AM, Dean Troyer 
mailto:dtro...@gmail.com>> wrote:
On Thu, Jan 16, 2014 at 9:37 AM, Jesse Noller 
mailto:jesse.nol...@rackspace.com>> wrote:
On Jan 16, 2014, at 9:26 AM, Justin Hammond 
mailto:justin.hamm...@rackspace.com>> wrote:
I'm not sure if it was said, but which httplib using being used (urllib3
maybe?). Also I noticed many people were talking about supporting auth
properly, but are there any intentions to properly support 'noauth'
(python-neutronclient, for instance, doesn't support it properly as of
this writing)?
Can you detail out noauth for me; and I would say the defacto httplib in python 
today is python-requests - urllib3 is also good but I would say from a 
*consumer* standpoint requests offers more in terms of usability / extensibility

requests is built on top of urllib3 so there's that...

The biggest reaon I favor using Jamie Lennox's new session layer stuff in 
keystoneclient is that it better reflects the requests API instead of it being 
stuffed in after the fact.  And as the one responsible for that stuffing, it 
was pretty blunt and really needs to be cleaned up more than Alessio did.

only a few libs (maybe just glance and swift?) don't use requests at this point 
and I think the resistance there is the chunked transfers they both do.

There's a few more items here that are needed for glance to be able to work 
with requests (which we really really want).
1) Support for 100-expect-continue is probably going to be required in glance 
as well as swift
2) Support for turning off tls/ssl compression (our streams are already 
compressed)

I feel like we *must* have somebody here who is able and willing to add these 
features to requests, which seems like the right approach.

Let me talk to upstream about this; I know a lot of people involved. Patches 
from us probably needed, but I’ll ask



I'm really curious what 'noauth' means against APIs that have few, if any, 
calls that operate without a valid token.

dt

--

Dean Troyer
dtro...@gmail.com<mailto:dtro...@gmail.com>

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Jesse Noller]

On Jan 16, 2014, at 9:54 AM, Alexei Kornienko 
mailto:alexei.kornie...@gmail.com>> wrote:

On 01/16/2014 05:25 PM, Jesse Noller wrote:

On Jan 16, 2014, at 9:07 AM, Joe Gordon 
mailto:joe.gord...@gmail.com>> wrote:




On Thu, Jan 16, 2014 at 9:45 AM, Jesse Noller 
mailto:jesse.nol...@rackspace.com>> wrote:

On Jan 16, 2014, at 5:53 AM, Chmouel Boudjnah 
mailto:chmo...@enovance.com>> wrote:


On Thu, Jan 16, 2014 at 12:38 PM, Chris Jones 
mailto:c...@tenshu.net>> wrote:
Once a common library is in place, is there any intention to (or resistance 
against) collapsing the clients into a single project or even a single command 
(a la busybox)?


that's what openstackclient is here for 
https://github.com/openstack/python-openstackclient

After speaking with people working on OSC and looking at the code base in 
depth; I don’t think this addresses what Chris is implying: OSC wraps the 
individual CLIs built by each project today, instead of the inverse: a common 
backend that the individual CLIs can wrap - the latter is an important 
distinction as currently, building a single binary install of OSC for say, 
Windows is difficult given the dependency tree incurred by each of the wrapped 
CLIs, difference in dependencies, structure, etc.

Also, wrapping a series of inconsistent back end Client classes / functions / 
methods means that the layer that presents a consistent user interface (OSC) to 
the user is made more complex juggling names/renames/commands/etc.

In the inverted case of what we have today (single backend); as a developer of 
user interfaces (CLIs, Applications, Web apps (horizon)) you would be able to:

from openstack.common.api import Auth
from openstack.common.api import Compute
from openstack.common.util import cli_tools

my_cli = cli_tools.build(…)

def my_command(cli):
compute = Compute(Auth(cli.tentant…, connect=True))
compute.list_flavors()

This would mean that *even if the individual clients needed or wanted to keep 
their specific CLIs, they would be able to use a not “least common denominator” 
back end (each service can have a rich 
common.api.compute.py<http://common.api.compute.py/> or api.compute/client.py 
and extend where needed. However tools like horizon / openstackclient can 
choose not to leverage the “power user/operator/admin” components and present a 
simplified user interface.

I’m working on a wiki page + blueprint to brainstorm how we could accomplish 
this based off of what work is in flight today (see doug’s linked blueprint) 
and sussing out a layout / API strawman for discussion.

Some of the additions that came out of this email threads and others:

1. Common backend should provide / offer caching utilities
2. Auth retries need to be handled by the auth object, and each sub-project 
delegates to the auth object to manage that.
3. Verified Mocks / Stubs / Fakes must be provided for proper unit testing

I am happy to see this work being done, there is definitely a lot of work to be 
done on the clients.

This blueprint sounds like its still being fleshed out, so I am wondering what 
the value is of the current patches 
https://review.openstack.org/#/q/topic:bp/common-client-library-2,n,z

Those patches mainly sync cliutils and apiutils from oslo into the assorted 
clients. But if this blueprint is about the python API and not the CLI (as that 
would be the openstack-pythonclient), why sync in apiutils?

Also does this need to go through oslo-incubator or can this start out as a 
library? Making this a library earlier on will reduce the number of patches 
needed to get 20+ repositories to use this.


Alexei and others have at least started the first stage of a rollout - the 
blueprint(s) needs additional work, planning and discussion, but his work is a 
good first step (reduce the duplication of code) although I am worried that the 
libraries and APIs / namespaces will need to change if we continue these 
discussions which potentially means re-doing work.

If we take a step back, a rollout process might be:

1: Solidify the libraries / layout / naming conventions (blueprint)
2: Solidify the APIs exposed to consumers (blueprint)
3: Pick up on the common-client-library-2 work which is primarily a migration 
of common code into oslo today, into the structure defined by 1 & 2

So, I sort of agree: moving / collapsing code now might be premature. I do 
strongly agree it should stand on its own as a library rather than an oslo 
incubator however. We should start with a single, clean namespace / library 
rather than depending on oslo directly.
Knowing usual openstack workflow I'm afraid that #1,#2 with a waterfall 
approach may take years to be complete.
And after they'll be approved it will become clear that this architecture is 
already outdated.
We try to use iterative approach for clients refactoring.
We started our work from removing code duplication because it already gives a 
direct positive effect on client pr

Re: [openstack-dev] a "common" client library

[Jesse Noller]

On Jan 16, 2014, at 9:26 AM, Justin Hammond 
mailto:justin.hamm...@rackspace.com>> wrote:

I'm not sure if it was said, but which httplib using being used (urllib3
maybe?). Also I noticed many people were talking about supporting auth
properly, but are there any intentions to properly support 'noauth'
(python-neutronclient, for instance, doesn't support it properly as of
this writing)?


Can you detail out noauth for me; and I would say the defacto httplib in python 
today is python-requests - urllib3 is also good but I would say from a 
*consumer* standpoint requests offers more in terms of usability / extensibility

On 1/15/14 10:53 PM, "Alexei Kornienko" 
mailto:alexei.kornie...@gmail.com>> wrote:

I did notice, however, that neutronclient is
conspicuously absent from the Work Items in the blueprint's Whiteboard.

It will surely be added later. We already working on several things in
parallel and we will add neutronclient soon.

Do you need another person to make neutron client?


I would love to see a bit more detail on the structure of the
lib(s), the blueprint really doesn't discuss the
design/organization/intended API of the libs.  For example, I would hope
the distinction between the various layers of a client stack don't get
lost, i.e. not mixing the low-level REST API bits with the higher-level
CLI parsers and decorators.
Does the long-term goals include a common caching layer?


Distinction between client layers won't get lost and would only be
improved. My basic idea is the following:

1) Transport layer would handle all transport related stuff - HTTP, JSON
encoding, auth, caching, etc.

2) Model layer (Resource classes, BaseManager, etc.) will handle data
representation, validation

3) API layer will handle all project specific stuff - url mapping, etc.
(This will be imported to use client in other applications)

4) Cli level will handle all stuff related to cli mapping - argparse,
argcomplete, etc.


I believe the current effort referenced by the blueprint is focusing on
moving existing code into the incubator for reuse, to make it easier to
restructure later. Alexei, do I have that correct?

You are right. The first thing we do is try to make all clients look/work
in similar way. After we'll continue our work with improving overall
structure.






2014/1/16 Noorul Islam K M mailto:noo...@noorul.com>>

Doug Hellmann mailto:doug.hellm...@dreamhost.com>> 
writes:

Several people have mentioned to me that they are interested in, or
actively working on, code related to a "common" client library --
something
meant to be reused directly as a basis for creating a common library for
all of the openstack clients to use. There's a blueprint [1] in oslo,
and I
believe the keystone devs and unified CLI teams are probably interested
in
ensuring that the resulting API ends up meeting all of our various
requirements.

If you're interested in this effort, please subscribe to the blueprint
and
use that to coordinate efforts so we don't produce more than one common
library. ;-)



Solum is already using it https://review.openstack.org/#/c/58067/

I would love to watch this space.

Regards,
Noorul

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev







___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Jesse Noller]

On Jan 16, 2014, at 9:07 AM, Joe Gordon 
mailto:joe.gord...@gmail.com>> wrote:




On Thu, Jan 16, 2014 at 9:45 AM, Jesse Noller 
mailto:jesse.nol...@rackspace.com>> wrote:

On Jan 16, 2014, at 5:53 AM, Chmouel Boudjnah 
mailto:chmo...@enovance.com>> wrote:


On Thu, Jan 16, 2014 at 12:38 PM, Chris Jones 
mailto:c...@tenshu.net>> wrote:
Once a common library is in place, is there any intention to (or resistance 
against) collapsing the clients into a single project or even a single command 
(a la busybox)?


that's what openstackclient is here for 
https://github.com/openstack/python-openstackclient

After speaking with people working on OSC and looking at the code base in 
depth; I don’t think this addresses what Chris is implying: OSC wraps the 
individual CLIs built by each project today, instead of the inverse: a common 
backend that the individual CLIs can wrap - the latter is an important 
distinction as currently, building a single binary install of OSC for say, 
Windows is difficult given the dependency tree incurred by each of the wrapped 
CLIs, difference in dependencies, structure, etc.

Also, wrapping a series of inconsistent back end Client classes / functions / 
methods means that the layer that presents a consistent user interface (OSC) to 
the user is made more complex juggling names/renames/commands/etc.

In the inverted case of what we have today (single backend); as a developer of 
user interfaces (CLIs, Applications, Web apps (horizon)) you would be able to:

from openstack.common.api import Auth
from openstack.common.api import Compute
from openstack.common.util import cli_tools

my_cli = cli_tools.build(…)

def my_command(cli):
compute = Compute(Auth(cli.tentant…, connect=True))
compute.list_flavors()

This would mean that *even if the individual clients needed or wanted to keep 
their specific CLIs, they would be able to use a not “least common denominator” 
back end (each service can have a rich 
common.api.compute.py<http://common.api.compute.py/> or api.compute/client.py 
and extend where needed. However tools like horizon / openstackclient can 
choose not to leverage the “power user/operator/admin” components and present a 
simplified user interface.

I’m working on a wiki page + blueprint to brainstorm how we could accomplish 
this based off of what work is in flight today (see doug’s linked blueprint) 
and sussing out a layout / API strawman for discussion.

Some of the additions that came out of this email threads and others:

1. Common backend should provide / offer caching utilities
2. Auth retries need to be handled by the auth object, and each sub-project 
delegates to the auth object to manage that.
3. Verified Mocks / Stubs / Fakes must be provided for proper unit testing

I am happy to see this work being done, there is definitely a lot of work to be 
done on the clients.

This blueprint sounds like its still being fleshed out, so I am wondering what 
the value is of the current patches 
https://review.openstack.org/#/q/topic:bp/common-client-library-2,n,z

Those patches mainly sync cliutils and apiutils from oslo into the assorted 
clients. But if this blueprint is about the python API and not the CLI (as that 
would be the openstack-pythonclient), why sync in apiutils?

Also does this need to go through oslo-incubator or can this start out as a 
library? Making this a library earlier on will reduce the number of patches 
needed to get 20+ repositories to use this.


Alexei and others have at least started the first stage of a rollout - the 
blueprint(s) needs additional work, planning and discussion, but his work is a 
good first step (reduce the duplication of code) although I am worried that the 
libraries and APIs / namespaces will need to change if we continue these 
discussions which potentially means re-doing work.

If we take a step back, a rollout process might be:

1: Solidify the libraries / layout / naming conventions (blueprint)
2: Solidify the APIs exposed to consumers (blueprint)
3: Pick up on the common-client-library-2 work which is primarily a migration 
of common code into oslo today, into the structure defined by 1 & 2

So, I sort of agree: moving / collapsing code now might be premature. I do 
strongly agree it should stand on its own as a library rather than an oslo 
incubator however. We should start with a single, clean namespace / library 
rather than depending on oslo directly.

jesse


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Jesse Noller]

On Jan 16, 2014, at 5:53 AM, Chmouel Boudjnah 
mailto:chmo...@enovance.com>> wrote:


On Thu, Jan 16, 2014 at 12:38 PM, Chris Jones 
mailto:c...@tenshu.net>> wrote:
Once a common library is in place, is there any intention to (or resistance 
against) collapsing the clients into a single project or even a single command 
(a la busybox)?


that's what openstackclient is here for 
https://github.com/openstack/python-openstackclient

After speaking with people working on OSC and looking at the code base in 
depth; I don’t think this addresses what Chris is implying: OSC wraps the 
individual CLIs built by each project today, instead of the inverse: a common 
backend that the individual CLIs can wrap - the latter is an important 
distinction as currently, building a single binary install of OSC for say, 
Windows is difficult given the dependency tree incurred by each of the wrapped 
CLIs, difference in dependencies, structure, etc.

Also, wrapping a series of inconsistent back end Client classes / functions / 
methods means that the layer that presents a consistent user interface (OSC) to 
the user is made more complex juggling names/renames/commands/etc.

In the inverted case of what we have today (single backend); as a developer of 
user interfaces (CLIs, Applications, Web apps (horizon)) you would be able to:

from openstack.common.api import Auth
from openstack.common.api import Compute
from openstack.common.util import cli_tools

my_cli = cli_tools.build(…)

def my_command(cli):
compute = Compute(Auth(cli.tentant…, connect=True))
compute.list_flavors()

This would mean that *even if the individual clients needed or wanted to keep 
their specific CLIs, they would be able to use a not “least common denominator” 
back end (each service can have a rich common.api.compute.py or 
api.compute/client.py and extend where needed. However tools like horizon / 
openstackclient can choose not to leverage the “power user/operator/admin” 
components and present a simplified user interface.

I’m working on a wiki page + blueprint to brainstorm how we could accomplish 
this based off of what work is in flight today (see doug’s linked blueprint) 
and sussing out a layout / API strawman for discussion.

Some of the additions that came out of this email threads and others:

1. Common backend should provide / offer caching utilities
2. Auth retries need to be handled by the auth object, and each sub-project 
delegates to the auth object to manage that.
3. Verified Mocks / Stubs / Fakes must be provided for proper unit testing

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Jesse Noller]

On Jan 16, 2014, at 5:42 AM, "Chris Jones" 
mailto:c...@tenshu.net>> wrote:

Hi

Once a common library is in place, is there any intention to (or resistance 
against) collapsing the clients into a single project or even a single command 
(a la busybox)?

(I'm thinking reduced load for packagers, simpler installation for users, etc)

Cheers,
--
Chris Jones

Based on the email I sent; ideally we'd expand the blueprint to include these 
details and speak to the PTLs for each project.

Ideally (IMO) yes - long term they would be collapsed into one or two packages 
to install. However at very least the code can be made so that the core 
functionality is in the common/single sdk backend and openstackcli, and each 
project CLI can derive their named/branded project specific one for that.

For standalone installs of nova or swift for example; not pushing users to use 
the entire openstackclient makes sense, instead guiding users to use a tool 
dedicated to that.

There are pros and cons on both sides, meeting in the middle at least collapses 
the heavy lifting into the common code base.


On 15 Jan 2014, at 19:37, Doug Hellmann 
mailto:doug.hellm...@dreamhost.com>> wrote:

Several people have mentioned to me that they are interested in, or actively 
working on, code related to a "common" client library -- something meant to be 
reused directly as a basis for creating a common library for all of the 
openstack clients to use. There's a blueprint [1] in oslo, and I believe the 
keystone devs and unified CLI teams are probably interested in ensuring that 
the resulting API ends up meeting all of our various requirements.

If you're interested in this effort, please subscribe to the blueprint and use 
that to coordinate efforts so we don't produce more than one common library. ;-)

Thanks,
Doug


[1] https://blueprints.launchpad.net/oslo/+spec/common-client-library-2
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Jesse Noller]

> On Jan 16, 2014, at 2:09 AM, "Flavio Percoco"  wrote:
> 
>> On 15/01/14 21:35 +, Jesse Noller wrote:
>> 
>>> On Jan 15, 2014, at 1:37 PM, Doug Hellmann  
>>> wrote:
>>> 
>>> Several people have mentioned to me that they are interested in, or 
>>> actively working on, code related to a "common" client library -- something 
>>> meant to be reused directly as a basis for creating a common library for 
>>> all of the openstack clients to use. There's a blueprint [1] in oslo, and I 
>>> believe the keystone devs and unified CLI teams are probably interested in 
>>> ensuring that the resulting API ends up meeting all of our various 
>>> requirements.
>>> 
>>> If you're interested in this effort, please subscribe to the blueprint and 
>>> use that to coordinate efforts so we don't produce more than one common 
>>> library. ;-)
>>> 
>>> Thanks,
>>> Doug
>>> 
>>> 
>>> [1] https://blueprints.launchpad.net/oslo/+spec/common-client-library-2
>> 
>> *raises hand*
>> 
>> Me me!
>> 
>> I’ve been talking to many contributors about the Developer Experience stuff 
>> I emailed out prior to the holidays and I was starting blueprint work, but 
>> this is a great pointer. I’m going to have to sync up with Alexei.
>> 
>> I think solving this for openstack developers and maintainers as the 
>> blueprint says is a big win in terms of code reuse / maintenance and 
>> consistent but more so for *end-user developers* consuming openstack clouds.
>> 
>> Some background - there’s some terminology mismatch but the rough idea is 
>> the same:
>> 
>> * A centralized “SDK” (Software Development Kit) would be built condensing 
>> the common code and logic and operations into a single namespace.
>> 
>> * This SDK would be able to be used by “downstream” CLIs - essentially the 
>> CLIs become a specialized front end - and in some cases, only an argparse or 
>> cliff front-end to the SDK methods located in the (for example) 
>> openstack.client.api.compute
>> 
>> * The SDK would handle Auth, re-auth (expired tokens, etc) for long-lived 
>> clients - all of the openstack.client.api.** classes would accept an Auth 
>> object to delegate management / mocking of the Auth / service catalog stuff 
>> to. This means developers building applications (say for example, horizon) 
>> don’t need to worry about token/expired authentication/etc.
>> 
>> * Simplify the dependency graph & code for the existing tools to enable 
>> single binary installs (py2exe, py2app, etc) for end users of the command 
>> line tools.
>> 
>> Short version: if a developer wants to consume an openstack cloud; the would 
>> have a single SDK with minimal dependencies and import from a single 
>> namespace. An example application might look like:
>> 
>> from openstack.api import AuthV2
>> from openstack.api import ComputeV2
>> 
>> myauth = AuthV2(…., connect=True)
>> compute = ComputeV2(myauth)
>> 
>> compute.list_flavors()
> 
> I know this is an example but, could we leave the version out of the
> class name? Having something like:
> 
> from openstack.api.v2 import Compute
> 
>   or
> 
> from openstack.compute.v2 import Instance
> 
> (just made that up)
> 
> for marconi we're using the later.

Definitely; it should be based on namespaces. 

> 
>> This greatly improves the developer experience both internal to openstack 
>> and externally. Currently OpenStack has 22+ (counting stackforge) potential 
>> libraries a developer may need to install to use a full deployment of 
>> OpenStack:
>> 
>> * python-keystoneclient (identity)
>> * python-glanceclient (image)
>> * python-novaclient (compute)
>> * python-troveclient (database)
>> * python-neutronclient (network)
>> * python-ironicclient (bare metal)
>> * python-heatclient (orchestration)
>> * python-cinderclient (block storage)
>> * python-ceilometerclient (telemetry, metrics & billing)
>> * python-swiftclient (object storage)
>> * python-savannaclient (big data)
>> * python-openstackclient (meta client package)
>> * python-marconiclient (queueing)
>> * python-tuskarclient (tripleo / management)
>> * python-melangeclient (dead)
>> * python-barbicanclient (secrets)
>> * python-solumclient (ALM)
>> * python-muranoclient (application catalog)
>> * python-manilaclient (shared filesystems)
>> * python-libraclient (load 

Re: [openstack-dev] a "common" client library

[Jesse Noller]

On Jan 15, 2014, at 6:16 PM, Doug Hellmann  wrote:

> 
> 
> 
> On Wed, Jan 15, 2014 at 5:39 PM, Dean Troyer  wrote:
> On Wed, Jan 15, 2014 at 1:37 PM, Doug Hellmann  
> wrote:
> Several people have mentioned to me that they are interested in, or actively 
> working on, code related to a "common" client library -- something meant to 
> be reused directly as a basis for creating a common library for all of the 
> openstack clients to use. There's a blueprint [1] in oslo, and I believe the 
> keystone devs and unified CLI teams are probably interested in ensuring that 
> the resulting API ends up meeting all of our various requirements.
> 
> I would love to see a bit more detail on the structure of the lib(s), the 
> blueprint really doesn't discuss the design/organization/intended API of the 
> libs.  For example, I would hope the distinction between the various layers 
> of a client stack don't get lost, i.e. not mixing the low-level REST API bits 
> with the higher-level CLI parsers and decorators.
> 
> Does the long-term goals include a common caching layer?
> 
> Good questions. I believe the current effort referenced by the blueprint is 
> focusing on moving existing code into the incubator for reuse, to make it 
> easier to restructure later. Alexei, do I have that correct?
> 
> I was starting to feel like an (unnecessary) bottleneck, so my goal for this 
> email thread was to step out of the middle of the conversation, and make sure 
> everyone who is doing the work knows about each other so you can all 
> coordinate directly. 
> 
> If we need more details for requirements and a design, let's start writing 
> them up. Maybe a wiki page or etherpad to get started, and turn that into 
> blueprints when ready? Who wants to take that on?

I’ll take that on: If I schedule an IRC meeting next week; who would like to 
join?

> 
> Doug
> 
>  
> 
> dt
> 
> -- 
> 
> Dean Troyer
> dtro...@gmail.com
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Jesse Noller]

On Jan 15, 2014, at 4:55 PM, "Renat Akhmerov" 
mailto:rakhme...@mirantis.com>> wrote:

Great idea, fully support it. We’re interested in that too. One specific thing 
that was mentioned is the ability to mock auth service seems to be very useful 
for some test scenarios, we came across that recently.

Renat Akhmerov
@ Mirantis Inc.

I'd add that verified fakes/mocks/stubs for all network traffic calls are 
critical for end users wanting to build/test code against the common code/sdk 
so they don't need a fully running OpenStack install that incurs charges. For 
fast unit test it's critical



On 15 Jan 2014, at 14:07, Sylvain Bauza 
mailto:sylvain.ba...@gmail.com>> wrote:

Hi Doug,

Count me in. Climate is currently working on delivering its first 
python-climateclient but it would be great if we could leverage any olso lib 
for this.

-Sylvain


2014/1/15 Doug Hellmann 
mailto:doug.hellm...@dreamhost.com>>
Several people have mentioned to me that they are interested in, or actively 
working on, code related to a "common" client library -- something meant to be 
reused directly as a basis for creating a common library for all of the 
openstack clients to use. There's a blueprint [1] in oslo, and I believe the 
keystone devs and unified CLI teams are probably interested in ensuring that 
the resulting API ends up meeting all of our various requirements.

If you're interested in this effort, please subscribe to the blueprint and use 
that to coordinate efforts so we don't produce more than one common library. ;-)

Thanks,
Doug


[1] https://blueprints.launchpad.net/oslo/+spec/common-client-library-2

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Jesse Noller]

On Jan 15, 2014, at 1:37 PM, Doug Hellmann  wrote:

> Several people have mentioned to me that they are interested in, or actively 
> working on, code related to a "common" client library -- something meant to 
> be reused directly as a basis for creating a common library for all of the 
> openstack clients to use. There's a blueprint [1] in oslo, and I believe the 
> keystone devs and unified CLI teams are probably interested in ensuring that 
> the resulting API ends up meeting all of our various requirements.
> 
> If you're interested in this effort, please subscribe to the blueprint and 
> use that to coordinate efforts so we don't produce more than one common 
> library. ;-)
> 
> Thanks,
> Doug
> 
> 
> [1] https://blueprints.launchpad.net/oslo/+spec/common-client-library-2

*raises hand* 

Me me! 

I’ve been talking to many contributors about the Developer Experience stuff I 
emailed out prior to the holidays and I was starting blueprint work, but this 
is a great pointer. I’m going to have to sync up with Alexei.

I think solving this for openstack developers and maintainers as the blueprint 
says is a big win in terms of code reuse / maintenance and consistent but more 
so for *end-user developers* consuming openstack clouds.

Some background - there’s some terminology mismatch but the rough idea is the 
same:

* A centralized “SDK” (Software Development Kit) would be built condensing the 
common code and logic and operations into a single namespace.

* This SDK would be able to be used by “downstream” CLIs - essentially the CLIs 
become a specialized front end - and in some cases, only an argparse or cliff 
front-end to the SDK methods located in the (for example) 
openstack.client.api.compute 

* The SDK would handle Auth, re-auth (expired tokens, etc) for long-lived 
clients - all of the openstack.client.api.** classes would accept an Auth 
object to delegate management / mocking of the Auth / service catalog stuff to. 
This means developers building applications (say for example, horizon) don’t 
need to worry about token/expired authentication/etc. 

* Simplify the dependency graph & code for the existing tools to enable single 
binary installs (py2exe, py2app, etc) for end users of the command line tools.

Short version: if a developer wants to consume an openstack cloud; the would 
have a single SDK with minimal dependencies and import from a single namespace. 
An example application might look like:

from openstack.api import AuthV2
from openstack.api import ComputeV2

myauth = AuthV2(…., connect=True)
compute = ComputeV2(myauth)

compute.list_flavors()

This greatly improves the developer experience both internal to openstack and 
externally. Currently OpenStack has 22+ (counting stackforge) potential 
libraries a developer may need to install to use a full deployment of OpenStack:

  * python-keystoneclient (identity)
  * python-glanceclient (image)
  * python-novaclient (compute)
  * python-troveclient (database)
  * python-neutronclient (network)
  * python-ironicclient (bare metal)
  * python-heatclient (orchestration)
  * python-cinderclient (block storage)
  * python-ceilometerclient (telemetry, metrics & billing)
  * python-swiftclient (object storage)
  * python-savannaclient (big data)
  * python-openstackclient (meta client package)
  * python-marconiclient (queueing)
  * python-tuskarclient (tripleo / management)
  * python-melangeclient (dead)
  * python-barbicanclient (secrets)
  * python-solumclient (ALM)
  * python-muranoclient (application catalog)
  * python-manilaclient (shared filesystems)
  * python-libraclient (load balancers)
  * python-climateclient (reservations)
  * python-designateclient (Moniker/DNS)

If you exclude the above and look on PyPI:

On PyPi (client libraries/SDKs only, excluding the above - not maintained by 
openstack):

 * hpcloud-auth-openstack 1.0
 * python-openstackclient 0.2.2
 * rackspace-auth-openstack 1.1
 * posthaste 0.2.2
 * pyrax 1.6.2
 * serverherald 0.0.1
 * warm 0.3.1
 * vaporize 0.3.2
 * swiftsc (https://github.com/mkouhei/swiftsc)
 * bookofnova 0.007
 * nova-adminclient 0.1.8
 * python-quantumclient 2.2.4.3
 * python-stackhelper 0.0.7.1.gcab1eb0
 * swift-bench 1.0
 * swiftly 1.12
 * txAWS 0.2.3 
 * cfupload 0.5.1
 * python-reddwarfclient 0.1.2
 * python-automationclient 1.2.1
 * rackspace-glanceclient 0.9
 * rackspace-novaclient 1.4

If you ignore PyPI and just want to install the base say - 7 services, each one 
of the python-** clients has its own dependency tree and may or may not build 
from one of the others. If a vendor wants to extend any of them, it’s basically 
a fork instead of a clean plugin system.

On the CLI front - this would *greatly* simplify the work openstackclient needs 
to do - it would be able to import from the main SDK and simply provide the 
noun-verb command line and any other end-user sugar it wanted to. Even if each 
service wanted to keep its own python-X client instead of relying on 
openstackclient it wou

Re: [openstack-dev] How to best make User Experience a priority in every project

[Jesse Noller]

On Nov 20, 2013, at 9:09 AM, Thierry Carrez  wrote:

> Hi everyone,
> 
> How should we proceed to make sure UX (user experience) is properly
> taken into account into OpenStack development ? Historically it was hard
> for UX sessions (especially the ones that affect multiple projects, like
> CLI / API experience) to get session time at our design summits. This
> visibility issue prompted the recent request by UX-minded folks to make
> UX an official OpenStack program.
> 
> However, as was apparent in the Technical Committee meeting discussion
> about it yesterday, most of us are not convinced that establishing and
> blessing a separate team is the most efficient way to give UX the
> attention it deserves. Ideally, UX-minded folks would get active
> *within* existing project teams rather than form some sort of
> counter-power as a separate team. In the same way we want scalability
> and security mindset to be present in every project, we want UX to be
> present in every project. It's more of an advocacy group than a
> "program" imho.
> 
> So my recommendation would be to encourage UX folks to get involved
> within projects and during project-specific weekly meetings to
> efficiently drive better UX there, as a direct project contributor. If
> all the UX-minded folks need a forum to coordinate, I think [UX] ML
> threads and, maybe, a UX weekly meeting would be an interesting first step.
> 
> There would still be an issue with UX session space at the Design
> Summit... but that's a well known issue that affects more than just UX:
> the way our design summits were historically organized (around programs
> only) made it difficult to discuss cross-project and cross-program
> issues. To address that, the plan is to carve cross-project space into
> the next design summit, even if that means a little less topical
> sessions for everyone else.
> 
> Thoughts ?

Hello again everyone - let me turn this around a little bit, I’m working on 
proposing something based on the Oslo work and openstack-client, and overall 
looking at the *Developer Experience* focused around application developers and 
end-users more so than the individual UX issues (configuration, UI, IxD, etc).

I’ve spoken to Everett and others about discussions had at the summit around 
ideas like developer.openstack.org - and I think the idea is a good start 
towards improving the lives of downstream application developers. However, one 
of the problems (as I and others see it) is that there’s a series of 
disconnects between the needs of the individual projects to have a command line 
client for administrative / basic usage and the needs of application developers 
and end-users (not Openstack admins, just end users).

What I’d like to propose is a team that’s not focused on the overarching UX 
(from horizon to **) but rather a team / group focused on some key areas:

1: Creating an *application developer* focused SDK for openstack services 
2: Unifying the back-end code and common tools for the command line clients 
into 1 
3: Providing extension points for downstream vendors to add custom extensions 
as needed
4: Based on 1; make deriving project-specific CLIs a matter of 
importing/subclassing and extending 

This is a bit of a hybrid between what the awesome openstackclient team has 
done to make a unified CLI, but takes a step back to focus on a unified back 
end with clean APIs that can not only power CLIs, but also act as an SDK. This 
would allow many vendors (Rackspace, for example) to willingly drop their SDKs 
and leverage this unified back end.

In my “perfect world” you’d be able to, as an application developer targeting 
Openstack providers, do something close to (code sketch):

from openstack.api.auth import AuthClass
from openstack.api.nova import NovaClient
from openstack.api.nova import NovaAdmin

auth = AuthClass(…)

nova = NovaClient(auth)
nova.server.create(… block=True)

nova_admin = NovaAdmin(auth)
nova_admin.delete_flavor(…)

Downstream vendors could further extend each of these and either create very 
thin shims or meta packages that add provider specific services, e.g:

from openstack.vendor.rackspace.api.auth AuthClass 

…

The end goals being:

1: provide a common rest client back end for all the things
2: Collapse all common functions (such as error retries) into a common lib
3: DO NOT DICTATE a concurrency system: no eventlet, no greenlet. Just Python; 
allow application developers to use what they need to.
4: Provide a cliff based extension system for vendors 
5: Document everything.
6: Python 3 & 2 compatible code base

As I said earlier; this would build on work already in flight within openstack, 
and additionally within vendors such as rackspace to contribute to this effort 
directly and reduce the proliferation of SDKs/clis/etc. Existing SDKs could be 
end-of-lifed. The team working on this would be comprised of people focused on 
working across the openstack projects not just as dictators of supreme design, 
but actua

Re: [openstack-dev] How to best make User Experience a priority in every project

[Jesse Noller]

> On Nov 21, 2013, at 10:43 AM, "Ben Nemec"  wrote:
> 
>> On 2013-11-21 10:20, Jesse Noller wrote:
>>> On Nov 20, 2013, at 9:09 AM, Thierry Carrez  wrote:
>>> Hi everyone,
>>> How should we proceed to make sure UX (user experience) is properly
>>> taken into account into OpenStack development ? Historically it was hard
>>> for UX sessions (especially the ones that affect multiple projects, like
>>> CLI / API experience) to get session time at our design summits. This
>>> visibility issue prompted the recent request by UX-minded folks to make
>>> UX an official OpenStack program.
>>> However, as was apparent in the Technical Committee meeting discussion
>>> about it yesterday, most of us are not convinced that establishing and
>>> blessing a separate team is the most efficient way to give UX the
>>> attention it deserves. Ideally, UX-minded folks would get active
>>> *within* existing project teams rather than form some sort of
>>> counter-power as a separate team. In the same way we want scalability
>>> and security mindset to be present in every project, we want UX to be
>>> present in every project. It's more of an advocacy group than a
>>> "program" imho.
>>> So my recommendation would be to encourage UX folks to get involved
>>> within projects and during project-specific weekly meetings to
>>> efficiently drive better UX there, as a direct project contributor. If
>>> all the UX-minded folks need a forum to coordinate, I think [UX] ML
>>> threads and, maybe, a UX weekly meeting would be an interesting first step.
>>> There would still be an issue with UX session space at the Design
>>> Summit... but that's a well known issue that affects more than just UX:
>>> the way our design summits were historically organized (around programs
>>> only) made it difficult to discuss cross-project and cross-program
>>> issues. To address that, the plan is to carve cross-project space into
>>> the next design summit, even if that means a little less topical
>>> sessions for everyone else.
>>> Thoughts ?
>> Hello again everyone - let me turn this around a little bit, I’m
>> working on proposing something based on the Oslo work and
>> openstack-client, and overall looking at the *Developer Experience*
>> focused around application developers and end-users more so than the
>> individual UX issues (configuration, UI, IxD, etc).
>> I’ve spoken to Everett and others about discussions had at the summit
>> around ideas like developer.openstack.org - and I think the idea is a
>> good start towards improving the lives of downstream application
>> developers. However, one of the problems (as I and others see it) is
>> that there’s a series of disconnects between the needs of the
>> individual projects to have a command line client for administrative /
>> basic usage and the needs of application developers and end-users (not
>> Openstack admins, just end users).
>> What I’d like to propose is a team that’s not focused on the
>> overarching UX (from horizon to **) but rather a team / group focused
>> on some key areas:
>> 1: Creating an *application developer* focused SDK for openstack services
>> 2: Unifying the back-end code and common tools for the command line
>> clients into 1
>> 3: Providing extension points for downstream vendors to add custom
>> extensions as needed
>> 4: Based on 1; make deriving project-specific CLIs a matter of
>> importing/subclassing and extending
>> This is a bit of a hybrid between what the awesome openstackclient
>> team has done to make a unified CLI, but takes a step back to focus on
>> a unified back end with clean APIs that can not only power CLIs, but
>> also act as an SDK. This would allow many vendors (Rackspace, for
>> example) to willingly drop their SDKs and leverage this unified back
>> end.
>> In my “perfect world” you’d be able to, as an application developer
>> targeting Openstack providers, do something close to (code sketch):
>> from openstack.api.auth import AuthClass
>> from openstack.api.nova import NovaClient
>> from openstack.api.nova import NovaAdmin
>> auth = AuthClass(…)
>> nova = NovaClient(auth)
>> nova.server.create(… block=True)
>> nova_admin = NovaAdmin(auth)
>> nova_admin.delete_flavor(…)
>> Downstream vendors could further extend each of these and either
>> create very thin shims or meta packages that add provider specific
>> services, e.g:
>> from openstack.vendor.rackspace.api.auth AuthClass
&g

Re: [openstack-dev] How to best make User Experience a priority in every project

[Jesse Noller]

On Nov 20, 2013, at 10:27 AM, Thierry Carrez  wrote:

> Anne Gentle wrote:
>> It's nigh-impossible with the UX resources there now (four core) for
>> them to attend all the project meetings with an eye to UX. Docs are in a
>> similar situation. We also want docs to be present in every project.
>> Docs as a program makes sense, and to me, UX as a program makes sense as
>> well. The UX program can then prioritize what to focus on with the
>> resources they have. 
> 
> The key difference between docs and UX is that documentation is a
> separate deliverable, and is reviewed by the docs core team. UX work
> ends up in each project's code, and gets reviewed by the project's core
> team, not the UX team.
> 
> Blessing a separate team with no connection with the project core team
> is imho a recipe for disaster, potentially with tension between a team
> that recommends work to be done and a team that needs to actually get
> the work coded, reviewed and merged.
> 
> That's why I see UX more like security or scalability, than like
> documentation. A design goal rather than a deliverable. And design goals
> need to be baked in the team that ends up writing and reviewing the
> code. Making it separate will just make it less efficient.

What if “the UX” team was not focused on dictating standards; but rather 
outlining them and then directly working with the individual teams to actually 
do the work? This is something I’ve been actually thinking about heavily and 
planned on typing up a proposal on, primarily focused on SDK / API / CLI 
consistency and developer experience.

This is something near and dear to my heart - I’ve actually been talking to 
Doug Hellmann and others about this and trying to find a balance between the 
competing forces you describe: I don’t think a stand alone team that doesn’t 
write the actual code/backend in conjunction with the individual projects can 
work; but I do feel there needs to be a unified vision and common plan held 
together by a team focused on this subject.

So in *my* ideal world: the “ux” team for this would be a mixture of individual 
project members, people solely focused on this but writing code *and* 
specifications and generally working across teams.

Jesse
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev