Re: [Openstack] API Spec

2011-08-27 Thread Michael Shuler
Preface: I am not very familiar with Nova.

On 08/26/2011 01:19 PM, Devin Carlen wrote:
 I believe we should have had a Rackspace API module just like we have
 an EC2 API module.  Then the OpenStack API wouldn't have been
 burdened by the historical decisions around the Rackspace API.
 
 But that is ancient history at this point.

So that I can better understand, as well as for others that may not
know, would it be possible to get a basic explanation of why it is
hard to modify Nova, now, to modularize the Rackspace API?

basic == non-technical highlights in simple english, please.  This
sounds to me like a bad architectural decision that people are living
with - so, why not admit that and make it a priority to change?

-- 
Kind regards,
Michael

snipped it's been over and year and it is hard sounding excuses(?)

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp


Re: [Openstack] API Spec

2011-08-27 Thread George Reese
The Rackspace compute API is actually one of the better cloud APIs out there. 
It's a pretty damn good basis for an OpenStack Nova API. 

EC2, on the other hand, is a pretty obnoxious API and I've already outlined my 
thoughts on its role as a defacto API on my blog.

Having said that, there is basically no difference between the OpenStack API 
and a Rackspace API that has not changed in 2 years. In other words, there's 
been no attempt to improve the API based on two year's of learning, no attempt 
to step back and model the problem of cloud computing in general, it uses some 
goofy Rackspace terminology like Flavor, and there's been no attempt to 
expose the features of OpenStack that don't exist in the current Rackspace API.

The biggest difference is authentication. That's been left up to the 
implementor. Yuck! Pick an authentication scheme and live with it!

The most important thing right now would not be to re-write the API, but to 
create specs for other parts of OpenStack like volumes and integrate them into 
the structure of the current API. It shouldn't be too hard. I absolutely don't 
understand why, after a year of work, all we have is clone of the Rackspace API 
labeled 1.1.

-George

On Aug 27, 2011, at 12:34 PM, Michael Shuler wrote:

 Preface: I am not very familiar with Nova.
 
 On 08/26/2011 01:19 PM, Devin Carlen wrote:
 I believe we should have had a Rackspace API module just like we have
 an EC2 API module.  Then the OpenStack API wouldn't have been
 burdened by the historical decisions around the Rackspace API.
 
 But that is ancient history at this point.
 
 So that I can better understand, as well as for others that may not
 know, would it be possible to get a basic explanation of why it is
 hard to modify Nova, now, to modularize the Rackspace API?
 
 basic == non-technical highlights in simple english, please.  This
 sounds to me like a bad architectural decision that people are living
 with - so, why not admit that and make it a priority to change?
 
 -- 
 Kind regards,
 Michael
 
 snipped it's been over and year and it is hard sounding excuses(?)
 
 ___
 Mailing list: https://launchpad.net/~openstack
 Post to : openstack@lists.launchpad.net
 Unsubscribe : https://launchpad.net/~openstack
 More help   : https://help.launchpad.net/ListHelp

--
George Reese - Chief Technology Officer, enStratus
e: george.re...@enstratus.comt: @GeorgeReesep: +1.207.956.0217f: 
+1.612.338.5041
enStratus: Governance for Public, Private, and Hybrid Clouds - @enStratus - 
http://www.enstratus.com
To schedule a meeting with me: http://tungle.me/GeorgeReese



smime.p7s
Description: S/MIME cryptographic signature
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp


Re: [Openstack] API Spec

2011-08-27 Thread George Reese
A cloud platform simply isn't functional without an API. It is a core 
requirement.

No API, no cloud.

-George

On Aug 27, 2011, at 7:04 PM, Tim Bell wrote:

 I'm also a non-API expert but getting a stable open cloud engine with a 
 reasonable API would seem to be a good target before we look to enhance it.
 
 There are lots of potential users of Nova (including Rackspace) who would 
 like to get Nova into production.  An API will fully exploits all of the 
 underlying functionality should be discussed/planned in the longer term but 
 let's get Diablo out and deployable first. 
 
 Tim Bell
 CERN
 
 
 ___
 Mailing list: https://launchpad.net/~openstack
 Post to : openstack@lists.launchpad.net
 Unsubscribe : https://launchpad.net/~openstack
 More help   : https://help.launchpad.net/ListHelp

--
George Reese - Chief Technology Officer, enStratus
e: george.re...@enstratus.comt: @GeorgeReesep: +1.207.956.0217f: 
+1.612.338.5041
enStratus: Governance for Public, Private, and Hybrid Clouds - @enStratus - 
http://www.enstratus.com
To schedule a meeting with me: http://tungle.me/GeorgeReese



smime.p7s
Description: S/MIME cryptographic signature
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp


Re: [Openstack] API Spec

2011-08-27 Thread Tim Bell
I have an api, diablo nova v1.1.

What we are talking about is if it covers 100% functionality.

I can start my deployment testing with v1.1.  The limiting factor is not v1.1 
vs v1.x for most sites. It is packaging, user exits and integration, not 
whether feature X is in the latest API.

Tim.

- Reply message -
From: George Reese george.re...@enstratus.com
To: Tim Bell tim.b...@cern.ch
Cc: openstack@lists.launchpad.net openstack@lists.launchpad.net
Subject: [Openstack] API Spec
Date: Sat, Aug 27, 2011 20:47



A cloud platform simply isn't functional without an API. It is a core 
requirement.

No API, no cloud.

-George

On Aug 27, 2011, at 7:04 PM, Tim Bell wrote:

 I'm also a non-API expert but getting a stable open cloud engine with a 
 reasonable API would seem to be a good target before we look to enhance it.

 There are lots of potential users of Nova (including Rackspace) who would 
 like to get Nova into production.  An API will fully exploits all of the 
 underlying functionality should be discussed/planned in the longer term but 
 let's get Diablo out and deployable first.

 Tim Bell
 CERN


 ___
 Mailing list: https://launchpad.net/~openstack
 Post to : openstack@lists.launchpad.net
 Unsubscribe : https://launchpad.net/~openstack
 More help   : https://help.launchpad.net/ListHelp

--
George Reese - Chief Technology Officer, enStratus
e: george.re...@enstratus.comt: @GeorgeReesep: +1.207.956.0217f: 
+1.612.338.5041
enStratus: Governance for Public, Private, and Hybrid Clouds - @enStratus - 
http://www.enstratus.com
To schedule a meeting with me: http://tungle.me/GeorgeReese

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp


Re: [Openstack] API Spec

2011-08-27 Thread Bryan Taylor


On 8/26/11 1:19 PM, Devin Carlen devin.car...@gmail.com wrote:

There have been a lot of efforts lately to bring the feature set of the
OpenStack API in line with the EC2 API, and this is admirable.  But this
has NOT been happening at the architect level.  This has been happening
at the developer level, and it is being done with API extensions which
make it sound like the feature is somehow not complete or not supported
fully, because it's not part of the core API.

I think you are hitting on a perception problem around extensions that
might be solvable by taking some steps to make the intent clear. I propose
two things: always keep the next version of the API in play and improve
the classification of the extensions.

Developers have to choose to contribute in a way that either works with
the current APIs or not. Mixing these is bad, and fortunately unnecessary.
The two tracks should probably have different tech leads.

How about this as classification system for extensions:

temporary extension ­ accepted for addition to core api in the next
backwards compatible version uprev
backport extension ‹ accepted for addition to core api in the next
non-backwards compatible version uprev
default extension  ­ viewed as standard, but not incorporated into the
core to allow opt-out by an operator
common extension ‹  viewed as the preferred solution, but must be an
opt-in by an operator
candidate extension ‹ an extension viewed as trying to prove itself worthy
for a higher level
private extension ‹ an extension not offered or not accepted for a higher
level

Thoughts?

This email may include confidential information. If you received it in error, 
please delete it.


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp


Re: [Openstack] API Spec

2011-08-27 Thread Jonathan Bryce
This is on the agenda for Tuesday's policy board meeting (in #openstack-meeting 
1 hour before the weekly OpenStack team meeting for those interested). Sounds 
like a potentially acceptable solution is to set some cross-project API 
standards and then push the remainder of the API definition and implementation 
into each project. Then the API can progress with the underlying project's 
features as the developers see fit.

Jonathan.


On Aug 27, 2011, at 4:16 PM, Tim Bell wrote:

 I have an api, diablo nova v1.1.
 
 What we are talking about is if it covers 100% functionality.
 
 I can start my deployment testing with v1.1.  The limiting factor is not v1.1 
 vs v1.x for most sites. It is packaging, user exits and integration, not 
 whether feature X is in the latest API.
 
 Tim.
 
 - Reply message -
 From: George Reese george.re...@enstratus.com
 To: Tim Bell tim.b...@cern.ch
 Cc: openstack@lists.launchpad.net openstack@lists.launchpad.net
 Subject: [Openstack] API Spec
 Date: Sat, Aug 27, 2011 20:47
 
 
 
 A cloud platform simply isn't functional without an API. It is a core 
 requirement.
 
 No API, no cloud.
 
 -George
 
 On Aug 27, 2011, at 7:04 PM, Tim Bell wrote:
 
  I'm also a non-API expert but getting a stable open cloud engine with a 
  reasonable API would seem to be a good target before we look to enhance it.
  
  There are lots of potential users of Nova (including Rackspace) who would 
  like to get Nova into production.  An API will fully exploits all of the 
  underlying functionality should be discussed/planned in the longer term but 
  let's get Diablo out and deployable first. 
  
  Tim Bell
  CERN
  
  
  ___
  Mailing list: https://launchpad.net/~openstack
  Post to : openstack@lists.launchpad.net
  Unsubscribe : https://launchpad.net/~openstack
  More help   : https://help.launchpad.net/ListHelp
 
 --
 George Reese - Chief Technology Officer, enStratus
 e: george.re...@enstratus.comt: @GeorgeReesep: +1.207.956.0217f: 
 +1.612.338.5041
 enStratus: Governance for Public, Private, and Hybrid Clouds - @enStratus - 
 http://www.enstratus.com
 To schedule a meeting with me: http://tungle.me/GeorgeReese
 
 ___
 Mailing list: https://launchpad.net/~openstack
 Post to : openstack@lists.launchpad.net
 Unsubscribe : https://launchpad.net/~openstack
 More help   : https://help.launchpad.net/ListHelp

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp


Re: [Openstack] API Spec

2011-08-27 Thread George Reese
I don't necessarily agree with that approach. I'm not sure if that's the way 
AWS does things or not, but you will note that the AWS APIs are very fragmented 
across projects. 

I think there are several principles that may at times be in conflict that need 
to be in place:

* Any GA feature should be exposed via API at the time of its GA-ness.
* There needs to be a gatekeeper (possibly the wrong word) ensuring that the 
APIs are self-consistent


And the understanding should exist that modeling something for functionality 
may not be the same as the way it is modeled in the API. In fact, the 
underlying model will likely be refactored many times by the API must be 
timeless (but evolvable).

-George

On Aug 28, 2011, at 2:29 AM, Jonathan Bryce wrote:

 This is on the agenda for Tuesday's policy board meeting (in 
 #openstack-meeting 1 hour before the weekly OpenStack team meeting for those 
 interested). Sounds like a potentially acceptable solution is to set some 
 cross-project API standards and then push the remainder of the API definition 
 and implementation into each project. Then the API can progress with the 
 underlying project's features as the developers see fit.
 
 Jonathan.
 
 
 On Aug 27, 2011, at 4:16 PM, Tim Bell wrote:
 
 I have an api, diablo nova v1.1.
 
 What we are talking about is if it covers 100% functionality.
 
 I can start my deployment testing with v1.1.  The limiting factor is not 
 v1.1 vs v1.x for most sites. It is packaging, user exits and integration, 
 not whether feature X is in the latest API.
 
 Tim.
 
 - Reply message -
 From: George Reese george.re...@enstratus.com
 To: Tim Bell tim.b...@cern.ch
 Cc: openstack@lists.launchpad.net openstack@lists.launchpad.net
 Subject: [Openstack] API Spec
 Date: Sat, Aug 27, 2011 20:47
 
 
 
 A cloud platform simply isn't functional without an API. It is a core 
 requirement.
 
 No API, no cloud.
 
 -George
 
 On Aug 27, 2011, at 7:04 PM, Tim Bell wrote:
 
  I'm also a non-API expert but getting a stable open cloud engine with a 
  reasonable API would seem to be a good target before we look to enhance it.
  
  There are lots of potential users of Nova (including Rackspace) who would 
  like to get Nova into production.  An API will fully exploits all of the 
  underlying functionality should be discussed/planned in the longer term 
  but let's get Diablo out and deployable first. 
  
  Tim Bell
  CERN
  
  
  ___
  Mailing list: https://launchpad.net/~openstack
  Post to : openstack@lists.launchpad.net
  Unsubscribe : https://launchpad.net/~openstack
  More help   : https://help.launchpad.net/ListHelp
 
 --
 George Reese - Chief Technology Officer, enStratus
 e: george.re...@enstratus.comt: @GeorgeReesep: +1.207.956.0217f: 
 +1.612.338.5041
 enStratus: Governance for Public, Private, and Hybrid Clouds - @enStratus - 
 http://www.enstratus.com
 To schedule a meeting with me: http://tungle.me/GeorgeReese
 
 ___
 Mailing list: https://launchpad.net/~openstack
 Post to : openstack@lists.launchpad.net
 Unsubscribe : https://launchpad.net/~openstack
 More help   : https://help.launchpad.net/ListHelp
 
 ___
 Mailing list: https://launchpad.net/~openstack
 Post to : openstack@lists.launchpad.net
 Unsubscribe : https://launchpad.net/~openstack
 More help   : https://help.launchpad.net/ListHelp

--
George Reese - Chief Technology Officer, enStratus
e: george.re...@enstratus.comt: @GeorgeReesep: +1.207.956.0217f: 
+1.612.338.5041
enStratus: Governance for Public, Private, and Hybrid Clouds - @enStratus - 
http://www.enstratus.com
To schedule a meeting with me: http://tungle.me/GeorgeReese



smime.p7s
Description: S/MIME cryptographic signature
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp


Re: [Openstack] API Spec

2011-08-26 Thread Anne Gentle

 That analysis omits a very important third party: users of the API.


+1

Users of the API are one of the audiences for docs.openstack.org. Another
audience is system admins standing up clouds.

I'd like to revise the docs landing page to better serve users of all of the
OpenStack APIs - Compute, Object Storage, Keystone, Image Service. Input is
welcome. I'm tracking moving the Image Service API doc (source and
presentation) via a blueprint:
https://blueprints.launchpad.net/openstack-manuals/+spec/glance-api-doc.

I'm also working on a consistent information architecture for all OpenStack
projects, where certain information is known to reside in particular areas.

For presentation of information, we have to take a bigger picture view here.
Where the source is stored is less important to me, but review is very
important.

Just wanted to let people know we're not losing sight of users.

Anne
*Anne Gentle*
a...@openstack.org
 my blog http://justwriteclick.com/ | my
bookhttp://xmlpress.net/publications/conversation-community/|
LinkedIn http://www.linkedin.com/in/annegentle |
Delicioushttp://del.icio.us/annegentle|
Twitter http://twitter.com/annegentle
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp


Re: [Openstack] API Spec

2011-08-26 Thread Mark Collier
+1




On 8/26/11 1:19 PM, Devin Carlen devin.car...@gmail.com wrote:

Hey all,

I've been following the code vs architect debate that's been unfolding
over the past week or so.  Here are some of the problems I've seen from
my point of view.  Fundamentally, the process we have now for defining
API specs is broken.  I don't believe that this can be argued.  The first
major misstep in my opinion was forcing backwards compatibility with the
Rackspace API in the OpenStack API.

I believe we should have had a Rackspace API module just like we have an
EC2 API module.  Then the OpenStack API wouldn't have been burdened by
the historical decisions around the Rackspace API.

But that is ancient history at this point.

But we have to look at this pragmatically, and realize that 1 year later,
the OpenStack API (as spec'd) is still not even close to exposing the
underlying core functionality that exists within Nova.  For the most
part, the OpenStack API is a subset of functionality of the EC2 API.
This is a big reason why the Dashboard project used the EC2 API for its
underlying communication for so long.

There have been a lot of efforts lately to bring the feature set of the
OpenStack API in line with the EC2 API, and this is admirable.  But this
has NOT been happening at the architect level.  This has been happening
at the developer level, and it is being done with API extensions which
make it sound like the feature is somehow not complete or not supported
fully, because it's not part of the core API.

So the question to all in favor of architecting up front:

How do you explain lacking feature parity with the underlying components
for over a year now?


In my opinion, this has been a big problem in gaining traction around the
OpenStack API.



Devin


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

This email may include confidential information. If you received it in error, 
please delete it.


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp


Re: [Openstack] API Spec

2011-08-26 Thread Soren Hansen
2011/8/25 Jorge Williams jorge.willi...@rackspace.com:
 On Aug 24, 2011, at 12:02 PM, Soren Hansen wrote:
 2011/8/24 Jorge Williams jorge.willi...@rackspace.com:
 Okay, I do remember these, but the talks focus on documenting what the
 implementation does.  I'm all for it, docstrings are good, but that's not
 what I usually think of a specification.  So  we are having a semantic
 disconnect here

Indeed.

docstrings can have any format, language, verbosity. It's a string.
You can put as much or as little stuff in them as you please (if you
have one at all!). Anything you can possibly think of putting in what
you refer to as an API spec, you can put in a docstring.

 OpenStack was created to stop every company wanting to do offer cloud
 computing from reinventing everything and have everyone work
 together.  What's the use of competing implementations? We'll be
 right back to square one.
 Because whether you like it or not there are competing vendors...

I don't think making our lives more difficult to make our competitor's
easier is a way a strategy we'll get very far with, to be honest.

-- 
Soren Hansen        | http://linux2go.dk/
Ubuntu Developer    | http://www.ubuntu.com/
OpenStack Developer | http://www.openstack.org/

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp


Re: [Openstack] API Spec

2011-08-26 Thread Dolph Mathews
++

-Dolph


On 08/26/2011 02:46 PM, Mark Collier wrote:
 +1




 On 8/26/11 1:19 PM, Devin Carlendevin.car...@gmail.com  wrote:

 Hey all,

 I've been following the code vs architect debate that's been unfolding
 over the past week or so.  Here are some of the problems I've seen from
 my point of view.  Fundamentally, the process we have now for defining
 API specs is broken.  I don't believe that this can be argued.  The first
 major misstep in my opinion was forcing backwards compatibility with the
 Rackspace API in the OpenStack API.

 I believe we should have had a Rackspace API module just like we have an
 EC2 API module.  Then the OpenStack API wouldn't have been burdened by
 the historical decisions around the Rackspace API.

 But that is ancient history at this point.

 But we have to look at this pragmatically, and realize that 1 year later,
 the OpenStack API (as spec'd) is still not even close to exposing the
 underlying core functionality that exists within Nova.  For the most
 part, the OpenStack API is a subset of functionality of the EC2 API.
 This is a big reason why the Dashboard project used the EC2 API for its
 underlying communication for so long.

 There have been a lot of efforts lately to bring the feature set of the
 OpenStack API in line with the EC2 API, and this is admirable.  But this
 has NOT been happening at the architect level.  This has been happening
 at the developer level, and it is being done with API extensions which
 make it sound like the feature is somehow not complete or not supported
 fully, because it's not part of the core API.

 So the question to all in favor of architecting up front:

 How do you explain lacking feature parity with the underlying components
 for over a year now?


 In my opinion, this has been a big problem in gaining traction around the
 OpenStack API.



 Devin


 ___
 Mailing list: https://launchpad.net/~openstack
 Post to : openstack@lists.launchpad.net
 Unsubscribe : https://launchpad.net/~openstack
 More help   : https://help.launchpad.net/ListHelp
 This email may include confidential information. If you received it in error, 
 please delete it.


 ___
 Mailing list: https://launchpad.net/~openstack
 Post to : openstack@lists.launchpad.net
 Unsubscribe : https://launchpad.net/~openstack
 More help   : https://help.launchpad.net/ListHelp
This email may include confidential information. If you received it in error, 
please delete it.


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp


Re: [Openstack] API Spec

2011-08-26 Thread George Reese
You couldn't be more correct.

The words I would use to describe this scenario are: unacceptable and 
inexcusable.

On Aug 26, 2011, at 7:19 PM, Devin Carlen wrote:

 Hey all,
 
 I've been following the code vs architect debate that's been unfolding over 
 the past week or so.  Here are some of the problems I've seen from my point 
 of view.  Fundamentally, the process we have now for defining API specs is 
 broken.  I don't believe that this can be argued.  The first major misstep in 
 my opinion was forcing backwards compatibility with the Rackspace API in the 
 OpenStack API.   
 
 I believe we should have had a Rackspace API module just like we have an EC2 
 API module.  Then the OpenStack API wouldn't have been burdened by the 
 historical decisions around the Rackspace API.
 
 But that is ancient history at this point.
 
 But we have to look at this pragmatically, and realize that 1 year later, the 
 OpenStack API (as spec'd) is still not even close to exposing the underlying 
 core functionality that exists within Nova.  For the most part, the OpenStack 
 API is a subset of functionality of the EC2 API.   This is a big reason why 
 the Dashboard project used the EC2 API for its underlying communication for 
 so long.
 
 There have been a lot of efforts lately to bring the feature set of the 
 OpenStack API in line with the EC2 API, and this is admirable.  But this has 
 NOT been happening at the architect level.  This has been happening at the 
 developer level, and it is being done with API extensions which make it 
 sound like the feature is somehow not complete or not supported fully, 
 because it's not part of the core API.
 
 So the question to all in favor of architecting up front:
 
 How do you explain lacking feature parity with the underlying components for 
 over a year now? 
 
 
 In my opinion, this has been a big problem in gaining traction around the 
 OpenStack API.
 
 
 
 Devin
 
 
 ___
 Mailing list: https://launchpad.net/~openstack
 Post to : openstack@lists.launchpad.net
 Unsubscribe : https://launchpad.net/~openstack
 More help   : https://help.launchpad.net/ListHelp

--
George Reese - Chief Technology Officer, enStratus
e: george.re...@enstratus.comt: @GeorgeReesep: +1.207.956.0217f: 
+1.612.338.5041
enStratus: Governance for Public, Private, and Hybrid Clouds - @enStratus - 
http://www.enstratus.com
To schedule a meeting with me: http://tungle.me/GeorgeReese



smime.p7s
Description: S/MIME cryptographic signature
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp


Re: [Openstack] API Spec

2011-08-26 Thread Jesse Andrews
Devin,

Thank you for putting this so eloquently. I cannot agree more

+1000
On Aug 26, 2011 11:38 AM, Devin Carlen devin.car...@gmail.com wrote:
 Hey all,

 I've been following the code vs architect debate that's been unfolding
over the past week or so. Here are some of the problems I've seen from my
point of view. Fundamentally, the process we have now for defining API specs
is broken. I don't believe that this can be argued. The first major misstep
in my opinion was forcing backwards compatibility with the Rackspace API in
the OpenStack API.

 I believe we should have had a Rackspace API module just like we have an
EC2 API module. Then the OpenStack API wouldn't have been burdened by the
historical decisions around the Rackspace API.

 But that is ancient history at this point.

 But we have to look at this pragmatically, and realize that 1 year later,
the OpenStack API (as spec'd) is still not even close to exposing the
underlying core functionality that exists within Nova. For the most part,
the OpenStack API is a subset of functionality of the EC2 API. This is a big
reason why the Dashboard project used the EC2 API for its underlying
communication for so long.

 There have been a lot of efforts lately to bring the feature set of the
OpenStack API in line with the EC2 API, and this is admirable. But this has
NOT been happening at the architect level. This has been happening at the
developer level, and it is being done with API extensions which make it
sound like the feature is somehow not complete or not supported fully,
because it's not part of the core API.

 So the question to all in favor of architecting up front:

 How do you explain lacking feature parity with the underlying components
for over a year now?


 In my opinion, this has been a big problem in gaining traction around the
OpenStack API.



 Devin


 ___
 Mailing list: https://launchpad.net/~openstack
 Post to : openstack@lists.launchpad.net
 Unsubscribe : https://launchpad.net/~openstack
 More help : https://help.launchpad.net/ListHelp
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp


Re: [Openstack] API Spec

2011-08-25 Thread Thierry Carrez
And now, the architects vs. developers discussion :)

I hear both sides of the argument, and I think both have very valid points.

However, so far we have been following the architect way (design
upfront and separately from code), and it's failing miserably.

I'm a bit tired to see developers spending all their time trying to
catch up with a spec they had little influence on. I regret that some
key features in a release end up being exposed in some strange
extension. Most of all, I feel sorry for not having any release fully
compliant with an API that is under our control.

So I'm more than ready to give the developers way its chance.
Especially with safeguards like an API freeze and inclusion of
architects in the review process when API is touched. I think we can
mitigate the risk of anarchistic design, and actually end up with code
that matches API.

Regards,

-- 
Thierry Carrez (ttx)
Release Manager, OpenStack

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp


Re: [Openstack] API Spec

2011-08-25 Thread Soren Hansen
2011/8/25 Mellquist, Peter peter.mellqu...@hp.com:
  but ... in regard to API design this approach leaves something to be 
 desired. There is much value in designing the APIs up front to ensure 
 consistency, a cohesive presentation model and to ensure that API best 
 practices are followed. APIs need to be designed and this requires an up 
 front definition and review.

Once again:

 This is completely orthogonal to when the discussion about the API
 happens. It can happen as early as you like, you can (informally)
 publish the expected API as early as you want.

Just because we stick the API spec in the code repository so that they
can be kept in sync, doesn't mean that it just magically appears out
of nowhere with no prior discussions. We can right now, today, this
very instant, start discussing and defining the API for some feature
that won't land for another 7 releases. I don't care. I just don't
want it to be published as part of something called The OpenStack
API any sooner or later than when OpenStack actually supports it.

We have something called the OpenStack API v1.1. There's no such thing
as OpenStack 1.1, and there's no other version of OpenStack that
supports the OpenStack API v1.1. Why is fixing this so controversial?

-- 
Soren Hansen        | http://linux2go.dk/
Ubuntu Developer    | http://www.ubuntu.com/
OpenStack Developer | http://www.openstack.org/

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp


Re: [Openstack] API Spec

2011-08-24 Thread Soren Hansen
2011/8/24 Jorge Williams jorge.willi...@rackspace.com:
 Let me start by saying that I have no idea why we're having this
 discussion again. We talked about it at the design summit and we
 agreed we'd move forward in pretty much exactly the way Vish describes
 it.
 I believe I attended all discussions on the compute API.   We agreed
 to design the spec in a more open and agile manner -- I'm all for
 that.  I don't remember ever discussing generating the spec from code.

Generating the spec from code (or docstrings living in code, rather) was
informally discussed in San Antonio in November.

 I also don't remember ever discussing where the spec will live.  Not
 that I really care where it lives, I just don't remember talking about
 it.

This was the subject for an entire session in Santa Clara. Making code
and docuemtation live together to ensure they'd be in sync at all times.

 2011/8/23 Jorge Williams jorge.willi...@rackspace.com:
 I don't have a problem moving the spec out of docs manuals and into another
 project even the nova repo.   But, I do have a number of issues with the
 approach that you're proposing. First, I think that fundamentally there
 should be a decoupling of the spec and the implementation.   If you have the
 spec generated from the code than essentially the spec is whatever the code
 does.
 I think you're making unsound assumptions about the implementation of
 the API docs live in code concept.

 First of all, this is primarily about consolidating the maintenance of
 the code and the API into one place. Having API docs that are ahead of
 the code is pointless. It's describing an API that doesn't exist.
 Having it lag behind the code also helps noone. To address this, we
 keep the docs in the code repository. When I write a patch that alters
 the API in some way, the corresponding doc change should accompany it.
 Whether it's written by hand or autogenerated based on introspection
 is beside the point. The point is to be able to look at any point in
 version control and see what API I can expect to work with that code.
 Having the spec ahead of time is not pointless.  That's like saying
 that blueprints are pointless.  It allows us to talk about features
 before they make it in the code and gives us a path forward. For
 example, we had heated debates about the JSON format while drafting
 the spec. We changed the examples here and there until we found an
 approach that we could all live with.

I know it's going to be an annoying discussion when I have to quote
myself. Nevertheless:

 This is completely orthogonal to when the discussion about the API
 happens. It can happen as early as you like, you can (informally)
 publish the expected API as early as you want.

Please let me know if (and how) I should elaborate on this.

 Imagine if all of those changes had to be followed by a patch that
 change the representation 3..4..5 times, that would have been a big
 waste of coding effort.

Noone is suggesting writing roughly the same code 3, 4 or 5 times. Part
of the discussion about a feature is discussing the API. We've
historically not been very good at discussing blueprints in very much
detail, but that's different issue.

 Also, the spec allows us to talk about the API as a whole

In what way are we unable to talk about the API as a whole when we're
having these discussions?

 Looking at the API as a whole is important when we talk about
 consistency, correctness etc.

Of course it is. And when we discuss the API for a feature, we have to
consider it in the context of the rest of the API.

 Having a spec separate from the Impl is also important when you have
 multiple groups writing an implementation to the spec..

It's the same people defining the API. Whether they write it by hand or
have parts of it be generated based on introspection, the output will be
the *exact* same. Who cares?

 that's happening today with at least compute and keystone, and it's
 something which I think we should continue to encourage.

OpenStack was created to stop every company wanting to do offer cloud
computing from reinventing everything and have everyone work together.
What's the use of competing implementations? We'll be right back to square one.

 Multiple groups can both simultaneously strive for compatible
 implementations, and the spc gives them a place where they can come to
 an agreement.

Yes? Can you explain how any of this is impeded by this proposal? I'm
not seeing it at all.

 I don't see how we can make this possible if a particular
 implementation's code dictates what the spec is.

I don't see the connection at all.

 This is completely orthogonal to when the discussion about the API
 happens. It can happen as early as you like, you can (informally)
 publish the expected API as early as you want. Expected is the
 keyword. When you sit down and start implementing something, sometimes
 you just realise that the API you were expecting to implement is
 really, really awkward to do, but a slightly 

Re: [Openstack] API Spec

2011-08-23 Thread Thor Wolpert
That sounds fair.  I listened to Linus talk about a similar thing when
in the early days they tried to pack in features, and only later
started to try and constrain what got into the core.  There was some
interesting debate about the new features as they often didn't know
how they would be best used and grow until after people started using
them.  As such they looked to try and maintain backwards compatibility
as best they could until people had enough warning to migrate
(measured in years).

Sounds like a similar dilemma to me.

Thor HW

On Mon, Aug 22, 2011 at 9:49 PM, Jorge Williams
jorge.willi...@rackspace.com wrote:

 I say we up the version number when we can't ensure backward compatibility.  
 As to how long older versions should be supported.  Hard to say.  It depends 
 on a lot of factors, and at the end of the day it may come up to how popular 
 a version is and how  willing and able operators and client devs are to 
 upgrading.

 -jOrGe W.


 On Aug 22, 2011, at 8:49 PM, Thor Wolpert wrote:

 I agree the Specs shouldn't change often ... but just to use your
 examples, they where all simplifications of larger specs that took
 years to create.

 If an API changes and is deprecated, how long does backwards
 compatibility stay in place?

 Thanks,
 Thor W

 On Mon, Aug 22, 2011 at 5:49 PM, Jan Drake jan_dr...@hotmail.com wrote:
 +1




 On Aug 22, 2011, at 5:06 PM, Jay Pipes jaypi...@gmail.com wrote:

 ++

 On Mon, Aug 22, 2011 at 7:59 PM, Jorge Williams
 jorge.willi...@rackspace.com wrote:
 Hi Vish,
 I don't have a problem moving the spec out of docs manuals and into 
 another
 project even the nova repo.   But, I do have a number of issues with the
 approach that you're proposing. First, I think that fundamentally there
 should be a decoupling of the spec and the implementation.   If you have 
 the
 spec generated from the code than essentially the spec is whatever the 
 code
 does. It's very difficult to interoperate with specs that are generated 
 this
 way as the specs tend to be very brittle and opaque (since you have to 
 study
 the code). If you introduce a  bug in the code that bug filters it's way 
 all
 the way to the spec (this was a big problem with SOAP and CORBA). It's
 difficult to detect errors because you cant validate. By keeping the
 implementation and the spec separate you can validate one against the 
 other.

 Second, I don't think that the core OpenStack API should change with every
 OpenStack release. There are a number of efforts to provide multiple
 implementation of an existing OpenStack API.  We should encourage this, 
 but
 it becomes difficult if the core spec is in constant flux.  Certainly you
 can use the extension mechanism to bring functionality out to market
 quickly, but the process of deciding what goes into the core should be 
 more
 deliberate. Really good specs, shouldn't need to change very often, think
 HTTP, X11, SMTP, etc. We need to encourage clients to write support for 
 our
 spec and we need to also encourage other implementors to write
 implementations for it. These efforts become very difficult if the spec is
 in constant flux.
 -jOrGe W.
 On Aug 22, 2011, at 5:43 PM, Vishvananda Ishaya wrote:

 Hey Everyone,
 We discussed at the Diablo design summit having API spec changes be 
 proposed
 along with code changes and reviewed according to the merge process that 
 we
 use for code.  This has been impossible up until now because the canonical
 spec has been in the openstack-manuals project.
 My suggestion is that we move the openstack-compute spec into the nova
 source tree.  During a six-month release we can propose changes to the 
 spec
 by proposing along with the code that changes it.  In the final freeze for
 the release, we can increment the spec version number and copy the current
 version of the spec into openstack-manuals and that will be the locked 
 down
 spec for that release.
 This means that openstack 1.1 will be the official spec for diablo, at 
 which
 point we will start working on a new api (we can call it 1.2 but it might 
 be
 best to use a temporary name like 'latest') during the essex release 
 cycle,
 then at essex release we lock the spec down and it becomes the new version
 of the openstack api.
 Ultimately I would like the spec to be generated from the code, but as a
 first pass, we should at least be able to edit the future version of the
 spec as we make changes.  I've proposed the current version of the spec
 here:
 https://code.launchpad.net/~vishvananda/nova/add-api-docs/+merge/72506
 Are there any issues with this approach?
 Vish

 This email may include confidential information. If you received it in
 error, please delete it.
 ___
 Mailing list: https://launchpad.net/~openstack
 Post to     : openstack@lists.launchpad.net
 Unsubscribe : https://launchpad.net/~openstack
 More help   : https://help.launchpad.net/ListHelp



 ___
 

Re: [Openstack] API Spec

2011-08-23 Thread Thierry Carrez
Vishvananda Ishaya wrote:
 Ultimately I would like the spec to be generated from the code, but as a
 first pass, we should at least be able to edit the future version of the
 spec as we make changes.  I've proposed the current version of the spec
 here:
 
 https://code.launchpad.net/~vishvananda/nova/add-api-docs/+merge/72506
 
 Are there any issues with this approach?

Looks like there are diverging opinions. That sounds like work for... a
design summit brainstorm session !

Any reason why we'd need to discuss this *now*, as opposed to after
Diablo is released ?

-- 
Thierry Carrez (ttx)
Release Manager, OpenStack

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp


Re: [Openstack] API Spec

2011-08-23 Thread Jorge Williams
I'm not proposing that the API be frozen solid.  I'm all for making changes to 
the API in a backward compatible manner.  In fact, I think most changes should 
come in this way, which is why versions of the *core* API shouldn't have to 
change frequently, but the API itself may be under constant rapid development.  
New features should be added in a backward compatible manner as frequently as 
you can dream them up.  Go to town.

We are in a different position than github, though.  To my knowledge,  GitHub, 
Inc is in complete control of the API.  They can add backward compatible 
changes without jumping through hoops. In our case, anybody has access to the 
source code for our APIs and different vendors may build solutions based of of 
them and may extend them. The vendors may not want to contribute the changes 
back to the main repo -- either because it doesn't make a lot of sense (think 
pricing extensions) or because the vendor wants a way in which to differentiate 
their deployment from other deployments in order to find a niche and remain 
competitive. Certainly that's their prerogative -- if we can make them 
successful then OpenStack will be successful.  This means, however that the 
development of our APIs, unlike the github APIs,  can happen in a completely 
decentralized manner. Imagine that Rackspace comes up with a feature to perform 
backups and places it in /backups.  HP comes up with it's own backup feature 
and also puts it in /backups. The features are different so a client expecting 
Rackspace backup will break when it encounters an HP /backup.  The idea of 
extensions is to prevent this from happening.   What's actually in the core is 
what gets protected  because that's the functionality that the client can 
*always* depend on.  If Rackspace and HP want to get together and work out the 
differences between their backup features and propose them as part of the core 
-- there is a method by which this can be accomplished but the process then 
becomes more deliberate and centralized.  That said the feature could have been 
developed rapidly and in a completely decentralized manner.

-jOrGe W.


On Aug 23, 2011, at 12:46 AM, Christopher MacGown wrote:

This is just a robustness principal argument. If a client breaks because we've 
added a new key to a JSON dict, it's the fault of the client's developer. If 
the client breaks because the primitives have changed and a dictionary has been 
changed into a list, then it's ours. The features that have been proposed that 
would drive functionality in the API from nova-core aren't changing the 
semantics of the API, they're adding additional fields.

There are examples of well-written APIs that commonly add new features to their 
API without breaking backward compatibility. If you take, for example, github 
(not to start a git/bzr argument, it's just late and I know the API), they 
regularly add features to their v2 and v3 API. The well-written clients don't 
break when new features come out, even the mediocrely written ones like mine 
don't break when new features come out.

If even mediocre programers like me can handle programming against a constantly 
changing API without causing any breakage in my client, I'm not sure what 
justification there is to freeze the API solid and not allow additional 
features. We shouldn't be building our API for people who write terrible 
clients that break when a feature gets added, because it'll waste everyone's 
time waiting for new features that may not get common support across service 
providers, and because no one will bother using those clients anyway.


Christopher MacGown
Piston Cloud Computing, Inc.
w: (650) 24-CLOUD
m: (415) 300-0944
ch...@pistoncloud.commailto:ch...@pistoncloud.com

On Aug 22, 2011, at 9:18 PM, Jorge Williams wrote:

Comments inline

On Aug 22, 2011, at 9:05 PM, Vishvananda Ishaya wrote:

Inline
On Aug 22, 2011, at 4:59 PM, Jorge Williams wrote:

Hi Vish,

I don't have a problem moving the spec out of docs manuals and into another 
project even the nova repo.   But, I do have a number of issues with the 
approach that you're proposing. First, I think that fundamentally there should 
be a decoupling of the spec and the implementation.   If you have the spec 
generated from the code than essentially the spec is whatever the code does. 
It's very difficult to interoperate with specs that are generated this way as 
the specs tend to be very brittle and opaque (since you have to study the 
code). If you introduce a  bug in the code that bug filters it's way all the 
way to the spec (this was a big problem with SOAP and CORBA). It's difficult to 
detect errors because you cant validate. By keeping the implementation and the 
spec separate you can validate one against the other.

The spec SHOULD BE exactly what the code does, verifiably.  I'm proposing that 
we have exactly the existing document, only that the xml and json examples in 
the spec are actually generated from the code so that 

Re: [Openstack] API Spec

2011-08-23 Thread Jay Pipes
No problems with anything you say below...

-jay

On Mon, Aug 22, 2011 at 9:59 PM, Vishvananda Ishaya
vishvana...@gmail.com wrote:
 Inline

 On Aug 22, 2011, at 4:15 PM, Jay Pipes wrote:

 It may be just me, but having DocBookXML in the source tree is hideous
 to me. Not only does it clutter the source tree with non-RST
 documentation, but as you know, reviewing diffs of XML is just about
 makes people want to slit their throats with a spoon. There is a
 special type of person (Anne!) that seem to be impervious to the
 throat-slitting urge and can successfully digest such a review
 request, but for many, the process is too painful.

 I hate xml changes as well, but anne + people in manuals don't really have 
 the expertise to know if something belongs in the api.  *-core should be 
 responsible for the spec for the project.

 In addition to the above gripe, the development, stability, and
 enhancement of the OpenStack APIs can and should (IMHO) be managed
 separately from the source code of the project. The API can evolve in
 the openstack-manuals project and developers can code the OpenStack
 subproject to match the API documented in openstack-manuals project
 (at the tagged version). So, for instance, if the compute API needs to
 change, the API documentation changes would be proposed in
 openstack-manuals, reviewed, discussed and approved. The new API docs
 would be published to something like
 http://docs.openstack.org/compute/2.0/ and developers coding Essex
 features having to do with implementing such a 2.0 API would refer to
 the API documentation there while writing the feature...

 If we want to separate the xml code out, we can do a separate nova-spec repo 
 for the spec, but this should be managed by nova-core


 Just my 2 cents,
 -jay


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp


Re: [Openstack] API Spec

2011-08-23 Thread Anne Gentle
Couple of reasons from my perspective:

We've got to get me out of the loop for reviews of the API spec - I'm happy
to review and would like to be a reviewer of all the API docs for OpenStack,
but we need better governance of API updates.

Also, my goal was to move doc source to github prior to October 3. Based on
this discussion, it also makes a decision about how many books per repo we
have. I'm now leaning towards one repo per book deliverable.

Anne
*Anne Gentle*
a...@openstack.org
 my blog http://justwriteclick.com/ | my
bookhttp://xmlpress.net/publications/conversation-community/|
LinkedIn http://www.linkedin.com/in/annegentle |
Delicioushttp://del.icio.us/annegentle|
Twitter http://twitter.com/annegentle
On Tue, Aug 23, 2011 at 3:01 AM, Thierry Carrez thie...@openstack.orgwrote:

 Vishvananda Ishaya wrote:
  Ultimately I would like the spec to be generated from the code, but as a
  first pass, we should at least be able to edit the future version of the
  spec as we make changes.  I've proposed the current version of the spec
  here:
 
  https://code.launchpad.net/~vishvananda/nova/add-api-docs/+merge/72506https://code.launchpad.net/%7Evishvananda/nova/add-api-docs/+merge/72506
 
  Are there any issues with this approach?

 Looks like there are diverging opinions. That sounds like work for... a
 design summit brainstorm session !

 Any reason why we'd need to discuss this *now*, as opposed to after
 Diablo is released ?

 --
 Thierry Carrez (ttx)
 Release Manager, OpenStack

 ___
 Mailing list: https://launchpad.net/~openstack
 Post to : openstack@lists.launchpad.net
 Unsubscribe : https://launchpad.net/~openstack
 More help   : https://help.launchpad.net/ListHelp

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp


Re: [Openstack] API Spec

2011-08-23 Thread Soren Hansen
Let me start by saying that I have no idea why we're having this
discussion again. We talked about it at the design summit and we
agreed we'd move forward in pretty much exactly the way Vish describes
it.


2011/8/23 Jorge Williams jorge.willi...@rackspace.com:
 I don't have a problem moving the spec out of docs manuals and into another
 project even the nova repo.   But, I do have a number of issues with the
 approach that you're proposing. First, I think that fundamentally there
 should be a decoupling of the spec and the implementation.   If you have the
 spec generated from the code than essentially the spec is whatever the code
 does.

I think you're making unsound assumptions about the implementation of
the API docs live in code concept.

First of all, this is primarily about consolidating the maintenance of
the code and the API into one place. Having API docs that are ahead of
the code is pointless. It's describing an API that doesn't exist.
Having it lag behind the code also helps noone. To address this, we
keep the docs in the code repository. When I write a patch that alters
the API in some way, the corresponding doc change should accompany it.
Whether it's written by hand or autogenerated based on introspection
is beside the point. The point is to be able to look at any point in
version control and see what API I can expect to work with that code.

This is completely orthogonal to when the discussion about the API
happens. It can happen as early as you like, you can (informally)
publish the expected API as early as you want. Expected is the
keyword. When you sit down and start implementing something, sometimes
you just realise that the API you were expecting to implement is
really, really awkward to do, but a slightly different API for the
same functionality would be simple and elegant. In those cases, I see
no particular reason to stick to the originally published, expected
API. Of course we need a deadline (probably around feature freeze) at
which point no more changes can be made to the exposed API's so we
don't rip the rug out from under everyone every day up until release.

Second of all, if this is done right, it could be used as a tool to
ensure that you don't accidentally make a change that breaks the
published API.

 It's very difficult to interoperate with specs that are generated this
 way as the specs tend to be very brittle and opaque (since you have to study
 the code).

You seem to be assuming that people will just throw random bits and
bytes into patches and others will blindly approve these patches.

Validating API (and corresponding docs) changes is of course part of
the review process, and there are plenty of things we can do to make
this robust and reliable and automatically catch it if it changes
unexpectedly.

 Second, I don't think that the core OpenStack API should change with every
 OpenStack release.
[...]
 implementations for it. These efforts become very difficult if the spec is
 in constant flux.

Again, it's not like the API will change randomly every half hour. It
will change when the developers want it to change. There's a review
process. And mindful developers. And even if they aren't mindful, you
can freeze the documentation or maintain it separately all you want,
but if the code changes, it changes.

-- 
Soren Hansen        | http://linux2go.dk/
Ubuntu Developer    | http://www.ubuntu.com/
OpenStack Developer | http://www.openstack.org/

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp


Re: [Openstack] API Spec

2011-08-22 Thread Jay Pipes
Hi Vish! Comments inline...

On Mon, Aug 22, 2011 at 6:43 PM, Vishvananda Ishaya
vishvana...@gmail.com wrote:
 Hey Everyone,
 We discussed at the Diablo design summit having API spec changes be proposed
 along with code changes and reviewed according to the merge process that we
 use for code.  This has been impossible up until now because the canonical
 spec has been in the openstack-manuals project.

Well, it's not been impossible :) It's just that the openstack-manuals
project doesn't go through the same code review process as Nova,
Swift, and Glance :) That can easily be changed, no?

 My suggestion is that we move the openstack-compute spec into the nova
 source tree.  During a six-month release we can propose changes to the spec
 by proposing along with the code that changes it.  In the final freeze for
 the release, we can increment the spec version number and copy the current
 version of the spec into openstack-manuals and that will be the locked down
 spec for that release.

It may be just me, but having DocBookXML in the source tree is hideous
to me. Not only does it clutter the source tree with non-RST
documentation, but as you know, reviewing diffs of XML is just about
makes people want to slit their throats with a spoon. There is a
special type of person (Anne!) that seem to be impervious to the
throat-slitting urge and can successfully digest such a review
request, but for many, the process is too painful.

In addition to the above gripe, the development, stability, and
enhancement of the OpenStack APIs can and should (IMHO) be managed
separately from the source code of the project. The API can evolve in
the openstack-manuals project and developers can code the OpenStack
subproject to match the API documented in openstack-manuals project
(at the tagged version). So, for instance, if the compute API needs to
change, the API documentation changes would be proposed in
openstack-manuals, reviewed, discussed and approved. The new API docs
would be published to something like
http://docs.openstack.org/compute/2.0/ and developers coding Essex
features having to do with implementing such a 2.0 API would refer to
the API documentation there while writing the feature...

Just my 2 cents,
-jay

 This means that openstack 1.1 will be the official spec for diablo, at which
 point we will start working on a new api (we can call it 1.2 but it might be
 best to use a temporary name like 'latest') during the essex release cycle,
 then at essex release we lock the spec down and it becomes the new version
 of the openstack api.
 Ultimately I would like the spec to be generated from the code, but as a
 first pass, we should at least be able to edit the future version of the
 spec as we make changes.  I've proposed the current version of the spec
 here:
 https://code.launchpad.net/~vishvananda/nova/add-api-docs/+merge/72506
 Are there any issues with this approach?
 Vish
 ___
 Mailing list: https://launchpad.net/~openstack
 Post to     : openstack@lists.launchpad.net
 Unsubscribe : https://launchpad.net/~openstack
 More help   : https://help.launchpad.net/ListHelp



___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp


Re: [Openstack] API Spec

2011-08-22 Thread Jorge Williams
Hi Vish,

I don't have a problem moving the spec out of docs manuals and into another 
project even the nova repo.   But, I do have a number of issues with the 
approach that you're proposing. First, I think that fundamentally there should 
be a decoupling of the spec and the implementation.   If you have the spec 
generated from the code than essentially the spec is whatever the code does. 
It's very difficult to interoperate with specs that are generated this way as 
the specs tend to be very brittle and opaque (since you have to study the 
code). If you introduce a  bug in the code that bug filters it's way all the 
way to the spec (this was a big problem with SOAP and CORBA). It's difficult to 
detect errors because you cant validate. By keeping the implementation and the 
spec separate you can validate one against the other.

Second, I don't think that the core OpenStack API should change with every 
OpenStack release. There are a number of efforts to provide multiple 
implementation of an existing OpenStack API.  We should encourage this, but it 
becomes difficult if the core spec is in constant flux.  Certainly you can use 
the extension mechanism to bring functionality out to market quickly, but the 
process of deciding what goes into the core should be more deliberate. Really 
good specs, shouldn't need to change very often, think HTTP, X11, SMTP, etc. We 
need to encourage clients to write support for our spec and we need to also 
encourage other implementors to write implementations for it. These efforts 
become very difficult if the spec is in constant flux.

-jOrGe W.

On Aug 22, 2011, at 5:43 PM, Vishvananda Ishaya wrote:

Hey Everyone,

We discussed at the Diablo design summit having API spec changes be proposed 
along with code changes and reviewed according to the merge process that we use 
for code.  This has been impossible up until now because the canonical spec has 
been in the openstack-manuals project.

My suggestion is that we move the openstack-compute spec into the nova source 
tree.  During a six-month release we can propose changes to the spec by 
proposing along with the code that changes it.  In the final freeze for the 
release, we can increment the spec version number and copy the current version 
of the spec into openstack-manuals and that will be the locked down spec for 
that release.

This means that openstack 1.1 will be the official spec for diablo, at which 
point we will start working on a new api (we can call it 1.2 but it might be 
best to use a temporary name like 'latest') during the essex release cycle, 
then at essex release we lock the spec down and it becomes the new version of 
the openstack api.

Ultimately I would like the spec to be generated from the code, but as a first 
pass, we should at least be able to edit the future version of the spec as we 
make changes.  I've proposed the current version of the spec here:

https://code.launchpad.net/~vishvananda/nova/add-api-docs/+merge/72506

Are there any issues with this approach?

Vish

This email may include confidential information. If you received it in error, 
please delete it.
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp


Re: [Openstack] API Spec

2011-08-22 Thread Jay Pipes
++

On Mon, Aug 22, 2011 at 7:59 PM, Jorge Williams
jorge.willi...@rackspace.com wrote:
 Hi Vish,
 I don't have a problem moving the spec out of docs manuals and into another
 project even the nova repo.   But, I do have a number of issues with the
 approach that you're proposing. First, I think that fundamentally there
 should be a decoupling of the spec and the implementation.   If you have the
 spec generated from the code than essentially the spec is whatever the code
 does. It's very difficult to interoperate with specs that are generated this
 way as the specs tend to be very brittle and opaque (since you have to study
 the code). If you introduce a  bug in the code that bug filters it's way all
 the way to the spec (this was a big problem with SOAP and CORBA). It's
 difficult to detect errors because you cant validate. By keeping the
 implementation and the spec separate you can validate one against the other.

 Second, I don't think that the core OpenStack API should change with every
 OpenStack release. There are a number of efforts to provide multiple
 implementation of an existing OpenStack API.  We should encourage this, but
 it becomes difficult if the core spec is in constant flux.  Certainly you
 can use the extension mechanism to bring functionality out to market
 quickly, but the process of deciding what goes into the core should be more
 deliberate. Really good specs, shouldn't need to change very often, think
 HTTP, X11, SMTP, etc. We need to encourage clients to write support for our
 spec and we need to also encourage other implementors to write
 implementations for it. These efforts become very difficult if the spec is
 in constant flux.
 -jOrGe W.
 On Aug 22, 2011, at 5:43 PM, Vishvananda Ishaya wrote:

 Hey Everyone,
 We discussed at the Diablo design summit having API spec changes be proposed
 along with code changes and reviewed according to the merge process that we
 use for code.  This has been impossible up until now because the canonical
 spec has been in the openstack-manuals project.
 My suggestion is that we move the openstack-compute spec into the nova
 source tree.  During a six-month release we can propose changes to the spec
 by proposing along with the code that changes it.  In the final freeze for
 the release, we can increment the spec version number and copy the current
 version of the spec into openstack-manuals and that will be the locked down
 spec for that release.
 This means that openstack 1.1 will be the official spec for diablo, at which
 point we will start working on a new api (we can call it 1.2 but it might be
 best to use a temporary name like 'latest') during the essex release cycle,
 then at essex release we lock the spec down and it becomes the new version
 of the openstack api.
 Ultimately I would like the spec to be generated from the code, but as a
 first pass, we should at least be able to edit the future version of the
 spec as we make changes.  I've proposed the current version of the spec
 here:
 https://code.launchpad.net/~vishvananda/nova/add-api-docs/+merge/72506
 Are there any issues with this approach?
 Vish

 This email may include confidential information. If you received it in
 error, please delete it.
 ___
 Mailing list: https://launchpad.net/~openstack
 Post to     : openstack@lists.launchpad.net
 Unsubscribe : https://launchpad.net/~openstack
 More help   : https://help.launchpad.net/ListHelp



___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp


Re: [Openstack] API Spec

2011-08-22 Thread Vishvananda Ishaya
Inline

On Aug 22, 2011, at 4:15 PM, Jay Pipes wrote:

 It may be just me, but having DocBookXML in the source tree is hideous
 to me. Not only does it clutter the source tree with non-RST
 documentation, but as you know, reviewing diffs of XML is just about
 makes people want to slit their throats with a spoon. There is a
 special type of person (Anne!) that seem to be impervious to the
 throat-slitting urge and can successfully digest such a review
 request, but for many, the process is too painful.

I hate xml changes as well, but anne + people in manuals don't really have the 
expertise to know if something belongs in the api.  *-core should be 
responsible for the spec for the project.
 
 In addition to the above gripe, the development, stability, and
 enhancement of the OpenStack APIs can and should (IMHO) be managed
 separately from the source code of the project. The API can evolve in
 the openstack-manuals project and developers can code the OpenStack
 subproject to match the API documented in openstack-manuals project
 (at the tagged version). So, for instance, if the compute API needs to
 change, the API documentation changes would be proposed in
 openstack-manuals, reviewed, discussed and approved. The new API docs
 would be published to something like
 http://docs.openstack.org/compute/2.0/ and developers coding Essex
 features having to do with implementing such a 2.0 API would refer to
 the API documentation there while writing the feature...

If we want to separate the xml code out, we can do a separate nova-spec repo 
for the spec, but this should be managed by nova-core

 
 Just my 2 cents,
 -jay

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp


Re: [Openstack] API Spec

2011-08-22 Thread Thor Wolpert
I agree the Specs shouldn't change often ... but just to use your
examples, they where all simplifications of larger specs that took
years to create.

If an API changes and is deprecated, how long does backwards
compatibility stay in place?

Thanks,
Thor W

On Mon, Aug 22, 2011 at 5:49 PM, Jan Drake jan_dr...@hotmail.com wrote:
 +1




 On Aug 22, 2011, at 5:06 PM, Jay Pipes jaypi...@gmail.com wrote:

 ++

 On Mon, Aug 22, 2011 at 7:59 PM, Jorge Williams
 jorge.willi...@rackspace.com wrote:
 Hi Vish,
 I don't have a problem moving the spec out of docs manuals and into another
 project even the nova repo.   But, I do have a number of issues with the
 approach that you're proposing. First, I think that fundamentally there
 should be a decoupling of the spec and the implementation.   If you have the
 spec generated from the code than essentially the spec is whatever the code
 does. It's very difficult to interoperate with specs that are generated this
 way as the specs tend to be very brittle and opaque (since you have to study
 the code). If you introduce a  bug in the code that bug filters it's way all
 the way to the spec (this was a big problem with SOAP and CORBA). It's
 difficult to detect errors because you cant validate. By keeping the
 implementation and the spec separate you can validate one against the other.

 Second, I don't think that the core OpenStack API should change with every
 OpenStack release. There are a number of efforts to provide multiple
 implementation of an existing OpenStack API.  We should encourage this, but
 it becomes difficult if the core spec is in constant flux.  Certainly you
 can use the extension mechanism to bring functionality out to market
 quickly, but the process of deciding what goes into the core should be more
 deliberate. Really good specs, shouldn't need to change very often, think
 HTTP, X11, SMTP, etc. We need to encourage clients to write support for our
 spec and we need to also encourage other implementors to write
 implementations for it. These efforts become very difficult if the spec is
 in constant flux.
 -jOrGe W.
 On Aug 22, 2011, at 5:43 PM, Vishvananda Ishaya wrote:

 Hey Everyone,
 We discussed at the Diablo design summit having API spec changes be proposed
 along with code changes and reviewed according to the merge process that we
 use for code.  This has been impossible up until now because the canonical
 spec has been in the openstack-manuals project.
 My suggestion is that we move the openstack-compute spec into the nova
 source tree.  During a six-month release we can propose changes to the spec
 by proposing along with the code that changes it.  In the final freeze for
 the release, we can increment the spec version number and copy the current
 version of the spec into openstack-manuals and that will be the locked down
 spec for that release.
 This means that openstack 1.1 will be the official spec for diablo, at which
 point we will start working on a new api (we can call it 1.2 but it might be
 best to use a temporary name like 'latest') during the essex release cycle,
 then at essex release we lock the spec down and it becomes the new version
 of the openstack api.
 Ultimately I would like the spec to be generated from the code, but as a
 first pass, we should at least be able to edit the future version of the
 spec as we make changes.  I've proposed the current version of the spec
 here:
 https://code.launchpad.net/~vishvananda/nova/add-api-docs/+merge/72506
 Are there any issues with this approach?
 Vish

 This email may include confidential information. If you received it in
 error, please delete it.
 ___
 Mailing list: https://launchpad.net/~openstack
 Post to     : openstack@lists.launchpad.net
 Unsubscribe : https://launchpad.net/~openstack
 More help   : https://help.launchpad.net/ListHelp



 ___
 Mailing list: https://launchpad.net/~openstack
 Post to     : openstack@lists.launchpad.net
 Unsubscribe : https://launchpad.net/~openstack
 More help   : https://help.launchpad.net/ListHelp

 ___
 Mailing list: https://launchpad.net/~openstack
 Post to     : openstack@lists.launchpad.net
 Unsubscribe : https://launchpad.net/~openstack
 More help   : https://help.launchpad.net/ListHelp


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp


Re: [Openstack] API Spec

2011-08-22 Thread Anne Gentle
I think it makes sense to have an openstack-api project with all the API
docs (specs and learning materials) gathered in one place. I think it's
preferable to have the API separated out from the code for several reasons -
ease of review, ease of check out, also for learning materials for the API
itself.

I'd envision these would go in it for starters:

Compute API (docbook, core + extensions)
Glance API (RST to docbook, core)
Keystone API (docbook, incubation, core + extensions)
Swift API (docbook, core)

Notes:
- Yep, Keystone moved their docbook out of the core code base due to the
overhead associated with a large-ish document ensconced with the code.
- Glance's API is documented in a single RST page. We have a simple RST to
docbook tool inhouse at Rackspace that lets me take updates and move them
into docbook.
- Just today I had a request to bring the Load Balancing API into
openstack-manuals for comments and review from
http://wiki.openstack.org/Atlas-LB, since our wiki doesn't enable comments.
I'm not sure what to do with nascent APIs for review that aren't yet in
incubation.

So these are some of my observations having worked with the API docs for a
while, to consider while seeking the ideal solution:
Incubation - how do we indicate an API is incubated?
Pre-incubation/review period - how could a team offer an API up for
community review and commenting?
Core - how do we indicate what is core and how to get extensions? At
Rackspace the Compute API team is working on a solution to getting
extensions and telling people how to use them once they're available.
Source - DocBook is the source for the two biggest API docs, Compute and
Object Storage, Keystone is a close third, and I can get DocBook out of
Glance. Do we need to set DocBook as the standard source?
Output - I'd also like to focus on not only API specs but also deliverables
that help people learn the APIs, such as the frameworks recently opensourced
by Mashery (example: http://developer.klout.com/iodocs) and Wordnik (
http://swagger.wordnik.com/). If we also deliver this type of web tool, we'd
also need JSON or XML as source files (many of which are already embedded
into the DocBook).

I'd like the best of both worlds - API specs and self-documenting APIs. I
think we can get there, and I think a separate API project with a core
review team moves us in that direction.

Thanks for the good discussion here.
Anne

* *
*Anne Gentle*
http://www.facebook.com/conversationandcommunity
my blog http://justwriteclick.com/ | my
bookhttp://xmlpress.net/publications/conversation-community/|
LinkedIn http://www.linkedin.com/in/annegentle |
Delicioushttp://del.icio.us/annegentle|
Twitter http://twitter.com/annegentle
On Mon, Aug 22, 2011 at 7:49 PM, Jan Drake jan_dr...@hotmail.com wrote:

 +1




 On Aug 22, 2011, at 5:06 PM, Jay Pipes jaypi...@gmail.com wrote:

  ++
 
  On Mon, Aug 22, 2011 at 7:59 PM, Jorge Williams
  jorge.willi...@rackspace.com wrote:
  Hi Vish,
  I don't have a problem moving the spec out of docs manuals and into
 another
  project even the nova repo.   But, I do have a number of issues with the
  approach that you're proposing. First, I think that fundamentally there
  should be a decoupling of the spec and the implementation.   If you have
 the
  spec generated from the code than essentially the spec is whatever the
 code
  does. It's very difficult to interoperate with specs that are generated
 this
  way as the specs tend to be very brittle and opaque (since you have to
 study
  the code). If you introduce a  bug in the code that bug filters it's way
 all
  the way to the spec (this was a big problem with SOAP and CORBA). It's
  difficult to detect errors because you cant validate. By keeping the
  implementation and the spec separate you can validate one against the
 other.
 
  Second, I don't think that the core OpenStack API should change with
 every
  OpenStack release. There are a number of efforts to provide multiple
  implementation of an existing OpenStack API.  We should encourage this,
 but
  it becomes difficult if the core spec is in constant flux.  Certainly
 you
  can use the extension mechanism to bring functionality out to market
  quickly, but the process of deciding what goes into the core should be
 more
  deliberate. Really good specs, shouldn't need to change very often,
 think
  HTTP, X11, SMTP, etc. We need to encourage clients to write support for
 our
  spec and we need to also encourage other implementors to write
  implementations for it. These efforts become very difficult if the spec
 is
  in constant flux.
  -jOrGe W.
  On Aug 22, 2011, at 5:43 PM, Vishvananda Ishaya wrote:
 
  Hey Everyone,
  We discussed at the Diablo design summit having API spec changes be
 proposed
  along with code changes and reviewed according to the merge process that
 we
  use for code.  This has been impossible up until now because the
 canonical
  spec has been in the openstack-manuals project.
  My suggestion is that we move 

Re: [Openstack] API Spec

2011-08-22 Thread Vishvananda Ishaya
Inline
On Aug 22, 2011, at 4:59 PM, Jorge Williams wrote:

 Hi Vish,
 
 I don't have a problem moving the spec out of docs manuals and into another 
 project even the nova repo.   But, I do have a number of issues with the 
 approach that you're proposing. First, I think that fundamentally there 
 should be a decoupling of the spec and the implementation.   If you have the 
 spec generated from the code than essentially the spec is whatever the code 
 does. It's very difficult to interoperate with specs that are generated this 
 way as the specs tend to be very brittle and opaque (since you have to study 
 the code). If you introduce a  bug in the code that bug filters it's way all 
 the way to the spec (this was a big problem with SOAP and CORBA). It's 
 difficult to detect errors because you cant validate. By keeping the 
 implementation and the spec separate you can validate one against the other. 

The spec SHOULD BE exactly what the code does, verifiably.  I'm proposing that 
we have exactly the existing document, only that the xml and json examples in 
the spec are actually generated from the code so that we know they are 
accurate.  This is a minor point to me though, and could be accomplished by 
testing the spec against the code as well.

 
 Second, I don't think that the core OpenStack API should change with every 
 OpenStack release. There are a number of efforts to provide multiple 
 implementation of an existing OpenStack API.  We should encourage this, but 
 it becomes difficult if the core spec is in constant flux.  Certainly you can 
 use the extension mechanism to bring functionality out to market quickly, but 
 the process of deciding what goes into the core should be more deliberate. 
 Really good specs, shouldn't need to change very often, think HTTP, X11, 
 SMTP, etc. We need to encourage clients to write support for our spec and we 
 need to also encourage other implementors to write implementations for it. 
 These efforts become very difficult if the spec is in constant flux.

Incrementing the version number is only a problem if we fail to support the old 
versions.  At the rate we are adding new functionality, I think we will easily 
need a new spec every six months for the foreseeable future.  If there are no 
reasonable changes in a six month period, we can skip a release and not have a 
new version of the spec

 
 -jOrGe W.


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp


Re: [Openstack] API Spec

2011-08-22 Thread Jorge Williams
Comments inline

On Aug 22, 2011, at 9:05 PM, Vishvananda Ishaya wrote:

 Inline
 On Aug 22, 2011, at 4:59 PM, Jorge Williams wrote:
 
 Hi Vish,
 
 I don't have a problem moving the spec out of docs manuals and into another 
 project even the nova repo.   But, I do have a number of issues with the 
 approach that you're proposing. First, I think that fundamentally there 
 should be a decoupling of the spec and the implementation.   If you have the 
 spec generated from the code than essentially the spec is whatever the code 
 does. It's very difficult to interoperate with specs that are generated this 
 way as the specs tend to be very brittle and opaque (since you have to study 
 the code). If you introduce a  bug in the code that bug filters it's way all 
 the way to the spec (this was a big problem with SOAP and CORBA). It's 
 difficult to detect errors because you cant validate. By keeping the 
 implementation and the spec separate you can validate one against the other. 
 
 The spec SHOULD BE exactly what the code does, verifiably.  I'm proposing 
 that we have exactly the existing document, only that the xml and json 
 examples in the spec are actually generated from the code so that we know 
 they are accurate.  This is a minor point to me though, and could be 
 accomplished by testing the spec against the code as well.

Let's say you generate the samples from the code.  Everything works great.  The 
samples represent exactly what the code is doing.  A client developer takes a 
look at the spec samples and develops an app against it.  Then a merge request 
goes through that changes the format of the JSON inadvertently in a subtle way. 
 Now the client is out of synch and breaks. The client dev looks back at the 
spec,  not only has the code changed but the spec has changed too since it's 
generated, -- the client therefore  assumes that the bug is his bad -- so he 
changes the code to match the spec.  Things work great, until the service 
developer notices that the merge changed the format of the JSON and changes it 
back to what it used to be -- now the client is broken againAnyway that 
illustrates why these type of approaches are brittle. 

Here's what I think is a better approach.  Design your API and write the 
samples by hand. You may design schema for your XML representation and verify 
that your samples validate. Write tests that check the generated XML and JSON  
that comes out of your service against the hand written versions and, if 
applicable, against the schema.  You know the API is done when the tests pass.  
If a merge  comes in that changes the representation then the tests will fail. 
Either way there's not much you can do in your code that can change's the spec 
from under your client's feet because the spec and the code are separate.  Sure 
during early development the spec and the code may be out of synch, but at 
least the client knows where you're headed and can work to meet you there.

So yes, you should be using those samples for testing. You just shouldn't be 
generating the samples from the code. The code SHOULD DO what the spec says 
verifiably -- not the other way around.


 
 
 Second, I don't think that the core OpenStack API should change with every 
 OpenStack release. There are a number of efforts to provide multiple 
 implementation of an existing OpenStack API.  We should encourage this, but 
 it becomes difficult if the core spec is in constant flux.  Certainly you 
 can use the extension mechanism to bring functionality out to market 
 quickly, but the process of deciding what goes into the core should be more 
 deliberate. Really good specs, shouldn't need to change very often, think 
 HTTP, X11, SMTP, etc. We need to encourage clients to write support for our 
 spec and we need to also encourage other implementors to write 
 implementations for it. These efforts become very difficult if the spec is 
 in constant flux.
 
 Incrementing the version number is only a problem if we fail to support the 
 old versions.  At the rate we are adding new functionality, I think we will 
 easily need a new spec every six months for the foreseeable future.  If there 
 are no reasonable changes in a six month period, we can skip a release and 
 not have a new version of the spec
 

Things have been changing because we've been working hard to realize what the 
code API should be.  Once we have this settled, I don't see a big reason why 
the core spec should change every 6 months. In fact, functionality wise the 
core API you see in 1.1 today hasn't really changed all that much in comparison 
to the 1.0 API.


-jOrGe W.

This email may include confidential information. If you received it in error, 
please delete it.


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp


Re: [Openstack] API Spec

2011-08-22 Thread Jorge Williams

On Aug 22, 2011, at 8:59 PM, Vishvananda Ishaya wrote:

 Inline
 
 On Aug 22, 2011, at 4:15 PM, Jay Pipes wrote:
 
 It may be just me, but having DocBookXML in the source tree is hideous
 to me. Not only does it clutter the source tree with non-RST
 documentation, but as you know, reviewing diffs of XML is just about
 makes people want to slit their throats with a spoon. There is a
 special type of person (Anne!) that seem to be impervious to the
 throat-slitting urge and can successfully digest such a review
 request, but for many, the process is too painful.
 
 I hate xml changes as well, but anne + people in manuals don't really have 
 the expertise to know if something belongs in the api.  *-core should be 
 responsible for the spec for the project.


Agreed there should be someone in the loop that can help validate the changes. 
A manuals person alone should not suffice to let those type of changes go into 
the spec.  I think we can adjust our process to be able to accomplish this.   
Doesn't gerrit help for this sort of stuff? 

 
 In addition to the above gripe, the development, stability, and
 enhancement of the OpenStack APIs can and should (IMHO) be managed
 separately from the source code of the project. The API can evolve in
 the openstack-manuals project and developers can code the OpenStack
 subproject to match the API documented in openstack-manuals project
 (at the tagged version). So, for instance, if the compute API needs to
 change, the API documentation changes would be proposed in
 openstack-manuals, reviewed, discussed and approved. The new API docs
 would be published to something like
 http://docs.openstack.org/compute/2.0/ and developers coding Essex
 features having to do with implementing such a 2.0 API would refer to
 the API documentation there while writing the feature...
 
 If we want to separate the xml code out, we can do a separate nova-spec repo 
 for the spec, but this should be managed by nova-core

Having a separate nova-spec repo is a good idea. I've never been a fan of 
having them all in a single repo.  You still want a writer in the loop to check 
style, apply the latest templates, do general editing etc.

-jOrGe W.
This email may include confidential information. If you received it in error, 
please delete it.


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp


Re: [Openstack] API Spec

2011-08-22 Thread Jorge Williams
Inline

On Aug 22, 2011, at 9:12 PM, Anne Gentle wrote:

I think it makes sense to have an openstack-api project with all the API docs 
(specs and learning materials) gathered in one place. I think it's preferable 
to have the API separated out from the code for several reasons - ease of 
review, ease of check out, also for learning materials for the API itself.


+1

I'd envision these would go in it for starters:

Compute API (docbook, core + extensions)
Glance API (RST to docbook, core)
Keystone API (docbook, incubation, core + extensions)
Swift API (docbook, core)

Notes:
- Yep, Keystone moved their docbook out of the core code base due to the 
overhead associated with a large-ish document ensconced with the code.
- Glance's API is documented in a single RST page. We have a simple RST to 
docbook tool inhouse at Rackspace that lets me take updates and move them into 
docbook.
- Just today I had a request to bring the Load Balancing API into 
openstack-manuals for comments and review from 
http://wiki.openstack.org/Atlas-LB, since our wiki doesn't enable comments. I'm 
not sure what to do with nascent APIs for review that aren't yet in incubation.

So these are some of my observations having worked with the API docs for a 
while, to consider while seeking the ideal solution:
Incubation - how do we indicate an API is incubated?
Pre-incubation/review period - how could a team offer an API up for community 
review and commenting?
Core - how do we indicate what is core and how to get extensions? At Rackspace 
the Compute API team is working on a solution to getting extensions and telling 
people how to use them once they're available.
Source - DocBook is the source for the two biggest API docs, Compute and Object 
Storage, Keystone is a close third, and I can get DocBook out of Glance. Do we 
need to set DocBook as the standard source?
Output - I'd also like to focus on not only API specs but also deliverables 
that help people learn the APIs, such as the frameworks recently opensourced by 
Mashery (example: http://developer.klout.com/iodocs) and Wordnik 
(http://swagger.wordnik.com/). If we also deliver this type of web tool, we'd 
also need JSON or XML as source files (many of which are already embedded into 
the DocBook).

I'd like the best of both worlds - API specs and self-documenting APIs. I think 
we can get there, and I think a separate API project with a core review team 
moves us in that direction.


+1

Thanks for the good discussion here.
Anne


Anne Gentle
http://www.facebook.com/conversationandcommunity
my bloghttp://justwriteclick.com/ | my 
bookhttp://xmlpress.net/publications/conversation-community/ | 
LinkedInhttp://www.linkedin.com/in/annegentle | 
Delicioushttp://del.icio.us/annegentle | 
Twitterhttp://twitter.com/annegentle

On Mon, Aug 22, 2011 at 7:49 PM, Jan Drake 
jan_dr...@hotmail.commailto:jan_dr...@hotmail.com wrote:
+1




On Aug 22, 2011, at 5:06 PM, Jay Pipes 
jaypi...@gmail.commailto:jaypi...@gmail.com wrote:

 ++

 On Mon, Aug 22, 2011 at 7:59 PM, Jorge Williams
 jorge.willi...@rackspace.commailto:jorge.willi...@rackspace.com wrote:
 Hi Vish,
 I don't have a problem moving the spec out of docs manuals and into another
 project even the nova repo.   But, I do have a number of issues with the
 approach that you're proposing. First, I think that fundamentally there
 should be a decoupling of the spec and the implementation.   If you have the
 spec generated from the code than essentially the spec is whatever the code
 does. It's very difficult to interoperate with specs that are generated this
 way as the specs tend to be very brittle and opaque (since you have to study
 the code). If you introduce a  bug in the code that bug filters it's way all
 the way to the spec (this was a big problem with SOAP and CORBA). It's
 difficult to detect errors because you cant validate. By keeping the
 implementation and the spec separate you can validate one against the other.

 Second, I don't think that the core OpenStack API should change with every
 OpenStack release. There are a number of efforts to provide multiple
 implementation of an existing OpenStack API.  We should encourage this, but
 it becomes difficult if the core spec is in constant flux.  Certainly you
 can use the extension mechanism to bring functionality out to market
 quickly, but the process of deciding what goes into the core should be more
 deliberate. Really good specs, shouldn't need to change very often, think
 HTTP, X11, SMTP, etc. We need to encourage clients to write support for our
 spec and we need to also encourage other implementors to write
 implementations for it. These efforts become very difficult if the spec is
 in constant flux.
 -jOrGe W.
 On Aug 22, 2011, at 5:43 PM, Vishvananda Ishaya wrote:

 Hey Everyone,
 We discussed at the Diablo design summit having API spec changes be proposed
 along with code changes and reviewed according to the merge process that we
 use for code.  This has been 

Re: [Openstack] API Spec

2011-08-22 Thread Jorge Williams

I say we up the version number when we can't ensure backward compatibility.  As 
to how long older versions should be supported.  Hard to say.  It depends on a 
lot of factors, and at the end of the day it may come up to how popular a 
version is and how  willing and able operators and client devs are to upgrading.

-jOrGe W.


On Aug 22, 2011, at 8:49 PM, Thor Wolpert wrote:

 I agree the Specs shouldn't change often ... but just to use your
 examples, they where all simplifications of larger specs that took
 years to create.
 
 If an API changes and is deprecated, how long does backwards
 compatibility stay in place?
 
 Thanks,
 Thor W
 
 On Mon, Aug 22, 2011 at 5:49 PM, Jan Drake jan_dr...@hotmail.com wrote:
 +1
 
 
 
 
 On Aug 22, 2011, at 5:06 PM, Jay Pipes jaypi...@gmail.com wrote:
 
 ++
 
 On Mon, Aug 22, 2011 at 7:59 PM, Jorge Williams
 jorge.willi...@rackspace.com wrote:
 Hi Vish,
 I don't have a problem moving the spec out of docs manuals and into another
 project even the nova repo.   But, I do have a number of issues with the
 approach that you're proposing. First, I think that fundamentally there
 should be a decoupling of the spec and the implementation.   If you have 
 the
 spec generated from the code than essentially the spec is whatever the code
 does. It's very difficult to interoperate with specs that are generated 
 this
 way as the specs tend to be very brittle and opaque (since you have to 
 study
 the code). If you introduce a  bug in the code that bug filters it's way 
 all
 the way to the spec (this was a big problem with SOAP and CORBA). It's
 difficult to detect errors because you cant validate. By keeping the
 implementation and the spec separate you can validate one against the 
 other.
 
 Second, I don't think that the core OpenStack API should change with every
 OpenStack release. There are a number of efforts to provide multiple
 implementation of an existing OpenStack API.  We should encourage this, but
 it becomes difficult if the core spec is in constant flux.  Certainly you
 can use the extension mechanism to bring functionality out to market
 quickly, but the process of deciding what goes into the core should be more
 deliberate. Really good specs, shouldn't need to change very often, think
 HTTP, X11, SMTP, etc. We need to encourage clients to write support for our
 spec and we need to also encourage other implementors to write
 implementations for it. These efforts become very difficult if the spec is
 in constant flux.
 -jOrGe W.
 On Aug 22, 2011, at 5:43 PM, Vishvananda Ishaya wrote:
 
 Hey Everyone,
 We discussed at the Diablo design summit having API spec changes be 
 proposed
 along with code changes and reviewed according to the merge process that we
 use for code.  This has been impossible up until now because the canonical
 spec has been in the openstack-manuals project.
 My suggestion is that we move the openstack-compute spec into the nova
 source tree.  During a six-month release we can propose changes to the spec
 by proposing along with the code that changes it.  In the final freeze for
 the release, we can increment the spec version number and copy the current
 version of the spec into openstack-manuals and that will be the locked down
 spec for that release.
 This means that openstack 1.1 will be the official spec for diablo, at 
 which
 point we will start working on a new api (we can call it 1.2 but it might 
 be
 best to use a temporary name like 'latest') during the essex release cycle,
 then at essex release we lock the spec down and it becomes the new version
 of the openstack api.
 Ultimately I would like the spec to be generated from the code, but as a
 first pass, we should at least be able to edit the future version of the
 spec as we make changes.  I've proposed the current version of the spec
 here:
 https://code.launchpad.net/~vishvananda/nova/add-api-docs/+merge/72506
 Are there any issues with this approach?
 Vish
 
 This email may include confidential information. If you received it in
 error, please delete it.
 ___
 Mailing list: https://launchpad.net/~openstack
 Post to : openstack@lists.launchpad.net
 Unsubscribe : https://launchpad.net/~openstack
 More help   : https://help.launchpad.net/ListHelp
 
 
 
 ___
 Mailing list: https://launchpad.net/~openstack
 Post to : openstack@lists.launchpad.net
 Unsubscribe : https://launchpad.net/~openstack
 More help   : https://help.launchpad.net/ListHelp
 
 ___
 Mailing list: https://launchpad.net/~openstack
 Post to : openstack@lists.launchpad.net
 Unsubscribe : https://launchpad.net/~openstack
 More help   : https://help.launchpad.net/ListHelp
 
 
 ___
 Mailing list: https://launchpad.net/~openstack
 Post to : openstack@lists.launchpad.net
 Unsubscribe : https://launchpad.net/~openstack
 More help   :