Re: [openstack-dev] [api] Optional Properties in an Entity

[Kevin L. Mitchell]

On Mon, 2015-01-19 at 19:55 +, Douglas Mendizabal wrote:
> I’m curious about something that came up during a bug discussion in
> one of the Barbican weekly meetings.  The question is about optional
> properties in an entity.  e.g. We have a Secret entity that has some
> properties that are optional, such as the Secret’s name.  We were
> split on what the best approach for returning the secret
> representation would be when an optional property is not set.
> 
> In one camp, some developers would like to see the properties returned
> no matter what.  That is to say, the Secret dictionary would include a
> key for “name” set to null every single time.  i.e.
[snip]
> On the other camp, some developers would like to see optional
> properties omitted if they were not set by the user.
> 
> The advantage of always returning the property is that the response is
> easier to parse, since you don’t have to check for the existence of
> the optional keys.  The argument against it is that it makes the API
> more rigid, and clients more fragile.

I keep trying to come up with technical arguments for or against, and
the only one I can come up with that has any true meaning is that
omitting properties reduces bandwidth usage a little…but I don't really
think that's something we've particularly concerned about.  Thus, from a
technical perspective, either way is perfectly fine, and we just have to
answer consistency.

When we look at consistency, we look at everything else in OpenStack.
From the standpoint of the nova API (with which I am the most familiar),
I am not aware of any property that is ever omitted from any payload
without versioning coming in to the picture, even if its value is null.
Thus, I would argue that we should encourage the first situation, where
all properties are included, even if their value is null.
-- 
Kevin L. Mitchell 
Rackspace


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [api] Optional Properties in an Entity

[Dean Troyer]

On Mon, Jan 19, 2015 at 3:54 PM, Kevin L. Mitchell <
kevin.mitch...@rackspace.com> wrote:

> When we look at consistency, we look at everything else in OpenStack.
> From the standpoint of the nova API (with which I am the most familiar),
> I am not aware of any property that is ever omitted from any payload
> without versioning coming in to the picture, even if its value is null.
> Thus, I would argue that we should encourage the first situation, where
> all properties are included, even if their value is null.
>

Independent of actual implementations in OpenStack, I prefer always
including null/empty properties here because it is slightly more
self-documenting.  Having spent the morning chasing down attributes for an
API to be named at a later date by looking at server code, we do not help
ourselves or the users of our APIs by omitting this sort of thing.

dt

-- 

Dean Troyer
dtro...@gmail.com
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [api] Optional Properties in an Entity

[Chris Dent]

On Mon, 19 Jan 2015, Dean Troyer wrote:


Independent of actual implementations in OpenStack, I prefer always
including null/empty properties here because it is slightly more
self-documenting.  Having spent the morning chasing down attributes for an
API to be named at a later date by looking at server code, we do not help
ourselves or the users of our APIs by omitting this sort of thing.


+1

--
Chris Dent tw:@anticdent freenode:cdent
https://tank.peermore.com/tanks/cdent

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [api] Optional Properties in an Entity

[Ian Cordasco]

On 1/20/15, 04:13, "Chris Dent"  wrote:

>On Mon, 19 Jan 2015, Dean Troyer wrote:
>
>> Independent of actual implementations in OpenStack, I prefer always
>> including null/empty properties here because it is slightly more
>> self-documenting.  Having spent the morning chasing down attributes for
>>an
>> API to be named at a later date by looking at server code, we do not
>>help
>> ourselves or the users of our APIs by omitting this sort of thing.
>
>+1

I’m in much the same position as Kevin. I’ve tried coming up with
reasoning for both positions but I don’t see a really compelling reason
for either side. On the one hand though, having strict schema about what
is returned can be valuable, so not allowing something to be omitted
**may** catch a bug somewhere else. Allowing things to be nullable is
perfectly reasonable to me.

It seems like a few of us are in agreement with this direction. Perhaps
one of us should write a proposal for the API-WG to review about this.

Cheers,
Ian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [api] Optional Properties in an Entity

[Brian Rosmaita]

From: Kevin L. Mitchell [kevin.mitch...@rackspace.com]
Sent: Monday, January 19, 2015 4:54 PM

> When we look at consistency, we look at everything else in OpenStack.
> From the standpoint of the nova API (with which I am the most familiar),
> I am not aware of any property that is ever omitted from any payload
> without versioning coming in to the picture, even if its value is null.
> Thus, I would argue that we should encourage the first situation, where
> all properties are included, even if their value is null.

That is not the case for the Images API v2:

"An image is always guaranteed to have the following attributes: id,
status, visibility, protected, tags, created_at, file and self. The other
attributes defined in the image schema below are guaranteed to
be defined, but is only returned with an image entity if they have
been explicitly set." [1]


[1] 
http://docs.openstack.org/api/openstack-image-service/2.0/content/image-entities.html
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [api] Optional Properties in an Entity

[Jay Pipes]

On 01/19/2015 02:55 PM, Douglas Mendizabal wrote:

Hi API WG,

I’m curious about something that came up during a bug discussion in one of the 
Barbican weekly meetings.  The question is about optional properties in an 
entity.  e.g. We have a Secret entity that has some properties that are 
optional, such as the Secret’s name.  We were split on what the best approach 
for returning the secret representation would be when an optional property is 
not set.

In one camp, some developers would like to see the properties returned no 
matter what.  That is to say, the Secret dictionary would include a key for 
“name” set to null every single time.  i.e.

{
   …
   “secret”: {
 “name”: null,
 …
   }
   ...
}

On the other camp, some developers would like to see optional properties 
omitted if they were not set by the user.

The advantage of always returning the property is that the response is easier 
to parse, since you don’t have to check for the existence of the optional keys. 
 The argument against it is that it makes the API more rigid, and clients more 
fragile.

I was wondering what the API Working Group’s thoughts are on this?


My opinion is that attributes should always be in the returned result 
(that corresponds to a particular version of an API), set to null when 
there is no value set.


Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [api] Optional Properties in an Entity

[Jay Pipes]

On 01/20/2015 10:54 AM, Brian Rosmaita wrote:

From: Kevin L. Mitchell [kevin.mitch...@rackspace.com]
Sent: Monday, January 19, 2015 4:54 PM


When we look at consistency, we look at everything else in OpenStack.
 From the standpoint of the nova API (with which I am the most familiar),
I am not aware of any property that is ever omitted from any payload
without versioning coming in to the picture, even if its value is null.
Thus, I would argue that we should encourage the first situation, where
all properties are included, even if their value is null.


That is not the case for the Images API v2:

"An image is always guaranteed to have the following attributes: id,
status, visibility, protected, tags, created_at, file and self. The other
attributes defined in the image schema below are guaranteed to
be defined, but is only returned with an image entity if they have
been explicitly set." [1]


This was a mistake, IMHO. Having entirely extensible schemas means that 
there is little guaranteed consistency across implementations of the API.


This is the same reason that I think API extensions are an abomination.

Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [api] Optional Properties in an Entity

[Joe Gordon]

On Mon, Feb 9, 2015 at 1:22 PM, Jay Pipes  wrote:

> On 01/20/2015 10:54 AM, Brian Rosmaita wrote:
>
>> From: Kevin L. Mitchell [kevin.mitch...@rackspace.com]
>> Sent: Monday, January 19, 2015 4:54 PM
>>
>>  When we look at consistency, we look at everything else in OpenStack.
>>>  From the standpoint of the nova API (with which I am the most familiar),
>>> I am not aware of any property that is ever omitted from any payload
>>> without versioning coming in to the picture, even if its value is null.
>>> Thus, I would argue that we should encourage the first situation, where
>>> all properties are included, even if their value is null.
>>>
>>
>> That is not the case for the Images API v2:
>>
>> "An image is always guaranteed to have the following attributes: id,
>> status, visibility, protected, tags, created_at, file and self. The other
>> attributes defined in the image schema below are guaranteed to
>> be defined, but is only returned with an image entity if they have
>> been explicitly set." [1]
>>
>
> This was a mistake, IMHO. Having entirely extensible schemas means that
> there is little guaranteed consistency across implementations of the API.
>

+1, Subtle hard to discover differences between clouds is a pain for
interchangeability.


>
> This is the same reason that I think API extensions are an abomination.
>
> Best,
> -jay
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [api] Optional Properties in an Entity

[Morgan Fainberg]

On February 9, 2015 at 1:25:58 PM, Jay Pipes (jaypi...@gmail.com) wrote:
On 01/20/2015 10:54 AM, Brian Rosmaita wrote: 
> From: Kevin L. Mitchell [kevin.mitch...@rackspace.com] 
> Sent: Monday, January 19, 2015 4:54 PM 
> 
>> When we look at consistency, we look at everything else in OpenStack. 
>> From the standpoint of the nova API (with which I am the most familiar), 
>> I am not aware of any property that is ever omitted from any payload 
>> without versioning coming in to the picture, even if its value is null. 
>> Thus, I would argue that we should encourage the first situation, where 
>> all properties are included, even if their value is null. 
> 
> That is not the case for the Images API v2: 
> 
> "An image is always guaranteed to have the following attributes: id, 
> status, visibility, protected, tags, created_at, file and self. The other 
> attributes defined in the image schema below are guaranteed to 
> be defined, but is only returned with an image entity if they have 
> been explicitly set." [1] 

This was a mistake, IMHO. Having entirely extensible schemas means that 
there is little guaranteed consistency across implementations of the API. 

This is the same reason that I think API extensions are an abomination. 


This right here! +1. +more than +1 if I get more votes on this.

Best, 
-jay 

__ 
OpenStack Development Mailing List (not for usage questions) 
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe 
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev 
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [api] Optional Properties in an Entity

[Brian Rosmaita]

On 2/9/15, 8:44 PM, "Joe Gordon" 
mailto:joe.gord...@gmail.com>> wrote:

On Mon, Feb 9, 2015 at 1:22 PM, Jay Pipes 
mailto:jaypi...@gmail.com>> wrote:
On 01/20/2015 10:54 AM, Brian Rosmaita wrote:
From: Kevin L. Mitchell 
[kevin.mitch...@rackspace.com]
Sent: Monday, January 19, 2015 4:54 PM

When we look at consistency, we look at everything else in OpenStack.
 From the standpoint of the nova API (with which I am the most familiar),
I am not aware of any property that is ever omitted from any payload
without versioning coming in to the picture, even if its value is null.
Thus, I would argue that we should encourage the first situation, where
all properties are included, even if their value is null.

That is not the case for the Images API v2:

"An image is always guaranteed to have the following attributes: id,
status, visibility, protected, tags, created_at, file and self. The other
attributes defined in the image schema below are guaranteed to
be defined, but is only returned with an image entity if they have
been explicitly set." [1]

This was a mistake, IMHO. Having entirely extensible schemas means that there 
is little guaranteed consistency across implementations of the API.

+1, Subtle hard to discover differences between clouds is a pain for 
interchangeability.

Jay and Joe, thanks for weighing in.  I’m still not convinced that the course 
taken in the Images v2 API was a mistake, though.  (I wasn’t involved in its 
initial design, so this isn’t personal, just curiosity.)  Here are a few 
reasons why, maybe someone can set me straight?

(1) Leaving null elements out is parsimonious.
As long as there’s a JSON schema, the client has a good idea what to expect.  
If you include
  “whatever”: null
in the response, I don’t see what that buys you.  If you simply don’t include 
the “whatever” element, the recipient knows it’s not set.  If you do include it 
set to null, you know that it’s not set … and you increased the size of the 
response payload without increasing its informativeness.  Further, even if you 
include the “whatever” element set to null, the client is still going to have 
to check it to handle the null case, so it’s really just a matter of how the 
client checks, not whether it has to check.

(2) Leaving null elements out doesn’t affect interchangeability.
If our convention is that unset elements aren’t included, and we’ve got a JSON 
schema, then everyone knows what’s up.  Further, looking specifically at the 
use cases for images in Glance, different clouds have different sets of image 
properties that they use for specific purposes that may be unique to their 
cloud.  For example, some may put a hyperlink to licensing info in an image 
property, or versioning info, or package lists, or whatever you can fit in 255 
chars.  So a client (intelligent or not) connecting to various clouds can’t 
expect to find the same set of properties defined in every cloud (except for 
the ones guaranteed by contract, which are listed above).  Thus, you’re going 
to have to deal with the problem of non-existent elements when you get to the 
additionalProperties in JSON no matter what.  But as long as you know this, 
you’re OK.  I think it’s a much bigger problem when you’ve got a mixture of 
null, “”, {} and other ways of conveying empty elements in a response.  By 
simply leaving properties out, there’s no question that they’re not set.

(3) A little consistency is a good thing.
Jay mentions that having entirely extensible schemas means that there’s little 
guaranteed consistency across implementations of the API.  In the Images API v2 
case, the schema isn’t entirely extensible, you can add string-valued 
additionalProperties.  So there’s that.  But the bigger picture is that we’re 
in at the infancy of clouds and cloud management, there’s no way we can 
anticipate the set of Image properties that will be sufficient for all 
deployers.  So as long as the consistency guarantees are met for the small set 
of properties they’re guaranteed for, I don’t have a problem with the majority 
of image properties being variable … as long as we know what type each is, 
which we do, they’re all strings.


cheers,
brian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [api] Optional Properties in an Entity

[Jay Pipes]

Hi Brian, thanks for the response. Some comments inline :)

On 02/11/2015 09:57 AM, Brian Rosmaita wrote:

On 2/9/15, 8:44 PM, "Joe Gordon" mailto:joe.gord...@gmail.com>> wrote:


On Mon, Feb 9, 2015 at 1:22 PM, Jay Pipes mailto:jaypi...@gmail.com>> wrote:

On 01/20/2015 10:54 AM, Brian Rosmaita wrote:

From: Kevin L. Mitchell [kevin.mitch...@rackspace.com
]
Sent: Monday, January 19, 2015 4:54 PM

When we look at consistency, we look at everything else
in OpenStack.
  From the standpoint of the nova API (with which I am
the most familiar),
I am not aware of any property that is ever omitted from
any payload
without versioning coming in to the picture, even if its
value is null.
Thus, I would argue that we should encourage the first
situation, where
all properties are included, even if their value is null.


That is not the case for the Images API v2:

"An image is always guaranteed to have the following
attributes: id,
status, visibility, protected, tags, created_at, file and
self. The other
attributes defined in the image schema below are guaranteed to
be defined, but is only returned with an image entity if
they have
been explicitly set." [1]


This was a mistake, IMHO. Having entirely extensible schemas
means that there is little guaranteed consistency across
implementations of the API.


+1, Subtle hard to discover differences between clouds is a pain for
interchangeability.


Jay and Joe, thanks for weighing in.  I’m still not convinced that the
course taken in the Images v2 API was a mistake, though.  (I wasn’t
involved in its initial design, so this isn’t personal, just curiosity.)
  Here are a few reasons why, maybe someone can set me straight?

(1) Leaving null elements out is parsimonious.
As long as there’s a JSON schema, the client has a good idea what to
expect.  If you include
   “whatever”: null
in the response, I don’t see what that buys you.  If you simply don’t
include the “whatever” element, the recipient knows it’s not set.  If
you do include it set to null, you know that it’s not set … and you
increased the size of the response payload without increasing its
informativeness.  Further, even if you include the “whatever” element
set to null, the client is still going to have to check it to handle the
null case, so it’s really just a matter of how the client checks, not
whether it has to check.


Agreed, it doesn't buy you much at all. I'm more interested in just 
being consistent across APIs regarding this.



(2) Leaving null elements out doesn’t affect interchangeability.
If our convention is that unset elements aren’t included, and we’ve got
a JSON schema, then everyone knows what’s up.  Further, looking
specifically at the use cases for images in Glance, different clouds
have different sets of image properties that they use for specific
purposes that may be unique to their cloud.


And this, right here, is not something we should encourage. Tag images 
with whatever free-form tags you wish, as a user, but deployers of the 
Glance image service should be able to say attribute XYZ means the same 
thing across different deployments of Glance. Otherwise, there's no use 
to having those attributes, IMO, since you cannot rely on them meaning 
the same thing.


>  For example, some may put a

hyperlink to licensing info in an image property, or versioning info, or
package lists, or whatever you can fit in 255 chars.  So a client
(intelligent or not) connecting to various clouds can’t expect to find
the same set of properties defined in every cloud (except for the ones
guaranteed by contract, which are listed above).  Thus, you’re going to
have to deal with the problem of non-existent elements when you get to
the additionalProperties in JSON no matter what.  But as long as you
know this, you’re OK.  I think it’s a much bigger problem when you’ve
got a mixture of null, “”, {} and other ways of conveying empty elements
in a response.  By simply leaving properties out, there’s no question
that they’re not set.


I do not think that additionalProperties should ever be anything other 
than false for any public API.



(3) A little consistency is a good thing.
Jay mentions that having entirely extensible schemas means that there’s
little guaranteed consistency across implementations of the API.  In the
Images API v2 case, the schema isn’t entirely extensible, you can add
string-valued additionalProperties.  So there’s that.  But the bigger
picture is that we’re in at the infancy of clouds and cloud management,
there’s no way we can anticipate the set of Image properties that will
be sufficient for a