[PROPOSAL] Collect and provide generic key/value pairs to storage plug-ins

[Mike Tutkowski]

Hi,

Please feel free to review the following proposal for 4.5:

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=42566111

Here is the summary:

Today the way you associate a Compute Offering (CO) or a Disk Offering (DO)
with a Primary Storage (PS) is via storage tagging.

This has some benefits and drawbacks.

One benefit is being able to have some level of vendor independence from
the point of view of the CO or DO. For example, if the storage tag of a DO
is "Fast", then this can be satisfied by PS that describes itself as
"Fast", regardless of vendor.

A major drawback with the storage-tagging approach, however, is that you
are not easily able to leverage vendor-specific features, which is often
why you bought storage from the vendor in question to begin with.

Ideally we do not want to add each vendor's features into the system as
properties that can be seen by the admin regardless of whether or not the
underlying storage he's actually using supports the feature in question.
Traditionally, however, this has been business as usual in the CloudStack
codebase.

Going forward, we want to implement a more fine-grain and generic approach.

For example, in the GUI we would like to have a storage provider field for
the CO and DO windows (this equates to the name of one and only one storage
provider). If the admin inputs a specific storage provider, he can enter in
an arbitrary number of key/value pairs in another text field (perhaps we
would provide a nice entry dialog to make this easier in the GUI). These
key value pairs can be passed into the storage driver when it's asked to
create (or update) a volume and the storage driver can decide what each and
every key/value pair means, if anything.

Thanks!

-- 
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkow...@solidfire.com
o: 303.746.7302
Advancing the way the world uses the cloud
*™*

Re: [PROPOSAL] Collect and provide generic key/value pairs to storage plug-ins

[Wido den Hollander]

On 06/11/2014 12:26 AM, Mike Tutkowski wrote:

Hi,

Please feel free to review the following proposal for 4.5:

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=42566111



Is this what we discussed at CCC13 in Amsterdam? Seems like it!

I'm in favor, since this could add a lot of potentials to the Ceph 
storage as well.


Wido


Here is the summary:

Today the way you associate a Compute Offering (CO) or a Disk Offering (DO)
with a Primary Storage (PS) is via storage tagging.

This has some benefits and drawbacks.

One benefit is being able to have some level of vendor independence from
the point of view of the CO or DO. For example, if the storage tag of a DO
is "Fast", then this can be satisfied by PS that describes itself as
"Fast", regardless of vendor.

A major drawback with the storage-tagging approach, however, is that you
are not easily able to leverage vendor-specific features, which is often
why you bought storage from the vendor in question to begin with.

Ideally we do not want to add each vendor's features into the system as
properties that can be seen by the admin regardless of whether or not the
underlying storage he's actually using supports the feature in question.
Traditionally, however, this has been business as usual in the CloudStack
codebase.

Going forward, we want to implement a more fine-grain and generic approach.

For example, in the GUI we would like to have a storage provider field for
the CO and DO windows (this equates to the name of one and only one storage
provider). If the admin inputs a specific storage provider, he can enter in
an arbitrary number of key/value pairs in another text field (perhaps we
would provide a nice entry dialog to make this easier in the GUI). These
key value pairs can be passed into the storage driver when it's asked to
create (or update) a volume and the storage driver can decide what each and
every key/value pair means, if anything.

Thanks!

Re: [PROPOSAL] Collect and provide generic key/value pairs to storage plug-ins

[Mike Tutkowski]

Yep :) This is what we talked about at CCCEU13.

On Wednesday, June 11, 2014, Wido den Hollander  wrote:

> On 06/11/2014 12:26 AM, Mike Tutkowski wrote:
>
>> Hi,
>>
>> Please feel free to review the following proposal for 4.5:
>>
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=42566111
>>
>>
> Is this what we discussed at CCC13 in Amsterdam? Seems like it!
>
> I'm in favor, since this could add a lot of potentials to the Ceph storage
> as well.
>
> Wido
>
>  Here is the summary:
>>
>> Today the way you associate a Compute Offering (CO) or a Disk Offering
>> (DO)
>> with a Primary Storage (PS) is via storage tagging.
>>
>> This has some benefits and drawbacks.
>>
>> One benefit is being able to have some level of vendor independence from
>> the point of view of the CO or DO. For example, if the storage tag of a DO
>> is "Fast", then this can be satisfied by PS that describes itself as
>> "Fast", regardless of vendor.
>>
>> A major drawback with the storage-tagging approach, however, is that you
>> are not easily able to leverage vendor-specific features, which is often
>> why you bought storage from the vendor in question to begin with.
>>
>> Ideally we do not want to add each vendor's features into the system as
>> properties that can be seen by the admin regardless of whether or not the
>> underlying storage he's actually using supports the feature in question.
>> Traditionally, however, this has been business as usual in the CloudStack
>> codebase.
>>
>> Going forward, we want to implement a more fine-grain and generic
>> approach.
>>
>> For example, in the GUI we would like to have a storage provider field for
>> the CO and DO windows (this equates to the name of one and only one
>> storage
>> provider). If the admin inputs a specific storage provider, he can enter
>> in
>> an arbitrary number of key/value pairs in another text field (perhaps we
>> would provide a nice entry dialog to make this easier in the GUI). These
>> key value pairs can be passed into the storage driver when it's asked to
>> create (or update) a volume and the storage driver can decide what each
>> and
>> every key/value pair means, if anything.
>>
>> Thanks!
>>
>>
>

-- 
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkow...@solidfire.com
o: 303.746.7302
Advancing the way the world uses the cloud
*™*

Re: [PROPOSAL] Collect and provide generic key/value pairs to storage plug-ins

[John Burwell]

Mike,

I completely agree with the need to attach driver-specified key-value pairs to 
a storage driver (we have been batting this around for nearly a year).  
However, I think this facility should be generalized to support all drivers 
(e.g. network, compute, etc) where storage maybe the first layer to implement 
it.  For example, we have a raft of places where vendor specific data/concepts 
leak into the object model and schema.  For example, in networking, we have 
vendor specific API calls. One of the main value propositions of a cloud 
platform is to provide a unified abstraction for underlying infrastructure 
components.  Furthermore, requiring vendor implements to add API calls further 
increases the complexity and effort required to support CloudStack — 
discouraging their participation.

In terms of the design of itself, I don’t feel that the Map 
getPropertiesAndTypes is a rich enough semantic for such an extension facility. 
 First, it does not provide a mechanism to validate that the form and content 
of the values being set.  Second, it is incredible lose contract that does not 
exploit the Java compiler to ensure drivers are providing well-formed metadata 
to the CloudStack orchestration components.  Finally, it does not support 
notions such as hinting which fields are required (very useful for creating 
useful UIs) and constrained values (rendered as drop downs).  My thoughts are 
as follows:

Define an ExtenstionPropertyType enumeration with the following types permitted:
STRING
DATE
NUMBER
BOOLEAN
The following methods as well:
Class getType(): The underlying Java type used to represent the type
Object convertFromString(String aValue): Converts a value from a string to the 
underlying Java type
String convertToString(Object aValue): Converts a value to a string from the 
underlying Java type
Define a ExtensionPropertyDefinition interface that describes the following 
attributes about a extension property:
String getName(): The name of the property conforming to JavaBean standards 
(while we don’t support reflection, it’s a well known standard by Java 
developers …)
ExtensionPropertyType getType():  The type of the property
boolean isRequired(): Whether or not a value for the property is required
boolean isCollected(): Whether or not the property’s value is a single value or 
collected (default to a list)
Set getConstraints: A list of the values that constrain the values of 
this extension property.
List validate(Object aValue): A callback to validate a value for this 
property definition — returning a list of strings describing the validation 
errors.  An empty list implies a valid value.  An additional nicety for this 
interface would be to define a type which the definition would expect of the 
underlying value to be at runtime. 
Introduce an Extendable interface that indicates that a type can have its data 
extended with a set of key-value pairs.  I would define the following methods:
Set getExtenstionPropertyDefinitions(): The set 
of extension property descriptions
Object getExtensionPropertyValue(String aPropertyName): Get a value for an 
extension property
List validateExtensionProperties(): Validates the values of the 
extension properties

There are likely pieces I missed and refinements to improve this design — I 
jotted it down off the top of my head.  However, we have to ensue that the 
semantic is rich enough to render friendly UIs, as well as, protect against 
GIGO (garbage in, garbage out).  

Finally, I would like to see the design expanded to explain when and where this 
information will pass into drivers for use.  As the design stands now, it does 
not explain how the data would actually be used during operation.  The scope of 
these changes is important to understand for both impact analysis and driver 
developers and implementors.  Also, I don’t understand how the addition of 
generic key-values to drivers would eliminate tags.  As I understand them, tags 
are used to establish valid combinations of drivers (e.g. mutual exclusion) to 
avoid defining an unimplementable infrastructure configuration.  While I agree 
with the desire to eliminate them, I don’t understand how such a facility would 
contribute to that goal.

Great work putting together an initial design, and getting this important 
conversation started.

Thanks,
-John

On Jun 10, 2014, at 6:26 PM, Mike Tutkowski  
wrote:

> Hi,
> 
> Please feel free to review the following proposal for 4.5:
> 
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=42566111
> 
> Here is the summary:
> Today the way you associate a Compute Offering (CO) or a Disk Offering (DO) 
> with a Primary Storage (PS) is via storage tagging.
> This has some benefits and drawbacks.
> One benefit is being able to have some level of vendor independence from the 
> point of view of the CO or DO. For example, if the storage tag of a DO is 
> "Fast", then this can be satisfied by PS that describes itself as "Fast", 
> regardless of 

Re: [PROPOSAL] Collect and provide generic key/value pairs to storage plug-ins

[Daan Hoogland]

+1 to Johns first comment; I think we need it in a more generic way
then for storage plugins. I am sure network
providers/loadbalancers/firewalls need such properties as well.

John, should we make your ExtensionPropertyType a generic and let the
convert-methods deal with the parameter types instead of with Objects?

greatings

On Fri, Jun 13, 2014 at 4:38 PM, John Burwell  wrote:
> Mike,
>
> I completely agree with the need to attach driver-specified key-value pairs 
> to a storage driver (we have been batting this around for nearly a year).  
> However, I think this facility should be generalized to support all drivers 
> (e.g. network, compute, etc) where storage maybe the first layer to implement 
> it.  For example, we have a raft of places where vendor specific 
> data/concepts leak into the object model and schema.  For example, in 
> networking, we have vendor specific API calls. One of the main value 
> propositions of a cloud platform is to provide a unified abstraction for 
> underlying infrastructure components.  Furthermore, requiring vendor 
> implements to add API calls further increases the complexity and effort 
> required to support CloudStack — discouraging their participation.
>
> In terms of the design of itself, I don’t feel that the Map 
> getPropertiesAndTypes is a rich enough semantic for such an extension 
> facility.  First, it does not provide a mechanism to validate that the form 
> and content of the values being set.  Second, it is incredible lose contract 
> that does not exploit the Java compiler to ensure drivers are providing 
> well-formed metadata to the CloudStack orchestration components.  Finally, it 
> does not support notions such as hinting which fields are required (very 
> useful for creating useful UIs) and constrained values (rendered as drop 
> downs).  My thoughts are as follows:
>
> Define an ExtenstionPropertyType enumeration with the following types 
> permitted:
> STRING
> DATE
> NUMBER
> BOOLEAN
> The following methods as well:
> Class getType(): The underlying Java type used to represent the type
> Object convertFromString(String aValue): Converts a value from a string to 
> the underlying Java type
> String convertToString(Object aValue): Converts a value to a string from the 
> underlying Java type
> Define a ExtensionPropertyDefinition interface that describes the following 
> attributes about a extension property:
> String getName(): The name of the property conforming to JavaBean standards 
> (while we don’t support reflection, it’s a well known standard by Java 
> developers …)
> ExtensionPropertyType getType():  The type of the property
> boolean isRequired(): Whether or not a value for the property is required
> boolean isCollected(): Whether or not the property’s value is a single value 
> or collected (default to a list)
> Set getConstraints: A list of the values that constrain the values of 
> this extension property.
> List validate(Object aValue): A callback to validate a value for this 
> property definition — returning a list of strings describing the validation 
> errors.  An empty list implies a valid value.  An additional nicety for this 
> interface would be to define a type which the definition would expect of the 
> underlying value to be at runtime.
> Introduce an Extendable interface that indicates that a type can have its 
> data extended with a set of key-value pairs.  I would define the following 
> methods:
> Set getExtenstionPropertyDefinitions(): The set 
> of extension property descriptions
> Object getExtensionPropertyValue(String aPropertyName): Get a value for an 
> extension property
> List validateExtensionProperties(): Validates the values of the 
> extension properties
>
> There are likely pieces I missed and refinements to improve this design — I 
> jotted it down off the top of my head.  However, we have to ensue that the 
> semantic is rich enough to render friendly UIs, as well as, protect against 
> GIGO (garbage in, garbage out).
>
> Finally, I would like to see the design expanded to explain when and where 
> this information will pass into drivers for use.  As the design stands now, 
> it does not explain how the data would actually be used during operation.  
> The scope of these changes is important to understand for both impact 
> analysis and driver developers and implementors.  Also, I don’t understand 
> how the addition of generic key-values to drivers would eliminate tags.  As I 
> understand them, tags are used to establish valid combinations of drivers 
> (e.g. mutual exclusion) to avoid defining an unimplementable infrastructure 
> configuration.  While I agree with the desire to eliminate them, I don’t 
> understand how such a facility would contribute to that goal.
>
> Great work putting together an initial design, and getting this important 
> conversation started.
>
> Thanks,
> -John
>
> On Jun 10, 2014, at 6:26 PM, Mike Tutkowski  
> wrote:
>
>> Hi,
>>
>> Please feel free to review the follow

Re: [PROPOSAL] Collect and provide generic key/value pairs to storage plug-ins

[Mike Tutkowski]

Thanks for the detailed response, John!

When Chris Suich (formerly from NetApp) and I talked about this several
months ago, we were thinking that this would replace the need for storage
tagging. However, as you point out, I now think it does not.

We were originally figuring that by specifically selecting a vendor (by
picking the applicable storage plug-in) that this removed the need for
storage tagging, but I see now that you might still want to further refine
within a given storage provider which primary storage CloudStack selects.

That being the case, if the user selects a specific storage plug-in, that
will be the first step in filtering and storage tagging will be the next.

Once I update the design, maybe you can take a look at it again. I can
include the new data types you listed.

Thanks!


On Fri, Jun 13, 2014 at 8:38 AM, John Burwell  wrote:

> Mike,
>
> I completely agree with the need to attach driver-specified key-value
> pairs to a storage driver (we have been batting this around for nearly a
> year).  However, I think this facility should be generalized to support all
> drivers (e.g. network, compute, etc) where storage maybe the first layer to
> implement it.  For example, we have a raft of places where vendor specific
> data/concepts leak into the object model and schema.  For example, in
> networking, we have vendor specific API calls. One of the main value
> propositions of a cloud platform is to provide a unified abstraction for
> underlying infrastructure components.  Furthermore, requiring vendor
> implements to add API calls further increases the complexity and effort
> required to support CloudStack — discouraging their participation.
>
> In terms of the design of itself, I don’t feel that the Map String> getPropertiesAndTypes is a rich enough semantic for such an
> extension facility.  First, it does not provide a mechanism to validate
> that the form and content of the values being set.  Second, it is
> incredible lose contract that does not exploit the Java compiler to ensure
> drivers are providing well-formed metadata to the CloudStack orchestration
> components.  Finally, it does not support notions such as hinting which
> fields are required (very useful for creating useful UIs) and constrained
> values (rendered as drop downs).  My thoughts are as follows:
>
>
>- Define an ExtenstionPropertyType enumeration with the following
>types permitted:
>   - STRING
>   - DATE
>   - NUMBER
>   - BOOLEAN
>   - The following methods as well:
>  - Class getType(): The underlying Java type used to represent
>  the type
>  - Object convertFromString(String aValue): Converts a value from
>  a string to the underlying Java type
>  - String convertToString(Object aValue): Converts a value to a
>  string from the underlying Java type
>   - Define a ExtensionPropertyDefinition interface that describes the
>following attributes about a extension property:
>   - String getName(): The name of the property conforming to JavaBean
>   standards (while we don’t support reflection, it’s a well known 
> standard by
>   Java developers …)
>   - ExtensionPropertyType getType():  The type of the property
>   - boolean isRequired(): Whether or not a value for the property is
>   required
>   - boolean isCollected(): Whether or not the property’s value is a
>   single value or collected (default to a list)
>   - Set getConstraints: A list of the values that constrain
>   the values of this extension property.
>   - List validate(Object aValue): A callback to validate a
>   value for this property definition — returning a list of strings 
> describing
>   the validation errors.  An empty list implies a valid value.  An 
> additional
>   nicety for this interface would be to define a type which the definition
>   would expect of the underlying value to be at runtime.
>- Introduce an Extendable interface that indicates that a type can
>have its data extended with a set of key-value pairs.  I would define the
>following methods:
>   - Set
>   getExtenstionPropertyDefinitions(): The set of extension property
>   descriptions
>   - Object getExtensionPropertyValue(String aPropertyName): Get a
>   value for an extension property
>   - List validateExtensionProperties(): Validates the values
>   of the extension properties
>
>
> There are likely pieces I missed and refinements to improve this design —
> I jotted it down off the top of my head.  However, we have to ensue that
> the semantic is rich enough to render friendly UIs, as well as, protect
> against GIGO (garbage in, garbage out).
>
> Finally, I would like to see the design expanded to explain when and where
> this information will pass into drivers for use.  As the design stands now,
> it does not explain how the data would actually be used during operation.
>  The scope of