Re: [openstack-dev] [Glance][Glare] External locations design

[Fox, Kevin M]

Has the use case been considered where the compute nodes have no nat to the 
internet?

The glare api server might have to pass the data through to the requesting 
client in that case or download/cache an internal copy.

Thanks,
Kevin

From: Mikhail Fedosin [mfedo...@mirantis.com]
Sent: Monday, August 08, 2016 4:08 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Glance][Glare] External locations design

Thanks for you comments Stuart!

I also have some thoughts on this matter: first of all, I never liked how 
locations are implemented in glance v2, and not just because of inconsistencies 
in multiple locations per one image, but for the reason that there are no 
differences between internal and external locations.
So, I think we're on right track with glare's implementation. But I suppose we 
can improve it even more:  in my understanding, the service generally does not 
work with external data. I will try to explain with an example:

Let's assume that we have a link to external ubuntu image ( 
http://releases.ubuntu.com/16.04.1/ubuntu-16.04.1-server-amd64.iso ). When user 
adds this link to his image artifact, Glare doesn't validate it, it just 
creates a blob instance in DB (optionally user may specify a checksum).
When user requests 'show' on his image, the blob url will look like this: 
{"url": "http://releases.ubuntu.com/16.04.1/ubuntu-16.04.1-server-amd64.iso"}, 
and if user wants to download it, the image won't be proxied with glare - it 
will be a direct download from ubuntu.com<http://ubuntu.com> from the client.

In case of internal locations their urls are {"url": 
"/artifacts/images//file"} and these downloads are done with Glare as 
a proxy between the client and internal cloud storage, like Swift.

For compatibility, internal urls should work with external blobs as well, but 
there will be a redirect to the external url: 
"/artifacts/images//file" -> 
"http://releases.ubuntu.com/16.04.1/ubuntu-16.04.1-server-amd64.iso;.

We can discuss it on the artifact meeting at 1730 UTC on Mondays at 
#openstack-meeting-alt

Best,
Mike

On Tue, Aug 2, 2016 at 1:16 PM, 
<stuart.mcla...@hp.com<mailto:stuart.mcla...@hp.com>> wrote:
Hi Kairat,

I think it's great to try and tease through the various issues here.
I added some comments to the etherpad.

-Stuart

Hello all,

I would like to start to describe some design decisions we made in Glare
code (https://review.openstack.org/#/q/topic:bp/glare-api+status:open).  If
you are not familiar with Glare I suggest you to read the following spec:

https://github.com/openstack/glance-specs/blob/master/specs/newton/approved/glance/glare-api.rst

I hope it will help other folks to understand Glare approach and provide
some constructive feedback for Glare. I think that we can also use Glare
solution for Glance in near future to address some drawbacks we have in
Glance.

Glare locations

Glance and Glare have possibility to set some external url as
image(artifact) location. This feature is quite useful for users who would
like to refer to some external image or artifact (for example, Fedora image
on official Fedora site) and not to store this image or artifact in the
cloud.

External locations in Glance have several specialities:

  1.

  It is possible to setup multiple locations for an image. Glance uses
  special location strategy to define which location to use. This strategy
  defined in glance codebase and can be configured in glance conf.
  2.

  Glance doesn?t differ image locations specified by url and image
  locations uploaded to Glance backend. Glance has some restrictions about
  which urls to use for locations (see Glance docs for more info).


Glare external locations designed in different way to address some
drawbacks we have in Glance. So the approach is the following:

  1.

  Glare doesn?t support multiple locations, you can specify dict of blobs
  in artifact type and add url for each blob in dict. User must define a
  name(f.e. region name or priority) for blob in dict and this name can be
  used to retrieve this blob from artifact. So decision about which location
  to use will be outside of Glare.
  2.

  Glare adds a special flag to database for external locations. So they
  will be treated differently in Glare when delete artifact. If blob value is
  external url then we don?t need to pass this url to backend and just delete
  the record in DB. For now, Glare allows only http(s) locations set but it
  may be extended in future but the idea still the same: external location
  are just records in DB.
  3.

  Glare saves blob size and checksum when specifying external url. When
  user specified url Glare downloads the blob by url, calculates its size and
  checksum. Of course, it leads to some performance degradation but we can
  ensure that the external blob is immutable. We made t

Re: [openstack-dev] [Glance][Glare] External locations design

[Mikhail Fedosin]

Thanks for you comments Stuart!

I also have some thoughts on this matter: first of all, I never liked how
locations are implemented in glance v2, and not just because of
inconsistencies in multiple locations per one image, but for the reason
that there are no differences between internal and external locations.
So, I think we're on right track with glare's implementation. But I suppose
we can improve it even more:  in my understanding, the service generally
does not work with external data. I will try to explain with an example:

Let's assume that we have a link to external ubuntu image (
http://releases.ubuntu.com/16.04.1/ubuntu-16.04.1-server-amd64.iso ). When
user adds this link to his image artifact, Glare doesn't validate it, it
just creates a blob instance in DB (optionally user may specify a checksum).
When user requests 'show' on his image, the blob url will look like this:
{"url": "http://releases.ubuntu.com/16.04.1/ubuntu-16.04.1-server-amd64.iso"},
and if user wants to download it, the image won't be proxied with glare -
it will be a direct download from ubuntu.com from the client.

In case of internal locations their urls are {"url":
"/artifacts/images//file"} and these downloads are done with
Glare as a proxy between the client and internal cloud storage, like Swift.

For compatibility, internal urls should work with external blobs as well,
but there will be a redirect to the external url:
"/artifacts/images//file" -> "
http://releases.ubuntu.com/16.04.1/ubuntu-16.04.1-server-amd64.iso;.

We can discuss it on the artifact meeting at 1730 UTC on Mondays at
#openstack-meeting-alt

Best,
Mike

On Tue, Aug 2, 2016 at 1:16 PM,  wrote:

> Hi Kairat,
>
> I think it's great to try and tease through the various issues here.
> I added some comments to the etherpad.
>
> -Stuart
>
> Hello all,
>>
>> I would like to start to describe some design decisions we made in Glare
>> code (https://review.openstack.org/#/q/topic:bp/glare-api+status:open).
>> If
>> you are not familiar with Glare I suggest you to read the following spec:
>>
>> https://github.com/openstack/glance-specs/blob/master/specs/
>> newton/approved/glance/glare-api.rst
>>
>> I hope it will help other folks to understand Glare approach and provide
>> some constructive feedback for Glare. I think that we can also use Glare
>> solution for Glance in near future to address some drawbacks we have in
>> Glance.
>>
>> Glare locations
>>
>> Glance and Glare have possibility to set some external url as
>> image(artifact) location. This feature is quite useful for users who would
>> like to refer to some external image or artifact (for example, Fedora
>> image
>> on official Fedora site) and not to store this image or artifact in the
>> cloud.
>>
>> External locations in Glance have several specialities:
>>
>>   1.
>>
>>   It is possible to setup multiple locations for an image. Glance uses
>>   special location strategy to define which location to use. This strategy
>>   defined in glance codebase and can be configured in glance conf.
>>   2.
>>
>>   Glance doesn?t differ image locations specified by url and image
>>   locations uploaded to Glance backend. Glance has some restrictions about
>>   which urls to use for locations (see Glance docs for more info).
>>
>>
>> Glare external locations designed in different way to address some
>> drawbacks we have in Glance. So the approach is the following:
>>
>>   1.
>>
>>   Glare doesn?t support multiple locations, you can specify dict of blobs
>>   in artifact type and add url for each blob in dict. User must define a
>>   name(f.e. region name or priority) for blob in dict and this name can be
>>   used to retrieve this blob from artifact. So decision about which
>> location
>>   to use will be outside of Glare.
>>   2.
>>
>>   Glare adds a special flag to database for external locations. So they
>>   will be treated differently in Glare when delete artifact. If blob
>> value is
>>   external url then we don?t need to pass this url to backend and just
>> delete
>>   the record in DB. For now, Glare allows only http(s) locations set but
>> it
>>   may be extended in future but the idea still the same: external location
>>   are just records in DB.
>>   3.
>>
>>   Glare saves blob size and checksum when specifying external url. When
>>   user specified url Glare downloads the blob by url, calculates its size
>> and
>>   checksum. Of course, it leads to some performance degradation but we can
>>   ensure that the external blob is immutable. We made this because
>> security
>>   seems more important for Glare than performance. Also there are plans to
>>   extend this approach to support subscriptions for external locations so
>> we
>>   can increase secureness of that operation.
>>
>>
>> I think that some of the features above can be implemented in Glance. For
>> example, we can treat our locations as only read-only links if external
>> flag will be implemented.  It will allow us to 

Re: [openstack-dev] [Glance][Glare] External locations design

[stuart . mclaren]

Hi Kairat,

I think it's great to try and tease through the various issues here.
I added some comments to the etherpad.

-Stuart


Hello all,

I would like to start to describe some design decisions we made in Glare
code (https://review.openstack.org/#/q/topic:bp/glare-api+status:open).  If
you are not familiar with Glare I suggest you to read the following spec:

https://github.com/openstack/glance-specs/blob/master/specs/newton/approved/glance/glare-api.rst

I hope it will help other folks to understand Glare approach and provide
some constructive feedback for Glare. I think that we can also use Glare
solution for Glance in near future to address some drawbacks we have in
Glance.

Glare locations

Glance and Glare have possibility to set some external url as
image(artifact) location. This feature is quite useful for users who would
like to refer to some external image or artifact (for example, Fedora image
on official Fedora site) and not to store this image or artifact in the
cloud.

External locations in Glance have several specialities:

  1.

  It is possible to setup multiple locations for an image. Glance uses
  special location strategy to define which location to use. This strategy
  defined in glance codebase and can be configured in glance conf.
  2.

  Glance doesn?t differ image locations specified by url and image
  locations uploaded to Glance backend. Glance has some restrictions about
  which urls to use for locations (see Glance docs for more info).


Glare external locations designed in different way to address some
drawbacks we have in Glance. So the approach is the following:

  1.

  Glare doesn?t support multiple locations, you can specify dict of blobs
  in artifact type and add url for each blob in dict. User must define a
  name(f.e. region name or priority) for blob in dict and this name can be
  used to retrieve this blob from artifact. So decision about which location
  to use will be outside of Glare.
  2.

  Glare adds a special flag to database for external locations. So they
  will be treated differently in Glare when delete artifact. If blob value is
  external url then we don?t need to pass this url to backend and just delete
  the record in DB. For now, Glare allows only http(s) locations set but it
  may be extended in future but the idea still the same: external location
  are just records in DB.
  3.

  Glare saves blob size and checksum when specifying external url. When
  user specified url Glare downloads the blob by url, calculates its size and
  checksum. Of course, it leads to some performance degradation but we can
  ensure that the external blob is immutable. We made this because security
  seems more important for Glare than performance. Also there are plans to
  extend this approach to support subscriptions for external locations so we
  can increase secureness of that operation.


I think that some of the features above can be implemented in Glance. For
example, we can treat our locations as only read-only links if external
flag will be implemented.  It will allow us to ensure that only blobs
uploaded through the Glance will be managed.

Additionally, if we will calculate checksum and size for external urls, we
can ensure that  all multiple locations refers to the same blob. So
management of multiple locations(deletion/creation) can be more secure.
Also we can ensure that the external url blob was not changed.

I understand that we need a spec for that but I would like to discuss this
at high level first. Here is etherpad for discussion:
https://etherpad.openstack.org/p/glare-locations


Best regards,
Kairat Kushaev
-- next part --
An HTML attachment was scrubbed...
URL: 


--

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


End of OpenStack-dev Digest, Vol 52, Issue 2




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Glance][Glare] External locations design

[Kairat Kushaev]

Hello all,

I would like to start to describe some design decisions we made in Glare
code (https://review.openstack.org/#/q/topic:bp/glare-api+status:open).  If
you are not familiar with Glare I suggest you to read the following spec:

https://github.com/openstack/glance-specs/blob/master/specs/newton/approved/glance/glare-api.rst

I hope it will help other folks to understand Glare approach and provide
some constructive feedback for Glare. I think that we can also use Glare
solution for Glance in near future to address some drawbacks we have in
Glance.

Glare locations

Glance and Glare have possibility to set some external url as
image(artifact) location. This feature is quite useful for users who would
like to refer to some external image or artifact (for example, Fedora image
on official Fedora site) and not to store this image or artifact in the
cloud.

External locations in Glance have several specialities:

   1.

   It is possible to setup multiple locations for an image. Glance uses
   special location strategy to define which location to use. This strategy
   defined in glance codebase and can be configured in glance conf.
   2.

   Glance doesn’t differ image locations specified by url and image
   locations uploaded to Glance backend. Glance has some restrictions about
   which urls to use for locations (see Glance docs for more info).


Glare external locations designed in different way to address some
drawbacks we have in Glance. So the approach is the following:

   1.

   Glare doesn’t support multiple locations, you can specify dict of blobs
   in artifact type and add url for each blob in dict. User must define a
   name(f.e. region name or priority) for blob in dict and this name can be
   used to retrieve this blob from artifact. So decision about which location
   to use will be outside of Glare.
   2.

   Glare adds a special flag to database for external locations. So they
   will be treated differently in Glare when delete artifact. If blob value is
   external url then we don’t need to pass this url to backend and just delete
   the record in DB. For now, Glare allows only http(s) locations set but it
   may be extended in future but the idea still the same: external location
   are just records in DB.
   3.

   Glare saves blob size and checksum when specifying external url. When
   user specified url Glare downloads the blob by url, calculates its size and
   checksum. Of course, it leads to some performance degradation but we can
   ensure that the external blob is immutable. We made this because security
   seems more important for Glare than performance. Also there are plans to
   extend this approach to support subscriptions for external locations so we
   can increase secureness of that operation.


I think that some of the features above can be implemented in Glance. For
example, we can treat our locations as only read-only links if external
flag will be implemented.  It will allow us to ensure that only blobs
uploaded through the Glance will be managed.

Additionally, if we will calculate checksum and size for external urls, we
can ensure that  all multiple locations refers to the same blob. So
management of multiple locations(deletion/creation) can be more secure.
Also we can ensure that the external url blob was not changed.

I understand that we need a spec for that but I would like to discuss this
at high level first. Here is etherpad for discussion:
https://etherpad.openstack.org/p/glare-locations


Best regards,
Kairat Kushaev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev