Re: [OpenStack-Infra] Space for Sahara artifacts (disk images)
On 2017-05-30 12:49:37 +0200 (+0200), Luigi Toscano wrote: > On Friday, 26 May 2017 15:27:02 CEST Jeremy Stanley wrote: [...] > > We can see about extending the available free space for > > tarballs.openstack.org after we relocate it off the same server > > where we store our job logs, but we don't have a timeline for > > when that will be. I'm sorry I don't have better news on that > > front. > > It wouldn't be a problem to wait a bit. Even if you don't have a timeline, do > you know the approximate timing? Is it more 6 months, one year, or more? [...] Aside from it being something we've talked about probably wanting (putting tarballs content in AFS and being able to present a local cache to workers in each provider/region we use for our CI system), I don't think it's on anyone's radar yet from an execution perspective. We've had more recent discussions about relocating our logs.openstack.org site off that same server and into another service provider... doing that would also free up plenty of space for growing the current tarballs site and may well happen sooner (though again I have no solid timeline for that work, we need a spec for things like the 404->301 forwarding configuration on whichever end has the DNS record while waiting out the retention period). Rough guess is at least a few months given our current team throughput and priorities. > > If you need a periodic job to upload and replace pregenerated > > branch-tip snapshot images for consumption by other CI jobs, we > > should be able to work something out pretty easily (that's what > > the other projects I mentioned above have been doing). > > This is not the use case described in my original request. > Nevertheless, it could be useful for some of our scenario jobs. > But wouldn't this be constrained by the lack of space as well? Based on the numbers you gave, we could fairly confidently provide sufficient space for images built from the tips of supported branches since they would be replaced rather than accumulating new ones for each tag. Hosting image sets for every point release requires a fair amount more available space by comparison. -- Jeremy Stanley signature.asc Description: Digital signature ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] Space for Sahara artifacts (disk images)
On Friday, 26 May 2017 15:27:02 CEST Jeremy Stanley wrote:
> On 2017-04-28 14:39:51 +0200 (+0200), Luigi Toscano wrote:
> > The Sahara project has been providing pre-built images containing
> > the Hadoop/ Spark/$bigdata frameworks since the beginning of the
> > project, so that users can be immediately productive.
> >
> > The generated qcow2 images have been living so far here:
> > http://sahara-files.mirantis.com/images/upstream/
> >
> > As a team we were wondering whether we could store those images on
> > some shared and publicly accessible space on openstack.org (like
> > tarballs.openstack.org).
> >
> > I guess that the main concern could be the disk usage. Currently
> > the space used for the older releases (from kilo to newton) is
> > around ~110GB. The estimate for Ocata is ~35GB and the number is
> > going to grow. Of course we can drop old images when a certain
> > release reaches its end-of- life (unless there is a place to store
> > some archived artifacts).
> >
> > About the update frequency: the images are currenctly rebuilt with
> > with every commit in sahara-image-elements (and soon in sahara
> > with a different build method) by the tests. I don't think that we
> > would need to update the images in this stored space with every
> > commit, but at most once every month or, even better, when a new
> > release of sahara-image-elements is tagged.
> >
> > Please note that we already store some artifacts on
> > tarballs.openstack.org, even if their size is not definitely not
> > the same of those disk images.
> > https://review.openstack.org/#/c/175395/
> > https://review.openstack.org/#/c/367271/
> >
> > To summarize: would it be possible for us to use some shared
> > space, and if yes, which are the conditions?
>
> Apologies for the delay in responding. We don't currently have
> sufficient free space to store the quantity of data you're talking
> about (some projects like Ironic, Trove and Kolla do or have in the
> past uploaded guest images there, but those are far fewer and much
> smaller than what you're requesting). We can see about extending the
> available free space for tarballs.openstack.org after we relocate it
> off the same server where we store our job logs, but we don't have a
> timeline for when that will be. I'm sorry I don't have better news
> on that front.
It wouldn't be a problem to wait a bit. Even if you don't have a timeline, do
you know the approximate timing? Is it more 6 months, one year, or more?
>
> On a separate note, what degree of security support is being
> provided for those images (as far as known vulnerabilities in
> non-OpenStack software aggregated within them)? There is still some
> concern expressed within the community around producing images of
> this nature for any purpose other than use within our CI system, in
> which case long-term archiving of release versions of those images
> is unnecessary.
I'm aware and I've been following the discussion about container images on
openstack-dev@ ("do we want to be publishing binary container images"). I
understand (as I wrote there) that the decision made there could be relevant
for this case too.
On one side the security issues are definitely important. On the other side
the ready-to-use images can definitely help users to reduce the time needed to
test the system.
It would be nice to have a clear way to mark the images as "unsupported, here
be dragons"; there were some suggestions about this in the other thread, even
if my understanding from the other discussion is that this is not considered
possible or acceptable.
> If you need a periodic job to upload and replace
> pregenerated branch-tip snapshot images for consumption by other CI
> jobs, we should be able to work something out pretty easily (that's
> what the other projects I mentioned above have been doing).
This is not the use case described in my original request. Nevertheless, it
could be useful for some of our scenario jobs. But wouldn't this be
constrained by the lack of space as well?
--
Luigi
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] Space for Sahara artifacts (disk images)
On 2017-05-26 12:34:44 +0400 (+0400), Evgeny Sikachev wrote: > I found the project which is using tarballs for storing images. > https://tarballs.openstack.org/trove/images/ > > We would like to use the same space for storing sahara-images if > it is possible. [...] It sounded from the previous E-mail like the interest was in long-term publication of release images, while what you're linking there are periodically refreshed branch-tip snapshots for consumption within CI jobs (technically the Trove team has ceased using those, but Ironic and Kolla are still doing something similar to that). The use cases, storage needs and security/support concerns are vastly different. -- Jeremy Stanley ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] Space for Sahara artifacts (disk images)
On 2017-04-28 14:39:51 +0200 (+0200), Luigi Toscano wrote: > The Sahara project has been providing pre-built images containing > the Hadoop/ Spark/$bigdata frameworks since the beginning of the > project, so that users can be immediately productive. > > The generated qcow2 images have been living so far here: > http://sahara-files.mirantis.com/images/upstream/ > > As a team we were wondering whether we could store those images on > some shared and publicly accessible space on openstack.org (like > tarballs.openstack.org). > > I guess that the main concern could be the disk usage. Currently > the space used for the older releases (from kilo to newton) is > around ~110GB. The estimate for Ocata is ~35GB and the number is > going to grow. Of course we can drop old images when a certain > release reaches its end-of- life (unless there is a place to store > some archived artifacts). > > About the update frequency: the images are currenctly rebuilt with > with every commit in sahara-image-elements (and soon in sahara > with a different build method) by the tests. I don't think that we > would need to update the images in this stored space with every > commit, but at most once every month or, even better, when a new > release of sahara-image-elements is tagged. > > Please note that we already store some artifacts on > tarballs.openstack.org, even if their size is not definitely not > the same of those disk images. > https://review.openstack.org/#/c/175395/ > https://review.openstack.org/#/c/367271/ > > To summarize: would it be possible for us to use some shared > space, and if yes, which are the conditions? Apologies for the delay in responding. We don't currently have sufficient free space to store the quantity of data you're talking about (some projects like Ironic, Trove and Kolla do or have in the past uploaded guest images there, but those are far fewer and much smaller than what you're requesting). We can see about extending the available free space for tarballs.openstack.org after we relocate it off the same server where we store our job logs, but we don't have a timeline for when that will be. I'm sorry I don't have better news on that front. On a separate note, what degree of security support is being provided for those images (as far as known vulnerabilities in non-OpenStack software aggregated within them)? There is still some concern expressed within the community around producing images of this nature for any purpose other than use within our CI system, in which case long-term archiving of release versions of those images is unnecessary. If you need a periodic job to upload and replace pregenerated branch-tip snapshot images for consumption by other CI jobs, we should be able to work something out pretty easily (that's what the other projects I mentioned above have been doing). -- Jeremy Stanley ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] Space for Sahara artifacts (disk images)
Hi! I found the project which is using tarballs for storing images. https://tarballs.openstack.org/trove/images/ We would like to use the same space for storing sahara-images if it is possible. Please answer on Luigi's questions. Best Regards, Evgeny Sikachev QA Engineer Mirantis, Inc On 28 Apr 2017, 16:39 +0400, Luigi Toscano , wrote: > Hi, > > The Sahara project has been providing pre-built images containing the Hadoop/ > Spark/$bigdata frameworks since the beginning of the project, so that users > can be immediately productive. > > The generated qcow2 images have been living so far here: > http://sahara-files.mirantis.com/images/upstream/ > > As a team we were wondering whether we could store those images on some shared > and publicly accessible space on openstack.org (like tarballs.openstack.org). > > I guess that the main concern could be the disk usage. Currently the space > used for the older releases (from kilo to newton) is around ~110GB. The > estimate for Ocata is ~35GB and the number is going to grow. > Of course we can drop old images when a certain release reaches its end-of- > life (unless there is a place to store some archived artifacts). > > About the update frequency: the images are currenctly rebuilt with with every > commit in sahara-image-elements (and soon in sahara with a different build > method) by the tests. > I don't think that we would need to update the images in this stored space > with every commit, but at most once every month or, even better, when a new > release of sahara-image-elements is tagged. > > Please note that we already store some artifacts on tarballs.openstack.org, > even if their size is not definitely not the same of those disk images. > https://review.openstack.org/#/c/175395/ > https://review.openstack.org/#/c/367271/ > > To summarize: would it be possible for us to use some shared space, and if > yes, which are the conditions? > > -- > Luigi > ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Space for Sahara artifacts (disk images)
Hi, The Sahara project has been providing pre-built images containing the Hadoop/ Spark/$bigdata frameworks since the beginning of the project, so that users can be immediately productive. The generated qcow2 images have been living so far here: http://sahara-files.mirantis.com/images/upstream/ As a team we were wondering whether we could store those images on some shared and publicly accessible space on openstack.org (like tarballs.openstack.org). I guess that the main concern could be the disk usage. Currently the space used for the older releases (from kilo to newton) is around ~110GB. The estimate for Ocata is ~35GB and the number is going to grow. Of course we can drop old images when a certain release reaches its end-of- life (unless there is a place to store some archived artifacts). About the update frequency: the images are currenctly rebuilt with with every commit in sahara-image-elements (and soon in sahara with a different build method) by the tests. I don't think that we would need to update the images in this stored space with every commit, but at most once every month or, even better, when a new release of sahara-image-elements is tagged. Please note that we already store some artifacts on tarballs.openstack.org, even if their size is not definitely not the same of those disk images. https://review.openstack.org/#/c/175395/ https://review.openstack.org/#/c/367271/ To summarize: would it be possible for us to use some shared space, and if yes, which are the conditions? -- Luigi ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
