Yes, I expect we'll see the same issue with S3, as well. Punith - Is this something you might have time to investigate? Perhaps Edison can point us in the right direction here.
On Mon, Aug 25, 2014 at 5:17 AM, Francois Gaudreault < fgaudrea...@cloudops.com> wrote: > Punith, > > I highly anticipate the same issue with S3... it shares a lot of code with > swift. > > My focus would be swift, but we should fix for both :) > > FG > On Aug 25, 2014 6:33 AM, "Punith S" <punit...@cloudbyte.com> wrote: > > > thanks for opening this thread mike, > > > > since i only use nfs as my secondary storage provider, i didn't see this > > issue till date. > > > > is this issue occurring even using a S3 secondary storage with staging > nfs > > store ? > > > > if so like edison pointed we need to fetch the virtual size from the nfs > > store instead of S3 in the deploymentmanager. > > > > thanks > > > > > > On Sat, Aug 23, 2014 at 3:45 AM, Mike Tutkowski < > > mike.tutkow...@solidfire.com> wrote: > > > > > Hey Edison, > > > > > > Do you know how difficult/easy of a fix this is, who might be available > > to > > > put this fix in, and for what release (hopefully 4.4.1) this fix could > > find > > > its way in? > > > > > > Thanks! > > > Mike > > > > > > > > > On Fri, Aug 22, 2014 at 3:37 PM, Francois Gaudreault < > > > fgaudrea...@cloudops.com> wrote: > > > > > > > Min, > > > > > > > > Ok, but this is not the behavior I see. Even without requesting a VM > > > > create, the template is pushed to the staging NFS at least once. Is > it > > > > downloaded there or pushed after download, that I am not sure. I was > > > > assuming the swift upload bash script was executed after the template > > is > > > on > > > > the staging. > > > > > > > > Anyway... the focus is on the virt size, and you all know the code > > better > > > > than I do :) > > > > > > > > FG > > > > On Aug 22, 2014 5:28 PM, "Min Chen" <min.c...@citrix.com> wrote: > > > > > > > >> No. For S3/Swift, register template will directly upload templates > to > > S3 > > > >> without going through staging NFS. It will only be copied to staging > > NFS > > > >> when we first use that template to provision a VM. > > > >> > > > >> Thanks > > > >> -min > > > >> > > > >> On 8/22/14 2:25 PM, "Francois Gaudreault" <fgaudrea...@cloudops.com > > > > > >> wrote: > > > >> > > > >> >Edison, > > > >> > > > > >> >Isnt the templates downloaded to the Staging NFS first? > > > >> > > > > >> >FG > > > >> >On Aug 22, 2014 5:20 PM, "Edison Su" <edison...@citrix.com> wrote: > > > >> > > > > >> >> I know the reason why the size of template doesn¹t have correct > > > virtual > > > >> >> size if it¹s registered in S3/Swift: > > > >> >> In case of s3/swift, the template is directly stored into > s3/swift > > > >> >>through > > > >> >> swift/s3 api, there is no place for cloudstack to look into > > template, > > > >> to > > > >> >> find out the virtual size during template registration. > > > >> >> While, if secondary storage is NFS, the template is first stored > on > > > >> >> NFS(which has file system), cloudstack can unzip the template(if > > > it¹s a > > > >> >> zipped file), and read virtual size from the file, then report > back > > > to > > > >> >>mgt > > > >> >> server. > > > >> >> In order to fix it, we can add some code as: all the templates > > > >> >>registered > > > >> >> on Swift/S3, need to be downloaded to a NFS intermediate storage > > > before > > > >> >>it > > > >> >> can be consumed by primary storage. After the download finished, > > then > > > >> we > > > >> >> check virtual size of template, and report back to mgt > > server/update > > > DB > > > >> >>etc. > > > >> >> > > > >> >> From: Mike Tutkowski [mailto:mike.tutkow...@solidfire.com] > > > >> >> Sent: Friday, August 22, 2014 1:38 PM > > > >> >> To: dev@cloudstack.apache.org > > > >> >> Cc: Edison Su > > > >> >> Subject: S3/Swift Problem around Virtual Size > > > >> >> > > > >> >> Hi, > > > >> >> > > > >> >> This was brought up in a different e-mail thread, but I wanted to > > > make > > > >> >>it > > > >> >> more clear that it's related to CloudStack's download code around > > > >> >>S3/Swift, > > > >> >> so I'm opening up a new thread. > > > >> >> > > > >> >> Francois (from CloudOps) noticed today that when he downloaded a > > > >> >>template > > > >> >> (VHD format) to Swift (but it looks like the same applies for S3) > > > that > > > >> >>the > > > >> >> physical and virtual sizes are set to be the same. > > > >> >> > > > >> >> This appears to have the following consequence: > > > >> >> > > > >> >> You can download a template with a physical size of, say, 3 GB > and > > a > > > >> >>root > > > >> >> disk that's supposed to be, say, 20 GB. Instead of the virtual > size > > > >> >>showing > > > >> >> as 20 GB, it shows as 3 GB. > > > >> >> > > > >> >> This is not an issue with NFS. In that situation, the two sizes > are > > > >> >> correctly accounted for. > > > >> >> > > > >> >> What later can happen is the template is downloaded from Swift > and > > > >> takes > > > >> >> up an unexpected amount of space on the XenServer storage > > repository > > > >> >>(SR). > > > >> >> > > > >> >> If there is enough space on the SR, this isn't too big of a deal. > > > >> >>However, > > > >> >> for so-called managed storage plug-ins (examples are SolidFire > and > > > >> >> CloudByte), this will lead to them dynamically creating a SAN > > volume > > > of > > > >> >>the > > > >> >> wrong size. > > > >> >> > > > >> >> Francois opened up the following ticket: > > > >> >> > > > >> >> https://issues.apache.org/jira/browse/CLOUDSTACK-7406 > > > >> >> > > > >> >> Thanks! > > > >> >> > > > >> >> -- > > > >> >> Mike Tutkowski > > > >> >> Senior CloudStack Developer, SolidFire Inc. > > > >> >> e: mike.tutkow...@solidfire.com<mailto: > > mike.tutkow...@solidfire.com> > > > >> >> o: 303.746.7302 > > > >> >> Advancing the way the world uses the cloud< > > > >> >> http://solidfire.com/solution/overview/?video=play> > > > >> >> > > > >> > > > >> > > > > > > > > > -- > > > *Mike Tutkowski* > > > *Senior CloudStack Developer, SolidFire Inc.* > > > e: mike.tutkow...@solidfire.com > > > o: 303.746.7302 > > > Advancing the way the world uses the cloud > > > <http://solidfire.com/solution/overview/?video=play>*™* > > > > > > > > > > > -- > > regards, > > > > punith s > > cloudbyte.com > > > -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloud <http://solidfire.com/solution/overview/?video=play>*™*
