Min, I apologize for the confusions. Files greater than 5 GB must uploaded using multipart and no part can exceed 5 GB in size. The original S3 implementation in 4.1.0 didn't support multi-part uploads for templates. Therefore, I didn't implement it in the s3xen plugin. Now that we support it for all other operations, the s3xen plugin needs to be retrofitted for multipart.
Thanks, -John On Oct 31, 2013, at 1:50 PM, Min Chen <min.c...@citrix.com> wrote: > Hi John, > I have a related question on this issue for you regarding RiakCS. In 4.2.0 > release, backup snapshot to S3 is done through your s3xen plugin, where S3 > single part upload API is used. If customer is using RiakCS as their S3 > object store, can they still backup snapshot larger than 5GB? Based on your > comments, it seems that they should be able to? > > Thanks > -min > > From: John Burwell <jburw...@basho.com<mailto:jburw...@basho.com>> > Reply-To: <dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>> > Date: Thursday, October 31, 2013 10:30 AM > To: Darren Shepherd > <darren.s.sheph...@gmail.com<mailto:darren.s.sheph...@gmail.com>> > Cc: "dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>" > <dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>> > Subject: Re: ACS + Riak-CS, works? > > Darren, > > Since S3/Riak CS can receive the contents of a file out of order and > re-assemble upon completion of transfer of all parts, the underlying driver > uses the TransferManager which automatically breaks files up into multi-part > uploads to increase parallelism to decrease the time required to upload to > the object store. Therefore, multi-part uploads are not employed solely to > upload files larger than 5 GB. > > Have you tried to upload the file to Riak CS using s3cmd outside of > CloudStack? > > Thanks, > -John > > On Oct 31, 2013, at 1:23 PM, Darren Shepherd > <darren.s.sheph...@gmail.com<mailto:darren.s.sheph...@gmail.com>> wrote: > > My template is less than 5gb and it still uses multipart. I actually don't > think this is a CloudStack issue but more of an issue with Riak-CS. I was > just wondering if anybody has specifically ran the setup or knows more about > Riak-CS and can tell me whats up. I've been trying off and on for the last > week to try to get this working with 4.2 but it just keeps failing. I think > I'll just try ceph rados gateway if I can't get this working. > > Darren > > > On Wed, Oct 30, 2013 at 11:17 PM, Sanjeev Neelarapu > <sanjeev.neelar...@citrix.com<mailto:sanjeev.neelar...@citrix.com>> wrote: > Hi, > > Cloudstack with Riak-CS works fine. If we register template more than 5GB we > use multipart upload to push the files to Riak. But this does not work for > snapshots and volumes. This will be fixed in CS4.2.1 . But for template > registration it should work fine. > > Thanks, > Sanjeev > > -----Original Message----- > From: Darren Shepherd > [mailto:darren.s.sheph...@gmail.com<mailto:darren.s.sheph...@gmail.com>] > Sent: Thursday, October 31, 2013 11:18 AM > To: dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org> > Subject: ACS + Riak-CS, works? > > Has anybody gotten CloudStack and Riak-CS to work together. No matter what I > try when it goes to download and save the template, after its posted a series > of multi part uploads it ends up sending a 403. The access log looks like > > 127.0.0.1 - - [31/Oct/2013:05:41:11 +0000] "PUT > /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=14 > HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic > OpenJDK_64-Bit_Server_VM/20.0-b12" > 127.0.0.1 - - [31/Oct/2013:05:41:28 +0000] "PUT > /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=15 > HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic > OpenJDK_64-Bit_Server_VM/20.0-b12" > 127.0.0.1 - - [31/Oct/2013:05:41:42 +0000] "PUT > /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=16 > HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic > OpenJDK_64-Bit_Server_VM/20.0-b12" > 127.0.0.1 - - [31/Oct/2013:05:41:51 +0000] "PUT > /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=17 > HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic > OpenJDK_64-Bit_Server_VM/20.0-b12" > 127.0.0.1 - - [31/Oct/2013:05:41:51 +0000] "PUT > /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=18 > HTTP/1.1" 403 311 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic > OpenJDK_64-Bit_Server_VM/20.0-b12" > > And then in the error log I always get > > 2013-10-31 05:41:51.867 [error] > <0.5117.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user record > for s3 failed. Reason: disconnected > > I'm able to interact with Riak-CS with s3cmd just fine but its not using > multipart. I think I'm going to try to download a new s3cmd that does > multipart and see if even that will work. > > Darren > >
signature.asc
Description: Message signed with OpenPGP using GPGMail
