Thanks for your clarification, John. Yes, we have fixed in 4.2.1. -min
On 10/31/13 10:58 AM, "John Burwell" <jburw...@basho.com> wrote: >Min, > >I apologize for the confusions. Files greater than 5 GB must uploaded >using multipart and no part can exceed 5 GB in size. The original S3 >implementation in 4.1.0 didn't support multi-part uploads for templates. >Therefore, I didn't implement it in the s3xen plugin. Now that we >support it for all other operations, the s3xen plugin needs to be >retrofitted for multipart. > >Thanks, >-John > >On Oct 31, 2013, at 1:50 PM, Min Chen <min.c...@citrix.com> wrote: > >> Hi John, >> I have a related question on this issue for you regarding RiakCS. In >>4.2.0 release, backup snapshot to S3 is done through your s3xen plugin, >>where S3 single part upload API is used. If customer is using RiakCS as >>their S3 object store, can they still backup snapshot larger than 5GB? >>Based on your comments, it seems that they should be able to? >> >> Thanks >> -min >> >> From: John Burwell <jburw...@basho.com<mailto:jburw...@basho.com>> >> Reply-To: <dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>> >> Date: Thursday, October 31, 2013 10:30 AM >> To: Darren Shepherd >><darren.s.sheph...@gmail.com<mailto:darren.s.sheph...@gmail.com>> >> Cc: "dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>" >><dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>> >> Subject: Re: ACS + Riak-CS, works? >> >> Darren, >> >> Since S3/Riak CS can receive the contents of a file out of order and >>re-assemble upon completion of transfer of all parts, the underlying >>driver uses the TransferManager which automatically breaks files up into >>multi-part uploads to increase parallelism to decrease the time required >>to upload to the object store. Therefore, multi-part uploads are not >>employed solely to upload files larger than 5 GB. >> >> Have you tried to upload the file to Riak CS using s3cmd outside of >>CloudStack? >> >> Thanks, >> -John >> >> On Oct 31, 2013, at 1:23 PM, Darren Shepherd >><darren.s.sheph...@gmail.com<mailto:darren.s.sheph...@gmail.com>> wrote: >> >> My template is less than 5gb and it still uses multipart. I actually >>don't think this is a CloudStack issue but more of an issue with >>Riak-CS. I was just wondering if anybody has specifically ran the setup >>or knows more about Riak-CS and can tell me whats up. I've been trying >>off and on for the last week to try to get this working with 4.2 but it >>just keeps failing. I think I'll just try ceph rados gateway if I can't >>get this working. >> >> Darren >> >> >> On Wed, Oct 30, 2013 at 11:17 PM, Sanjeev Neelarapu >><sanjeev.neelar...@citrix.com<mailto:sanjeev.neelar...@citrix.com>> >>wrote: >> Hi, >> >> Cloudstack with Riak-CS works fine. If we register template more than >>5GB we use multipart upload to push the files to Riak. But this does not >>work for snapshots and volumes. This will be fixed in CS4.2.1 . But for >>template registration it should work fine. >> >> Thanks, >> Sanjeev >> >> -----Original Message----- >> From: Darren Shepherd >>[mailto:darren.s.sheph...@gmail.com<mailto:darren.s.sheph...@gmail.com>] >> Sent: Thursday, October 31, 2013 11:18 AM >> To: dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org> >> Subject: ACS + Riak-CS, works? >> >> Has anybody gotten CloudStack and Riak-CS to work together. No matter >>what I try when it goes to download and save the template, after its >>posted a series of multi part uploads it ends up sending a 403. The >>access log looks like >> >> 127.0.0.1 - - [31/Oct/2013:05:41:11 +0000] "PUT >> >>/buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2 >>209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw >>==?partNumber=14 >> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic >>OpenJDK_64-Bit_Server_VM/20.0-b12" >> 127.0.0.1 - - [31/Oct/2013:05:41:28 +0000] "PUT >> >>/buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2 >>209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw >>==?partNumber=15 >> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic >>OpenJDK_64-Bit_Server_VM/20.0-b12" >> 127.0.0.1 - - [31/Oct/2013:05:41:42 +0000] "PUT >> >>/buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2 >>209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw >>==?partNumber=16 >> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic >>OpenJDK_64-Bit_Server_VM/20.0-b12" >> 127.0.0.1 - - [31/Oct/2013:05:41:51 +0000] "PUT >> >>/buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2 >>209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw >>==?partNumber=17 >> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic >>OpenJDK_64-Bit_Server_VM/20.0-b12" >> 127.0.0.1 - - [31/Oct/2013:05:41:51 +0000] "PUT >> >>/buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2 >>209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw >>==?partNumber=18 >> HTTP/1.1" 403 311 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic >>OpenJDK_64-Bit_Server_VM/20.0-b12" >> >> And then in the error log I always get >> >> 2013-10-31 05:41:51.867 [error] >> <0.5117.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user >>record for s3 failed. Reason: disconnected >> >> I'm able to interact with Riak-CS with s3cmd just fine but its not >>using multipart. I think I'm going to try to download a new s3cmd that >>does multipart and see if even that will work. >> >> Darren >> >> >
