On Jun 8, 2011, at 7:09 PM, [email protected] wrote: ACK to whole serie from my. I was able to stream a 10mb file to S3 thought this without problems.
-- Michal > > * incorporates comments by David Lutterkort in response to Revision 1 (date: > patch on 07 June 2011, comments on 08 June, Subject: "Adds blob streaming > uploads using PUT - REVISION 1" ) - Patch 1 > > * removes the necessity for sinatra >= 1.2.4 (no longer calling the POST blob > route from PUT blob) > > * moves formatting nits into seperate patches (patches 2+3) > > * moves extraction of blob metadata headers, and renaming of headers per > provider, into helpers methods (patch 4) > > > marios > > > Message from original patch copied here for convenience: > ======================================================= > > > Streaming for uploading blobs to provider. Currently to create a blob we > receive the blob from the client, load it into memory and then stream it up > to the provider. i.e.: > > client ---TEMP_FILE---> deltacloud ---STREAM---> provider > > Obviously this is not very useful for large blobs (e.g. an image) - > especially when there can be any number of 'create blob' requests which the > server would have to create temp_files for. > > You can try it out using > > curl -H 'content-type: text/html' -H 'X-Deltacloud-Blobmeta-Name:mariosblob' > -H 'X-Deltacloud-Blobmeta-Version:niceone' -H > 'X-Deltacloud-Blobmeta-Author:me' -iv --upload-file > "/home/marios/Desktop/larger.txt" --user 'awskey:aws_secretkey' > http://localhost:3001/api/buckets/thebucket_name/the_new_blob_name?format=xml > > Since Net::HTTP doesn't provide a way for us to do this 'nicely' the only way > to go was with monkey patching (both to Thin::Request and Net::HTTP). > Basically we hook into the 'receive' method of the thin server so as the data > is read in from the client, it starts getting sent out to the provider. Also, > aws gem (appoxy) doesn't provide this functionality so I do the PUT directly > using (a monkey patched version of) Net::HTTP. At this point this is an ec2 > specific solution - only because its rev. 1 - we will add other providers if > this seems like a sane way to proceed. > > Also, since theres no way to do a 'put' using browser (the sinatra 'method > override' stuff is not helpful here since that code kicks in way later than > where I am starting to stream) I leave the original 'POST > /api/buckets/:bucket' rule in there as an alternative (i.e. using the html > interface and a HTTP multipart/form data upload) though this is non streaming. ------------------------------------------------------ Michal Fojtik, [email protected] Deltacloud API: http://deltacloud.org
