Re: Streaming Uploads Discussion

2008-04-05 Thread Ivan Sagalaev
Ivan Sagalaev wrote: > Nobody stops a developer from doing both things in parallel: storing on > disk and streaming to S3. > they will gain heavily from not doing writes and > reads of the whole file on local disk. Looks like I'm contradicting myself a bit here :-). But it's not the point act

Re: Streaming Uploads Discussion

2008-04-05 Thread Ivan Sagalaev
Malcolm Tredinnick wrote: > Then those people deserve to be beaten heavily about the head and > shoulders. S3 is NOT a reliable upload endpoint. They (Amazon) say > there'll be approximately a 1% failure rate for attempted uploads. As 37 > signals have noted in the past (they use it for their whit

Re: Streaming Uploads Discussion

2008-04-05 Thread Ivan Sagalaev
Mike Axiak wrote: > I didn't want to use the extra setting either, but I finally caved in > after working with it (and discussing with Ivan). I will certainly > explore any other possibilities. My original reason for settings was that a single upload handler can't possibly know the semantics of

Re: Streaming Uploads Discussion

2008-04-04 Thread Malcolm Tredinnick
On Fri, 2008-04-04 at 13:23 -0700, Mike Axiak wrote: [...] > Let's look at the S3 uploading process. There are two ways we can > handle the upload and send to S3: > >1. Stream the data directly to S3. >2. Stream the data to disk, then send to S3. > > I think a lot of people might opt to

Re: Streaming Uploads Discussion

2008-04-04 Thread Mike Axiak
On Apr 4, 3:29 pm, "Marty Alchin" <[EMAIL PROTECTED]> wrote: > I admit I haven't been following this terribly closely, but now that > both #5361 and #2070 are nearing completion, I'm trying to get a good > handle on all of this in case there are any interactions between the > two that I can help w

Re: Streaming Uploads Discussion

2008-04-04 Thread Marty Alchin
On Fri, Apr 4, 2008 at 3:43 PM, Jacob Kaplan-Moss <[EMAIL PROTECTED]> wrote: > Yeah, one thing we'll need to figure out PDQ is what's appropriate for > an upload handler, and what's appropriate for a storage backend. > Hopefully the two of you can work out the breakdown. I'll read over the pat

Re: Streaming Uploads Discussion

2008-04-04 Thread Jacob Kaplan-Moss
On Fri, Apr 4, 2008 at 2:29 PM, Marty Alchin <[EMAIL PROTECTED]> wrote: > Maybe I'm missing something obvious, but why would there ever be an > S3UploadHandler? Shouldn't that be handled by a file storage backend? Yeah, one thing we'll need to figure out PDQ is what's appropriate for an upload

Re: Streaming Uploads Discussion

2008-04-04 Thread Marty Alchin
On Fri, Apr 4, 2008 at 2:04 PM, Mike Axiak <[EMAIL PROTECTED]> wrote: > However, the instant they pass the file to some remote location > (think: S3UploadHandler) or alter the content (think: > GZipUploadHandler) they will need their own way of defining what is > content and how it should be "

Re: Streaming Uploads Discussion

2008-04-04 Thread Mike Axiak
On Apr 4, 2:46 pm, "Jacob Kaplan-Moss" <[EMAIL PROTECTED]> wrote: > Hrm, good point. I'll chew a bit more, but I can't think of a good way > to avoid the extra setting (as much as I dislike creeping settings). I didn't want to use the extra setting either, but I finally caved in after working wit

Re: Streaming Uploads Discussion

2008-04-04 Thread Jacob Kaplan-Moss
On Fri, Apr 4, 2008 at 1:04 PM, Mike Axiak <[EMAIL PROTECTED]> wrote: > Composition gets a little tricky. I realized this when I wrote the > stuff that handles the composition. > Right now we have (InMemoryUploadHandler, TemporaryFileUploadHandler) > in a sequence, and now the memory handler d

Re: Streaming Uploads Discussion

2008-04-04 Thread Mike Axiak
Thanks for the review! On Apr 4, 12:28 pm, "Jacob Kaplan-Moss" <[EMAIL PROTECTED]> wrote: > >    TextFileForm(data={'description': u'Assistance'}, files={'file': > >  {'filename': 'test1.txt', 'content': 'hello world'}}) > > What would an equivalent line look like under the new system? That is, w

Re: Streaming Uploads Discussion

2008-04-04 Thread Jacob Kaplan-Moss
On Mon, Mar 24, 2008 at 1:02 PM, Mike Axiak <[EMAIL PROTECTED]> wrote: > Now that we actually have a working patch [1], there are a few details > I'd like to raise here. Woo! Thanks for your hard work. My thoughts on your questions follow inline: > Supporting dictionaries in form code >

Re: Streaming Uploads Discussion

2008-03-26 Thread Malcolm Tredinnick
On Tue, 2008-03-25 at 22:59 -0700, Mike Axiak wrote: > On Mar 25, 3:38 pm, Mike Axiak <[EMAIL PROTECTED]> wrote: > > Yes. :-) > Doh. I didn't realize you actually wanted me to do more work. I didn't either. I thought I may have just been missing something obvious. But now that you've written the

Re: Streaming Uploads Discussion

2008-03-25 Thread Mike Axiak
On Mar 25, 3:38 pm, Mike Axiak <[EMAIL PROTECTED]> wrote: > Yes. :-) Doh. I didn't realize you actually wanted me to do more work. The newly updated doc can be found at the same URL: http://orodruin.axiak.net/upload_handling.html Be sure to refresh as the browser caching is high on that page. -

Re: Streaming Uploads Discussion

2008-03-25 Thread Mike Axiak
Hey, On Mar 25, 3:32 pm, Malcolm Tredinnick <[EMAIL PROTECTED]> wrote: > Does this Just Work(tm) out of the box? Yes. :-) -Mike --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to t

Re: Streaming Uploads Discussion

2008-03-25 Thread Malcolm Tredinnick
On Mon, 2008-03-24 at 11:02 -0700, Mike Axiak wrote: > Now that we actually have a working patch [1], there are a few details > I'd like to raise here. I haven't had time to sit down and devote to reading through the whole patch, but I have a possibly very easy question that I can't answer from

Re: Streaming Uploads Discussion

2008-03-25 Thread Mike Axiak
Hey, On Mar 25, 6:04 am, Ivan Sagalaev <[EMAIL PROTECTED]> wrote: > I don't like #1 because there's no point to keep deprecated code in > Django where we can fix it. And #3 is indeed broken because there is > much code in the wild that uses request.FILES directly. It's a public > API after all. I

Re: Streaming Uploads Discussion

2008-03-25 Thread Ivan Sagalaev
Mike Axiak wrote: > 1. Leave forms code alone, causing (a) DeprecationWarnings and (b) > the files to be read into memory when saving. > 2. Modify the form code to access the attributes of the > UploadedFile, but on AttributeError use the old dict-style interface > and emit a DeprecationWarning.

Re: Streaming Uploads Discussion

2008-03-24 Thread Mike Axiak
Now that we actually have a working patch [1], there are a few details I'd like to raise here. Major Issues === Supporting dictionaries in form code While we can have file objects that support the dictionary lookup to make those backwards compatible, ther