Re: file upload fails with slow network bandwith in django

[Pablo Conesa]

El martes, 16 de febrero de 2016, 21:12:32 (UTC+1), James Schneider 
escribió:
>
> How long does the file upload work before receiving the error? 
>

 There is no response from the server (fiddler returns -1, whatever this 
 means) 

>>>
>>> Well, no response from the server doesn't necessarily mean that there 
>>> were no logs. Can you check the error and access logs of the server? 
>>>
>>
>>
>
> Have you checked the server logs?
>

Yesthere are no logs!! I knowjust joined the project and we are 
using the email handlersad! 

>
>  
>
>>  
>>
>>>
>>> Does Fiddler return an immediate response of -1, or does it seem like 
>>> there is a timeout period?
>>>
>>
>> immediate response, no time out period.
>>
>
>
> Without looking at the server logs, I can only guess that Gnuicorn is 
> unhappy with the headers that are being sent, and might be closing the 
> connection. A long timeout would probably indicate that Gnuicorn is waiting 
> for more data but something in the upload process is not triggering that it 
> has reached the end of the data stream.
>

I've done some more test and seems that the -1 is normal while data it's 
been uploaded. Nothing wrong with that. So I guess we need to config the 
logging and start looking at it. 

>
>  
>
>>
>>> After a quick glance at Fiddler itself, are you using any customizations 
>>> on the rules 
>>>
>>
>> no customization. Default "Simulate Modem speeds". I must say I use 
>> fiddle to be able to reproduce the error locally. If I hit the server 
>> (production) using wifi (slower than ethernet) I got the same thing, no 
>> fiddler in the middle.
>>
>
> But it works using a wired Ethernet connection? Honestly, the speed 
> difference between Wi-Fi and Ethernet for this purpose is probably not 
> relevant to the problem. The server logs would tell you more about what is 
> happening.
>  
>

logs, logs, logsI'm on it. :-( 

>
>
>>>
> Nice advice...I'll look into this. Do you know any nice example for 
 that JS library chunk upload.

>
>
>>>
>>> Probably best to stick with your existing JS framework of choice if you 
>>> are using one. All of the major ones have some version of a chunked 
>>> uploader (JQuery and Angular), and I come across resumable.js fairly 
>>> frequently. I've never personally had a need for one, so I can't really 
>>> comment on them in any sort of detail or make a recommendation. 
>>>
>>
>> Testing resumable.jslet's see how it goes. 
>>
>> So, far there is some response, but after a while got 404 response 
>> (testing locally against DAJNGO)will it make a difference against 
>> gunicorn?
>>
>>
>> 
>>
>
> This smells like either a memory exhaustion/ceiling issue or some kind of 
> internal connection/process timeout, neither of which are likely easy to 
> adjust using the Django dev server. Gunicorn probably has 10,000 knobs to 
> adjust those timers and limits, though. 
>
> Look at the server error/access logs.
>

againlogs, logs, logsI'll setup gunicorn locally and enable logging.

For someone wanting to move to a file chunk upload, I've tried resumable.js 
and a django component for it. it was failing. Now I'm happier with this 
other: https://github.com/juliomalegria/django-chunked-upload-demo (test 
over wifi ave worked fine). But again.maybe the problem is in our setup 
and not in those tools.

Thanks, James, you've been vary helpful.

>
> -James
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/ba2e9f2d-2c6d-4a4a-9259-14a8e95d9135%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: file upload fails with slow network bandwith in django

[Pablo Conesa]

Thanks once more James for your help and time. Answers below..

El martes, 16 de febrero de 2016, 8:33:08 (UTC+1), James Schneider escribió:
>
>
>
> On Mon, Feb 15, 2016 at 10:41 PM, Pablo Conesa <p.cones...@gmail.com 
> > wrote:
>
>> Thanks James. answers in line:
>>
>> El lunes, 15 de febrero de 2016, 23:04:48 (UTC+1), James Schneider 
>> escribió:
>>>
>>>
>>>
>>> On Mon, Feb 15, 2016 at 10:20 AM, Pablo Conesa <p.cones...@gmail.com> 
>>> wrote:
>>>
>>>> Hi, I posted this in stackoverflow. Now I'm trying here: 
>>>> http://stackoverflow.com/questions/35413649/file-upload-fails-with-slow-network-bandwith-in-django
>>>>
>>>> I've got a code working fine that uploads a file to a DJANGO server.
>>>>
>>>> It works fine on a fine connection.
>>>>
>>>> Now if, using fiddler to simulate a slow connectionpage freezes 
>>>> (Chrome pop up offering to kill the page).
>>>>
>>>> I've tested in different ways and the only factor that causes the error 
>>>> is a slow upload speed. e.g. If I choose a wi-fi network it fails but over 
>>>> the ethernet it works.
>>>>
>>>> File is 100MB!.
>>>>
>>>> python: 2.7.6 
>>>>
>>>> DJANGO: 1.5.5 or 1.9 (I updated to 1.9 and is not working either)
>>>>
>>>> Fiddler show -1 in the response, and says: "No response body".
>>>>
>>>> Any ideas?
>>>>
>>>>
>>>
>>> How long does the file upload work before receiving the error? 
>>>
>>
>> There is no response from the server (fiddler returns -1, whatever this 
>> means) 
>>
>
> Well, no response from the server doesn't necessarily mean that there were 
> no logs. Can you check the error and access logs of the server? 
>

 

>
> Does Fiddler return an immediate response of -1, or does it seem like 
> there is a timeout period?
>

immediate response, no time out period.

>
> After a quick glance at Fiddler itself, are you using any customizations 
> on the rules 
>

no customization. Default "Simulate Modem speeds". I must say I use fiddle 
to be able to reproduce the error locally. If I hit the server (production) 
using wifi (slower than ethernet) I got the same thing, no fiddler in the 
middle.

>
>
>>> My guess is that Fiddler is attempting to use an HTTP chunked transfer 
>>> to send the file 'slowly' (by rate-limiting the chunks it sends rather than 
>>> traffic shaping the data stream itself), which is how modern JS libraries 
>>> normally handle large file uploads. However, this takes advantage of 
>>> features built into the HTTP 1.1 standard, and I'm not entirely sure 
>>> whether or not the Django dev server fully implements HTTP 1.1. That would 
>>> explain why you are getting a response body error (since the Django dev 
>>> server probably doesn't understand chunked requests and doesn't know to 
>>> expect more data). See https://code.djangoproject.com/ticket/25619 
>>> <https://www.google.com/url?q=https%3A%2F%2Fcode.djangoproject.com%2Fticket%2F25619=D=1=AFQjCNH4D4wtKNpxg_c-DnWeAjGPKSk1hg>
>>>  
>>> That's purely a guess though, as I'm not familiar with how Fiddler works, 
>>> and Django's internal support for HTTP 1.1 is somewhat ambiguous. 
>>>
>>
>> Don't know either details about FIddler..but what you are saying sounds 
>> reasonable and in the right direction...
>>
>>>
>>> This type of testing should be done against the production server (which 
>>> likely fully implements HTTP 1.1), not the Django dev server, since 
>>> performance of the dev server cannot be correlated with a production server 
>>> that implements threading and/or multiple worker processes.
>>>
>>
>> I've done this. Actually the bug was reported in production (apache 
>> forwarding to guinicorn)
>>
>
> Hmm, Apache should handle that fine. It's had 1.1 support forever, and I 
> don't believe it requires any special configuration. Might be related to 
> gunicorn, though, if it is responsible for handling the chunked response. 
> Found this to be similar to your issue: 
> https://github.com/benoitc/gunicorn/issues/1125, although sadly no 
> resolution. Might point you in the right direction, though.
>
>
Looks like the same problem ... 

>  
>
>>  
>>
>>>
>>> On a side note, though. If you plan to have users regularly upload large 
>>> files 

Re: file upload fails with slow network bandwith in django

[Pablo Conesa]

Thanks James. answers in line:

El lunes, 15 de febrero de 2016, 23:04:48 (UTC+1), James Schneider escribió:
>
>
>
> On Mon, Feb 15, 2016 at 10:20 AM, Pablo Conesa <p.cones...@gmail.com 
> > wrote:
>
>> Hi, I posted this in stackoverflow. Now I'm trying here: 
>> http://stackoverflow.com/questions/35413649/file-upload-fails-with-slow-network-bandwith-in-django
>>
>> I've got a code working fine that uploads a file to a DJANGO server.
>>
>> It works fine on a fine connection.
>>
>> Now if, using fiddler to simulate a slow connectionpage freezes 
>> (Chrome pop up offering to kill the page).
>>
>> I've tested in different ways and the only factor that causes the error 
>> is a slow upload speed. e.g. If I choose a wi-fi network it fails but over 
>> the ethernet it works.
>>
>> File is 100MB!.
>>
>> python: 2.7.6 
>>
>> DJANGO: 1.5.5 or 1.9 (I updated to 1.9 and is not working either)
>>
>> Fiddler show -1 in the response, and says: "No response body".
>>
>> Any ideas?
>>
>>
>
> How long does the file upload work before receiving the error? 
>

There is no response from the server (fiddler returns -1, whatever this 
means) 

>
> My guess is that Fiddler is attempting to use an HTTP chunked transfer to 
> send the file 'slowly' (by rate-limiting the chunks it sends rather than 
> traffic shaping the data stream itself), which is how modern JS libraries 
> normally handle large file uploads. However, this takes advantage of 
> features built into the HTTP 1.1 standard, and I'm not entirely sure 
> whether or not the Django dev server fully implements HTTP 1.1. That would 
> explain why you are getting a response body error (since the Django dev 
> server probably doesn't understand chunked requests and doesn't know to 
> expect more data). See https://code.djangoproject.com/ticket/25619 
> <https://www.google.com/url?q=https%3A%2F%2Fcode.djangoproject.com%2Fticket%2F25619=D=1=AFQjCNH4D4wtKNpxg_c-DnWeAjGPKSk1hg>
>  
> That's purely a guess though, as I'm not familiar with how Fiddler works, 
> and Django's internal support for HTTP 1.1 is somewhat ambiguous. 
>

Don't know either details about FIddler..but what you are saying sounds 
reasonable and in the right direction...

>
> This type of testing should be done against the production server (which 
> likely fully implements HTTP 1.1), not the Django dev server, since 
> performance of the dev server cannot be correlated with a production server 
> that implements threading and/or multiple worker processes.
>

I've done this. Actually the bug was reported in production (apache 
forwarding to guinicorn)
 

>
> On a side note, though. If you plan to have users regularly upload large 
> files (>2MB), you should seriously look in to implementing chunked uploads 
> via a JS library. With a standard upload via a regular form, the worker 
> process in your server handling the upload are dedicated to that upload 
> until it finishes. Usually this is fine for small uploads, but for larger 
> uploads that can take minutes or hours, it becomes a large problem even on 
> a site with a small user base. If 5 users are uploading a large file, and 
> you only have 5 worker processes running, then literally nobody can access 
> your site until at least one of those uploads finishes. With chunked 
> transfers, the file is broken up into pieces and sent as many small 
> requests rather than a single large one. This allows a server worker 
> process to process the small upload, then step away to serve other users, 
> and then process the next small upload. The net result on a busy server is 
> usually a slower response, but at least there is a response.
>

Nice advice...I'll look into this. Do you know any nice example for that JS 
library chunk upload.

>
> -James
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/3d400904-8fe8-460c-b9cd-e8ba6521a96a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: file upload fails with slow network bandwith in django

[Pablo Conesa]

Thanks Ezequiel, answer below.

El lunes, 15 de febrero de 2016, 21:22:23 (UTC+1), Ezequiel Bertti escribió:
>
> You are using de runserver to develop and getting this error or something 
> like gunicorn and nginx?
>

In production  we are using gunicorn, locally I don't use guinicorn, and in 
both cases it happens the same.

Unfortunately there is no message, no time out, nothing from the server.

>
> Because the nginx have some time to proccess each request.
>
> http://stackoverflow.com/questions/6816215/gunicorn-nginx-timeout-problem 
> <http://www.google.com/url?q=http%3A%2F%2Fstackoverflow.com%2Fquestions%2F6816215%2Fgunicorn-nginx-timeout-problem=D=1=AFQjCNEaoPVXq87HfcUZARiv5oAZjTM8Fw>
>
> On Mon, Feb 15, 2016 at 4:20 PM, Pablo Conesa <p.cones...@gmail.com 
> > wrote:
>
>> Hi, I posted this in stackoverflow. Now I'm trying here: 
>> http://stackoverflow.com/questions/35413649/file-upload-fails-with-slow-network-bandwith-in-django
>>
>> I've got a code working fine that uploads a file to a DJANGO server.
>>
>> It works fine on a fine connection.
>>
>> Now if, using fiddler to simulate a slow connectionpage freezes 
>> (Chrome pop up offering to kill the page).
>>
>> I've tested in different ways and the only factor that causes the error 
>> is a slow upload speed. e.g. If I choose a wi-fi network it fails but over 
>> the ethernet it works.
>>
>> File is 100MB!.
>>
>> python: 2.7.6 
>>
>> DJANGO: 1.5.5 or 1.9 (I updated to 1.9 and is not working either)
>>
>> Fiddler show -1 in the response, and says: "No response body".
>>
>> Any ideas?
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-users...@googlegroups.com .
>> To post to this group, send email to django...@googlegroups.com 
>> .
>> Visit this group at https://groups.google.com/group/django-users.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-users/3625feed-bc42-4d80-a8cb-d6a8c94a48cf%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/django-users/3625feed-bc42-4d80-a8cb-d6a8c94a48cf%40googlegroups.com?utm_medium=email_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> -- 
> Ezequiel Bertti
> E-Mail: ebe...@gmail.com 
> Cel: (21) 99188-4860
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/0307bb9e-01dd-4f52-a004-b0f7277b9354%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

file upload fails with slow network bandwith in django

[Pablo Conesa]

Hi, I posted this in stackoverflow. Now I'm trying 
here: 
http://stackoverflow.com/questions/35413649/file-upload-fails-with-slow-network-bandwith-in-django

I've got a code working fine that uploads a file to a DJANGO server.

It works fine on a fine connection.

Now if, using fiddler to simulate a slow connectionpage freezes (Chrome 
pop up offering to kill the page).

I've tested in different ways and the only factor that causes the error is 
a slow upload speed. e.g. If I choose a wi-fi network it fails but over the 
ethernet it works.

File is 100MB!.

python: 2.7.6 

DJANGO: 1.5.5 or 1.9 (I updated to 1.9 and is not working either)

Fiddler show -1 in the response, and says: "No response body".

Any ideas?

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/3625feed-bc42-4d80-a8cb-d6a8c94a48cf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.