Re: Django/uWSGI/nginx under load

[Andy Robinson]

I'm unsure what you mean - do you mean "are we executing raw SQL queries 
and a Python db api, or using Django's ORM?"   If so, usually the latter 
but this is definitely not a problem of poorly written queries, we've 
profiled them.

- Andy

On Saturday, 20 February 2021 at 17:44:45 UTC jmccla...@gmail.com wrote:

> are you using django directly to run the queries or are you querying the 
> db. I had a similar problem a while back with a project and the load was 
> resolved by letting python do the work.
>
>
>
> On Sat, 20 Feb 2021 at 15:23, Andy Robinson  wrote:
>
>> Hi all,
>>
>> We're maintaining an application that is hitting scaling problems.   It 
>> helps run grass-roots sporting events all over Europe, so Saturdays can 
>> produce large and unpredictable loads, with both logged-in users and public 
>> viewers. Despite a powerful server seemingly being lightly loaded (htop 
>> suggests just 3-6 out of24 cores typically busy, and free memory 
>> available), it often "hangs" for long periods - simple requests return 
>> after 10sec, and at times we have had to restart nginx.   
>>
>> If anyone here has experience of this configuration under high loads, 
>> we'd be keen to hire an experienced expert to review and check we are not 
>> doing anything stupid with our nginx or uwsgi configuration.  
>>
>> Likewise, if you really know how to cache Django with nginx, and work 
>> around the various cookies and language settings so that logged-in pages 
>> still work but anonymous ones can be cached, we'd love to get a bit of 
>> help  
>>
>> Architecture is ubuntu 18.04, Django 3.0.12, with MariaDB and Mongo back 
>> ends used for slightly different things.
>>
>> Feel free to contact me by private email if you,
>>
>> Otherwise, if people have suggestions and questions here, I'll try to 
>> share what I can and am very grateful for any help...
>>
>> Best Regards
>>
>>
>> Andy Robinson
>>  
>>
>> -- 
>>
> You received this message because you are subscribed to the Google Groups 
>> "Django users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-users...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-users/4fcc006e-37bb-4497-a692-45dfb99bfad1n%40googlegroups.com
>>  
>> 
>> .
>>
>
>
> -- 
> John McClain
>
> Cell: 085-1977-823
> Skype: jmcclain0129
> Email: jmccla...@gmail.com
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/1aa1c853-8028-4f58-8c40-bf966caa5e56n%40googlegroups.com.

Re: Django/uWSGI/nginx under load

[Andy Robinson]

Thanks for this.  27k file descriptors out of about 370k allowed.  I 
presume that means this isn't a problem?

(We run a lot of different virtual hosts on this box, each with a few 
Django processes)

- Andy
On Saturday, 20 February 2021 at 16:15:12 UTC mob...@aldian.net wrote:

> Check the number of open file descriptor when you experienced the slow 
> response.
>
> On Sat, Feb 20, 2021 at 10:23 PM Andy Robinson  
> wrote:
>
>> Hi all,
>>
>> We're maintaining an application that is hitting scaling problems.   It 
>> helps run grass-roots sporting events all over Europe, so Saturdays can 
>> produce large and unpredictable loads, with both logged-in users and public 
>> viewers. Despite a powerful server seemingly being lightly loaded (htop 
>> suggests just 3-6 out of24 cores typically busy, and free memory 
>> available), it often "hangs" for long periods - simple requests return 
>> after 10sec, and at times we have had to restart nginx.   
>>
>> If anyone here has experience of this configuration under high loads, 
>> we'd be keen to hire an experienced expert to review and check we are not 
>> doing anything stupid with our nginx or uwsgi configuration.  
>>
>> Likewise, if you really know how to cache Django with nginx, and work 
>> around the various cookies and language settings so that logged-in pages 
>> still work but anonymous ones can be cached, we'd love to get a bit of 
>> help  
>>
>> Architecture is ubuntu 18.04, Django 3.0.12, with MariaDB and Mongo back 
>> ends used for slightly different things.
>>
>> Feel free to contact me by private email if you,
>>
>> Otherwise, if people have suggestions and questions here, I'll try to 
>> share what I can and am very grateful for any help...
>>
>> Best Regards
>>
>>
>> Andy Robinson
>>  
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-users...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-users/4fcc006e-37bb-4497-a692-45dfb99bfad1n%40googlegroups.com
>>  
>> 
>> .
>>
>
>
> -- 
> Regards,
>
> Aldian Fazrihady
> http://aldianfazrihady.com
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/c2093676-90f5-4e69-b646-868102361d18n%40googlegroups.com.

Re: Django/uWSGI/nginx under load

[John McClain]

are you using django directly to run the queries or are you querying the
db. I had a similar problem a while back with a project and the load was
resolved by letting python do the work.



On Sat, 20 Feb 2021 at 15:23, Andy Robinson  wrote:

> Hi all,
>
> We're maintaining an application that is hitting scaling problems.   It
> helps run grass-roots sporting events all over Europe, so Saturdays can
> produce large and unpredictable loads, with both logged-in users and public
> viewers. Despite a powerful server seemingly being lightly loaded (htop
> suggests just 3-6 out of24 cores typically busy, and free memory
> available), it often "hangs" for long periods - simple requests return
> after 10sec, and at times we have had to restart nginx.
>
> If anyone here has experience of this configuration under high loads, we'd
> be keen to hire an experienced expert to review and check we are not doing
> anything stupid with our nginx or uwsgi configuration.
>
> Likewise, if you really know how to cache Django with nginx, and work
> around the various cookies and language settings so that logged-in pages
> still work but anonymous ones can be cached, we'd love to get a bit of
> help
>
> Architecture is ubuntu 18.04, Django 3.0.12, with MariaDB and Mongo back
> ends used for slightly different things.
>
> Feel free to contact me by private email if you,
>
> Otherwise, if people have suggestions and questions here, I'll try to
> share what I can and am very grateful for any help...
>
> Best Regards
>
>
> Andy Robinson
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/4fcc006e-37bb-4497-a692-45dfb99bfad1n%40googlegroups.com
> 
> .
>


-- 
John McClain

Cell: 085-1977-823
Skype: jmcclain0129
Email: jmcclain0...@gmail.com

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAN-hv_q%3DH4D4%3DcDr9GJWR5j2WU%3DkCwRW-xKaQGBdK25Ca1m13g%40mail.gmail.com.

Re: Django/uWSGI/nginx under load

[Aldian Fazrihady]

Check the number of open file descriptor when you experienced the slow
response.

On Sat, Feb 20, 2021 at 10:23 PM Andy Robinson  wrote:

> Hi all,
>
> We're maintaining an application that is hitting scaling problems.   It
> helps run grass-roots sporting events all over Europe, so Saturdays can
> produce large and unpredictable loads, with both logged-in users and public
> viewers. Despite a powerful server seemingly being lightly loaded (htop
> suggests just 3-6 out of24 cores typically busy, and free memory
> available), it often "hangs" for long periods - simple requests return
> after 10sec, and at times we have had to restart nginx.
>
> If anyone here has experience of this configuration under high loads, we'd
> be keen to hire an experienced expert to review and check we are not doing
> anything stupid with our nginx or uwsgi configuration.
>
> Likewise, if you really know how to cache Django with nginx, and work
> around the various cookies and language settings so that logged-in pages
> still work but anonymous ones can be cached, we'd love to get a bit of
> help
>
> Architecture is ubuntu 18.04, Django 3.0.12, with MariaDB and Mongo back
> ends used for slightly different things.
>
> Feel free to contact me by private email if you,
>
> Otherwise, if people have suggestions and questions here, I'll try to
> share what I can and am very grateful for any help...
>
> Best Regards
>
>
> Andy Robinson
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/4fcc006e-37bb-4497-a692-45dfb99bfad1n%40googlegroups.com
> 
> .
>


-- 
Regards,

Aldian Fazrihady
http://aldianfazrihady.com

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAN7EoAbNUw2UEUH4g6_xf%2B4HYrvK_Vrdj5QObUE6BFbEDLMHwQ%40mail.gmail.com.

Django/uWSGI/nginx under load

[Andy Robinson]

Hi all,

We're maintaining an application that is hitting scaling problems.   It 
helps run grass-roots sporting events all over Europe, so Saturdays can 
produce large and unpredictable loads, with both logged-in users and public 
viewers. Despite a powerful server seemingly being lightly loaded (htop 
suggests just 3-6 out of24 cores typically busy, and free memory 
available), it often "hangs" for long periods - simple requests return 
after 10sec, and at times we have had to restart nginx.   

If anyone here has experience of this configuration under high loads, we'd 
be keen to hire an experienced expert to review and check we are not doing 
anything stupid with our nginx or uwsgi configuration.  

Likewise, if you really know how to cache Django with nginx, and work 
around the various cookies and language settings so that logged-in pages 
still work but anonymous ones can be cached, we'd love to get a bit of 
help  

Architecture is ubuntu 18.04, Django 3.0.12, with MariaDB and Mongo back 
ends used for slightly different things.

Feel free to contact me by private email if you,

Otherwise, if people have suggestions and questions here, I'll try to share 
what I can and am very grateful for any help...

Best Regards


Andy Robinson
 

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/4fcc006e-37bb-4497-a692-45dfb99bfad1n%40googlegroups.com.