Re: Channels - unbalanced load for threaded workers

2017-05-18 Thread Andrew Godwin 
Yes. Running with no --threads option will give you the best spread, but
result in more RAM usage. Rather than relying on process statistics via
htop or similar, I'd suggest benchmarking actual throughput, as that's the
thing you want to optimise for at the end of the day; it might be that one
way uses less CPU but has higher throughput, for example.

Andrew

On Thu, May 18, 2017 at 12:36 AM,  wrote:

> Yes, that was my concern that only some processes are loaded. So running
> more 'runworker' commands with less '--threads' should give better results?
>
> W dniu czwartek, 18 maja 2017 02:18:56 UTC+2 użytkownik Andrew Godwin
> napisał:
>>
>> Hi - I am not sure what you are saying exactly. That the load only goes
>> to some processes? Python threads are strange, the OS may not show stats
>> correctly and I'd generally advise multiple processes instead.
>>
>> Andrew
>>
>> On Wed, May 17, 2017 at 1:53 AM,  wrote:
>>
>>> I am trying to deploy Django project with Channels. After some trial and
>>> error I settled on 4 workers with 16 threads each. However as attached
>>> screen shows, the processes that are spawned by supervisor have CPU usage
>>> at ~15% while threads are around ~1-2%. Number of connections at the moment
>>> is less than 100. I tried load testing this setup, and as more messages are
>>> coming in the CPU usage is increasing both on processes that were spawned
>>> by supervisor and threads of these processes - so there is still
>>> disproportion (~50% and ~10%). Is it normal behavior? Any tips to configure
>>> it some other way to better balance the load?
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Django users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to django-users...@googlegroups.com.
>>> To post to this group, send email to django...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/django-users.
>>> To view this discussion on the web visit https://groups.google.com/d/ms
>>> gid/django-users/b566a9bc-5313-4573-9351-d6dd81915251%40googlegroups.com
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-users/b31841c7-ea17-4189-a9b8-f6f0e0d7dc79%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAFwN1ur21eNF%2BJh6TdhWo75YsULpWsSVV5Bxz2DYbLMZzYB%2B3w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Channels - unbalanced load for threaded workers

2017-05-18 Thread biwanczuk 
Yes, that was my concern that only some processes are loaded. So running 
more 'runworker' commands with less '--threads' should give better results?

W dniu czwartek, 18 maja 2017 02:18:56 UTC+2 użytkownik Andrew Godwin 
napisał:
>
> Hi - I am not sure what you are saying exactly. That the load only goes to 
> some processes? Python threads are strange, the OS may not show stats 
> correctly and I'd generally advise multiple processes instead.
>
> Andrew
>
> On Wed, May 17, 2017 at 1:53 AM, > wrote:
>
>> I am trying to deploy Django project with Channels. After some trial and 
>> error I settled on 4 workers with 16 threads each. However as attached 
>> screen shows, the processes that are spawned by supervisor have CPU usage 
>> at ~15% while threads are around ~1-2%. Number of connections at the moment 
>> is less than 100. I tried load testing this setup, and as more messages are 
>> coming in the CPU usage is increasing both on processes that were spawned 
>> by supervisor and threads of these processes - so there is still 
>> disproportion (~50% and ~10%). Is it normal behavior? Any tips to configure 
>> it some other way to better balance the load? 
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-users...@googlegroups.com .
>> To post to this group, send email to django...@googlegroups.com 
>> .
>> Visit this group at https://groups.google.com/group/django-users.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-users/b566a9bc-5313-4573-9351-d6dd81915251%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/b31841c7-ea17-4189-a9b8-f6f0e0d7dc79%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Channels - unbalanced load for threaded workers

2017-05-17 Thread Stephen J. Butler 
CPython uses a global interpreter lock, which means that only one thread
per process can ever be running Python code. Where threading really helps
is if your python project is mostly I/O bound (waiting on database work,
network connections, etc). If it's CPU bound then you probably won't see
much concurrency with multiple threads.

I'm guessing this is why you're seeing all your work being accounted to a
couple processes. You'd get better concurrency moving to a
multiprocess/fork model.

https://www.ploggingdev.com/2017/01/multiprocessing-and-multithreading-in-python-3/


On Wed, May 17, 2017 at 3:53 AM,  wrote:

> I am trying to deploy Django project with Channels. After some trial and
> error I settled on 4 workers with 16 threads each. However as attached
> screen shows, the processes that are spawned by supervisor have CPU usage
> at ~15% while threads are around ~1-2%. Number of connections at the moment
> is less than 100. I tried load testing this setup, and as more messages are
> coming in the CPU usage is increasing both on processes that were spawned
> by supervisor and threads of these processes - so there is still
> disproportion (~50% and ~10%). Is it normal behavior? Any tips to configure
> it some other way to better balance the load?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-users/b566a9bc-5313-4573-9351-d6dd81915251%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAD4ANxXhDp3TfhoWVkhhJFT_%3DZ08bHxeQoBP2S7sSOCGbxhYew%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Channels - unbalanced load for threaded workers

2017-05-17 Thread Andrew Godwin 
Hi - I am not sure what you are saying exactly. That the load only goes to
some processes? Python threads are strange, the OS may not show stats
correctly and I'd generally advise multiple processes instead.

Andrew

On Wed, May 17, 2017 at 1:53 AM,  wrote:

> I am trying to deploy Django project with Channels. After some trial and
> error I settled on 4 workers with 16 threads each. However as attached
> screen shows, the processes that are spawned by supervisor have CPU usage
> at ~15% while threads are around ~1-2%. Number of connections at the moment
> is less than 100. I tried load testing this setup, and as more messages are
> coming in the CPU usage is increasing both on processes that were spawned
> by supervisor and threads of these processes - so there is still
> disproportion (~50% and ~10%). Is it normal behavior? Any tips to configure
> it some other way to better balance the load?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-users/b566a9bc-5313-4573-9351-d6dd81915251%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAFwN1uoRQydaH8aC5bx%3DQReZT0JQVVBB5pHt1HMxbiZERxSHrA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.